akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Log_softmax vs NLLLoss

Gaurav_Pant: <p>Is -log_softmax the same as NLLLoss applied on log_softmax, and if so why is a separate function required ?</p>

ptrblck: <p>No, you won’t get the same result.<br>
<code>nn.NLLLoss</code> calculates the loss value by reducing the log probability for each sample using the target index.<br>
<code>log_softmax</code> will just calculate the log probabilities for the complete tensor in the specified dimension.<br>
The reduction type can be specified and is the <code>mean</code> by default.<br>
So in the default use case, <code>nn.NLLLoss</code> will return a single loss value, while the output tensor of <code>log_softmax</code> will have the same shape as the input tensor.</p>
How to speed up data loading from disc?

hadaev8: <p>So i need 5 sizes of every picture.<br>
I preprocess and save everything with numpy, but loading files from disc (probably) take most time (much more then model/tensor processing).</p>
<p>My dataloader looks like this:</p>
<pre><code>class GanDataset(Dataset):
	def __init__(self, samples):
		self.samples = samples

	def __getitem__(self, index):
		sample = self.samples[index]
		return sample
	
	def __len__(self):
		return len(self.samples)

class GanCollate():
	def __init__(self, sizes):
		self.sizes = sizes
	def __call__(self, batch):
		output = []            
		for size in self.sizes:
			output.append(torch.stack([torch.from_numpy(np.load(os.path.join(str(size), i) + '.npy')) for i in batch]))
		return output

dataloader = DataLoader(GanDataset(pics), batch_size=batch_size,shuffle=True,
						num_workers=workers, pin_memory=False, drop_last=True, collate_fn=GanCollate(sizes))
</code></pre>
<p>Full code<br>
<a href="https://colab.research.google.com/drive/1yWkyqyzE0chTs2ciBcLn_l2T3MXDEjaP" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1yWkyqyzE0chTs2ciBcLn_l2T3MXDEjaP</a></p>
Appending different 1D tensors to 3rd dimension of 3D tensor

Dylan_Yung: <blockquote>
<p>def to_rnn_embed_tensor(tensor: Tensor, cond: list, letter_count=LETTERS_COUNT, cond_to_tensor_func = embed_line_to_tensor) → list:<br>
input_sz, batch_sz = tensor.shape<br>
ret = torch.zeros(input_sz, batch_sz, letter_count)<br>
for i in range(input_sz):<br>
for j in range(batch_sz):<br>
cond_j = cond[j]<br>
cond_tensor = cond_to_tensor_func(cond_j)<br>
ret[i, j, int(tensor[i, j])] = 1<br>
return ret</p>
</blockquote>
<p>So I have a list of strings that get converted to tensors that are 1x7 that I’m trying to append to third dimension for a tensor that’s going into a RNN. So ret should be input_sz x batch_sz x letter_count + 7. The issue is that the tensor that gets appended to the third dimension at j should correlate to the jth index in cond list.</p>
<p>Essentially cond_tensor should be appended to ret[i,j] for all j</p>
Polar coordinates transformation layer

mlearner: <p>Hello,<br>
in the <code>forward</code> function of a neural network I would like, in the propagation between two layers, to insert a polar coordinate conversion (the output of the first layer is intended to be a set of 2D Cartesian coordinates).<br>
How can I achieve that? Can I put it directly in the forward function or is it needed to create a new layer to do that? How does the backpropagation deal with that?<br>
Sorry for the dumb question, but I am quite new to PyTorch.<br>
Thanks in advance</p>

albanD: <p>Hi,</p>
<p>If you use a function that only use pytorch Tensors and pytorch functions, No need for a new layer. You can use the following in your forward pass (make changes for batch-size if you need):</p>
<pre><code class="lang-python">def to_polar(x, y):
  return (x**2 + y**2).sqrt(), torch.atan(y/x)
</code></pre>

mlearner: <p>Thank you for the answer!<br>
What if I have a <code>[21, 2]</code> size tensor in which the first column is <code>x</code> and the second <code>y</code>?<br>
What is an efficient way to make that conversion?</p>

albanD: <p>Then you need to replace the addition with a <code>.sum(-1)</code> and the division by <code>t.select(-1, -1)/t.select(-1, 0)</code>.<br>
If you want a single Tensor for the output, you can use <code>torch.stack([r, theta], -1)</code> to get an output of size [21, 2].</p>

mlearner: <p>Perfect, I implemented it (your second strategy) and it was working fine as soon as I was having only one batch.</p>
<p>The problem is that I actually have a <code>[BATCH_SIZE, 42]</code> (intermediate) input and I was reshaping it as <code>[21,2]</code> when the batch was <code>1</code> without any problem, simply using something like <code>h.view(21, 2)</code>.<br>
How to deal with that when <code>BATCH_SIZE</code> is different from 1? How should I reshape it to fit the function <code>to_polar</code>?</p>
<p>Sorry for the dumb question, but I am not very familiar with these sizing issues.</p>

albanD: <p>Hi,</p>
<p>You can make it into a 3D Tensor of size [batch, 21, 2] and use the same function as above. Since we index with -1 (last dimension), it does not really matter how many there are before.<br>
Also if the input is always of size 42, you can do <code>.view(-1, 21, 2)</code>.</p>
Error while downloading pre-trained GAN model in Pytorch : 'memory' file not found

SurajSubramanian: <p>I was following the steps given in <a href="https://modelzoo.co/model/pytorch-cyclegan-and-pix2pix" rel="nofollow noopener">https://modelzoo.co/model/pytorch-cyclegan-and-pix2pix</a> to download a pre-trained model.</p>
<p>These where the first 3 commands given there :</p>
<pre><code class="lang-auto">git clone https://github.com/pytorch/vision
cd vision
**python setup.py install**
</code></pre>
<p>However when i ran the third line, i got an error :</p>
<p>fatal error: ‘memory’ file not found</p>
<p><code>#include &lt;memory&gt;</code></p>
<p>error: command ‘gcc’ failed with exit status 1</p>
<p>If anyone has some idea on how to overcome this error, it would be really helpful.</p>
<p>( I use a Mac OS, you can refer ptrblck’s answer if you use a Linux OS )</p>
<p>Thanks in advance !</p>

ptrblck: <p>Could you check, if your libstdc++ is properly installed?<br>
You should find this file in <code>/usr/include/c++/&lt;version&gt;/memory</code>.</p>
<p>In case you can’t find it, you might need to install it via:</p>
<pre><code class="lang-auto">apt-get install build-essential
</code></pre>

SurajSubramanian: <p>I use a Mac OS, so this command doesn’t work for me !<br>
I cant check the path too<br>
Thanks a lot for your response !</p>

SurajSubramanian: <p>Do you know the equivalent command for a Mac</p>

ptrblck: <p>Unfortunately, I’m not familiar with macOS and don’t know how these headers are installed.<br>
A quick search seems to point to <code>xcode</code> or <code>brew</code>, but again I couldn’t decipher how it should work. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

SurajSubramanian: <p>I will try to figure it out then<br>
Thanks a lot, really appreciate it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Extra dimension in data loader?

f3ba: <p>I am using the following code to load the MNIST dataset:</p>
<pre><code class="lang-auto">batch_size = 64
train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor()
                   ])),
    batch_size=batch_size)
</code></pre>
<p>If I try to load one batch:</p>
<pre><code class="lang-auto">for data, target in train_loader:
    print(data.shape)
    break
</code></pre>
<p>this prints:</p>
<pre><code class="lang-auto">torch.Size([64, 1, 28, 28])
</code></pre>
<p>What I don’t understand is why the size is <code>64 x 1 x 28 x 28</code> instead of just <code>64 x 28 x 28</code>? What is this extra dimension of length 1?</p>

ptrblck: <p><code>dim1</code> represents the channel dimension. Since <code>MNIST</code> uses grayscale images, the channel dim has a value of 1.<br>
<code>nn.Conv2d</code> and other 2-dimensional layers expect an input of <code>[batch_size, channels, height, width]</code>.</p>
Multiply torch tensor along a dimension

jwillette: <p>lets say I have a tensor <code>t</code> with the dimensions <code>(b, m, n)</code> and I have a vector <code>v</code> of size <code>(b)</code>. I want to multiply these together and have each <code>(m, n)</code> entry of <code>t</code> multiplied by the corresponding b_i scalar value in the vector <code>v</code>.</p>
<p>Is there a way to do this with normal pytorch operations?</p>

G.M: <p>Try this:</p>
<pre><code class="lang-auto">v = v.view(-1, 1, 1) # same number of dimension as *t*
ans = v * t
</code></pre>
Apply LSTM in inner dimensions

howard222: <p>Normally, we have data like</p>
<pre><code class="lang-auto">inputs = [
  ['how', 'are', 'you', '?'],
  ['I', 'am', 'good', '.'],
  ...
]
</code></pre>
<p>the dimension is (batch size, seq len)<br>
So, we can apply LSTM directly</p>
<pre><code class="lang-auto">self.lstm = nn.LSTM()
self.lstm(inputs)
</code></pre>
<p>However, I have data like</p>
<pre><code class="lang-auto">inputs = [
  [
    ['U.S.', 'president'], ['is', 'not'], ['clinton', 'hillary']
  ],
  [
    ['The', 'boss'], ['likes'], ['apple', 'pies'],
  ],
  ...
]
</code></pre>
<p>I’d like to apply LSTM in the most inner dimension such as [‘U.S.’, ‘president’] and [‘is’, ‘not’], and so on.<br>
And then, I’ll apply attention for <code>[ output of LSTM(['U.S.', 'president']), output of LSTM(['is', 'not']), output of LSTM(['clinton', 'hillary']) ]</code>.</p>
<p>My question is that how to apply LSTM for the inner dimensions.</p>
<p>Please help!</p>

G.M: <p>Perhaps using <code>transpose</code> and <code>view</code> to change the dimensions?</p>

howard222: <p>Thank you!<br>
I’ll try it!</p>
What is the dilation in the convolutional layer output size formula?

jwillette: <p>I have been searching around and I cannot find any easy answers to how to dynamically calculate the output size of a set of convolutional layers. I see the formula <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d" rel="nofollow noopener">here</a> and most of the terms are obvious except for the dilation term.</p>
<p>With <code>stride=1</code>, <code>kernel=3</code></p>
<p>I am using omniglot</p>
<ul>
<li>inputs: (batch, 1, 28, 28)</li>
<li>outputs: (batch, 64, 1, 1)</li>
<li>dialtion: 14.5???</li>
</ul>
<p>and imagenet</p>
<ul>
<li>inputs: (batch, 3, 84, 84)</li>
<li>outputs: (batch, 64, 5, 5)</li>
<li>dilation: 40.5???</li>
</ul>
<p>These dont seem to make sense</p>

albanD: <p>Hi,</p>
<p>This is usually <code>dilation=1</code> for most models. Where did you get these numbers from?</p>
<p>Also <a href="https://towardsdatascience.com/review-dilated-convolution-semantic-segmentation-9d5a5bd768f5">this</a> blogpost has a nice visualization of what this parameter is doing.</p>

jwillette: <p>I wasn’t used to working with convolutions. I forgot that the max pooling layer was what was causing most of the sizing issues and not the conv2d itself.</p>
BUG: inception_v3 training error

mingruimingrui: <p>Hi inference using the inception_v3 model from torchvision seems to be working in an unintended manner.<br>
Judging by the error log, it seems that the error has to do with how batch norm is implemented</p>
<p>I’m using<br>
python 3.5<br>
pytorch 0.4.1<br>
torchvision 0.2.1</p>
<pre><code class="lang-auto">import torch
import torchvision

model = torchvision.models.inception_v3()

inputs = torch.Tensor(1, 3, 299, 299)
outputs = model(inputs)
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "scrapbook.py", line 7, in &lt;module&gt;
    outputs = model(inputs)
  File "/data/mingrui/anaconda3/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/mingrui/anaconda3/lib/python3.5/site-packages/torchvision-0.2.1-py3.5.egg/torchvision/models/inception.py", line 109, in forward
  File "/data/mingrui/anaconda3/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/mingrui/anaconda3/lib/python3.5/site-packages/torchvision-0.2.1-py3.5.egg/torchvision/models/inception.py", line 308, in forward
  File "/data/mingrui/anaconda3/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/mingrui/anaconda3/lib/python3.5/site-packages/torchvision-0.2.1-py3.5.egg/torchvision/models/inception.py", line 326, in forward
  File "/data/mingrui/anaconda3/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/mingrui/anaconda3/lib/python3.5/site-packages/torch/nn/modules/batchnorm.py", line 66, in forward
    exponential_average_factor, self.eps)
  File "/data/mingrui/anaconda3/lib/python3.5/site-packages/torch/nn/functional.py", line 1251, in batch_norm
    raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
ValueError: Expected more than 1 value per channel when training, got input size [1, 768, 1, 1]
</code></pre>

mingruimingrui: <p>My bad, there isn’t actually any bugs</p>
<pre><code class="lang-auto">import torch
import torchvision

model = torchvision.models.inception_v3()

inputs = torch.Tensor(2, 3, 299, 299)
outputs = model(inputs)
</code></pre>
<p>The batch size I tested with is just 1.</p>

zarzen: <p>why the batch-size cannot be 1?<br>
I think it is still a bug?</p>

ptrblck: <p>During inference it can be <code>1</code>, during training and it cannot as the batch norm layers cannot compute the stats from a single sample.</p>
nn.LogSoftmax() equation

oasjd7: <p>As far as I know, <code>CrossEntropy()</code> equals to<br>
<code>torch.mean(torch.sum(-target * nn.LogSoftmax(input), dim=1))</code></p>
<p>How can I express <code>nn.LogSoftmax</code> in equation?</p>
<p><code>nn.LogSoftmax</code>  equals to <code>torch.mean(torch.sum(F.softmax(input)*torch.log(F.softmax(input)),1)</code> ?</p>

Eta_C: <pre><code class="lang-python">import torch
import torch.nn as nn
import torch.nn.functional as F


x = torch.rand(4, 5) * 100
y_1 = nn.LogSoftmax(dim=-1)(x)
y_2 = torch.log(F.softmax(x, dim=-1))

print((y_1 - y_2).abs().max())  # 3e-8
</code></pre>
<p>See <a href="https://pytorch.org/docs/stable/nn.html?highlight=logsoftmax#torch.nn.LogSoftmax" rel="nofollow noopener">here</a>.</p>
Training independent models simultaneously

sawal86: <p>Is it possible to train multiple models simultaneously?<br>
For instance, suppose by nettwork class is Net.</p>
<p>net1 = Net()<br>
net2 = Net()</p>
<p>Is it possible to train net 1 and net 2 simultaneously?</p>
<p>Thanks.</p>

mrshenli: <p>I assume you need to do this because you want to use different training data for the models? In that case, yes, that should be possible. I think you can wrap them with a wrapper module, sth like:</p>
<pre><code class="lang-python">class WrapperModule(nn.Module):
    def __init__(self):
        super(WrapperModule, self).__init__()
        self.net0 = Net()
        self.net1 = Net()

    def forward(inputs):
        return [self.net0(inputs[0]), self.net1(inputs[1])]

net = WrapperModule()
opt = SomeOptimizer(net.parameters())
ddp = DistributedDataParallel(net)
ddp.forward(inputs).backward()
opt.step()
</code></pre>

mrshenli: <p>Sorry, just realized you didn’t mention DistributedDataParallel in the question. Is this for distributed training? Could you please provide more contexts?</p>

TinfoilHat0: <p>You can pass each of your model to a different GPU. See <a href="https://discuss.pytorch.org/t/how-to-deploy-different-scripts-on-different-gpus/61011/3" class="inline-onebox">How to deploy different scripts on different GPUs?</a></p>
PyTorch 1.4 / torchvision 0.5 vs Detectron

chandrachud: <p>Hi guys,</p>
<p>I just noticed that with the latest <a href="https://pytorch.org/blog/pytorch-1-dot-4-released-and-domain-libraries-updated/" rel="nofollow noopener">PyTorch 1.4</a> release:  <a href="https://github.com/pytorch/vision/releases" rel="nofollow noopener">torchvision 0.5</a> provides better deployment support in terms of exporting to ONNX and quantisation. What’s the timeline for the same features on detectron2?</p>
<p>Also, there seems to be significant overlap between torchvision and detectron. What’s the vision for the two? Do you recommend one over the other for prototyping and deployment?</p>
<p>Thanks</p>
PyTorch+Windows: is parallelization over multiple GPUs now possible?

SmoothPQ: <p>Hi,</p>
<p>I am planning to add a new GPU on my computer. Using Pytorch on Windows, I wonder if it will be possible for me to use parallelism.</p>
<p>In 2017 it wasn’t possible(<a href="https://github.com/pytorch/pytorch/issues/4391" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/4391</a>).<br>
Has it changed? Will I have to switch to Linux? And is there a guide of how to install Pytorch for DataParallelism?</p>
<p>Thank you for all the library!</p>

rvarm1: <p>Hi, I am not aware of a NCCL binary from nvidia that supports windows, so parallelization over multiple GPUs on windows is still not possible.</p>

SmoothPQ: <p>Thank you very much for the answer. Is nccl installed automatically when installing CUDA on Linux, or do I need to add something else?</p>

MLnut: <p>Im also on a Windows system.  I was able to use dataparallel on my model without any apparent errors.  However, the performance was actually worse;  which makes me think that it’s not actually using multiple gpus.  Why am I able to use multiple gpus in tensorflow on a windows system, but not pytorch?  There must be some hack to get be able to do this.</p>

mrshenli: <aside class="quote no-group" data-username="SmoothPQ" data-post="3" data-topic="59971" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/7ba0ec/48.png" class="avatar"> SmoothPQ:</div>
<blockquote>
<p>Thank you very much for the answer. Is nccl installed automatically when installing CUDA on Linux, or do I need to add something else?</p>
</blockquote>
</aside>
<p>On Linux, NCCL and torch.distributed will be enabled by default. On MacOs, with PyTorch 1.3.1+, you need to conda install libuv and pkg-config explicitly set USE_DISTRIBUTED=1 when compiling from source. For Windows, torch.distributed is not enabled yet.</p>

mrshenli: <blockquote>
<p>I was able to use dataparallel on my model without any apparent errors. However, the performance was actually worse; which makes me think that it’s not actually using multiple gpus.</p>
</blockquote>
<p>DataParallel is single-process-multi-thread data parallelism, and it replicates the input module in every forward pass, which is expected to be slow, but this is a very convenient entry point for enabling parallelism. You don’t need to do anything to enable that, and it should work fine if the batch size is large enough (to shadow the model replicating overhead)</p>
<blockquote>
<p>Why am I able to use multiple gpus in tensorflow on a windows system</p>
</blockquote>
<p>We are working on using libuv to enable that, as <a class="mention" href="/u/pietern">@pietern</a> did for Windows, but timeline is TBD</p>
DistributedDataParallel broadcast_buffers

dashesy: <p>I find this in the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/parallel/distributed.html" rel="noopener nofollow ugc">documentation</a>:</p>
<blockquote>
<pre><code>     broadcast_buffers: flag that enables syncing (broadcasting) buffers of
                      the module at beginning of the forward function.
                      (default: True)
</code></pre>
</blockquote>
<p>But what exactly does that mean? If I set it to False, from code it looks like the gradients are still reduced together, which should result in the same buffers on all the replicas on all the nodes? Please let me know if I am wrong.</p>

Ali2500: <p>What you’re saying is true for learnable parameters, but some modules like BatchNorm keep track of statistics (e.g. mean, variance) of the tensors that pass through them in buffers. In a multi-gpu setup, different GPUs will receive different inputs and so these statistics will be different. It is therefore necessary to synchronize them (which is what the <code>broadcast_buffers</code> flag does).</p>
<p>My guess is that this incurs a slight performance overhead, so if you’re not using such modules in your model you can disable it.</p>

dashesy: <p>I think that documentation should mention exactly this information. That BN variance is also a buffer may not be obvious at first.</p>
Use torch.nonzero() as index

dashesy: <p>Here is a numpy example to use <code>nonzero</code>.</p>
<pre><code class="lang-python">    a = np.random.randint(6,size=(4,5,3))
    idx = np.nonzero(a)
    a[idx] = 0
</code></pre>
<p>This is PyTorch</p>
<pre><code class="lang-python">    a = torch.randint(6,size=(4,5,3))
    idx = torch.nonzero(a)  # idx = a.nonzero()
    # a[idx] = 0  # this throws an error because `nonzero` cannot be used as index
    a[(idx[:, 0], idx[:, 1], idx[:, 2])] = 0  # this is a little counter-intuitive, can we instead accept multi-dim tensors as index?

</code></pre>
<p>I actually like <code>torch.nonzero</code>'s single tensor better than the tuple that <code>numpy</code> returns, it is more elegant. But I do not know of a better way to use it as index</p>

tom: <p>Maybe it is a bit unfortunate that nonzero has the dimensions in that order… <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=6" title=":confused:" class="emoji" alt=":confused:"><br>
You can make the split explicit and have something that is close to NumPy’s nonzero indexing:</p>
<pre><code class="lang-python">a = torch.threshold(torch.randn(5, 5, 5), 1, 0)
idx = torch.nonzero(a).split(1, dim=1)
a[idx]
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

dashesy: <p>Thanks! <code>.split</code> makes it easier. As a suggestion if PyTorch tensors could also be indexed by another tensor (of coordinates) that could be even more intuitive than numpy.</p>

Zhixin_Ma: <p>You just need to set as_tuple=True:</p>
<blockquote>
<p>a[ a.nonzero(as_tuple=True) ]</p>
</blockquote>

dashesy: <p>Hurray, this is now available!</p>
How to reshape list?

oasjd7: <p>Hi,all</p>
<p>How to reshape list from<br>
[64,256,16,16] (list length : 1) → [256,16,16] * 64 (list length) ?</p>

ptrblck: <p>Could you explain your use case a bit more, please?<br>
Is the first list a shape of a tensor or a Python list containing 4 values (which would then have a length of 4)?</p>

oasjd7: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I have features which have [64,256,16,16] shape.(Batch_size, channel, width, height)<br>
and I want to append this features to the empty list.<br>
<code>list_a.append(features)</code><br>
then list_a have only one length. (whole features included in list_a[0])</p>
<p>But I want to make this list have 64 length and every element of this list have [256,16,16] features.</p>

ptrblck: <p>In that case, you could <code>split</code> the tensor:</p>
<pre><code class="lang-python">x = torch.randn([64, 256, 16, 16])
x_list = x.split(1, 0)
print(len(x_list))
&gt; 64
print(x_list[0].shape)
&gt; torch.Size([1, 256, 16, 16])
</code></pre>

oasjd7: <p>Thanks a lot ! It works well.<br>
I have one more question.<br>
I want to use data_loader with this list, then where to append the labels (shape:[64]) for this list?<br>
<code>x_data_loader = torchdata.DataLoader(x_list, batch_size=...)</code></p>

ptrblck: <p>You could pass the input tensor directly with the target tensor to a <code>TensorDataset</code> and pass it to a <code>DataLoader</code> without splitting:</p>
<pre><code class="lang-python">x = torch.randn(64, 256, 16, 16)
y = torch.randint(0, 10, (64,))
dataset = TensorDataset(x, y)
loader = DataLoader(
    dataset,
    batch_size=2,
    num_workers=2,
    shuffle=True
)
</code></pre>

oasjd7: <p>Actually, I have many [64,256,16,16] features, so I want to append all features to one list and then use it for data_loader. (Maybe this approach is inefficient.)</p>
<pre><code class="lang-auto">whole_list = []
for step in data_loader:
   ...
   x # shape: [64,256,16,16]
   y # shape : [64]
   ...
   whole_list.append(x_list, labels) # It failed. How to append labels?

new_data_loader = torchdata.DataLoader(x_list, batch_size=...)


</code></pre>

ptrblck: <p>Could you explain your data shape a bit?<br>
If you are concatenating these <code>[64, 256, 16, 16]</code> features, you would end up with a tensor of <code>[N, 64, 256, 16, 16]</code>.<br>
Would dim0 in this case refer to the batch size?<br>
If so, you could call <code>whole_list = torch.stack(whole_list)</code>, and pass it to a <code>TensorDataset</code> as shown in my example.</p>
Cannot quantize nn.Conv2d with dynamic Quantization

babak_hss: <p>I’ve read the pytorch <a href="https://pytorch.org/docs/stable/quantization.html" rel="nofollow noopener">quantization document</a> , and I think it should quantize nn.Conv2d module as well as nn.Linear. But, when I try the dynamic quantization, it only converts the nn.Linear.</p>
<p>I used the following simple dummy test:</p>
<pre><code class="lang-auto">class dumy_CNN(nn.Module):
    def __init__(self, ni, no):
        super().__init__()              
        self.conv = nn.Conv2d(ni, no, 8, 2, 3)
        self.lin = nn.Linear(1024, 4)
    def forward(self, x):
        out = self.lin(self.conv(x))
        return out    

model_test = dumy_CNN(2,10)
model_qn = torch.quantization.quantize_dynamic(
        model_test, {nn.Linear, nn.Conv2d} , dtype= torch.qint8
        )
</code></pre>
<p>But the model_qn looks like:</p>
<pre><code class="lang-auto">dumy_CNN(
 (conv): Conv2d(2, 10, kernel_size=(8, 8), stride=(2, 2), padding=(3, 3))
  (lin): DynamicQuantizedLinear(in_features=1024, out_features=4, scale=1.0, zero_point=0)
)
</code></pre>
<p>I also checked the weights to make sure about the above issue:</p>
<pre><code class="lang-auto">model_qn.conv.weight.data[0,0,0,0].item()
0.02230245992541313
</code></pre>

raghuramank100: <p>Hi <a class="mention" href="/u/babak_hss">@babak_hss</a>, Dynamic quantization is currently supported only for nn.Linear and nn.LSTM, please see: <a href="https://pytorch.org/docs/stable/quantization.html#torch.quantization.quantize_dynamic" rel="nofollow noopener">https://pytorch.org/docs/stable/quantization.html#torch.quantization.quantize_dynamic</a></p>
How to take gradient with respect to activation values using torch.autograd.grad?

gebrahimi: <p>Hi,</p>
<p>I am new to pytorch and register hook and I am wondering how i can get the gradient with respect to intermediate values when i calculate hessian vector product using autograd:</p>
<pre><code> grad_grad = torch.autograd.grad(grad_vec, self.model.parameters(), grad_outputs=vec, retain_graph=True)
</code></pre>
<p>where grad_vec shows gradient and vec is the vector i want to multiply by hessian.</p>
C++ DCGAN Tutorial, 'transposed' error

Maziar: <p>Hi,</p>
<p>Compiling the <em>DCGAN</em> tutorial code: <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_frontend.html</a>, I receive this error:</p>
<pre><code class="lang-auto">.../pytorch/examples/cpp/dcgan/dcgan.cpp:48:23: error: ‘struct torch::nn::ConvOptions&lt;2&gt;’ has no member named ‘transposed’ 48 | .transposed(true)), |
</code></pre>
<p>Any idea how to handle this?</p>

ptrblck: <p>If you are using <code>1.4.0</code>, you would have to change the code to <code>torch:nn:ConvTranspose{1,2,3}d</code>.<br>
From the release notes:</p>
<blockquote>
<p>If users have <code>transposed</code> originally set to <code>true</code> in <code>torch::nn::Conv{1,2,3}dOptions</code> , they should migrate their code to use <code>torch::nn::ConvTranspose{1,2,3}d</code> layers instead.</p>
</blockquote>
<p>The tutorial should be updated.<br>
CC <a class="mention" href="/u/yf225">@yf225</a> Do you know, whom to ping?</p>

yf225: <p>I just updated <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_frontend.html</a> and the accompanying example code to fix the error. Please feel free to try it again <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Composition of convolution kernels with unfold and fold

cfinlay: <p>Hi</p>
<p>I’m working on a project that involves composing two kernels, but where the first kernel is pixel dependent (say given by a MLP). This is related to this <a href="https://discuss.pytorch.org/t/convolution-that-takes-a-function-as-kernel/25588">post</a>, but not exactly.</p>
<p>To implement this, I’m thinking I’ll need nn.functional’s fold and unfold functions. To wrap my head around these functions, and as an intermediate step to coding the above, I would like to know: is it possible to write the composition of two kernels explicitly with unfold-matmul-fold?</p>
<p>For example:</p>
<pre><code class="lang-auto">k1 = torch.randn(5,2,3,3)
k2 = torch.randn(4,5,3,3)

# How to compute K = k2*k1 ?
# K should have shape (4,2,5,5)
</code></pre>
<p>Supposing we did have K explicitly, then we could double check it is correct with</p>
<pre><code class="lang-auto">x = torch.randn(10,2,8,8)
ycorrect = F.conv2d(F.conv2d(x,k1, padding=1), k2, padding=1)
ypredicted = F.conv2d(x, K, padding=2)
err = (ycorrect-ypredicted).abs().max()
print('Error: %.3g'%err)
</code></pre>
<p>There is a horribly slow way to compute the composed kernel K using linear indexing, but I’d rather avoid that approach. Maybe there is another way to get K using only F.conv2d, but I haven’t figured it out…</p>
<p>Any help would be greatly appreciated!</p>
How to set requires_grad = false in C++?

cruzas: <p>Hello there!</p>
<p>I was wondering if it’s possible to set requires_grad = false for a model’s parameters in C++?</p>
<p>For instance, if I understood correctly, in Python this would be done as follows:</p>
<pre><code class="lang-auto">for param in model.parameters():
    param.requires_grad = False
</code></pre>
<p>What is the equivalent of this in C++?</p>
<p>Thank you very much.</p>

Oktai15: <p>Just add the following line in your code:</p>
<pre><code class="lang-auto">torch::NoGradGuard no_grad;
</code></pre>

cruzas: <p>Hi Oktai15!</p>
<p>Thank you for your reply!</p>
<p>If I understood correctly, the NoGradGuard should, in essence, emulate the behaviour of requires_grad=false, correct?</p>
<p>If so, no parameters should change if I have the following, right? I.e., I should see the same bias before and after?</p>
<pre><code class="lang-auto">std::cout &lt;&lt; "Biases before:\n" &lt;&lt; policy-&gt;affine2-&gt;bias.data() &lt;&lt; std::endl;

{
   torch::NoGradGuard no_grad;
   loss.backward();
   optimizer.step();
}

std::cout &lt;&lt; "Biases after:\n" &lt;&lt; policy-&gt;affine2-&gt;bias.data() &lt;&lt; std::endl;
</code></pre>
<p>It turns out it still changed the biases, do you know why?</p>

Oktai15: <p>Hmm, try to add this line to the same scope with model and loss</p>
Choose backends

junonia: <p>It is not obvious to me how to select different backends.  In torch, you can mix layers from cudnn and cunn easily.  Is there an easy way to do the same in pytorch?</p>

apaszke: <p>Right now there’s no way to pick which layers will use cuDNN and which not. It will be automatically picked, based on what’s the fastest option. There’s only a global switch <code>torch.backends.cudnn.enabled</code> that you can set to <code>False</code> to globally disable cuDNN.</p>
<p>Just out of curiosity, what’s the reason why you need to select a particular backend?</p>

junonia: <p>Thanks for the information.  Sometimes I extract extra information from the layers.  For example, for maxpooling layer,  I have cases that make use of the indices in the cunn implementation.  The cudnn implementation does not have the indices variable available.</p>
<p>So in cases there is no corresponding cudnn implementation, cunn option will be selected for that layer?</p>

apaszke: <p>Yes, if you request the pooling function/module to return indices and cuDNN doesn’t support it, it will be automatically ignored. We’re always picking the fastest backend that supports all specified options.</p>

junonia: <p>Just to clarify,  if I have a network with some layers supported by cudnn, but if the net contains a layer that cudnn does not support (e.g.  FractionalMaxPooling),  will the entire net be run using cunn or just the FractionalMaxPooling layer?</p>

apaszke: <p>The decision is made on a per-operation basis. When we have a choice of multiple backends for a given op, we filter out these that don’t support the options you chose and pick the fastest one.</p>
<p>For example cuDNN 5 doesn’t support dilated convolutions. If your model has both dilated and regular convolutions, THCUNN will be used for dilated and cuDNN for regular ones.</p>
<p>This has the upside of always running at the max speed, across many different systems, that have only a subset of backends (e.g. no CUDA support), based on a single model definition.</p>

supakjk: <p>In my case, since I’d like to get deterministic results, I have to set <code>torch.backends.cudnn.enabled = False</code>, which shows much slow performance than using CUDNN.</p>
<p>I’d like to use CUDNN libraries for deterministic modules but not for non-deterministic modules such as CNN libraries. If I can choose backends for each module, I would be able to get some degree of speedup while keeping determinic results.</p>
<p>You don’t really have a plan to allow manually choosing backends?</p>

apaszke: <p>We want to add the ability to pick the backends in a more fine-grained manner, but we haven’t discussed any solutions yet.</p>

Shubhankar: <p>Any update on this? Is there a way to choose conv algorithms?</p>
Convert model output to vector of strings

Oktai15: <p>I use trained model in Python API in C++ API. My forward in Python returns <code>List[str]</code>. How can I get output as <code>std::vector&lt;std::string&gt;</code> in C++?</p>
<pre><code class="lang-auto">// how to convert outs?
auto outs = module.forward(dummy_input); 
</code></pre>
How to generate disorderly variable data with GAN when small data set

davidlee: <p>Hi,</p>
<p>I’m training <strong>Conditional GAN with Small dataset</strong><br>
and i’m wondering if there is i can do for generating disorderly variable data with GAN, which is very light network (MLP based 3 layers)<br>
I mean, once the GAN is trained with the limited number of samples,<br>
I can select the random z values for generating the fake data<br>
but the generated ones are quite easily overfitted but I want for the generator to produce randomly and disorderly distributed fake data.</p>
<p>Please help!</p>
JIT does not autodiff conv & batchnorm

ililiilliilill: <p>Hi,</p>
<p>recently I found that JIT does not create DifferentiableGraph nodes corresponding to Conv and BN operations.</p>
<p>Both ops are not registered in symbolic_script.h and considered to be “non-differentiable” in the sense that their gradients are always computed by Autograd engine, which loses the opportunity of making bigger DifferentiableGraphs.</p>
<p>What’s the reason behind such non-optimal behavior of JIT?</p>
Changing a Linear layer in pretrained model changes activation

Asa-Nisi-Masa: <p>Hello. I am using a pretrained model, but I want to change the final layer Linear to accompany a different number of classes. But it seems that while originally there was no activation associated with it (i.e. f(x)=x), after changing it the activation becomes hyperbolic tangent (f(x)=tanh(x)). Here is the code</p>
<pre><code class="lang-auto">import torch
from torchvision import models


model = models.mobilenet_v2(pretrained=True)
# last layer(s) defined in model.classifier

print(model.classifier)
# Sequential(
# (0): Dropout(p=0.2, inplace=False)
# (1): Linear(in_features=1280, out_features=1000, bias=True)
# )

x = torch.randn((1, 3, 224, 224))
y = model.forward(x)

print(y.min().item(), y.max().item())
# -3.4289, 2.9577

# change the last layer to predict 3 classes
model.classifier[1] = torch.nn.Linear(in_features=1280, out_features=3, bias=True)

y = model.forward(x)

print(y.min().item(), y.max().item())
# -0.185, 0.6268
</code></pre>
<p>Repeating this shows that new classifier always predicts ‘y’ to be in range [-1, 1]. What’s happening here?</p>

vainaijr: <aside class="quote no-group" data-username="Asa-Nisi-Masa" data-post="1" data-topic="66949">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/asa-nisi-masa/48/24803_2.png" class="avatar"> Asa-Nisi-Masa:</div>
<blockquote>
<p>model.classifier[1] = torch.nn.Linear(in_features=1280, out_features=3, bias=True) y = model.forward(x) print(y.min().item(), y.max().item()) # -0.185, 0.6268</p>
</blockquote>
</aside>
<p>I think if we change the input, then we will get values outside of [-1, 1] range.<br>
I was able to get greater than 1 value,</p>
<pre><code class="lang-auto">x = torch.randn(1, 3, 224, 224)
model.classifier[1] = torch.nn.Linear(in_features=1280, out_features=3, bias=True)
y = model.forward(x)
print(y.min().item(), y.max().item())
</code></pre>
<p>-0.0004289243370294571 1.1706701517105103</p>

Asa-Nisi-Masa: <p>Weird. I am even putting the <code>y = model.forward(x)</code> inside a loop and never seem to get values outside [-1, 1]…</p>
Introducing the Pytorch Toolkit - Keras-like API for Pytorch

mjbhobe: <p>The <strong>Pytorch Toolkit</strong> is a library of functions and classes I wrote to <strong>provide a Keras-like interface to train &amp; evaluate Pytorch models and make predictions</strong>. It works with Numpy data as well as torchvision datasets.</p>
<p>You can download it from my Github repository - <a href="https://github.com/mjbhobe/dl-pytorch" rel="nofollow noopener">https://github.com/mjbhobe/dl-pytorch</a> (it includes examples as well)</p>
<p>I also include a <a href="https://github.com/mjbhobe/dl-pytorch/blob/master/Tutorial.md" rel="nofollow noopener">Step-by-Step Tutorial</a> to get you started quickly.</p>
<p>Hope you love using this as much as I loved writing it.</p>

ptrblck: <p>Thanks for posting your work here! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Could you describe some advantages or use cases of your library compared to e.g. Lightning or Ignite?</p>

mjbhobe: <p>To be honest, I have not used Lightening or Ignite, so I can’t really comment.<br>
I came to Pytorch from a primary Keras over Tensorflow background. I hated using pure Tensorflow as you had to write a lot of verbose non-pythonic code. Keras was a welcome relief. Then I heard that Pytorch was more Pythonic in it’s approach, so though I’d give it a try. But to my dismay, I saw that yet again I had to write a lot of boilerplate code to train/test my DL models.<br>
So to alleviate my problem, I wrote some re-useable code, primarily to avoid writing repeated boilerplate code. Though I’d share my work with the community in the hope that someone else also finds it useful. This is a very simple library (just 1 file of code), so I don’t doubt that Ignite or Lightening would be far more capable. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Ah OK, sounds like a reasonable approach <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Maybe your library (or the other mentioned ones) could benefit from the design decisions made, so feel free if you think some kind of collaboration would be useful. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

henrique: <p>Well, there is also the fast.ai thing! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Storing learning data in tensors, data loading optimization

Tavados: <p>I’m wondering if it could reduce training time by storing learning data (e.g. tenthousands of images) in a big tensor instead of loading them one by one with a dataloader.</p>
<p>Can this only speed up the initialization process because the data - once loaded - is stored in the memory and in tensor format anyways? How do we differentiate between datasets so small that they should stay in memory and so big, that they have to be partially loaded for every epoch?</p>
<p>I don’t care about a faster initialization-phase but can we improve learning time in any way with? If tenthousands (or more) of image have to be decoded (from .jpg or .png or alike) every epoch, this seems like an easy optimization to do once.</p>
<p>For the case that the images are loaded again and again for every epoch and should not be stored as a tensor (for whatever reason): Which image format would be optimal? Are there mentionably less decoding steps for a .tiff, .jpg., .png etc., or does it not matter, because they are just loaded once?</p>

ptrblck: <p>You could preload all images and store them in a large tensor, if you have enough RAM and are OK with spending the loading time during initialization.<br>
However, once you are dealing with image tensors, note that you won’t be able to use a lot of <code>torchvision.transforms</code> anymore, as they are using <code>PIL.Image</code>s.</p>
<p>If you are using a <code>DataLoader</code> to load each sample separately, you could lazily load the data using multiple workers in the background, while the GPU is busy with the model training, thus hiding the data loading.</p>
Multi-gpu trainining does not exist correctly

coincheung: <p>Hi,</p>
<p>I used the distributed method to train models with multi gpus. When the training process is done, the program does not exist and I can see from <code>nvidia-smi</code> that all the gpus are released except the first gpu which still occupies around 1G memory. I stopped the program by <code>ctrl + C</code> and the memory in the first gpu is thus released.</p>
<p>I can see from the log message after my pressing <code>ctrl + C</code> like this:</p>
<p>File “/miniconda/envs/py36/lib/python3.6/site-packages/torch/distributed/launch.py”, line 246, in main<br>
process.wait()<br>
File “/miniconda/envs/py36/lib/python3.6/subprocess.py”, line 1477, in wait<br>
(pid, sts) = self._try_wait(0)<br>
File “/miniconda/envs/py36/lib/python3.6/subprocess.py”, line 1424, in _try_wait<br>
(pid, sts) = os.waitpid(self.pid, wait_flags)<br>
KeyboardInterrupt</p>
<p>What is the cause of this please, and how could I make it work correctly ?</p>

yikaiw: <p>Exactly the same issue, have you had any solution so far?</p>

ptrblck: <p>Is this behavior reproducible?<br>
Do you only encounter it in a distributed setup or also with a single GPU?<br>
Could you post a code snippet to reproduce this issue and post some information about your current system and setup?</p>
<p>CC <a class="mention" href="/u/yikaiw">@yikaiw</a></p>

coincheung: <p>Yes, it maybe some memory leakage problem about dataloader. I have this problem when I add the line of <code>torch.multiprocessing.set_sharing_strategy('file_system')</code>. If I do not have this line in my code, my program can exit correctly.</p>

yikaiw: <p>I only encounter it in a distributed setup. Either running the code on two GPUs or four GPUs meets the same problem. For a single GPU, there is no problem. A quick code to reproduce shows below:</p>
<p>import torch<br>
torch.cuda.set_device(0)<br>
torch.distributed.init_process_group(backend=“nccl”, init_method=“env://”)<br>
torch.distributed.barrier()</p>
<p>And the running command is: CUDA_VISIBLE_DEVICES=3,4 python -m torch.distributed.launch --nproc_per_node=2 quick_code.py</p>
<p>where quick_code.py contains the four-line code.</p>
<p>The problem appears in V100 GPUs. However, the same code is OK when I run it on GeForce GTX-1080 GPUs or RTX-2080 GPUs.</p>

yikaiw: <p>I don’t have this line in my code. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
How to implement stochastic pooling in pytorch?

smani: <p>I am trying to implement Stochastic pooling. But I am very much unclear about how to sample from a multi-nominal distribution to get the location from where we need to pool. Could you please provide any example code?</p>

Eta_C: <p>Are you looking for this?<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/renebidart/adv-denoising/blob/1400e147ee517667fb9aeb67fa3b38a233c4ab3b/models/StochasticPool2d.py#L61-L78" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/renebidart/adv-denoising/blob/1400e147ee517667fb9aeb67fa3b38a233c4ab3b/models/StochasticPool2d.py#L61-L78" target="_blank" rel="nofollow noopener">renebidart/adv-denoising/blob/1400e147ee517667fb9aeb67fa3b38a233c4ab3b/models/StochasticPool2d.py#L61-L78</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="61" style="counter-reset: li-counter 60 ;">
<li># def gen_random(values):</li>
<li>#     if torch.sum(values) != 0:</li>
<li>#         idx = torch.multinomial(values, num_samples=1)</li>
<li>#     else:</li>
<li>#         idx = torch.randint(0, values.shape[0], size=(1,)).type(torch.LongTensor)</li>
<li>#     return values[idx]</li>
<li>
</li>
<li># init_size = x.shape</li>
<li>
</li>
<li># x = F.pad(x, self._padding(x), mode='reflect')</li>
<li># x = x.unfold(2, self.kernel_size[0], self.stride[0]).unfold(3, self.kernel_size[1], self.stride[1])</li>
<li># x = x.contiguous().view(-1, 4)</li>
<li>
</li>
<li># x = torch.stack([</li>
<li>#          gen_random(x_i) for i, x_i in enumerate(torch.unbind(x, dim=0), 0)</li>
<li>#         ], dim=0)</li>
<li># x = x.contiguous().view(init_size[0], init_size[1], int(init_size[2]/2), int(init_size[3]/2))</li>
<li># return x</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

smani: <p>Thank you very much for the implementation. Do you think the run time of this can be somehow improved, for e.g. by somehow replacing the for loop?</p>

Nirthika_Rajendran: <p>Hi, In the stochastic pooling, the author used <strong>probabilistic weighting at test time</strong>. I tried that, but the testing is not performing well. Did you have any solution for that?</p>

Eta_C: <p>Hi, <a class="mention" href="/u/smani">@smani</a> <a class="mention" href="/u/nirthika_rajendran">@Nirthika_Rajendran</a><br>
Sorry I haven’t studied this code carefully. I just happened to see it before, so I shared it. <img src="https://discuss.pytorch.org/images/emoji/apple/worried.png?v=9" title=":worried:" class="emoji" alt=":worried:"></p>