akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
nn.Linear() with variable-sized input

username: arjung
cooked: <p>Can nn.Linear() work with variable-sized inputs?</p>
<p>I have an input of shape (2048, x) and would like to convert it to be of shape (2048) using a fully-connected layer – is this possible?</p>
<p>When I passed in x as an argument to the model, the only issue I had was with saving/loading the weights since x would change the number of parameters in the layer. Any ideas about how this issue can be resolved?</p>

username: tymokvo
cooked: <p>Welcome to the forums!</p>
<p>As you noted, it will “work” in that you can dynamically create an <code>nn.Linear</code> with an input dimension that varies with <code>x</code>. But this necessarily changes the shape of the weight tensor, so, once saved, it will not match all of your data points.</p>
<p>I think the typical solution is to make a <code>nn.Linear</code> that matches the max <code>x</code> value of your data and pad the small data points with zeros.</p>

username: Balamurali_M
cooked: <p>In my experience, the networks with fully connected networks are restricted by their input image dimensions. Change in image dimensions will result in input to the linear layer having different dimensions. This will throw an error. The fully convolutional networks are designed for this problem of handling multiple image sizes. Because not all data can be resized to the expected dimensions. Imagenet the dimensions are restricted by 224 * 224. It is because of fully connected network only.  But in general available datasets will have fixed dimension. So the model can be saved for that specific dimension. Resizing is generally done for classification tasks. Padding is also an alternative.</p>
Saving for training later

username: Paulo_Mann
cooked: <p>I see that in the tutorials, from PyTorch website, we have a way to save the entire model along with the loss and optimizer parameters. That being said, I don’t understand what is the “loss” in this code below (from PyTorch website - <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training</a>) :</p>
<pre><code class="lang-auto">model = TheModelClass(*args, **kwargs)
optimizer = TheOptimizerClass(*args, **kwargs)

checkpoint = torch.load(PATH)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']

model.eval()
# - or -
model.train()
</code></pre>
<p>Is it the final value of the loss before stopping training? Or is it the entire loss history (aka, a list of losses per epoch)?</p>
<p>I also have a StepRL, can its state be saved like the others? Since I need to recover the lr decay.</p>
<p>Thanks!</p>

username: alex.veuthey
cooked: <p>The loss in the tutorial corresponds to the loss value when it’s saved. It’s not mandatory to save it when you want to resume training, but it can be useful to have (if you want to check consistency for example…)</p>
<p>With step LR, you can give the last epoch as an argument when constructing it. This will update the scheduling accordingly, but you need to save the last step with the checkpoint.</p>

username: Paulo_Mann
cooked: <p>I see! Thanks <a class="mention" href="/u/alex.veuthey">@alex.veuthey</a></p>
Why doesn't backward() work on nonscalar outputs?

username: johnmwu
cooked: <p>For example,</p>
<pre><code class="lang-auto">t1 = torch.Tensor([1, 3])
t1.requires_grad_(True)
t2 = t1 * t1

t2.backward() # fails
</code></pre>
<p>Why can’t it just put the Jacobian matrix in t1?</p>

username: Balamurali_M
cooked: <p>Pytorch is designed in a way that you try to calculate the error between predicted and ground truth sum the error and try to minimise that sum. In deep learning literature, that’s how backprop has been defined.</p>
Out-of-place ReLU operation in PyTorch Neural style transfer tutorial

username: Nikhil_Chigali
cooked: <p>What’s the point of changing the mode of <code>nnReLU</code> operation from inplace to out-of-place in PyTorch Neural style transfer tutorial. I dont think that it would make any difference even if the relu is inplace.</p>
<p>It’s mentioned in the document that,</p>
<pre><code class="lang-auto"># The in-place version doesn't play very nicely with the ContentLoss and 
# StyleLoss we insert below. So we replace with out-of-place 
# ones here.
</code></pre>
<p>Why is that exactly true?</p>
Chatbot Tutorial on Pytorch documentation, gathering indices that are invalid

username: tnguyen
cooked: <p>Hi,<br>
From this tutorial here: <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/chatbot_tutorial.html</a>,<br>
at the part where we calculate the loss using:</p>
<pre><code class="lang-auto">def maskNLLLoss(inp, target, mask):
    nTotal = mask.sum()
    crossEntropy = -torch.log(torch.gather(inp, 1, target.view(-1, 1)).squeeze(1))
    loss = crossEntropy.masked_select(mask).mean()
    loss = loss.to(device)
    return loss, nTotal.item()
</code></pre>
<p>Why are we using target.view(-1, 1) as the index to gather from? target contains word indices, if I’m not mistaken, which, given a large corpus can be very huge, and thus won’t be valid indices for the function to use to select from.</p>
<p>Also, we do we only gather from the first dimension?</p>

username: ptrblck
cooked: <p><code>inp</code> will probably contain the prediction probabilities as <code>[batch_size, nb_words]</code>.<br>
If that’s the case, the <code>gather</code> operation will get all predictions for the current target word index in each sample:</p>
<pre><code class="lang-python">batch_size = 5
nb_words = 10

inp = torch.randn(batch_size, nb_words)
target = torch.randint(0, nb_words, (batch_size,))
torch.gather(inp, 1, target.view(-1, 1))
</code></pre>

username: tnguyen
cooked: <p>Ah, I see, so this function requires the resulting probability to be of size (batch_size, number of words).</p>
<p>I’ve been reading articles about implementing a chatbot using the Transformer model from Google. I was borrowing the training loop here. However, the Transformer model requires the user to input an output size or as the paper from Google calls it, d_model. So currently, I am getting decoder outputs of size [batch_size, d_model]. How would I go about calculating the loss for this, do you have any pointer? Thanks.</p>
Multiple a real-value quantity

username: aldeka12
cooked: <p>Hello, I made a model to predict multiple a real-value quantity, exactly two of them, some values could be negative, but now I need to train the model. Which are loss function and optimizers suited for this? Thanks in advance!</p>

username: tymokvo
cooked: <p>Welcome to the forums!</p>
<p>You have probably noticed that PyTorch can only implicitly create gradients for scalar outputs. Thus, you need to combine them in some way that will make a loss calculation meaningful for your task. A simple objective function like <code>nn.MSELoss</code> might be a good place to start.</p>
<p>You can also take multiple backward passes by specifying <code>output.backward(retain_graph=True)</code></p>
<p>This is a contrived example that you should be able to translate into your problem:</p>
<pre><code class="lang-auto">x = torch.randn(1, 3, requires_grad=True)
y = torch.randn(3, 2, requires_grad=True)

z = x @ y
z0 = z[:, 0]
z1 = z[:, 1]

z0.backward(retain_graph=True)
z1.backward()
</code></pre>
<p>Good luck!</p>
StepLR(_LRScheduler) based on batch_sizes

username: MasayoMusic
cooked: <p>I am going through the transfer learning tutorial, and looked at the source code of StepLR.<br>
It seems if I wanted to change the StepLR to be based on batches instead of epochs,<br>
I just need to run <code>scheduler.step()</code> every <code>x</code> of amount batches, where x is the amount desired and set step_size within lr_scheduler to 1?</p>

username: alex.veuthey
cooked: <p>That’s actually pretty straightforward, but I can understand not thinking about it, because it’s not explicit from the documentation.</p>
<p>What you want to do is define your <code>StepLR</code> normally in the beginning, but instead of having values correspond to epochs, you set them up w.r.t. batches. Then you call the <code>.step()</code> method of the scheduler at each batch instead of each epoch…</p>

username: MasayoMusic
cooked: <p>Okay. Yeah I had written that in the OP, but just wanted to make sure. Thank you.</p>
Difference between batchnorm1d and batchnorm2d

username: sapNou
cooked: <p>Batchnorm2d is meant to take an input of size NxCxHxW where N is the batch size and C the number of channels. But is it the same if I fold the two last dimensions together, call Batchnorm1d and then unfold them after the normalization?<br>
Thanks a lot.</p>

username: chittiman
cooked: <p>If I am not wrong , I think when you are talking about batchnorm2d you mean Spatial Batchnormalization which intakes an input of size (N,C,H,W) and returns back the same size.<br>
In spatial BN mean and variances are calculated for each color channel© across all the images. So if you want to use batchnorm1d to perform spatial BN , you need to input a 2dim array of size(N<em>H</em>W, C,). So, just folding last 2 dimensions will give you array of size (N, C<em>H</em>W) which is incorrect input. So you need to swap first 2 axes to (C,N,H,W) then fold last 2 dims (C,N<em>H</em>W) and then transpose it (N<em>H</em>W, C) and now pass it to BN1d and then perform reverse steps on the output to get the desired result.</p>
<p>Hope this helps!!</p>
<p>P.S: I am new to the pytorch community. Please correct me if I am wrong.</p>
Loss for video - sum up loss for frames

username: danielafrimi
cooked: <p>Hi,<br>
I want to use my loss for images for my video, what im doing is looping over the frames and then call the loss function and summing it up.</p>
<p>there is another way to do it?<br>
this is what i’m doing:</p>
<p>def dist_per_frame(self, frames_real, frames_fake):<br>
rec_loss = 0<br>
for i in range(31):<br>
rec_loss += self.dist_frame(frames_fake[i], frames_real[i])<br>
return rec_loss</p>
<p>**self.dist_frame is my loss function</p>

username: JuanFMontesinos
cooked: <p>Can you pose a pytorch-related problem? Do you have a model? Pure math question?</p>

username: danielafrimi
cooked: <p>yes, i have a model and i’m using perceptual loss (VGG16). the VGG16 net works well woth images (for reconstruction) but when im using the net with video i want to take into account all the frames.<br>
so i wonder if what i wrote above is the correct way to do so.</p>
<p>thanks</p>

username: JuanFMontesinos
cooked: <p>Proper way would be to squash all those frames into batch dimension, apply vgg, unsquash and then sum them up</p>

username: danielafrimi
cooked: <p>what does it mean to squash them?<br>
i pass a tensor of size [batch_size, 3, 64,64] to VGG16 so i think i squashes them up into a batch dimension (?).</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: JuanFMontesinos
cooked: <p>As you are working with video I was expecting something like</p>
<p>batch,frames,3,64,64 --&gt; batch*frames,3,64,64</p>
ModuleNotFoundError: No module named 'torch' in spyder

username: JohnCwok
cooked: <p>Hi !</p>
<p>I installed pytorch recently, following the command on the pytorch website (i am using anaconda).<br>
When I use the anaconda prompt to open a python intepreter the <code>import torch</code> line works perfectly, but when I try it in spyder I get the error :<br>
<code>import torch ModuleNotFoundError: No module named 'torch'</code><br>
I tried creating a conda environment, activating it and importing pytorch inside of it but it did’t change anything.<br>
I am running on window 10 without cuda.</p>
<p>Do you guys know what I can do ? I know this issue has already been posted but none of the proposed solution worked for me which is why i am posting.</p>

username: JuanFMontesinos
cooked: <p>Hi, have you verified that spyder points to proper python iterpreter? Realize you cannot install several spyder versions (namely, one for python2 and another for python3).</p>
<p>That’s probably the issue you are facing (somehow)</p>

username: tymokvo
cooked: <p>You can follow <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> advice with:</p>
<pre><code class="lang-auto">import sys
print(sys.executable)
</code></pre>
<p>And ensure the executable that is pointed to is the one with <code>torch</code> installed.</p>

username: JohnCwok
cooked: <aside class="quote no-group" data-username="tymokvo" data-post="3" data-topic="47648">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tymokvo/48/9527_2.png" class="avatar"> tymokvo:</div>
<blockquote>
<p>import sys print(sys.executable)</p>
</blockquote>
</aside>
<p>This returns <code>C:\Users\cnelias\Anaconda3\pythonw.exe</code> which i think should be correct ? When I run <code>conda info --envs</code><br>
I only have the following :<br>
<code>base     *    C:\Users\cnelias\.julia\conda\3    C:\Users\cnelias\Anaconda3</code>. Or am I doing this wrong ?</p>

username: tymokvo
cooked: <p>Ugh, it’s hard to help debug an environment setup. It also looks like you have installed <code>julia</code> which I have had totally ruin a <code>Conda</code> setup on my windows machine before. To be honest, I have entirely stopped using environment managers like <code>Conda</code> as I think they cause more problems than they solve.</p>
<p>Could you try making a native python virtual environment to test? That will isolate config problems to <code>Spyder</code> or <code>Conda</code>.</p>
<p>I use something like:<br>
(from a command prompt)</p>
<pre><code class="lang-auto">&gt;python -m venv pytorch # venv is the python native virtual environment manager
&gt;.\pytorch\Scripts\activate
(pytorch) &gt;pip install [enter the correct pip package here]
</code></pre>
<p>And then launch python from that environment and try some torchin’.</p>
<p>The <code>venv</code> command will create a directory with the name that you give it which contains the whole virtual environment. So I keep all of mine in a directory called <code>PythonEnvironments</code>.</p>
Indices of a dataset sampled by DataLoader

username: sparseinference
cooked: <p>Is there a recommended method of obtaining the indices of the dataset that are sampled by a <code>torch.utils.data.DataLoader</code>?</p>
<p>I know that a RandomSampler will return a list of indices, but there doesn’t seem to be a way to access those indices once the sampler has been passed to a <code>torch.utils.data.DataLoader</code>.</p>

username: Oli
cooked: <p>Hoi there! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I don’t know if there is a recommended method but I managed to do it in the way below. This will produce the same indexes until you call sampler.shuffle(). So if you want the indexes, you could just save your sampler and produce the indexes like so -&gt; <code>indexes = list(iter(sampler))</code> Don’t know if you need that iter part tbh but can’t test now.</p>
<p>This comes at the downside of manually shuffling your indexes at the end of every epoch via the shuffle method.</p>
<p>Do you think something like this would work for you?</p>
<pre><code class="lang-auto">class RandomSampler(Sampler):
  def __init__(self, data_source):
    self.data_source = data_source
    self.shuffle()

  def shuffle(self):
    self.seed = random.randint(0, 2**32 - 1)

  def __iter__(self):
    n = len(self.data_source)
    indexes = list(range(n))
    random.Random(self.seed).shuffle(indexes)
    return iter(indexes)

  def __len__(self):
    return len(self.data_source)
</code></pre>

username: sparseinference
cooked: <p>OK thanks.<br>
I see the idea - generate a random permutation of the indexes ahead of time, then just pass that to a <code>SequentialSampler</code>.<br>
Since the order of indexes is fixed I can just extract <code>batch_size</code> indexes at a time from that list.</p>
Combining any two rows of a tensor

username: paolo
cooked: <p>I have a tensor T with size (m, n) and I would like to concatenate any two rows. For example:</p>
<pre><code class="lang-auto">T = [[0,1], 
     [2,3], 
     [4,5]]
concat = [[0,1,2,3], [0,1,4,5], [2,3,0,1], [2,3,4,5], [4,5,0,1], [4,5,2,3]]
</code></pre>
<p>Is there any function able to do this? The only way that I have found is to use two nested for loops. Thanks.</p>

username: sparseinference
cooked: <pre><code class="lang-auto">from itertools import permutations
torch.stack([torch.cat(rows,0) for rows in permutations(T, 2)])
</code></pre>
<p>results in:</p>
<pre><code class="lang-auto">tensor([[0, 1, 2, 3],
        [0, 1, 4, 5],
        [2, 3, 0, 1],
        [2, 3, 4, 5],
        [4, 5, 0, 1],
        [4, 5, 2, 3]])
</code></pre>
Install torchvision with pytorch nighlty

username: amirhf
cooked: <p>I’m trying to install pytorch-nightly with torchvision, but when I try the following command: <code>conda install pytorch-nightly torchvision cudatoolkit=10.0 -c pytorch</code> it tries to install the stable pytorch instead of pytorch-nightly.</p>
<p>I also install torchvision only first with: <code>conda install torchvision -c pytorch</code>, then when I want to remove the pytorch that comes with it with <code>conda remove pytorch</code>, it removes the torchvision as well!</p>

username: Carlos_Barrios
cooked: <p>I have the same problem. I don’t see how to only get <code>torchvision</code> with the pytorch nightly.</p>
Extract weights from LSTM / GRU cells

username: davidmrau
cooked: <p>How can I access and extract weights from all cells in a LSTM / GRU layer?</p>

username: hughperkins
cooked: <p>In general, you can always do <code>list(r.parameters())</code>, to get all the parameters of any model <code>r</code>.</p>

username: anantguptadbl
cooked: <p>A better alternative would be list(r.named_parameters()). This would give you a dict kind of structure</p>
Using autograd when requires_grad=False

username: chrome
cooked: <p>I am trying to implement the GradNorm from <a href="https://github.com/hosseinshn/GradNorm/blob/master/GradNormv8.ipynb" rel="nofollow noopener">https://github.com/hosseinshn/GradNorm/blob/master/GradNormv8.ipynb</a> wherein I am using a pretrained word2vec embedding layer for which I have set requires_grad=False.<br>
On cell 4 of the Notebook there is<br>
<code>G1R = torch.autograd.grad(l1, param[0], retain_graph=True, create_graph=True)</code>.<br>
The above line throws the following error<br>
<code>G1R = torch.autograd.grad(l1, param[0], retain_graph=True, create_graph=True) File "/usr/lib64/python3.6/site-packages/torch/autograd/__init__.py", line 145, in grad inputs, allow_unused) RuntimeError: One of the differentiated Tensors does not require grad</code>.<br>
Is there anyway to implement this code for my usecase where pretrained embedding can have requires_grad=False ?</p>

username: albanD
cooked: <p>What is <code>param[0]</code> supposed to be here? The weights of your embedding layer?</p>

username: chrome
cooked: <pre><code class="lang-auto"># Getting gradients of the first layers of each tower and calculate their l2-norm 
        param = list(MTL.parameters())
</code></pre>
<p>where MTL is the class <code>class MTLnet(nn.Module):</code><br>
I myself am not sure why they’re using <code>param[0]</code></p>

username: albanD
cooked: <p>I guess you want to check what <code>param[0]</code> is and why they use that.<br>
Then make sure that this requires gradients.<br>
Note that if gradients are needed to compute your gradient penalty loss but you don’t want to update your embedding, you can keep the embedding as requiring gradient but don’t give it to the optimizer so that it will never be updated.</p>
How exactly dataloader works with csv file

username: ASHUTOSH_CHANDRA
cooked: <p>Pytorch data loader is supposed to take a dataset as input. The dataset is an instance of Dataset class which defines the “len” and “get” functionality.</p>
<p>What if instead of dataset, I give simple csv file as input to pytroch data loader. I tried and the data loader is working, but I’m not sure about the behavior of the created dataloader. I mean how will it get the len and get defined.</p>

username: ptrblck
cooked: <p>We had recently a similar discussion <a href="https://discuss.pytorch.org/t/is-numpy-array-a-dataset/47376">in this topic</a>.<br>
The <code>DataLoader</code> might take inputs, where <code>__getitem__</code> and <code>__len__</code> is defined.<br>
E.g. this code will work:</p>
<pre><code class="lang-python">x = [0, 1, 2]
print(x.__getitem__)
print(x.__len__)

loader = DataLoader(x)
for data in loader:
    print(data)
</code></pre>
<p>However, I would still recommend to derive your custom dataset from <code>Dataset</code> and pass it to the <code>DataLoader</code>, as this will give you more control of how <code>__getitem__</code> and <code>__len__</code> is implemented.</p>
Input/output error

username: Kaushal_Aggarwal
cooked: <p>I am using google colaboratory, in my custom dataset class(inheriting from Dataset class) I am passing a file consisting of strings like ‘1.npy’,'16003.npy which I need to load from my drive.<br>
For example i passed the file as path_x and path_x[1] = ‘100.npy’ , now while loading it using np.load(path_x[1]) I am getting error: OSError: [Errno 5] Input/output error: ‘100.npy’<br>
Can anyone please help me with this?</p>

username: ptrblck
cooked: <p>This looks like a Colab issue and is probably not directly related to PyTorch.<br>
It seems it’s being tracked <a href="https://github.com/googlecolab/colabtools/issues/510" rel="nofollow noopener">here</a>.</p>
Debugger fails while Running .py works

username: Haydnspass
cooked: <p>Hey there,</p>
<p>I work on a growing project and suddenly have the problem that the debugger fails while running the python script works …<br>
So if I want to run let’s say:</p>
<pre><code class="lang-auto">import torch

if __name__ == '__main__':
    x = torch.rand((5, 5, 5, 5))
    print(x)
</code></pre>
<p>that works fine, but as soon as I want to debug it I get the following error:</p>
<pre><code class="lang-auto">pydev debugger: process 77281 is connecting

Connected to pydev debugger (build 191.7479.30)
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1758, in &lt;module&gt;
    main()
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1752, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1147, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/lucasmueller/Repositories/deepsmlm/deepsmlm/generic/utils/dummy.py", line 1, in &lt;module&gt;
    import torch
  File "/Users/lucasmueller/anaconda/envs/embl/lib/python3.7/site-packages/torch/__init__.py", line 42, in &lt;module&gt;
    import numpy as _np
  File "/Users/lucasmueller/anaconda/envs/embl/lib/python3.7/site-packages/numpy/__init__.py", line 142, in &lt;module&gt;
    from . import core
  File "/Users/lucasmueller/anaconda/envs/embl/lib/python3.7/site-packages/numpy/core/__init__.py", line 40, in &lt;module&gt;
    from . import multiarray
  File "/Users/lucasmueller/anaconda/envs/embl/lib/python3.7/site-packages/numpy/core/multiarray.py", line 12, in &lt;module&gt;
    from . import overrides
  File "/Users/lucasmueller/anaconda/envs/embl/lib/python3.7/site-packages/numpy/core/overrides.py", line 46, in &lt;module&gt;
    """)
RuntimeError: implement_array_function method already has a docstring

</code></pre>
<p>What kind of problem could that be?</p>
Print gradient on the parameters

username: Jake_Zhao
cooked: <p>What is the solution to print out the gradient on the parameters, in the newest version of PyTorch?</p>
<p>I did:</p>
<pre><code class="lang-auto">for p in model.parameters():
    print(p.grad.norm())
</code></pre>
<p>It gave me that <code>p.grad</code> is <code>None</code>.</p>

username: ptrblck
cooked: <p>The loop should print gradients, if they have been already calculated.<br>
Make sure to call <code>backward</code> before running this code.<br>
Also, if some parameters were unused during the forward pass, their gradients will stay <code>None</code>.</p>
How to decide the margin in Triplet Loss to be used in the training of a Siamese Network

username: n0obcoder
cooked: <p>I am not sure how much <strong>margin</strong> should i keep in my Triplet Loss. My model is not training.<br>
Here is my implementation of the <a href="https://www.kaggle.com/n0obcoder/siamese-net" rel="nofollow noopener">Siamese Network</a></p>
Torch.add() get different results

username: Maox
cooked: <p>In Docs, there are torch.add(input,  value,  out=None) and torch.add(input,  value, others,  out=None).<br>
I expect ‘b’ and ‘c’ in the following code should be the same in value but not:</p>
<pre><code class="lang-python">import torch

lr = 1e-2
torch.manual_seed(666)
a = torch.randn(3,3)
a.grad = torch.randn(3,3)

b = torch.add(a, -lr, a.grad)
c = torch.add(a, -lr*a.grad)
print(torch.abs(b-c).sum())
</code></pre>
<p>So, can you please help me on why they are different?</p>

username: jerinphilip
cooked: <pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt;
&gt;&gt;&gt; lr = 1e-2
&gt;&gt;&gt; torch.manual_seed(666)
&lt;torch._C.Generator object at 0x7f278fb0dc10&gt;
&gt;&gt;&gt; a = torch.randn(3,3)
&gt;&gt;&gt; a.grad = torch.randn(3,3)
&gt;&gt;&gt;
&gt;&gt;&gt; b = torch.add(a, -lr, a.grad)
&gt;&gt;&gt; c = torch.add(a, -lr*a.grad)
&gt;&gt;&gt; print(torch.abs(b-c).sum())
tensor(1.8626e-09)
</code></pre>
<p>Aren’t they same? 1e-9 ~= 0.</p>

username: Maox
cooked: <p>Yes,  the final value is 1.8626e-9. But it should be a strict 0.0 . There may be precision problem in float computing but the result should be the same.</p>

username: tom
cooked: <p>What happens is that those operations are implemented differently and that for floating point operations this different ordering matters.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Maox
cooked: <p>Thanks for your reply.</p>
<p>So, you mean these two operations are implemented differently in C codes and that causes different results? Would you please give me a little more information about the differences?</p>
Loading column major stored matrices in n-dimensions

username: Haydnspass
cooked: <p>Hey,</p>
<p>I have a n-dim data where the memory layout is column major (or rather first index first for n dimensions) instead of C++ default row major layout (last index first).</p>
<p>So if I load it with std::memcpy I get the following in 2D for illustration:</p>
<p>input<br>
0.8147    0.0975    0.1576    0.1419<br>
0.9058    0.2785    0.9706    0.4218<br>
0.1270    0.5469    0.9572    0.9157<br>
0.9134    0.9575    0.4854    0.7922<br>
0.6324    0.9649    0.8003    0.9595</p>
<p>tensor:<br>
0.8147  0.9058  0.1270  0.9134<br>
0.6324  0.0975  0.2785  0.5469<br>
0.9575  0.9649  0.1576  0.9706<br>
0.9572  0.4854  0.8003  0.1419<br>
0.4218  0.9157  0.7922  0.9595</p>
<p>Is there an interface for libtorch to automatically convert this? Or does someone know a good source? All I can find works for 2 dimensions, which is comparably easy compared to at compile time unknown number dimensions.</p>
<p>Cheers!</p>
Stacked LSTM with same hidden input

username: lanka
cooked: <p>i need to be stacked two LSTM with same parallel hidden input, how can I add one after another,</p>
<p>class DecoderRNN(nn.Module):<br>
def <strong>init</strong>(self, embed_size, hidden_size, vocab_size, num_layers, max_seq_length):<br>
“”“Set the hyper-parameters and build the layers.”""<br>
super(DecoderRNN, self).<strong>init</strong>()<br>
self.embed = nn.Embedding(vocab_size, embed_size)<br>
self.lstm1 = nn.LSTM(embed_size, hidden_size, num_layers, batch_first=True)<br>
self.linear = nn.Linear(hidden_size, vocab_size)<br>
self.max_seg_length = max_seq_length<br>
self.init_weights() # initialize weigts<br>
def init_weights(self):<br>
“”“Initialize the weights.”""<br>
self.linear.weight.data.normal_(0.0, 0.02)<br>
self.linear.bias.data.fill_(0)</p>
The use of loss function calculated with Numba in Pytorch

username: zehra_uslu
cooked: <p>Hi All. I have a model written in Pytorch; however, I would like to calculate loss function by using Numba. I have searched online and could not find any appropriate solution. What I made up is to calculate loss by using a loss function written for Pytorch variables, let’ s say MSE loss, and then change “loss.data” to what I calculate with my loss function in Numba.    I do want to ask if there is any more elegant way to incorporate my loss function to Pytorch.</p>

username: albanD
cooked: <p>Hi,</p>
<p>This won’t work as the computed gradients won’t be the ones corresponding to what you computed in numba.<br>
If you can, you should reimplement your loss function with pytorch operations to have automatic differentiation of it.<br>
Otherwise, you will need to write a new <code>autograd.Function</code> as discussed <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="nofollow noopener">here</a> to specify what the backward should be.</p>

username: AndreasBuhr
cooked: <p>This sounds like an interesting question.<br>
Could you share your source code to investigate?</p>

username: zehra_uslu
cooked: <p>Thank you for your answer. It makes sense.</p>

username: zehra_uslu
cooked: <p>Sorry. I am not allowed to share my code.</p>
No acc in the validation

username: omarabdelaziz
cooked: <p>Hello,<br>
I’m building a Cnn_lstm for classification and it gives me pretty good accuracy in the training but in the validation or testing it gives me pretty bad accuracy</p>
<pre><code class="lang-auto">
class EncoderCNN(nn.Module):
    def __init__(self, embed_size):
        """Load the pretrained ResNet-152 and replace top fc layer."""
        super(EncoderCNN, self).__init__()
        self.conv1 = nn.Conv2d(1, 3, kernel_size=2, stride=2, padding=3,
                               bias=False)
        resnet = models.resnet152(pretrained=True)
        modules = list(resnet.children())[:-1]      # delete the last fc layer.
        
        self.resnet = nn.Sequential(*modules)
        self.linear = nn.Linear(resnet.fc.in_features, embed_size)
        self.bn = nn.BatchNorm1d(embed_size, momentum=0.01)
        
    def forward(self, images):
      
        """Extract feature vectors from input images."""
        C,batch_size, H, W = images.size()
        images = images.view(batch_size , C, H, W)
        images = self.conv1(images)
        with torch.no_grad():
            features = self.resnet(images)
          
        features = features.reshape(features.size(0), -1)
        features = self.bn(self.linear(features))
        return features


class DecoderRNN(nn.Module):
    def __init__(self, embed_size, hidden_size, vocab_size, num_layers, max_seq_length=20):
        """Set the hyper-parameters and build the layers."""
        super(DecoderRNN, self).__init__()
        self.embed = nn.Embedding(vocab_size, embed_size)
        self.lstm = nn.LSTM(embed_size, hidden_size, num_layers, batch_first=True)
        self.linear = nn.Linear(hidden_size , vocab_size)
        self.max_seg_length = max_seq_length
        
    def forward(self, features, captions, lengths):
        """Decode image feature vectors and generates captions."""
        embeddings = self.embed(captions)
        embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)
        packed = pack_padded_sequence(embeddings, lengths, batch_first=True) 
        hiddens, _ = self.lstm(packed)
        outputs = self.linear(hiddens[0])
        return outputs
    
    def sample(self, features, states=None):
        """Generate captions for given image features using greedy search."""
        
        inputs = features.unsqueeze(1)
        hiddens, states = self.lstm(inputs, states)          # hiddens: (batch_size, 1, hidden_size)
        outputs = self.linear(hiddens[0])            # outputs:  (batch_size, vocab_size)

            
        return outputs
</code></pre>
<p>I’m using the sample method for testing if anyone has any Ideas what’s wrong here please tell me</p>
How to train multiple parameters with a GAN in PyTorch?

username: Taxaria
cooked: <p>Hey <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I’m implementing actually a GAN, which should generate MNIST-Superpixel Graphs.<br>
My model should learn how the nodes are connected and which node is important to represent the number in the graph.<br>
So the connection problem does it solve very well, but how can my model the node features additionally?</p>
<p>Thank you in andvance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Cropping tensor

username: ash_gamma
cooked: <p>Hi,</p>
<p>I’ve a tensor of size N x 3 x 364 x364, where N is the number of images. I’d like to “center crop” the tensor to of size N x 3 x 256 x 256. What would be most efficient way to achieve this in cpp? (I’m not using opencv)</p>
<p>Currently I do the following:</p>
<pre><code class="lang-auto">batch = tensor of size N x 3 x 364 x 364;
tensor_sliceH = batch.slice(2, start_height, end_height);
tensor_sliceW = tensor_sliceH.slice(3, start_width, end_width);
</code></pre>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=7" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: jinfagang
cooked: <p>I thin no good way to do this</p>
Slice a tensor on its outermost dimension

username: vimalthilak
cooked: <p>Is there a better or more obvious way to do the following operation:</p>
<pre><code class="lang-auto">A = torch.randn(3, 3, 4)
temp_m, temp_am = A.max(dim=0)
print(A)
print(temp_am.shape, temp_m.shape)
print("---max----")
print(temp_m)
print("---indexed---")
indexed_max = torch.gather(A, 0, temp_am.expand_as(A))[0]
print(indexed_max)
print(torch.all(indexed_max == temp_m))
</code></pre>
<p>My intent is to compute the max value of one tensor and use the argmax output to index other tensors</p>

username: LeviViana
cooked: <p>Maybe this is what you want:</p>
<pre><code class="lang-python">import torch
A = torch.randn(3, 3, 4)
temp_m, temp_am = A.max(dim=0)
print(A)
print(temp_am.shape, temp_m.shape)
print("---max----")
print(temp_m)
print("---indexed---")
indexed_max = torch.gather(A, 0, temp_am.expand_as(A))[0]
print(indexed_max)
print(torch.all(indexed_max == temp_m))

### More direct way ####

idx = A.argmax(0)
result = A.gather(0, idx.unsqueeze(0))
print(result)
print(torch.all(result == temp_m))
</code></pre>
Question on generator.zero_grad() in DCGAN

username: marchinidavide
cooked: <p>TL;DR<br>
Accumulate gradient over multiple batches on GANs with low memory.</p>
<p>Hi everybody,<br>
I would like to implement gradient updates over multiple minibatches (as described by <a class="mention" href="/u/alband">@albanD</a> in <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/18">https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/18</a>) in a GAN.</p>
<p>I would like to call optimizerG.step(), let’s say, only every 4 batches and accumulate gradients for the generator as described in the albanD second example in the answer  at the link above.<br>
In DCGAN example <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html#training" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html#training</a> we call for every batch the netG.zero_grad() just before the update of the generator and this prevents the gradient accumulation.</p>
<p>Any way I can deal with it?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I am not sure actually how to do that.<br>
Maybe you can do multiple minibatches updating just one network, then a few minibatches updating just the other?</p>

username: marchinidavide
cooked: <p>Thank you, had already started in this direction …<br>
I stopped because I wasn’t sure about optimising using data from different batches to first get D(G(fake_z_d)) (and then optimize D) and then optimize G under loss from D(G(fake_z_g)) …</p>
<p>Anyone aware of any “empirical rule” saying one should optimize D and G using z from the same batch?</p>
<p>(I think it shouldn’t matter … LF to post a MWE when and if I get it)</p>
What is the most convenient way to make model weights dependent on some parameter?

username: universome
cooked: <p>Imagine I have a neural network <code>f(x; w)</code>, where <code>x</code> is my input and <code>w</code> is its weights vector.<br>
Weights vector <code>w</code> itself depends on some parameter <code>u</code>, i.e. <code>w = g(u)</code>, where <code>g(.)</code> is some function. I want to do optimization for <code>u</code>. What is the most convenient way to do it?</p>
<p>To provide some example, let <code>w = sin(u) / ||u||</code> (here <code>||u||</code> is the norm of <code>u</code>). And our model architecture <code>f(x; w)</code> is ResNet-18. How can we optimize for <code>u</code> in this case? As far as I understand, if I just make model to take additional parameter <code>u</code> during initialization, compute <code>w</code> and set layers parameters to <code>w</code> this will not work. The problem here is that each layer uses <code>nn.Parameter()</code> under the hood and  <code>nn.Parameter()</code> ignores the history of computation (we have computed <code>w = torch.sin(u) / torch.norm(u)</code> and set layers parameters to <code>w</code>) and gradients for <code>u</code> will not be computed. Currently, I see the following two solutions:</p>
<ul>
<li>Before each iteration, compute <code>w</code> from <code>u</code> and update each layer parameters. Then, when gradient with respect to <code>w</code> is computed, update <code>u</code> manually (if the gradient <code>dw/du</code> is not that difficult to compute manually).</li>
<li>Rewrite ResNet-18 completely from scratch in such a way, that it takes computed <code>w</code> as input and does not use nn.Parameter() and calls everything via torch.nn.functional. In this way the gradient with respect to <code>u</code> will be computed automatically (so, it’s like a tensorflow before 2.0 version).</li>
</ul>
<p>Both of these solutions are quite tedious. Are there any better alternatives?</p>

username: universome
cooked: <p>I went via the second way. For example, here is my code for Linear layer:</p>
<pre><code class="lang-auto">import torch.nn.functional as F

class LinearOp:
    def __init__(self, weights, bias=None):
        self.weights = weights
        self.bias = bias

    def __call__(self, X):
        return F.linear(X, self.weights, self.bias)
</code></pre>
<p>It’s not handy to write such a class for each layer (convolutional, batchnorm, etc)</p>
Why the forward time are different?

username: pipibo
cooked: <p>I set up a Net, and test the forward time with the random input:</p>
<blockquote>
<p>test_data = torch.randn(1, 500, 3).cuda()<br>
net = Net().cuda()<br>
net.load_state_dict(torch.load(‘Net.pth’))<br>
net.eval()<br>
t1 = time.process_time()<br>
output = net(test_data)<br>
t2 = time.process_time()<br>
print('forward time: ',t2-t1)</p>
</blockquote>
<p>and get the forward time:</p>
<blockquote>
<p>forward time:  0.0874945260000004</p>
</blockquote>
<p>but when I use this model in real-world data in my task, I get a different time:</p>
<blockquote>
<p>forward time:  0.3340185280000014</p>
</blockquote>
<p>the real-world data has the same shape of my test data, why the forward time are different? and how to reduce the forward time in  real-world task?</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you’re running on gpu, you need to add a <code>torch.cuda.syncrhonize()</code> before every call to <code>time.process_time()</code> as the cuda api is asynchronous.</p>

username: pipibo
cooked: <p>Hi, I try to add this, but the time doesn’t change much, the first is about 0.08 and the later is 0.3+</p>

username: albanD
cooked: <p>Are you running both experiements on the same computer? With the exact same code?<br>
Does your real world data has some weird properties (like almost zero)?</p>

username: pipibo
cooked: <p>Yes, all experiments are on the same computer with the same code, the real-world data is ordinary.</p>

username: albanD
cooked: <p>Could you send a small sample so that I can try and reproduce it here locally?</p>

username: pipibo
cooked: <p>Sorry, this code belongs to our lab and is not allowed to be distributed without permission.<br>
What causes the difference in forward time between the test and the real world task in general?</p>

username: albanD
cooked: <p>Well I can’t think of anything really <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
It may be some discrepancy in your timing scripts?<br>
It may be external factors?<br>
It may be that they don’t actually have the same size?<br>
It may be that your net does different things depending on the values (like a rnn where the length is controlled by the network itself)?</p>

username: pipibo
cooked: <p>Alright, I’m going to examine the code and the data carefully. thanks!</p>
How to append tensors

username: vaibhavkumar049
cooked: <p>How can we append tensors<br>
So my situation is I have a for loop that goes through all rows from a 2-D tensor and then does argmax in each row, and returns a tensor position of the maximum value in that row. And finally, I want to append all those a value and creates a 1-D tensor<br>
How can I achieve that</p>

username: ptrblck
cooked: <p>If you want to get the maximal indices for each row, you could just call <code>.argmax(1)</code>:</p>
<pre><code class="lang-python">x = torch.randn(10, 15)
x.argmax(1)
</code></pre>

username: vaibhavkumar049
cooked: <p>thank you <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
it works</p>
Caffe2 cmake exports INTERFACE_COMPILE_OPTIONS when building from source

username: ondrej-smola
cooked: <p>I am having trouble creating libtorch binary distribution to be exactly the same as the one downloadable from Pytorch website (windows 10, pytorch master, VS2017, cmake 3.14, python 3.7, c++ 17, cuda 10, cudnn 7.6) - need to compile with protobuf 3.7.1 due to <a href="https://github.com/pytorch/pytorch/issues/14573" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/14573</a></p>
<pre><code class="lang-auto">python tools/build_libtorch.py
python setup.py sdist --format=zip
</code></pre>
<p>Everything compiles OK but caffe2 cmake module <code>Caffe2Targets.cmake</code> exports</p>
<pre><code class="lang-auto">set_target_properties(caffe2 PROPERTIES
  INTERFACE_COMPILE_DEFINITIONS "_THP_CORE;AT_PARALLEL_OPENMP=1"
  INTERFACE_COMPILE_OPTIONS "/Z7;/EHa;/DNOMINMAX;/wd4267;/wd4251;/wd4522;/wd4522;/wd4838;/wd4305;/wd4244;/wd4190;/wd4101;/wd4996;/wd4275;/bigobj;-openmp"
  INTERFACE_INCLUDE_DIRECTORIES "C:/Program Files/NVIDIA Corporation/NvToolsExt/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "protobuf::libprotobuf;c10;Threads::Threads"
)
</code></pre>
<p>they come from <code>setup.py</code></p>
<p>but these options are not present in the downloadable libtorch archive. Only</p>
<pre><code class="lang-auto">set_target_properties(caffe2 PROPERTIES
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "protobuf::libprotobuf;c10;Threads::Threads;caffe2::mkl"
)
</code></pre>
<p>Resulting in multiple<br>
<code>cl : Command line warning D9025 : overriding '/EHs' with '/EHa'</code>.<br>
warnings during my source compilation</p>
<p>I am fairly new to c++ and cmake so maybe I am just missing some simple part.</p>
<p>Thank you</p>
Pytorch tutorial - Training a classifier : TypeError with Dataloader on pytorch classifier with CIFAR 10 dataset

username: rickantonais
cooked: <p>I’m using Pytorch version 1.1.0 installed with fast.ai library and trying to get to know more about the framework about how it works.</p>
<p>I’ve been going through the blitz tutorial and on the one with training a classifier with CIFAR10 dataset, I’m stuck at a certain point where we use the Data Loader on the training sample :</p>
<pre><code class="lang-auto"># get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()
</code></pre>
<p>When I want to run this code, it gives me the following error :</p>
<pre><code class="lang-auto">TypeError: object() takes no parameters
</code></pre>
<p>I don’t understand why it does this error, I’ve tried to uninstall and re-install yet it still keep doing the same error</p>

username: tymokvo
cooked: <p>Welcome to the PyTorch community! <img src="https://discuss.pytorch.org/images/emoji/apple/smile_cat.png?v=9" title=":smile_cat:" class="emoji" alt=":smile_cat:"></p>
<p>It’s a bit hard to tell the context without seeing the rest of your code. Also, I’m not sure what kind of wrapping fast.ai has done, but this could be coming from their library.</p>
<p>The typical way to interact with datasets in PyTorch looks like something like this:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; mnist = MNIST(r"some root directory", download=True, train=True, transform=ToTensor())
&gt;&gt;&gt; data = DataLoader(mnist)
&gt;&gt;&gt; for num, (input, label) in enumerate(data):
...     print(num)
</code></pre>
<p><code>data</code> in this case, an instance of the <code>DataLoader</code> class is itself an iterator. You can use normal python looping constructs on this. You shouldn’t have to call <code>.next()</code> directly.</p>

username: rickantonais
cooked: <p>Thank you for your answer!<br>
The code comes from the official PyTorch training a classifier tutorial <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py" rel="nofollow noopener">here</a></p>
<p>EDIT : Just found the mistake…</p>
<p>In the code below, I’ve not put “()” after the function <code>ToTensor</code></p>
<pre><code class="lang-auto">transform = transforms.Compose([transforms.ToTensor, 
                               transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))])
</code></pre>
<p>Which led to the TypeError. But I’ve got confused as the error pointed at the lines I’ve mentioned before on my first post.</p>
<p>Sorry for the inconvenience and thank you for your answer!</p>
Combinatorial Concatenation

username: Soumyadip_Ghosh
cooked: <p>Hi,</p>
<p>I have tensor X of shape (a,b) and tensor Y of shape (c,d)<br>
I would like the output tensor Z to be of shape (a,c,(b+d)), such that</p>
<p>Z[i][j] = [ith row of X, jth row of Y]</p>
<p>What is the best way to do this?</p>
<p>Best,<br>
Soumyadip</p>

username: Pranavan_Theivendira
cooked: <p>Hi ,</p>
<p>Try the following solution.</p>
<pre><code class="lang-auto">import torch

a, b, c, d = 2, 3, 4, 4
A = torch.tensor([[10, 20, 30], [40, 50, 60]]) # a = 2, b = 3
B = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]]) # c = 4 , d = 4

A = A [:, None, :]
B = B [None, :, :]

A = A.repeat_interleave(c, 1)
B = B.repeat_interleave(a, 0)

C = torch.cat((A, B), dim=2)

print(C.size())
</code></pre>
<p>Thanks</p>