akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Backpropagate through gradient step

Mammothbagpipe: <p>Sorry for my questions, though I am fairly new into this topic. I am trying to learn a model to guide a gradient step in MAP restoration. My first question is simply if it is possible to backpropagate a gradient step (with adam optimizer)?<br>
My second question is how to do this?</p>
<p>My code, for one step, is following (shotened):</p>
<pre><code class="lang-auto">...
input_img = nn.Parameter(input_img, requires_grad=False)
prior = nn.Parameter(prior, requires_grad=False)
img_ano = nn.Parameter(input_img.clone(),requires_grad=True)

MAP_optimizer = optim.Adam([img_ano], lr=map_step_size)
net_optimizer = optim.Adam(net.parameters(), lr=net_lr_rate)

gfunc = torch.sum((prior.view(-1, prior.numel()) - img_ano.view(-1,img_ano.numel())).pow(2))

MAP_optimizer.zero_grad()
gfunc.backward()

net.train()
out = net(img_ano.data())
img_ano.grad = img_ano.grad + out

MAP_optimizer.step() # Update img_ano
     
loss = diceloss(img_ano, input_seg)
loss.backward()  
net_optimizer.step() # Update network parameters
</code></pre>
<p>The problem I am having is that after loss.backward(), network parameters gradient is of “Nonetype” (zero) which is not true if we were aloud to backpropagate through the gradient step. How do I solve my problem?</p>

albanD: <p>Hi,</p>
<p>If you’re new to this, I would recommend using a library built for this like higher: <a href="https://github.com/facebookresearch/higher/">https://github.com/facebookresearch/higher/</a></p>

Mammothbagpipe: <p>Thank you for the library! But I don’t think it solves my problem, as I want to update the gradient before the optimizer step with:</p>
<pre><code class="lang-auto">out = net(img_ano.data())
img_ano.grad = img_ano.grad + out
</code></pre>
<p>This does not seem possible in the library as you need to send the loss function with the gradient step (optim.loss(loss_function)) and in my case I want to change the gradient I get after gfunc.backward(). Any suggestions?</p>

albanD: <p>You want to change that gradient in the inner step or the outer step?</p>

Mammothbagpipe: <p>The inner step. As described below:</p>
<pre><code class="lang-auto">inner_loss = l2_loss(img_ano,prior)
inner_loss.backward()

img_ano.grad += out(img_ano.data()) # Changing the gradient by adding the output of network

img_ano = img_ano + step_size * img_ano.grad # (restore_optimizer.step()) Here I want to take a corresponding step with adam optimizer instead

loss = diceloss(img_ano,target_img)
loss.backward() # Gather network params gradients, ie. backpropagate the gradient step 
net_optimizer.step() # Update network params 
</code></pre>
<p>Thanks for the help!</p>

albanD: <p>I think you can specify a <code>grad_callback</code> in higher if you want to modify them before the optimizer step <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Mammothbagpipe: <p>Oh, missed that argument. Thanks a lot! Hopefully this will do the work.</p>
How do linear combination using pytorch?

11152: <p>I am building a network that creates filters in a linear combination. But I don’t know if the approach is right. And it is also slow.</p>
<p>And I want to use the return value as weight.<br>
How do you do it?</p>
<pre><code class="lang-auto">class Layer (torch.nn.Module):
    def __init __ (self, in_ch, out_ch):
        super (Layer, self) .__ init __ ()
        self.out_ch = out_ch
        self.in_ch = in_ch
        self.weights = torch.nn.Parameter (torch.Tensor (out_ch, in_ch, 5))

    def forward (self, x):
        w = torch.empty (self.out_ch, self.in_ch, 3, 3) .to ('cuda')

        for i in range (self.out_ch):
            for j in range (self.in_ch):
                w [i] [j] = x [0] .mul (self.weights [i] [j] [0]) + x [1] .mul (self.weights [i] [j] [1]) + x [2] .mul (self.weights [i] [j] [2]) \
                            + x [3] .mul (self.weights [i] [j] [3]) + x [4] .mul (self.weights [i] [j] [4])
        return w

class Net (nn.Module):
    def __init__(self):
       self.layer = Layer (3, 16)
       
    def forward (self, x):
        weight = self.layer(x) # x shape is (3, 3, 3)
        x = F.conv2d (x, weight)
        return x
</code></pre>
<ol>
<li>Computation Speed ​​improvement</li>
<li>How to update parameters with parameters</li>
</ol>

albanD: <p>Hi,</p>
<p>You can use the version given in nn: <code>torch.nn.Linear()</code>.</p>
<p>To update the parameters, you can use the builtin optimizers in gneeral. You should check the tutorial on how to write a training loop.</p>
Argument 'input' (position 1) must be Tensor, not str

Jacob_K: <p>Hi,</p>
<p>I am loading images into a custom <code>Dataset</code> and trying to classify them. When I pass the image through my neural net, I get</p>
<p><code>TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not str</code></p>
<p>You can see a short notebook reproducing the problem <a href="https://s3.eu-central-1.amazonaws.com/sftpgateway-i-0d1ba0c6742828040/pytorch-speedo.html" rel="nofollow noopener">here</a>.</p>
<p>Can anyone spot what I am doing wrong?</p>
<p>Thanks,<br>
Jacob</p>

justusschock: <p>Can you print the inputs? Maybe they are still a dict or a tuple of key and value and you are trying to feed the key (which is a string) to the network?</p>

vmirly1: <p>In the last cell in that notebook, you have the following line:</p>
<pre><code class="lang-auto">   inputs, labels = data
</code></pre>
<p>since <code>data</code> is a dictionary, that assignment will only assigns the keys. I made an example dictionary <code>d</code> with two elements:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; d = {'a':1, 'b':2}
&gt;&gt;&gt; x, y = d
&gt;&gt;&gt; x, y
('a', 'b')
</code></pre>
<p>Yet, another problem is in your dataset class, where in <code>__getitem__</code> you defined sample as a dictionary, and then pass a dictionary to <code>self.transform()</code> function. The transformation functions only accept PIL Image as input not dictionary.</p>

Jacob_K: <p>Thanks that solved it for me.</p>

mikey_t: <p><a class="mention" href="/u/vmirly1">@vmirly1</a> what would be the proper way to set up the sample in <code>__getitem__</code> for the <a href="https://s3.eu-central-1.amazonaws.com/sftpgateway-i-0d1ba0c6742828040/pytorch-speedo.html" rel="nofollow noopener">example provided in this question</a>? To me, it looks like <a class="mention" href="/u/jacob_k">@Jacob_K</a>  did the same thing as <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">this example from pytorch</a>. In the pytorch example, they use <code>sample = {'image': image, 'landmarks': landmarks}</code> in creating the dataset class, and Jacob used <code>sample = {'image': image, 'label': label}</code>.<br>
Sorry if this is an obvious answer, but I’m new to pytorch and having a pretty hard time loading image files from a csv. <a class="mention" href="/u/jacob_k">@Jacob_K</a> would it be possible to post the code that you eventually got to work for this problem.</p>
Combine multiple faster RCNN models

loicdtx: <p>Hi,</p>
<p>I’m trying to train separately different components of a faster rcnn model (<code>torchvision.models.detection.fasterrcnn_resnet50_fpn</code>) on different datasets and combine them into a single model. My final objective is to detect tree species over a forest using overhead imagery. My training data is non exhaustive (most of the trees are unlabelled/of unknown species).<br>
For example I first train backbone + RPN + 2 classes classifier (tree/background) on a dataset with all trees identified (regardless of the species). Then I freeze the backbone and train a new multiclass classifier on a synthetic data made of trees of known species placed on a textured background image.<br>
Finally I combine the common backbone with rpn of first model and <code>RoIHeads</code> of second model.<br>
When taken separately the two models work quite well, but combining them hasn’t worked too well so far.</p>
<ul>
<li>Am I right to think different components of a model can be trained separately and later combined?</li>
<li>Where is the <code>box_regression</code> part of the model? Is it part of <code>model.RoIHeads.box_predictor</code>? Can it be separated from it?</li>
<li>Any other suggestion?</li>
</ul>
<p>KInd regards,<br>
Loïc</p>
Backpropagating with gradient function instead of loss function

Wolf_Nuyts: <p>Hi,</p>
<p>I have a model and i need to train it with a self calculated gradient instead of a loss function but how do i backpropagate this gradient? I have a model and optimizer:</p>
<p>model = Model()<br>
optimizer = optim.Adam(model.parameters(), lr=1e-4)</p>
<p>now i can calculate the output and the gradient:</p>
<p>optimizer.zero_grad()<br>
out = model(input)<br>
gradient = my_custom_gradient_function(out)</p>
<p>And normally i use now loss.backward() and optimizer.step() when i have a loss function but how can i backpropagate now with the calculated gradient?</p>

Wolf_Nuyts: <p>Is using the following correct?:</p>
<p>out.backward(gradient)<br>
optimizer.step()</p>
<p>and if yes why and what is it doing exactly?</p>

albanD: <p>Hi,</p>
<p>Yes that looks good.<br>
Another option is to write a custom Function as described <a href="https://pytorch.org/docs/stable/notes/extending.html">here</a>. And then you can wrap your loss function with this.</p>
Non-autodiff-computationally tractable gradients in forward pass

drezap: <p>Hi -</p>
<p>Dumb question! May be I haven’t encountered it in the docs.</p>
<p>Suppose due to transformation and some bad functional form, I’m not able to build an expression graph in the forward pass. However, I’m able to do some analytic calculations and the gradient for the ill-formed loss function is tractable.</p>
<p>Is there a way to substitute <code>grad</code> or gradients WRT a loss function, with something I’ve implemented, for example, using finite differencing?</p>
<p>I’m sure my deep learning understanding is rather poor.</p>
<p>Thanks,</p>
<p>Andre</p>

albanD: <p>Hi,</p>
<p>You can check <a href="https://pytorch.org/docs/stable/notes/extending.html">this section</a> of the doc that explains how to specify the backward formula for a forward that is not differentiable.</p>
ModuleNotFoundError: No module named 'Custom_Net'

Mthathsara: <pre><code> Custom_Net is one of my own network architecture.I trained this network in my own computer GPU. But in order to test it I just had to transfer to Google Colab, then I had to face the above Error. 
 
I figured out the problem and I just got resolved it. This topic is for helping someone who is going to struggle with this error.

Since I have used 'torch.save(PATH)' when I'm saving my model after training, this problem has occurred. This code snippet just keeps path to class module, which is having the network architecture, not the class. That was the reason for the problem I faced. 

So I trained my model again in Google Colab, then I got resolved from the problem. Since my network is small it got trained very quickly. Thanks. 
Think this will help someone!!!!!!! :smiley: :smiley: :smiley:</code></pre>
What is the meaning of a tensor shape starting with 0?

WaterKnight: <p>When I prints shape from a tensor I get: <code>torch.Size([0, 1, 1004, 1002])</code>.</p>
<p>This means that there is no element in that tensor?</p>

albanD: <p>Hi,</p>
<p>Yes that means that the Tensor is empty.</p>
How gradient flows across torch.where(), tensor.min()?

Deepak: <p><strong>Question:</strong></p>
<ol>
<li>When I will do: <code>e1.backward()</code>, would the gradient flow properly back to the parameters of the network?</li>
<li>How does this <code>torch.where()</code> statement varies from <code>d1[c2 &lt;= 0] = 0.0</code> , or if-else statements (if implementable here) ?</li>
<li>Kindly suggest some good implementations of the mask, threshold operations allowing gradient flow across them?</li>
</ol>
<p><strong>Context:</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6bd8e46d03304aa65c552b827c8adf82a49d4070.png" alt="computatuion_graph" data-base62-sha1="fo3Mib7fsQl74LVsN0RijGxMnsY" width="224" height="417"><br>
Please see the attached image for the computation flow (roughly).</p>
<p><code>d1</code> is the modified <code>c1</code> based on the condition or mask created by <code>c2</code></p>
<p><code>a</code>: is a tensor of shape [16,3,256,256] <em># rgb image batch</em><br>
<code>c1, c2</code>: single-channel tensors [16,1,256,256]</p>
<p>The chain a-b2-c2 includes operations like torch.round(), modulo, view, squeeze, stack, min, clone, etc.</p>
<p>I want to update the parameters of the ‘Network’ based on the gradient calculated from <code>e1</code>, considering the fact that both <code>c1</code>, <code>c2</code> depends upon <code>a</code> (output of network).</p>
<p><strong>Code:</strong></p>
<pre><code class="lang-auto">...
c2 = b2.min(dim = 0, keepdim = True)[0]
d1 = torch.where(c2 &lt;= 0, torch.tensor(0,device='cuda:0',dtype = torch.float32,requires_grad=True), c1)
...
e1.backward()
</code></pre>

albanD: <p>Hi,</p>
<p>The autograd will compute the gradient of the op.<br>
So a good mental model is how much the output varies if you change the input by an epsilon.<br>
Here you see that if c2 moves by eps, nothing changes (except close to 0 but that is a special case). So the gradient flowing back along c2 will be 0.<br>
For c1, all the values that were overwritten as well, if you change them, it does not change the result. So the gradient is going to be 0 for them<br>
For the values that were kept as-is, then the gradient will be exactly the same as the one flowing from e1.</p>
<p>All the indexing/torch.where, etc will follow this logic and will give you the same gradient.</p>
How to freeze layer on mobilenet v2?

Wei_Wong: <p>What should I do to freeze the first N layers in (features) of mobilenetV2?<br>
Thank you!</p>

sambaths: <p>Please see this <a href="https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088">post</a></p>
How to use spacy word vectors in torchtext

greedsin: <p>spacy provides german word vectors <a href="https://spacy.io/models/de" rel="nofollow noopener">spacy model de</a>, so I want to make use of them with pytorch and torchtext.</p>
<pre><code>word_vectors = torch.FloatTensor(spacy_de.vocab.vectors.data)
emb = nn.Embedding.from_pretrained(word_vectors)
TEXT = data.Field(tokenize=tokenize,lower=True)
TEXT.build_vocab(vocab,vectors=word_vectors)
</code></pre>
<p>However when i try to build <code>TEXT.build_vocab</code> I get the following error message:</p>
<pre><code>ValueError: Got input vectors of type &lt;class 'torch.Tensor'&gt;, expected str or Vectors object
</code></pre>
<p>So i looked  <code>Vector</code> up in the torchtext documentation <a href="https://torchtext.readthedocs.io/en/latest/vocab.html#vectors" rel="nofollow noopener">torchtext vectors</a> and it seems I need to load them first. Is there a solution where I do not have to save the spacy word vectors to a file first? And if not, what is the correct way to save spacy word vectors that later I can load them ?</p>

sai_m: <p>wondering, whether have you tried loading vectors directly instead of converting into tensors?</p>

greedsin: <p>No this this does not work because spacy word vectors are of type spacy.vectors.Vectors which is a different class than torchtext’s Vector class</p>
How to generation / update parameter?

11152: <p>Hello everybody!!</p>
<p>i want weight generation and weight update<br>
So I want to know how to update the network that generates the weights.</p>
<pre><code class="lang-auto">class Network:
  def __init__(self):
    self.conv = nn.Conv2d(3, 64, kernel_size=kernel_size, stride=1, padding=1)

  def forward(self, x)
    filters = generation_weight_network()
    self.conv.weight = torch.nn.Parameter(filters)
    x = self.conv(x)
    return x
</code></pre>
<p>what is the problem??</p>

YeongHwa_Jin: <p>If you want to assign new weight in conv module, you delete the weight in the module first.</p>

11152: <p>I want to create a filter and assign it as the weight of conv.<br>
So I want to update both the weight of the network that creates the filter and the original network.</p>
Memory efficient weighted sum of channels

Soham_Tamba: <p>Inputs:</p>
<ol>
<li>I = Tensor of dim (N, C, X)      (Input)</li>
<li>W = Tensor of dim (N, X, Y)    (Weight)<br>
Output:</li>
<li>O = Tensor of dim (N, C, Y)     (Output)</li>
</ol>
<p>I want to compute:</p>
<pre><code class="lang-auto">I = I.view(N, C, X, 1)
W = W.view(N, 1, X, Y)
PROD = I*W
O = PROD.sum(dim=2)
return O
</code></pre>
<p>without incurring <strong>N * C * X * Y memory overhead</strong>.</p>
<p>Basically I want to calculate the weighted sum of a feature map wherein the weights are the same along the channel dimension, without incurring memory overhead per channel.</p>
<hr>
<p>Maybe I could use</p>
<pre><code class="lang-auto">from itertools import product

O = torch.zeros(N, C, Y)
for n, x, y in product(range(N), range(X), range(Y):
    O[n, :, y] += I[n, :, x]*W[n, x, y]
return O
</code></pre>
<p>but that would be slower (no broadcasting) and I’m not sure how much memory overhead would be incurred by saving variables for the backward pass.</p>

googlebot: <p>You can change the loop you’ve written to iterate over X only. I’m not 100% sure that this will work well with autograd, but I think it will (i.e. copies of O won’t be stored because of in-place addition, and I&amp;W slicing overhead is constant). Well, there is an option to put this loop in autograd.Function or checkpoint… Unfortunately, I don’t know of better ways to do this, maybe addbmm.</p>
How to totally free allocate memory in CUDA?

themoonboy: <p>Let me use a simple example to show the case</p>
<pre><code class="lang-auto">import torch 
a = torch.rand(10000, 10000).cuda()  # memory size: 865 MiB
del a
torch.cuda.empty_cache()   # still have 483 MiB
</code></pre>
<p>That seems very strange, even though I use “del Tensor” + torch.cuda.empty_cache(), there are still more than half memory left in CUDA side (483 MB in my case above). Is there any approach to totally remove these unused memory from GPU when the old reference is no longer needed?</p>

ptrblck: <p>If you are checking the memory via <code>nvidia-smi</code>, note that the CUDA context will also use memory.<br>
The allocated and cached memory will be freed using your code snippet:</p>
<pre><code class="lang-python">print(torch.cuda.memory_allocated())
print(torch.cuda.memory_cached())
&gt; 0
&gt; 0

a = torch.rand(10000, 10000).cuda()  
print(torch.cuda.memory_allocated())
print(torch.cuda.memory_cached())
&gt; 400556032
&gt; 400556032

del a
print(torch.cuda.memory_allocated())
print(torch.cuda.memory_cached())
&gt; 0
&gt; 400556032

torch.cuda.empty_cache()   
print(torch.cuda.memory_allocated())
print(torch.cuda.memory_cached())
&gt; 0
&gt; 0
</code></pre>
Batch Normalization Over Sequence Does not Generalize to Validation

Amit_Moryossef: <p>I have videos of variable lengths, where each frame is represented as a feature vector I create of size 1000.<br>
With a batch size of 16, my tensors are in the shape <code>(16, 1000, LEN)</code> - where <code>LEN</code> is the maximum length of a video in this batch.</p>
<p>If I instantiate a batch norm considering the 1000 dimensions as different channels:</p>
<pre><code class="lang-python">self.batch_norm = nn.BatchNorm1d(1000)
</code></pre>
<p>and in the forward-pass run <code>self.batch_norm(tensor)</code>, my <code>training_loss</code> goes down, but my <code>validation_loss</code> stays stagnant - and thus, 0% accuracy for my task.</p>
<p>Without using the batch norm, my model trains well - <code>validation_loss</code> goes down, and accuracy is <code>65%~</code></p>
<hr>
<p>Perhaps this problem emerges from the padded sequences, where if I batch a video of 10 frames, with a video of 20 frames, I create 10 frames of zeros.</p>
<p>To check that, instead of padding with zeros, I tried padding with the last frame of that video, so it is a valid frame. However, the results are the same.</p>

ptrblck: <p>The batch size might be too small, if you are dealing with different distributions and the batchnorm layers might estimate noisy running stats.<br>
Would it be possible to increase the batch size and lower the sequence length for the sake of debugging?<br>
Alternatively you could also change the <code>momentum</code> to weight the updates a bit less.</p>

Amit_Moryossef: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Playing with momentum did help get this training correctly, although after sweeping through a large range of momentums, the best model on the validation set is without using batch norm at all.</p>
<p>Thanks again, consider this solved</p>
RuntimeError: expected device cpu but got device cuda:0

Mthathsara: <p>I’m getting this error for my following code snippet.<br>
def feature_map_running_mean(self, x):<br>
running_mean = torch.zeros(x.size(1))<br>
for i,im in enumerate(x,1):<br>
running_mean += im.mean(dim=(1,2))<br>
return running_mean<br>
This code is calling in between every convolutional layer in VGG16. Then I try to run my code for a pre-trained model of VGG16. I got the above error. Can anyone please solve this problem as quickly as possible.<br>
Thanks</p>

Mthathsara: <p>I got solved this problem. Issue was with</p>
<pre><code>   running_mean = torch.zeros(x.size(1))
</code></pre>
<p>line, it has been changed to</p>
<pre><code>  running_mean = torch.zeros(x.size(1)).cuda()
</code></pre>
<p>Think this will help others well. <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"> <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"> <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>
Reuse part of module without accumulating gradient

GenjiB: <p>With Encoder-Decoder based models, we can encode the feature from the inputs by encoding and decoding.</p>
<p>My question is how to prevent duplicate gradient calculating.<br>
e.g., <strong>input -&gt; enc -&gt; features  -&gt;  dec -&gt; outputs.</strong><br>
Then we would like to obtain the feature of outputs to compute loss like triplet loss: <strong>enc(outputs)</strong><br>
Due to enc(outputs) would pass encoder twice, we would like to obtain the gradient for the first time.</p>
<p>I get two options:<br>
1: <strong>enc(outputs.detach())</strong><br>
2: <strong>torch.no_grad: enc(outputs)</strong></p>
<p>I am not sure which one is the correct way.</p>

ptrblck: <p>If you would like to use the output of <code>enc(output)</code> as the target only and it thus wouldn’t require gradients, I think both methods would work.</p>
<p>Feel free to correct me, in case I misunderstood the use case.</p>

GenjiB: <p>Thanks for your reply. But, I would like to use the feature of <strong>enc(output)</strong>  for metric learning loss. Due to output already containing the gradient from <strong>encoder</strong>, I would like to ignore the computing gradient while passing the encoder for the second time.</p>
<p><strong>input -&gt; enc -&gt; features -&gt; dec -&gt; outputs -&gt; enc (not computing graident) -&gt; features</strong></p>

ptrblck: <p>If <code>features</code> from this calculation: <code>features = enc(output) # no gradient</code> does not require gradients, you should be fine using either approach.<br>
This would be the use case, e.g. if you are using this <code>features</code> output as the target for a specific loss calculation.</p>
What is train_iterator

Imene.b: <p>i am trying to understand how to create neural Network with pytorch, and i have find a code where it  utilize a variable not declared above ; train_iterator and valid_iterator<br>
what is this variables? and how can i create them?<br>
note:<br>
the code create train_data via ImageFolder<br>
‘’’ train_data = torchvision.datasets.ImageFolder(root= train_data_path,transform=transforms)’’’<br>
the code is:<br>
‘’'def train(model, optimizer, loss_fn, train_loader, val_loader, device=‘cpu’):<br>
epochs =20</p>
<pre><code>for epoch in range(epochs):
    print('training epoche  {}'.format(epoch))
    print('-------------------------------------------------')
    training_loss = 0.0
    valid_loss = 0.0
    model.train()
    for batch in train_loader:
       
        optimizer.zero_grad()    # reset the gradients to zero for the next batch iteration, if not the next batch
                                 # would have to deal with the previeuse batch's gradients as well as its own
        
        inputs, targets = batch    
        inputs = inputs.to(device)  # put inputs data to the device 
        target = targets.to(device)
        output = model(inputs)
        loss = loss_fn(output, target)
        loss.backward()             # compute gradients
        optimizer.step()            # use gradients, to adjust weights 
        training_loss += loss.data.item()
        print('loss =', loss)
    **training_loss /= len(train_iterator)**
</code></pre>
<p>‘’’</p>

ptrblck: <p>Unfortunately the creation of <code>train_iterator</code> is not shown in your code snippet, but I guess it might have been created via:</p>
<pre><code class="lang-python">train_iterator = iter(train_loader)
</code></pre>
<p>This would allow you to manually get the next batch via <code>next(train_iterator)</code>, but you would have to take care of the <code>StopIteration</code> manually.</p>

Imene.b: <p>tanks for your replay</p>
Transferring everything inside the nn.Module class to Cuda

DrHB: <p>Suppose I have this model:</p>
<pre><code class="lang-auto">class DummYmodel(nn.Module):
    def __init__(self, x, m):
        super().__init__()
        self.x = torch.nn.Parameter(torch.tensor(x))
        self.m = torch.tensor(m)
        
    def forward(self, x):
        pass
</code></pre>
<p>I will init and put it on <code>Cuda</code></p>
<p><code>dummy = DummYmodel(2.0, 3).cuda()</code></p>
<pre><code class="lang-auto">&gt;dummy.x

Parameter containing:
tensor(2., device='cuda:0', requires_grad=True)

&gt;dummy.m

tensor(3)
</code></pre>
<p><code>dummy.m</code> is not on <code>cuda</code>. Is there a way to make sure when I transfer to <code>Cuda</code> everything inside the class get transferred to Cuda ?</p>

ptrblck: <p>All <code>nn.Parameters</code>, <code>nn.Modules</code> and buffers will be transferred to the specified device.<br>
To push <code>self.m</code> to the device, you should register it as a buffer via:</p>
<pre><code class="lang-python">self.register_buffer('m', torch.tensor(m))
</code></pre>
[SOLVED] RuntimeError: Expected object of device type cuda but got device type cpu for argument #2 'mat2' in call to _th_mm

Dennis_Wu: <pre><code>
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 
torch.cuda.is_available() #True
word_model = word_model.to(device)
word_model.cuda()
for i,(x,y) in enumerate(dataset):
  if i &lt; 1:
    x = x.cuda()
    output = word_model(x)
    i+=1
    print(output)
  if(i&gt;=1):
    break
</code></pre>
<p>And it shows :</p>
<pre>RuntimeError: Expected object of device type cuda but got device type cpu for argument #2 'mat2' in call to _th_mm </pre>
<p>I guess that my input and model are not in the same device, but not sure why.</p>

Dennis_Wu: <p>below is my model, but i’m not sure if this is where the error come from</p>
<pre><code>
class MojiNet(nn.Module):

    def __init__(self,vocab_size,embedding_size,hidden_size,pretrained_embed,embed_dropout,model_dropout,num_layers,class_num):

        super(MojiNet,self).__init__()

        self.hidden_size = hidden_size

        if(pretrained_embed):
          self.embedding = nn.Embedding.from_pretrained(torch.FloatTensor(numpy_embed)).to(device)
        else:
          self.embedding = nn.Embedding(vocab_size,embedding_size)
        self.embed_dropout = nn.Dropout(embed_dropout)
        self.sent_gru = nn.GRU(embedding_size,hidden_size,num_layers,dropout=(0 if num_layers == 1 else model_dropout),bidirectional=True,batch_first=True)
        self.fc = nn.Linear(hidden_size,class_num)
        self.fc_dropout = nn.Dropout(0.5)
        # self.attn = Attn(hidden_size)
    def forward(self,x):
        x = self.embedding(x)
        x = self.embed_dropout(x)
        #(batch,max_len,embedding_size)
        y,_ = self.sent_gru(x)
        #batch,max_len,2*hidden_size(bidirectional)
        y = y[:,:,:hidden_size] + y[:,:,hidden_size:]
        #batch,max_len,hidden_size(head + tail)
        alpha = Attn(y,hidden_size)
        #batch,1,max_len
        r = alpha.bmm(y).squeeze(1)
        #batch,hidden_size
        h = torch.tanh(r)
        #batch,hidden_size
        output = self.fc(h)
        #batch,class_num
        output = self.fc_dropout(output)
        return output
</code></pre>

Dennis_Wu: <p>I create the Attention function Attn() outside the model, but im not sure if this is the problem,since it didn’t showed the information of it</p>

Dennis_Wu: <p>Okay, i just solved the problem by myself, the reason of this is the Attn() function which i wrote outside the model class as another def() function, and the Attn() function will not be moved to the GPU, so I create a new nn.Module class for Attn and i wrote : self.attn = Attn(hidden_size) in the model.</p>
Distributing binary python wheels which depend on pytorch with custom CUDA Kernels

bkmi: <p>The github project e3nn is attempting to get on pypi. We have pytorch as a dependency as well as custom cuda kernels. Our goal is to distribute precompiled binaries much like how pytorch is distributed, i.e. <code>pip install e3nn=0.0.1+cu101</code> for version 0.0.1 with binaries which are compatible with cuda 10.1. I am looking for guidance on how to get this done.</p>
<ol>
<li>
<p>We will need to compile to the manylinux standard. I have seen how manylinux pytorch itself is compiled and uploaded to pypi in the <a href="https://github.com/pytorch/builder/tree/master/manywheel" rel="nofollow noopener">builder repo</a>. Is it possible to get access to this docker image in order to compile our code against this environment? Where can I find it?</p>
</li>
<li>
<p>Furthermore, how does one create the +cu101 version option in pypi? This feature is completely undocumented and looking at the <a href="https://pypi.org/simple/torch/" rel="nofollow noopener">contents of pytorch’s repo on pypi.org</a>, there are no +cu101 versions available. How can we get this functionality as well?</p>
</li>
</ol>

ptrblck: <p>CC <a class="mention" href="/u/seemethere">@seemethere</a>, who might be familiar with the binary packages. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Changing the size of nn.Linear depending on example size

Bixqu: <p>I am using the nn.Linear layer as part of a graph classifier (in pytorch geometric). The network looks like this:</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = GraphConv(in_channels=768, out_channels=16) 
        self.conv2 = GraphConv(in_channels=16, out_channels=2) 
        # PROBLEM
        self.fc1 = nn.Linear(16 * 2, 2)

    def forward(self, data):
        x, edge_index, edge_weight = data.x, data.edge_index, data.edge_attr

        x = self.conv1(x, edge_index, edge_weight)
        x = F.relu(x)
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index, edge_weight)
        x = F.relu(x)

        #PROBLEM
        x = x.view(-1, 16 * 2)
        x = self.fc1(x)

        return F.softmax(x, dim=1)
</code></pre>
<p>Now ,I am feeding a list of graphs of different sizes, such as:</p>
<pre><code class="lang-auto">
[Data(edge_attr=[218, 1], edge_index=[2, 218], x=[203, 768], y=[1]),
 Data(edge_attr=[1306, 1], edge_index=[2, 1306], x=[1281, 768], y=[1]),
 Data(edge_attr=[244, 1], edge_index=[2, 244], x=[234, 768], y=[1])]
</code></pre>
<p>So I need to somehow be able to read the 0th shape of x before defining the size of my linear layer and doing the view reshaping. How can this be done?</p>

ptrblck: <aside class="quote no-group" data-username="Bixqu" data-post="1" data-topic="79523">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/958977/48.png" class="avatar"> Bixqu:</div>
<blockquote>
<p>How can this be done?</p>
</blockquote>
</aside>
<p>You could use the functional API via <code>F.linear</code> and define the <code>weight</code> and <code>bias</code> depending on your input shape.<br>
However, this would create a new parameter for each input shape and I’m not sure, if that’s what you want.</p>
<p>Usually you would add e.g. an adaptive pooling layer to make sure the incoming activation has always a defined shape.</p>
How to automatically free CUDA memory when using same reference (variable name) in torch operation?

themoonboy: <p>Hi, here is one toy code about this issue:</p>
<pre><code class="lang-auto">import torch 

torch.cuda.set_device(3)
a = torch.rand(10000, 10000).cuda()
# monitor cuda:3 by "watch -n 0.01 nvidia-smi"
a = torch.add(a, 0.0)
# keep monitoring
</code></pre>
<p>When using same variable name “a” in torch.add, I find the old a’s memory is not freed in cuda, it still exists even though the reference is updated, and I cannot reach tp original memory since the reference is gone. How could I make the memory of old reference automatically freed when use same variable name in left/right of torch operations?</p>

ptrblck: <p><code>a</code> will be freed automatically, if no reference points to this variable.<br>
Note that PyTorch uses a memory caching mechanism, so <code>nvidia-smi</code> will show all allocated and cached memory as well as the memory used by the CUDA context.</p>
<p>Here is a small example to demonstrate this behavior:</p>
<pre><code class="lang-python"># Should be empty
print('allocated ', torch.cuda.memory_allocated() / 1024**2)
print('cached ', torch.cuda.memory_cached() / 1024**2)
&gt; allocated  0.0
&gt; cached  0.0

# Initial setup
a = torch.rand(1024, 1024, 128).cuda()
print('allocated ', torch.cuda.memory_allocated() / 1024**2)
print('cached ', torch.cuda.memory_cached() / 1024**2)
&gt; allocated  512.0
&gt; cached  512.0

# torch.add will use a temp variable, as it's not inplace
a = torch.add(a, 0.0)
print('allocated ', torch.cuda.memory_allocated() / 1024**2)
print('cached ', torch.cuda.memory_cached() / 1024**2)
&gt; allocated  512.0
&gt; cached  1024.0

# Delete reference
a = 1.
print('allocated ', torch.cuda.memory_allocated() / 1024**2)
print('cached ', torch.cuda.memory_cached() / 1024**2)
&gt; allocated  0.0
&gt; cached  1024.0
</code></pre>
What does _single , _pair, and _triple do?

seyeeet: <p>i see<br>
<code>from torch.modules.utils import _single, _pair, _triple </code></p>
<p>what does <code>_pair </code>, <code>_single</code> ,  and <code>_triple</code>, do?</p>

ptrblck: <p>These are internal methods (marked via the underscore at the beginning), which repeat some arguments if necessary.<br>
E.g. for <code>nn.Conv2d</code> you can pass the <code>kernel_size</code> as a single integer or as a tuple.<br>
In the former case, the tuple will be created via <code>_pair</code> in <a href="https://github.com/pytorch/pytorch/blob/78d5707041ea4c3f9e5e8aa13f6858d54de5a6da/torch/nn/modules/conv.py#L334">this line of code</a>.<br>
So it’s used for convenience, otherwise you would have to specify the complete shapes for each argument.</p>
Modifying the parameters of a layer manually

sezer: <p>Hi,</p>
<p>I used to train the last fully connected layer manually and write it back to the model as follows:</p>
<pre><code class="lang-auto">for param in model.fc.parameters():
    fc_params = param.data.clone().detach().t()

fc_params = custom_train(fc_params)

state_dict = model.fc.state_dict()
for name, param in state_dict.items():
    state_dict[name].copy_(fc_params.t())
</code></pre>
<p>I calculate the gradients in my own function. So I define the optimizer without the last layer:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam([{"params": model.layer1.parameters(), "lr": lr},
                              {"params": model.layer2.parameters(), "lr": lr}])
</code></pre>
<p>I have just updated to PyTorch 1.5 and now it doesn’t work. The error message says:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [50, 16]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<p>What is the proper way of doing this?</p>
<p>Thank you.</p>
Looking for real-use case Datasets to perform Domain Adaptation for Image Classification/ Recognition/ Obj. Detection

Rajat_Sharma: <p>Hi PyTorch Community,</p>
<p>I am trying to test my architecture (proposed model) for Domain Adaption (Semi-Supervised or unsupervised) for Image Classifications or Object Detection (at later stage).</p>
<p>Can somebody suggest me similar datasets so that I can test my proposed technique ?</p>
<p>I found some related datasets:<br>
MNIST→ USPS<br>
USPS → MNIST<br>
SVHN → MNIST<br>
Office 31 dataset</p>
<p>Can somebody suggest me some real use case datasets or scenarios. If you find some related threads, please provide me.</p>
<p>Thanks in advance !</p>
What does copy_initial_weights do in the higher library?

pinocchio: <p>I was reading the docs (<a href="https://higher.readthedocs.io/en/latest/toplevel.html" class="inline-onebox" rel="noopener nofollow ugc">Top-Level Functions — higher 0.2.1 documentation</a>) for the context manager ( <code>higher.</code> <code>innerloop_ctx</code>) and it said:</p>
<blockquote>
<p><code>higher.</code> <code>innerloop_ctx</code> ( <em>model</em> ,  <em>opt</em> ,  <em>device=None</em> ,  <em>copy_initial_weights=True</em> ,  <em>override=None</em> ,  <em>track_higher_grads=True</em> )<br>
…<br>
<strong>copy_initial_weights</strong>  – if true, the weights of the patched module are copied to form the initial weights of the patched module, and thus are not part of the gradient tape when unrolling the patched module. If this is set to False, the actual module weights will be the initial weights of the patched module. This is useful when doing MAML, for example.</p>
</blockquote>
<p>but I don’t understand what that means. Perhaps because english is not my first language.</p>
<p>What is confusing to me is that I thought the whole point of this library is to be able to differentiate the unrolled optimizer. Thus, not being able to have the weights as part of the unrolled optimization seems strange to me. Why would anyone want to set this flag to true…ever? So I don’t understand what:</p>
<blockquote>
<p>thus are not part of the gradient tape when unrolling the patched module</p>
</blockquote>
<p>that means I guess.</p>
<p>I do understand what MAML is suppose to be.</p>
<hr>
<p>Related:</p>
<ul>
<li><a href="https://github.com/facebookresearch/higher/issues/30" class="inline-onebox" rel="noopener nofollow ugc">Improving documentation for copy_initial_weights · Issue #30 · facebookresearch/higher · GitHub</a></li>
<li>SO: <a href="https://stackoverflow.com/questions/60311183/what-does-the-copy-initial-weights-documentation-mean-in-the-higher-library-for" class="inline-onebox" rel="noopener nofollow ugc">machine learning - What does the copy_initial_weights documentation mean in the higher library for Pytorch? - Stack Overflow</a></li>
</ul>

pinocchio: <p>all links:</p>
<ul>
<li>
<p>gitissue: <a href="https://github.com/facebookresearch/higher/issues/30" rel="nofollow noopener">https://github.com/facebookresearch/higher/issues/30</a></p>
</li>
<li>
<p>SO: <a href="https://stackoverflow.com/questions/60311183/what-does-the-copy-initial-weights-documentation-mean-in-the-higher-library-for" rel="nofollow noopener">https://stackoverflow.com/questions/60311183/what-does-the-copy-initial-weights-documentation-mean-in-the-higher-library-for</a></p>
</li>
<li>
<p>What does copy_initial_weights do in the higher library? <a href="https://discuss.pytorch.org/t/what-does-copy-initial-weights-do-in-the-higher-library/70384" class="inline-onebox">What does copy_initial_weights do in the higher library?</a></p>
</li>
<li>
<p>Why does MAML need copy_initial_weights=False? <a href="https://discuss.pytorch.org/t/why-does-maml-need-copy-initial-weights-false/70387" class="inline-onebox">Why does MAML need copy_initial_weights=False?</a></p>
</li>
</ul>