akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Masked_fill_ behaves differently on cpu and gpu

username: stan
cooked: <p><code>masked_fill_</code> seems to behave differently on cpu and gpu. It also produces an output different from its out-of-place counterpart <code>masked_fill</code> on cpu. Here is a minimum working example to demonstrate the behavior on cpu:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.5.0'
&gt;&gt;&gt; tensor_cpu = torch.LongTensor([[0], [1]]).expand(2, 4)
&gt;&gt;&gt; tensor_cpu
tensor([[0, 0, 0, 0],
        [1, 1, 1, 1]])
&gt;&gt;&gt; mask_cpu = torch.BoolTensor(
...     [[False,  True, False, False],
...      [False, False, False, False]]
...     )
&gt;&gt;&gt; mask_cpu
tensor([[False,  True, False, False],
        [False, False, False, False]])
&gt;&gt;&gt; tensor_cpu.masked_fill(mask_cpu, 3)        # expected behavior
tensor([[0, 3, 0, 0],
        [1, 1, 1, 1]])
&gt;&gt;&gt; tensor_cpu.masked_fill_(mask_cpu, 3)       # unexpected behavior?
tensor([[3, 3, 3, 3],
        [1, 1, 1, 1]])
</code></pre>
<p>and here is the gpu equivalent:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; tensor_cuda = torch.LongTensor([[0], [1]]).expand(2, 4).to('cuda')
&gt;&gt;&gt; tensor_cuda
tensor([[0, 0, 0, 0],
        [1, 1, 1, 1]], device='cuda:0')
&gt;&gt;&gt; mask_cuda = torch.BoolTensor(
...     [[False,  True, False, False],
...      [False, False, False, False]]
...     ).to('cuda')
&gt;&gt;&gt; mask_cuda
tensor([[False,  True, False, False],
        [False, False, False, False]], device='cuda:0')
&gt;&gt;&gt; tensor_cuda.masked_fill(mask_cuda, 3)      # expected behavior
tensor([[0, 3, 0, 0],
        [1, 1, 1, 1]], device='cuda:0')
&gt;&gt;&gt; tensor_cuda.masked_fill_(mask_cuda, 3)     # expected behavior
tensor([[0, 3, 0, 0],
        [1, 1, 1, 1]], device='cuda:0')
</code></pre>
<p>Apparently, the in-place <code>masked_fill_</code> gives a different output on the cpu. I’m guessing this has something to do with <code>expand</code> while initializing <code>tensor_cpu</code>; if I do not use <code>expand</code>, I get the expected output:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; other_cpu = torch.LongTensor(
...     [[0, 0, 0, 0],
...      [1, 1, 1, 1]]
...     )
&gt;&gt;&gt; other_cpu
tensor([[0, 0, 0, 0],
        [1, 1, 1, 1]])
&gt;&gt;&gt; other_cpu.masked_fill_(mask_cpu, 3)        # expected behavior
tensor([[0, 3, 0, 0],
        [1, 1, 1, 1]])
</code></pre>
<p>Am I missing something here?</p>

username: ptrblck
cooked: <p>Also tracked <a href="https://github.com/pytorch/pytorch/issues/39639">here</a>.</p>
Indexing Matrix

username: Alex31
cooked: <p>Suppose I have the following tensors:</p>
<pre><code class="lang-auto">N = 2
k = 3
d = 2

L = torch.arange(N * k * d * d).view(N, k, d, d)
L
tensor([[[[ 0,  1],
          [ 2,  3]],

         [[ 4,  5],
          [ 6,  7]],

         [[ 8,  9],
          [10, 11]]],


        [[[12, 13],
          [14, 15]],

         [[16, 17],
          [18, 19]],

         [[20, 21],
          [22, 23]]]])


index = torch.Tensor([0,1,0,0]).view(N,-1)
index
tensor([[0., 1.],
        [0., 0.]])
</code></pre>
<p>I now would like to use the index vector to pick out the corresponding matrices on the second dimension, i.e. I would like to get something like:</p>
<pre><code class="lang-auto">tensor([[[[ 0,  1],
          [ 2,  3]],

         [[ 4,  5],
          [ 6,  7]],]],


        [[[12, 13],
          [14, 15]],

         [[[[12, 13],
          [14, 15]]])
</code></pre>
<p>Any idea how I could achieve this?<br>
Thank you so much!</p>

username: ptrblck
cooked: <p>This should work:</p>
<pre><code class="lang-python">N = 2
k = 3
d = 2

L = torch.arange(N * k * d * d).view(N, k, d, d)
index = torch.tensor([0,1,0,0]).view(N,-1)
L[torch.arange(L.size(0)).unsqueeze(1), index]
</code></pre>
Highly varied results for different training seeds

username: anubhavm11
cooked: <p>I am trying CIFAR10 classification using an Alexnet type network and a multistepLR schedule, which reduces lr at epochs = 100, 150 and wd=1e-2. The training runs smoothly when the initial lr = 1e-3, but when I am using initial lr= 4e-3, I get random accuracy (~10%) with pytorch seed = 0, and around the expected accuracy (~85%) when I set the seed = 2. Is there any possible explanation for this behaviour, or is this solely due to a bug in my code? I am not posting any code for now as it is distributed among many files. Also, I am using kaiming initialization for the weights.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="anubhavm11" data-post="1" data-topic="84457">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/e79b87/48.png" class="avatar"> anubhavm11:</div>
<blockquote>
<p>Is there any possible explanation for this behaviour, or is this solely due to a bug in my code?</p>
</blockquote>
</aside>
<p>It’s most likely not a bug, but just a “lucky” and “bad” seed. Don’t try to optimize your code for a specific seed, as your training will be overall unstable and might break easily.<br>
Based on your description it seems the learning rate is too high, but apparently some seeds force the initialization methods to generate working parameters for the setup.</p>
Extracting weights from previous layers- EfficientNet

username: akshat91
cooked: <p>Hi Team,<br>
I have trained an image regression model using EfficientNetB5.<br>
I wanted to extract the weights from it’s previous layers because i want to use those weights to further train a boosting model.</p>
<p>Previously I used Densenet and I was able to extract it’s weights using indexing and using forward hooks.</p>
<p>Now since EfficientNet models doesn’t support indexing.<br>
Is their any way to extract the weights from the previous layers?</p>
<p>Any help would be really appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Regards,<br>
Akshat</p>

username: ptrblck
cooked: <p>Most likely your previous model used some <code>nn.Sequential</code> containers, which can be indexed to get the corresponding layer.<br>
If <code>EfficientNet</code> doesn’t use <code>nn.Seuqential</code> modules, you would have to directly access the desired modules via:</p>
<pre><code class="lang-python">model.layer.weight
</code></pre>
<p>where <code>layer</code> would be the name of the layer you would like to access.<br>
To get the layer names you could either <code>print(model)</code> or iterate all modules using <code>model.named_modules()</code>.</p>
The output of SyncBatchNorm

username: as754770178
cooked: <p>I view the code of <code>SyncBatchNorm</code> in <code>/usr/local/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py</code>.</p>
<pre><code class="lang-auto">@classmethod
    def convert_sync_batchnorm(cls, module, process_group=None):
        module_output = module
        if isinstance(module, torch.nn.modules.batchnorm._BatchNorm):
            module_output = torch.nn.SyncBatchNorm(module.num_features,
                                                   module.eps, module.momentum,
                                                   module.affine,
                                                   module.track_running_stats,
                                                   process_group)
            if module.affine:
                module_output.weight.data = module.weight.data.clone().detach()
                module_output.bias.data = module.bias.data.clone().detach()
                # keep reuqires_grad unchanged
                module_output.weight.requires_grad = module.weight.requires_grad
                module_output.bias.requires_grad = module.bias.requires_grad
            module_output.running_mean = module.running_mean
            module_output.running_var = module.running_var
            module_output.num_batches_tracked = module.num_batches_tracked
        for name, child in module.named_children():
            module_output.add_module(name, cls.convert_sync_batchnorm(child, process_group))
        del module
        return module_output
</code></pre>
<ol>
<li>
<p>why the weight and bias of bn layer need to exec <code>module_output.weight.data = module.weight.data.clone().detach()</code>, not  <code>module_output.weight.data = module.weight.data</code> as <code>running_mean</code> and <code>running_var</code>.</p>
</li>
<li>
<p>why must exec <code>convert_sync_batchnorm</code> after <code>torch.distributed.new_group</code></p>
</li>
<li>
<p>What parameters to synchronize by SyncBatchNorm? <code>weight</code>, <code>bias</code>, <code>running_mean</code> and <code>running_var</code> all synchronized?</p>
</li>
</ol>
Custom Loss for single object detection task

username: arjun_pukale
cooked: <p>My task is to create a single object detection model, It is doing 2 task: 1. Classification between 2 classes [cat, no object]<br>
2: Regression x and y co_ordinates.<br>
So I want to design a loss function such that it gives me 2 losses for classification and regression each.<br>
for classification I am using CrossEntropy Loss, but for regression I want to calculate the MSE loss only for the images containing cat. So here is a dummy example of such loss function I have created please check if it is right or wrong.</p>
<pre><code class="lang-auto">output = torch.tensor([[0.7,0.3,0.5,0.6], #(prob_cat, prob_no_object, x,y)
                       [0.2,0.8,100,100],
                       [0.8,0.2,0.4,0.5]])
target = torch.tensor([[1,0,0.4,0.8],
                       [0,1,0,0],
                       [1,0,0.3,0.4]])
def myloss(output,target):
    #classification loss (cat, no object)
    target_class = target[:,:2].argmax(1) #True class 
    output_class = output[:,:2] #Predicted Class
    l1 = nn.CrossEntropyLoss(reduction='sum')
    Loss1 = l1(output_class,target_class)

    #regression loss (x,y)
    prob_no_object = target[:,[1]]
    prob_object=(prob_no_object==0).float()
    target_bbx = target[:,[2,3]]
    output_bbx = output[:,[2,3]]
    l2 = nn.MSELoss(reduction='none')
    L2 = l2(output_bbx,target_bbx)
    # print(L2.shape)
    Loss2 = torch.sum(torch.mul(prob_object,L2))

    return(Loss1,Loss2/torch.sum(prob_object))
    #classification loss, regression_loss_per_cat_img_in_batch
</code></pre>

username: ptrblck
cooked: <p>The code looks generally alright. The <code>output</code> should contain raw logits, not probabilities for the <code>nn.CrossEntropyLoss</code>, but I assume you just used these values for easier debugging.</p>
<p>If you are concerned about the indexing, you could just use separate target tensors (and potentially different output layers in your model) for both tasks.</p>
Apply Linear layers after LSTM

username: jackeown
cooked: <p>I’m trying to implement a model which I would describe as an LSTM autoencoder although I’m not sure if it strictly meets a definition of one. I’m trying to make a model which can learn to represent a set of variable length sequences as fixed length vectors.</p>
<p>The forward function should take in a list (batch) of lists (sequences) of lists of floats.<br>
the <code>torch.nn.utils.rnn.pack_sequence</code> function is then used on this input before feeding it to an LSTM to encode it to a fixed length representation. Unlike a seq2seq model, I actually want a bottleneck, since the whole point of my model is obtaining a fixed-length representation. Therefore, I take only the hidden state as input to the decoder LSTM. (Also since the encoder LSTM can have multiple layers, I only take the last layer.) I actually use <code>repeat</code> to repeat the hidden states for each example in the batch to its appropriate length and then use <code>pack_sequence</code> again before feeding this into the decoder LSTM. LSTMs in pytorch have 3 components to their output : <code>output</code>, <code>hidden state</code>, and <code>cell state</code>. For the output of the decoder, I take the “output” component.</p>
<p>I would like to perform a <code>nn.Linear</code> layer to each timestep of the decoder output. The problem is that the <code>output</code> component of the decoder LSTM is a PackedSequence object.<br>
Is there a good way to do this? I wish that the Linear layer could just be applied to PackedSequence objects. What is the alternative? I can post my code below if it’s helpful. It’s somewhat minimal.</p>
Variable with requires_grad=False

username: codeknight11
cooked: <p>I am following someone else’s codebase for personal research purposes.<br>
While going through the code I came across several code snippets such as:</p>
<pre><code class="lang-auto">target_var = Variable(target_vec, requires_grad=False)

Variable(last_acc[name], requires_grad=False).view(-1)
</code></pre>
<p>I know that Variable has been deprecated and we should be using Tensors now. What doesn’t make sense to me here is that wasn’t Variable supposed to be used if gradients had to be calculated?</p>
<p>What’s the point of using “Variable” and then setting <code>requires_grad=False</code> ?<br>
That seems redundant to me.</p>
<p>In tensor lingo, the declaration should be like this right?</p>
<pre><code class="lang-auto">target_var = torch.Tensor(target_vec, requires_grad=False) 
</code></pre>

username: tom
cooked: <aside class="quote no-group" data-username="codeknight11" data-post="1" data-topic="84487">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/codeknight11/48/22986_2.png" class="avatar"> codeknight11:</div>
<blockquote>
<p>What’s the point of using “Variable” and then setting <code>requires_grad=False</code> ?</p>
</blockquote>
</aside>
<p>Before Variable and Tensor were merged, you could only do computation between tensors or between variables, but not mix them.</p>
<aside class="quote no-group" data-username="codeknight11" data-post="1" data-topic="84487">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/codeknight11/48/22986_2.png" class="avatar"> codeknight11:</div>
<blockquote>
<pre><code class="lang-auto">target_var = torch.Tensor(target_vec, requires_grad=False) 
</code></pre>
</blockquote>
</aside>
<p><a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/" rel="noopener nofollow ugc">We don’t use Tensor constructors anymore  but only torch.tensor factory methods</a>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: codeknight11
cooked: <aside class="quote no-group">
<blockquote>
<p><a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/" rel="noopener nofollow ugc">We don’t use Tensor constructors anymore  but only torch.tensor factory methods</a>.</p>
</blockquote>
</aside>
<p>Sorry if I sound like a noob but I don’t understand what that implies. Do you mean I cannot use torch.Tensor() to create a new tensor from another one?</p>
How to get gradient of loss with respect to predicted output

username: 0xfede
cooked: <p>I’m trying to get the gradient of the final output of a nn with respect to the loss function like so:</p>
<pre><code class="lang-auto">x = torch.FloatTensor(...)
x = Variable(x, requires_grad=True)
y = torch.FloatTensor(...)
y = Variable(y)

model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H, False),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out, False),
)

loss_fn = torch.nn.L1Loss(size_average=False)
y_pred = model(x)
loss = loss_fn(y_pred, y)
loss.backward()
print(y_pred.grad)

</code></pre>
<p>But the gradient comes back ‘None’. What am I doing wrong? For reference, I can get the linear layer’s weights just fine with a similar approach. Also, y_pred (created by model(x)) does have ‘requires_grad=True’</p>

username: richard
cooked: <p>Gradients don’t accumulate in non-leaf nodes. You should register a backward hook on the model: <a href="http://pytorch.org/docs/0.3.1/nn.html?highlight=backward%20hook#torch.nn.Module.register_backward_hook">http://pytorch.org/docs/0.3.1/nn.html?highlight=backward%20hook#torch.nn.Module.register_backward_hook</a> to view the gradient of the loss wrt y_pred</p>

username: NoobCoder
cooked: <p>Hey,</p>
<p>Did you figure out how to use it for finding the gradient respect to prediction?</p>
<p><a class="mention" href="/u/richard">@richard</a> Can you provide an example to how to do this?</p>
Dropout is not working in train mode on

username: orhunguley
cooked: <p>Hello,<br>
I am trying to do Monte Carlo sampling to estimate uncertainty. Even though I turn on the train mode  by model.train(), dropout is not working and all the samples are getting the same value. Can anybody help me to identify what am I doing wrong here? Thanks in advance!</p>
<p>Model looks like something like below:</p>
<pre><code class="lang-auto">LSTM_Base(
  (lstm): LSTM(167, 256, num_layers=3, batch_first=True, dropout=0.5)
  (fc): Linear(in_features=12800, out_features=256, bias=True)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc2): Linear(in_features=256, out_features=43, bias=True)
)
</code></pre>
<p>-My sampling function looks like something like this</p>
<pre><code class="lang-auto">def sampler(X, model, T=128):
    y_dist = []
    model.train()
    hidden_init = (torch.zeros(n_layers, 1, hidden_dim).cuda(), 
                       torch.zeros(n_layers, 1, hidden_dim).cuda())
    if model.training:
        for i in range(T):
            sample_out, _ = model.train().forward(X, hidden_init)
            y_dist.append(torch.clone(sample_out))
    
    return y_dist
</code></pre>
<ul>
<li>Note that model is already trained, I am trying to do sampling in train mode, after I trained the model.</li>
</ul>
<p><strong>EDIT</strong> : I just realized that there is no problem about the working of train mode. Because before training, the model is able to predict different outputs in train mode. But after training with more than 10 epochs, all outputs become the same. After realizing that, I also tried to train for just one epoch, and see that outputs are different. So it is clear that the problem is not about train mode. Can it be vanishing gradients? I am looking forward to read you suggestions!</p>
Train model on GPU and CPU

username: Kun_Han
cooked: <p>Hi, I have a model with two components, one on gpu and the other on cpu.<br>
Let’s say <code>Loss_1</code> is from gpu, <code>Loss_2</code> is from cpu. And I do <code>Loss = Loss_1 + Loss_2.to('cuda')</code>.  I’m wondering how the loss.backward() compute since there are two device?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The <code>.to()</code> operator is differentiable (just move values to the right device). And so the backward will work exactly the same as if  everything was on a single device. The backward ops will run on the same device as the forward ones.</p>
How to use GPU while training a model?

username: raoashish10
cooked: <p>I am running a code to train a resnet model on a kaggle notebook. I have chosen the accelerator as GPU so I haven’t made any mistakes there. I am training the model using the following code:</p>
<pre><code class="lang-auto">model.cuda()
for epoch in range(10):
  model.train(True)
  trainloss=0
  for x,y in trainloader:
    
    x,y=x.cuda(),y.cuda()

    yhat=model(x)
    optimizer.zero_grad()
    loss=criterion(yhat,y)
    loss.backward()
    optimizer.step()
    trainloss+=loss.item()
  
  print('Epoch {}  Loss: {}'.format(epoch,(trainloss/len(trainloader.dataset))))
  model.eval()
  testcorrect=0
  with torch.no_grad():
    for test_x,test_y in testloader:
      test_x,test_y=test_x.cuda(),test_y.cuda()
      yhat=model(test_x)
      _,z=yhat.max(1)
      testcorrect+=(test_y==z).sum().item()
      
print('Model Accuracy: ',(testcorrect/len(testloader.dataset)))
</code></pre>
<p>If you see I have used the .cuda() function on both my model as well as the tensors(inside the training part as well as validation part). However the GPU usage shown for the kaggle notebook is 0% while my CPU usage is up to 99%. Am I missing any code which is required to train the model using the GPU?</p>

username: Vatsal_Malaviya
cooked: <p>I use .to(device) methods and works perfectly fine for me,<br>
Also to be sure that your script has accessible cuda, you can check by the following code</p>
<pre><code class="lang-auto">device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
print(device)
</code></pre>
Gradient flow for auto-regressive RL model

username: Limmen1
cooked: <p>Hi, if I have a auto-regressive RL model where my network has two heads, where the first head predicts action a1 and the second head predicts action a2 conditioned on a1, where both a1 and a2 are sampled using PyTorch categorical distribution, will the gradients be able to backpropagate through the sample operation? See code snippet below.</p>
<pre><code class="lang-auto">shared_latent = self.test_shared_latent(obs.float())
values_latent = self.test_value_net(shared_latent)

a_1_logits = self.test_protocol_port_target_net(shared_latent)
a_1_dist = Categorical(logits=a_1_logits)
a_1 = a_1_dist.sample()                
a_2_logits = self.test_source_net(a_1.float())
a_2_dist = Categorical(logits=a_2_logits)
                    
a_2 = a_2_dist.sample()
a_1_log_prob = a_1_dist.log_prob(a_1.float())
a_2_log_prob = a_2_dist.log_prob(a_2)
</code></pre>
<p>The a_1_log_prob and a_2_log_prob are used later to create the loss as follows:</p>
<pre><code class="lang-auto">a_1_ratio = th.exp(a_1_log_prob - a_1_old_log_prob)
a_2_ratio = th.exp(a_2_log_prob - rollout_data.a_2_old_log_prob)

# clipped surrogate loss for target
a_1_loss_1 = advantages * a_1_ratio
a_1_loss_2 = advantages * th.clamp(a_1_ratio, 1 - clip_range, 1 + clip_range)
a_1_loss = -th.min(a_1_loss_1, a_2_loss_2).mean()

# clipped surrogate loss for source
a_2_loss_1 = advantages * a_2_ratio
a_2_loss_2 = advantages * th.clamp(a_2_ratio, 1 - clip_range, 1 + clip_range)
a_2_loss = -th.min(a_2_loss_1, a_2_loss_2).mean()

policy_loss = a_1_loss + a_2_loss
pg_losses.append(policy_loss.item())
</code></pre>
Problem with importing/installing PyTorch in PyCharm on Mac

username: chris_verdence
cooked: <p>I have successfully installed torch in my virtual environment in PyCharm (it shows up in Preferences -&gt; Project Interpreter). However, when I try to import torch by running: “import torch” I receive the following error:</p>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py”, line 21, in do_import<br>
module = self._system_import(name, *args, **kwargs)<br>
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/<strong>init</strong>.py”, line 136, in <br>
from torch._C import *<br>
File “/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py”, line 21, in do_import<br>
module = self._system_import(name, *args, **kwargs)<br>
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/_C.cpython-38-darwin.so, 2): Library not loaded: <span class="mention">@rpath</span>/libc++.1.dylib<br>
Referenced from: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/_C.cpython-38-darwin.so<br>
Reason: image not found</p>
<p>I have tried to copy the libc++.1.dylib file from ‎⁨Macintosh HD⁩ ▸ ⁨opt⁩ ▸ ⁨anaconda3⁩ ▸ ⁨pkgs⁩ ▸ ⁨libcxx-4.0.1-hcfea43d_1⁩ ▸ ⁨lib⁩ into the torch/lib folder, which removes that error, but gives me the following one:</p>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py”, line 21, in do_import<br>
module = self._system_import(name, *args, **kwargs)<br>
File “/Users/knaess/Documents/coding/mouth_detection/venv/lib/python3.8/site-packages/torch/<strong>init</strong>.py”, line 138, in <br>
<strong>all</strong> += [name for name in dir(_C)<br>
NameError: name ‘_C’ is not defined</p>
<p>How can i fix this?</p>
Does model.load_state_dict(strict=False) ignore new parameters, introduced in my model's constructor?

username: cbao001
cooked: <p>For <code>load_state_dict</code>, the documentation states:<br>
<code>Whether you are loading from a partial  *state_dict* , which is missing some keys, or loading a  *state_dict*  with more keys than the model that you are loading into, you can set the  </code>strict<code> argument to  **False**  in the </code>load_state_dict()<code>  function to ignore non-matching keys.</code> (from <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#id4" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html#id4</a>)</p>
<p>I have created a new model by augmenting the architecture slightly, doubling the number of trainable parameters (by introducing more modules), but I want to retain the parameters from a checkpoint (but keep them trainable for fine-tuning in my model) with a different dataset. I want to make sure that the newly introduced parameters will be trained, instead of ignored when I set <code>strict=False</code>, or if I should look into doing something else. Thanks!</p>

username: chetan_patil
cooked: <p>The <code>load_state_dict()</code> just loads the weights for you. Once you’ve loaded the weights using either of the arguments in <code>strict</code>, it’s done.</p>
<p>Suppose there is <code>modelA</code> which you trained previously and saved its weights as “modelA.pth”<br>
Now, you create an exact model called <code>modelB</code>, and you want to initialise its weights with <code>modelA</code>'s.<br>
In this case,<br>
<code>ModelB.load_state_dict(torch.load("ModelA.pth"))</code> would work.<br>
Note, that if you use <code>strict= False</code> or <code>strict=True</code> here, there won’t be any error thrown.<br>
The reason is, ModelA and and ModelB have the same kind of layers, there won’t be any problem in loading them.</p>
<p>But consider a scenario where your <code>ModelB</code> has some extra layers or is missing some layers.<br>
Now, if you use something like<br>
<code>ModelB.load_state_dict(torch.load("ModelA.pth"),strict=True)</code>, it will throw error.<br>
Keeping the parameter <code>strict=True</code> is like assuring PyTorch that both your models are identical.<br>
On the contrary, if you use <code>strict=False</code>, you inform PyTorch that <code>ModelB</code> and <code>ModelA</code> are not identical, so it just initialises the parameters of layers which are present in both and ignores the rest.</p>
<p>P.S: The parameter <code>strict</code> has nothing to do with the parameters being trained or not.</p>
GPU running out of memory at only 50% consumption

username: santient
cooked: <p>I’m getting the following error (during the backward pass):<br>
RuntimeError: CUDA out of memory. Tried to allocate 4.93 GiB (GPU 0; 23.65 GiB total capacity; 9.11 GiB already allocated; 10.44 GiB free; 12.34 GiB reserved in total by PyTorch) (malloc at /opt/conda/conda-bld/pytorch_1587428398394/work/c10/cuda/CUDACachingAllocator.cpp:289)</p>
<p>I have exclusive access to this GPU while the process is running. It seems like PyTorch isn’t reserving enough memory? Clearly there is enough free memory (10.44 GiB), unless memory fragmentation is so bad that I can’t use ~50% of the GPU memory. This always happens on the backward pass - I’ve even tried using torch.utils.checkpoint, but it didn’t make much of a difference since the forward pass does not take up nearly as much memory even with grad.<br>
My model uses Transformer layers as well as sparse-dense matrix multiplication, with variable-length sequences, so I do expect some fragmentation, but could it be causing it to this degree when I call loss.backward()? How could I fix this?<br>
Does backpropagation on sparse-dense matrix multiplication w.r.t. the sparse tensor return a sparse tensor? If it is returning a dense tensor, this could be the reason for high memory consumption.</p>
LSTM always predict same values for all inputs in the batch

username: Yilin_Liu
cooked: <p>Hi everyone, I want to apply LSTM for a regression problem, and for each pixel it needs to predict two values. somehow the LSTM model keeps output same values for all inputs in the batch.</p>
<p>('out: ', tensor([[0.2576, 0.0891],<br>
[0.2576, 0.0891],<br>
[0.2576, 0.0891],<br>
[0.2576, 0.0891],<br>
[0.2576, 0.0891],<br>
[0.2576, 0.0891],<br>
[0.2576, 0.0891],<br>
[0.2576, 0.0891],<br>
[0.2576, 0.0891],<br>
[0.2576, 0.0891]], device=‘cuda:0’, grad_fn=))<br>
('Tmaps: ', tensor([[0.2760, 0.2000],<br>
[0.5120, 0.2000],<br>
[0.8000, 1.0000],<br>
[0.0700, 0.7000],<br>
[0.2580, 0.0400],<br>
[0.0120, 0.0200],<br>
[0.0160, 0.1600],<br>
[0.0200, 0.2000],<br>
[1.0000, 1.0000],<br>
[0.1200, 0.2800]], device=‘cuda:0’))</p>
<p>This is how the model is implemented:</p>
<p>class MRF_RNN(nn.Module):<br>
def <strong>init</strong>(self, input_dim, hidden_dim, output_dim, n_layers, batch_size):<br>
super(MRF_RNN, self).<strong>init</strong>()</p>
<pre><code>    self.batch_size = batch_size
    self.n_layers = n_layers
    self.hidden_dim = hidden_dim
    
    self.rnn = nn.LSTM(input_dim, hidden_dim, n_layers, batch_first=True)
    self.fc = nn.Linear(hidden_dim, output_dim)

    for m in self.modules():
        if isinstance(m, nn.LSTM):
           for param in m.parameters():
               if len(param.shape) &gt;= 2:
                  nn.init.orthogonal_(param.data)
               else:
                  nn.init.normal_(param.data)
         
def forward(self, x):
    out, (hidden, cell) = self.rnn(x)
    out = self.fc(out[:, -1, :])
    return out
</code></pre>
<p>Any inputs would be greatly appreciated!!</p>

username: GuacaQuock
cooked: <p>Hello Yilin_Liu,</p>
<p>did you be any chance figure out what went wrong in your model?</p>
<p>i am facing the exact same problem with an identical model class and obtain the same results regardless of the input.</p>
<p>I would greatly appreciate your help!</p>

username: dwlmt
cooked: <p>Did either of you get to the bottom of this? I get the same issue. Whatever the input the LSTM it returns the same output for each row of the batch. The only difference is the model trains correctly. It is only when it is reloaded and used for predictions this is an issue. Running the LSTM using batch first.</p>

username: Michael-Equi
cooked: <p>Have any of you figured out this issue? I am having a similar issue and made a post describing my particular problem in more detail <a href="https://discuss.pytorch.org/t/outputs-of-lstm-model-all-very-similar-or-the-same/84370" class="inline-onebox">Outputs of LSTM Model all Very Similar or the Same</a></p>
Copyright holder for examples repository

username: grahamh
cooked: <p>The <a href="https://github.com/pytorch/examples/blob/master/LICENSE" rel="nofollow noopener">LICENSE in the PyTorch examples repository on GitHub</a> does not contain any names for the copyright holder (or contributors). Who is/are the copyright holder(s)? Is it the intention that copyright belongs to the individuals who have committed to the repository?</p>
What is the use of fused modules in quantization?

username: Midhilesh
cooked: <p>I am working on static quantization and have found that quantizing model with fuse module gives more accuracy than the models without applying fuse model.</p>
<p>What is the meaning of ConvRelu2d? whether the batchnormalization part is fused with the convolution weights or bachnormalization is removed?</p>
<p>Please do help me understand this concept.</p>

username: chetan_patil
cooked: <p>Hi <a class="mention" href="/u/midhilesh">@Midhilesh</a>,<br>
What I understand from <code>ConvReLU2d()</code> is that it just converts from<br>
<code>Conv2d()</code>-&gt; <code>ReLU()</code>  to something like <code>ReLU(Conv2d())</code> .<br>
However, there is no <code>BatchNorm2d</code> here, if you want to fuse even the BatchNorm layer, you could use this <code>ConvBnReLU2d()</code> .</p>
<p>You will find more info from the quantization doc <a href="https://pytorch.org/docs/stable/quantization.html#torch.nn.intrinsic.ConvReLU2d" rel="nofollow noopener">https://pytorch.org/docs/stable/quantization.html#torch.nn.intrinsic.ConvReLU2d</a></p>
torchvision.transforms.Resize not existing?

username: Behrooz_Sepehry
cooked: <p>There is clearly a torchvision.transforms.Resize documented here:</p>
<p><a href="http://pytorch.org/docs/master/torchvision/transforms.html#torchvision.transforms.Resize" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/docs/master/torchvision/transforms.html#torchvision.transforms.Resize</a></p>
<p>but when I use it</p>
<p>transforms.Compose([<br>
transforms.Resize(size=(224, 224)),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
])</p>
<p>I get the following error:</p>
<p>AttributeError: ‘module’ object has no attribute ‘Resize’</p>
<p>As soon as I change transforms.Resize(size=(224, 224)), to something else like transforms.CenterCrop(224), it starts working.</p>
<p>What is the reason for this problem and how can it be solved?<br>
Thanks.</p>

username: jpeg729
cooked: <p>Are you sure your version of torchvision is up to date?<br>
If you have an old version of torchvision transforms.Scale might work.</p>

username: Behrooz_Sepehry
cooked: <p>yes, the problem was resolved after upgrading torchvision.</p>

username: jpeg729
cooked: <p>The link to the docs that you provided is for the current version of the source code.<br>
If you replace “master” with “0.3.0” you get the docs for pytorch version 0.3.0 (which is currently the latest release) …</p>
<p><a href="http://pytorch.org/docs/0.3.0/torchvision/transforms.html#torchvision.transforms.Resize" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/docs/0.3.0/torchvision/transforms.html#torchvision.transforms.Resize</a></p>
<p>… which doesn’t make that much sense when you consider that the latest release of torchvision is version 0.2.0.</p>

username: Behrooz_Sepehry
cooked: <p>yes, I should mention that I actually upgraded both pytorch and torchvision to the latest version, and the problem was fixed.</p>

username: Ehsan_Moravveji
cooked: <p>I’m facing the same issue with PyTorch version 1.0.1 (the latest stable release). I have installed torchvision via pip.<br>
When I try to check the version of torchvision, I do not see the <strong>version</strong> class attribute. How do I make sure that mhy torchvision is up-to-date?</p>

username: shahadatku
cooked: <p>I am facing the same problem. How can I upgrade the torchvision?</p>
<p>Thanks in advance for your cooperation.</p>

username: chetan_patil
cooked: <p>You can use this command</p>
<pre><code class="lang-auto">pip install torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
Batched matrices from 1D tensors

username: Paran0
cooked: <p>Hi guys, Let’s say we want to generate Rotation Matrices from Batch of angles ( Bx1), we could use the <code>torch.cos()</code> and <code>torch.sin()</code> to convert  (Bx1) angles to (Bx1) tensors of cos and sin.</p>
<p>How I can stack the given angles to produce the batched version of <a href="https://en.wikipedia.org/wiki/Rotation_matrix" rel="nofollow noopener">Rotation Matrices</a>.<br>
Here is an example:</p>
<p><code>[ [  [cos(ang(0)), -sin(ang(0))],  [sin(ang(0)), cos(ang(0))] ], [  [cos(ang(1)), -sin(ang(1))], [sin(ang(1)), cos(ang(1))] ],  .... ] </code></p>

username: chetan_patil
cooked: <p>Hi <a class="mention" href="/u/paran0">@Paran0</a>, this code does the work.</p>
<pre><code class="lang-auto">import torch

n = 10 #from 0 upto 9 degrees
thetas = np.arange(n, dtype = np.float32)

cosine = torch.from_numpy(np.cos(np.deg2rad(thetas)))
sine = torch.from_numpy(np.sin(np.deg2rad(thetas)))

batch = torch.zeros((n,2,2), dtype = cosine.dtype)

batch[:,0,0] = cosine
batch[:,0,1] = -sine
batch[:,1,0] = sine
batch[:,1,1] = cosine
print(batch.shape)
</code></pre>

username: Paran0
cooked: <p>Yeah, that’s great, I tried this method too, any suggestion for performance improvements? for example, somehow we omit the pre-allocations …</p>

username: chetan_patil
cooked: <p>Yes, you can do that.!<br>
But your tensor shape will be (2,2, N).</p>
<pre><code class="lang-auto">batch = np.asarray([[cosine, -sine],
 					[sine, cosine]])
</code></pre>
Parsing meta.bin for Imagenet in torchvision

username: paganpasta
cooked: <p>The ImageNet <a href="https://pytorch.org/docs/stable/_modules/torchvision/datasets/imagenet.html#ImageNet" rel="nofollow noopener">class</a>  allows to load the data from the image directory. As we know, we need to download the data before hand as it is not longer supported.</p>
<p>My concern is regarding parsing the <code>meta.bin</code> file which is supposed to be in <code>devkit</code> folder. Let’s focus on classification task. Therefore, <code>devkit/data/ILVSRC2015_clsloc_validation_ground_truth.txt</code> and <code>devkit/data/meta_clsloc.mat</code> are firstly needed to be renamed in order to be read successfully.</p>
<p>The function <code>parse_meta</code> is unable to parse the data still. I found that the below modifications help.</p>
<pre><code class="lang-auto">
def parse_meta(devkit_root, path='data', filename='meta.mat'):
    import scipy.io as sio

    metafile = os.path.join(devkit_root, path, filename)
    meta = sio.loadmat(metafile, squeeze_me=True)['synsets']
    # nums_children = list(zip(*meta))[4]
    # meta = [meta[idx] for idx, num_children in enumerate(nums_children)
    #         if num_children == 0]
    idcs, wnids, classes = list(zip(*meta))[:3]
    classes = [tuple(clss.split(', ')) for clss in classes]
    idx_to_wnid = {idx: wnid for idx, wnid in zip(idcs, wnids)}
    wnid_to_classes = {wnid: clss for wniddef parse_meta(devkit_root, path='data', filename='meta.mat'):
</code></pre>
<p>For example, <code>meta</code> just after reading from filename appears like <code>[(1, n02119789, 'kit fox, vulpis macrotis', 'some description', 1300),(...),...]</code> . I am not sure what is the purpose of the lines I have commented. uncommented version results in <code>meta = [ ]</code>.<br>
Can someone confirm if this is the right approach? Or am I looking into wrong meta files. I downloaded the dataset from academic torrent. Pytorch version: <code>1.3.0</code><br>
Below is a sample snippet to replicate the problem.</p>
<pre><code class="lang-auto">from torchvision.datasets.imagenet import ImageNet, parse_devkit
class CustomImgNet(ImageNet):
    def __init__(self, root, **kwargs):
        self.root = root
        meta = parse_devkit(root+'/devkit')
        self._save_meta_file(*meta)
        super().__init__(root+'/data', **kwargs)

    def __getitem__(self, item):
        img, y = super().__getitem__(item)

        return img, y


if __name__ == '__main__':
    from torch.utils.data.dataloader import DataLoader
    from torchvision.transforms import transforms

    val_transform = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225])
    ])
    dataset = ImageNet16(root='/path/to/data', download=False, split='val', transform=val_transform)
</code></pre>
Preference of learning scheduler

username: Sravani_Lekkala
cooked: <p>when do we prefer which learning scheduler<br>
cosine annealing learning scheduler<br>
exponential learning scheduler<br>
is it trial and error only or can we prefer one over other based on situations</p>

username: tom
cooked: <aside class="quote no-group" data-username="Sravani_Lekkala" data-post="1" data-topic="84485">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sravani_lekkala/48/25065_2.png" class="avatar"> Sravani_Lekkala:</div>
<blockquote>
<p>is it trial and error only or can we prefer one over other based on situations</p>
</blockquote>
</aside>
<p>Probably trial and error + intuition. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Things I often see is fixed learning rate throughout or a learning rate that is decreased at 2-3 discrete points in time during learning. And then there is the 1-cycle policy advocated by by <a href="https://arxiv.org/abs/1803.09820" rel="noopener nofollow ugc">L. Smith in his papers</a> that has seens lots of success (see below).</p>
<p>Also, it depends a bit on the domain. Training purely convolutional networks in vision usually work differently than LSTMs in NLP.<br>
For the common ones, there are extensive hyperparameter studies and recipies. In no particular order</p>
<ul>
<li>David Page has a <a href="https://myrtle.ai/how-to-train-your-resnet/" rel="noopener nofollow ugc">series on training resnets</a> that provides lots of insights. There also are various “bags of tricks”-papers that collect principles.</li>
<li>For NLP, S. Merity et al’s paper on <a href="https://arxiv.org/abs/1708.02182" rel="noopener nofollow ugc">AWD-LSTM</a> would seem to be very influential in presenting a good training approach.</li>
<li><a class="mention" href="/u/jphoward">@jphoward</a> discusses L. Smith’s 1-cycle-policy on his <a href="https://fast.ai/" rel="noopener nofollow ugc">fast.ai courses</a> and lots of people following the courses wrote blog posts about their successes with it. (I think he also has stuff on AWD-LSTM for NLP, but I forget.)</li>
<li>For transformer language models (BERT/GPT and co), Huggingface has some info on training transformers in general in the documentation for their <a href="https://huggingface.co/transformers/" rel="noopener nofollow ugc">great library</a>. There are some paper emerging about efficient training, but I think the effort to bring down the total GPU ressources used is still underway.</li>
<li>In general, I’d look for “well done papers with good results and code” in your area of interest to see what they’re doing. Not all papers take great care in squeezing out the last bit of performance (because it’s not their focus), but usually they have experimented to find a method that generally works well.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>
Extracting non square/rectangular patch

username: vainaijr
cooked: <p>is there a way to extract non square/rectangular patch, like, extract a + or a circle, from an image, is there any research going on in this direction?<br>
what would the shape of such a tensor be?</p>

username: tom
cooked: <p>Typically, one uses masks. Either directly or to index with a boolean mask, which gives you an arbitray selection of pixels in a flattened tensor, e.g. for a circle, you can use</p>
<p><code>mask = (torch.linspace(-1, 1)[None]**2+torch.linspace(-1, 1)[:, None]**2)&lt;0.5**2</code></p>
<p>Best regards</p>
<p>Thomas</p>

username: vainaijr
cooked: <p>hello, thanks for your reply, the technique you describe do appear to work if my neural network already know the shape that it has to extract, but the problem be that it has no idea what shape it has to extract, it is only given an image, so information about what does a particular shape look like is not an input.</p>
<p>for example, if I see an image that I have never seen before, and I do not know what is that image about, still I am able to extract patches from it, of non square/rectangular shape.</p>
<p>currently the only way I know is to use unfold, that also, to extract square/rectangular patch, that also, I will have to manually input the shape of the patch that I want to extract.</p>

username: tom
cooked: <p>Yeah, well. You can combine the masking with unfold to get a series of patches.<br>
For “extracting the right thing if you don’t know what it is” (which is how the first paragraph sounds to me) you likely need someone more competent to answer.</p>
<p>Best regards</p>
<p>Thomas</p>
Dynamic quantization error: Mixed serialization of script and non-script modules is not supported

username: baheytharwat
cooked: <p>Hi, I have an issue in quantizing yolov3 model. I am working on this implementation: <a href="https://github.com/eriklindernoren/PyTorch-YOLOv3" rel="nofollow noopener">https://github.com/eriklindernoren/PyTorch-YOLOv3</a><br>
I followed the dynamic quantization tutorial and the quantization code as following: loading the model, loading pretrained weights from Darknet and using the dynamic quantization.</p>
<pre><code class="lang-auto">model = Darknet(opt.model_def, img_size=opt.img_size).to(device)

if opt.weights_path.endswith(".weights"):
    # Load darknet weights
    model.load_darknet_weights(opt.weights_path)
else:
    # Load checkpoint weights
    model.load_state_dict(torch.load(opt.weights_path))

qmodel = torch.quantization.quantize_dynamic(model, dtype=torch.qint8)
</code></pre>
<p>But I got this error message:</p>
<p>“For purely script modules use my_script_module.save() instead.”)<br>
_pickle.PickleError: ScriptModules cannot be deepcopied using copy.deepcopy or saved using torch.save. Mixed serialization of script and non-script modules is not supported. For purely script modules use my_script_module.save() instead.</p>
<p>Do you have any suggestions guys?</p>

username: huoge
cooked: <p>It looks like you are trying to quantize the scripted net.<br>
The correct order seems like first quantize your net then script it!</p>
L2 regularization

username: netaglazer
cooked: <p>hi, I use cross-entropy<br>
how can I know how much from the final loss is the part of the regularization ?<br>
thanks</p>

username: Alpha
cooked: <p>In fact, you can’t.</p>
<p>You can refer to this <a href="https://discuss.pytorch.org/t/how-to-inspect-the-regularization-loss-if-i-use-weight-decay-for-some-of-the-layers/28419/4">answer</a></p>
ValueError: optimizer got an empty parameter list New

username: 111336
cooked: <p>Hey there, I have built a neural Network to classify MNIST but there’s an error below is my code<br>
depth=[4,8]<br>
class ConvNet(nn.Module):<br>
def <em>init</em>(self):<br>
super(ConvNet,self).<strong>init()</strong> ?<br>
self.conv1=nn.Conv2d(1,4,5,padding=2)<br>
self.blocks=nn.ModuleList(block)<br>
self.pool=nn.MaxPool2d(2,2)<br>
<span class="hashtag">#No2</span>.Convolution<br>
self.conv2=nn.Conv2d(depth[0],depth[1],5,padding=2)<br>
self.fc1=nn.Linear(image_size//4<em>image_size//4</em>depth[1],512)<span class="hashtag">#int</span> division<br>
self.fc2=nn.Linear(512,num_classes)<br>
def forward(self,x):<br>
x=self.conv1(x)<br>
x=F.relu(x)<br>
x=self.pool(x)<br>
x=self.conv2(x)<br>
x=F.relu(x)<br>
x=self.pool(x)<br>
x=x.view(-1,image_size//4<em>image_size//4</em>depth[1])<br>
x=F.relu(self.fc1(x))<br>
x=F.dropout(x,training=self.training)<br>
x=self.fc2(x)<br>
x=F.log_softmax(x,dim=1)<br>
return x<br>
def retreive_features(self,x):<span class="hashtag">#get_features</span> maps<br>
features_map1=F.relu(self.conv1(x))<br>
x=self.pool(feature_map1)<br>
features_map2=F.relu(self.conv2(x))<br>
return (features_map1,features_map2)</p>
<p>net=ConvNet().cuda()<br>
optimizer=optim.SGD(params=net.parameters(),lr=0.001,momentum=0.9)</p>
<p>error raise:optimizer got an empty parameter list<br>
How can i do?</p>

username: chetan_patil
cooked: <p>Can you change it to from <code>init()</code> to <code>__init()__</code> ?</p>

username: 111336
cooked: <p>yes mine is <code>__init()__</code>  above i just forget to use underline , if you need i can give you whole python code</p>

username: 111336
cooked: <p>I solve it thank you like you say i for got to add  <strong>init()</strong>  double underline thx</p>