akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dimension error for a newbie

username: Subhankar_Ghosh
cooked: <pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
      super().__init__()
      self.fc1 = nn.Linear(M, N)
      self.conv1 = nn.Conv2d(1, 64, 2, 1) #1 input channel, 64 output channel, kernel size 2, stride length 1
      self.conv2 = nn.Conv2d(64, 32, 2, 1)
      self.conv3 = nn.Conv2d(32, 1, 2, 1)
      
    def forward(self, x):
      x = F.relu(self.fc1(x))
      x = x.view(int(math.sqrt(N)), -1)
      x = F.relu(self.conv1(x))
      x = F.relu(self.conv2(x))
      x = F.relu(self.conv3(x))
      return x

model = Net().to(device).double()
x_train = x_train.double() #torch.Size([1000, 25])
y_train = y_train.double() #torch.Size([1000, 5])
x_pred = model.forward(y_train[0])
</code></pre>
<p>I am getting this error:</p>
<blockquote>
<p>RuntimeError: Expected 4-dimensional input for 4-dimensional weight 64 1 2 2, but got 2-dimensional input of size [5, 5] instead</p>
</blockquote>
<p>in:</p>
<blockquote>
<p>—&gt; 12       x = F.relu(self.conv1(x))</p>
</blockquote>
<p>Can you all please suggest what I am doing wrong? Thanks!</p>

username: fanl
cooked: <p>Conv1 is expected to receive 4-D input, but x is viewed as 2-D.</p>

username: Subhankar_Ghosh
cooked: <p>Thanks!<br>
If I convert x to [1,1,5,5] does that agree with the architecture?</p>

username: fanl
cooked: <p>It seems okay.  You can try, it’s easy to debug.</p>
Why fnl_loss is not working

username: dato_nefaridze
cooked: <p>here i have a bug.if i replace  F.nll_loss function by nn.MSE class it works but why?fll_loss should have worked too.(for nll_loss i am modifying target values(you can see it in this example), but for MSE it is not required). please help me why F.nll_loss is not working, and help me to use it in this code(there i am training model cats vs dogs above i flattened image 50<em>50</em>3 to grayscale and used neural network (without convolutions), activation function=relu).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c9cc361e9944044dd2cc42042bde66429289b4ee.png" data-download-href="https://discuss.pytorch.org/uploads/default/c9cc361e9944044dd2cc42042bde66429289b4ee" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c9cc361e9944044dd2cc42042bde66429289b4ee_2_690x389.png" alt="image" data-base62-sha1="sNbux4Ua7epyXzH6TOExJf8FKcS" width="690" height="389" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c9cc361e9944044dd2cc42042bde66429289b4ee_2_690x389.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c9cc361e9944044dd2cc42042bde66429289b4ee_2_1035x583.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/9/c9cc361e9944044dd2cc42042bde66429289b4ee.png 2x" data-dominant-color="252626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1360×768 33.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>What exactly is not working using <code>F.nll_loss</code>? Is your model not learning or do you get some errors?<br>
Could you post the shapes of your model output and the target?</p>

username: dato_nefaridze
cooked: <p>whenever i use F.nll_loss, everything is fine it works very vell, but during test set it outputs the same class, it predicts 1 (dogs =1 cats=0) all time.at firs all images are 50x50x3 then iconvert them to 50x50x1, target values at firs are vectors [1,0] =dog [0,1]=cat. then i convert them to single number.also int network i am using softmax function</p>

username: ptrblck
cooked: <p><code>F.nll_loss</code> expects the model output to be log probabilities, so you should use <code>F.log_softmax</code> as your last activation.<br>
Also, the target tensor should contain the class index only (have a look at the docs for an example), while it seems you are passing one-hot encoded targets.<br>
Usually this should raise an Exception, so I’m not sure, why the code runs.</p>
<p>Could you post a minimal code snippet to see, how you are using this criterion and what model you are using?</p>

username: dato_nefaridze
cooked: <p>thanks it works with log_softmax, it was problem.it runs if i encode this.thaks a lot.</p>
Undefined symbol: _ZTIN5torch8autograd4NodeE

username: handsomeyang
cooked: <p>When ‘import torch’ in Python3, it shows the following problem:</p>
<pre><code class="lang-auto">Python 3.7.5rc1 (default, Oct  2 2019, 04:19:31) 
[GCC 9.2.1 20190909] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/opt/python3.7/lib/python3.7/site-packages/torch/__init__.py", line 81, in &lt;module&gt;
    from torch._C import *
ImportError: /opt/python3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so: undefined symbol: _ZTIN5torch8autograd4NodeE
&gt;&gt;&gt; 
</code></pre>
<p>Any idea what the problem would be? Thanks.</p>

username: ptrblck
cooked: <p>Did you install PyTorch from source or a binary?<br>
Was PyTorch working at some point in the past or is this your first installation?</p>
<p>Could you post which command you’ve used to install PyTorch?</p>

username: handsomeyang
cooked: <p>I used pip to install PyTorch as in</p>
<pre><code class="lang-auto">pip3 install torch torchvision
</code></pre>
<p>PyTorch used to work on my system (Debian 9) and it broke after I upgraded some system components (I don’t remember what exactly).</p>

username: ptrblck
cooked: <p>In that case I would recommend to reinstall the latest PyTorch version.<br>
Let us know, if you still have trouble.</p>

username: handsomeyang
cooked: <p>After updating Python3.7 and reinstalling PyTorch, the error persists:</p>
<pre><code class="lang-auto">Python 3.7.5 (default, Oct 27 2019, 15:43:29) 
[GCC 9.2.1 20191022] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/opt/python3.7/lib/python3.7/site-packages/torch/__init__.py", line 81, in &lt;module&gt;
    from torch._C import *
ImportError: /opt/python3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so: undefined symbol: _ZTIN5torch8autograd4NodeE
&gt;&gt;&gt; 
</code></pre>
Module 'torch.nn.init' has no attribute 'zeros_'

username: han.liu
cooked: <p>I’m using Pytorch 0.4.0, torchvision 0.2.1, Python 3.6.4, cuda9.2<br>
When running into <code>nn.init.zeros_(self.rnn.bias)</code><br>
I got this error:<br>
<code>AttributeError: module 'torch.nn.init' has no attribute 'zeros_'</code><br>
Why is this happend?<br>
Thank you!</p>

username: ptrblck
cooked: <p>This method was apparently introduced after the <code>0.4</code> release. You could call <code>self.rnn.bias.zeros_()</code> in this older release, but I would recommend to update to the latest stable release (<code>1.3.1</code>), since a lot of bugs were fixed and also a lot of improvements as well as features were added.</p>

username: han.liu
cooked: <p>Thank you for your reply! Of course I’d like to use the latest version. And I tried the latest version. I got different result from 0.4.1, and the result is much worse. And I don’t know where goes wrong. So I have to use the older version.</p>
<p>So Pytorch 0.4.0 is not <code>0.4</code> release?</p>

username: ptrblck
cooked: <p>Yes, <code>0.4.0</code> is the <code>0.4</code> release I’ve mentioned and the function is not available in this old release.<br>
Is my workaround working for you?<br>
Once this works, we should figure out, why the results are worse in the latest release, so please feel free to create a new topic with the issue here.</p>

username: han.liu
cooked: <p>I tried not using <code>nn.init.zeros_(self.rnn.bias)</code>. And I get another error. Pytorch 0.4.0 seems do not have <code>num_batches_tracked </code>. So I gave up on Pytorch 0.4.0.</p>
ATen/cuda/CUDAContext.h: No such file or directory

username: han.liu
cooked: <p>I’m trying to compile this project. (<a href="https://github.com/NVIDIA/flownet2-pytorch" rel="nofollow noopener">https://github.com/NVIDIA/flownet2-pytorch</a>). They used Pytorch 0.4.1. I’m also using Pytorch 0.4.1. And I get this error: <code>fatal error: ATen/cuda/CUDAContext.h: No such file or directory</code>. I tried with Pytorch 1.0.0, it can compile successfully. So it seems this file is introduced in newer Pytorch version. How can I have this file in Pytorch 0.4.1? Thank you!</p>
Gradient Clipping makes training slow

username: shuaitang
cooked: <p>Hi all,</p>
<p>I am new to pyTorch, but I have been using Torch for more than a year now.</p>
<p>Basically, I implemented an autoencoder model with gru as encoder and decoder, and it is a multi-gpu implementation. When I incorporate gradient clipping to training, the training gets slower, and it is probably 4 times slower than the training without gradient clipping.</p>
<p>Is it normal? Or is it an issue with my implementation?</p>
<p>Thank you so much for your time.</p>

username: chenyuntc
cooked: <p>I don’t think gradient clipping would be time-consuming. I guess it’s simply an operation on tensor which should be fast.</p>

username: shuaitang
cooked: <p>Yes, I agree.</p>
<p>I guess the issue was caused by calling the following line every iteration</p>
<p><code>torch.nn.utils.clip_grad_norm(model.parameters(), config.clip)</code></p>
<p>Instead, I tried with another way.</p>
<p>First, call</p>
<p><code>params = model.parameters()</code></p>
<p>before training, and then call</p>
<p><code>torch.nn.utils.clip_grad_norm(params, config.clip)</code></p>
<p>in every iteration. Right now, the training speed is reasonable now compared to my torch implementation.</p>

username: shuaitang
cooked: <p>The other reason that slows down training was that my lstm model is way too small, and multi-gpu setting didn’t help. Now I switched to single GPU mode, and it runs super fast.</p>

username: srama
cooked: <p>Why does this seem to cause such a drastic change in performance? Is it because the iterators are recreated or something?</p>

username: imy
cooked: <p>But program like that won’t take effect on the actual parameters.</p>

username: david-macleod
cooked: <p>Expanding on <a class="mention" href="/u/imy">@imy</a>’s reply…</p>
<p><code>model.parameters()</code>  returns a generator which can only be iterated over once before it is exhausted. It has to be re-instantiated every time you want to iterate over it. The proposed code is faster because it is only grad clipping your parameters once, the first time you call  <code>torch.nn.utils.clip_grad_norm</code> . Successive calls to  <code>torch.nn.utils.clip_grad_norm</code>  are operating on an exhausted iterator which effectively does nothing, and so is faster.</p>
Modifying batch size during training

username: TinfoilHat0
cooked: <p>Is it possible to decrease/increase the batch size during training loop assuming I use a DataLoader to fetch my batches?</p>
<p>For example, I can configure the learning rate during training via the scheduler of the optimizer. Is there an equivalent of this for the batch size?</p>

username: paganpasta
cooked: <p>Is this something you are looking for ?</p><aside class="quote" data-post="1" data-topic="13840">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hij/48/14113_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloader-for-variable-batch-size/13840">Dataloader for variable batch size</a> 
  </div>
  <blockquote>
    Hi I am new to this and for most application I have been using the dataloader in utils.data to load in batches of images. However I am now trying to load images in different batch size. For example my first iteration loads in batch of 10, second loads in batch of 20. 
Is there a way to do this easily? Thank you.
  </blockquote>
</aside>
Saving distributed models

username: iwkim
cooked: <p>Hi,</p>
<p>I’m trying to train multi-agent reinforcement learning. To do so, each agent has distributed (separate) network model. Therefore, the distributed model has a number of nn.Module for each separate model.</p>
<p>I want to save the entire networks’ parameters to evaluate the trained model. How can I save the entire parameters?</p>
<p>As I know, using</p>
<blockquote>
<p>d=model.state_dict()</p>
</blockquote>
<p>and</p>
<blockquote>
<p>torch.save(d, path)</p>
</blockquote>
<p>is appropriate. But the <em>model</em> used in the above command seems to be linked for only one network model (not entire separate model).</p>
<p>How can I save all separate model?</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>I’m not sure what “distributed” means in your use case.<br>
Are you working with different models? If so, you could just save the <code>state_dict</code> of each model using a separate file.<br>
Or are you working in a distributed setup, where the models are scattered and gathers using different nodes?<br>
In that case, you could most likely want to reduce the model to the main node and just store this <code>state_dict</code>.</p>

username: iwkim
cooked: <p>Thank you for your reply.<br>
Specifically, I have a number of agents and each agent has own policy network.<br>
So I said it as ‘distributed’ but it was quite vague… sorry for the inconvenience.</p>
<p>I think the first one you gave me is applicable, right?<br>
Because I generated a number of policy networks (models) and it is necessary to store separately.<br>
Am I right?</p>
<p>Thank you</p>
Using output of one network in another with only one loss function

username: Kuppharish
cooked: <p>I am using two neural networks say NN1 and NN2. Output of NN1 is used in forward function of NN2 and output of forward function of NN2 is the loss required. So in the training loop if I write</p>
<pre><code class="lang-auto">loss = NN2(data) 
loss.backward()
</code></pre>
<p>Will this update the weights of both NN1 and NN2?</p>

username: ptrblck
cooked: <p>If I’m understanding your use case correctly, you are sequentially calling NN1 and NN2.<br>
If that’s the case, your code should be fine and gradients should be calculated for both models.<br>
Basically you can treat models (like NN1 and NN2) as layers, since both are derived from <code>nn.Module</code>.<br>
There will be no difference between sequentially calling e.g. <code>nn.Linear</code> layer instances and your models.</p>
How to solve:--- RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

username: Praful
cooked: <p>Traceback (most recent call last):<br>
File “train_search.py”, line 265, in <br>
main()<br>
File “train_search.py”, line 179, in main<br>
train_acc, train_obj = train(train_queue, valid_queue, model, architect, criterion, optimizer, lr)<br>
File “train_search.py”, line 218, in train<br>
architect.step(input, target, input_search, target_search, lr, optimizer, unrolled=args.unrolled)<br>
File “/raid/shubhangi/Akshay/NAS/darts-depth/cnn/architect.py”, line 36, in step<br>
self._backward_step(input_valid, target_valid)<br>
File “/raid/shubhangi/Akshay/NAS/darts-depth/cnn/architect.py”, line 44, in _backward_step<br>
loss.backward()<br>
File “/home/cvprlab/.conda/envs/praful/lib/python3.5/site-packages/torch/tensor.py”, line 107, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/cvprlab/.conda/envs/praful/lib/python3.5/site-packages/torch/autograd/<strong>init</strong>.py”, line 93, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>

username: ptrblck
cooked: <p>I guess you are detaching the output or loss in <code>architect.step</code> from the computation graph.<br>
If you are only using PyTorch methods, Autograd will be able to record all operations and calculate the gradients in the backward pass using the created computation graph.<br>
However, if you are using other libraries, e.g. numpy, you are detaching the tensors from the computation graph, and Autograd won’t be able to automatically compute the backward pass.</p>
<p>Let us know, if this might be the case or post some code snippets so that we can have a look.</p>
<p>PS: you can add code snippets by wrapping them in three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
The size of mobilenetv2 become much bigger when convert to torchscript model

username: dalalaa
cooked: <p>When I convert mobilenet_v2 network to torchscript model, the size increased from 13.5MB to 43.1MB. It’s too big to deploy in Android.</p>
<p>How can I compress the torchscript model ?</p>
<p>My code:</p>
<pre><code class="lang-auto">from torchvision.models import mobilenet_v2
net = mobilenet_v2()
torch.save(net,"net.pth")
script_module = torch.jit.script(net)
torch.jit.save(script_module,"net.pt")
</code></pre>
<p>The size of models:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50baae5dd0d83acf4bb1d329cab187910f14a6db.png" alt="%5D%7D%60OQ1C_(8%40U_2%409%24JSCNWO" data-base62-sha1="bwa9VnE0WndtSFVvICHCibR7d1N" width="297" height="45"></p>

username: driazati
cooked: <p>Are you on the latest version of PyTorch (1.3)? We made some changes to significantly decrease the size of TorchScript binaries. For <code>mobilenet_v2</code> on the nightly version of PyTorch, I get very similar sizes between eager PyTorch and TorchScript:</p>
<pre><code class="lang-bash">$ ls -la --block-size=M
-rw-------.  1 user user 15M Nov 15 13:06 net.pt
-rw-------.  1 user user 14M Nov 15 13:06 net.pth
</code></pre>
<p>If you are on PyTorch 1.3 and still are getting this problem, could you try using the nightly PyTorch package and see if that fixes it?</p>

username: dalalaa
cooked: <p>Thank you, the problem solved when I upgraded torch from 1.2 to 1.3.</p>
Jit for tolist()

username: huoge
cooked: <p>Hello everyone. I attempt to use torch.jit.script on the something like hyp_ctc_state_prev.tolist(), but it doesn’t work。I build the pytorch from source and the torch version is 1.4.0a0+2e7dd54。<br>
I’d appreciate if anybody can help me! Or if there is a workable implementation, please let me know! Thanks in advance!<br>
here is the log:</p>
<details>
<summary>
logs</summary>
<p>RuntimeError:                                                                                                                   │                    # will be (2 x beam) hyps at most<br>
Tried to access nonexistent attribute or method ‘tolist’ of type ‘Tensor’.:<br>
hyps_score = [hyp_score]<br>
hyps_ctc_state_prev = [hyp_ctc_state_prev.tolist()]<br>
~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;— HERE<br>
hyps_ctc_score_prev = [hyp_ctc_score_prev]<br>
ended_hyps_score = []</p>
</details>

username: driazati
cooked: <p>This is a known bug <a href="https://github.com/pytorch/pytorch/issues/26752" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/26752</a></p>
<p>You can workaround by implementing it yourself, but it can be cumbersome due to the reasons explained in the bug report (plus the fact that TorchScript does not support generic functions or recursion):</p>
<pre><code class="lang-python">def my_1d_tolist(x):
    result: List[float] = []
    for i in x:
        result.append(i.item())
    return result

@torch.jit.script
def my_2d_tolist(x):
    result: List[List[float]] = []
    for i in x:
        result.append(my_1d_tolist(i))
    return result

x = torch.ones(2, 2)

# These calls should be the same
print(x.tolist())
print(my_2d_tolist(x))
</code></pre>

username: huoge
cooked: <p>Thanks a lot! I implemented it by myself yesterday and I find that use c++ to load torchscript model only support the function name ‘forward’. If I change the function name ‘forward’ to ‘infer’，and use the decorator <span class="mention">@torch.jit.export</span> , transfer the model to torchscript.however, when i use c++ to load the model like  torch::jit::script::Module module = torch::jit::load(‘model.pt’);  and then use module.infer(inputs), the cpp file won’t be built. Do you know why it happen?</p>

username: driazati
cooked: <p>For methods other than <code>forward</code> you have to explicitly get the method and run it. For this <code>Module</code></p>
<pre><code class="lang-python">class M(nn.Module):
    @torch.jit.export
    def infer(self, x):
        return x + 10

torch.jit.script(M()).save("m.pt")
</code></pre>
<p>You can run it in C++ with <code>script::Module::get_method</code></p>
<pre><code class="lang-cpp">int main() {
  auto module = torch::jit::load("m.pt");
  auto result = module.get_method("infer")({torch::ones({2, 2})});
  std::cout &lt;&lt; result &lt;&lt; "\n";
}
</code></pre>
<p>We have an open issue to improve our C++ documentation to make things like this more clear in the future.</p>

username: huoge
cooked: <p>Thanks a lot! it’s very helpful to me!</p>
How to create pytorch bindings for language X?

username: tastyminerals
cooked: <p>I would like to try and create <strong>pytorch</strong> bindings for X language (for learning purposes) with no prior experience. Where should I start and what should I read?</p>

username: tom
cooked: <p>So PyTorch’s own bindings are generated from <code>native_functions.yaml</code> by code in <code>tools/autograd</code> in addition with the data there.<br>
There also are a number of existing PyTorch bindings for other languages, but I’d probably try to generate the bindings by adapting the code in <code>tools/autograd</code> rather than parsing the <code>native_functions.yaml</code> myself.</p>
<p>If you’re not set on the task, an alternative  to creating bindings could be to do stuff within PyTorch. There are issues up for grabs in a number of areas and with varying degrees of difficulty (e.g. look at “good first issue” or “bootcamp” tags).</p>
<p>Best regards</p>
<p>Thomas</p>
Converting model for usage of batch input

username: matthias.l
cooked: <p>Hi! I am wondering if make sense to feed in the complete mini-batch during training at once to the model.<br>
This might increase the utilization of the GPU for RNNs?<br>
Or does pytorch this automatically if it detects a loop around you model?</p>
<p>If so, is there a possibility to convert a a model taking one sequence into a model taking a batch of sequences?</p>
<p>Do you have to split and concat your tensors-flow in your model?<br>
How to archieve that with <code>nn.Linear</code> and <code>nn.LSTM</code> for instance?</p>
<p>Thank you very much!</p>

username: alx
cooked: <p>Not sure to clearly understand, did you set a batch size in your dataloader?</p>

username: matthias.l
cooked: <p>Yes, that’s why I need my model eating batches.</p>

username: matthias.l
cooked: <p>Ok, I think I got it. Most of the builtin torch modules support both: Single samples and batches as input out of the box. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
My network is not learning at all

username: dato_nefaridze
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/3/632615116a1a0d19b352450e00f8e3a97cc5d2fb.png" data-download-href="https://discuss.pytorch.org/uploads/default/632615116a1a0d19b352450e00f8e3a97cc5d2fb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/632615116a1a0d19b352450e00f8e3a97cc5d2fb_2_690x387.png" alt="image" data-base62-sha1="e96RlPSddoFbdEVWXbx37d5Yz5F" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/632615116a1a0d19b352450e00f8e3a97cc5d2fb_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/632615116a1a0d19b352450e00f8e3a97cc5d2fb_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/3/632615116a1a0d19b352450e00f8e3a97cc5d2fb.png 2x" data-dominant-color="2B2B2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1366×768 47.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>  this is loss which is not improvin at all, i am trying to fix bug for 5 days but failed, please help me.</p>

username: ptrblck
cooked: <p>We would need some more information about the model and your use case in general.</p>
<p>PS: you can add code snippets by wrapping it in three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
This would make it easier to help in debugging without rewriting your complete code.</p>

username: dato_nefaridze
cooked: <p>thanks for reading my code, i am idiot, i will post required and easily readable code in future <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Visualizing loss surface of models

username: TinfoilHat0
cooked: <p>Is there a way to visualize (the approximation of) the loss surface of my model as well as how parameters are updated over it in PyTorch?</p>
<p>Something like they did over here:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://deeplearning.ai/ai-notes/assets/images/layout/favicon.png" class="site-icon" width="16" height="16">
      <a href="https://deeplearning.ai/ai-notes/optimization/?utm_campaign=BlogAINotesOptimizationNovember122019&amp;utm_content=105388620&amp;utm_medium=social&amp;utm_source=linkedin&amp;hss_channel=lcp-18246783" target="_blank" rel="nofollow noopener">deeplearning.ai</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:132/63;"><img src="https://www.deeplearning.ai/content/uploads/2019/04/ai-notes-image.png" class="thumbnail"></div>

<h3><a href="https://deeplearning.ai/ai-notes/optimization/?utm_campaign=BlogAINotesOptimizationNovember122019&amp;utm_content=105388620&amp;utm_medium=social&amp;utm_source=linkedin&amp;hss_channel=lcp-18246783" target="_blank" rel="nofollow noopener">AI Notes: Parameter optimization in neural networks - deeplearning.ai</a></h3>

<p>AI Notes: Parameter optimization in neural networks - deeplearning.ai</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Tensor operation bug?

username: Raul-Ronald_Galea
cooked: <p>Running the following code yields very strange results:</p>
<pre><code class="lang-auto">for i in range(100):
    a = torch.rand((189,4))
    b = torch.rand((189,4), dtype=torch.float64)
    print(torch.max(a[:,:2]+b[:,:2]))
</code></pre>
<p>Gives:</p>
<pre><code class="lang-auto">tensor(2.8119e+275, dtype=torch.float64)
tensor(nan, dtype=torch.float64)
tensor(nan, dtype=torch.float64)
tensor(2.8119e+275, dtype=torch.float64)
tensor(9.2142e+269, dtype=torch.float64)
tensor(9.2142e+269, dtype=torch.float64)
...
</code></pre>
<p>Note that for this to happen the tensors need to have different types and the operation has to involve a slice (simple a + b behaves normally)</p>
<p>Am I missing something? Thanks in advance.</p>
<p>Pytorch 1.3.0</p>

username: ptrblck
cooked: <p>Could you update to <code>1.3.1</code>, as some indexing/assignment operations with type promotion were broken in <code>1.3.0</code>?</p>
IOError: [Errno 121] Remote I/O error

username: KaiyangZhou
cooked: <p>Hi guys,</p>
<p>I have been receiving the following errors when loading images with PIL.Image and could not find ways to solve them. Could anyone help me?</p>
<pre><code class="lang-auto"> for i, inputs in enumerate(data_loader):
  File "/homes/.local/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 210, in __next__
    return self._process_next_batch(batch)
  File "/homes/.local/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 230, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
IOError: Traceback (most recent call last):
  File "/homes/.local/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 42, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/homes/my-project/utils/data/preprocessor.py", line 23, in __getitem__
    return self._get_single_item(indices)
  File "/homes/my-project/utils/data/preprocessor.py", line 30, in _get_single_item
    img = Image.open(fpath).convert('RGB')
  File "/homes/.local/lib/python2.7/site-packages/PIL/Image.py", line 2530, in open
    fp = builtins.open(filename, "rb")
IOError: [Errno 121] Remote I/O error: '/homes/my-project/images/571_02_0001.jpg'
</code></pre>
<p>The <code>_get_single_item</code> function in <code>preprocessor.py</code> looks like:</p>
<pre><code class="lang-auto">def _get_single_item(self, index):
        fpath, others = self.dataset[index]
        img = Image.open(fpath).convert('RGB')
        if self.transform is not None:
            img = self.transform(img)
        return img
</code></pre>
<p>I am sure there is no problem with those images that cause the errors, because I tried opening those images that caused the errors in an independent console and they can be successfully opened.</p>

username: smth
cooked: <p>it’s possible that the files are not accessible intermittently, or under heavy use of the I/O system.</p>
<p>In the terminal when you access them once, it’s possible that you dont see this error…</p>

username: Rajat-Mehta
cooked: <p>I am also getting the same error every time I train a pytorch model. Is there any solution for this problem?</p>
How to validate a neural network on another GPU?

username: tim4ous
cooked: <p>Hello!<br>
I have two video cards and I want to train a neural network on one video card, and after each epoch to test it on another video card. Please tell me how it can be implemented?</p>
<p>Thank you in advance!</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/tim">@tim</a>,</p>
<p>What’s the rationale behind this training architecture over the classical multi-GPU distributed training?</p>
<p>The framework and the underlying libraries are optimized to give you the best throughput in distributed training. You can either use <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="nofollow noopener">torch.distributed</a> or <a href="https://github.com/horovod/horovod" rel="nofollow noopener">Horovod</a> to achieve that, and you will certainly have a better GPU occupancy.</p>

username: tim4ous
cooked: <p>thank you for answer! I’m using nn.DataParallel, but I have problem: RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1</p>

username: spanev
cooked: <p>What’s your command line?</p>
<p><a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255" rel="nofollow noopener">Here</a> you can find a small tutorial explaining how to train on multiple devices and the best practices.</p>

username: tim4ous
cooked: <pre><code class="lang-auto">model = nn.DataParallel(Neural())
model = model.to(torch.device('cuda'))
images_first, images_second = (images_first.to(torch.device('cuda'), torch.float32), images_second.to(torch.device('cuda'), torch.float32))
        
output = model(images_first, images_second) &lt;--- here I have problem
</code></pre>
Compute squared gradient

username: Yonathan_Aflalo
cooked: <p>Hi. I have a network f that take a batch of input pair of image and label (x,y) and compute a cross entropy loss. I understand that if I use nn.CrossEntropyLoss(), the loss will be averaged over every input of the batch. If I access any weight (let say module.conv1.weight), I can obtain the gradient of the loss that is the average of the cross entropy over the batch with respect to the parameter  module.conv1.weight by accessing  module.conv1.weight.grad.</p>
<p>My question is the following. I would like to be able to compute the mean of the squared gradient of the loss with respect to the parameter over the batch instead of the mean of the gradient itself.</p>
<p>In other words, instead of having module.conv1.weight.grad that contains the mean of the gradient of the loss with respect to every element of the batch, I would like to extract a variable that would contain  the mean of the squared gradient of the loss, i.e. a vector g whose entries g_i are (module.conv1.weight.grad_i)^2.</p>
<p>Is it possible to do that?</p>
<p>Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>Pytorch computes the gradients wrt the loss you computed. So if your loss is the average, then the gradients will be the average as well.</p>
<p>If you don’t average the loss, then you will get as many losses as the batch size. And you will need to run as many backwards as the batch size to get all these gradients.</p>

username: Yaroslav_Bulatov
cooked: <p>There’s a trick from Ian Goodfellow to do this in a single pass: <a href="https://arxiv.org/abs/1510.01799" rel="nofollow noopener">https://arxiv.org/abs/1510.01799</a></p>
<p>This post gives PyTorch recipe – <a href="https://discuss.pytorch.org/t/efficient-per-example-gradient-computations/17204/3" class="inline-onebox">Efficient Per-Example Gradient Computations</a></p>

username: Yonathan_Aflalo
cooked: <p>Thanks a lot for this answer!</p>
How to put tensor on a custom Function to cuda device?

username: feifei
cooked: <p>Hi,<br>
I custom a Function, in forward method. I have a numpy array x and a cuda tensor input.How to copy value of x to input tensor ? I have try, in forward method, torch.to(device) don’t work.</p>
<p>Thanks</p>
<pre><code class="lang-auto">class MyFunction(torch.autograd.Function):

    @staticmethod
    def forward(ctx, input):
        x = np.zeros(shape)
        # do something for x
        # and how to copy data of x to gpu tensor input
        
        return ...

    @staticmethod
    def backward(ctx, grad_output):
          '''
</code></pre>

username: zetyquickly
cooked: <p>Here is an answer, that might work for you.<br>
You should create Tensor from NumPy array and then transfer it to device</p><aside class="quote" data-post="2" data-topic="19423">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/converting-numpy-array-to-tensor-on-gpu/19423/2">Converting numpy array to tensor on GPU</a> 
  </div>
  <blockquote>
    You should transform numpy arrays to PyTorch tensors with torch.from_numpy. 
Otherwise some weird issues might occur. 
img = torch.from_numpy(img).float().to(device)
  </blockquote>
</aside>


username: feifei
cooked: <aside class="quote no-group quote-modified" data-username="zetyquickly" data-post="2" data-topic="61171" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zetyquickly/48/22489_2.png" class="avatar"> zetyquickly:</div>
<blockquote>
<p>Here is an answer, that might work for you.<br>
You should create Tensor from NumPy array and then transfer it to device<br>
<a href="https://discuss.pytorch.org/t/converting-numpy-array-to-tensor-on-gpu/19423/2" class="inline-onebox">Converting numpy array to tensor on GPU - #2 by ptrblck</a></p>
</blockquote>
</aside>
<p>Hi, I create a tensor from numpy array. But the tensor to cuda device don’t work.</p>

username: zetyquickly
cooked: <p>At first you should check if CUDA devices are available.<br>
Then set the <code>device</code> variable with some value (e.g. <code>'cpu', 'cuda:0'</code>) and pass it to <code>your_tensor.to()</code> function.<br>
Note: set a constant string value for the <code>device</code> is not an only option (if you want use <code>tensor.to()</code> for transfering to device), you may pass there a <code>device</code> value of some other tensor</p>
Combining different types of features for classification

username: kevalmorabia97
cooked: <p>I am using first few layers of the pretrained alexnet to extract features from a given image and apply RoI pool layer to get a fixed dimensional feature vector corresponsing to one of the Bounding Boxes (BBox) in the image that I pass to a single layered classifier.<br>
My question is that, now I also want to use some extra features to improve the classification performance. I have about 3000 dimensional visual features and 4 other features (x, y, width, height) of the BBox. Should I just concatenate the 3000 dim visual features with the 4 extra features?</p>
<ol>
<li>Is it possible that the 3000 dim will overpower the classifier and not let the classifier learn something significant from the 4 extra features since 3000 is much greater than 4?</li>
<li>Should I do some kind of normalization before concatenating the 3000 dim features with the 4 dim features?</li>
<li>Is it a good idea to add a batch norm layer on the 3000 dim features and another batch norm on the 4 dim features and then concatenate these?</li>
<li>Should I project the 3000 dims to a smaller dimension or the 4 dims to a higher dimension and then concatenate them?</li>
<li>Is there a better way than concatenation to use the two types of features for classification?</li>
</ol>
<p>Thanks!</p>

username: Ashish_Gupta1
cooked: <p><a class="mention" href="/u/kevalmorabia97">@kevalmorabia97</a></p>
<ol>
<li>Yes its better to use batch norm for 3000 dim features. Your 4 dim are bounding box location? Try normalizing both. Otherwise that will leads to abrupt result.</li>
</ol>
<p>Try it(Not sure if It will work in your case)…<br>
1.Use Upsampling(deconv) layers to increase the dimension of 4 dim.<br>
2. Reduce the dimension of 3000 by using conv. I think if you try flattening kind then you will miss a lot of information.</p>
<p>Please explain a little more about the 3000 dim features.Because concatenation in your case sounds little different to me.</p>
How to make a heatmap of weights?

username: Rojin
cooked: <p>I would like to look at my decoder weight matrix ( as a heatmap). I am a bit confused about the shapes, shouls I user the 5th matrix? ( the size can be seen in the model.parameters().</p>
<pre><code class="lang-auto">Model(
  (encoder): Encoder(
    (model): ModuleList(
      (0): Linear(in_features=6000, out_features=100, bias=True)
      (1): BatchNorm1d(100, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
      (3): Dropout(p=0.2)
    )
  )
  (decoder): Decoder(
    (model): ModuleList(
      (0): Linear(in_features=100, out_features=6000, bias=True)
      (1): BatchNorm1d(6000, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
      (3): Dropout(p=0.2)
    )
  )
)
</code></pre>
<p>model parameters size:</p>
<pre><code class="lang-auto">
torch.Size([100, 6000])
torch.Size([100])
torch.Size([100])
torch.Size([100])
torch.Size([6000, 100])
torch.Size([6000])
torch.Size([6000])
torch.Size([6000])
</code></pre>

username: ptrblck
cooked: <p>You could get more information, if you print the name and parameter using <code>model.named_parameters()</code>.<br>
Most likely the first and 5th outputs are the weight matrices from the linear layers.<br>
The other (1dim parameters) would be the bias of the linear layers, <code>weight</code> and <code>bias</code> of the batch norm layers.</p>

username: Rojin
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="61152">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>model.named_parameters()</p>
</blockquote>
</aside>
<p>Thank you !<br>
When I did what you said, the namings are as follow:</p>
<pre><code class="lang-auto">encoder.model.0.weight
encoder.model.0.bias
encoder.model.1.weight
encoder.model.1.bias
decoder.model.0.weight
decoder.model.0.bias
decoder.model.1.weight
decoder.model.1.bias
</code></pre>
<p>Just to make sure that I got it right, the <code>encoder.model.1.weight encoder.model.1.bias</code> corresponds to  the Relu activation?</p>

username: ptrblck
cooked: <p>No, <code>ReLU</code> does not contain any parameters. They correspond to the <code>nn.Batchnorm</code> layer in the encoder.<br>
The index in <code>encoder.model.1</code> is corresponding to the <code>ModuleList</code> index when you print the model.</p>
PyTorch useful tools/packages

username: Er_Hall
cooked: <p>Is there a list of tools/packages for parameter tuning/visualization of loss function using PyTorch? or in general tools/libraries useful for PyTorch?</p>
<p>This will be useful for someone newbie in PyTorch, even 1-2 packages.</p>
<p>Please post if you know</p>
<p>Thanks</p>

username: Eta_C
cooked: <p>I think <a href="https://pytorch.org/ecosystem/" rel="nofollow noopener">PyTorch Ecosystem</a> may help you.</p>
Which is the right loss?

username: Jordan_Howell
cooked: <p>Should i use softmax on the last layer with Cross entropy loss for a binary classification?  Is there a cheat sheet out there on how to pair up the last layer and loss criteria?</p>

username: ptrblck
cooked: <p>The docs for the loss functions (e.g. <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss">nn.CrossEntropyLoss</a>) provide the necessary information on how to pass the inputs and targets.<br>
<code>nn.CrossEntropyLoss</code> expects logits, so you shouldn’t apply a softmax on your model outputs.</p>

username: Jordan_Howell
cooked: <p>So I don’t see a ‘logits’ in the docs.  I just let the linear layers run when constructing the model class and apply the loss to whatever the final layer puts out?</p>

username: ptrblck
cooked: <p>Yes, just pass the output of your last (linear) layer directly to <code>nn.CrossEntropyLoss</code>, as internally <code>F.log_softmax</code> and <code>nn.NLLLoss</code> will be applied.</p>
<p>In the docs “scores” is used, so you are right about the missing “logits”.</p>
<blockquote>
<p>The input is expected to contain raw, unnormalized scores for each class.</p>
</blockquote>

username: Jordan_Howell
cooked: <p>That’s awesome.  Thanks.</p>

username: Jordan_Howell
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>.  I expect when i take the exponent of the predicted values after sending the data through, they would equal 1 after being summed.  I don’t get that.  Am I missing something?</p>

username: ptrblck
cooked: <p>If you apply <code>F/log_softmax</code> manually, the exponent of this output should sum to one.<br>
Which output are you using at the moment? The model output or the (unreduced) loss function output?</p>

username: Jordan_Howell
cooked: <p>the out put is just the output layer by itself.  I will switch to log_softmax(x)?</p>

username: ptrblck
cooked: <p>In that case, you cannot expect the exponent of logits to sum to one.<br>
To get the probabilities, you could use softmax on the output of the last layer.<br>
Just make sure to <em>not</em> pass the softmax’ed output to <code>nn.CrossentropyLoss</code>.</p>

username: Jordan_Howell
cooked: <p>So i use <code>Log_softmax()</code> on output with <code>nn.CrossentropyLoss</code>?  Or <code>log_softmax()</code> with<code> NLLloss()</code></p>

username: ptrblck
cooked: <p>You can use:</p>
<ul>
<li>raw logits (no activation function at the end, just the raw output of the last layer) + <code>nn.CrossEntropyLoss</code>
</li>
<li>
<code>F_log_softmax</code> on the model output + <code>nn.NLLLoss</code>
</li>
</ul>
<p>If you need to see the probabilities for debug/printing purpose:</p>
<ul>
<li>use softmax and print the output</li>
<li>use <code>exp()</code> and print the output</li>
</ul>
<p>Make sure to <em>not</em> pass the probabilities to the loss functions.</p>
Custom implementation of attention

username: antgr
cooked: <p>Hi,<br>
I have implemented the following attention function, adapting it from an other implementation<br>
(<a href="https://github.com/littleflow3r/attention-bilstm-for-relation-classification/blob/master/model.py" rel="nofollow noopener">https://github.com/littleflow3r/attention-bilstm-for-relation-classification/blob/master/model.py</a>),<br>
and the final result is:</p>
<pre><code class="lang-auto">def attention(out, hidden):
    out = out.permute(1,0,2)
    hidden = hidden.squeeze(0)
    attn_weights = torch.einsum('pqr,pr-&gt;pq', [out, hidden])
    soft_attn_weights = F.softmax(attn_weights, 1)
    new_hid = torch.einsum('pqr,pq-&gt;pr', [out, soft_attn_weights])
    return new_hid
</code></pre>
<p>this is intended to be used after an lstm, and works in my code.</p>
<p>Calling it with</p>
<p>out: torch.Size([2, 2, 3])<br>
hidden: torch.Size([2, 3])</p>
<p>gives torch.Size([2, 3])</p>
<p>But if I use allennlp attention implementations, I have:</p>
<pre><code class="lang-auto">linear = CosineAttention(normalize=False)
output = linear(
            torch.FloatTensor([[0, 0, 0], [1, 1, 1]]),
            torch.FloatTensor([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]]),
        )
output.shape
torch.Size([2, 2])

[2, 3], [2, 2, 3] ---&gt; [2, 2]

linear = DotProductAttention(normalize=False)
output = linear(
  torch.FloatTensor([[0, 0, 0], [1, 1, 1]]),
  torch.FloatTensor([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]]),
)
output.shape
torch.Size([2, 2])

[2, 3], [2, 2, 3] ---&gt; [2, 2]
</code></pre>
<p>Can you provide some feedback? Is my implementation wrong?<br>
Is just an implementation for another use case? I need some theoretical feedback.</p>
Where do I check details of pytorch version update details

username: MinkyuChoi
cooked: <p>I was using pytorch 1.0 and after I update it to 1.3.1, my model’s performance became worse.<br>
I want to check if some functions have been updated drastically. I checked pytorch website but it is hard for me to find out the summary of updates. Is there a page shows things that updated for each version?<br>
I found migration guide but it only shows very big updates (0.4 --&gt; 1.0).</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can check all the details of the different releases on the github page: <a href="https://github.com/pytorch/pytorch/releases">https://github.com/pytorch/pytorch/releases</a></p>
How does `torchvision.datasets.ImageFolder()` make images coherent with pretrained CNNs?

username: B4D_0M3N
cooked: <p>Hi everybody,<br>
I retrained SqueezeNet with greyscale 128 by 128 images, only using the <code>torchvision.transforms.ToTensor()</code> transform, via <code>torchvision.datasets.ImageFolder()</code>. It yielded no error and training went fine, although the original network should only accept RGB 224 by 224 images.<br>
However, when I try to load a single image through <code>PIL</code> in order to classify it with the model I trained previously, I get a dimension mismatch error (<code>Invalid dimensions for image data</code>). How does exactly <code>torchvision.datasets.ImageFolder()</code> does in order to accommodate my images into the preexisting architecture? How can I reproduce the same transformations <code>torchvision.datasets.ImageFolder()</code> does to data, in order to have a successful isolated classification?</p>
<p><strong>Edit:</strong> I figured out, going through <code>torchvision.datasets.ImageFolder()</code> code, that by default, <code>PIL</code> is used to load the images and it is automatically converted to RGB, by using <code>img = Image.open(f)</code> and <code>img.convert('RGB')</code>. However, I still do not know how the difference between the spatial resolutions is handled.</p>