akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 1 7 7, but got 3-dimensional input of size [462, 2, 14] instead

Mohamed_Nabih: <p>class CNN(nn.Module):</p>
<pre><code>def __init__(self):
    super(CNN, self).__init__()

    self.cnn1 = nn.Conv2d(in_channels=1, out_channels=32, kernel_size=7, stride=4, padding=3)
    self.relu1 = nn.ReLU()Preformatted text
    self.maxpool1 = nn.MaxPool2d(kernel_size=2)
    self.cnn2 = nn.Conv2d(in_channels=16, out_channels=64, kernel_size=5, stride=1, padding=2)
    self.relu2 = nn.ReLU()
    self.maxpool2 = nn.MaxPool2d(kernel_size=2)
    self.fc1 = nn.Linear(3136, 10)
    self.fcrelu = nn.ReLU()

def forward(self, x):
    out = self.cnn1(x)
    out = self.relu1(out)
    out = self.maxpool1(out)

    out = self.cnn2(out)
    out = self.relu2(out)
    out = self.maxpool2(out)

    out = out.view(out.size(0), -1)

    out = self.fc1(out)
    out = self.fcrelu(out)

    return out</code></pre>

ptrblck: <p>A 2D layer, e.g. <code>nn.Conv2d</code>, expects the input to have the shape <code>[batch_size, channels, height, width]</code>.<br>
The <code>in_channels</code> of the first conv layer correspond to the <code>channels</code> in your input.</p>
<p>Based on the definition of <code>self.cnn1</code> it seems you want to pass an input with a single channel, so you might want to call <code>x = x.unsqueeze(1)</code> to create the channel dimension.<br>
This would only work, if your current input is defined as <code>[batch_size, height, width]</code>.</p>
AssertionError: torch.nn.quantized.ReLU does not support inplace

Ashish_Gupta1: <p>I tries to set relu in qconfig in prepare method of quantisation and came across this errors during .convert</p>
<pre><code class="lang-auto">args=args=[criterion,evaluation_loader, converter, opt, True]
            self.validation(fuse_m,args)
            
            # Convert to Quantize Model
            quantise_model=torch.quantization.convert(fuse_m, inplace=True)
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test_5step_quant.py", line 474, in &lt;module&gt;
    m.model("//media/ai/OCR_DATA/IIIT5k",15)
  File "test_5step_quant.py", line 448, in model
    self.test(opt)
  File "test_5step_quant.py", line 315, in test
    quantise_model=torch.quantization.convert(fuse_m, inplace=False)
  File "/home/remote/.local/lib/python3.6/site-packages/torch/quantization/quantize.py", line 293, in convert
    convert(mod, mapping, inplace=True)
  File "/home/remote/.local/lib/python3.6/site-packages/torch/quantization/quantize.py", line 293, in convert
    convert(mod, mapping, inplace=True)
  File "/home/remote/.local/lib/python3.6/site-packages/torch/quantization/quantize.py", line 294, in convert
    reassign[name] = swap_module(mod, mapping)
  File "/home/remote/.local/lib/python3.6/site-packages/torch/quantization/quantize.py", line 316, in swap_module
    new_mod = mapping[type(mod)].from_float(mod)
  File "/home/remote/.local/lib/python3.6/site-packages/torch/nn/quantized/modules/activation.py", line 44, in from_float
    return ReLU(mod.inplace)
  File "/home/remote/.local/lib/python3.6/site-packages/torch/nn/quantized/modules/activation.py", line 34, in __init__
    assert not inplace, 'torch.nn.quantized.ReLU does not support inplace'
AssertionError: torch.nn.quantized.ReLU does not support inplace

</code></pre>

jerryzh168: <p>should be fixed in <a href="https://github.com/pytorch/pytorch/pull/33105" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/33105</a>, cc <a class="mention" href="/u/raghuramank100">@raghuramank100</a></p>
Quantized convolution and NHWC layout

masahi: <p>Hi, I’m confused with the comment in FbgemmConv function below:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/ecbf6f99e6a4e373105133b31534c9fb50f2acca/aten/src/ATen/native/quantized/cpu/qconv.cpp#L268-L276" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/ecbf6f99e6a4e373105133b31534c9fb50f2acca/aten/src/ATen/native/quantized/cpu/qconv.cpp#L268-L276" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/ecbf6f99e6a4e373105133b31534c9fb50f2acca/aten/src/ATen/native/quantized/cpu/qconv.cpp#L268-L276</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="268" style="counter-reset: li-counter 267 ;">
<li>// Quantized kernels are all written with NHWC (channels last) layout in</li>
<li>// mind. Ideally, we'd be compatible with conv2d behavior and preserve the</li>
<li>// inputs layout as is (doing necessary upconversions).</li>
<li>//</li>
<li>// However, to be more robust, for now we just force output layout to always</li>
<li>// be NHWC (channels last), thus opportunistically improving perf.</li>
<li>//</li>
<li>// This might change when full memory format support lands</li>
<li>// See https://github.com/pytorch/pytorch/issues/23403</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>It says quantized convolution operates on NHWC layout and an output are kept in NHWC. But since the input is assumed to be in NCHW layout, there has to be a conversion step from NHWC back to NCHW. Where does this layout conversion happens?</p>
<p>I’m comparing outputs of quantized convolution in Pytorch and the same operations translated to and executed on TVM (via TVM’s WIP torch frontend). Even though the operations are just quantize, qconv, and dequantize, the two results doesn’t match. I’m trying to figure out where the difference comes from.</p>

hx89: <p>cc <a class="mention" href="/u/raghuramank100">@raghuramank100</a> .</p>

jerryzh168: <p>I think the logical layout is still NHWC, using memory_format we are re-arranging the physical memory format so that it will help kernels implementations.</p>
Return the tensor part from the generator class

Mohamed_Nabih: <p>Hi</p>
<p>I am using this command to read the ark.files<br>
train_feat = torchaudio.kaldi_io.read_mat_ark(’/home/mnabih/kaldi/egs/timit/s5/mfcc/cmvn_train.ark’)<br>
The output type is Generator [(str, torch.Tensor)]<br>
How can I extract the tensor part only to train my network.<br>
Also,<br>
d = {u: d for u, d in torchaudio.kaldi_io.read_mat_ark(’/home/mnabih/kaldi/egs/timit/s5/mfcc/cmvn_train.ark’)}</p>
<p>it gives me class  with key: string and value: torch.tensor<br>
I want to remove the key and using the tensor to train my network</p>

vincentqb: <p>Have you tried?</p>
<pre><code class="lang-auto">g = torchaudio.kaldi_io.read_mat_ark(’/home/mnabih/kaldi/egs/timit/s5/mfcc/cmvn_train.ark’)

d = {u: d for u, d in g}  # dictionary
t = [d for _, d in g]  # list of tensors
</code></pre>
When quantized::max_pool2d is used?

masahi: <p>In my IR that came from the quantized resnet model from torchvision, I have one max pool op which operates on quantized tensors:</p>
<pre><code class="lang-auto">  %input.3 : QUInt8(1, 64, 56, 56) = aten::max_pool2d(%input.2, %1279, %1282, %1285, %1288, %1289), scope: __module.maxpool # /home/masa/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:488:0
</code></pre>
<p>Does this dispatch into <code>quantized::max_pool2d</code> below?<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qpool.cpp#L414" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qpool.cpp#L414" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qpool.cpp#L414</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="404" style="counter-reset: li-counter 403 ;">
<li>    #ifdef USE_PYTORCH_QNNPACK</li>
<li>    if (at::globalContext().qEngine() == at::QEngine::QNNPACK &amp;&amp; qx.scalar_type() == kQUInt8) {</li>
<li>      return qnnpack_maxpool(qx, kernel_size, stride, padding, dilation, ceil_mode);</li>
<li>    }</li>
<li>    #endif</li>
<li>    return at::max_pool2d(qx, kernel_size, stride, padding, dilation, ceil_mode);</li>
<li>  }</li>
<li>};</li>
<li>
</li>
<li>static auto registry = torch::RegisterOperators().op(</li>
<li class="selected">    "quantized::max_pool2d(Tensor qx, "</li>
<li>    "int[] kernel_size, "</li>
<li>    "int[] stride, "</li>
<li>    "int[] padding, "</li>
<li>    "int[] dilation,"</li>
<li>    "bool ceil_mode) -&gt; Tensor",</li>
<li>    torch::RegisterOperators::options().kernel&lt;QMaxPool2D_arr_args&gt;(</li>
<li>        DispatchKey::QuantizedCPUTensorId));</li>
<li>
</li>
<li>} // namespace</li>
<li>} // namespace native</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Or more generally, I want to know about aten dispatch mechanism. Any pointer is appreciated.</p>
<p>cc <a class="mention" href="/u/raghuramank100">@raghuramank100</a> <a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>

jerryzh168: <p>Yes, that is correct. it is dispatch here: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Pooling.cpp#L128" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Pooling.cpp#L128</a><br>
We have multiple ways to do dispatch right now in PyTorch, one common place is in native_functions.yaml, you can take a look at: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/README.md" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/README.md</a></p>
How to escape from a saturated loss landscape while using Adam optimizer

vinaykumar2491: <p>Hi,<br>
I’m training an AutoEncoder network with Adam optimizer (<code>amsgrad=True</code>). My loss decreases very rapidly initially but slowly it starts saturating.</p>
<p>How can I accelerate the learning to decrease my loss? A standard strategy is to decay the learning rate; but when I do this to Adam optimizer my loss jumps abruply.</p>
<p>I’m using pytorch’s <code>scheduler.MultiStepLR()</code> to decay my learning rate.</p>
<p>Below is my loss plot.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/1/199a080a70d1f5d554fc8e55671304ab4edb3815.png" alt="loss_vocals_100epochs_multiStepLR" data-base62-sha1="3EtWKjO6fvJ0W7PcbozhQNir8SV" width="640" height="480"></p>

muammar: <p>Did you find a solution for this problem?</p>

vinaykumar2491: <p>Not even yet. I would be happy to know if anyone on the forum can experiments on this. I have also opened a thread on stack overflow.ai with more details.</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://ai.stackexchange.com/questions/8063/loss-jumps-abruptly-when-i-decay-the-learning-rate-with-adam-optimizer-in-pytorc" target="_blank" rel="nofollow noopener">ai.stackexchange.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://ai.stackexchange.com/users/8720/vinaykumar2491" target="_blank" rel="nofollow noopener">
    <img alt="vinaykumar2491" src="https://i.stack.imgur.com/mYycw.jpg?s=128&amp;g=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://ai.stackexchange.com/questions/8063/loss-jumps-abruptly-when-i-decay-the-learning-rate-with-adam-optimizer-in-pytorc" target="_blank" rel="nofollow noopener">Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch</a>
</h4>

<div class="tags">
  <strong>machine-learning, deep-learning, optimization, autoencoders</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://ai.stackexchange.com/users/8720/vinaykumar2491" target="_blank" rel="nofollow noopener">
    vinaykumar2491
  </a>
  on <a href="https://ai.stackexchange.com/questions/8063/loss-jumps-abruptly-when-i-decay-the-learning-rate-with-adam-optimizer-in-pytorc" target="_blank" rel="nofollow noopener">01:14PM - 20 Sep 18 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Implementing Backpropagation Through Time

FrankYang: <p>Hello,</p>
<p>I want to try an easier way to practice a recurrent network of fix length sequence input, and identical length sequence output. I make a gif animation to illustrate the forward and backward flow of backpropagation through time as below to ensure I know the concept clearly.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48d0a4e410cdc79d4dd03ad2032b7e8095956d90.gif" alt="BPTT" data-base62-sha1="ao9oxtzJU67BiAL9rjtTuor04fK" width="690" height="388" class="animated"><br>
My question is should I compute the gradients <code>∇_{U_i}L, ∇_{V_i}L, ∇_{W_i}L</code> of each time step simultaneously, sum them together and then do parameter updates using the summed gradient as shown in animation? Or should I do it sequentially? I mean compute gradient of time step 8 and then back propagate to time step 7, and so on till time step 1.<br>
Which one is correct?</p>
<p>This problem might not be directly related to pytorch, and since I am using pytorch to practice it, I hope to know how to obtain the gradients for each time step in pytorch.</p>
Why is the broadcasting semantics of PyTorch so weird? Also, why do we need broadcasting semantics?

Megh_Bhalerao: <p>Please let me know why is it that way</p>

albanD: <p>You can check the note about broadcasting <a href="https://pytorch.org/docs/stable/notes/broadcasting.html">here</a>. But the main point in there is the first sentence “Many PyTorch operations support <code>NumPy Broadcasting Semantics</code> .”: we follow numpy’s semantic because many users are used to them.</p>
Changing to cuda turns my model to None

mahsa: <p>I have a normal class (which is not an instance of anything like nn.Module or something), and inside that, I have some instances from two other classes which are inheritances of nn.Module.<br>
For using my model on GPU, I’ve defined a function which calls .cuda() for every nn.Module instances. But after running the code, maybe by calling this function, my main model turns to None. Actually, the structure of my models is something like as follows:</p>
<pre><code class="lang-auto">class main:
    def __init__(self):
        self.a = A()
        self.b = []
        for i in range(5):
            self.b.append(B())

    def cuda(self):
        self.a.cuda()
        for b in self.b:
            b.cuda()


class A(nn.Module):
    def __init__(self):
        ...


class B(nn.Module):
    def __init__(self):
        ...
</code></pre>

albanD: <p>What do you mean by turns to None?<br>
This looks like it should work properly.<br>
Could you give a small code sample that show the problem?</p>
How to use Transformer.DecoderLayer?

Rafael_R: <p>Hi,</p>
<p>The transformer decoder layer takes in input and the encoder embeddings. What is the shape of the output?</p>
<p>Unlike LSTM how do we do the beam search using the transformer decoder layer?  Since transformer decoder gives the entire prediction at once i.e. output shape is the same as the target shape.</p>
<p>Any pointers?</p>

shahensha: <p><a class="mention" href="/u/rafael_r">@Rafael_R</a> Were you able to figure out how to do it?</p>

zhangguanheng66: <p>The shape of transformer decoder is same as that of transformer encoder.</p>
Accuracy Metric for multi-class image segmentation

bing: <p>I am trying to model a multi-class image segmentation problem. On doing some literature survey,<br>
f-1 score and Jaccard index are coming to be a preferable way of measuring accuracy.<br>
My classes are highly unbalanced. Which metric will give the most accurate representation?<br>
Also, my model is predicting poorly on most of the classes, so most of the times, the outcome for a certain class is 0, so I can’t use Jaccard index in those cases as they’ll give undefined values according to this <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html" rel="nofollow noopener">https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html</a></p>
<p>What should I do?<br>
Also, any suggestions on how to improve the model performance, it is really bad and I don’t have any clue , how to improve it?</p>
Reshaping both the dimensions of images while loading dataset using dataloader

Suraj_Subramanian: <p>This is how I loaded my dataset. The shape of each batch while loading is (32, 3, 218, 178)</p>
<pre><code class="lang-auto">temp_dataset = ImageFolder('./rude_dir', transforms.Compose([ transforms.ToTensor() ]))
temp_loader = DataLoader(dataset=temp_dataset, batch_size=batch_size, shuffle=True, num_workers=8, drop_last=True)

temp_dataset = ImageFolder('./rude_dir', transforms.Compose([ transforms.ToTensor() ]))
temp_loader = DataLoader(dataset=temp_dataset, batch_size=batch_size, shuffle=True, num_workers=8, drop_last=True)

a[0].shape
torch.Size([32, 3, 218, 178])
</code></pre>
<p>How to reshape these images by just modifying the transforms function ?</p>
<pre><code class="lang-auto">transformations_exp = transforms.Compose([
    transforms.Resize(128),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
temp_dataset = ImageFolder('./rude_dir', transformations_exp)
temp_loader = DataLoader(dataset=temp_dataset, batch_size=batch_size, shuffle=True, num_workers=8, drop_last=True)
for a in temp_loader:
    break
a[0].shape
torch.Size([32, 3, 156, 128])
</code></pre>
<p>As you can see, when I specify transforms.Resize(128), only one dimension changes to 128 and another dimension changes to some other value.<br>
Thank you so much in advance !</p>

beaupreda: <p>Hello,</p>
<p>Looking at the <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Resize" rel="nofollow noopener">docs</a> for <code>transforms.Resize</code>, you need to specify a sequence <code>(h, w)</code> if you want to reshape the image in both dimensions. If you only specify one number (like you did), it will resize the smaller edge and keep the aspect ratio for the other one.</p>
<p>Good luck!</p>

Suraj_Subramanian: <p>Thanks, I tried it<br>
But got the following error</p>
<pre><code class="lang-auto">ValueError: Caught ValueError in DataLoader worker process 0.
 ValueError: unknown resampling filter
</code></pre>

beaupreda: <p>I tried you <code>transformations_exp</code> on an image and everything works on my end. Could you post your image folder code? Also, make sure to give a tuple to <code>transforms.Resize((128, 128))</code> or else the second value will be interpreted as the interpolation!</p>

Suraj_Subramanian: <p>It worked, thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I didn’t specify the input of transforms.Resize as a tuple !</p>
NaN in Mask R-CNN training on MS COCO

sigma_x: <p>It only happens to MS COCO, not Pascal VOC or ICDAR. I fixed the class ids:</p>
<pre><code class="lang-auto">coco_interface = COCO("instances_train2017.json")
all_cats = coco_interface.getCatIds()
all_cats.insert(0,0)
print(all_cats, len(all_cats))
</code></pre>
<p>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 67, 70, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 84, 85, 86, 87, 88, 89, 90]</p>
<pre><code class="lang-auto">selected_class_ids = coco_interface.getCatIds(catNms=['person'])
adjusted_class_ids = {}
for id, cl in enumerate(all_cats):
    adjusted_class_ids[cl] = id
print ("ADJUSTED CLASS IDS:")
print(adjusted_class_ids)
</code></pre>
<p>{0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 13: 12, 14: 13, 15: 14, 16: 15, 17: 16, 18: 17, 19: 18, 20: 19, 21: 20, 22: 21, 23: 22, 24: 23, 25: 24, 27:25, 28: 26, 31: 27, 32: 28, 33: 29, 34: 30, 35: 31, 36: 32, 37: 33, 38: 34, 39: 35, 40: 36, 41: 37, 42: 38, 43: 39, 44: 40, 46: 41, 47: 42, 48: 43, 49: 44, 50: 45, 51: 46, 52…88: 78, 89: 79, 90: 80}</p>
<p>The class label is converted so that it’s between 0 and 80. The total number of classes is 81. Learning rate is very low. What else can I do?</p>
Pytorch crashing while trying to iterate through a loaded training set

noobdog_170: <p>I am using the code:</p>
<pre><code>import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)

trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2);

dataiterator = iter(trainloader)
</code></pre>
<p>However once the program reaches the line:<br>
<code>dataiterator = iter(trainloader)</code><br>
It crashes and asks me twice whether I want to kill the python program(Like I pressed the close window button). The program is then unable to continue and hangs, this problem still persists after changing the:<br>
<code>iter()</code><br>
to a for loop:<br>
<code>for image, label in trainloader</code><br>
It always crashes the program. Additionally every time it asks to close it leaves a python3 program running in the background and they each use about 350MB. I am using linux mint 18.1 with a lenovo thinkpad x230. Has anybody come across this error before?</p>

albanD: <p>What do you mean by “the program crashes” exactly?</p>
<p>You may want to change <code>num_workers=1</code>, this may solve your issue.</p>

noobdog_170: <p>The error shown below keeps popping up whenever I run the program.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/c/cbc07a878ba83fe525358a9a91db03f2be8f6cfb.png" alt="image" data-base62-sha1="t4tjdTkwonYOKwhlnYJButa0xXJ" width="301" height="113"></p>
<p>The <code>num_workers=1</code> Didn’t work</p>

noobdog_170: <p>Tried it on another computer running ubuntu 16.04 and the same issue occurs. It crashes when reaching the trainloader.</p>

noobdog_170: <p>After some trial and error, removing the line <code>num_workers</code> completely fixes the issue. Does anybody know what the significance of removing this line is? Looking online it seems to be related to memory, I have 16GB of RAM and python doesn’t consume very much when running the program. It seems to be similar to this issue <a href="https://github.com/pytorch/pytorch/issues/1355" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/1355</a></p>

being_saurabh: <p>In my case,<br>
this problem occurred with pyTorch v1.0 as per the latest release, but degrading and installing the version to v0.4.0, solved for me.<br>
Thanks!</p>

PorkPy: <p>Thank you. This solved my similar issue where the execution would just hang when calling iter().<br>
Changing ‘num_workers’ to 1 had no affect.<br>
I’m using PyTorch 1.4.0 on Ubuntu 16.04.</p>
Where Dropout should be inserted?

oasjd7: <p>I want to use Dropout as a regularization.</p>
<p>I have ResNet Encoder and GAP Classifier.<br>
If I add one <code>Dropout()</code>, where should I insert this?</p>
<pre><code class="lang-auto">class Classifier(nn.Module):
    def __init__(self):
        super(Classifier, self).__init__()
        self.conv = nn.Conv2d(64, kernel_size=1)
        self.avgpool = nn.AvgPool2d(8)

    def forward(self, x):
        x = self.conv(x)
        x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        return x
</code></pre>
Variation of the same loss to optimize two different networks

manurare: <p>Hi,</p>
<p>Coming from the same criterion I would like to generate two different losses to optimize two networks. The 2nd networks receives images and transform them to meet the requirements from the resnet18. Subsequently, I want the resnet to minimize its loss but I only want the 2nd network to minimize the loss coming from a particular class.</p>
<pre><code class="lang-auto">resnet = resnet18()
network_2 = other_network()
criterion = nn.CrossEntropyLoss()
optimizer1 = optim.Adam(resnet.parameters())
optimizer2 = optim.Adam(network_2.parameters())

generated_samples = network_2(inputs)

classifier_outputs = resnet(generated_samples)
loss_1 = criterion(classifier_outputs, labels)
optimizer1.zero_grad()
loss_1.backward(retain_graph=True)
optimizer1.step()

indexes = torch.from_numpy(np.where(labels.cpu() == 1)[0]) #Take only the samples of the class == 1
new_outputs = torch.index_select(classifier_outputs, 0, indexes)
new_labels = torch.index_select(labels, 0, indexes)
loss_2 = criterion(new_outputs, new_labels)
optimizer2.zero_grad()
loss_2.backward()
optimizer2.step()
</code></pre>
<p>But this is not working as it should because I think there exists a conflict between gradients which I am unable to understand.</p>
LSTM variable perdiction performance

Roaldb86: <p>I’m running an LSTM model on a AWS t2.medium instance (2,4GHZ 2 cores).<br>
I use sequence length of 120 and have 16 features.</p>
<p>The time it takes to predict and cpu utilization seems very variable. prediction takes around 50 ms most of the time but all of a sudden it can take 5 seconds for one predictions while cpu usage goes above 100%<br>
See the model below, any ideas?</p>
<pre><code class="lang-auto">class LSTMmodel(nn.Module):

    def __init__(self, output_size, hidden_dim, n_layers, features, drop_prob=0.3):
        """
        Initialize the model by setting up the layers.
        """
        super().__init__()

        self.output_size = output_size
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim
        self.features = features

     
        self.lstm = nn.LSTM(self.features, hidden_dim, n_layers,
                            dropout=drop_prob)

        # dropout layer
        self.dropout = nn.Dropout(0.3)

        self.fc = nn.Linear(hidden_dim, output_size)

    def forward(self, x, hidden):

        batch_size = x.size(1)
        lstm_out, hidden = self.lstm(x, hidden)

        # stack up lstm outputs
        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)

        # dropout and fully-connected layer
        out = self.dropout(lstm_out)
        out = self.fc(out)

        # reshape to be batch_size first
        out = out.view(batch_size, -1, self.output_size)
        out = out[:, -1] # get last batch of labels

        return out, hidden



    def init_hidden(self, batch_size, train_on_gpu=False):
        ''' Initializes hidden state '''

        weight = next(self.parameters()).data

        if (train_on_gpu):
            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda(),
                  weight.new(self.n_layers, batch_size,  self.hidden_dim).zero_().cuda())
        else:
            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),
                      weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())

        return hidden


    def predict(self, inp):

        self.eval()
        h = self.init_hidden(1)
        inp = torch.FloatTensor(inp)
        with torch.no_grad():
                outputs, h = self(inp, h)
                _, prediction = torch.max(outputs, 1)
        return F.softmax(outputs, dim=1).numpy()[0], prediction.numpy()[0]


</code></pre>
Torch.save/load does not support partial read

Carl: <p>Greetings comrades,</p>
<p>I would expect the following to run smoothly, since it’s a valid pattern with raw <code>pickle</code>:</p>
<pre><code class="lang-auto">In[203]: with open('file.pt', 'wb') as f:
    ...:     torch.save(torch.rand(4, 4), f)
    ...:     torch.save(torch.rand(4, 4), f)
    ...:     
In[204]: with open('file.pt', 'rb') as f:
    ...:     print(torch.load(f))
    ...:     print(torch.load(f))
    ...:     
tensor([[ 0.0685,  0.9256,  0.7928,  0.5120],
        [ 0.8782,  0.6241,  0.7041,  0.0264],
        [ 0.3485,  0.4451,  0.7395,  0.4364],
        [ 0.6275,  0.4573,  0.2679,  0.7629]])
Traceback (most recent call last):
  File "/home/clemaire/miniconda-mio/envs/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2963, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "&lt;ipython-input-204-07261aa68cb4&gt;", line 3, in &lt;module&gt;
    print(torch.load(f))
  File "/home/clemaire/miniconda-mio/envs/venv/lib/python3.6/site-packages/torch/serialization.py", line 303, in load
    return _load(f, map_location, pickle_module)
  File "/home/clemaire/miniconda-mio/envs/venv/lib/python3.6/site-packages/torch/serialization.py", line 459, in _load
    magic_number = pickle_module.load(f)
_pickle.UnpicklingError: invalid load key, '\x10'.
</code></pre>
<p>My use case is being able to read only the metadata associated with a training checkpoint without having to load all the network weights in memory. I would save the file like this:</p>
<pre><code class="lang-auto">with open('checkpoint.pt', 'wb') as f:
    torch.save({'epoch': 123, ...}, f)
    torch.save(net.state_dict(), f)
</code></pre>
<p>Why doesn’t this work? Do you think it should be supported?</p>
<p>Thanks!</p>

kshitijpatil09: <p>My question is similar in some sense so posting it here. I want to save partial state dict and architecture and be able to load it back. Like saving the weights of only encoder and use that as a part of other model.</p>
<p>Thanks!</p>
Undocumented behavior with functional.normalize()

Ryan_Keathley: <p>I have a 4-d tensor  and I would like to  (in one go) L1 normalize each of the 3-d slices</p>
<pre><code class="lang-auto">tensor[0], tensor[1], etc
</code></pre>
<p>In pytorch I  was able to accomplish this with</p>
<pre><code class="lang-auto">tensor = torch.nn.functional.normalize(tensor, dim=[1,2,3], p=1)
</code></pre>
<p>Which is odd since the documentation  says that dim must be an int. Either way, I would like to replicate this in libtorch, but I have no idea how.</p>
Image segmentation 'cropping after reszing' vs 'resizing after cropping'

Sunshower076: <p>Hi, although it’s not a topic about pytorch, I wrote to get your help.<br>
I am studying image segmentation and I have to do cropping images and do resizing.</p>
<p>I have two datasets which have different image size. The reason I concern the order of these functions is because of ‘scale problem’</p>
<p>If cropping is done prior to resizing, I think the scale problem will be caused.</p>
<p>For example, I have two datasets that have the sizes ‘300x300’ and ‘600x600’ then, I crop these images with size of ‘256x256’.  In the former case, the ratio is 300/256 and latter case, the ratio is 600/256. I think that it will make degrading performance.</p>
<p>If you have some tips about this case. please give me some advice thanks!</p>

ptrblck: <p>I would also share your concern.<br>
However, given that <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.RandomResizedCrop"><code>torchvision.transforms.RandomResizedCrop</code></a> uses a scale of <code>[0.08, 1]</code> and seems to work fine, I would recommend to just try out both approaches.<br>
If I’m not mistaken, these settings were used in the original Inception model.</p>
How do we collect all the gradients from a model and store it?

pinocchio: <p>I did this:</p>
<pre><code class="lang-auto">grads = OrderedDict([ (name, None ) for name, w in mdl.named_parameters() ])
for name, w in mdl.named_parameters():
  grads[name] = w.grad
</code></pre>
<p>is this correct?</p>

ptrblck: <p>This seems to be a good way.<br>
However, note that you are storing references, so if your model updates the gradients or zeroes them out, your dict will reflect these changes.<br>
You could call <code>w.grad.clone()</code> to avoid this (also a check, if <code>grad</code> is not None should be added in this case).</p>
How to duplicate the R channel weights in DeepLapV3?

Mohammad_Elias_AlHus: <p>I’ve changed the number of input channels in DeepLabV3, because I have NRGB image (N=NIR=Near infrared), the next step is I have to duplicate the weights for the Red channel to make the model be fully pre-trained.</p>
<p>Any ideas ?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class MyDeepLab(nn.Module):

    def __init__(self, in_channels=1):
        super(MyDeepLab, self).__init__()
        self.model = torch.hub.load('pytorch/vision:v0.5.0', 'deeplabv3_resnet101', pretrained=True)
        self.model.backbone.conv1 = nn.Conv2d(4, 64, kernel_size=3, stride=2, padding=1, bias=False)

    def forward(self, x):
        return self.model(x)


my_deeplab = MyDeepLab()
print(my_deeplab.state_dict())
</code></pre>

ptrblck: <p>This code should work:</p>
<pre><code class="lang-python">
class MyDeepLab(nn.Module):

    def __init__(self, in_channels=1):
        super(MyDeepLab, self).__init__()
        self.model = torch.hub.load('pytorch/vision:v0.5.0', 'deeplabv3_resnet101', pretrained=True)
        with torch.no_grad():
            conv_weight = self.model.backbone.conv1.weight.clone()
            self.model.backbone.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3, bias=False)
            self.model.backbone.conv1.weight[:, 1:] = conv_weight
            self.model.backbone.conv1.weight[:, 0:1] = conv_weight[:, 0:1]
        

    def forward(self, x):
        return self.model(x)
</code></pre>
<p>It will repeat the first channel and reuse the others.<br>
Note that you have a different kernel shape and padding in your code than the original first conv layer.</p>
Building offline doc does not work

Hesiod_Theogonist: <pre><code class="lang-auto">git clone https://github.com/pytorch/pytorch.git -b v1.4.0-docs 
cd pytorch/docs
pip install -r requirements.txt
make dirhtml
</code></pre>
<p>result:</p>
<pre><code class="lang-auto">  File "source/scripts/build_activation_images.py", line 8, in &lt;module&gt;
    import torch.nn.modules.activation
ModuleNotFoundError: No module named 'torch'
Makefile:17: recipe for target 'figures' failed

</code></pre>
<p>What am I missing?</p>

ptrblck: <p>Based on the error it looks like PyTorch isn’t installed in your environment.<br>
I think the <code>requirements.txt</code> just mentions additional requirements besides PyTorch?</p>
How to use nn.LSTMCell()?

chunchun: <p>coule please tell me how to use nn.LSTMCell,including the arguments it needs and how to achieve LSTM with it?<br>
thanks a lot for your attention.</p>

G.M: <p>According to <a href="https://pytorch.org/docs/1.0.1/nn.html#recurrent-layers" rel="nofollow noopener">here</a>, u just have to manually feed each time step to the cell to achieve LSTM with the LSTMCell.</p>
How to make a model differentiate between two visually similar classes?

alx: <p>I have two classes that are extremely similar, in fact, my model predicts 99% of class 1 as class 0.</p>
<p>Someone recently recommended to enhance the differences between the two classes by applying some transformations to one of the class. I’m not too familiar with the process so I was wondering if it would work and if so, what should I change, Brightness? Exposure?</p>

ziqipang: <p>This is actually a topic in computer vision research called “fine-grained image classification.” You may take a look at the literature and pick a most relevant one. I personally recall some methods like “tripplet loss.” You may want to take a look at it.</p>
Input and output dimensions in PyTorch

Lightning_Blade: <p>I have a table with 100 rows and 11 columns, first 10 columns represent the explanatory variables (X1 to X10) and the last column represents the response variable (Y), should my input data be 100 by 10 matrix and output data be 100 by 1 column vector? does the feedforward operation in PyTorch go by xW+b rather than Wx+b?</p>
<p>Example:</p>
<pre><code>X = torch.Tensor(matrix(100, 10))
Y = torch.Tensor(vector(100, 1))

class NN(nn.Module):
    def __init__(self, input_dim = 10, output_dim=1):
        super(NN, self).__init__()
        self.lin1 = nn.Linear(input_dim, 5)
        self.lin2 = nn.Linear(5, output_dim)

    def forward(self, x):
        x = self.lin1(x)
        x = F.sigmoid(x)
        x = self.lin2(x)
        return x
</code></pre>
<p>Sorry for the primitive question.</p>

ziqipang: <p>According to my understanding, you do not have to care about the questions like Wx + b or xW + b in Pytorch.</p>
<p>When you have 100 * 10 input, it means you have the input data with batch size 100 and dimmension number 10. After the processing, the output of the neural network will be batch_size * output_dim, which is 100 * 1. So your current way of writing this is right.</p>

Lightning_Blade: <p>Thank you very much for the answer.</p>
Multiple sub-task loss backward for specific part of model

JWarlock: <p>My model has 3 sub-task lossess.  I want to calculate grad through each loss.backward() in 1 forward, and then use optimizer.step() for different parts of the model. The model is defined as below:</p>
<pre><code class="lang-auto">Class SomeNet(nn.Module):
    def __init__(self, num_classes):
        self.encoder = resnet50(pretrained=True)
        self.cls1 = nn.Linear(2048, num_classes)
        self.cls2 = nn.Linear(2048, num_classes)

    def forward(x):
        feature = self.encoder(x)
        output1 = self.cls1(feature)
        output2 = self.cls2(feature)
        return output1, output2
</code></pre>
<p>Currently I am using optimizer.zero_grad() before calling each loss.backward() to ensure only a specific part is updated. Concretly, I update encoder and cls1 with loss1, cls2 with loss2 and encoder with loss3. See demo code below:</p>
<pre><code class="lang-auto">
model = SomeNet(num_classes)
optimizer_fea = torch.optim.SGD({'params': model.encoder.parameters()})
optimizer_cls1 = torch.optim.SGD({'params': model.cls1.parameters()})
optimizer_cls2= torch.optim.SGD({'params': model.cls2.parameters()})

for data, label1, label2, label3 in data_loader:
    output1, output2 = model(data)
    loss1 = criterion1(output1, label1)
    loss2 = criterion2(output2, label2)
    loss3 = criterion2(output2, label3)

    loss1.backward(retain_graph=True)
    optimizer_fea.step()
    optimizer_cls1.step()
    optimizer_fea.zero_grad()
    optimizer_cls1.zero_grad()

    loss2.backward(retain_graph=True)
    optimizer_cls2.step()
    optimizer_fea.zero_grad()
    optimizer_cls2.zero_grad()

    loss3.backward()
    optimizer_fea.step()
    optimizer_fea.zero_grad()
    optimizer_cls2.zero_grad()
    
</code></pre>
<p>I have 2 questions for above code:</p>
<ol>
<li>
<p>If I use loss3.backward without retain_graph=True, then most of computational graph is freed. However, loss3 does not involve the node of cls1 in computational graph, so I guess there is some graph is not freed. Is there any convenient way to free all computational graph after I’ve called loss3.backward()?</p>
</li>
<li>
<p>The above code is tediously long. Is there any convenient way to only compute gradient for a specific layer of model?</p>
</li>
</ol>

JWarlock: <p>Seriously, does no one know  how to do this?</p>
<p>I want to use <code>loss1</code> to update <code>model.encoder</code> and <code>model.cls1</code>, which is simple. I can just use <code>loss1.backward()</code>.<br>
But for <code>loss2</code> and <code>loss3</code> the situation is different. <code>loss2</code> and <code>loss3</code> is computed on the exact same computational graph. Yet I want <code>loss2</code> only update <code>model.cls2</code> and <code>loss3</code> only update <code>model.encoder</code>.</p>
<p>Can I do that in an elegant way?</p>

JWarlock: <p>I think <code>torch.autograd.grad</code> may do the job. But I’m not sure how to implement it.</p>
Cnn convolution layer return nans

netaglazer: <p>hi,<br>
i have a cnn model,<br>
the first convolutional layer returns nan values:</p>
<pre><code class="lang-auto">x tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], device='cuda:0',
       grad_fn=&lt;SoftmaxBackward&gt;)
</code></pre>
<p>it is not happening immidiatly, only after some batches<br>
i have bo idea why</p>

JuanFMontesinos: <p>Check the input data contains no NaNs, NaNs in convolutions usually come from backprop of NaNs.<br>
Are you performing any risky operation?</p>

netaglazer: <p>i have a costume loss, mabey this is the reason?</p>

JuanFMontesinos: <p>Custom losses tend to be way less stable <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
But just check you are not passing negative values to a log, doing anything/0 these kind of things.<br>
Also have a look at<br>
<a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly</a></p>
<p>Still recommend you to check the input data if you apply any more suspicious transform. (Realize normalization of a signal whose values are close to 0 leads to a 0-division for example)</p>

netaglazer: <p>my inpet data is only ones and zeros<br>
and my forward is:</p>
<pre><code class="lang-auto"> def forward(self, x):
        x = self.dropout_input(x)
        x = x.transpose(1, 2)
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.conv3(x)
        x = self.conv4(x)
        x = self.conv5(x)
        # x = self.conv6(x)
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        x = self.softmax(x)
        return x
</code></pre>

netaglazer: <p>i only have a sequencial layer, and the data is ones and zeros</p>

JuanFMontesinos: <p>Soo have a look at your custom loss because there is no way you to find the issue with those layers.</p>
Need help with Go binding

Yitao_Sun: <p>Hey,</p>
<p>My colleague and I have been working on binding the most updated Torchlib to Go for our PyTorch production model. I attached the link below.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/nextcaller/go-torch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/5240414?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/nextcaller/go-torch" target="_blank" rel="nofollow noopener">nextcaller/go-torch</a></h3>

<p>LibTorch (PyTorch) bindings for Golang. Contribute to nextcaller/go-torch development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>We reached the bottleneck where we tried to convert the Compilation unit to Module, and the compilation unit is a new thing. I wonder if we can get any help from the official PyTorch developers to help explain some C++ functions and classes.</p>
<p>Thanks!</p>
Dice loss implementation issues (on smooth/eps factor)

ai_padawan: <p>I’m running into a wall, where I “can’t win”, in my current soft dice implementation.</p>
<p>To recap, soft dice is:<br>
<code>2TP / (2TP + FP + FN)</code></p>
<p>One way to smooth it is to at eps at say 1e-8:<br>
<code>2TP / (2TP + FP + FN + eps)</code><br>
When TP is exactly zero, it works, in the sense you don’t get NaNs. However, if TP is smaller than eps, you get a division over a big number which blows up to infinity. I’m getting a NaN/inf error when training with this loss function right now. I don’t think is NaN, because of eps, so I can’t be diving by zero. By elimination, the error has to be inf?</p>
<p>The other way to smooth is to add 1. This is what I did previously.<br>
<code>(2TP + 1) / (2TP + FP + FN + 1)</code><br>
This ONLY works if TP is large. If you have a situation where there are no positive pixels, and for simplicity no FP and no FN, you get 1/1=1, which is zero loss (since loss is 1 - dice), and I’m not a 100% sure I want that. That’s because I have half my dataset with no positive pixels, and so this earlier loss implementation is “incorrectly” giving it a good score, when all the model is learning to predict is say everything is background pixel.</p>
<p>The 3rd option, I’m thinking is eps both numerator and denominator.<br>
<code>(2TP + eps) / (2TP + FP + FN + eps)</code><br>
But the issue is see with this is that if TP, FP, FN = 0, then I will get eps/eps =1, which is also not want I want!</p>
<p>How do I fix this? It seems that in all scenarios, I get some kind of edge case that ruins it.</p>
Pretrained quantized models' export to ONNX fails

Zeenath_Shaikh: <p>I am trying to export the pretrained quantized models to ONNX, but it fails. I’ve tried for GoogleNet, ResNet18 and Mobilenet v2, but none of those exported.</p>
<p>I got the following error for GoogleNet and ResNet18</p>
<p>RuntimeError: self.qscheme() == at::kPerTensorAffine INTERNAL ASSERT FAILED at /pytorch/aten/src/ATen/native/quantized/QTensor.cpp:162, please report a bug to PyTorch. clone for quantized Tensor only works for PerTensorAffine scheme right now</p>

ggeor: <p>did you find a solution for this?</p>

stihl1210: <p>Hey, i’ve changed qconfig to torch.quantization.get_default_qconfig(‘qnnpack’) and error RuntimeError: self.qscheme() == at::kPerTensorAffine INTERNAL ASSERT FAILED is not an issue any more. This is just workaround.</p>
<p>Now i have an error KeyError: ‘conv2d_relu’ after small changed.</p>
<p>I fused layers for static quantization and it seems onnx doesn’t support fused layers in this format.  <a href="https://github.com/onnx/onnx/blob/master/docs/Operators.md#Conv" rel="nofollow noopener">https://github.com/onnx/onnx/blob/master/docs/Operators.md#Conv</a></p>
<p>This key doesn’t exist in symbolic_opset (any) so i don’t expect it would work.</p>

jerryzh168: <p>We probably need a patch to support clone for per channel quantized tensor: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/QTensor.cpp#L160" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/QTensor.cpp#L160</a></p>

jerryzh168: <p>created an issue: <a href="https://github.com/pytorch/pytorch/issues/33309" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/33309</a><br>
I’ll fix it when I have time, but feel free to submit a PR as well and I’m happy to review it.</p>