akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Saving FusionGroup as part of ScriptModule

Poorna: <p>Hello,</p>
<p>I am able to see the FusionGroup in the graph dump as part of optimization passes. But, I am not seeing that in the saved ScriptModule. How can I be able to save the last executed optimized graph (last_executed_optimized_graph, which includes FusionGroup) in the ScriptModule?</p>
<p>Thank you</p>

driazati: <p>The code for a <code>ScriptModule</code> is saved in its un-optimized form for a number of stability and performance related reasons. When it’s loaded, it gets re-compiled and re-optimized, so you don’t have to worry about the <code>FusionGroup</code> getting saved.</p>

Poorna: <p>Thank you for the response <a class="mention" href="/u/driazati">@driazati</a> . While I agree that re-compilation and re-optimization could be done during load time, not all optimizations are quick enough to do for every load. If there is an opportunity to save the optimized model (on the target hardware), we wont be incurring the optimization cost. Do you recommend or suggest ways in which we can achieve this?</p>

vdantu: <p>Bumping this question up. Is it possible to save an optimized graph along with its fusiongroups/subgraphs in a scriptmodule?</p>

driazati: <p>This is not possible today but may be enabled in the future as we build out support for different backends (e.g. <a href="https://github.com/pytorch/pytorch/pull/32178">this PR to enable lowered graphs for consumption by backend passes</a>)</p>

vdantu: <p><a class="mention" href="/u/driazati">@driazati</a>:  Awesome . Thanks a ton for this pointer. I will go through this PR. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Gradients and the way it is computed

learner47: <p>I have three neural networks, <code>model1</code>, <code>model2</code> and <code>model3</code>. I feed an input to <code>model1</code> and output of this network is fed to <code>model2</code> and further, the output of the <code>model2</code> is fed to <code>model3</code>. I have two loss functions. One for <code>model2</code> alone and one a global loss function which optimizes <code>model1</code> and <code>model3</code> weights only. <code>model2</code> is optimized only for a few epochs, after which it is run in forward prop only.</p>
<p>I have defined the optimizers like below:</p>
<pre><code class="lang-python">global_opt = optim.Adam(list(model1.params())+list(model3.params()))
model2_opt = optim.Adam(model2.params())
</code></pre>
<p>For the first few epochs I compute the following:</p>
<pre><code class="lang-auto">criterion = nn.MSE()
global_loss = criterion(obtained, expected)
global_loss.backward()
global_opt.step()

model2_crit = nn.BCE()
model2_loss = model2_crit(a, b)
model2_loss.backward()
model2_opt.step()
</code></pre>
<p>While executing these, all networks are set to train using <code>model1.train()</code>, <code>model2.train()</code> and <code>model3.train()</code>.</p>
<p>After the first few epochs, I want to stop training <code>model2</code>. I can’t set <code>model2.eval()</code> because my <code>model2</code> contains RNNs and the gradients have to flow through <code>model2</code> to <code>model1</code>.</p>
<p>So in the training loop I only have:</p>
<pre><code class="lang-python">criterion = nn.MSE()
global_loss = criterion(obtained, expected)
global_loss.backward()
global_opt.step()
</code></pre>
<p>My question is (it might be silly), does <code>model2</code> still update its weights after the training has subsided (when the last piece of code is the only running code in the training loop)? Thank you for reading this question.</p>

ptrblck: <p><code>model2</code>'s parameters won’t be updated unless you call <code>model2_opt.step()</code>.<br>
However, if your model contains batchnorm layers, the running estimates will be updated during the forward pass, if you keep these layers in <code>.train()</code> mode.<br>
Also, dropout will be applied, but of course there is nothing to update.</p>
<p>That being said, the <code>.grad</code> attributes of <code>model2</code> will accumulate the gradients, if you don’t set the <code>requires_grad</code> attribute of the parameters to <code>False</code>.</p>

learner47: <p>Thank you for your response. I am calling  <code>model2_opt.zero_grad()</code>  in each training loop. Won’t this suffice in order to prevent gradients from accumulating?</p>

ptrblck: <p>Yes, this should be sufficient. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Torch.cat throws error for tensor list when compiling with torchscript

ParticularlyPythonic: <p>Torch.cat throws error for tensor lists when used within torchscript.<br>
Kindly let me know of a fix/workaround.</p>
<p>Here is a minimal example to reproduce the bug.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

"""
Smallest working bug for torch.cat torchscript
"""


class Model(nn.Module):
    """dummy model for showing error"""

    def __init__(self):
        super(Model, self).__init__()
        pass

    def forward(self):
        a = torch.rand([6, 1, 12])
        b = torch.rand([6, 1, 12])
        out = torch.cat([a, b], axis=2)
        return out


if __name__ == '__main__':
    model = Model()
    print(model())  # works
    torch.jit.script(model)  # throws error

</code></pre>
<pre><code class="lang-auto">This code throws the following error:
File "/home/anil/.conda/envs/rnn/lib/python3.7/site-packages/torch/jit/__init__.py", line 1423, in _create_methods_from_stubs
    self._c._create_methods(self, defs, rcbs, defaults)
RuntimeError: 
Arguments for call are not valid.
The following operator variants are available:
  
  aten::cat(Tensor[] tensors, int dim=0) -&gt; (Tensor):
  Keyword argument axis unknown.
  
  aten::cat.out(Tensor[] tensors, int dim=0, *, Tensor(a!) out) -&gt; (Tensor(a!)):
  Argument out not provided.

The original call is:
at smallest_working_bug_torch_cat_torchscript.py:19:14
    def forward(self):
        a = torch.rand([6, 1, 12])
        b = torch.rand([6, 1, 12])
        out = torch.cat([a, b], axis=2)
              ~~~~~~~~~ &lt;--- HERE
        return out
</code></pre>
<p>Thank you for your consideration</p>

driazati: <p>PyTorch supports the <code>axis</code> keyword arg for numpy compatibility, but it looks like there is a bug where this isn’t translating into TorchScript. Most TorchScript ops use <code>dim</code> in place of <code>axis</code> (the meaning is the same), so if you change that in your code it should work, i.e. <code>torch.cat([a, b], axis=2)</code> becomes <code>torch.cat([a, b], dim=2)</code>.</p>
Batch size memory problem and normalization across batches

Emre_Bayram: <p>I am a SW eng trying to learn pytorch and deep learning.<br>
I am trying to train a resnext50_32x4d model.<br>
I have limited resources; so I can not increase the batch size to more than 16.<br>
But just to try I used a high capacity cloud machine for a single day and increased batch size to 128 and I got a lot better results.<br>
I think this is because of the normalization health.<br>
I am ok with the slow training because of the limited resources but I am not ok with the normalization defect.<br>
Is there a way to do the batch normalization across 8 batches or something like that?</p>
<p>Thanks in advance!</p>
<p>Edit: Not sure if I posted according to the forum rules; not sure which category to select. Please feel free to let me know if I did wrong.</p>

ptrblck: <p>You might try to change the <code>momentum</code> of your batch norm layers, so that the running estimates will be smoothed more.<br>
Besides that you could try to trade compute for memory using <a href="https://pytorch.org/docs/stable/checkpoint.html"><code>torch.utils.checkpoint</code></a>.</p>

Emre_Bayram: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> playing with optimizer parameters solved the problem.</p>
Gradient calculation with symmetric matrix

lmnop: <p>Hi,<br>
I am trying to compute the gradient of output from model with respect to the input, which is a symmetric matrix. I expect the output of gradient calculation to be symmetric. Can someone help explain why the gradient output is not symmetric? This is the code I have been trying.</p>
<p>SM = Variable(matrix,requires_grad=True)<br>
out = model(SM)<br>
output = F.softmax(out)<br>
model.zero_grad()<br>
out_class = output.max(1)[1].data[0]<br>
one_hot_output = torch.zeros(output.size())<br>
one_hot_output[0, output_class] = 1<br>
output.backward(gradient=one_hot_output)<br>
grad_out_in =  SM.grad.data[0]</p>
<p>The output of above code is for example looks like this:<br>
tensor([[[-0.0465, -0.0377, -0.0602,  …,  0.0117,  0.0323, -0.0426],<br>
[ 0.0018, -0.0711,  0.0594,  …,  0.0302, -0.1398,  0.0213],<br>
[-0.0123,  0.0490,  0.0875,  …, -0.0314,  0.0194, -0.0434],<br>
…,<br>
[ 0.0363,  0.0121,  0.0098,  …,  0.0108,  0.0057,  0.0425],<br>
[ 0.0596, -0.0743,  0.0541,  …,  0.0290, -0.1835,  0.1229],<br>
[-0.0395,  0.0283, -0.0428,  …, -0.0294,  0.1277, -0.0793]]]</p>
<p>Using Pytorch 1.4 with cuda 9.2</p>

albanD: <p>Hi,</p>
<p>Why do you expect the gradient to be symmetric?<br>
If I get a matrix, and pass it through a function that only takes the first row and sum it. The gradient will be 1s in the first row. Whether the input matrix is symmetric or not does not influence that.</p>
How to set requires_grad

zenghang: <p>a=torch.ones(3,3)<br>
print(a.requires_grad)  <span class="hashtag">#False</span></p>
<p>print(a[0][0].requires_grad)  <span class="hashtag">#False</span></p>
<p>a[0][0].requires_grad=True</p>
<p>print(a[0][0].requires_grad)  <span class="hashtag">#False</span></p>
<p>I want to know why?How to set the gradient of part parameters inside a nucleus?</p>

albanD: <p>Hi,</p>
<p><code>requires_grad</code> is a field on the whole Tensor, you cannot do it only on a subset of it.<br>
You will need to do <code>a.requires_grad=True</code> and then extract the part of the gradient of interest after computing all of it: <code>a.grad[0][0]</code>.</p>

zenghang: <p>Thanks,<br>
But for a convolution kernel, for example, I want to specify the parameters of the diagonal to make it fixed, and learn the parameters of other locations.So I want to set the diagonal parameter to requires_grad = False,but this is not possible，is there any other way to do this?</p>

albanD: <p>Hi,</p>
<p>I’m afraid it’s not possible.<br>
If you don’t use regularization, one way would be to set the gradients for the diagonal elements to <code>0</code>.<br>
But if you have regularization they will still change.</p>
<p>Otherwise, you can save your parameters diagonal manually before training and then re-modify the weights after your optimizer step to the original value.</p>

zenghang: <p>Thank you very much ,I will try</p>

Ajna_Ram: <p>How to you extract the gradient of interest exactly?<br>
Thank you.</p>

albanD: <p>Hi,</p>
<p>As mentionned in the post <code>a.grad[0][0]</code> will give you the right subset of the gradient.</p>
Building a chatbot with pytorch seems cool and easy but bulky, I could build a more easier means with research and all if i can get support from pytorch admins

Lord_Mona: <p>Compared with tensorflow which i don’t like, Pytorch means of creating a chatbot is easier but bulky, If it could be made easier like creating a rasa-core or rasa-nlu model, or supporting such initiatives, it would be cool… I don’t think pytorch give full attention on its subAI models like chatbot… Anyways if i can create something easier, would i be supported??</p>

albanD: <p>Hi,</p>
<p>By something easier you mean to revamp the tutorial?<br>
Or do a easy-to-use chatbot implementation?</p>

Lord_Mona: <p>to create a easy to use chatbot implementation…</p>
Per-batch loss for semi-supervised learning

Jeremy_Rutman: <p>I have a semi-supervised problem as follows:<br>
I only know ground-truth for batches of examples, e.g. for batch 1 with examples b1=(e1,e2,…) there should be at least one high value from the outputs o1=(o1,o2,…) while for batch 2 there shouldnt be any high outputs.  Is there a way to setup a per-batch loss such as</p>
<p><code>L=(max(o1,o2,...)-E(b))**2</code></p>
<p>or</p>
<p><code>L=E(b)*log(max(o1,o2,...)</code></p>
<p>or the like, where E is the known expectation for batch b?</p>
<p>At inference time I’d like to send instances and not batches, to get back  individual outputs- so I blv each example must generate a prediction.</p>

albanD: <p>Hi,</p>
<p>The two loss you’ve written above are differentiable. So you can use these as the loss for your network. Do you have any issue with doing that?</p>

Jeremy_Rutman: <p>The losses shown e.g. <a href="https://pytorch.org/docs/master/notes/extending.html" rel="nofollow noopener">here</a> don’t seem to have access to the entire batch but rather only single x,y values, if I understood correctly.<br>
Another  example <a href="https://discuss.pytorch.org/t/fcn-implementation-loss-function/8435">here</a> , and the forum won’t let me post a third link as I am too new.<br>
Anyway  while I am happy to try writing the loss (which iiuc would require implementing forward and backward methods as in the first link) I think it may not have access to the  full batch, which I need for this case. I’ll try implementing a simple example, perhaps the x,y are batches of input,output and not individuals.</p>
<p>Also now that I think of it, differentiating max(f(x1),f(x2),…) is not obvious for me.  I suppose I can just select the max of f and then evaluate the derivative for this point only.</p>
<p>At a math forum I did see an approach where max(f(x),g(x)) = ( f(x)+g(x) + abs(f(x)-g(x)) )/2 and then one need only consider when the abs() changes sign but I blv this is a different case, as my inputs x_i are different for each output.</p>

albanD: <p>Hi,</p>
<p>The first link your posted is about extending the autograd engine for things it does not support.<br>
Everything you need here is supported, so no need for you to implement any backward <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Usually, given a batch of samples, your model will give you one output score (?) for each sample (the oi).<br>
Then you can write a function that given these, will compute the scalar loss on that batch.<br>
For the first one, that would be:</p>
<pre><code class="lang-auto">def my_loss(outs):
  return (outs.max() - out.max()) ** 2
</code></pre>
<p>Then you can do as usual:</p>
<pre><code class="lang-auto">outs = model(inp)
loss = my_loss(outs)
opt.zero_grad()
loss.backward()
opt.step()
</code></pre>

Jeremy_Rutman: <p>Thanks, I will give a whirl.</p>
Intel openVINO cpu extension LogSoftmax

spiovesan: <p>I am trying to optimize a converted onnx pytorch model containing a LogSoftmax output layer with Intel OpenVINO.I get this error:<br>
<code>[ ERROR ]  There is no registered "infer" function for node "output1" with op = "LogSoftmax". Please implement this function in the extensions.</code><br>
Of course I should implement LogSoftmax cpu extension. Is there a LogSoftmax cpu extension already implemented by someone?</p>

addisonklinke: <p>If you are using <code>nn.LogSoftmax</code> + <code>nn.NLLLoss</code> for training, you can switch to <code>nn.CrossEntropy</code> which combines the two (during training) as discussed in this <a href="https://discuss.pytorch.org/t/is-log-softmax-nllloss-crossentropyloss/9352/7">post</a>. Then, add a plain <code>nn.functional.softmax</code> to your model output during evaluation mode (since this operation is supported by OpenVino)</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self, num_hidden, num_classes):
        super(Net, self).__init__()
        self.fc = nn.Linear(num_hidden, num_classes)

    def forward(self, input):
        output = self.fc(input)
        if not self.training:
            output = F.softmax(output, dim=1)
        return output
</code></pre>
Equivalent of Keras GlobalMaxPooling1D

lorenzo_fabbri: <p>I’m trying to translate my Keras code to PyTorch. In my Keras code I use GlobalMaxPooling1D after the last 1D convolutional layer:</p>
<pre><code class="lang-auto">result = GlobalMaxPooling1D()(previous_result).
</code></pre>
<p>In Pytorch I’m trying to use MaxPool1d. I guess the stride should be 0 but I’ve no idea about the value of kernel_size. Should it be the same size as the kernel in the last convolutional layer?</p>

Ryanzsun: <p>It should be equivalent to <a href="https://pytorch.org/docs/stable/nn.html?highlight=adaptive#torch.nn.AdaptiveMaxPool1d" rel="nofollow noopener">torch.nn.AdaptiveMaxPool1d</a></p>

lorenzo_fabbri: <p>Thanks for the answer. So, in this case the <code>output_size</code> should match the input number of features?</p>

Ryanzsun: <p>The “output_size” is just output size, not about the feature map, not batch_size. You can check the example in documentation. By printing the shape of input and output, you should get what you want.</p>

mbp28: <p>A bit late to the party, but I just created this:</p>
<blockquote>
<p>class GlobalMaxPooling1D(nn.Module):</p>
<pre><code>def __init__(self, data_format='channels_last'):
    super(GlobalMaxPooling1D, self).__init__()
    self.data_format = data_format
    self.step_axis = 1 if self.data_format == 'channels_last' else 2

def forward(self, input):
    return torch.max(input, axis=self.step_axis).values
</code></pre>
</blockquote>
<p>Based on<br>
<a href="https://github.com/keras-team/keras/blob/master/keras/layers/pooling.py#L557" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/keras-team/keras/blob/master/keras/layers/pooling.py#L557</a></p>
Creation of Model

XYZCODE123: <p>Hello, am trying to recreate the model from (<a href="https://arxiv.org/pdf/1207.0580.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1207.0580.pdf</a> appendix G - Models for Cifar10).  So I should construct a CNN with 3 convolutional layers each followed by a subsam-<br>
pling/pooling layer. The convolutional layers should consist of 64 5x5 filters and the subsampling layers implement the max-pooling function over a window of size 3x3 using a stride of 2. In the end, I must connect the 3rd pooling layer to a 10-way softmax output layer. But when am trying to create this model, am having technical difficulties.</p>
<h2>So output shape for each layer would look like this:</h2>
<pre><code>    Layer (type)               Output Shape         
</code></pre>
<p>================================================================<br>
Input                  [3,32,32]<br>
Conv2d-1          [64, 28, 28]<br>
MaxPool2d-2        [64, 13, 13]<br>
Conv2d-3           [64, 9, 9]<br>
MaxPool2d-4        [64, 4, 4]<br>
///////////////////////////////////////////////////////// I can’t add another Conv2vd with kernal_size = 5 x 5,<br>
///////////////////////////////////////////////////////// because it would lead to<br>
Conv2d-5             [64, 4 - 5 , 4 - 5 ]   - &gt; error.</p>
<p>================================================================</p>
<p>So this model can’t be created? Or am doing something wrong?</p>

simaiden: <p>You can’t add the last conv layer because the kernel size is larger than the input in spatial dim (4x4 input and 5x5 kernel). This happen because you are using the conv layer without padding thus the output is smaller than input.</p>
<p>Try adjusting padding that preserve the input spatial dim</p>
Iterate over each layer in a network for later saving?

Jesper.Lindberg: <p>Hello!</p>
<p>I want to export my model as a ostream. I can sucessfully save each layer by:</p>
<pre><code>std::ofstream apa ("new.txt", std::ofstream::binary);
torch::save(model_-&gt;first_hidden_layer, apa);
</code></pre>
<p>However, I would prefer to do this of the entire model, not per layer if possible?<br>
If not possible, I would like to create a function that iterates over each layer in the network, get the networks name and save a file with the same name and the ostream in that file. However, I have not found any clear way to iterate over each layer, get the name and then the weights related to that layer.</p>
<p>So basically, instead of:</p>
<pre><code>std::ofstream first_hidden_layer("first_hidden_layer.txt", std::ofstream::binary);
torch::save(model_-&gt;first_hidden_layer, first_hidden_layer);
std::ofstream input_layer("input_layer.txt", std::ofstream::binary);
torch::save(model_-&gt;input_layer, input_layer);
std::ofstream output_layer("output_layer.txt", std::ofstream::binary);
torch::save(model_-&gt;output_layer, output_layer)
</code></pre>
<p>I would like to have a for-loop so I don’t have to hard-code the names and the amout of layers.</p>
<p>I found the following python post:<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/54203451/how-to-iterate-over-layers-in-pytorch" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2580505/infintyyy" target="_blank" rel="nofollow noopener">
    <img alt="Infintyyy" src="https://www.gravatar.com/avatar/e32b8d6bc3fcc28e31e65484cbe1ab3a?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/54203451/how-to-iterate-over-layers-in-pytorch" target="_blank" rel="nofollow noopener">How to iterate over layers in Pytorch</a>
</h4>

<div class="tags">
  <strong>python, machine-learning, neural-network, deep-learning</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/2580505/infintyyy" target="_blank" rel="nofollow noopener">
    Infintyyy
  </a>
  on <a href="https://stackoverflow.com/questions/54203451/how-to-iterate-over-layers-in-pytorch" target="_blank" rel="nofollow noopener">04:54PM - 15 Jan 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>However, I did not manage to do the same in my c++ code…</p>
How to share weights between two nets?

Andybert: <p>In torch, we can share parameters between two nets like this,</p>
<pre><code class="lang-auto">local r1 = nn.Recurrent(
        opt.embeddingSize,
        cnn,
        h2h,
        nn.Identity(),
        opt.sampleSeqLength)
local r2 = nn.Recurrent(
        opt.embeddingSize,
        cnn:clone('weight','bias','gradWeight','gradBias'),
        h2h:clone('weight','bias','gradWeight','gradBias'),
        nn.Identity(),
        opt.sampleSeqLength)
</code></pre>
<p>so, how to share weights between two nets in pytorch?</p>

shufanwu: <p>I think you can refer to this topic:</p><aside class="quote quote-modified" data-post="2" data-topic="398">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/48/21_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-create-model-with-sharing-weight/398/2">How to create model with sharing weight?</a> 
  </div>
  <blockquote>
    EDIT: we do support sharing Parameters between modules, but it’s recommended to decompose your model into many pieces that don’t share parameters if possible. 
We don’t support using the same Parameters in many modules. Just reuse the base for two inputs: 
class MyModel(nn.Module):
    def __init__(self):
        self.base = ...
        self.head_A = ...
        self.head_B = ...

    def forward(self, input1, input2):
        return self.head_A(self.base(input1)), self.head_B(self.base(input2))…
  </blockquote>
</aside>


smth: <p>you can simply use the same nets on different outputs. In Torch, each net is tied to a particular output, in PyTorch it is not.</p>

Andybert: <p>Thanks a lot! I got it.</p>

Baichuan: <p>Hi, can you explain it ? Thanks</p>

Baichuan: <p>we can implement weight sharing among the innermost layers by simply reusing the same Module multiple times when defining the forward pass. please see <a href="https://pytorch.org/tutorials/beginner/examples_nn/dynamic_net.html#pytorch-control-flow-weight-sharing" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/examples_nn/dynamic_net.html#pytorch-control-flow-weight-sharing</a></p>

Andybert: <p>Yes, you’re right,this is how pytorch creates model with sharing weight.</p>

Baichuan: <p>Do you use torch instead of pytorch?</p>

Andybert: <p>No, I use pytorch instead of torch.</p>
Variable Length Data Audio

shakeel608: <p><span class="mention">@ptrblck_de</span>  I have a variable length data and I want to first sort the data then select the batch sizes from that. (Data Loader). Any Suggestions Please</p>

G.M: <p>U should write a custom <code>collate_fn</code> and sort the data inside it, below is an example:</p>
<pre><code class="lang-auto">def collate_fn(batch):
    inp, labels = zip(*batch)
    inp = SORTED_VERSION
    return pad_sequence(inp), tc.stack(labels)
</code></pre>
<p>(I assume each index from ur dataset returns a tuple with 2 tensors)</p>

zhangguanheng66: <p>Another example <a href="https://github.com/pytorch/text/issues/664" rel="nofollow noopener">here</a>. Search <code>pad_data</code> func</p>

shakeel608: <p>Thank you, but I think bucketing will work here otherwise I wont be able to shuffle the dataset. Do you have any link/suggestion  <a class="mention" href="/u/g.m">@G.M</a></p>

G.M: <p>Maybe this helps? <a href="https://pytorch.org/text/data.html#bucketiterator" rel="nofollow noopener">https://pytorch.org/text/data.html#bucketiterator</a></p>
Conv3D with RNN

hash-ir: <p>I want to do classification with a sequence of images. I essentially want the local features of each frame in a sequence obtained through convolution layers to go into an RNN layer. For this, I am using <code>nn.Conv3d()</code> with <code>nn.RNN()</code>. My input is of the shape <code>(samples, channels, timesteps, height, width)</code>. I am not getting the intuition behind 3d convolutions. Because I have to preserve the <code>timesteps</code> dimension, I am using <code>kernel_size=1</code> for the <code>timesteps</code> dimension. So far, I have the following setup:</p>
<pre><code class="lang-python"># input: (batchsize, channels, timesteps, height, width)
layer1 = nn.Sequential(
    nn.Conv3d(1, 8, (1, 5, 5),
    nn.ReLU(),
    nn.MaxPool3d((1, 2, 2), (1, 2, 2)),
)
layer2 = nn.Sequential(
    nn.Conv3d(8, 16, (1, 3, 3),
    nn.ReLU(),
    nn.MaxPool3d((1, 2, 2), (1, 2, 2)),
)
layer3 = ...
# output: (batchsize, channels', timesteps, height', width')
# features: channels' x height' x width'

# I am combining the channels', width' and height' dimension to represent input for the RNN
output = output.view(output.size(0), output.size(2), -1)
n_features = output.size(2)

# (batchsize, timesteps, features)
rnn = nn.RNN(
    input_size=n_features,
    hidden_size=100,
    num_layers=1,
    batch_first=True
)
# Taking the output from the last state of the rnn and concatening the features for a Linear layer
output = output[:, -1, :].view(output.size(0), -1)
n_features = output.size(-1)
fc = nn.Linear(n_features, n_classes)

# loss function
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.01, betas=(0.9, 0.999), eps=1e-8, weight_decay=1e-6)
</code></pre>
<p>As a sanity check, I tried to overfit the model for 200 epochs. The loss went down initially but then became constant. I want to know whether I am doing right with the 3d convolution layer and concatenating the dimensions as features for the RNN layer.</p>
RuntimeError: expected device cuda:0 but got device cpu

mia-minyoung: <p>Hi,</p>
<p>While developing my GNNs, I finally faced a problem that I can’t understand without help.<br>
The problem is</p>
<p>/opt/conda/envs/py36/lib/python3.6/site-packages/torch/nn/functional.py in l1_loss(input, target, size_average, reduce, reduction)<br>
2189     else:<br>
2190         expanded_input, expanded_target = torch.broadcast_tensors(input, target)<br>
-&gt; 2191         ret = torch._C._nn.l1_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))<br>
2192     return ret<br>
2193</p>
<p>RuntimeError: expected device cuda:1 but got device cpu.</p>
<p>I checked that cuda is available and there are three GTX Titan V on a remote server. ( I’m working on this server)<br>
I’m using pytorch 1.4.0, CUDA toolkit 10.1 cuDNN 7603.</p>
<p>There aren’t any errors before that line.<br>
Thank you for your help in advance!</p>

simaiden: <p>I guess you have to pass tensor to CUDA using <code>input = input.cuda()</code> and <code>target=target.cuda()</code> in your training script</p>

mia-minyoung: <p>You are right! Thank you so much!</p>
Derivative of selected hypothesis of neural network

Beyonder: <p>Hello guys! I hope you are all well.</p>
<p>I have a problem which I try (not successfully) to solve <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">… (It’s about my master’s thesis).<br>
Anyway, I coded a fully connected feed forward neural network for supervised regression with PyTorch (4 layers, 32 neurons per layer). It is one dimensional (1 input dim., 1 output dim.), i.e. the selected hypothesis of the NN can be plotted in a x-y plane. So now imagine I plot the hypothesis in x-y coordinates, i.e. I use a created data set X and apply Function = model(X), then plot(Function). I now would like to have the information about the slope of that Function. I know, in PyTorch it is a Tensor containing data points in the size of X. Nevertheless, is there any way to get the information about the derivative/slope of the Function? I don’t want the slope w.r.t. the weights, I need the slope of x w.r.t. y!!</p>
<p>Please help!! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tom: <p>if you do <code>x.requires_grad_()</code> you can compute the derivative of y w.r.t. x using <code>torch.autograd.grad</code>. The derivative of x w.r.t. y is the inverse (1/x) by the <a href="https://en.wikipedia.org/wiki/Inverse_function_theorem#Statement_of_the_theorem" rel="nofollow noopener">inverse function theorem</a>.<br>
Of course, for 1d, approximation by finitite differences also works well.</p>
<p>Best regards</p>
<p>Thomas</p>

Beyonder: <p>Hi Tom,</p>
<p>thanks for your response. I would like to try autograd, but I don’t know where to put it. Currently I put it after training my NN like that:</p>
<pre><code class="lang-auto">model.eval()
with torch.no_grad():
    Predict = model(XTest)
    Function = model(X)

XGrad = Variable(X, requires_grad=True)
FGrad= Variable(Function, requires_grad=True)
Grad = torch.autograd.grad(outputs=FGrad, inputs=X,
                           grad_outputs=torch.ones_like(FGrad), allow_unused=True)
</code></pre>
<p>But, if I print Grad, it is basically (None,).</p>
<p>Could you be a little more precise please how to implement it :)?</p>
Copy and restore the hidden state of a RNN network

Luca_Pamparana: <p>I am using a LSTM in my model and my <code>forward</code> function is  defined as:</p>
<pre><code class="lang-auto">...
self.rnn = nn.LSTM(...)

def forward(self, input, hidden):
   #  dropout, linear embedding
    emb = self.drop(self.encoder(input.contiguous().view(-1, self.num_in_features)))  
    emb = emb.view(-1, input.size(1), self.rnn_hid_size)

    output, hidden = self.rnn(emb, hidden)
    output = self.drop(output)
    # [(seq_len x batch_size) * feature_size]
    decoded = self.decoder(output.view(output.size(0) * output.size(1), output.size(2)))
    # [ seq_len, batch_size, feature_size]
    decoded = decoded.view(output.size(0), output.size(1), decoded.size(1))

    return decoded, hidden, output

</code></pre>
<p>I am doing some inference as follows:</p>
<pre><code class="lang-auto">model.eval()
hidden = model.init_hidden(1)

with torch.no_grad():
    for i in range(end_point):
        if i &gt;= start_point:
            out, hidden, _ = model.forward(out, hidden)
       else:
            out, hidden, _ = model.forward(my_input_seq[i].unsqueeze(0), hidden)
</code></pre>
<p>Now when I run this statement</p>
<pre><code class="lang-auto">out, hidden, _ = model.forward(out, hidden)
</code></pre>
<p>After I get  the output, I want to undo this statement i.e. restore the LSTM state to before the call. I am guessing this would mean somehow undoing or restoring the hidden state to before the call. What would be a fast (and hopefully easy) way to achieve this in pytorch?</p>

christopherkuemmel: <p>Hi <a class="mention" href="/u/luca_pamparana">@Luca_Pamparana</a>,</p>
<p>I hope I don’t get you wrong, but can’t you just omit the return values of the hidden states in that line:</p>
<aside class="quote no-group" data-username="Luca_Pamparana" data-post="1" data-topic="71199">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luca_pamparana/48/14310_2.png" class="avatar"> Luca_Pamparana:</div>
<blockquote>
<p>out, hidden, _ = model.forward(out, hidden)</p>
</blockquote>
</aside>
<p>And change it to</p>
<pre data-code-wrap="python"><code class="lang-python">out, _, _ = model.forward(out, hidden)
</code></pre>
<p>Otherwise if you need the information of the hidden states at a different line in your code I would go for creating an extra variable. If you run into memory issues with that on gpu, you could pass this value to the cpu (maybe add an extra param within the models forward for memory allocation).</p>
Pytorch - Linear Interpolation is DIFFERENT from PIL/OpenCV

soulslicer: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/4/3428afc474e3ad639b1f18d813da8ecf9a8aca35.png" alt="image" data-base62-sha1="7rpZWtzfydk8nXtcuh2CK1TS2NL" width="674" height="455"></p>
<p>This image shows us downsampling/applying linear interpolation from a 8x8 image to a 2x2 image.</p>
<p>The result in red is the result from using PIL. The result in black is from F.interpolate.</p>
<p>You can see that the results/ordering is completely different. Pytorch indexes from the top left while PIL and OpenCV index from bottom right.</p>
<p>This tiny bug has caused me some hell for the last 2 weeks and I have finally figured it out. Word of advice, don’t mix different tools for doing data processing.</p>

ptrblck: <p>I’m sorry to hear the painful debugging. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
Feel free to ask about such issues, in case you suspect other differences, as we already discussed it <a href="https://discuss.pytorch.org/t/torchvision-resize-vs-cv2-resize/47530/2">here</a> (which includes a link to the <a href="https://github.com/python-pillow/Pillow/issues/2718#issuecomment-333669547">PIL issue</a>).</p>
Question about image segmentation

Yolkandwhite: <p>Hi I’m trying to do image segmentation.</p>
<p>but I can not understand how it works.</p>
<p>I do know how CNN works in classification (something like MNIST classification) by the way</p>
<p>I just can’t understand how the mask image and labels work in the network.</p>
<p>do I need the datasets having like this???</p>
<p>(image, mask image)<br>
or<br>
[(image, label), (mask image, label)]</p>
<p>having both mask image and label makes me really confusing.</p>
<p>specifically I am trying to do segementation the brain cancer in Brain MRI image<br>
so do I need the dataset like<br>
(brain MRI image, mask image)<br>
or<br>
[(brain MRI, cancer or not),(mask image, cancer or not)]</p>
<hr>
<p>and another question.<br>
does the visualized weights shows the predicted mask image??</p>

ptrblck: <p>You would need the former, i.e. an input image and a target mask.<br>
The target mask would now have the shape <code>[batch_size, height, width]</code> and contain the class indices for each pixel in the range <code>[0, nb_classes-1]</code> (similar to a vanilla classification task, but with the additional spatial dimensions).</p>
<aside class="quote no-group" data-username="Yolkandwhite" data-post="1" data-topic="71123">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yolkandwhite/48/17604_2.png" class="avatar"> Yolkandwhite:</div>
<blockquote>
<p>does the visualized weights shows the predicted mask image??</p>
</blockquote>
</aside>
<p>No. Your model could use e.g. a conv layer as the output layer, which would return class logits in the shape <code>[batch_size, nb_classes, height, width]</code>. Again this would be similar to a standard classification use case, but your output would also have the additional spatial dimensions.</p>
Model.forward() returning batch of tensors

jkmm: <p>Hi,<br>
I want help to translate a PyTorch model into c++  for inference. The forward method is as below:<br>
def forward(self,inputs):<br>
blen = inputs.size(0) <span class="hashtag">#1</span><br>
tlen = inputs.size(1)<span class="hashtag">#no</span> of frames (no of samples/128)</p>
<pre><code>    h1 = torch.Tensor(blen,L1).uniform_(-1.0,1.0).to(self.dev)
    h2 = torch.Tensor(blen,L2).uniform_(-1.0,1.0).to(self.dev)
    h  = []

    for t in range(tlen):
        x  = inputs[:,t,:]
        h1 = self.gru1(x,h1)
        h2 = self.gru2(h1,h2)
        h3 = self.fc3(h2).clamp(min=0)
        h4 = self.fc4(h3)
        h.append(h4)
    y = torch.stack(h).permute(1,0,2)
    return F.log_softmax(y,dim=-1)
</code></pre>
<p>The returned tensor dimension is (1,24,2)   , where 24 is number of frames for a certain wav file.<br>
In the C++ code,<br>
auto input_tensor = torch::from_blob(ai-&gt;input_data_cat, {1,24,64},at::kFloat);<br>
std::vector<a>torch::jit::IValue</a> inputs;<br>
inputs.push_back(input_tensor);<br>
auto output = model.forward(inputs).toTensor();<br>
output dimension is only (1,1,2), where as I am expecting (1,24,2).</p>
<p>How can I fix this ?<br>
I am using libtorch-cxx11-abi-shared-with-deps-1.4.0+cpu.zip  and torch ver 1.4</p>
<p>Regards<br>
JK</p>
Pytorch install error 'ProtocolError('Connection aborted.', ConnectionResetError(10054

Abb_ey: <p>I am trying to install pytorch version 1.2.0 on Windows 10  and getting error : <strong>connection broken by 'ProtocolError(‘Connection aborted.’, ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/9/693882825d799a177947124691a88d3f3cd6141d.png" data-download-href="https://discuss.pytorch.org/uploads/default/693882825d799a177947124691a88d3f3cd6141d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/9/693882825d799a177947124691a88d3f3cd6141d.png" alt="image" data-base62-sha1="f0PcMcLw7eL5injnGufZ9k1wBSB" width="690" height="75" data-dominant-color="0D0C02"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1912×209 20.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I also tried with  pip install torch==1.2.0<br>
but getting another error:<br>
ERROR: Could not find a version that satisfies the requirement torch==1.2.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)<br>
ERROR: No matching distribution found for torch==1.2.0</p>

albanD: <p>Hi,</p>
<p>Do you use a 32bit python by any chance? Our windows binaries only support the 64bit python.</p>

peterjc123: <p>It seems that your connection to our download server is broken. Would you please retry a few times or switch from https to http? BTW, are you behind a proxy or sth?</p>
Mask tensors from indexes (e.g from multinomial sampling)

ludc: <p>Hi,</p>
<p>I am looking for a ‘good’ way to transform matrix of indexes to a onehot matrix (line by line). For example,</p>
<p>Transforming<code>[[1,3],[2,4]]</code> to <code>[[0,1,0,1,0],[0,0,1,0,1]]</code></p>
<p>Any idea ? Is there a way for example to extend <code>index_fill_</code> to take into account 2D tensors instead of 1D tensor ?</p>

apaszke: <p>I’d recommend using <code>scatter_</code>:</p>
<pre><code class="lang-python">&gt;&gt;&gt; x = torch.LongTensor([[1, 3], [2, 4]])
&gt;&gt;&gt; torch.zeros(2, 5).scatter_(1, x, 1)

 0  1  0  1  0
 0  0  1  0  1
[torch.FloatTensor of size 2x5]
</code></pre>

1112: <p>how to reverse this step ?<br>
i.e. from 2d mask tensor to 2d  index ?</p>

ptrblck: <p>You could use this code snippet to get all indices of the value you are comparing to:</p>
<pre><code class="lang-python">x = torch.LongTensor([[1, 3], [2, 4]])
res = torch.zeros(2, 5).scatter_(1, x, 1)
(res == 1.).nonzero()
</code></pre>
Transfer learning for 3D data

nevermoar: <p>Hi,</p>
<p>I am working on an image classification task which uses a labelled dataset of 3D medical images (volumetric CT brain scans).<br>
I want to try transfer learning but wasn’t sure if it’s possible to use pre-trained 2D CNNs on my dataset (and if so, how to do that).<br>
Alternatively, is it simply best to use pre-trained 3D CNNs (the only one I currently know of being this <a href="https://github.com/kenshohara/3D-ResNets-PyTorch" rel="nofollow noopener">3D ResNet for Action Recognition</a>?</p>
<p>I’m quite new at this so any help is appreciated. Many thanks.</p>

ptrblck: <p>Using the pretrained parameters of a 2D CNN won’t work out of the box, as the depth dimension is missing e.g. in the kernels.<br>
You could try to create this dimension, e.g. by repeating the filters, but I’m not sure, if this would be helpful.</p>
<p>Would it be possible to use each slice as a 2D image first and apply a standard transfer learning approach?<br>
This would make the workflow easier and could give you a signal, if your 3D model would benefit from the pretrained parameters.</p>
The grad in optimizer whether exec detach

as754770178: <p>I recently see optimizer code, I don’t understand whether exec <code>p.grad.detach_()</code>?</p>
<pre><code class="lang-python">    def zero_grad(self):
        r"""Clears the gradients of all optimized :class:`torch.Tensor` s."""
        for group in self.param_groups:
            for p in group['params']:
                if p.grad is not None:
                    p.grad.detach_()
                    p.grad.zero_()
</code></pre>
The size of tensor a (3) must match the size of tensor b (224) at non-singleton dimension 4

Levis: <p>Hi,</p>
<p>I have an annoying issue that I can’t handle.</p>
<p>Some say that it is from batch size, others from the metric used for a non/binary segmentation, etc…<br>
I tested different batch size, it is a binary segmentation, so I want to use IoU.</p>
<p>Quite lost, may you help me please ?</p>
<pre><code class="lang-auto">
from catalyst.dl.callbacks import DiceCallback, IouCallback, \
  CriterionCallback

runner.train(
    model=model,
    criterion=criterion,
    optimizer=optimizer,
    scheduler=scheduler,
    
    # our dataloaders
    loaders=loaders,
    
    callbacks=[
        # Each criterion is calculated separately.
        CriterionCallback(
            input_key="mask",
            prefix="loss_iou",
            criterion_key="iou"
        ),
        
        # metrics
        IouCallback(input_key="mask"),
    ],
    # path to save logs
    logdir=logdir,
    
    num_epochs=num_epochs,
    
    # save our best checkpoint by IoU metric
    main_metric="iou",
    # IoU needs to be maximized.
    minimize_metric=False,
    
    # for FP16. It uses the variable from the very first cell
    fp16=fp16_params,
    
    # for external monitoring tools, like Alchemy
    monitoring_params=monitoring_params,
    
    # prints train logs
    verbose=True,
)
</code></pre>

ptrblck: <p>Based on the shapes in the error message it seems you are trying to apply an operation on a channel-first and another channel-last tensor.<br>
Which line of code throws this error?</p>

Levis: <p>Here is the output, I agree it is about the shape but seeing this topic, I’m not so sure : <a href="https://forums.fast.ai/t/runtimeerror-the-size-of-tensor-a-115-must-match-the-size-of-tensor-b-64-at-non-singleton-dimension-1/50953/17" rel="nofollow noopener">https://forums.fast.ai/t/runtimeerror-the-size-of-tensor-a-115-must-match-the-size-of-tensor-b-64-at-non-singleton-dimension-1/50953/17</a></p>
<pre><code class="lang-auto">1/20 * Epoch (train):   0% 0/1 [00:00&lt;?, ?it/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-78-330e12609d6c&gt; in &lt;module&gt;()
     38 
     39     # prints train logs
---&gt; 40     verbose=True,
     41 )

14 frames
/usr/local/lib/python3.6/dist-packages/catalyst/utils/criterion/iou.py in iou(outputs, targets, classes, eps, threshold, activation)
     42         _sum = torch.sum
     43 
---&gt; 44     intersection = _sum(targets * outputs)
     45     union = _sum(targets) + _sum(outputs)
     46     # this looks a bit awkward but `eps * (union == 0)` term

RuntimeError: The size of tensor a (3) must match the size of tensor b (224) at non-singleton dimension 4
</code></pre>

ptrblck: <p>Could you print the shapes of <code>targets</code> and <code>outputs</code>, please?</p>
Validation Log loss lower than Training

brtrentini: <p>Hi everyone,<br>
this is my first post in the forum and I believe you can help me clarify one thing: is it acceptable to have validation log loss lowe (or much lower) than training log loss? And if so, how much this gap should be at most?</p>
<p>I am using Inception V3 to classify 120 classes of images with 30K training examples which I split into 21K images for training and 9k for testing using Stratified Random Sampling (30% of each class chosen at random for the validation set).</p>
<p>My model is pretty much the same as the pre-trained one and the only major change the number of classes which I set to 120.</p>
<p>Here’s my most recent callback:</p>
<pre><code class="lang-auto">Epoch 1/1499
⚠ train Loss: 6.3281 Acc: 0.0770
⚠ val Loss: 4.2735 Acc: 0.0965
(...)
Epoch 25/1499
⚠ train Loss: 4.2652 Acc: 0.3317
⚠ val Loss: 3.4767 Acc: 0.2571
</code></pre>
<p>I’m aware that if the gap between the two is too big and that training error is much smaller than validation error then I am over-fitting.</p>
<p>And then there’s also this thread: <a href="https://twitter.com/aureliengeron/status/1110839223878184960" rel="nofollow noopener">https://twitter.com/aureliengeron/status/1110839223878184960</a></p>
<p>Where Aurélien Geron,author of the book Hands-On MachineLearning with ScikitLearn, Keras and TensorFlow explains that this might be normal… but I’m not sure how credible this is.</p>
<p>This happens regardless of data augmentation technique (even if train/val has the same transforms using just cropsize and ToTensor()), learning rate or optimiser. More recently I’ve been exploring cyclical learning rate as well… Changed batch sizes, used different techniques for sampling… nothing seems to change this behaviour.</p>
<p>I also based myself mostly on: <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html</a><br>
to build the scaffolding for my code. Difference is the dataset and some data augmentation changes.</p>
<p>So…Is this normal or is it likely that my code is badly implemented?</p>
<p>Thanks a lot!</p>

ptrblck: <p>Aurelien’s explanations seem reasonable and it’s what I observe usually.<br>
If you want to make sure dropout and the running estimates might be the cause for the gap, you could set your model to <code>model.eval()</code> and use the training dataset after the epoch (same as where you are calculating the validation loss) and compare these two values.</p>

brtrentini: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a> - Did as suggested and, indeed, the values have changed. I am now experiment a few other pre-trained models. Cheers!</p>