akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Compile torch model to not need loading from file

rtkaratekid: <p>Hi everyone, just first wanted to say that I really enjoy working with libtorch.</p>
<p>Is there a way to compile my C++ code with the model in such a way that when the executable is run, it doesn’t need to load the model from the file, but rather has the model “hard coded” into the program itself? That way I could run the executable with no external file dependencies.<br>
Is this something that could be done with torchscript?</p>
<p>I ask because that would be really awesome for deployment purposes where I try to make my executable as light and as fast as possible. If that’s not the case, I might have to set it up as a daemon communicating on local sockets.</p>
<p>Cheers!</p>
<p>Edit: to be really clear, my end goal is to incorporate this into C code where it doesn’t need to load the file each time the function is called since the file loading is quite the bottleneck.</p>

albanD: <p>This is very likely a bad idea but can’t you bundle the file content into a static string when you compile and load from that string?</p>

rtkaratekid: <p><a class="mention" href="/u/alband">@albanD</a> Haha! That was my first idea as well and I was hoping there was something a little less hacky. I may try it out and report back.</p>
<p>It would be really cool if I could compile the model itself down into a binary without such shenanigans though. I’m just not sure how it would be done.</p>

rtkaratekid: <p><a class="mention" href="/u/alband">@albanD</a> it is indeed a bad idea! The reasoning is actually interesting though. I thought (for some reason) that the loading the model from the file was the bottleneck, but the bottleneck is actually libtorch reading the model settings and creating the algorithm itself. Because of that there is no speed boost from just hard coding the string into the file.</p>
<p>That being said, since the libtorch library essentially “compiles” the model, it would be cool if there were a functionality to save the model as an executable. Unfortunately I don’t have the time to hack on something like that currently. But maybe in the future if I’m looking for a project I’ll work on that.</p>
<p>As for right now I’m just setting my model up as a daemon communicating with other processes through local sockets and seems adequate.</p>
Type object 'TabularDataset' has no attribute

Tank: <p>I keep running into errors of the form: “type object ‘TabularDataset’ has no attribute”:</p>
<pre><code>def tokenize_text(text):
   return [letter for letter in text]

TEXT = Field(tokenize=tokenize_text,
         eos_token = '&lt;eos&gt;')

train_datafields = [("SRC", TEXT), 
              ("TRG", TEXT)]

train_data, valid_data = TabularDataset.splits( 
    train = 'train.csv',
    validation = 'valid.csv',
     format = 'csv',
     skip_header = True,
     fields = train_datafields)
</code></pre>
<p>specific error in this case is: “type object ‘TabularDataset’ has no attribute ‘name’”</p>
<p>However, if I do the following:</p>
<pre><code>def tokenize_text(text):
    return [letter for letter in text]

TEXT = Field(tokenize=tokenize_text,
         eos_token = '&lt;eos&gt;')

 train_datafields = [("SRC", TEXT), 
              ("TRG", TEXT)]


train_data = TabularDataset(
        path= "train.csv",
        format='csv',
        skip_header=True, 
        fields=train_datafields)

TEXT.build_vocab(train_data, min_freq = 2)

Batch_size = 128
train_iterator = BucketIterator.splits((train_data), 
                      batch_size = Batch_size)
</code></pre>
<p>I get the following error: “AttributeError: ‘Example’ object has no attribute ‘sort_key’”</p>
Placing dropout when putting model.eval

Lewis: <p>Hi. In evaluation mode, do we need to still put the line of dropout? This is the code</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

x = torch.randn(4,4)
drop = nn.Dropout(0.5)
model = nn.Linear(4,6)
model.train()
model(drop(x))

model.eval()
d = model(drop(x))
n = model(x)
print(d == n)
</code></pre>
<p>This gives a matrix of 0s (all False). So which one should be correct? <code>model(drop(x))</code> or <code>model(x)</code>? I’ve seen some GitHub codes use <code>model(x)</code>, but then in this case how would the model scale the output at test time if <code>drop(x)</code> is removed at test time?</p>

ptrblck: <p>Usually you would create a model containing all layers. If you then call <code>model(inputs)</code>, the forward pass will be executed (in your case with the linear layer and dropout). If your model is quite simple, you could use <code>nn.Sequential</code>, otherwise you would just write a custom <code>nn.Module</code>.</p>
<p>In your example code, you would have to call <code>drop.eval()</code>, to set the dropout layer to evaluation, as <code>model</code> does not contain this layer.</p>

Lewis: <p>Thanks alot <a class="mention" href="/u/ptrblck">@ptrblck</a> for your answer. If I had them in a class like:</p>
<pre><code class="lang-auto">class Model(nn.Module):

    def __init__():

        super(Model, self).__init__()
        self.layer = nn.Linear(4,6)
        self.dropout = nn.Dropout(p=0.5)
        self.classification = nn.Linear(6,3)

    def forward(x):
        out = self.dropout(self.layer(x))
        return self.classification(out)
</code></pre>
<p>I know that if I do model.eval, then dropout would switch to evaluation mode. However, assuming that I do not want to follow the forward function (call the layers manually), do I need to put <code>model.dropout</code> in between, as in:</p>
<pre><code class="lang-auto">model = Model()
model.eval()
x = torch.randn(4,4)
out1 = model.layer(x)
# Do I need to put model.dropout(out1)?
out2 = model.classification(out1)
</code></pre>
<p>Thanks!</p>

ptrblck: <p>If you want to call the modules manually, you would have to call <code>self.dropout</code> as well.</p>

Fawaz_Sammani: <p>Actually it dosent matter whether you place self.dropout in evaluation or not. PyTorch uses inverted dropout, so the scaling is done at training time only, so that during evaluation time nothing needs to be scaled.</p>
How to compute paired-wise cosine distance using nn.CosineSimilarity

John1231983: <p>I am using toch.bmm to compute the paired-wise cosine distance between <code>BxDxN</code> and <code>BxDxN</code>. It will return a matrix size of NxN instead of a triangle vector in the matrix in the  <code>nn.CosineSimilarity</code>. How to use  <code>nn.CosineSimilarity</code>  to get full cosine matrix as torch.bmm did?  I cannot use <code>torch.bmm</code> because of CUDA memory error. This is my code.</p>
<pre><code class="lang-auto">input1 = torch.randn(2, 4, 4)
input2 = torch.randn(2, 4, 4)
#Using bmm
x_norm = input1 / torch.norm(input1, p=2, dim=1, keepdim=True)
y_norm = input2 / torch.norm(input2, p=2, dim=1, keepdim=True)
cosine_sim = torch.bmm(x_norm.transpose(2,1), y_norm)
print('Using bmm: \n', cosine_sim)
# Pytorch CosineSimilarity
cos = nn.CosineSimilarity(dim=1, eps=1e-6)
cosine_sim = cos(input1, input2)
print('Using nn.CosineSimilarity: \n', cosine_sim)
</code></pre>
<p>The output is</p>
<pre><code class="lang-auto">Using bmm: 
 tensor([[[-0.0230,  0.2983,  0.0487,  0.3974],
         [-0.5747,  0.5513, -0.6436, -0.1389],
         [-0.3876, -0.2107,  0.7093, -0.4929],
         [-0.3446, -0.5347,  0.6372, -0.6423]],

        [[-0.3842, -0.0349,  0.1621,  0.6400],
         [ 0.6776, -0.4812, -0.3169, -0.7976],
         [-0.5251, -0.1258,  0.9381, -0.2379],
         [-0.1517,  0.7164,  0.8332,  0.1668]]])
Using nn: 
 tensor([[-0.0230,  0.5513,  0.7093, -0.6423],
        [-0.3842, -0.4812,  0.9381,  0.1668]])
</code></pre>
Training Loss increases suddenly

surya00060: <p>I am trying to train a network on Imagenet dataset and i observed an unusual Behaviour with SGD. My loss function keeps on decreasing and suddenly increases. I am using SGD with momentum with no schedulers. I have attached my train and test plots. Also attached the code for it. I used lr=1e-3.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/9/797c2827229387a8d3dfa5b1c6291a82ef5a1d36.png" alt="Train" data-base62-sha1="hkHMhu9JTQqYAsVloSfnuoayYYK" width="376" height="275"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc60fc618cf8980f5ff2fe962fd81fe176867741.png" alt="Test" data-base62-sha1="ta1c6rMEO3TrwST12E9meYABXtT" width="373" height="282"></p>
<p>I am not able to understand why loss suddenly increases.<br>
Thanks in advance for any responses.</p>

isalirezag: <p>One explanation is that your network was going to local minimum and suddenly got out of it</p>
How can I use my custom distance in TripletMarginLoss?

isalirezag: <p>I’d like to use  <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.TripletMarginLoss" rel="nofollow noopener">TripletMarginLoss</a>, but i like to modify the distance <code>d</code> with some custom fuction.</p>
<p>Can anyone please help and show me how I can rewrite the TripletMarginLoss in a way that i can modify the parts.<br>
Looking at the source code it gets me to <code>def triplet_margin_loss</code>  which is not that much useful either <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

Oli: <p>You can write a normal python function. You have to return the loss as a scalar, and all the operations have to be through pytorch tensors.</p>
<p>So, its quite chill <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

isalirezag: <p>Thanks for your answer.<br>
I dont think you can write a normal python function and use it in pytorch. Unless they change it.<br>
It has to be in pytorch to be able to do backpropagation.<br>
Can you give me a simple example?</p>

Oli: <p>Sure. As I said, you can write a normal python function as long as you don’t break the pytorch <em>magic</em> i.e. do the calculations on tensors to not break the gradients.</p>
<pre><code class="lang-auto">import torch

def myloss(t1, t2):
    loss_per_batch = (t1 - t2).abs().mean(dim=1)
    return loss_per_batch.mean()


tensor1 = torch.randn(2, 3)
tensor2 = torch.randn(2, 3)

loss = myloss(tensor1, tensor2)
print(loss)
</code></pre>

isalirezag: <p><a class="mention" href="/u/oli">@Oli</a> is it something new in pytorch?, i.e. I thought using normal python function  will always break the paytorch gradient</p>

Oli: <p>No, not new. Don’t know if it was always like that</p>
PyTorch 1.0 Wheel for Raspberry Pi 3 B+

acburigo: <p>Has anyone built a PyTorch 1.0 wheel for Raspberry Pi 3 B+?</p>

tom: <p>I didn’t - mainly because I don’t have a Raspberry Pi - but it should be relatively straightforward if you do it on a raspberry itself. I think Soumith and others did work on compiling for ARM for NVidia Jetson.<br>
If you need to cross-compile, maybe the cross-compilation of <a href="https://github.com/pytorch/pytorch/pull/16242" rel="nofollow noopener">libtorch on android that I recently built</a> can be of inspiration. That does not build the Python module.<br>
One thing to keep in mind is that you absolutely want NNPack when running on an ARM CPU.</p>
<p>PyTorch on Raspberry Pi seems a very exciting project!</p>
<p>Best regards</p>
<p>Thomas</p>

LeviViana: <p>This might be useful: <a href="https://discuss.pytorch.org/t/pytorch-1-3-wheels-for-raspberry-pi-python-3-7/58580" class="inline-onebox">PyTorch 1.3 wheels for Raspberry Pi (Python 3.7)</a></p>
Confusion with LR Scheduler step() In Pytorch 1.1.0 and Later

t.ouyang: <p>Today I upgraded pytorch from 1.1.0 to 1.2.0 for some reasons. But when running the same code on pytorch 1.2.0 I get a warning as follows:</p>
<blockquote>
<p>/usr/local/lib/python3.5/dist-packages/torch/optim/lr_scheduler.py:82: UserWarning: Detected call of <code>lr_scheduler.step()</code> before <code>optimizer.step()</code>. <strong>In PyTorch 1.1.0 and later</strong>, you should call them in the opposite order: <code>optimizer.step()</code> before <code>lr_scheduler.step()</code>.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule.See more details at <a href="https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate" class="inline-onebox" rel="noopener nofollow ugc">torch.optim — PyTorch 2.1 documentation</a></p>
</blockquote>
<p>What is the first value of the lr schedule? Is that means the scheduler will start the lr from 0.0005 for the first epoch if I set lr=0.001 and gamma=0.5 and call <code>scheduler.step()</code> before <code>optimizer.step()</code>?</p>
<p>BUT I printed the optimizer and <code>scheduler.get_lr()</code> every iteration and found both of the calling order (i.e. <code>scheduler.step()+optimizer.step()</code> and <code>optimizer.step()+scheduler.step()</code>) start the lr from 0.001 for the first epoch.</p>
<p>Is there anything I misunderstand and what’s going on here?</p>
<p>Thanks a lot.</p>
Can not load model

Norfo: <p>Hello every one!</p>
<p>I  use pytorch for GCN-slam (<a href="https://github.com/jiexiong2016/GCNv2_SLAM" rel="nofollow noopener">https://github.com/jiexiong2016/GCNv2_SLAM</a>). I have a problem when I try to load model by torch::jit::load(). I constantly get this error: “type_ != nullptr INTERNAL ASSERT FAILED at C:\w\1\s\tmp_conda_3.6_171155\conda\conda-bld\pytorch_1570813991702\work\torch/csrc/jit/ir.h:165, please report a bug to PyTorch.  (type at C:\w\1\s\tmp_conda_3.6_171155\conda\conda-bld\pytorch_1570813991702\work\torch/csrc/jit/ir.h:165)<br>
(no backtrace available)”. I use win10, torch 1.3.0a0+b728ffa, cuda v10.1, Visual Studio 2017.</p>
<p>I will appreciate if someone can help me.</p>

driazati: <p>This seems like a bug, could you repost this as an <a href="https://github.com/pytorch/pytorch/issues/new?template=bug-report.md" rel="nofollow noopener">issue on GitHub</a>? That will let us track this better.</p>
<p>Also could you upload the <code>.pt</code> file you are loading to your repo so we can try to repro this issue directly?</p>

Norfo: <p>Thanks! I create issue <span class="hashtag">#28276</span> (<a href="https://github.com/pytorch/pytorch/issues/28276" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/28276</a>). You can download models from <a href="https://github.com/jiexiong2016/GCNv2_SLAM/tree/master/GCN2" rel="nofollow noopener">https://github.com/jiexiong2016/GCNv2_SLAM/tree/master/GCN2</a> (any *.pt model). I can also check any working model for this bug if you share it with me.</p>
How to custom sigmoid activation function

astri: <p>Hello all<br>
I am beginner in deep learning who recently researching using keras and pytorch. I want to make custom activation function that based on sigmoid with a little change like below.</p>
<p>new sigmoid = (1/1+exp(-x/a))</p>
<p>what i do in keras is like below</p>
<pre><code class="lang-auto">#CUSTOM TEMP SIGMOID
def tempsigmoid(x):
    nd=3.0 
    temp=nd/np.log(9.0) 
    return K.sigmoid(x/(temp)) 
</code></pre>
<p>i tried by making def class in pytorch but not succeed, how could i custom sigmoid in pytorch?</p>

ptrblck: <p>Your code should work, if you replace the numpy as Keras calls with their PyTorch equivalent:</p>
<pre><code class="lang-python">def tempsigmoid(x):
    nd=3.0 
    temp=nd/torch.log(torch.tensor(9.0)) 
    return torch.sigmoid(x/(temp)) 
</code></pre>

astri: <p>thanks for your reply,</p>
<p>i tried but another error comes it is said that</p>
<p>temp=nd.torch.log(0.9)</p>
<p>log():argument ‘input’(position 1) must be Tensor, not float</p>
<p>i guess the calculation can not be done because i use 0.0 number as float?</p>

ptrblck: <p><code>torch.log</code> expects a tensor, not a <code>float</code> value.<br>
My code should work <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

astri: <p>I tried your code and when i called the funcion in the model by</p>
<pre><code class="lang-auto">tempsigmoid(X_train)
</code></pre>
<p>there is an error <strong>TypeError: len() of a 0-d tensor</strong> ?</p>

ptrblck: <p>What shape does <code>X_train</code> have?<br>
I’ve tested it with different shapes and am not sure, which function throws this error:</p>
<pre><code class="lang-python">def tempsigmoid(x):
    nd=3.0 
    temp=nd/torch.log(torch.tensor(9.0)) 
    return torch.sigmoid(x/(temp)) 

tempsigmoid(torch.randn(1))
tempsigmoid(torch.tensor(1.))
tempsigmoid(torch.randn(1, 1))
</code></pre>

astri: <p>thank you for your reply</p>
<p>my X_train data shape is ('train data shape : ', (42000, 785))</p>
<p>i am using reference from here <a href="https://www.kaggle.com/pinocookie/pytorch-simple-mlp/notebook" rel="nofollow noopener">kaggle</a> and want to make some modification</p>
Is it possible to have torch.cat in model definition?

falmasri: <p>Is it possible to have torch.cat in the model definition so when I use <a href="https://github.com/lutzroeder/netron" rel="nofollow noopener">Netron</a>  I can visualize the skip connections?</p>

Nikronic: <p>Hi,<br>
If I understood correctly, you mean something like Unet or resnet by skip-connections, right? If yes, the answer is yes too! I have not used Netron but when I tried visualizing model (Unet: skip connection between encoder and decoder layers) using Tensorboard or TensorboardX, it was ok and I could see the connections.</p>
<p>Here is the link to my <a href="https://github.com/Nikronic/CoarseNet/blob/master/CoarseNet.py" rel="nofollow noopener">implementation on GitHub</a></p>
<p>But the main part of code that do the <code>cat</code> thing is here:</p>
<pre><code class="lang-auto">class Expand(nn.Module):
    def __init__(self, input_channel, output_channel, ks=4, s=2):
        """
        This path consists of an up sampling of the feature map followed by a
        4x4 convolution ("up-convolution" or Transformed Convolution) that halves the number of
        feature channels, a concatenation with the correspondingly cropped feature map from Contract phase
        :param input_channel: input channel size
        :param output_channel: output channel size
        """
        super(Expand, self).__init__()
        self.layers = CE(input_channel * 2, output_channel, ks, s)

    def forward(self, x1, x2):
        delta_x = x1.size()[2] - x2.size()[2]
        delta_y = x1.size()[3] - x2.size()[3]
        x2 = F.pad(x2, pad=(delta_x // 2, delta_y // 2, delta_x // 2, delta_y // 2), mode='constant', value=0)
        x = torch.cat((x2, x1), dim=1)
        x = self.layers(x)
        return x
</code></pre>
<p>And for visualizing I did not specified any particular arg, etc and you do not need too, because the graph of the model constructed by the flow of data through network, so if you have a skip-connection, the data would flow that way too.</p>

falmasri: <p>Yes exactly, this way of doing concatenation doesn’t appear on Netron visualization. I converted the model to OMNX first and it worked correctly. I think the problem with torch.save function.</p>

Nikronic: <p>The skip-connections does not contain any weights right, so <code>torch.save</code> won’t save them I think. I am not sure about this or maybe eager execution fails the <code>save</code> function to cover all aspects?</p>

martinr: <p>Yes, you are correct. To visualise a Pytorch model with Netron, you have to save it in ONNX as it also saves the graph. I use <code>torch.onnx.export()</code> for this. With <code>torch.save()</code> you can just visualise individual blocks in Netron and see the weights, but not the network graph.</p>
Tensor.gather triggers in place operation error when calling backward

M_Tu: <p>Pytorch 1.2 on linux with cuda 10.0.</p>
<p>Got the tracing info by applying torch.autograd.set_detect_anomaly(True).</p>
<p>The error shows the</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [2, 1, 1024]] is at version 1; expected version 0 instead. Hint: the backtrace further above shows       the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>and it points to</p>
<pre><code class="lang-auto">states = hidden_states.gather(-2, positions)
</code></pre>
<p>Any ideas? Thanks.</p>

albanD: <p>Hi,</p>
<p>For inplace errors, the anomaly mode points to the operation that saved the Tensor for later use. Not the one that modified it inplace.</p>
<p>But given that it conplains that a LongTensor is faulty. I’m fairly confident in the fact that you change <code>positions</code> inplace later in your code <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> That would be causing the issue as <code>gather()</code> needs these positions to be able to compute its backward.</p>

M_Tu: <p>Thanks a lot for your reply Alban.</p>
<p>I checked the position variable, I only changed it in one place within the forward() func:</p>
<pre><code class="lang-auto">positions = positions.unsqueeze(-1).expand(-1, -1, fdim)
</code></pre>
<p>and the same error still exists. If I don’t use positions as an input argument, actually this error disappears.</p>
<p>Also, I’m wondering what’s version of a variable means. I got different version numbers if I change some of the operations on positions.</p>
<p>Thanks again.</p>

albanD: <p>Versions tracks how many times it has been changed inplace.<br>
Inplace modifications are anything like <code>positions[foo] = </code> or <code>positions.add_(foo)</code> or <code>positions += 2</code>.<br>
a simple fix you can use is replace your gather call to:</p>
<pre><code class="lang-auto">states = hidden_states.gather(-2, positions.clone())
</code></pre>
<p>The clone here will make sure that you don’t share memory between the version used by gather and the one used in the rest of your code. That way inplace operations in the rest of your code won’t be a problem for gather.</p>

M_Tu: <p>Thanks Alban. I used clone() and it works now. Wondering if this will affect the backward behavior.</p>

albanD: <p>No, the gradients will flow through the clone op and will be computed properly <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Select rows of the tensor whose first element is equal to some value

vvanirudh: <p>Given a tensor like</p>
<pre><code>a =  1  1  1
     1  1  1
     1  1  1
     2  1  1
     2  1  1
</code></pre>
<p>I would like to obtain all rows starting with 2, i.e. I would like to obtain</p>
<pre><code>b = 2 1 1 
    2 1 1
</code></pre>
<p>Indexing using <code>a[a[:, 0]==2]</code> doesn’t seem to work and <code>index_select</code> requires indices to be known and so does <code>gather</code>.</p>

jekbradbury: <p><code>a[(a[:, 0] == 2).nonzero().squeeze(1)]</code></p>

hendy: <p>Thanks! What if I need two criteria columns at once? for example, select rows where column A equals 2 and column B equals 5 ?</p>

Cheng_Zhang: <p>Hi Hendy, did you find a solution for this?</p>

Cheng_Zhang: <aside class="quote no-group" data-username="hendy" data-post="3" data-topic="1718" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hendy/48/6167_2.png" class="avatar"> hendy:</div>
<blockquote>
<p>Thanks! What if I need two criteria columns at once? for example, select rows where column A equals 2 and column B equals 5 ?</p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/jekbradbury">@jekbradbury</a> Could you please also kind respond to this question? Thanks a lot!</p>
Include directory structure

markl: <p>I am using bazel as my build system instead of cmake. I noticed that the file <code>torch.h</code> is located at <code>include/torch/csrc/api/include/torch/torch.h</code> instead of <code>include/torch/torch.h</code></p>
<p>I assume then I should configure the project to have the files at  <code>include/torch/csrc/api/include/torch/xxx</code> made available via <code>&lt;torch/xxx&gt;</code>.</p>
<p>Is that correct? And are there other include paths that should be configured unconventionally? And what is the rational for this unconventional directory structure?</p>

albanD: <p>I usually look at the <a href="https://github.com/pytorch/pytorch/blob/a5ac7f6387c517cee61f869db3a3c80274a811a3/torch/utils/cpp_extension.py#L501" rel="nofollow noopener">cpp_extension include path</a> to answer this.<br>
In particular, I don’t think you should be using this particular include anymore. But I have to admit I don’t the rational behind this.</p>

markl: <p>Thanks <a class="mention" href="/u/alband">@albanD</a>. I think your link is broken though?</p>
<p>As for the header file, it is needed for this example <a href="https://pytorch.org/cppdocs/frontend.html#end-to-end-example" rel="nofollow noopener">https://pytorch.org/cppdocs/frontend.html#end-to-end-example</a></p>

albanD: <p>Fixed the link, sorry about that.</p>
<p>Then I don’t know about this include, sorry <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Training multiple models on one GPU simultaneously

limo94: <p>So I’m trying to train multiple (let’s say 10) models simultaneously on a single GPU. The way I’m trying to go about this is have a list of models, and optimizers. The models are stored on CPU, and I iterate through the model list, putting one model on the GPU, training for an epoch, and then putting the model back on CPU and moving to the next model on the list. The problem is, usually one of the models in the list has a very high loss value (nan sometimes). I’m wondering if there is some sort of leak happening?</p>

albanD: <p>Hi,</p>
<p>This should work properly if you move models with <code>.cuda()</code> and <code>.cpu()</code>.<br>
Can you give more details about what you’ve tried? Does running a single model works?</p>

limo94: <p>Hi,</p>
<p>Thanks for the quick response. I start with the list of 10 networks, and move them to CPU with</p>
<p>model[i].to(‘cpu’)</p>
<p>and I put them in eval mode. Then, when I want to train one of them for an epoch, I say</p>
<p>model[i].to(‘cuda’)<br>
model[i].train()<br>
optimizer[i].zero_grad()<br>
model[i].zero_grad() – I know this is redundant, just thought I’d try it<br>
…<br>
loss.backward()<br>
optimizer[i].step()<br>
…<br>
models[i].eval()<br>
models[i].to(‘cpu’)</p>
<p>Running a small number of models (2) seems to work fine.</p>

albanD: <p>You do call <code>optimizer[i].zero_grad()</code> (or the model version) at every iteration during your epoch right? You can check this discussion if you do not: <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903" class="inline-onebox">Why do we need to set the gradients manually to zero in pytorch?</a></p>
<p>Otherwise, it looks good !</p>

limo94: <p>Yes. I call it after every batch is fed through the network and the loss is calculated/backward is called.</p>

albanD: <p>Then given what you shared, I don’t see any reason for a model to behave differently than the others.<br>
Is it always the same model in the list that lags behind? Do you see any pattern?</p>
Recover from errors in data loader: What is the best way?

blackberry: <p>What is the best way to recover from errors in our dataset’s <code>__getitem__</code> function and skip the problematic item(s) in the dataset without crashing? For example, the image could not be loaded due to some problem (eg, loading from a network disk).</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
  def __init__(self, image_dir, image_list_file, transform=None):

    self.transform = transform
    self.image_dir = image_dir
    self.image_list = pickle_load(image_list_file)

  def __len__(self):
    return len(self.image_list)

  def __getitem__(self, index):
    image_file, label = self.image_list[index]
    image_path = os.path.join(self.image_dir, image_file)
    image = Image.open(image_path).convert('RGB')
    if self.transform:
         image = self.transform(image)
    return image, label

for images, labels in dataloader:
      # process the batch: if an error in __getitem__ the program crashes here
</code></pre>
<p>I am thinking of wrapping the for loop (<code>for images, labels in dataloader</code>) in a <code>try-catch</code> block. Is there a better way you know of?</p>

JuanFMontesinos: <p>Do a “fake” getitem and then catch it in the real <strong>getitem</strong> generating a random idx if fails</p>
NaNs in input data breaking gradients

Alexey_Nikulkov: <p>I have a usecase for which I want some components of my input data to be NaN (slate data with variable number of items per slate, I want to insert NaNs instead of item features for empty item slots). The NaN propagate through the forward pass properly and I ignore them when calculating the loss (by using the nanmean function from <a href="https://github.com/pytorch/pytorch/issues/21987" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/21987</a>)</p>
<p>The problem is that I am now getting NaN gradients even when NaN outputs are not used as inputs to the loss function. Minimal example:</p>
<pre><code class="lang-auto">lin_layer = nn.Linear(1, 2)
X_ones = torch.ones(2, 1)
X_ones[0, :] = float('NaN') # [[nan], [1]]

# Approach 1: remove NaN data from input to the linear layer.
# Gradients calculated succesfully

X_input = X_ones[1,:] # select non-NaN inputs
output = lin_layer(X_input)
loss = output.mean()
print(f'loss={loss} (mean of {output})')
lin_layer.zero_grad()
loss.backward()
print(f'grad={lin_layer.weight.grad}')

# Approach 2: remove NaN data from input to the loss function.
# Gradient calculation fails

output = lin_layer(X_ones)
loss = output[1,:].mean() # select non-NaN outputs
print(f'loss={loss} (mean of {output[1,:]})')
lin_layer.zero_grad()
loss.backward()
print(f'grad={lin_layer.weight.grad}')
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">loss=-0.25358104705810547 (mean of tensor([-0.2736, -0.2336], grad_fn=&lt;AddBackward0&gt;))
grad=tensor([[0.5000],
        [0.5000]])
loss=-0.25358104705810547 (mean of tensor([-0.2736, -0.2336], grad_fn=&lt;SliceBackward&gt;))
grad=tensor([[nan],
        [nan]])
</code></pre>

albanD: <p>Unfortunately, any <code>nan</code> will create <code>nan</code> for any number it touches. So they have a tendancy to propagate. And this is the expected behavior here.<br>
You definitely want to perform the masking before using them in any computations as much as possible.</p>

Alexey_Nikulkov: <p>Thanks for the reply! In Approach 2, the loss depends only on output[1,:], the elements of which are non-NaN, therefore the loss is non-NaN as well. Wouldn’t we expect the gradients to propagate properly in this case since the NaN tensor elements are “irrelevant” - they don’t feed into the loss function?</p>
<p>Or is it the case that if at least one component of a tensor is NaN, then the gradients can’t flow through this tensor?</p>

albanD: <p>Unfortunately pytorch cannot know about the “irrelevant” part and only uses the chain rule. And in the chaine rule, you get 0 * nan = nan.<br>
There are a few issues on github that discuss this problem: <a href="https://github.com/pytorch/pytorch/issues/15506" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/15506</a> and <a href="https://github.com/pytorch/pytorch/issues/26799" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/26799</a> for example.</p>
nn.DataParallel - "Was asked to gather along dimension 0, but all input tensors were scalars"

param_eter: <p>I’m using a BERT model from the transformers package (by huggingface). I’ve now got access to more than 1 GPU and so am wrapping my model in nn.DataParallel() to take advantage of them.</p>
<p>I get the following warning: <code>UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.</code><br>
This is generated by this bit of code<br>
<code>train_output = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)</code></p>
<p>This returns loss tuple. I extract the loss and take the mean as I’m spreading the batches across 2 GPUs, and so receive 2 losses.</p>
<pre><code class="lang-auto">loss = train_output[0]
loss = loss.mean()
</code></pre>
<p>One thing I want to understand is why the forward pass is giving me this warning? I’m assuming that nn.DataParallel is doing something to the inputs that wasn’t happening when it was a single GPU.</p>

ptrblck: <p>As the warning says, <code>nn.DataParallel</code> will chunk the data in dim0 (batch dimension) and send each chunk to the corresponding device.<br>
E.g. if you are dealing with an input of <code>[16, 3, 100]</code> and two GPUs, each GPU will receive an input of <code>[8, 3, 100]</code>.</p>
<p><code>b_input_ids</code> doesn’t seem to have the right shape, so could you check, what type and shape you are using at the moment?</p>

param_eter: <p>So my initial input is a dataset of around 800 rows and 64 columns (tokenised texts of length 64).</p>
<p>I’m using a batch size of 64, and my data consists of tokenised texts of length 64, so the I print the size of <code>b_input_ids</code>, I get <code>[64,64]</code>.</p>
<p>Any ideas what Pytorch is doing to the batch given it doesn’t have this third dimension like in your example?</p>
Time of DataLoader with O0 and O1 is quite different,

luyang2019: <pre><code class="lang-auto">    # in the train process
    end = time.time()
    for i, data_in in enumerate(train_loader):
        # measure data loading time
        if data_in is None:
            print("empty batch")
            continue
        input, target = data_in
        torch.cuda.synchronize()
        data_time.update(time.time() - end)
        input_image = input.cuda()
        target = target.cuda()
        
</code></pre>
<p>I try the train a reset50 use my own dataset with apex. The time of  O0 and O1 is almost the same. It seems that O1 always apend a lot more time than O0 when coming up with big pictures in dataloader, that will offsets the speed up taken by the fp16 calculation. my data process include random resize, flip, jitter and normalize.</p>

ptrblck: <p>Could you time your data loading as done in the <a href="https://github.com/pytorch/examples/blob/ee964a2eeb41e1712fe719b83645c79bcbd0ba1a/imagenet/main.py#L280" rel="nofollow noopener">ImageNet example</a>?</p>
<p>If your bottleneck is the data loading, you won’t get any significant speedup using AMP.</p>
How does module.children() know how to sort submodules?

Hovnatan_Karapetyan: <p>When I run</p>
<pre><code>class A(nn.Module):                  
    def __init__(self):              
        super().__init__()           
        self.c = nn.Conv2d(55, 50, 3)
        self.b = nn.Conv2d(50, 50, 3)
        self.a = nn.Conv2d(25, 25, 3)

 a = A()
 print(a)
</code></pre>
<p>it outputs submodules in the order of addition in the class A, i.e., <code>c, b, a</code>, instead of alphabetical order <code>a, b, c</code> I would expect. What python technique is used here to achieve this functionality? How does the base class (<code>nn.Module</code>) inspect live the derived classes’ (<code>A</code>) submodules addition? Thanks.</p>

tom: <p>Children are kept in an <code>collections.OderedDict</code>, a Python dictionary to preserve order.<br>
The <code>nn.Module</code> class does some clever magic around <code>__setattr__</code> and friends to keep the submodules (and parameters and buffers) in separate dictionaries.<br>
You can look <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py" rel="nofollow noopener">at the definition</a>, it is clever yet reasonably straightforward.</p>
<p>Best regards</p>
<p>Thomas</p>

Hovnatan_Karapetyan: <p>But how does the base class inspect the order of member modules in the derived class? I updated the question.</p>

alex.veuthey: <p>Well as code is executed in a linear way (downwards…), and <code>self.something = ...</code> calls <code>__setattr__</code>, it is called in the correct order (c, b, a), and the <code>OrderedDict</code> will keep the order by definition. Is that clearer?</p>

Hovnatan_Karapetyan: <p>Ok, thanks! <code>nn.Module</code> basically overrides the <code>__setattr__</code> method to achieve this.</p>
How torch.norm() works? and How it calculates L1 and L2 loss?

muhibkhan: <p>I don’t understand how torch.norm() behave and it calculates the L1 loss and L2 loss? When p=1, it calculates the L1 loss, but on p=2 it fails to calculate the L2 loss…</p>
<p>Can somebody explain it?</p>
<p>a, b = torch.rand((2,2)), torch.rand((2,2))</p>
<p>var1 = torch.sum(torch.abs((a * b)), 1)<br>
print("L1 Distance is : ", var1)</p>
<p>var2 = torch.norm(((a * b)), 1, -1)<br>
print("Torch NORM L1 Distance is : ", var2)</p>
<p>var3 = torch.sum(((a * b)) ** 2, 1)<br>
print("L2 SDistance is : ", var3)</p>
<p>var4 = torch.norm(((a * b)), 2, -1)<br>
print("Torch NORM L2 Distance is : ", var4)</p>
<p>And the computed output is as:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba48be3dd8d6ea928aaa4f177c5a89cc67052c6b.png" alt="aaaa" data-base62-sha1="qzWEYFjdF0LMBSsrno3Qw0KhXOb" width="529" height="109"></p>

JuanFMontesinos: <pre><code class="lang-auto">import torch
a, b = torch.rand((2,2)), torch.rand((2,2))

var1 = torch.sum(torch.abs((a * b)), 1)
print("L1 Distance is : ", var1)

var2 = torch.norm(((a * b)), 1, -1)
print("Torch NORM L1 Distance is : ", var2)

var3 = torch.sum(((a * b)) ** 2, 1).sqrt()
print("L2 SDistance is : ", var3)

var4 = torch.norm(((a * b)), 2, -1)
print("Torch NORM L2 Distance is : ", var4)
</code></pre>
<p>Your code was wrong.</p>
<pre><code class="lang-auto">L1 Distance is :  tensor([0.0924, 0.2528])
Torch NORM L1 Distance is :  tensor([0.0924, 0.2528])
L2 SDistance is :  tensor([0.0893, 0.2327])
Torch NORM L2 Distance is :  tensor([0.0893, 0.2327])
</code></pre>

muhibkhan: <p>THANKS <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>. Its a big help.</p>
GPU hangs when calling model.cuda()

youali: <p>Hi,</p>
<p>I have this strange problem where the GPU hangs the minute <code>model.cuda()</code> is called.</p>
<ul>
<li>GPU: V100 32GB</li>
<li>Nvidia Driver: 418.87.00</li>
<li>CUDA version: 10.1</li>
<li>Pytorch version: tested with 1.3.0 and 1.2.0.</li>
</ul>
<p>it happens within the training script, but even in a python console, where I only create a torchvision model and try to send it to the GPU, and then the node hangs (the resource manager is Slurm if that’s relevant). I note that I don’t have this problem with other GPUs (notably K80).</p>
<p>Thanks.</p>

spanev: <p>Hi <a class="mention" href="/u/youali">@youali</a>,</p>
<p>I would be tempted to thing that it is related to this <a href="https://stackoverflow.com/questions/58344480/pytorch-tocuda-or-cuda-doesnt-work-and-just-get-stuck" rel="nofollow noopener">issue</a>, but you say that it also fails with 1.2.0.</p>
<p>Does it happen only with the torchvision model you are trying to send to the GPU or even with a toy network (let’s say, several Conv2d and BNs)?</p>

alex.veuthey: <p>Apparently, there’s also an issue with 1.2 here: <a href="https://discuss.pytorch.org/t/cuda-problem-i-cant-transfer-objects-to-the-gpu/57988" class="inline-onebox">.cuda() Problem - I can't transfer objects to the GPU</a>.</p>
<p>Could you try reinstalling from conda, as suggested <a href="https://github.com/pytorch/pytorch/issues/27807#issuecomment-541363889" rel="nofollow noopener">here</a>?</p>

Bruno_Oliveira: <blockquote>
<p>Apparently, there’s also an issue with 1.2 here: <a href="https://discuss.pytorch.org/t/cuda-problem-i-cant-transfer-objects-to-the-gpu/57988">.cuda() Problem - I can’t transfer objects to the GPU</a>.</p>
</blockquote>
<p>Indeed, I had this problem while running PyTorch 1.2. It seems it was related to a Conda PyTorch install (version doesn’t seem to matter) and CUDA 10.1. This was fixed and all you need to do is reinstall it from Conda.</p>

youali: <p>Thanks for the answers, I was able to solve it by re-installing pytorch in the same node where the script will be run (with a V100 card). Even if I install it with cuda but not within a node with V100 card, the script hangs (but works fine for other / older cards), but after re-installing, it works fine across the GPUs.</p>
Tensorboard HParams Granularity

stevejpurves: <p>Hi Folks!</p>
<p>I am using the <code>add_hparams</code> function on the integrated Tensorboard SummaryWriter, which is great.<br>
I notice though that in tensorboard that we do not get any control over the hyperparameter values in tensorboard itself. e.g. compare this image</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac34bed7531a8574208e43691624913f2cfc6732.png" alt="image" data-base62-sha1="ozp89LkpviV9qJKAZvipYR4SJJE" width="424" height="268"></p>
<p>to the screenshot in teh tensorboard docs:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.tensorflow.org/tensorboard/hyperparameter_tuning_with_hparams">
  <header class="source">
      <img src="https://www.gstatic.com/devrel-devsite/prod/vb47a36f3a983ed748bf281529457db47955fe57e2b5ea15e7e9641c5e7b5032e/tensorflow/images/favicon.png" class="site-icon" width="32" height="32">

      <a href="https://www.tensorflow.org/tensorboard/hyperparameter_tuning_with_hparams" target="_blank" rel="noopener nofollow ugc">TensorFlow</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/387;"><img src="https://www.tensorflow.org/static/site-assets/images/project-logos/tensorboard-logo-social.png" class="thumbnail" width="690" height="388"></div>

<h3><a href="https://www.tensorflow.org/tensorboard/hyperparameter_tuning_with_hparams" target="_blank" rel="noopener nofollow ugc">Hyperparameter Tuning with the HParams Dashboard &nbsp;|&nbsp; TensorBoard...</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I am assuming that this is due to the simpler implementation here (<code>add_hparams</code> just receives a dict of params when metrics are logged) where there is no prior declaration of the hyperparameter ranges etc that is available when using the hparams and TF direct.</p>
<p>My question is is there any way currently to register those hparams in tensorboard via the pytorch API or otherwise get the same behavior?<br>
or any plans to extend in that direction? ( I see no open issues on github)</p>
<p>cheers!</p>
How to make custom callback (stop training at certain value)

astri: <p>Hello all</p>
<p>I am beginner in deep learning and doing research in keras dan pytorch. I am making custom stop training by watching the loss value and make condition I have succeeded to make a custom stop training in keras and i want make the same thing in pytorch but i am facing problems. First how should i write loss metrics for my stop training function and after I make custom function for callbacks how i called that function.</p>
<p>I am considering using ignite and make a custom but i dont know how to do it</p>
<p>this is my custom stop training in Keras</p>
<pre><code class="lang-auto">class StopatLossValue(Callbacks):
    def on_batch_end(self, batch, logs={}):
        THR = 0.1
        if logs.get('loss') &lt; np.square(THR):
            self.model.stop_training = True

</code></pre>
<p>Below is my pytorch model</p>
<pre><code class="lang-auto"># CREATE MODEL
class MLP(nn.Module):
    def __init__(self):
        super(MLP, self).__init__()
        self.layers = nn.Sequential(
            nn.Linear(784, 100),
           nn.Sigmoid(),
           nn.Linear(100, 10)
        )
        
    def forward(self, x):
        x = x.view(x.size(0), -1)
        x = self.layers(x)
        return x

model = MLP()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
loss_fn = nn.CrossEntropyLoss()

for epoch in range(epochs):
    model.train()
    
    train_losses = []
    valid_losses = []
    for i, (images, labels) in enumerate(train_loader): #Loop for every training ENUMERATE (one epoch)
        
        optimizer.zero_grad()
        
        outputs = model(images) 
        loss = loss_fn(outputs, labels) 
        loss.backward() 
        optimizer.step()
        
        train_losses.append(loss.item())
        
        if (i * 128) % (128 * 100) == 0:
            print(f'{i * 128} / 50000') 

</code></pre>

albanD: <p>Hi,</p>
<p>In your pytorch model, you already have access to both loss at every iteration and train_losses at the end of the epoch.<br>
So you don’t need a callback actually in pytorch. You can write a function with whatever you need directly.</p>

astri: <p>thank you for your reply, i tried write and there is no error , but i am not sure whether is right or not.</p>
<pre><code class="lang-auto">class StopatLossValue:
    def on_epoch_end(self, train_losses, **kwargs):
        # if the monitored metrics got worst set a flag to stop training
         eo = 0.1
         if train_losses &lt; eo.pow(2):
             return{'stop_training': True}
</code></pre>
<p>and called it in my model</p>
<pre><code class="lang-auto">  StopatLossValue()
</code></pre>
Average pool an image without decreasing the size?

barry_t: <p>For a given n x n tensor and window size k, is there a quick way to set every k x k window to the average value of the window in the original tensor? This would basically be average pooling, but without resizing, this would effectively just blur the image.</p>
<p>Thanks</p>

albanD: <p>Hi,</p>
<p>If you use <a href="https://pytorch.org/docs/stable/nn.functional.html#avg-pool2d" rel="nofollow noopener">average pooling</a> with a kernel of k (must be odd) and a padding of floor(k/2). Then you will maintain the image size.</p>

barry_t: <p>I wasn’t clear enough in my first post. What I mean is the following:</p>
<p>Given a 4 x 4 image [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15]] and a window size of 2, what I’d like to get is the following:</p>
<p>[[2.5, 2.5, 4.5, 4.5], [2.5, 2.5, 4.5, 4.5], [10.5, 10.5, 12.5, 12.5], [10.5, 10.5, 12.5, 12.5]]</p>
<p>Thanks</p>

albanD: <p>Ho,</p>
<p>I’m not sure there is a very clean way to do this but the following will work:</p>
<pre><code class="lang-python">import torch

kern_size = 2

# Add 0th dimension because avg_pool expect batch dimension
inp = torch.arange(16).float().view(1, 4, 4)
pooled = torch.nn.functional.avg_pool2d(inp, kern_size)
# remove 0th dimension
pooled.squeeze_(0)
rep_col = []
for c in range(pooled.size(1)):
  col = pooled.select(1, c)
  rep_col += [col] * kern_size
repeated = torch.stack(rep_col, 1)
rep_row = []
for r in range(repeated.size(0)):
  row = repeated.select(0, r)
  rep_row += [row] * kern_size
output = torch.stack(rep_row, 0)

print(output)
</code></pre>

barry_t: <p>Thanks a lot for this</p>

barry_t: <p>Another solution would be to average pool, and then use nn.UpsamplingNearest2d()</p>

pmeier: <p>This upsampling is a <a href="https://en.wikipedia.org/wiki/Kronecker_product" rel="nofollow noopener">Kronecker product</a> with a matrix of ones. Unfortunately, <code>torch</code> does not have an implementation of it. If you are not dependent on the GPU it might be faster to do it with <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.kron.html" rel="nofollow noopener"><code>numpy</code></a>:</p>
<pre><code class="lang-python">import numpy as np
import torch
from torch.nn.functional import avg_pool2d

image = torch.arange(16, dtype=torch.float).view(4, 4)

kernel_size = 2
stride = 2

image_pooled = image.view(1, 1, *image.size())
image_pooled = avg_pool2d(image_pooled, kernel_size=kernel_size, stride=stride)
image_pooled = image_pooled.squeeze()

image_blured = image_pooled.numpy()
image_blured = np.kron(image_blured, np.ones(image_blured.shape))
image_blured = torch.from_numpy(image_blured)

print(image_blured)
</code></pre>

albanD: <p>Also you cannot use this if you need gradients !</p>

pmeier: <p>You are right, I totally forgot about that.</p>