akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
GPU utilization is low while using CUDA extension on multi-gpu

username: jhd
cooked: <p>Hi, I built my CUDA extension module following <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">this</a> link. Everything works well when I only use 1 GPU. And its utilization is high (&gt;90%). However, when I integrated it into my neural work and trained with 2 GPU, the utilization of each gpu is pretty low (≈50%).<br>
Any ideas? thanks!<br>
My environments:<br>
Pytorch:1.1<br>
CUDA:10.1<br>
OS:Ubuntu 18<br>
GPU: RTX2080ti</p>

username: Mendel123
cooked: <p>Maybe you can analysis the running time of each part, such as data loading, model forwarding, output processing. This may not be caused by forwarding.</p>

username: jhd
cooked: <p>Hi, <a class="mention" href="/u/mendel123">@Mendel123</a>  thanks for your reply.<br>
After analysis the running time of  each part, Forward has taken the largest part.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th style="text-align:left"></th>
<th style="text-align:left">Forward</th>
<th style="text-align:left">backward</th>
<th style="text-align:center">Data</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left">2 GPU(32 batchsize)</td>
<td style="text-align:left">1.1 sec</td>
<td style="text-align:left">0.27 sec</td>
<td style="text-align:center">0.001</td>
</tr>
<tr>
<td style="text-align:left">1 GPU(16 batchsize)</td>
<td style="text-align:left">0.4 sec</td>
<td style="text-align:left">0.27 sec</td>
<td style="text-align:center">0.001</td>
</tr>
</tbody>
</table>
</div>

username: jhd
cooked: <p>I used APEX mix-precision training, will it be related to apex? Besides, I didn’t implement <code>half()</code> operation in my module , so I just convert inputs to<code> float()</code> before my invoking and convert it to <code>half()</code> after my invoking.</p>

username: jhd
cooked: <p>Checking line by line, I finally found that declaring variables and allocating CUDA memory in CUDA extension of Pyotrch will greatly reduce GPU efficiency. By removing the following statements,</p>
<pre><code class="lang-auto">float *dist_Dev; 
gpuErrchk(cudaMalloc((void**)&amp;dist_Dev, myParameter.obj_num * myParameter.cluster_num * sizeof(float)));
int *obj_num_Dev; 
gpuErrchk(cudaMalloc((void**)&amp;obj_um_Dev, myParameter.cluster_num * sizeof(int)));
int *num_per_classt;
gpuErrchk(cudaMalloc((void**)&amp;num_per_classt, myParameter.t * myParameter.cluster_num * sizeof(int)));
</code></pre>
<p>, the utilization of each gpu is high. And each variable will be created in pytorch and then passed to CUDA module.<br>
However, it still can’t expalin why single GPU works. I also compiled C++/CUDA into a <code>.so</code> file and invoked by <code>ctypes</code>.The result proves that declearing variables like the method above is fine.<br>
So I guess there might be a problem with the way CUDA extension works.</p>
Fix momentum bug in CyclicLR

username: Mughees
cooked: <p>Hey, An issue was resolved in <a href="https://github.com/pytorch/pytorch/pull/20401" rel="nofollow noopener">20401</a> but I am still facing that. Any recommendations on how can I implement that with my network. Thanks.</p>
<p>optimizer = optim.Adam(net.parameters(), lr=0.01, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)<br>
scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.001, max_lr=0.1,cycle_momentum=True)</p>
Is there any difference between different kinds of <grad_fn>?

username: yukino
cooked: <p>I would like a function such as <code>1 - x</code> with autograd, but I come up with some questions about the difference between the following code.</p>
<pre><code class="lang-auto">x = 1.0 - nn.Parameter(torch.zeros(1, 3, 1, 1))
print(x)
y = torch.ones(1, 3, 1, 1) - nn.Parameter(torch.zeros(1, 3, 1, 1))
print(y)
z = - nn.Parameter(torch.zeros(1, 3, 1, 1)) + 1.0
print(z)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor([[[[1.]],

         [[1.]],

         [[1.]]]], grad_fn=&lt;RsubBackward1&gt;)
tensor([[[[1.]],

         [[1.]],

         [[1.]]]], grad_fn=&lt;SubBackward0&gt;)
tensor([[[[1.]],

         [[1.]],

         [[1.]]]], grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>the &lt;grad_fn&gt; between them <strong>look</strong> different, but I dont konw how to test them to know the <strong>difference</strong> between them.</p>
<p>Someone have any idea ? Thanks a lot.</p>

username: ptrblck
cooked: <p>Since different operations are used in the forward calculations, Autograd will create the corresponding <code>grad_fn</code>s. How would you like to test them?<br>
If you would like to see the gradients, you could access it via the <code>.grad</code> attribute.</p>

username: yukino
cooked: <p>whatever I choose, the output will be the same?<br>
some people use <code>torch.ones_like(x)</code> and some people use <code>1.0</code>, I want to know whether the choice matter the output and <strong>Which</strong> is the preferable one to use in practice.</p>

username: yukino
cooked: <p>If they are the same maybe we can use <code>1.0 - x</code> in models, and at the end wrtie <code>models.to(device)</code>, but not <code>torch.ones_like()</code> every where.</p>

username: Mendel123
cooked: <p>I think there is no difference in the gradient aspect.</p>

username: ptrblck
cooked: <p>Sometimes you need to explicitly create a tensor with certain sizes, so there is nothing wrong in using <code>torch-ones_like</code>.<br>
Do you have some issues with this method?</p>
LSTM for high dimentional sequence prediction

username: shp
cooked: <p>I am new in coding pyton/pytorch… and I have a hard time in using pytorch lstm layers for predicting a high dimensional sequence. Let’s say I have a matrix of size 5* 10 of integer numbers as my data. This means sequence length is 10 and each point of my sequence has 5 features.</p>
<p>My goal is to build a network to predict this sequence. Let say I am going to feed two pints of this sequence (2<em>5) at each time to my network and network return me same size (2</em>5) just one step ahead in the sequence. Therefore data is something like</p>
<pre><code class="lang-auto">X=torch.LongTensor(5,10).random_(0, 10)) # 5 number of features and 10 size of sequence
</code></pre>
<p>My main problem is how to feed this data to my model? I find following code somewhere and change it a little bit. I read many posts about this issue but still confuse how to feed my data to this model</p>
<pre><code class="lang-auto">class LSTM(nn.Module):

    def __init__(self, input_dim, hidden_dim, batch_size, output_dim=1,num_layers=2):
        super(LSTM, self).__init__()
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.batch_size = batch_size
        self.num_layers = num_layers

        # LSTM layer
        self.lstm = nn.LSTM(self.input_dim, self.hidden_dim, self.num_layers)

        # output layer
        self.linear = nn.Linear(self.hidden_dim, output_dim)

    def init_hidden(self):
        return (torch.zeros(self.num_layers, self.batch_size, self.hidden_dim),
                torch.zeros(self.num_layers, self.batch_size, self.hidden_dim))

    def forward(self, input):
        lstm_out, self.hidden = self.lstm(input.view(len(input), self.batch_size, -1))
        
        y_pred = self.linear(lstm_out[-1].view(self.batch_size, -1))
        return y_pred.view(-1)

model = LSTM(lstm_input_size, h1, batch_size=num_train, output_dim=output_dim, num_layers=num_layers)

loss_fn = torch.nn.MSELoss(size_average=False)
optimiser = torch.optim.Adam(model.parameters(), lr=learning_rate)


# Training 
for t in range(num_epochs):
    model.zero_grad()
    model.hidden = model.init_hidden()
    
    # Forward pass
    y_pred = model(x_train)
    loss = loss_fn(x_train, x_train)

    optimiser.zero_grad()
    loss.backward()
    optimiser.step()

</code></pre>
<p>Any help appreciated. Also if have any other suggestion to predict such a sequence please let me know. thank you</p>
Resnet3D for regression

username: banikr
cooked: <p>Hello,<br>
I am doing a regression project where I have to predict 512 values of a vector from 3D images. The image dimension is 86,110,78.<br>
Now I am confused about using the Maxpool3D layer or not. If I use maxpool then I have to pad the images to shape 88,112,80… since there are 3 pooling layers the final shape of the featured image will be 11,14,10.<br>
And then using a fully connected layer(with tanh activation) I will shrink the output to 512.<br>
are there better ways to do this without pooling, I know about strides…I am not familiar with how to do that.<br>
Keeping in mind the residual connection/identity mapping.</p>
<p>Even if I zero-pad the image it will change the zero-mean unit variance normalization of the spatial values.</p>

username: ptrblck
cooked: <p>The padding approach seems to be fine.<br>
However, if you want to reduce the volumetric dimensions using strided convolutions, have a look <a href="http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html" rel="nofollow noopener">here</a> to get some information how strides work in conv layers.</p>

username: banikr
cooked: <p>Hi,<br>
Also, <a href="https://arxiv.org/pdf/1412.6806.pdf" rel="nofollow noopener">this paper</a> says <code>Maxpool</code> layers can be replaced by strided convolutional layers with the same kernel size.</p>
<p>I did that but didn’t see any change in my result though.</p>
Whats the difference between Optimizer.zero_grad() vs nn.Module.zero_grad()

username: banikr
cooked: <p>which one is used when?</p>
<p><a href="https://pytorch.org/docs/stable/optim.html" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html</a>  here it uses Optimizer.zero_grad()</p>
<p><a href="https://github.com/pytorch/examples/blob/master/dcgan/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/dcgan/main.py</a> codes here use NetG.zero_grad().</p>
<p>I know that optimizer.zero_grad() makes the existing gradients zero before backpropagating the loss. Then update network parameters. What is nn.Module.zero_grad() used for?</p>

username: albanD
cooked: <p>The nn.Module.zero_grad() also sets the gradients to 0 for all parameters.</p>
<p>If you ceated your optimizer like <code>opt = optim.SGD(model.paremeters(), xxx)</code>, then <code>opt.zero_grad()</code> and <code>model.zero_grad()</code> will have the same effect.<br>
The distinction is useful for people that have multiple models in the same optimizer, or multiple optimizer for different part of their model.</p>
Variable Input Size

username: tsut
cooked: <p>I have an input that its size is varied e.g. (102 x 128) or (102 x 1100).</p>
<p>I create a network that is theoretically valid on this different size of input as follows<br>
(In my model class, I define forward method as follows)</p>
<pre><code class="lang-auto">def forward(self, inputs):
        x = self.Batchnorm1(inputs)
        x, _ = self.LSTM(x)
        x, _  = torch.max(x, 1)
        x = self.linear(x)
        x = torch.sigmoid(x)
</code></pre>
<p>That is by using global max pooling, the dimension will match.</p>
<p>Besides, I also make some customization on the dataloader for training different size of inputs in the batch according to <a href="https://jdhao.github.io/2017/10/23/pytorch-load-data-and-make-batch/" rel="nofollow noopener">https://jdhao.github.io/2017/10/23/pytorch-load-data-and-make-batch/</a></p>
<p>However, the  problem still persists. That is the output from new customized collate_fn which is as fllows</p>
<pre><code class="lang-auto">def my_collate(batch):
    data = [item[0] for item in batch]  # just form a list of tensor

    target = [item[1] for item in batch]
    target = torch.LongTensor(target)
    return [data, target]
</code></pre>
<p>will still output the type of input data that I cannot feed into batch normalization (or other types of layers) in my forward function since it expects 3-d Tensor (Batch Size x dim1x dim2)<br>
How should I make any changes to my model to be able to receive the input.</p>
<p>ps. padding is not an option here</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/tsut">@tsut</a>,</p>
<p>I think that if you cannot pad your data, you won’t have other solutions that doing:</p>
<pre><code class="lang-auto">outputs = []
for datum in data:
    outputs.append(model(datum)) # you might need to add `.unsqueeze(0)` to datum
outputs = torch.stack(outputs)
</code></pre>

username: tsut
cooked: <p>Thanks for you reply.</p>
<p>By doing like that (feeding in 1-1 vs feeding entire batch at once), will I still have the same effect of batch normalization layer ?</p>

username: spanev
cooked: <p>Oh… no it’s not. I’m sorry, I missed your point about Batchnorm here.<br>
I don’t thinks it is possible to use the default batchnorm here. You would need to have your input data projected into same size tensors (some interpolation method?).</p>
Cuda Check - How to check if the device is available before assigning task

username: Anandh_Perumal_Konar
cooked: <p>model.to(device) or nn.DataParallel(model) moves the model to the GPU but is there a way to check if the particular device is already in use by any other process.<br>
The actual problem is when some else is training something on the server I don’t want my program to disturb their process and my program to fail so that I can allocate some other resource.</p>

username: albanD
cooked: <p>Unfortunately it is very hard to do a general check like this as it will depend a lot on how your manage your cluster with coworkers.<br>
You can use <code>nvidia-smi</code> to check memory usage and/or process running on the different GPUs potentially.<br>
You can check <a href="https://github.com/lberrada/waitGPU" rel="nofollow noopener">this</a> small tool for example on how to do such thing and adapt it to your needs.</p>

username: Anandh_Perumal_Konar
cooked: <p>nvidia-smi is more like a manual job.<br>
waitGPU is very helpful will little changes it should solve my problem.<br>
Thanks a lot.</p>
Torch 1.3.0.post2 published for MacOS but not other platforms in Pypi

username: mandubian
cooked: <p>Hi,</p>
<p>It looks like WHL files for other platforms than MacOS haven’t ben published to <a href="https://pypi.org/project/torch/#files" rel="nofollow noopener">https://pypi.org/project/torch/#files</a> and this creates issue for pip.</p>
<p>Thanks<br>
P.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can find the wheel for the other platforms at the 1.3.0 tag <a href="https://pypi.org/project/torch/1.3.0/#files" rel="nofollow noopener">https://pypi.org/project/torch/1.3.0/#files</a></p>
<p>We had some issues with the macos build and had to release a new set of binaries for it.</p>

username: mandubian
cooked: <p>Thanks.<br>
I know that because I’ve used 1.3.0 instead.<br>
But if you use a package manager such as pip with a version indicator &gt;=1.3, it considers 1.3.0.post2 as latest version and fails to get it so it fails to build.<br>
Maybe it’d also better to release builds for all other platforms too in 1.3.0.pos2 tag to avoid it.</p>

username: albanD
cooked: <p>Good to know,</p>
<p><a class="mention" href="/u/smth">@smth</a> do we want to release post2 versions of the other binaries?</p>

username: mandubian
cooked: <p>TBH I think you have to deploy it in any case because here is what you get under Linux for now:</p>
<blockquote>
<p>pip search torch<br>
torch (1.3.0.post2)                   - Tensors and Dynamic neural networks in Python with strong GPU acceleration<br>
INSTALLED: 1.3.0<br>
LATEST:    1.3.0.post2</p>
</blockquote>
<p>I’ve lost some time this morning because I thought I had made an error somewhere till I discovered the missing files on PyPi repo.<br>
Thks again</p>
Why do we need tensor.data

username: agadetsky
cooked: <p>Should <code>.data</code> attribute be deprecated in the future? Since we have <code>torch.no_grad</code> and other things that allow us to control availability of gradients we don’t need this attribute? <a class="mention" href="/u/apaszke">@apaszke</a>, <a class="mention" href="/u/smth">@smth</a></p>

username: albanD
cooked: <p>Hi,</p>
<p>It is already deprecated.<br>
It does not appear in the doc anywhere anymore (if we missed some spots, please let me know).</p>
<p>We can’t really remove the attribute itself for BC reasons unfortunately.</p>

username: agadetsky
cooked: <p>Thank you for your answer</p>
UNet different image size for the input

username: blackbirdbarber
cooked: <p>This is rather a theoretical question, but I like to know if one can create UNets for different input image sizes.</p>
<p>I already think I know the UNet input size may not match the output size.</p>

username: ptrblck
cooked: <p>That should be possible. If you stick to certain spatial sizes, e.g. powers of two, I would assume that most UNet implementations can handle the input.<br>
However, if you would like to use arbitrary input shapes, most likely you would have to add some checks and pad/crop manually for odd sizes etc.</p>
<p>The input shape and output shape in the original UNet paper are not matching, if I recall it correctly, but you could implement it in a matching way.</p>
<p>I’ve created a simple UNet style architecture <a href="https://github.com/ptrblck/pytorch_misc/blob/master/unet_demo.py" rel="nofollow noopener">here</a>, which could be useful as a code base to implement your model. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: blackbirdbarber
cooked: <p>I tried your <a href="https://github.com/ptrblck/pytorch_misc/blob/master/unet_demo.py" rel="noopener nofollow ugc">UNet demo</a>, with other input size like this:</p>
<pre><code class="lang-auto">x = torch.randn(1, 3, 240, 180)
y = torch.randint(0, nb_classes, (1, 240, 180))
</code></pre>
<p>and got</p>
<blockquote>
<p>RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 45 and 44 in dimension 3</p>
</blockquote>
<p>And I found <a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/295#issuecomment-402541300" rel="noopener nofollow ugc">online</a> tips you can resize or tweak UNet.</p>
<p>Looks like no way around.</p>
<p>That way it has lot of sense to create UNets that examine the input and output size and create the architecture dynamically based on that.</p>

username: blackbirdbarber
cooked: <p>Looks like UNets are really hard to work with if you have images of different size than the original for which it was created.</p>
<p>It has sense to create UNets that can adopt to any size of the input and output, still I haven’t saw this kind of dynamic architecture.</p>
<p>Is there any UNet like this, that I don’t need to modify the structure for specific input and output size I have?</p>

username: ptrblck
cooked: <p>Not sure, if there are any public implementations, but you could most likely add cropping or padding with some if/else conditions to use variable input shapes.</p>
How does CE loss get computed concretely on multi-class segmentation?

username: AlexLuya
cooked: <p>Hello,I found someone use it like this:<br>
<strong>celoss(logits,mask)</strong><br>
here<br>
logits has shape <strong>[2, 22, 256, 256]</strong><br>
mask has shape <strong>[2, 256, 256]</strong></p>
<p>and category type is encoded into mask,for example,if a pixel is belongs to a people,it will has value==17(human’s code in VOC dataset).</p>
<p>Will CELoss one-hot encode mask to generate 22 channels firstly,then pairing generated mask channels with logit channels one by one to compute cross entropy?</p>

username: albanD
cooked: <p>Hi,</p>
<p>CELoss can be implemented without having to create the one hot encoding. So to reduce memory needs, our CE function takes the label as a LongTensor that contains the index to the correct class.</p>
Failure to pass gradient check but the operation is reportedly correct

username: Guanqun_Yang
cooked: <h2>Problem</h2>
<p>I am trying to understand how the gradient reversal layer works.</p>
<p>I found a successful implementation as reported <a href="https://discuss.pytorch.org/t/solved-reverse-gradients-in-backward-pass/3589/6">here</a>. He claimed that by using the gradient reversal operation, he could reproduce results in the research papers. The operation is essentially the following (minor differences may arise from API changes)</p>
<pre><code class="lang-python">class GradientReverse(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        return x.view_as(x)

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output.neg()
</code></pre>
<p>However, when I tried to make sure this operation is correct</p>
<pre><code class="lang-python">test_input = autograd.Variable(torch.randn((3, 4), dtype=torch.float64), requires_grad=True)
flag = torch.autograd.gradcheck(GradientReverse.apply, test_input, eps=1e-3)
</code></pre>
<p>an error occurred</p>
<pre><code class="lang-auto">RuntimeError: Jacobian mismatch for output 0 with respect to input 0,
</code></pre>
<p>I am not sure what mistakes did I make when doing the gradient check.</p>
<p>Could someone help me? Thank you in advance!</p>

username: albanD
cooked: <p>Hi,</p>
<p>Well gradient check checks whether the gradients computed in the backward match the derivative of your forward function.<br>
In your case, your backward definitely don’t compute the gradients of the function that corresponds to the forward. So I would expect the gradcheck to fail.</p>
<p>PS: To avoid weird autograd behavior, change your forward() method to <code>return x.clone()</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Guanqun_Yang
cooked: <p>If I understand the role of <code>autograd.gradcheck()</code> properly, it only performs backpropagation in <strong>traditional sense</strong>.</p>
<p>This means there is <strong>no way</strong> for my function to pass the gradient check since adding minus sign to gradient is unconventional.</p>

username: albanD
cooked: <p><code>gradcheck</code> checks for “true gradients”.<br>
For your function, the “true gradient” would be <code>1</code>. But you deliberately set it to <code>-1</code>. So there is no way indeed it can pass the gradcheck.</p>
Slicing tensor with pin_memory=True

username: SPark9625
cooked: <p>Usually <code>pin_memory</code> is used when creating dataloaders, to allow CUDA to use DMA (direct memory access), but I noticed that it’s also possible to create tensors with <code>pin_memory=True</code>.</p>
<p>I’m curious what happens if I slice a tensor in pinned memory. My guess is that CPU would have to first copy the sliced tensor to pinned memory and then let CUDA copy from it. Am I right?</p>

username: albanD
cooked: <p>I assume that by slice you mean subset of a tensor right? like <code>t.select(0, 1)</code>?<br>
If so, the new tensor is actually looking at the same memory as the original one. So it is still pinned memory and no need to copy it.</p>

username: SPark9625
cooked: <p>Is that still the case of advanced indexing? For example, is the output tensor below also in pinned memory? I assumed that since now the subset of the tensor has non-contiguous memory, CUDA would have difficulties copying through DMA</p>
<pre><code class="lang-python">&gt;&gt;&gt; x = torch.randn(3,4)
&gt;&gt;&gt; x[range(3), [0,1,2]]
tensor([ 0.0127, -0.2941, -2.1625])
</code></pre>
<p>— EDIT—</p>
<p>I just found out that there’s <code>.is_pinned()</code> method I can use to check if the tensor is in pinned memory, and it seems like the above is not pinned.</p>
<pre><code class="lang-python">&gt;&gt;&gt; x = torch.randn(3, 4, pin_memory=True)
&gt;&gt;&gt; x.select(0,1).is_pinned()
True
&gt;&gt;&gt; x[range(3), range(3)].is_pinned()
False
</code></pre>

username: albanD
cooked: <p>You can indeed check with <code>.is_pinned()</code>.<br>
Advanced indexing does not make any guarantee about copying or not unfortunately when you get the new Tensor. <code>y = x[0]</code> will most likely share memory.<br>
Note though that <code>x[bar] = foo</code> will write into x even though <code>y = x[bar]; y = foo</code> might or might not <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
The trend of Loss changes strange when training

username: chen
cooked: <p>I am running a project about visual speech recognition task, and I found a strange phenomenon.<br>
In the first sevaral epochs, the loss can decrease at a normal speed. I mean, in a epoch, the loss would decrease with the increase of number of iterations. While in the latter epochs, the loss almost unchanged in the whole epoch, but would decrease at the start of next epoch.<br>
Sometimes I interruptted the running code after a epoch finishing and restart from the trained weights. The loss would decrease, too.<br>
I get very confused about this phenomenon. Can someone help me solve this problem. Thanks!</p>
assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors)

username: Safak_Altinok
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6c0cb4f08d6a7b093d362333dc8f627c2b4df61.png" data-download-href="https://discuss.pytorch.org/uploads/default/d6c0cb4f08d6a7b093d362333dc8f627c2b4df61" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6c0cb4f08d6a7b093d362333dc8f627c2b4df61.png" alt="Capture" data-base62-sha1="uDNeLZhpCibeNpsvXhgNbyyORRD" width="690" height="224" data-dominant-color="F3F1D7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1044×340 9.71 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <img src="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51590087741cb0387c80c4bfa8973696d21c9bc0.png" alt="Capture1" data-base62-sha1="bBDmohoijL0esEQLhDQ5afEjZ3a" width="534" height="368"></p>

username: ptrblck
cooked: <p>Your issue is similar to <a href="https://discuss.pytorch.org/t/how-can-i-create-a-single-dataloader-for-my-two-csv-files-i-have-tried-this/59099/8">this one</a>.</p>
<p>Basically, the passed tensors have a different length, which raises this error.</p>
<p>Let me know, if you need more information.</p>
Multilabel training for finetuning

username: Ruben_Maestro
cooked: <p>Hello guys! Is there any example about training a dataset for finetuning segmentation like the pedestrians one with more than 2 classes or labels?</p>
<p>The example about which im talking is the next one:</p>
<p><a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>
<p>Thank you.</p>
How to change the class name when loading a pre-trained model?

username: Lewis
cooked: <p>Hi, Ive trained two different models seperately in a sperate file, which both include the same class name but different architecture for that class. For example:</p>
<pre><code class="lang-auto">class CNN(nn.Module):
     some stuff

class CNN(nn.Module):
     some other stuff (other than the first)
</code></pre>
<p>I need to load and run the 2 models together. How can I do this? Which class will be assigned to which and how can i change the class name in that case?</p>

username: alex.veuthey
cooked: <p>This is related to Python itself, and not PyTorch, but have a look <a href="https://stackoverflow.com/questions/5352781/how-to-set-class-names-dynamically/54284495" rel="nofollow noopener">here</a>!</p>
How to verify the correctness of network implementation?

username: cckao
cooked: <p>How can I verify that the codes I wrote exactly implement the network architecture in my mind?</p>
<p>My idea is re-implementing all the tensor operations in NumPy and feeding in random inputs to see if the results are the same. But this is so dumb!</p>

username: ebarsoum
cooked: <p>And how you verify that your numpy implementation is correct.</p>
<p>If the network is totally novel, you can verify the expected shape after each node. Then, feed the network a constant, and check the output after each node. You can also fix the network parameter, start with a small network, check each section of the network on its own…etc.</p>

username: cckao
cooked: <p>Sounds reasonable. However, how do you know the number is correct when you check the output after each node?</p>
<p>Another way I can think of is to visualize the network and manually inspect if all the nodes and edges are connected correctly. But again, this is not clever. The verification cannot be wrapped into an unit test case and humans make mistakes.</p>
Training on distributed GPU system is out of sync

username: Cooper_Lee
cooked: <p>We use a distributed GPU system with four v100 GPUs for each node to train our model through pytorch.  We follow the demo of pytorch used on distributed GPU system. As we all know, each node has a copy of model. However, there is a great time interval for copy on different node. I wonder wether anyone else has this problem. Please give me some advice for this issue.</p>
Model parallel issue that disappears with CUDA_LAUNCH_BLOCKING=1

username: mbahri
cooked: <p>Hi</p>
<p>I’m trying to parallelize a somewhat large encoder-decoder model. I have the input data on GPU 0 fed into the encoder, then I transfer the latent code to GPU 1 and feed it to the decoder, then compute the losses on GPU 1.</p>
<p>One particular loss is implemented as a <code>torch.autograd.Function</code> and something triggers a device-side assert with out of bound indices in that snippet:</p>
<pre><code class="lang-auto">batchV = V.view((-1, 3))

# Compute half cotangents and double the triangle areas
C, TwoA = half_cotangent(V, faces)

batchC = C.view((-1, 3))

# Adjust face indices to stack:
offset = torch.arange(V.shape[0], device=V.device).view((-1, 1, 1)) * V.shape[1]

# Add the offset to the faces passed as parameters and save in a different tensor
F = faces + offset
batchF = F.view((-1, 3))

# import ipdb; ipdb.set_trace()
# Fails here if not run with CUDA_LAUNCH_BLOCKING=1
rows = batchF[:, [1, 2, 0]].view(
    1, -1
)  # 1,2,0 i.e to vertex 2-3 associate cot(23)
cols = batchF[:, [2, 0, 1]].view(
    1, -1
)
</code></pre>
<p>The code runs fine for 2 samples, then on the third I get the device-side assert. The dataloader is shuffling the samples, yet it consistently fails on the third sample.</p>
<p>Debugging in pdb gives me the following traceback:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 593, in &lt;module&gt;
    exp_flag,
  File "train.py", line 524, in compute_losses_real
    exp_real_and,
  File "&lt;thefile&gt;.py", line 23, in __call__
    Lx = self.laplacian.apply(V, self.F[mask])
  File "&lt;thefile&gt;.py", line 64, in forward
    rows = batchF[:, [1, 2, 0]].view(
RuntimeError: size is inconsistent with indices: for dim 0, size is 7380 but found index 4684893058448109737
</code></pre>
<p>I then ran the same exact code with CUDA_LAUNCH_BLOCKING=1 and the code doesn’t crash, and the loss decreases.</p>
<p>What I already tried:</p>
<ul>
<li>
<p>In case this might be related, I disabled pinned memory and non-blocking data transfers from host to GPU, but the problem persists.</p>
</li>
<li>
<p>I added <code>torch.cuda.synchronize()</code> right above <code>rows = batchF[:, [1, 2, 0]].view(</code> with no success.</p>
</li>
<li>
<p>This code works fine when the model is on a single GPU.</p>
</li>
</ul>
<p>Any help would be much appreciated!</p>

username: albanD
cooked: <p>Hi,</p>
<p>It does look like we’re missing a synchronization point…<br>
Could you provide a small code sample that triggers this issue so that we can reproduce locally please? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Yanli_Zhao
cooked: <p>Hi, would you mind providing your complete code? from the snippet you provided, it is hard to say the root cause.</p>
Embarassingly simple question about autograd

username: mechatron
cooked: <p>Hi,</p>
<p>I just wrote the following code and I am not getting the answer that I expected.</p>
<pre><code class="lang-auto">y = torch.tensor([10], dtype=torch.float32, requires_grad=True)
w = (np.pi*y)/180
f = torch.sin(w)
f.backward()
print(w.grad)
</code></pre>
<p>It prints ¸<code>None</code>. Shouldn’t the answer be 0.984807753012208 ? Here is my understanding. we have $f(w) = sin(w)$ where $w=\frac{\pi*y}{180}$. <code>w.grad</code> should give us $\frac{\partial f}{w}\big|_{w=0.1745329}$. However, I am not getting that.</p>
<p>Could someone please help me understand this ?</p>
<p>Thank you!</p>

username: albanD
cooked: <p>Hi,</p>
<p>When you call <code>.backward()</code>, then <code>.grad</code> field of only the leaf variables is populated (to reduce memory usage).<br>
A leaf variable is one that the user created with requires_grad=True. In your case, <code>y.is_leaf == True</code> but <code>w.is_leaf == False</code>.</p>
<p>If you want the <code>.grad</code> field to be populated for a Tensor that is not a leaf, you need to call <code>w.retain_grad()</code> on it before the backward pass.</p>

username: mechatron
cooked: <p><a class="mention" href="/u/alband">@albanD</a> Ah thank you! I understand your comment much better now after reading this <a href="https://github.com/pytorch/pytorch/issues/3625" rel="nofollow noopener">github issue</a></p>
Does torch.nn.EmbeddingBag() use multi-threads (or multi-cores)?

username: Taehun_Kim
cooked: <p>Hi,</p>
<p>When I run an application which contains the function torch.nn.EmbeddingBag(), it seems the utilization of one particular core is high(above 90%) but the utilization of the others is not that high.</p>
<p>I wonder if or not EmbeddingBag() uses multi-threads (or multi-cores), when it does embedding look up.</p>
<p>Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Pytorch multi-thread usage is function dependent and workload dependent.<br>
You would need to check the code for EmbeddingBag to check wether it uses multi-threading or not and if  so, what is the threshold that is used to start using multiple threads.</p>
Breaking up densenet121 results in a size mismatch?

username: ari
cooked: <p>Hi,</p>
<p>I am trying to create a custom nn module that wraps a pre-trained densenet121, so that I can access activations at the end of the convolutional part of the network.</p>
<p>As a starting point, I have just done the following:</p>
<pre><code class="lang-auto">import torch
from torchvision import models
import torch.nn as nn

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()

        self.model = models.densenet121(pretrained=True)

        self.features = self.model.features
        self.classifier = self.model.classifier

    def forward(self, x):
        x = self.features(x)
        x = self.classifier(x)
        return x

example = torch.ones((1,3,224,224))

model = MyModel()
output = model.forward(example)
</code></pre>
<p>However,  I get a mismatch error for line <code>x = self.classifier(x)</code> of the forward method:</p>
<pre><code class="lang-auto">RuntimeError: size mismatch, m1: [7168 x 7], m2: [1024 x 1000] at ../aten/src/TH/generic/THTensorMath.cpp:961
</code></pre>
<p>Also, as far as I can tell, densenet121 expects (batch_size, 1024) inputs to its classifier part, however when I print the size of x after <code>x = self.features(x)</code> in the forward method, it is (1, 1024, 7, 7)?</p>
<p>Thank you for the help, I am still very new to pytorch.</p>

username: albanD
cooked: <p>Hi,</p>
<p>From the <a href="https://github.com/pytorch/vision/blob/5eee0117f8829c7c13af23dbadd5d14afc01a5d9/torchvision/models/densenet.py#L214-L220" rel="nofollow noopener">original densenet code</a>, It looks like you’re missing an average pooling layer between the features and classifier</p>
Regarding F.conv2d implementation

username: vijaytida
cooked: <p>Hi,<br>
I need the underlying implementation code of the convolution operation. Can anyone help me where I can find this?<br>
Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>This question has already been answered in the forum, here: <a href="https://discuss.pytorch.org/t/source-code-for-conv2d/58398/2" class="inline-onebox">Source code for conv2d</a> for example.</p>
Seed only one GPU?

username: Evan_Weissburg
cooked: <p>I have 8 tasks running on 8 GPUs, potentially concurrently through a Jupyter notebook. I would like each task to be deterministic, so I call torch.cuda.manual_seed_all(seed) in the beginning of each task. However, the tasks are not deterministic, and I believe this is because GPUs are potentially being re-seeded during runtime by another task being launched. Am I correct in this assumption? If so, how do I fix this behavior? If not, what’s the issue? Thanks so much!</p>

username: ptrblck
cooked: <p>Based on the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a>, you would have to diable <code>cudnn.benchmark</code> and set <code>cudnn.deterministic=True</code> besides seeding.</p>
<p>Also, note that some CUDA operations are non-deterministic:</p>
<blockquote>
<p>There are some PyTorch functions that use CUDA functions that can be a source of non-determinism. One class of such CUDA functions are atomic operations, in particular  <code>atomicAdd</code> , where the order of parallel additions to the same value is undetermined and, for floating-point variables, a source of variance in the result. PyTorch functions that use  <code>atomicAdd</code>  in the forward include <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.index_add_"> <code>torch.Tensor.index_add_()</code> </a>, <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.scatter_add_"> <code>torch.Tensor.scatter_add_()</code> </a>, <a href="https://pytorch.org/docs/stable/torch.html#torch.bincount"> <code>torch.bincount()</code> </a>.</p>
</blockquote>
<blockquote>
<p>A number of operations have backwards that use  <code>atomicAdd</code> , in particular <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.embedding_bag"> <code>torch.nn.functional.embedding_bag()</code> </a>, <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.ctc_loss"> <code>torch.nn.functional.ctc_loss()</code> </a> and many forms of pooling, padding, and sampling. There currently is no simple way of avoiding non-determinism in these functions.</p>
</blockquote>

username: Evan_Weissburg
cooked: <p>I’m particularly concerned about accidentally re-seeding all 8 GPUs while another task is in the process of running, thus introducing non-deterministic behavior due to the seed reset. Is this possible?</p>

username: ptrblck
cooked: <p>The seed should be local to the current application.<br>
Did you see any weird behavior?</p>
Proper way of attach LibTorch built from source in C++ project

username: jlinuxkolosky
cooked: <p>Hey,</p>
<p>So far I’ve been using LibTorch downloaded from <a href="http://pytorch.org" rel="nofollow noopener">pytorch.org</a> (pre-built), however I realized that building PyTorch from source on my machine gives me huge speed increase, so I would like to do that with LibTorch too. I built it using <a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst" rel="nofollow noopener">these</a> instructions.</p>
<p>Can anyone explain me how to use built from source LibTorch in C++ project? Its structure slightly differs from this downloaded from official website and I get some compilation errors.</p>
<p>Thank you</p>

username: driazati
cooked: <p>You can also use your build directly without going through conda:</p>
<pre><code class="lang-bash">mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH=&lt;whatever&gt;/pytorch/torch/share/cmake ..
make
</code></pre>
<p>Where the <code>pytorch</code> is the top level folder you have the PyTorch repo checked out to (e.g. the folder produced when you run <code>git clone https://github.com/pytorch/pytorch.git</code></p>

username: tom
cooked: <p>The fastest way probably from the point where you are at is to just use the LibTorch from the PyTorch you built. It contains all you need in <code>torch/lib</code>, <code>torch/shared</code>, <code>torch/include</code> from the Python installed directory (<code>/usr/local/lib/python*/dist-packages/torch/*</code> for me, try <code>torch.__file__</code> if you can’t find it).</p>
<p>Best regards</p>
<p>Thomas</p>
What densenet variant is used in torchvision.models?

username: zapnuk
cooked: <p>Hi,<br>
I don’t think it is explicitly stated which variant of densent with regards to bottleneck+compression torchvision.models implements.</p>
<p>AFAIK the DenseNet paper defines three variants: DenseNet, DensNet-B, and DenseNet-BC.<br>
With DenseNet-B the input of a ConvLayer is downsampled to a feature map size of 4 * ‘growth rate’.<br>
In DenseNet-BC the transition layers downsample the feature map size by a factor of theta &lt; 1 (often 0.5)</p>
<p>I looked at the implementation and it seems to me that torchvision implements the DenseNet-BC, is that correct?</p>

username: ebarsoum
cooked: <p>From the paper: <a href="https://arxiv.org/pdf/1608.06993.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1608.06993.pdf</a>, ImageNet evaluation was done on DenseNet-BC.</p>
[solved] Thread safety issue in `torch::jit::load`

username: markl
cooked: <p>The following code triggers an <code>Access violation - code c0000005</code> on my computer (Windows 10).</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#include &lt;thread&gt;
#include &lt;vector&gt;

// custom file-path management code
#include "tools/cpp/runfiles/runfiles.h"
#include "util/runfiles.h"

using bazel::tools::cpp::runfiles::Runfiles;
void main(int argc, char** argv) {
  // custom file-path management code
  std::unique_ptr&lt;Runfiles&gt; runfiles = MakeRunfiles(argv[0]);
  const std::string net_location =
      runfiles-&gt;Rlocation("plasty/nets/segnet_backbone.pt");

  std::vector&lt;std::thread&gt; threads;
  for (int i = 0; i &lt; 10; ++i) {
    threads.emplace_back([&amp;]() {
      try {
        torch::jit::script::Module m = torch::jit::load(net_location);
      } catch (const c10::Error&amp; e) {
        std::cout &lt;&lt; "error loading the net from location " &lt;&lt; net_location
                  &lt;&lt; "\n"
                  &lt;&lt; e.what();
      }
    });
  }

  for (auto&amp; thread : threads) {
    thread.join();
  }
}

</code></pre>
<p>The call stack shows it happening somewhere in <code>caffe2::Workspace::bookkeeper</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29fd36b7e80e6d6ed1265dafa937cae840d0ce78.png" data-download-href="https://discuss.pytorch.org/uploads/default/29fd36b7e80e6d6ed1265dafa937cae840d0ce78" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29fd36b7e80e6d6ed1265dafa937cae840d0ce78.png" alt="Capture" data-base62-sha1="5Zs5dMTZfnfCVc7sa1gJiA5EBhS" width="690" height="329" data-dominant-color="E0E5E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">747×357 7.44 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is the <code>jit::load</code> function non-threadsafe or did I do something wrong?</p>

username: markl
cooked: <p>I think the error might be because I compiled in debug mode while using the release version of the torch library.</p>
Orthogonal vector to the given N dimensional vector using PyTorch

username: absathe
cooked: <p>Greetings,</p>
<p>I am new to PyTorch and want to use it in my undergraduate math project. I am working on a project where I need a vector v from the given vector u such that dot product of u and v is zero. If this is a 2 or 3 dimensional problem, it will be very easy. However, in my case the dimension will be arbitrary value (N). How can I get this efficiently using PyTorch ?</p>
<p>I have an approach in mind although I’m not sure if it will be correct. If I generate N-2 random vectors, then I can arrange N-1 vectors (N-2 random vectors plus my vector u) in a matrix (like we do for vector cross product). And then I can simply get the N components of normal/orthogonal vector using alternating determinant values (just like vector cross product). I’m not sure if vector cross product property holds in N dimensions. Also, can someone suggest me a way to speed up the determinant calculation using PyTorch ?</p>
<p>Summarized, my questions are as follows:</p>
<ol>
<li>How to find normal/orthogonal vector to the given N dimensional vector ? [I’m not sure that a normal vector and orthogonal vector are same things in N dimensional space, I need their dot product to be zero]</li>
<li>If my previously mentioned approach is correct, how can I speed up the determinant calculation process using PyTorch ?</li>
</ol>
<p>[Mods please let me know if this is the correct place to discuss this. Apologies if it isn’t]</p>

username: SimonW
cooked: <p>if you just need <strong>any</strong> orthogonal vector, you can do very simple things like</p>
<pre><code class="lang-auto">u = v.clone()
u[-1] = -(v[:-1] @ u[:-1]) / v[-1]
</code></pre>
Create model with different classes for finetuning

username: Ruben_Maestro
cooked: <p>Hello guys, i have applied the finetuning example (<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a>) with pedestrians and I applied it with my own object. In the example, the code differs only 1 class of person. But what i need is to train the ground, and 2 different objects (A and B)<br>
These are the different objects:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9b54a8f7176d3492c2caeade2ff7fb07594d066.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b9b54a8f7176d3492c2caeade2ff7fb07594d066" title="object"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9b54a8f7176d3492c2caeade2ff7fb07594d066_2_690x366.jpeg" alt="object" data-base62-sha1="quQKiuR0XqxurG2yZx2iKn1sgm2" width="690" height="366" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9b54a8f7176d3492c2caeade2ff7fb07594d066_2_690x366.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9b54a8f7176d3492c2caeade2ff7fb07594d066_2_1035x549.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9b54a8f7176d3492c2caeade2ff7fb07594d066_2_1380x732.jpeg 2x" data-dominant-color="F8FAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">object</span><span class="informations">2048×1088 103 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have trained the object A and with this model i were able to predict both objects (A and B) but i dont want it!<br>
I would like to differ the models and with modelA i dont want to recognise the object B. I only want to recognise object A with modelA. What should i change, the number of classes?</p>
<p>Thank you very much!</p>