akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.cuda.device_count() returns 1 even if the environment variable CUDA_VISIBLE_DEVICES set to 1,2,3,4,5,6,7,8,9,10

rahulraj1990: <p>I have about 10 GPU host indexes to be run on distributed mode. I need to use all the GPU machines available. But the problem is that torch.cuda.device_count() returns 1. I verified that the environment variable do have proper values ( 1,2,3,4,5,6,7,8,9,10 → indicating all 10 device indexes) .<br>
Can anyone tell me whats going wrong here? Really appreciate your time.</p>

agolynski: <p>Hi,</p>
<p>Just to clarify: you have 10 nodes and each node has a few GPUs (how many)? (torch.cuda.device_count returns number of GPU devices on a given machine)</p>

rahulraj1990: <p>There are 4 nodes and 10 GPU indices in total.<br>
Two nodes have 3 GPU indices each.<br>
Two nodes have 2 GPU indices each.</p>

agolynski: <p>if everything is setup properly torch.cuda.device_count() should return 2 or 3 respectively, not 1 or 10.</p>
<p>What environment variables do you mean here?</p>

rahulraj1990: <p>I’m referring to environment variable: <strong>CUDA_VISIBLE_DEVICES</strong><br>
This is set to <strong>1,2,3,4,5,6,7,8,9,10</strong> and verified the same while debugging.</p>
<p>So, I’m not sure what is going wrong here. Device count is proper if I print out from console, but getting 1 on code execution.  There’s nothing on the code that could mess up the device count.</p>
<ol>
<li>Is there any way by which the device count gets modified? (For example, with the use of CUDA_VISIBLE_DEVICES)</li>
<li>
<blockquote>
<p>if everything is setup properly torch.cuda.device_count() should return 2 or 3 respectively</p>
</blockquote>
</li>
</ol>
<p>Can you elaborate more on the setup?  I’m trying to figure out what is causing the issue.</p>

agolynski: <p>A couple of things: I think <strong>CUDA_VISIBLE_DEVICES</strong> is 0-based, so it should be set to something like “0, 1, …”</p>
<p>You have 4 machines with 3, 3, 2, 2 GPUs respectively, so CUDA_VISIBLE_DEVICES should be set on each of machines independently or you may just omit setting CUDA_VISIBLE_DEVICES and it should work as well (you’ll get all avail devices by default)</p>
Using NCCL and Gloo

ekurtic: <p>Hi everyone,</p>
<p>Is it possible to use NCCL backend for training of a DDP model and a new group with Gloo backend to do <code>gather</code> operation for <code>cpu</code> tensors?</p>
<p>I’ll try to illustrate my use case since there might be a cleaner/easier solution for it:</p>
<ol>
<li>I have a DDP model, training it on N GPUs with <code>nccl</code> backend.</li>
<li>I have attached some gradient hooks on the weight param of some layers, and I am storing these gradients in all processes.</li>
<li>After some time, I would like to gather stored gradients from all processes in the main process to do some computation with it.</li>
</ol>
<p>Since <code>gather</code> is not supported in <code>nccl</code> backend, I’ve tried to create a new group with <code>gloo</code> backend but for some reason the process hangs when it arrives at the: <code>torch.distributed.gather(..., group=my_gloo_group)</code>.</p>
<p>Note: using <code>all_gather</code> in <code>nccl</code> is not an option because gradients are stored as <code>cpu</code> tensors. Using <code>all_gather</code> with <code>gloo</code> is not an option since storing <code>world_size</code> times this large “storage of gradients” is impossible.</p>

agolynski: <p>Hi,</p>
<p>Could you share some code that reproduces the hang?</p>

ekurtic: <p>Hi <a class="mention" href="/u/agolynski">@agolynski</a> ,</p>
<p>I think I’ve managed to solve the issue. I didn’t know that I have to call <code>torch.distributed.gather(...)</code> in non-master processes. So the fix for the issue was basically changing this code snippet:</p>
<pre><code class="lang-python">if torch.distributed.get_rank() == 0:
    gathered_data = [...]
    torch.distributed.gather(tensor=my_tensor, gather_list=gathered_data, group=gloo_group_handle)
</code></pre>
<p>to:</p>
<pre><code class="lang-python">if torch.distributed.get_rank() == 0:
    gathered_data = [...]
    torch.distributed.gather(tensor=my_tensor, gather_list=gathered_data, group=gloo_group_handle)
else:
    torch.distributed.gather(tensor=my_tensor, group=gloo_group_handle)
</code></pre>
<p>I thought that non-master processes will somehow magically be pinged by the master process to send their tensors.</p>

agolynski: <p>Thanks for the update!</p>
<p>Non master process needs to know which tensors to send to master hence you need to call gather there too</p>
Error on calling backward() on complex tensor

Sherine_Brahma: <p>Hi, I got the following error when I called backward on a complex tensor. I would like to know if there is a way to solve it.</p>
<h1><a name="error-1" class="anchor" href="#error-1"></a>Error</h1>
<blockquote>
<p>RuntimeError: Expected isFloatingType(grad.scalar_type()) || (input_is_complex == grad_is_complex) to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)</p>
</blockquote>
<p>I have provided a very simple example below for which I am facing the error. The way I am feeding complex inputs to my network is by separating the real and imaginary part into different channels. Once I get the output I arrange the output again into a complex tensor and call backward based on the computation.</p>
<h1><a name="code-2" class="anchor" href="#code-2"></a>Code</h1>
<pre><code class="lang-auto">&gt; y1 = net(x1)
&gt; y2 = net(x2)
&gt; complex_y1 = (y1[0, :] + (1j * y1[1, :]))
&gt; complex_y2 = (y2[0, :] + (1j * y2[1, :]))
&gt; loss= (complex_y1 - complex_y2 ) * torch.conj(complex_y1 - complex_y2 )
&gt; loss.backward(torch.ones(loss.shape, dtype=torch.complex64).cuda())
</code></pre>
How to implement reusable dropout for freeLB in pytorch?

111491: <p>I’m trying to implement <a href="https://arxiv.org/abs/1909.11764" rel="noopener nofollow ugc">FreeLB adversarial training strategy</a> in my custom model, and find out that as it was mentioned in the paper, reusing dropout mask could be quite beneficial in adversarial training, which means you reuse or in other word “freeze” any dropout mask in your model when doing gradient ascent for generating perturbation. But I find out  in their source code it was implemented with pytorch1.4, quite complicated, and then get stuck <img src="https://discuss.pytorch.org/images/emoji/apple/sleepy.png?v=9" title=":sleepy:" class="emoji" alt=":sleepy:">. Is there any quick solution for this today in pytorch1.8?</p>

111491: <p>Just found out that  <strong>torch.manual_seed</strong> can help.</p>
Loss for imbalanced multi-label classification

zyc: <p>Hi,</p>
<p>I used multi-hot labeling for the multi-label cls problem. Initially I was using BCEWithLogitsLoss but as the dataset set is quite imbalanced, it soon predicts all 0. I have tried focal loss as following but the model just does not converge. Is there any suggestion?</p>
<pre><code class="lang-python">def focal_loss(self, pred, gt):
        ''' Modified focal loss. Exactly the same as CornerNet.
          Runs faster and costs a little bit more memory
        '''
        pos_inds = gt.eq(1).float()
        neg_inds = gt.lt(1).float()

        loss = 0

        pos_loss = torch.log(pred) * torch.pow(1 - pred, 2) * pos_inds
        neg_loss = torch.log(1 - pred) * torch.pow(pred, 2) * neg_inds

        num_pos  = pos_inds.sum()
        pos_loss = pos_loss.sum()
        neg_loss = neg_loss.sum()

        if num_pos == 0:
            loss = loss - neg_loss
        else:
            loss = loss - (pos_loss + neg_loss) / num_pos

        return loss
</code></pre>

KFrank: <p>Hi Zhiyuan!</p>
<aside class="quote no-group" data-username="zyc" data-post="1" data-topic="118309" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zyc/48/25460_2.png" class="avatar"> zyc:</div>
<blockquote>
<p>Initially I was using BCEWithLogitsLoss but as the dataset set is quite imbalanced, it soon predicts all 0.</p>
</blockquote>
</aside>
<p>Consider using <code> BCEWithLogitsLoss</code> with its <code>pos_weight</code> (or possibly<br>
its <code>weight</code>) constructor-argument to account for the data imbalance.</p>
<p>Best.</p>
<p>K. Frank</p>
How can i freeze weights in element-wise

tree5680: <p>i want to fix weights which have a value of ‘zero’ in model.parameters() (model is a neural network) it means i have to freeze it in element-wise so i made a code below.</p>
<p>in the code, model.parameters() will be input of ‘Lparam’ and when i ran the code, i got an error like 'nonetype has no attribute ‘zero_()’</p>
<p>and when i try another trick like <code>requires_grad=False</code>, error says like, “only leaf variables can use <code>requires_grad</code>”</p>
<p>then how can i approach leaf variables to fix the zero weights in the computation graph? here is my code.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/597e3cfdcf934146f39f272aa18acaa168d8b57f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/597e3cfdcf934146f39f272aa18acaa168d8b57f" title="KakaoTalk_20210413_104326868"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/597e3cfdcf934146f39f272aa18acaa168d8b57f_2_414x243.jpeg" alt="KakaoTalk_20210413_104326868" data-base62-sha1="cLGXsEU0kclshQDug0anX9bQSuP" width="414" height="243" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/597e3cfdcf934146f39f272aa18acaa168d8b57f_2_414x243.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/597e3cfdcf934146f39f272aa18acaa168d8b57f_2_621x364.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/597e3cfdcf934146f39f272aa18acaa168d8b57f_2_828x486.jpeg 2x" data-dominant-color="F8F9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">KakaoTalk_20210413_104326868</span><span class="informations">878×516 67.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

tom: <p>The conceptually clean way to fix some part of weights is to have buffers (with <code>self.register_buffer('weight_update_mask', the_mask)</code> in the module initialization for the mask of what should be updated and the fixed weights and then in the forward use <code>weight = torch.where(self.weight_update_mask, self.weight_param, self.weight_fixed)</code>.</p>
<p>Now you might get by with <code>with torch.no_grad(): param[fixed_mask] = 0</code> instead. But I would view this as an optimization attempt for the former and it is not true that it is the same when you consider optimizers that don’t work fully elementwise but take the weight/gradient in its entirety into consideration (e.g. LARS/LAMB etc.).</p>
<p>Best regards</p>
<p>Thomas</p>

tree5680: <p>i really appreciate your answer!<br>
i’m very new to pytorch so it took a little of time to understand your comment and now i have a one more question about optimizer.</p>
<p>i’m trying to fix the training weight using an SGD optim with momentum and i wonder ‘fixing weight in the part of the nn.parameters() manually under the SGD optimizer with momentum’ is possible?</p>
Argmax() Returns only Zeros

Tornike: <p>Hello, CrossEntropyLoss Returns 2 values in tensor. i want these tensors to return 0 or 1 as classes.</p>
<p>But.</p>
<blockquote>
<p>for i in range(len(y_test)):<br>
print(y_val[i].argmax().item())</p>
</blockquote>
<p>argmax().item() only returns zeros.</p>
<p>How can i retunr zero or 1 out of these tensors?</p>
<blockquote>
<p>for i in range(10):<br>
print(y_val[i])</p>
<p>tensor([ 4.0784, -4.6333])<br>
tensor([ 3.4433, -3.3892])<br>
tensor([ 2.8473, -2.9989])<br>
tensor([ 1.4762, -1.3092])<br>
tensor([ 4.6216, -5.5956])<br>
tensor([ 2.1936, -1.9820])<br>
tensor([ 2.4091, -1.7561])<br>
tensor([ 4.1027, -3.2408])<br>
tensor([ 2.3697, -2.1431])<br>
tensor([ 2.7150, -2.2833])</p>
</blockquote>

KFrank: <p>Hi Tornike!</p>
<aside class="quote no-group quote-modified" data-username="Tornike" data-post="1" data-topic="118270" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tornike/48/36047_2.png" class="avatar"> Tornike:</div>
<blockquote>
<p>CrossEntropyLoss Returns 2 values in tensor.<br>
…<br>
argmax().item() only returns zeros.<br>
…</p>
<blockquote>
<p>tensor([ 4.0784, -4.6333])<br>
tensor([ 3.4433, -3.3892])<br>
tensor([ 2.8473, -2.9989])<br>
…</p>
</blockquote>
</blockquote>
</aside>
<p>The problem is not <code>argmax()</code> – some other part of your code is not<br>
giving the values you expect for <code>y_val</code>.  They are not reasonable<br>
results from <code>CrossEntropyLoss</code> (if that’s what you are expecting).</p>
<p>In the examples you printed out the <code>[0]</code> element is positive, and the<br>
<code>[1]</code> element is negative, so the maximum value is the <code>[0]</code> element.<br>
<code>argmax()</code> is correctly returning <code>0</code> – the index of the maximum element.</p>
<p>Best.</p>
<p>K. Frank</p>
Hyperparameter optimising on pre-defined validation set

umefarp: <p>Hi all,</p>
<p>I am working on a binary classification task where there is significant class imbalance. Therefore, I would like to run 3 different experiments. I have a pre-defined training, validation and test set. The experiments are as follows:</p>
<ol>
<li>‘Normal’ data set that is split in to a train/validation/test set. Use validation set for hyperparameter tuning.</li>
<li>Use the train/validation/test sets and then upsample the minority class in the training set, find optimal hyperparameters on validation set, test on test set.</li>
<li>Use train/validation/test set, downsample the majority class in the training set and do as above.</li>
</ol>
<p>So my question is suitable to do in pytorch? I have done these experiments using SVM in sklearn where you can use pre-defined validation sets. But are there hyperparameter tuning libraries for pytorch that allow for this? I found <a href="https://github.com/cgnorthcutt/hypopt" rel="noopener nofollow ugc">this</a>, but it seems to be broken at the moment. I suppose I could do the tuning manually but this seems rather inefficient.</p>
How to align multi-channel signal with channels has different time steps

yousanke: <p>Hi,<br>
I am training an attention-based model. Multiple extractors are applied to my time-series input, resulted in multichannel signals with different timesteps. Apparently, I have to align/merge those channels before I can feed them to the model. I’d like to know what is the common practice for this kind of tasks.</p>
<p>I can think of a few ways to do this:<br>
a) padding with 0s</p>
<p>[a0 a1 a2 a3 a4]<br>
[b0 b1]<br>
[c0 c1 c2]</p>
<p>will become</p>
<p>[a0 a1 a2 a3 a4]<br>
[b0 b1 0   0   0  ]<br>
[c0 c1 c2  0   0  ]</p>
<p>The most obvious problem of this way is that the model has to align the timesteps for different channels. Especially when positional encoding is applied evenly among channels.</p>
<p>b) align signal with preprocessing<br>
[a0 a1 a2 a3 a4]<br>
[b0 b1]<br>
[c0 c1 c2]</p>
<p>will become</p>
<p>[a0 a1 a2 a3 a4]<br>
[b0 0   0   0   b1]<br>
[c0 0   c1  0   c2]</p>
<p>It makes more sense because the relative position is preserved, but I wonder will I have to consider the higher frequency components introduced by brutally inserting 0. Are there documents targeting these questions?</p>
How to change between train and evaluate mode in forward method?

Thabang_Lukhetho: <p>I’m trying to build a next frame generation model which behaves in the following manner"</p>
<ul>
<li>During training, the model expects both the input tensors, as well as targets and will return a dict containing the classification and regression losses:</li>
</ul>
<pre><code class="lang-auto">loss_dict = model(images, targets)
</code></pre>
<ul>
<li>During evaluation, the model requires only the input tensors, and returns the post-processed<br>
predictions:</li>
</ul>
<pre><code class="lang-auto">predictions = model(images)
</code></pre>
<p>How can I toggle between these two modes in <code>def forward(self, images):</code> of a PyTorch Lightning system?</p>

ptrblck: <p>Each <code>nn.Module</code> uses the internal <code>self.training</code> flag, which is changed by calling <code>model.train()</code> and <code>model.eval()</code>, and can be used as a condition to use different behaviors inside the module.</p>

Thabang_Lukhetho: <p>Thanks I understand.</p>
Freeze and Unfreeze the network

oasjd7: <pre><code class="lang-auto">[Input -&gt; Encoder -&gt; Classifier
                  -&gt; Classifier2]
</code></pre>
<pre><code class="lang-auto">def freeze(...) # defined to freeze the network
def unfreeze(...) # defined to unfreeze the network

# in train-loop:
freeze(encoder)
unfreeze(classifier)

enc_out = encoder(input)
class_out = classifier(enc_out)
class_loss = ce(class_out, target)
# -&gt; trained only classifier.

unfreeze(encoder)
unfreeze(classifier2)
class_out2 = classifier2(enc_out) # This part !!!
class_loss2 = ce(class_out2, target)
...
</code></pre>
<p>If I use <code>enc_out</code> when training <code>classifier2</code>, does the encoder also trained?<br>
Or do I have to do <code>enc_out = encoder(input)</code> again after <code>unfreeze(encoder)</code>?</p>

ptrblck: <p><code>enc_out</code> would also get gradients, if you pass its output to <code>classifier2</code> directly and call <code>backward()</code> on the calculated loss. To avoid it, you should <code>detach</code> the tensor:</p>
<pre><code class="lang-python">class_out2 = classifier2(enc_out.detach())
</code></pre>
How to find element in tensor A that is in tensor b

Bob_Li: <p>Given tensor A=torch.tensor([[2, 1, 5], [3, 2, 4]]) and B=torch.tensor([1, 3, 5])<br>
How can I return tensor C that tells whether element in A is in B, i.e. C=torch.tensor([[False, True, True], [True, False, False]]).</p>

iacob: <p>You can do this with a <a href="https://stackoverflow.com/q/66779647/9067615" rel="noopener nofollow ugc">for loop</a>:</p>
<pre><code class="lang-python">&gt;&gt;&gt; sum(A==i for i in B).bool()
tensor([[False,  True,  True],
        [ True, False, False]])
</code></pre>
Pytorch Cuda Device Count is 1 but I have 8 GPU cards

lamour_hey: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/ae6c3faa74633b7fb1231153d7d88715f09fb8d0.png" data-download-href="https://discuss.pytorch.org/uploads/default/ae6c3faa74633b7fb1231153d7d88715f09fb8d0" title="50"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae6c3faa74633b7fb1231153d7d88715f09fb8d0_2_689x417.png" alt="50" data-base62-sha1="oT106dlxnsgmeR3QCxSpuwc0Qy4" width="689" height="417" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae6c3faa74633b7fb1231153d7d88715f09fb8d0_2_689x417.png, https://discuss.pytorch.org/uploads/default/optimized/2X/a/ae6c3faa74633b7fb1231153d7d88715f09fb8d0_2_1033x625.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/ae6c3faa74633b7fb1231153d7d88715f09fb8d0.png 2x" data-dominant-color="F3F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">50</span><span class="informations">1106×670 95.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can somebody help me ?</p>

ASHUTOSH_CHANDRA: <p>By default pytorch uses only 1 gpu (device:0). If you want to use multiple gpu’s you need to configure it explicitly.</p>
<p>Here is a simple way to do it : <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#create-model-and-dataparallel" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#create-model-and-dataparallel</a></p>

lamour_hey: <p>Thanks for your reply. The issue is my torch.cuda.device_count() is 1. However, it should be 8.</p>

rahulraj1990: <p>I’m also getting the same issue<br>
No working solution so far found on the forum <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
What is the difference between using tensor.cuda() and tensor.to(torch.device(“cuda:0”))

Leockl: <p>Using PyTorch, what is the difference between the following two methods in sending a tensor to GPU:</p>
<p>Method 1:</p>
<pre><code class="lang-auto">X = np.array([[1, 3, 2, 3], [2, 3, 5, 6], [1, 2, 3, 4]])
X = torch.DoubleTensor(X).cuda()
</code></pre>
<p>Method 2:</p>
<pre><code class="lang-auto">X = np.array([[1, 3, 2, 3], [2, 3, 5, 6], [1, 2, 3, 4]])
X = torch.DoubleTensor(X)

device = torch.device("cuda:0")
X = X.to(device)
</code></pre>
<p>Similarly, is there any difference in the same two methods above when applied to sending a model to GPU:</p>
<p>Method A:</p>
<pre><code class="lang-auto">gpumodel = model.cuda()
</code></pre>
<p>Method B:</p>
<pre><code class="lang-auto">device = torch.device("cuda:0")
gpumodel = model.to(device)
</code></pre>
<p>Many thanks in advance!</p>

iffiX: <p>there is no difference</p>

ptrblck: <p>There might be a difference, if you were resetting the default CUDA device via <code>torch.cuda.set_device()</code> as seen in this code snippet:</p>
<pre><code class="lang-python">torch.cuda.set_device('cuda:1')
x = torch.randn(1).cuda()
print(x)
&gt; tensor([0.9038], device='cuda:1') # uses the default device now

y = torch.randn(1).to('cuda:0')
print(y)
&gt; tensor([-0.7296], device='cuda:0') # explicitly specify cuda:0
</code></pre>

Leockl: <p>Ok thanks <a class="mention" href="/u/iffix">@iffiX</a> for confirming they are both essentially doing the same thing.</p>

iffiX: <p>Ah yes, thats important, I forgot this <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

Leockl: <p>Ok many thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for the more detailed answer where the 2nd method is specifying which GPU device to use and the 1st method is just using the default GPU device.</p>

iacob: <p>Their syntax varies slightly, but <a href="https://stackoverflow.com/questions/62907815/pytorch-what-is-the-difference-between-using-tensor-cuda-and-tensor-totorch/66568863#66568863" rel="noopener nofollow ugc">they are equivalent</a>:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>⠀</th>
<th>.to(name)</th>
<th>.to(device)</th>
<th>.cuda()</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>CPU</strong></td>
<td><code>to('cpu')</code></td>
<td><code>to(torch.device('cpu'))</code></td>
<td><code>cpu()</code></td>
</tr>
<tr>
<td><strong>Current GPU</strong></td>
<td><code>to('cuda')</code></td>
<td><code>to(torch.device('cuda'))</code></td>
<td><code>cuda()</code></td>
</tr>
<tr>
<td><strong>Specific GPU</strong></td>
<td><code>to('cuda:1')</code></td>
<td><code>to(torch.device('cuda:1'))</code></td>
<td><code>cuda(device=1)</code></td>
</tr>
</tbody>
</table>
</div><p>Note: the current cuda device is <code>0</code> by default, but this can be set with <code>torch.cuda.set_device()</code>.</p>
Export onnx got RuntimeError: Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient

bigtree: <p>I am trying to export the model to onnx.<br>
I can achieve that using the torch.onnx.export()</p>
<pre><code class="lang-auto">class net_nn(torch.nn.Module):
   def __init__(self, training=True):
       ....
   def load_weights(self, path, args=None):
       ....
   def foward(self, x):
       ....

model = net_nn(training=False)
model .load_weights(args.trained_model, args=args)

dummy_input = torch.ones(2, 3, dtype=torch.long)

torch.onnx.export(model, dummy_input , 'net_nn.onnx', verbose=True)
</code></pre>
<p>But what I want to do further is adding the post-processing in the onnx. Here is what I did,</p>
<pre><code class="lang-auto">class net_nn(torch.nn.Module):
   def __init__(self, training=True):
       ....
   def load_weights(self, path, args=None):
       ....
   def foward(self, x):
       ....

model = net_nn(training=False)
model .load_weights(args.trained_model, args=args)


def post_processing(x):
    ....

class net_and_post(torch.nn.Module):
      def foward(self, x):
        pred = model(x)
        return post_processing(pred)

model_with_post = net_and_post()
dummy_input = torch.ones(2, 3, dtype=torch.long)
torch.onnx.export(model_with_post , dummy_input , 'net_nn_post.onnx', verbose=True)
</code></pre>
<p>But it ends by<br>
<code>RuntimeError: Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient</code></p>
<p>Please help.</p>
Is it possible to select a partial view of tensor with same indices from multiple dimensions?

Arna_Ghosh: <p>Is there an alternative to using <code>torch.diagonal()</code> which does not alter the ordering of dimensions of a tensor?<br>
For example, I have a 4D tensor, A, of shape (2,4,4,6). I want B = A[:,i,i,:] for all i in range(4).<br>
I did the following in PyTorch:</p>
<pre><code class="lang-auto">tmp = torch.diagonal(A,dim1=1,dim2=2)
B = tmp.permute(0,2,1).contiguous()
</code></pre>
<p>Is there a way to avoid the permute() operation?<br>
Many thanks. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

googlebot: <blockquote>
<p>torch.zeros(2,4,4,6).diagonal(0,1,2).shape<br>
torch.Size([2, 6, 4])<br>
torch.zeros(2,4,4,6).diagonal(0,1,2).stride()<br>
(96, 1, 30)</p>
</blockquote>
<p>Note the peculiar stride 30, diagonal() is returning a specially constructed view onto input’s memory. Hence, contiguous() is needed regardless of dimensions’ ordering, if you want to copy &amp; compact the output.</p>
Where should I look to solve "running_mean" error in ResNet transfer learning?

sguerin: <p>Hi! I’m trying to implement transfer learning on binary classification using ResNet model. I have tried the solution showed in  <a href="https://discuss.pytorch.org/t/modify-resnet-or-vgg-for-single-channel-grayscale/22762">this discussion</a> but I’m stuck in this error:</p>
<pre><code class="lang-bash">RuntimeError: running_mean should contain 128 elements not 64
</code></pre>
<p>This is the way I define and call the pretrained model:</p>
<pre><code class="lang-python">class ResnetPretrained(models.resnet.ResNet):
    def __init__(self, block, layers, num_classes=2):
        self.inplanes = 128
        super(ResnetPretrained, self).__init__(block, layers)
        self.conv1 = nn.Conv2d(1, 128, kernel_size=(7, 7), bias=False)
</code></pre>
<pre><code class="lang-python">model = ResnetPretrained(models.resnet.Bottleneck, [3, 8, 36, 3]).to(device)
</code></pre>
<p>My input is 128x128, 1 channel and I’m normalizing the datasets with its own mean and std parameters.</p>
<p>This line calls the error:</p>
<pre><code class="lang-python">for epoch in range(self.num_epochs):
    i = 0
    for images, labels in self.train_set_loader:
        images = images.to(device)
        labels = labels.to(device))
        outputs = model(images)       &lt;----------- this line calls the error
        loss = criterion(outputs, labels.long())  
</code></pre>
<p>The error reference points to this <a href="https://github.com/pytorch/pytorch/blob/c5f9e043e96f5c60e97292855572576335002c8a/torch/nn/functional.py#L2243" rel="noopener nofollow ugc">batch_norm nn.functional function</a></p>
<p>I don’t understand what this error means, where should I look to solve it?</p>
<p><strong>Thanks!</strong></p>

KaiHoo: <p>The original resnet’s first convolution out channel is 64, but you are using 128. Thus it does not work with the next batch norm as well as following layers.</p>
<p>Please use self.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), bias=False); self.inplanes = 128 or you have to change the entire network.</p>

sguerin: <p>Thanks! Now it’s working <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
How to do tensor product of tensors?

Capangas: <p>Hi, I’m wondering how can I do an outer product of two tensor, of shapes (batch_size, dim), (batch_size, dim), so that the outer product is applied in the last dimension, resulting in a tensor of shape (batch_size, dim, dim), according to the formula:</p>
<p>x_i_j times y_i_k =&gt; a_i_j_k = x_i_j y_i_k</p>
<p>Don’t know how to use latex here, I hope that’s clear. If the batch size was one, there would be two vectors of which I want the outer product. Since there is a batch, I would need to do the same for every sample of the batch.</p>
<p>In general, how can operations that don’t broadcast (like outer) be applied to specific dimensions of higher dimensional tensors, especially in the case you have an extra dimension (like the batch size) that won’t be involved in the operation?</p>

Capangas: <p>Answering my own question, einsum seems to fit perfectly with the problem (I can even use the same schematic notation I used in the question lol). If there is some way that is more specific then someone might feel free to comment.</p>

KFrank: <p>Hi Capangas!</p>
<aside class="quote no-group" data-username="Capangas" data-post="2" data-topic="118225" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/capangas/48/30412_2.png" class="avatar"> Capangas:</div>
<blockquote>
<p>einsum seems to fit perfectly with the problem</p>
</blockquote>
</aside>
<p><code>einsum()</code> is a fine solution – and a good general-purpose tool.</p>
<aside class="quote no-group">
<blockquote>
<p>If there is some way that is more specific then someone might feel free to comment.</p>
</blockquote>
</aside>
<p>In your specific case, you can use <code>torch.bmm()</code> with a couple of<br>
well-placed <code>unsqueeze()</code>s.  A good discussion is found in this thread:</p>
<aside class="quote" data-post="1" data-topic="4025">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/armenag/48/3597_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/batch-outer-product/4025">Batch Outer Product</a> 
  </div>
  <blockquote>
    Title says it all. Is there a way to compute a batch outer product. I noticed that pytorch conveniently has torch.ger which takes in two one-dimensional vectors and outputs there outer-product: (1, n) * (1, m) -&gt; (n, m) 
Is there a batch version of this operation? (b, n) * (b, m) -&gt; (b, n, m)
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>
Image processing suggestions for satellite image

WeNeedMore: <p>Trying to train a semantic segmentation network to segment clouds from the satellite images, but when I open the image by the rasterio, I find out many images are dark, the data owner told me those data are preprocessed by QGIS, they change the contrast manually and draw the mask. Problem is, I don’t know what kind of preprocessing the owners did when they creating the masks.</p>
<p>Is there a recommended solution to preprocess these kind of images? I simply divide the rgb bands by the maximum value</p>
<p><code>raw_rgb / np.iinfo(raw_rgb.dtype).max</code></p>
<p>Any suggestions, thanks.</p>
PyTorch equivalent of "make_one_shot_iterator"

tantheta01: <p>I am using the PyTorch DataLoader for MNIST dataset. I wanted to know if there is any equivalent of <code>make_one_shot_iterator</code> in pytorch. All I want to do is make a circular iterator so that we can do multiple passes through the dataset. I know we can achieve this by running multiple training iterations but I wanted to know if there is a way like this.</p>

ptrblck: <p>I’m not sure what <code>make_one_shot_iterator</code> does, as the <a href="https://www.tensorflow.org/api_docs/python/tf/compat/v1/data/make_one_shot_iterator">docs</a> only mention:</p>
<blockquote>
<p>Creates an iterator for elements of <code>dataset</code> .</p>
</blockquote>
<p>If you want to iterate the PyTorch <code>Dataset</code>, you could just use it in a loop:</p>
<pre data-code-wrap="python"><code class="lang-python">dataset = datasets.MNIST(root='PATH', transform=...)

for data, target in dataset:
    # ...
</code></pre>

tantheta01: <p>Currently when I iterate on the dataset, with a batch size of 64, it gives me an error at the last batch since the last batch has size 32. I want to know if there is any way I get the iterator to wrap around the dataset so I don’t get this error?</p>

ptrblck: <p>If you are using a <code>DataLoader</code>, you could drop the potentially smaller last batch via <code>drop_last=True</code>.</p>
Propagate method from MessagePassing class in PyTorch Geometric not determisitic

Oivind_Bakke: <p>Hi all,</p>
<p>I have an issue with not being able to reproduce the same output with the same input using the propagate method from the MessagePassing class from torch_geometric.nn (if it matters, aggr=‘mean’). I’ve narrowed it down to being that method specifically, by checking that everything in my own message method is deterministic. I’m otherwise using torch.manual_seed, but there seems to be some randomness induced by that specific method.</p>
<p>My question then follows:</p>
<ol>
<li>
<p>Why is there even randomness in that method to begin with? Am I missing something, or isn’t it desired to have the same aggregated message for the same input?</p>
</li>
<li>
<p>How can I overcome this, as it is inducing problems for my downstream task?</p>
</li>
</ol>
<p>I’m not exactly sure of the rules of conduct in regards to @'ing people, but I’ve seen other posts doing. If I’m overstepping, sorry in advance. <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@albanD</a></p>

Oivind_Bakke: <pre><code class="lang-auto">`    def forward(self, x, edge_index, edge_attr):
        node_embed = self.propagate(edge_index , x=x, edge_attr = edge_attr) 
        return node_embed
</code></pre>
<p>and:</p>
<pre><code class="lang-auto">sum(self.propagate(edge_index , x=x, edge_attr = edge_attr)-self.propagate(edge_index , x=x, edge_attr = edge_attr))
tensor([ 1.6689e-06, -8.1956e-07,  2.9802e-08, -1.0431e-07, -5.0664e-07,
         2.3842e-07, -1.0431e-07, -4.2468e-07, -1.4007e-06,  5.1409e-07,
        -9.5367e-07, -9.5367e-07, -2.9802e-08, -2.3842e-07, -2.8312e-07,
         1.7881e-07,  2.9802e-08, -9.5367e-07, -2.9802e-07, -2.9802e-08,
        -4.7684e-07, -2.3842e-07, -9.5367e-07,  0.0000e+00,  1.0431e-07,
         4.7684e-07,  4.7684e-07,  1.6689e-06,  0.0000e+00,  0.0000e+00,
        -9.5367e-07, -2.9802e-07], device='cuda:0', grad_fn=&lt;AddBackward0&gt;)
</code></pre>

ptrblck: <p>Could you check the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> and check, if you’ve followed all advices?</p>
<aside class="quote no-group" data-username="Oivind_Bakke" data-post="1" data-topic="118133">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oivind_bakke/48/37098_2.png" class="avatar"> Oivind_Bakke:</div>
<blockquote>
<p>I’m not exactly sure of the rules of conduct in regards to @'ing people, but I’ve seen other posts doing.</p>
</blockquote>
</aside>
<p>I’m not a huge fan of tagging certain users, as it could discourage others to post a valid answer, unless you want to directly address this user for some reason.</p>
NLLoss for framewise classification using LSTM not working with a batch dimension

HenryPowell: <p>I’m trying to do a frame by frame classification of some skeleton data using the torch.nn.NLLLoss().</p>
<p>The input is of size [batch_size, frames, features] and the network outputs logsoftmax values for each of the 12 classes for each of the frames so [batch_size, frames, classes=12]</p>
<p>I then want to compute the Negative Logliklihood Loss for this output based on the ground truth classes for each frame. At the moment the labels tensor is of size [batch_size, frames] where the second dimension contains the integer valued classes.</p>
<p>When I try to compute the loss I get the following error</p>
<pre><code class="lang-auto">ValueError: Expected target size (50, 12), got torch.Size([50, 175])
</code></pre>
<p>For a single example from a batch the loss seems to work but when I introduce the batch dimension it throws the above.</p>
<p>Any ideas on how to fix this would be greatly appreciated.</p>
<p>Cheers.</p>

ptrblck: <p>If you want to compute the loss for all frames, the output should have the shape <code>[batch_size=50, nb_classes=12, nb_frames=175]</code> while the target shape is correct in this case.<br>
I’m not sure, how your model processes the input and if you are expecting to get all frame outputs in a single tensor. If not, you would have to calculate the loss per frame by indexing the target in <code>dim1</code>.</p>

HenryPowell: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. Just to be sure I’m not making any errors. The models is as follows:</p>
<pre><code class="lang-auto">class SeqLSTM(nn.Module):
    def __init__(self, input_size, num_classes, num_layers, dropout):
        super(SeqLSTM, self).__init__()

        self.lstm_one = nn.LSTM(input_size=input_size, hidden_size=int(input_size * 1.5),
                                num_layers=num_layers, dropout=dropout, batch_first=True)
        self.linear = nn.Linear(in_features=int(input_size * 1.5), out_features=num_classes)
        self.softmax = nn.LogSoftmax(dim=1)


    def forward(self, input):
        x, (hidden, cell) = self.lstm_one(input)
        x = self.linear(x)
        x = self.softmax(x)
        return x
</code></pre>
<p>The aim is to have each output tensor represent the softmax probabilities that each frame is of a particular class. I’m guessing that I’ll just have to permute the output to get the <code>[batch_size=50, nb_classes=12, nb_frames=175]</code> order that you suggested?</p>
<p>Thanks again</p>

ptrblck: <p>Yes, you could permute it and you should check the output shape of the <code>self.lstm_one</code>.<br>
Based on the used config I assume the output would have the shape <code>[batch_size, seq_len, nb_features]</code>. If that’s the case, you could feed it into the linear layer and would then get an output in <code>[batch_size, seq_len, out_features=num_classes]</code>. Permute this output of the linear layer, and pass it to the log softmax (if you are using <code>nn.NLLLoss</code>) or remove the log softmax and pass it directly to <code>nn.CrossEntropyLoss</code>.</p>
Batch Normalization objects

shrutishrestha: <p>Suppose we initialized batchnorm:<br>
bn1=nn.BatchNorm2d(64,track_running_stats=False)</p>
<p>Can we use this bn1 after every convolution layer that outputs 64 channels?</p>
<p>or do we need to do:<br>
bn1=nn.BatchNorm2d(64,track_running_stats=False)<br>
bn2=nn.BatchNorm2d(64,track_running_stats=False)<br>
for using after 2 conv layers with 64 layers output each?</p>
<p>Do batch norm track weights i.e. running mean and variance from previous states?</p>

ptrblck: <p>You could reuse the same batchnorm layer, but this would indeed reuse the same affine parameters as well as update the running stats with each input, so it’s most likely not what you want.<br>
The standard approach would be to create a new batchnorm layer.</p>

shrutishrestha: <p>Ok. Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

shrutishrestha: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> is this the same for conv blocks too? Could we use one conv3x3 object for different layers or we should use different conv3x3 layers at different conv layers?</p>

ptrblck: <p>The same applied for all layers. Usually you would recreate them and use separate layers with their individual trainable parameters. However, there are of course use cases, where you explicitly want to reuse the parameters, in which case you could create a module and use it several times in the <code>forward</code> pass.</p>
Replace last layer of TorchScript model

fujenchu: <p>Dear experts,</p>
<p>Recently I have a need to load a torch scripted model and modify the last layer to finetune.<br>
An example is provided as below:</p>
<pre><code class="lang-auto">from torchvision import models
import torch
import torch.nn as nn
# Example of how I saved the model.
model_ft = models.resnet18(pretrained=True)
model_ft_scripted = torch.jit.script(model_ft) 
model_ft_scripted.save('r18_model_ft_scripted.pt')

# Load the model and adjust the last layer.
model_ft_scripted_loaded = torch.jit.load("r18_model_ft_scripted.pt")
model_ft_scripted_loaded.fc = nn.Linear(512, 2)
</code></pre>
<p>And it complains <code>Cannot re-assign modules in a ScriptModule with non-scripted module</code>.<br>
Then I tried:</p>
<pre><code class="lang-auto">linear_module = nn.Linear(512, 2)
linear_module_scripted = torch.jit.script(linear_module) 
model_ft_scripted_loaded.fc = linear_module_scripted
</code></pre>
<p>But this time the error is:</p>
<pre><code class="lang-auto">Expected a value of type '__torch__.torch.nn.modules.linear.Linear (of Python compilation unit at: 0x7f6bf29a53d0)' for field 'fc', but found '__torch__.torch.nn.modules.linear.___torch_mangle_25.Linear (of Python compilation unit at: 0x7f6bf31ff110)'
</code></pre>
<p>I searched online the the forum but I don’t find too much information to get me through this.<br>
Any suggestions are appreciated!</p>
<p>Also, I wonder if there is a way to copy the weights from loaded scripted model. In that case, if I have the model (say, Resnet18) then I could copy the weights to the model and replace the last layer without problems.</p>
<p>P.S. My use case is to save and load models in different machines. But both in python. Thanks so much!</p>
PyTorch performance on GPU dataset

raymank26: <p>Hello,</p>
<p>I’m trying to train my neural net with following code:</p>
<pre><code class="lang-auto">class Dataset(torch.utils.data.Dataset):
    def __init__(self, df, is_valid=False):
        self.l = len(df)
        df_numpy = df.to_numpy()
        self.x = torch.from_numpy(df_numpy[:,0:2]).cuda()
        self.target = torch.from_numpy(df_numpy[:,2:3]).float().cuda()
        
    def __getitem__(self, i):
        return [self.x[i], self.target[i]]
    
    def __len__(self): return self.l

class Word2Vec(nn.Module):
    def __init__(self, vocab_size, embedding_dim):
        super().__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.embeddings_bias = nn.Embedding(vocab_size, 1)
        self.context = nn.Embedding(vocab_size, embedding_dim)
        self.context_bias = nn.Embedding(vocab_size, 1)

    def forward(self, inputs):
        a = inputs[:,0]
        b = inputs[:,1]
        
        num = (self.embeddings(a) * self.context(b)).sum(dim=1, keepdim=True)
        num += self.embeddings_bias(a) + self.context_bias(b)
        return torch.sigmoid(num)

train_ds = Dataset(train) # train.shape == shape (52962501, 3) 
train_dl = torch.utils.data.DataLoader(train_ds, batch_size=1024, shuffle=True)

model = Word2Vec(vocab_size, 128) # vocab_size == 20000
model.cuda()
</code></pre>
<p>Environment:<br>
cuda = ‘10.1’<br>
torch version = 1.7.1<br>
GPU has 32GB of memory and CPU has 24 cores.</p>
<p>Learning loop is standard (feed forward, BCELoss().backward()). Unfortunately almost all the work is CPU bound (1 CPU core is 100%) and GPU usage is about 10%. Could you help me understand why CPU is bottleneck here in case of full dataset located on GPU?</p>

raymank26: <p>Answering my own question.</p>
<p>The fix is easy. First, the load is CPU bound indeed. To move the computation to GPU (as we want to) we can change the code above to introduce batching on Dataset level by specifying sampler in DataLoader or implement batching by ourselves.</p>
<p>I’ve followed the second approach (due to the fact that fastai DataLoader doesn’t offer sampler option). The code is below (it drops last non full batch but I have aligned data):</p>
<pre><code class="lang-auto">class Dataset(torch.utils.data.Dataset):
    def __init__(self, df, bs):
        self.l = len(df)
        df_numpy = df.to_numpy()
        self.x = torch.from_numpy(df_numpy[:,0:2]).cuda()
        self.target = torch.from_numpy(df_numpy[:,2:3]).float().cuda()
        self.bs = bs
        
    def __getitem__(self, i):
        return (self.x[self.bs*i:self.bs*i+self.bs,:], self.target[self.bs*i:self.bs*i+self.bs,:])
    
    def __len__(self): return int(self.l / self.bs)
</code></pre>
Parallelizing projected gradient descent attack

Sia_Rezaei: <p>I’m trying to parallelize a projected gradient descent attack (on a single node).<br>
The models parameters and buffers do not change, but the input images do.<br>
So there is no overhead of parameter/gradient synchronization between instances of the model running on different GPUs.<br>
This should provide a good opportunity for a great deal of speed up, since there is no need to move around parameters/gradients.</p>
<p>What is the best way to implement this, so I can take full advantage of the opportunity (provided by not needing to sync model instances)</p>
<p>If I were to use DataParallel is there a way to tell it not to sync model after a forward/backward iteration? Or is it smart enough not to sync them if they haven’t changed?</p>
<p>Thanks in advance</p>

Yanli_Zhao: <p>if you do not want to sync gradients for the models, and you are using DistributedDataParallel(), you can try this:</p>
<p>ddp_model = DistributedDataParallel(model)<br>
with ddp_model.no_sync():<br>
training loop</p>