akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Select and assign gives different values on GPU for same data

username: dgriffiths3
cooked: <p>When running a select and assign statement on GPU with the same data twice I do not get the same output. However, this works as expected on CPU. What am I missing here? Is selecting on GPU non-deterministic somehow?</p>
<pre><code class="lang-auto">import torch

n = 100
y = torch.randint(0, n, (n,))

store = []
x = torch.rand(n, 3)
for _ in range(2):
	z = torch.zeros_like(x)
	z[y] += x
	store.append(z)
print((store[0]==store[1]).all())

store = []
x = torch.rand(n, 3).cuda()
for _ in range(2):
	z = torch.zeros_like(x)
	z[y] += x
	store.append(z)
print((store[0]==store[1]).all())

# console output -&gt;
# tensor(True)
# tensor(False, device='cuda:0')
</code></pre>
Using custom autograd function in C++ to connect non-pytorch model in loss function

username: mmoelle1
cooked: <p>Dear all,</p>
<p>I plan to use PyTorch’s C++ front-end for developing physics-informed neural network applications. While the network shall be based on libtorch, the physics model is implemented atop the Eigen library and has full algorithmic differentiation capabilities (forward and backward mode) using the open-source <a href="https://github.com/SciCompKL/CoDiPack" rel="noopener nofollow ugc">CoDiPack</a> library.</p>
<p>From what I read at <a href="https://pytorch.org/tutorials/advanced/cpp_autograd.html#using-custom-autograd-function-in-c" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_autograd.html#using-custom-autograd-function-in-c</a> it should be possible wrap the external physics model in a custom autograd class that provides statically defined <code>forward</code> and <code>backward</code> methods in which (i) the connection with Eigen is made and (ii) the forward and backward propagation is continued using CoDiPack.</p>
<p>Does anyone have experience in connecting libtorch with an ‘external model’ via custom auto grad classes?</p>
<p>Thanks in advance,<br>
Matthias</p>
GPU memory consumption increases

username: lifeblack
cooked: <p>Hi<br>
GPU memory consumption increases during training… I thougth reshape is the reason of this. How can I fix it?<br>
its my code:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self,SR,block_size,phi):
        super(MHCSResNet,self).__init__()
        self.conv1= nn.Sequential(
            nn.Conv2d(..),
            nn.BatchNorm2d(..),
            nn.ReLU()
        )
        self.conv2= nn.Sequential(
            nn.Conv2d(...),
            nn.BatchNorm2d(...),
            nn.ReLU()
        )
        self.conv3= nn.Sequential(
            nn.Conv2d(...),
            nn.BatchNorm2d(...),
            nn.ReLU()
        )
        self.conv4= nn.Sequential(
            nn.Conv2d(...),
            nn.BatchNorm2d(...),
            nn.ReLU()
        )
        self.conv5= nn.Sequential(
            nn.Conv2d(...),
            nn.BatchNorm2d(...),
            nn.ReLU()
        )
        self.conv6= nn.Sequential(
            nn.Conv2d(...),
            nn.BatchNorm2d(...),
            nn.ReLU()
        )
        self.conv7= nn.Sequential(
            nn.Conv2d(...),
            
        )
        self.fc=nn.Linear(6400,64)

    def forward(self,kr,y,phi):
        out_conv1=self.conv1(kr)
        out_conv2=self.conv2(out_conv1)
        out_conv3=self.conv3(out_conv2)
        out_conv4=self.conv4(out_conv3))
        out_conv5=self.conv5(out_conv4)
        out_conv6=self.conv6(out_conv5)
        out_conv7=self.conv7(out_conv6)
        #print('out_conv7',out_conv7.shape)
        out_feedback=kr+out_conv7
        out_linear=self.fc(out_feedback.flatten(2))
        out_reshape=out_linear.reshape([out_feedback.shape[0],out_feedback.shape[1],8,8])
        outfc=torch.zeros(out_feedback.shape[0],out_feedback.shape[1],8,8)
        fxr=Block_Compressed_Sensing(outfc,phi)

        #print('fxr',fxr)
        return fxr,out_reshape,outfc
</code></pre>
<p>what should I do</p>

username: lifeblack
cooked: <p>Would you please give some advice on this problem? It seems that you have a good knowledge at Pytorch. THANKS VERY MUCH!!!</p>

username: ptrblck
cooked: <p>Please don’t tag specific users, as this might discourage others to post an answer and creates some noise.</p>

username: lifeblack
cooked: <p>Ok. I dont do it again… I really want to know your opinion about the reshape part</p>
What is numpy.linalg.multi_dot() equivalent in PyTorch

username: Leockl
cooked: <p>I am trying to perform matrix multiplication of multiple matrices in PyTorch and was wondering what is the equivalent of <code>numpy.linalg.multi_dot()</code> in PyTorch?</p>
<p>If there isn’t one, what is the next best way (in terms of speed and memory) I can do this in PyTorch?</p>
<p>Code:</p>
<pre><code class="lang-auto">import numpy as np
import torch

A = np.random.rand(3, 3)
B = np.random.rand(3, 3)
C = np.random.rand(3, 3)

results = np.linalg.multi_dot(A, B, C)

A_tsr = torch.tensor(A)
B_tsr = torch.tensor(B)
C_tsr = torch.tensor(C)

# What is the PyTorch equivalent of np.linalg.multi_dot()?
</code></pre>
<p>Many thanks!</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>I don’t think there is an equivalent function for that in PyTorch (not sure), but I just looked at numpy’s source code and it seems pretty simple as all functions are available in PyTorch to implement same method as numpy. It would not take much time as you just need to replace torch instead of numpy in some cases and maybe few tricks.</p>
<p><a href="https://github.com/numpy/numpy/blob/v1.19.0/numpy/linalg/linalg.py#L2621-L2814" rel="noopener nofollow ugc">numpy/linalg.py at v1.19.0 · numpy/numpy (github.com)</a></p>
<p>PS. For 3 matrices, it’s very simple.</p>
<p>Bests</p>

username: Leockl
cooked: <p>Ok, many thanks <a class="mention" href="/u/nikronic">@Nikronic</a> When I have time, I will put this on my list of to-do’s as a mini project to contribute to PyTorch <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Nikronic
cooked: <p>That’s really nice, maybe you can create a PR on official PyTorch github repo and help other people to use it too.<br>
AFAIK, PyTorch is trying to replicate all methods in <code>np.linalg</code>, so maybe you can add this to that list and take the responsibility.<br>
<a href="https://github.com/pytorch/pytorch/issues/42666" rel="noopener nofollow ugc">Linear Algebra tracking issue · Issue #42666 · pytorch/pytorch (github.com)</a></p>
<p>You can find <code>multi_dot</code> in the list of todos (Planned for PyTorch 1.9), and has not been assigned to anyone apparently.</p>
Custom conditional loss function

username: YazanShannak
cooked: <p>Hello all,</p>
<p>I am trying to implement a custom loss function, but with a condition</p>
<pre><code class="lang-auto">def ReverseSmoothL1Loss(output, target):
    absolute_errors = torch.abs(output - target)
    absolute_errors_squared = torch.squared(absolute_errors)
    loss = torch.where(absolute_errors &gt; 1, absolute_errors_squared, absolute_errors)
    return torch.mean(loss) 
</code></pre>
<p>I am getting valid results, but is this implementation valid for autograd?</p>
Setting diagonal terms of a tensor to zero

username: blade
cooked: <p>I have a symmetric tensor of size <code>[B, N, N, C]</code>. I would like to eliminate the diagonal elements. I was wondering what is the most efficient way to do this?</p>
<p>My solution:</p>
<p>Use</p>
<pre><code class="lang-auto">        idx1 = torch.triu_indices(N, N, 0)
        idx2 = torch.triu_indices(N, N, 1)
</code></pre>
<p>Then turn them into lists, subtract to get diagonal indices. Then use them to subtract diagonal term from the original tensor. But I’m wondering</p>
<ol>
<li>Would turning indices into list break backprop?</li>
<li>Also, I’m not sure if this is the most efficient way to do this.</li>
</ol>

username: RoySadaka
cooked: <p>Interesting, does it have to be inplace?<br>
How about creating an inverted identity diagonal matrix, and then multiply element wise?<br>
your_tensor *= (1 - torch.eye(N, N))</p>
<p>Also checkout <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.fill_diagonal_" rel="noopener nofollow ugc">fill_diagonal_</a> function it might fit perfectly</p>
<p>Roy.</p>
Changing size of network tensor during training?

username: Ameet_Rahane
cooked: <p>Hi,</p>
<p>I’m wondering if I can change the size of a tensor/replace it with a copy of itself + 1 row or something during training of a pytorch network? I’d be doing this update after the backprop has happened and before the forward prop for the next epoch has happened, so presumably the computation graph wouldn’t care about this?</p>
<p>Thanks,<br>
Ameet</p>
PyTorch istft does not return the stft lenght

username: kinwai_cheuk
cooked: <p>The istft is not entirely the inverse operation for stft when <code>center=False</code>.</p>
<pre><code class="lang-python">hl = 512
n_fft = 1024

dummy_spec = torch.stft(torch.randn(1,1536), n_fft, hop_length=hl, center=False, onesided=True)
print(torch.istft(dummy_spec, n_fft, hop_length=hl, center=False, onesided=True).shape)

output &gt;&gt;&gt; torch.Size([1, 1024]) 
</code></pre>
<p>Where the expected output shape should be (1, 1536).</p>
<p>In librosa, the same operation yield the correct result</p>
<pre><code class="lang-python">dummy_lib_spec = librosa.stft(np.random.randn(1536), n_fft, hop_length=hl, center=False)
print(librosa.istft(dummy_lib_spec, hop_length=hl, center=False).shape)
output &gt;&gt;&gt; (1536,)
</code></pre>
<p>I am using PyTorch 1.6.0</p>
Only training multiple_choice_head of my output model while leaving the rest of the weights untouched

username: Hyunjin_Cho
cooked: <p>Hello,<br>
If I set my optimizer and scheduler in the following way, will it prevent the weights for the <code>model_gpt2.transformer.h.parameters()</code> to <em><strong>not</strong></em> change during the training, while ensuring the weights from <code>model_gpt2.multiple_choice_head.parameters()</code> to change during the training? I don’t want the <code>model_gpt2.transformer.h.parameters()</code> to be altered during the training, I only want to adjust <code>model_gpt2.multiple_choice_head.parameters()</code> with my training loop:</p>
<pre><code class="lang-auto">optimizer_1 = torch.optim.SGD(model_gpt2.transformer.h.parameters(), lr=0.0)
optimizer_2 = torch.optim.Adam(model_gpt2.multiple_choice_head.parameters(), lr=0.00013)
scheduler_1 = torch.optim.StepLR(optimizer_1, step_size=1, gamma=1.5)
scheduler_2 = torch.optim.StepLR(optimizer_2, step_size=1, gamma=1.5)
</code></pre>
<p>Thank you,</p>

username: Abhilash_Srivastava
cooked: <p>Instead of modifying the optimizer like that, a better way of doing it would be:</p>
<pre><code class="lang-auto">for param in model_gpt2.transformer.h.parameters():
    param.requires_grad = False
for param in model_gpt2.multiple_choice_head.parameters():
    param.requires_grad = True

</code></pre>
Gradient is None on a root node!

username: turian
cooked: <p>I have read this post and understand about root nodes: <a href="https://discuss.pytorch.org/t/grad-is-none-even-when-requires-grad-true/29826" class="inline-onebox">Grad is None even when requires_grad=True</a><br>
I have followed its advice but I still get gradient of None</p>
<p>Here is the code I run:</p>
<pre><code class="lang-auto">import os
import torch
import torch.tensor as T
from fairseq.models.wav2vec import Wav2VecModel

MODEL = "wav2vec_small.pt"
fname = MODEL
os.system("wget -c https://dl.fbaipublicfiles.com/fairseq/wav2vec/%s" % MODEL)
cp = torch.load(fname)
model = Wav2VecModel.build_model(cp['args'], None)
model.load_state_dict(cp['model'], strict=False)
model.eval()

time = torch.linspace(0.0, 2.5, 64000)
def sine(f0):
  return T(0.5) * torch.cos(f0 * 2.0 * 3.14 * time)

def extractor(x):
  return model.feature_extractor(torch.tensor(x.view(1, -1)))
def wav2vecdist(x, y):
  return torch.mean(torch.abs(extractor(x) - extractor(y)))

ft = T(440.0)
# If you enable GPU, you have to add 'device="cuda"'
with torch.enable_grad():
  fp = T(660.0, requires_grad=True)
  fp.retain_grad()
  dist = torch.mean(torch.abs(extractor(sine(ft)) - extractor(sine(fp))))
  print(dist)
  dist.backward(retain_graph=False)
  print(fp.grad)
</code></pre>
<p>As you can see, the extractor in wav2vec is not passing a gradient and I’m not sure why.</p>
<p>A minimal google colab is here: <a href="https://colab.research.google.com/drive/1l61syPS7GO7O-S0o8cWnv8R0JwzVq58T?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1l61syPS7GO7O-S0o8cWnv8R0JwzVq58T?usp=sharing</a></p>

username: ptrblck
cooked: <p>You are rewrapping the input in a new tensor, which will break the computation graph and thus your input won’t get any gradients in this line of code:</p>
<pre><code class="lang-python">model.feature_extractor(torch.tensor(x.view(1, -1)))
</code></pre>
<p>Use this instead and it should work:</p>
<pre><code class="lang-python">model.feature_extractor((x.view(1, -1))
</code></pre>
<p>You should also get a <code>UserWarning</code> for the first use case:</p>
<pre><code class="lang-python">UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return model.feature_extractor(torch.tensor(x.view(1, -1)))
</code></pre>
Why would semantic segmentation results be blurry/fuzzy?

username: rzp0063
cooked: <p>Warm hello to everyone!</p>
<p>I am trying to implement the semantic segmentation using Resnet+Unet, for some reason my results are blurry. Any ideas of the source of the error?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/05177c37939d103cc34a7cc70ca10cd5bc58a0d5.png" alt="blurry_channel6" data-base62-sha1="J2HEZ5r8uZ288lKQzBnr7lFvsF" width="234" height="430"></p>
<p>I have an original colored image which i use K-Means to split into different colors, and then use these colors to make masks/channels.</p>
Min Operation On a Sparse Tensor / Variable Sizes Across Second Dimension

username: Caleb1
cooked: <p>Here’s the situation. I have a very large 2d tensor which is mostly sparse (like, &lt;0.1% density), everything else is filled with nan. I want to calculate the minimum value across the first dimension of the tensor: essentially torch.min(t, dim=1). But doing this for the entire shape of the tensor seems extremely wasteful when you’re only really looking for the min of 2-4 values out of tens of thousands.</p>
<p>I already need to grab the elements into an array for separate processing, which flattens the tensor into a 1d array, something similar to : a=~torch.isnan(b). I can also calculate how many real entries were in each column, so I’ll get another tensor which gives me how many elements I need to take the min of for each row: [3, 2, 4, 2, 4, 3, 2, …]. The expected value for all of these is roughly similar. I can convert this to index ranges by taking the cumulative sum [0, 3, 5, 9, 11…], so the first row of the original tensor is now stored in elements 0:3 of the new tensor, the second row from 3:5, third 5:9 etc…</p>
<p>After doing all of this, essentially what I’m looking to do is some parallel operation of torch.min where I feed in the ranges for each kernel to check through as a tensor. For example min_fancy(a, [0, 3, 5, 9, 11…]) would go through elements 0:3 and find the min and store it in row 0, go through 3:5 and store it in row 1, etc. I imagine this will involve writing a custom c++ extension, but I figured I should ask if there is any built in method I am missing or if anyone has any better ideas of how to do this, since I’m not the most familiar with writing pytorch extensions.</p>
Why is `id(tensors.storage())` different every time?

username: I-Love-U
cooked: <p>Demo:</p>
<pre><code class="lang-python">a = torch.randn(10, 10)
b = a[1:, :]
a.shape
Out[45]: torch.Size([10, 10])
b.shape
Out[46]: torch.Size([9, 10])
id(a)
Out[47]: 139889952764416
id(a)
Out[48]: 139889952764416
id(b)
Out[49]: 139889948003072
id(b)
Out[50]: 139889948003072
id(a.storage())
Out[51]: 139889948211136
id(a.storage())
Out[52]: 139889948120832
id(b.storage())
Out[53]: 139889952743616
id(b.storage())
Out[54]: 139892027430848
</code></pre>
<p>Why is <code>id(a.storage())</code> or <code>id(b.storage())</code>  different every time?</p>
<p>AND <strong>How can I tell if <code>a</code> and <code>b</code> here take up the same memory space?</strong></p>

username: SimonW
cooked: <p>the python storage object is construct when calling <code>.storage()</code>, so <code>id</code> is different.</p>
<p>test <code>x.storage().data_ptr()</code></p>
Batch_first = True in RNN. Question specific to a tutorial

username: nicofish
cooked: <p>I realize there are similar questions particularly this one: <a href="https://discuss.pytorch.org/t/could-someone-explain-batch-first-true-in-lstm/15402" class="inline-onebox">Could someone explain batch_first=True in LSTM</a></p>
<p>However my question pertains to a specific tutorial I found online: <a href="https://medium.com/dair-ai/building-rnns-is-fun-with-pytorch-and-google-colab-3903ea9a3a79" class="inline-onebox" rel="noopener nofollow ugc">Building RNNs is Fun with PyTorch and Google Colab | by elvis | DAIR.AI | Medium</a></p>
<p>I do not understand why they are not using batch_first=true.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f08261dde20e8a1f191bd04ee7f71e4853c6c91.png" data-download-href="https://discuss.pytorch.org/uploads/default/8f08261dde20e8a1f191bd04ee7f71e4853c6c91" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f08261dde20e8a1f191bd04ee7f71e4853c6c91_2_690x432.png" alt="image" data-base62-sha1="kpjJHMNxAh3HHF5d4nS3MWVyDtL" width="690" height="432" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f08261dde20e8a1f191bd04ee7f71e4853c6c91_2_690x432.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f08261dde20e8a1f191bd04ee7f71e4853c6c91_2_1035x648.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f08261dde20e8a1f191bd04ee7f71e4853c6c91.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1052×659 105 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I realized that the data being fed to the model is of the form [64, 28, 28].<br>
Because they are working with MNIST data and they specified a batch size of 64<br>
-64 is batch size<br>
-28 (number of rows) is seq_len<br>
-28 (number of cols) is features</p>
<p>Could someone please explain why they are not using batch_first=True?</p>
<p>Thank you very much for your help.<br>
edit: removed a part of question for simplicity</p>

username: ptrblck
cooked: <p>Most likely for performance reasons.<br>
The input will be permuted in the <code>forward</code> method via:</p>
<pre><code class="lang-python"> # transforms X to dimensions: n_steps X batch_size X n_inputs
X = X.permute(1, 0, 2) 
</code></pre>

username: nicofish
cooked: <p>Thank you. I had not seen permute call in the forward method.</p>
How to specify which optimizer to use for a loss

username: ChrisLiu2
cooked: <p>I have a NN with this type of structure:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b44bbb9999b87f113bd98b6b702ee3433ad85d5.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1b44bbb9999b87f113bd98b6b702ee3433ad85d5" title="WechatIMG9071"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b44bbb9999b87f113bd98b6b702ee3433ad85d5_2_499x375.jpeg" alt="WechatIMG9071" data-base62-sha1="3Te9qKqCXXSfQwh2GZSLk8WbM1f" width="499" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b44bbb9999b87f113bd98b6b702ee3433ad85d5_2_499x375.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b44bbb9999b87f113bd98b6b702ee3433ad85d5_2_748x562.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b44bbb9999b87f113bd98b6b702ee3433ad85d5_2_998x750.jpeg 2x" data-dominant-color="9F9991"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WechatIMG9071</span><span class="informations">4032×3024 1.18 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Where the L_est_A is a loss computed from A_init through L_estimate function, L_hat is the loss computed from random actions through L_estimate. L_estimate is a learned function.</p>
<p>In summary, what this net does is that it simultaneously learns an approximation of the loss of another network and also learns a conv_net that outputs some actions that minimizes the approximated loss, L_est_A.</p>
<p>My question is: I have separate optimizers for L_estimate and L_hat, and I would want L_est_A to only change the weight in conv_net when doing backward(), and L_hat only change the weight of L_estimate when doing backward(). But I didn’t find a way to specify which optimizer to use, for different losses. Any help is appreciated, thanks.</p>
Hello. Guys what are the functions we have for them in PyTorch? It would be a great help. Help required. Both are the lists of the folder contents with images+labels

username: Blizzard_boi2020
cooked: <pre><code class="lang-auto">dataset = tf.data.Dataset.from_tensor_slices((imagepath,labelpath))

    dataset = dataset.map(lambda image_path, labelpath: tuple(tf.py_func(
        self.input_parser, [imagepath, labelpath], [tf.float32,tf.int32])), num_parallel_calls=2)

</code></pre>

username: Blizzard_boi2020
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you please help?</p>

username: planet_pluto
cooked: <p>Are you working on semantic segmentation?</p>

username: Blizzard_boi2020
cooked: <p>Yes man! Can you kindly help me convert them?</p>

username: planet_pluto
cooked: <p>Assuming you have train_images, train_labels (segmentation masks) folders, in PyTorch you need to prepare a Dataset class and a Dataloader that feeds the data into your model for training. Here’s an example of a very simple dataset class and dataloader:</p>
<pre><code class="lang-auto"># The Dataset class definition:

import torch
import torchvision.transforms as T
import numpy as np
import os

from PIL import Image
from torch.utils.data import Dataset

class SegmentationDataset(Dataset):

    def __init__(self, images_dir, labels_dir):
        self.images = sorted(os.listdir(images_dir))
        self.labels = sorted(os.listdir(labels_dir))
        self.images_dir = images_dir
        self.labels_dir = labels_dir
        self.transform = T.Compose([
                T.ToTensor(),
            ])

    def __len__(self):
        return len(self.images)

    def __getitem__(self, i):

        img_path = self.images_dir + '/' + self.images[i]
        msk_path = self.labels_dir + '/' + self.labels[i]

        img = Image.open(img_path).convert('RGB')
        lbl = Image.open(msk_path).convert('L')

        img = img.resize((512, 512))
        lbl = lbl.resize((512, 512))

        image = self.transform(img)
        label = torch.from_numpy(np.array(lbl)).long()

        return image, label
</code></pre>
<p>The Dataloader:</p>
<pre><code class="lang-auto">train_dataset = SegmentationDataset('/path/to/train/images/', '/path/to/train/labels/')
train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=2, shuffle=True, num_workers=4)

# You could write a similar code for the validation
</code></pre>
<p>Model training:</p>
<pre><code class="lang-auto">for epoch in range(50):
    for idx, data in enumerate(train_dataloader):
        images, labels = data[0], data[1]
        # Rest of the training code
</code></pre>

username: Blizzard_boi2020
cooked: <p>Any suggestions on 3D dataset? I guess PIL will not work</p>

username: planet_pluto
cooked: <p>What is the format of your dataset? I think you should be able to make the necessary changes in the dataset class. Instead of loading the images/labels from the directory, load the 3d dataset (<code>np.array()</code> or <code>.nii</code>) and then crop the no. of slices accordingly depending upon your memory.</p>
Manual implementation of SGD

username: Muhammad_Usman_Qadee
cooked: <p>I have defined model using nn.module. I need to use grads() function instead of backward(). But unfortunaltely I am not getting the right results. The model is not converging. Here is snippet of my code for SGD implemenation. Is there any problem here?</p>
<pre><code class="lang-auto"> grads = grad(loss, model.parameters())
 with torch.no_grad():
    for (W, g) in zip(model.parameters(), grads):
          W.data -= step_size*g

</code></pre>
Hard negative mining for every class?

username: YASJAY
cooked: <p>Hi,</p>
<p>If anyone has experience in experimenting with hard negative mining in say object detection, I need some insight on its implementation:</p>
<ol>
<li>Should it be applied on background only?</li>
<li>Can it be applied on every object class? If yes, how do we decide the top-k threshold for every epoch?</li>
</ol>
Computing Backward one layer at a time

username: maralm
cooked: <p>Hi,<br>
I have a neural network and want to calculate the backward pass one layer at a time. Let’s say this is the network:</p>
<pre><code class="lang-auto">class DeepNN(nn.Module):
    def __init__(self):
        super(DeepNN, self).__init__()
        self.fc_1 = nn.Linear(D_in, H)
        self.fc = nn.Linear (H,H)
        self.fc_end = nn.Linear(H,D_out)

        self.module_list = []

        self.module_list.append(self.fc_1)
        for i in range(num_hidden):
            self.module_list.append(nn.Linear (H,H))

        self.layers = nn.ModuleList(self.module_list)
        self.f = nn.Sequential(*self.module_list)
        self._name = "DeepNN"
        
    
    def forward (self, x):
 
        return self.f(x)
</code></pre>
<p>Now, instead of loss.backward(), I want to calculate gradients layer by layer and change gradients of each layer and pass it to the other layer.<br>
Is there a proper way to do that?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I would see two ways:</p>
<ul>
<li>Return the output of each layer then use <code>autograd.grad</code> to backward only the part of the network that you want.</li>
<li>Use <code>register_hook()</code> on the output of each layer and have the hook function do the calculation you want and return the new gradient value from the hook.</li>
</ul>

username: mrqorib
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a> , thank you for your explanation.</p>
<p>I have read your explanation and the Autograd documentation page, but sorry I still don’t understand how to do it. Would you mind giving a small example of how to do it both methods you explained? Thank you very much for your help.</p>
<p>Warm regards,<br>
Reza Qorib</p>
When adding the new loss, the norm of gradient will increase

username: Kirito000
cooked: <p>When I add the loss weight or some new losses (keeping the original losses unchanged) to a model, I found the norm of gradient of the model will increase. For example, use the following code:</p>
<pre><code class="lang-auto">total_norm = 0.0
for p in net.parameters():
    param_norm = p.grad.data.norm(2)
    total_norm += param_norm.item() ** 2
    total_norm = total_norm ** (1. / 2)
print(total_norm)
</code></pre>
<p>I wonder if it will affect the model training stability or performance. Should I decrease the learning rate or scale the loss weight to match the original gradient norm? Does anyone can help clarify the relationship between gradient/norm of gradient vs learning rate vs step size? Thanks a lot!</p>

username: googlebot
cooked: <p>Gradient norm is related to current goodness of fit, optimizer adjustments may not be needed, however parameter sharing during multi-task training may sometimes by problematic by itself.</p>

username: Kirito000
cooked: <aside class="quote no-group" data-username="googlebot" data-post="2" data-topic="100407" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/ba8739/48.png" class="avatar"> googlebot:</div>
<blockquote>
<p>Gradient norm is related to current goodness of fit, optimizer adjustments may not be needed, however parameter sharing during multi-task training may sometimes by problematic by itself.</p>
</blockquote>
</aside>
<p>Thanks for your reply! But I’m not sure about the meaning of “parameter sharing during multi-task training”. Here, I just add some new losses or increase the loss weight in a single-task model. The gradient norm increases. I’m not sure about the effect and the underlying meaning of gradient norm.</p>

username: googlebot
cooked: <p>two losses = two objectives = two probability distributions being modeled = two “tasks”</p>
<p>It is usually fine if network capacity is big enough, tasks are related (“shared” feature space representations are possible) and value magnitudes are similar.</p>
<aside class="quote no-group" data-username="Kirito000" data-post="3" data-topic="100407">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/f0a364/48.png" class="avatar"> Kirito000:</div>
<blockquote>
<p>The gradient norm increases. I’m not sure about the effect and the underlying meaning of gradient norm.</p>
</blockquote>
</aside>
<p>Your network is not yet trained for the introduced objective, so gradients increase.</p>
Ceil Mode in pooling

username: Nilakshan_Kunanantha
cooked: <p>For the following parameter setting:<br>
input size 112x112<br>
kernel 3x3<br>
stride 2<br>
padding = 0<br>
Ceil Mode =True<br>
The output size is 56x56.</p>
<p>If the ceil mode is true, will there be any one-sided padding for input image at right and bottom before pooling?.<br>
With 108th column of input, outsize will be 55, in order to get 56 output size we will move to 110th column and since filter size is 3 we need an extra column. So there should be single-column padding, right?</p>

username: Pranavan_Theivendira
cooked: <aside class="quote no-group" data-username="Nilakshan_Kunanantha" data-post="1" data-topic="54646">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nilakshan_kunanantha/48/15470_2.png" class="avatar"> Nilakshan_Kunanantha:</div>
<blockquote>
<p>56x56</p>
</blockquote>
</aside>
<p>Hi,</p>
<p>Usually the output size is calculated as follow.</p>
<pre><code class="lang-auto">O = floor((W-K+2P)/S) + 1
</code></pre>
<p>If the ceil mode is on, ceil used instead of floor operation in the output. That is what said in the documentation. You can refer here.<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/pooling.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/_modules/torch/nn/modules/pooling.html</a></p>
<p>In the case of ceil mode, additional columns and rows are added at the right as well as at the down. (Not top and not left). It does not need to be one extra column. It depends on the stride value as well. I just wrote small code snippet where you can check how the populated values are pooled in either modes.</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch

test = torch.randn(2,3,32,32)
x = torch.tensor([[-2, 1, 2, 6, 4], [-3, 1, 7, 2, -2], [-4, 2, 3, -1 , -3], [-7, 1, 2, 3, 11], [5, -7, 8, 12, -9]]).float()
x = x.unsqueeze(0)
y_1 = nn.MaxPool2d(kernel_size=2,stride=2, padding=0)
y_2 = nn.MaxPool2d(kernel_size=2,stride=2, padding=0, ceil_mode=True)
print(y_1(x))
print(y_2(x))
</code></pre>
<p>Thanks</p>

username: Nilakshan_Kunanantha
cooked: <p>Thanks,clear now  <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>

username: LuKa_Chen
cooked: <p>In this doc<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html" class="inline-onebox" rel="noopener nofollow ugc">MaxPool2d — PyTorch 2.1 documentation</a>[torch nn MaxPool2D], why the output size is calculated differently<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7e543325bd11a78dc5ff8f2df3d8cf1b5a91f72d.png" alt="image" data-base62-sha1="i1yyCTgombwLChmWZPQ4nnKuzZb" width="639" height="137"></p>
RuntimeError: HIP out of memory

username: apalazzi
cooked: <p>Hi,</p>
<p>I’m using pytorch with an AMD card and rocm; I can train my model but when I try to detect something with it I run into an out of memory error:</p>
<pre><code class="lang-auto">RuntimeError: HIP out of memory. Tried to allocate 138.00 MiB (GPU 0; 7.98 GiB total capacity; 1.55 GiB already allocated; 6.41 GiB free; 1.57 GiB reserved in total by PyTorch)
</code></pre>
<p>It seems to me however that there is memory available, so why it fails to allocate the memory?<br>
Is it a rocm bug?</p>
<p>Thanks,<br>
Andrea</p>

username: apalazzi
cooked: <p>I rebooted the system and now it works, I guess there was some core component that crashed and the system was in an unclean state.</p>
<p>Bye<br>
Andrea</p>
Score function in RoI module at test time (Faster/Mask R-CNN) is not class-independent

username: sigma_x
cooked: <p>If there are C classes, each RoI outputs C score and bounding box predictions (two tensors, size <code>(1, C)</code> and <code>(2,C)</code> resp.) at test stage (<code>postprocess_detections</code> method). Non-max suppression is done independently of the class (i.e. boxes overlapping more than NMS are kept if they are different classes). But the normalization function is not class-independent:<br>
<code>pred_scores = F.softmax(class_logits, -1)</code><br>
So if there are two positive classes, <code>pred_scores</code> vector will be, e.g.<code> [0.9, 0.1]</code>, and at some point both of these scores will be compared to <code>box_score_thresh</code>. Obviously one of them is very likely to be rejected. Therefore, I don’t quite understand this implementation. It should be either:</p>
<pre><code class="lang-auto">pred_scores = F.sigmoid(class_logits, -1)
preds =  torch.nonzero(pred_scores.sigmoid()&gt;box_score_thresh)
</code></pre>
<p>to compute the scores independently, or</p>
<pre><code class="lang-auto">preds = class_logits.max(-1)
preds.values[preds.indices&gt;0].sigmoid()&gt;box_score_thresh 
</code></pre>
<p>to extract the best prediction from every RoI. Then the predictions will be genuinly independent. I think it needs to be re-implemented or at least added as an argument to choose from. Mask predictions are done independently in this way.</p>
How to implement a single-feature Encoder with Adaptive Pooling Layer for flexible finetuning?

username: mariekiii
cooked: <p>Hi there,</p>
<p>I am new to Pytorch with very little understanding of programming classes. While everything worked out so far, I don’t understand the following problem now:</p>
<p>In oder to allow my feed-forward network to process data with different input feature size, I was trying to implement an encoder, that</p>
<ol>
<li>processes each feature in  tensor of shape [730, 10] seperately, so a tensor of shape [730,1] with hidden size 64 to [730,64].</li>
<li>concatenates all of them together to [730,64,10]</li>
<li>activates it.<br>
Because I want to have the number of input features variable, I now apply adaptive pooling with arbitrary output size to this tensor.</li>
</ol>
<p>While the network is working on the first view, the loss that it outputs is:</p>
<p>tensor(nan, grad_fn=)</p>
<p>Am I missing something in the class implementation, such that I can’t do proper backpropagation?<br>
I would really appreciate any help on this!</p>
<p>The class I created looks like this:</p>
<p>class MLPmod(nn.Module):</p>
<pre><code>def __init__(self, hidden_features, dimensions, activation):
    
    super(MLPmod, self).__init__()
    self.hidden_features = hidden_features
    self.activation = activation()
    
    self.encoder = nn.Linear(1, hidden_features)
    self.avgpool = nn.AdaptiveAvgPool1d(hidden_features)
    self.classifier = self.mlp(dimensions, activation)
    
def forward(self, x):
    
    x = self.encode(x)
    x = self.avgpool(x).view(x.shape[0],-1)
    x = self.classifier(x)
    
    return(x)

def mlp(self, dimensions, activation):

    network = nn.Sequential()
    network.add_module(f"hidden0", nn.Linear(self.hidden_features*self.hidden_features, dimensions[0]))
    network.add_module(f'activation0', activation())

    for i in range(len(dimensions)-1):
        network.add_module(f'hidden{i+1}', nn.Linear(dimensions[i], dimensions[i+1]))
        if i &lt; len(dimensions)-2:
            network.add_module(f'activation{i+1}', activation())

    return(network)

def encode(self, x):
    
    x = x.unsqueeze(1)
    
    latent = torch.empty(x.shape[0], self.hidden_features, 1)
    
    for feature in range(x.shape[-1]):
        latent = torch.cat((latent, self.encoder(x[:,:,feature]).unsqueeze(2)),dim=2)
        
    latent = self.activation(latent)
    
    return(latent)</code></pre>

username: ptrblck
cooked: <p>In these lines of code:</p>
<pre><code class="lang-python">latent = torch.empty(x.shape[0], self.hidden_features, 1)
    
for feature in range(x.shape[-1]):
    latent = torch.cat((latent, self.encoder(x[:,:,feature]).unsqueeze(2)),dim=2)
</code></pre>
<p>you are appending an empty and thus uninitialized tensor to itself. Since you are not initializing the values of the tensor, it might contain invalid values such as Infs/NaNs etc.<br>
I would recommend to append the output of the <code>encoder</code> to a list and use <code>torch.stack</code> afterwards.</p>

username: mariekiii
cooked: <p>Thanks a lot! That fixed the issue.</p>
Reproducibility breaks down with weighted Cross Entropy loss

username: arsenzaryan
cooked: <p>Hello, the following code ceases to be reproducible when the weights in cross entropy are non-integers. Here’s the example:</p>
<pre><code class="lang-auto">import numpy as np
from collections import Counter

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader


h, w, in_ch, out_ch = 32, 32, 3, 5
class Dtst(Dataset):
    def __init__(self, N=20):
        self.X = [torch.randn([in_ch, h, w], dtype=torch.float32) for _ in range(N)]
        self.Y = [torch.randint(low=0, high=out_ch, size=(h,w), dtype=torch.int64) for _ in range(N)]
        
    def __getitem__(self, ix):
        return self.X[ix], self.Y[ix]
    
    def __len__(self):
        return len(self.Y)


class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Conv2d(in_channels=in_ch, out_channels=10, kernel_size=3, padding=1)
        self.drop = nn.Dropout2d(p=0.1)
        self.layer2 = nn.Conv2d(in_channels=10, out_channels=out_ch, kernel_size=3, padding=1)

    def forward(self, x):
        out = self.layer2(self.drop(self.layer1(x)))
        return out

seed = 4
torch.backends.cudnn.enabled = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = True
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
np.random.seed(seed)

dtst = Dtst()
model = Network()

device = 'cuda'
model.to(device)
class_weights = ((torch.arange(out_ch)+1).type(torch.FloatTensor)**0.5).to(device)

loss_fn = torch.nn.CrossEntropyLoss(weight=class_weights)
opt = torch.optim.Adam(model.parameters())

preds_dict = dict()
for e in range(1500):
    dtldr = DataLoader(dtst, batch_size=4)
    for x,y in dtldr:
        preds = model(x.to(device))
        loss = loss_fn(preds, y.to(device))
        loss.backward()
        opt.step()

        preds_argmax = preds.argmax(dim=1).flatten()
        preds_dict.update(Counter(preds_argmax.tolist()))

print(sorted(preds_dict.items(), key=lambda x: x[1]))
print(model.layer1.weight.data.norm(2).item())
</code></pre>
<p>It’s a very simple network with a very basic Dataset, and a simple train loop.<br>
This code is not reproducible. But when I remove the (**0.5) part from the class_weights it becomes reproducible. I.e., if the class weight values are actual floats, not integers cast to floats, then the code is not reproducible.</p>
<p>Also, the problem exists only on cuda. If the device is set to ‘cpu’, the code is reproducible again.<br>
I run this on Ubuntu 18. My environment is the following:<br>
pytorch 1.6.0<br>
cudatoolkit 10.1.243<br>
numpy 1.19.1</p>

username: arsenzaryan
cooked: <p>I’ve added this same issue on github as well, seems like there it is getting more attention. Here’s the link <a href="https://github.com/pytorch/pytorch/issues/46024" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/46024</a></p>
DeepLabV3 layers size

username: Avi
cooked: <p>Hey everyone,</p>
<p>I tried to use DeepLabV3+ head for segmentation, as a part of a new architecture model, and I couldn’t train it. while debugging I found out that there is an problem that seems to be inherent in this head class.</p>
<p>the head contains 4 ASSPconv classes, and one ASSPpool classes. the last one contains the following layers:</p>
<pre><code class="lang-auto">class ASPPPooling(nn.Sequential):
    def __init__(self, in_channels, out_channels):
        super(ASPPPooling, self).__init__(
            nn.veAdaptiAvgPool2d(1),
            nn.Conv2d(in_channels, out_channels, 1, bias=False),
            nn.BatchNorm2d(out_channels),
            nn.ReLU())

    def forward(self, x):
        size = x.shape[-2:]
        for mod in self:
            x = mod(x)
        return F.interpolate(x, size=size, mode='bilinear', align_corners=False)
</code></pre>
<p>the bug is hiding in the BN layer, which couldn’t get 1*1 shape tensor in “train mode”. but the first AdaptiveAvgPool2d layer is forcing that shape.</p>
<p>I appologize about the weak English,<br>
and I would aprritiate your help!</p>
<p>Avi</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Avi" data-post="1" data-topic="100380">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/b9bd4f/48.png" class="avatar"> Avi:</div>
<blockquote>
<p>the bug is hiding in the BN layer, which couldn’t get 1*1 shape tensor in “train mode”. but the first AdaptiveAvgPool2d layer is forcing that shape.</p>
</blockquote>
</aside>
<p>I don’t think this is necessarily an error, but might be a design decision.<br>
E.g. the <code>inception_v3</code> model also doesn’t accept single samples in <code>train()</code> mode for the same reason and fails in a batchnorm layer as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python"># setup
model = models.inception_v3(init_weights=False)

x = torch.randn(1, 3, 299, 299)
out = model(x) # error in batchnorm layer
&gt; ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 768, 1, 1])

x = torch.randn(2, 3, 299, 299)
out = model(x) # works
</code></pre>
How to apply augmentation to dataset

username: dlam0002
cooked: <p>I’d like to augment the training and validation dataset that I currently have. I’m not sure whereabouts to put this code in the main code:</p>
<pre><code class="lang-auto">transforms.Compose([
transforms.Resize((229,229)),
transforms.RandomResizedCrop((229,229)),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize(mean=[unknown],
std=[unknown])
])

</code></pre>
<p>MAIN CODE:</p>
<pre><code class="lang-auto">class STLData(Dataset):
    def __init__(self,trn_val_tst = 0, transform=None):
        data = np.load('hw3.npz')
        if trn_val_tst == 0:
            #trainloader
            self.images = data['arr_0']
            self.labels = data['arr_1']
        elif trn_val_tst == 1:
            #valloader
            self.images = data['arr_2']
            self.labels = data['arr_3']
        else:
            #testloader
            self.images = data['arr_4']
            self.labels = data['arr_5']
            
        self.images = np.float32(self.images)/1.0
        self.transform = transform

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
   
        sample = self.images[idx,:]
        labels = self.labels[idx]
        if self.transform:
            sample = self.transform(sample)
        return sample, labels

train_set = STLData(trn_val_tst=0, transform=torchvision.transforms.ToTensor()) 
val_set = STLData(trn_val_tst=1, transform=torchvision.transforms.ToTensor()) 
test_set = STLData(trn_val_tst=2, transform=torchvision.transforms.ToTensor()) 

batch_size = 100 
n_workers = multiprocessing.cpu_count()
trainloader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=n_workers)
valloader = torch.utils.data.DataLoader(val_set, batch_size=batch_size, shuffle=True, num_workers=n_workers)
testloader = torch.utils.data.DataLoader(test_set, batch_size=batch_size, shuffle=True, num_workers=n_workers)
</code></pre>

username: ptrblck
cooked: <p>You could pass the <code>transforms.Compose</code> as the <code>transform</code> argument to the <code>STLData</code>, if you would like to use it for this dataset.<br>
Note that <code>torchvision.transforms</code> work on <code>PIL.Image</code>s by default (in the nightly more transformation can also be applied to tensors directly) so you might need to transform the numpy arrays to <code>PIL.Image</code>s first.</p>
Extracting features from boxes in the pretrained Faster Rcnn

username: anshu0612
cooked: <p>How to extract features from the predicted bounding boxes in the pretrained Faster Rcnn in pytorch?</p>

username: ptrblck
cooked: <p>Which features would you like to extract?<br>
Generally you can use forward hooks as explained <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a> to get intermediate activations (features).</p>
Indexing a tensor from with a different mask size mask

username: Karan_Chhabra
cooked: <p>Hi,<br>
I have a tensor of size [24,14, 14, 30] ([Batch_size, window_H, window_W, predictions])  and the mask of size [24, 14, 14].</p>
<p>I want only the <strong>predictions</strong> for the <strong>windows</strong> where the mask is 1.<br>
So, my output vector could be of size [24,14, 14, 30] or just [30].</p>
<p>But I am unable to index using mask.</p>
<pre><code class="lang-auto">target_tensor.shape 
[24, 14, 14, 30]

contains_object_mask.shape
[24, 14, 14]

Result_tensor.shape
[24, 14, 14, 30]
or
[30]
</code></pre>
<p>Thank You</p>