akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CUDA runs out of memory after some epochs

username: Chame_call
cooked: <p>I’m using third party project for training age-gender predicting model.</p>
<p><a href="https://ideone.com/1HVB2h" rel="nofollow noopener">That’s the problem script</a>.<br>
The problem is with every next epoch the memory occupied by the script increased.<br>
The memory usage I observe via <code>nvidia-smi</code>.</p>
<p>I can decrease my batch-size and the training cal last 2 epochs but then cuda runs out of memory again.<br>
I suppose there’s a memory allocation in a wrong place or something like that.<br>
Maybe someone can find such detail in a quick look.<br>
any help would be appreciated.</p>

username: Chame_call
cooked: <p>That’s my error traceback:</p>
<pre><code class="lang-auto">File "train.py", line 641, in &lt;module&gt;
    a.train_model()
  File "train.py", line 294, in train_model
    gender_out, age_out = self.model(inputs)
  File "/home/algernone/anaconda3/envs/scienv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/algernone/DNNS/age-gender-pytorch/Age-Gender-Pred/agegenpredmodel.py", line 140, in forward
    last1 = self.get_resnet_convs_out(x)
  File "/home/algernone/DNNS/age-gender-pytorch/Age-Gender-Pred/agegenpredmodel.py", line 83, in get_resnet_convs_out
    x = self.resNet.layer1(x)   # out = [N, 64, 56, 56]
  File "/home/algernone/anaconda3/envs/scienv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/algernone/anaconda3/envs/scienv/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/algernone/anaconda3/envs/scienv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/algernone/anaconda3/envs/scienv/lib/python3.6/site-packages/torchvision/models/resnet.py", line 65, in forward
    out = self.bn2(out)
  File "/home/algernone/anaconda3/envs/scienv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/algernone/anaconda3/envs/scienv/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 107, in forward
    exponential_average_factor, self.eps)
  File "/home/algernone/anaconda3/envs/scienv/lib/python3.6/site-packages/torch/nn/functional.py", line 1670, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: CUDA out of memory. Tried to allocate 196.00 MiB (GPU 0; 5.77 GiB total capacity; 3.79 GiB already allocated; 185.38 MiB free; 4.66 GiB reserved in total by PyTorch)

</code></pre>
<p>that happens after first epoch</p>
Shall I pass hidden states to LSTM or not?

username: Balazs_Koncz
cooked: <p>Hi, I have a couple of weather related data like tempreature etc… in time series format.</p>
<p>Now I would like to experiment with regression and classification tasks. (Like predicting tomorrow’s weather,or just simply classifying day as sunny/rainy/cloud …)</p>
<p>For this task I have an LSTM network but I don’t know I shall retain the hidden states or reset them periodically.</p>
<p>Technically speaking:<br>
1.) Shall I invoke the recurrent layer without passing hiden states like this:</p>
<pre><code class="lang-auto">lstm_output, _ = lstm(inputs)
</code></pre>
<p>2.) Or save the hidden states somewhere and reset them at every minibatch:</p>
<pre><code class="lang-auto">lstm_output, new_hidden_states = self.lstm(inputs, hidden_states)
</code></pre>
<p>Any advice would be appreciated! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: bigyeet
cooked: <p>if each datapoint is already in timeseries form, the gradients are accumulated through time automatically, and you don’t need to save last batch’s hidden states. Also, since sequences between each batch are not continuous, keeping the hidden states will lead to the LSTM learning the jumps from sequence to sequence <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

username: G.M
cooked: <p>If u don’t pass the hidden states to the lstm, the hidden states will be initialized to zeros, so usually u don’t have to worry about the hidden states.</p>

username: Balazs_Koncz
cooked: <p>Thanks for the comments!</p>
<p>Also, I have to mention that the batches are sampled randomly. So In side the batch the data is sequential. But the next batch does not contains the next data points.</p>
<p>Does this make a difference?</p>

username: G.M
cooked: <p>I don’t think so. For lstm with <code>batch_first=False</code>, the input should be of shape [L, B, H]. For lstm with <code>batch_first=True</code>, the input should be  of shape [B, L, H].</p>
Handle video tensor

username: anderse
cooked: <p>Hi!<br>
I’m struggling with a video tensor in pytorch. I want to make some filtering on it for preparation.  I’m uncertain which shape I should use for the video. Currently I use shape: [frames, colors, height, width]<br>
e.g. rgb: [50, 3, 1080, 1920] or for gray [50, 1, 1080, 1920]</p>
<p>To prepare the video I would like to do some filtering/convolution.<br>
I would like to do some 2d-convolution separate for each frame and color channel.<br>
I tried torch.nn.functional Conv3d and Conv2d and different weights.<br>
If I try conv2d(video, filter, stride=1, bias=None,  padding=1)<br>
video having shape gray [50, 1, 1080, 1920]<br>
filter: [[ 0.0000,  0.5000,  0.0000],<br>
[ 0.0000, -1.0000,  0.0000],<br>
[ 0.0000,  0.5000,  0.0000]]<br>
And get.RuntimeError: weight should have at least three dimensions<br>
If i unsqeese(filter,0) I get:<br>
RuntimeError: expected stride to be a single integer value or a list of 1 values to match the convolution dimensions, but got stride=[1, 1]<br>
Could anyone show me how I should do it?<br>
Thanks!<br>
Best regards<br>
Anders</p>

username: anderse
cooked: <p>I used this as a solution:<br>
import torch<br>
import torch.nn.functional as F</p>
<p>x= torch.randn([10, 3, 4, 4], dtype=torch.float32)<br>
xsize= x.shape<br>
xx= x.view(xsize[0]*xsize[1], 1, xsize[2], xsize[3])</p>
<p>f= torch.tensor([[0,0.5,0],[0,-1,0],[0,0.5,0]], dtype=torch.float32)<br>
f=f.view(1,1,3,3)</p>
<p>y= F.conv2d(xx, f, bias= None, padding=1, stride=1)</p>
Threading vs multiprocessing for saturating gpu(s)

username: urw7rs
cooked: <p>Let’s say that you’re not doing any preprocessing to the data and you have multiple GPUs. In this case would using multiprocessing have a big advantage over threading? And would it make sense to use multiprocessing or multithreading for training GANs when discriminator and generator can be run in parallel?(e.g. discriminator calculating real loss and generator generating fake image)</p>
How to add my own bias to conv2d

username: Heshan_Anupama
cooked: <p>I want to add a conv2d layer and add my own bias instead of making it to automatically add the bias. I am already passing bias=false when I create the model. The bias that I have is a float torch vector.</p>
<pre><code class="lang-auto">m = nn.Conv2d(3, 2, (3, 3), bias=False))
</code></pre>

username: G.M
cooked: <p>Make your bias a <code>nn.Parameter</code> and manually add it in <code>forward</code>.</p>
On Dropout on train and eval mode

username: YinYang_Untalan
cooked: <p>if i call .eval() on my model that had dropout layers… does pytorch automatically scale the weights? and calling .train() will rescale accordingly as well?</p>

username: ptrblck
cooked: <p>Yes, the activations will be scaled automatically. In face this is done during training, while <code>model.eval()</code> just uses the model as it is.</p>

username: YinYang_Untalan
cooked: <p>thank you very much for this</p>
On Convolutions

username: YinYang_Untalan
cooked: <p>Say I have a 3x3 kernel for a conv. A normal conv2d would multiply the same 3x3 numbers to all the channels and add them up. What if, say i have 3 channels as input and instead of using the same 3x3 numbers for all the channels, i have different numbers effectively 3(channel) x 3 x 3 (kernel size). How would I do this? Thank you.</p>

username: ptrblck
cooked: <p>The second use case is actually a vanilla convolution and you can check the kernel shape via:</p>
<pre><code class="lang-python">print(nn.Conv2d(3, 1, 3).weight.shape)
&gt; torch.Size([1, 3, 3, 3]) # out_channels, in_channels, height, width
</code></pre>
<p>Have a look at <a href="http://cs231n.github.io/convolutional-networks/#conv">CS231n - Convolution layers</a>, which describes the underlying method quite well.</p>

username: YinYang_Untalan
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="72404">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>print(nn.Conv2d(3, 1, 3).weight.shape)</p>
</blockquote>
</aside>
<p>Ohhhh. okay thanks. I didn’t know sorry.</p>
About ignore_index

username: shuairenqin1
cooked: <p>When setting the label index to -100 (the same as the default ignore_index for NLLLoss), which is meaningless, the error occurs (RuntimeError: CUDA error: device-side assert triggered). But when setting both the ignore_index of NLLLoss and the meaningless label index to 255, no error is reported. I am completely confused.  What happens? Who can tell me. Thanks a lot.</p>

username: ptrblck
cooked: <p>Could you post a minimal code snippet to reproduce this issue, as this dummy code works fine:</p>
<pre><code class="lang-python">output = torch.randn(10, 10, requires_grad=True)
target = torch.randint(0, 10, (10,))
target[0] = -100

output = output.to('cuda')
target = target.to('cuda')

criterion = nn.CrossEntropyLoss()

loss = criterion(output, target)
loss.backward()
</code></pre>
[BERT] Implementation of the sliding window for long sequences

username: wasiahmad
cooked: <p>I was trying to implement the sliding window trick for BERT so that we can process long sequences. I am trying to implement it in plain PyTorch. I am unable to implement in Batches without running any loops. Basically, what is bothering me is how can we split a long sequence and then after getting the embeddings, how do we unpack them without running loops? If someone is familiar with the process, is it possible to describe the trick?</p>
<p>More detail about the issue - <a href="https://github.com/google-research/bert/issues/66" rel="nofollow noopener">https://github.com/google-research/bert/issues/66</a></p>
<p>Any help would be appreciated.</p>
Slow training on each step of epoch when folder has a lot of images

username: farhanrahman
cooked: <p>Hey,</p>
<p>So I noticed something when I was training a U-Net on my own data. I have 7000 images on one folder and 7000 ground truth tensors on another. I am using num_workers = 8. lets say if i am using a batch size of 4, I notice a pause in the epoch when its goes 32 steps. I notice another pause after 64. Each of these pauses make my training very slow.</p>
<p>Interestingly if I make my dataset way smaller, say I keep 100 images on the dataset folder the training gets a lot fast. Say for example training with 100 images would take 10 secs for a single epoch. So you would think if the training set had 7000 images, it would take somewhere around ~ (10 x 7000)/100 seconds which is roughly 12 mins. Whereas in reality it requires around 35 mins to complete the epoch.</p>
<p>Any insight or solution for this would be helpful. Thanks!</p>
Inception v3 taking too long to load

username: vishalagarwal
cooked: <p>Inception v3 is taking a lot of time to load compared to other torchvision models. Is there any bug?<br>
using torchvision 0.4.2</p>

username: ptrblck
cooked: <p>Yes, it’s a known <a href="https://discuss.pytorch.org/t/torchvisions-inception-v3-takes-much-longer-to-load-than-other-models/68756/6">bug</a> and here is the corresponding <a href="https://github.com/pytorch/vision/issues/1797">bug</a>.<br>
You should be able to pass <code>init_weights=False</code> in the nightly binaries to bypass the scipy issue.</p>
I need help to convert the following eq to tensor

username: nile649
cooked: <p>x is an image.<br>
R = x[0,0,:,:]<br>
G = x[0,1,:,:]<br>
B = x[0,2,:,:]</p>
<p>v_p = torch.tensor([R, G, B, torch.pow(R,2), torch.pow(G,2), torch.pow(B,2), R * G, G * B, B * R, 1.0])</p>
<p>The problem is many of the elements have different dtype.</p>
<blockquote>
<p>The following is the error,</p>
<p>ValueError                                Traceback (most recent call last)<br>
 in ()<br>
----&gt; 1 v_p = model.get_param(img)<br>
2 v_p.shape</p>
<p> in get_param(x)<br>
—&gt; 67         v_p = torch.tensor([R, G, B, torch.pow(R,2), torch.pow(G,2), torch.pow(B,2), R * G, G * B, B * R, 1.0])<br>
68         return v_p.reshape(1,10)<br>
69</p>
<p>ValueError: only one element tensors can be converted to Python scalars</p>
</blockquote>
<p>I don’t want to convert to numpy*</p>

username: ptrblck
cooked: <p>For different data types, you might need to store the tensors in a list, as I don’t think there is a workaround currently.<br>
Numpy will convert the arrays in the same way PyTorch does:</p>
<pre><code class="lang-python">tmp = np.array([np.zeros(1).astype(np.float32), np.zeros(1).astype(np.float64)])
print(tmp.dtype)
&gt; float64
print(tmp[0].dtype)
&gt; float64

tmp = torch.tensor([torch.randn(1), torch.randn(1).double()])
print(tmp.dtype)
&gt; torch.float64
</code></pre>
<p>The error might point instead to a shape mismatch.</p>
Why is the dataloder didn't load patches?

username: Quintia_Zhao
cooked: <p>hi, I’m trying to make patch (32,32) for every step 4 from a (256,256) image, it means that the patches are overlapped, I’ve got success when not feed the image to the Dataloader, but when I use Datalodaer to load the image, it shows, the function not work at all. the length of mr_patches return to 0.<br>
I don’t know why,  could anyone help me?</p>
<p>Any help would be greatly appreciated.<br>
here is my code to get patches:</p>
<pre><code class="lang-auto">def ext_patches_2d_ovl(imgs):
    patch_h=32
    patch_w=32
    stride_h=4  
    stride_w=4  
    p = []
    img_h = imgs.shape[1]  #height of the full image
    img_w = imgs.shape[0] #width of the full image
    i = 0
    for w in range((img_w-patch_w)//stride_w+1): #extract by row
        for h in range((img_h-patch_h)//stride_h+1):
            patch = imgs[(w*stride_w) : (w*stride_w + patch_w), (h*stride_h):(h*stride_h + patch_h)]
            patches= patch
            i +=1   #total
            p.append(patches)
    # print(p[0].shape)
    return p


test_mri = os.path.join(test_dir, "mri_data")
for root, dir, file in os.walk(test_mri):
    for name in file:
        test_data = MyDatasetPaired2d(data_dir=os.path.join(root,name))
        test_loader = DataLoader(dataset=test_data, batch_size=1)
        with torch.no_grad():
            for iter, input_mr_slices in enumerate(test_loader):   
                mr_patches = ext_patches_2d_ovl(input_mr_slices)
                print(len(mr_patches))

</code></pre>
<pre><code class="lang-auto">class MyDatasetPaired2d(Dataset):

    def __init__(self, data_dir, transform=None):  #???
        self.data_info_mri= self.get_img_info(data_dir)
        self.transform = transform

    def __getitem__(self, index): # ??index ????paired??
        mri_img = self.data_info_mri[index,:,:]
        mri_img = mri_img[np.newaxis,:,:]
        return mri_img
    def __len__(self):
        print(len(self.data_info_mri))
        return len(self.data_info_mri)
    @staticmethod
    def get_img_info(data_dir):
        mri_data = read_img(data_dir)
        return mri_data</code></pre>

username: ptrblck
cooked: <p>Your <code>DataLoader</code> will return a batch of samples and thus add a batch dimension to it.<br>
The height and width calculation in <code>ext_patches_2d_ovl</code> will thus be wrong:</p>
<pre><code class="lang-python">img_h = imgs.shape[1]  #height of the full image
img_w = imgs.shape[0] #width of the full image
</code></pre>
<p>and you might need to change it to index2 and index1.</p>

username: Quintia_Zhao
cooked: <p>thanks soooo much!! it works! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: Noob
cooked: <p>Hi ptrblck, can you help me? I post a new question for the multi-label classification.<br>
I try it but really hard to understand what is my problem</p>
Low mAP in multi-label classification

username: Noob
cooked: <p>Hi, everyone.<br>
I got a problem in multi-label classification.<br>
I have 20 numbers of classifications. For each picture, we may find some different categories at same time. For example, [0,1,1,0…]  means in this picture, we can find dog and cat.<br>
However, when I use mAP to measure my model, I only get very very low mAP. However, I don’t know what is going on and what is wrong with my code.</p>
<p>Here is my code：<br>
Loss function is nn.MultiLabelSoftMarginLoss().<br>
And after using logits = classifier(images.to(device)),  I use sigmoid and make thresholds.<br>
Can anyone help me why I get very low accuracy?</p>
<p>classifier = resnet101().to(device)<br>
optimizer = torch.optim.Adam(classifier.parameters(), lr= args.lr ,weight_decay=5e-4)<br>
criterion = nn.MultiLabelSoftMarginLoss()<br>
NUM_EPOCHS = 100<br>
TEST_FREQUENCY = 5<br>
for epoch in range(1, NUM_EPOCHS+1):<br>
print("Starting epoch number " + str(epoch))<br>
train_loss = train_classifier(train_loader, classifier, criterion, optimizer)<br>
print("Loss for Training on Epoch " +str(epoch) + " is “+ str(train_loss) + " lr:” + str(args.lr))</p>
<p>def train_classifier(train_loader, classifier, criterion, optimizer):<br>
classifier.train()<br>
<span class="hashtag">#exp_lr_scheduler</span>.step()<br>
train_loss = 0<br>
correct = 0<br>
total = 0<br>
losses = []<br>
y_true = np.zeros((0,20))<br>
y_score = np.zeros((0,20))<br>
for i, (images, labels, _) in enumerate(train_loader,0):<br>
images, labels = images.to(device), labels.to(device)<br>
<span class="hashtag">#zero</span> the parameter gradients<br>
optimizer.zero_grad()<br>
logits = classifier(images.to(device))<br>
<span class="hashtag">#pdb</span>.set_trace()<br>
predict = nn.functional.sigmoid(logits)<br>
zero = torch.zeros_like(logits)<br>
one = torch.ones_like(logits)<br>
predict = torch.where(logits &gt;= 0.5, one, predict)<br>
predict = torch.where(logits &lt; 0.5, zero, predict)<br>
<span class="hashtag">#pdb</span>.set_trace()<br>
y_true = np.concatenate((y_true, labels.cpu().detach().numpy()), axis=0)<br>
y_score = np.concatenate((y_score, predict.cpu().detach().numpy()), axis=0)<br>
loss = criterion(predict, labels)<br>
loss.backward()<br>
optimizer.step()<br>
losses.append(loss)</p>
<p>output is that:<br>
-------  Class: aeroplane        AP:   0.0452  -------<br>
-------  Class: bicycle          AP:   0.0488  -------<br>
…<br>
mAP: 0.0806</p>
How can I rotate images

username: YinYang_Untalan
cooked: <p>I have landscape photos (135,240) but I have photo in portrait (240,135): rotate by -90 deg. Is there a way to this in pytorch ? Thank you</p>

username: ptrblck
cooked: <p>I think <code>permute</code> and <code>flip</code> should work here:</p>
<pre><code class="lang-python">x = torch.zeros(135, 240)
# Draw reference lines
x[:, 0:10] = torch.arange(x.size(0)).unsqueeze(1).float()
x[0:10, :] = torch.arange(x.size(1)).unsqueeze(0).float() * 2

plt.imshow(x.numpy())

x_trans = x.permute(1, 0).flip(0)
plt.imshow(x_trans.numpy())
</code></pre>

username: YinYang_Untalan
cooked: <p>thanks. permute &amp; flip can used for multiples of 90 rotations. I got lucky, a simple transpose worked for me. thanks a lot.</p>
Splitting a concatenated sequence (advanced indexing )

username: Ohad_Rubin
cooked: <p>I have a batched tensor of size [k,max_seq_len,embedding_dim] (batch size is k)  which represents a concatenation two sequences, the first sequence is of size n_i and the second is of size m_i and I want to split this into two tensors where the first contains only the first sequence and the second has the second sequence, where the padding is according to the max n_i / m_i.<br>
For example:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1fb03fa7e65e7bc1ffd60ce54d23dea4c1ab82b3.png" data-download-href="https://discuss.pytorch.org/uploads/default/1fb03fa7e65e7bc1ffd60ce54d23dea4c1ab82b3" title="Screenshot from 2020-03-08 00-48-59"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1fb03fa7e65e7bc1ffd60ce54d23dea4c1ab82b3_2_665x500.png" alt="Screenshot from 2020-03-08 00-48-59" data-base62-sha1="4wkpzZ5zqS8gbuQKcRKPSKq3bA7" width="665" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1fb03fa7e65e7bc1ffd60ce54d23dea4c1ab82b3_2_665x500.png, https://discuss.pytorch.org/uploads/default/original/3X/1/f/1fb03fa7e65e7bc1ffd60ce54d23dea4c1ab82b3.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/f/1fb03fa7e65e7bc1ffd60ce54d23dea4c1ab82b3.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-03-08 00-48-59</span><span class="informations">763×573 21.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I can do this with loops, using torch.split, but I am interested in a solution without a loop on the batch dimension.<br>
Thanks!</p>
RNN Training Loss Computation and BPTT

username: Atul_Sharma
cooked: <p>Hi, I am trying to train an RNN on 1 million sentences (actually genomic reads), each of length = 100 characters.<br>
I want to use bptt = 3, that is, I will break down each sentence into overlapping sequences of length 3 with stride 1.<br>
Let us say, I am using a batch size of 5, that means for each batch, there will be 5 independent forward passes taking place (please correct me if I am wrong)<br>
I want to know if what I am doing is correct<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1db6ecf9fd115f3de1c8b72624cabe3b33b6667f.png" data-download-href="https://discuss.pytorch.org/uploads/default/1db6ecf9fd115f3de1c8b72624cabe3b33b6667f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1db6ecf9fd115f3de1c8b72624cabe3b33b6667f.png" alt="image" data-base62-sha1="4eRLjs8ddGym5LpQviw4kREnIaX" width="690" height="303" data-dominant-color="EFF0F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">879×387 38.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Following this, this is my input and hidden shape<br>
My input tensor looks like this : [a1 b1 c1 d1 e1<br>
a2 b2 c2 d2 e2<br>
a3 b3 c3 d3 e3]  (not showing the one hot encoding here)<br>
So, when a forward pass is happening, does the hidden state grow from a1 to a2 to a3? How is it handled internally by PyTorch?<br>
The corresponding target tensor (used after flattening, of course) looks like this<br>
[a2 b2 c2 d2 e2<br>
a3 b3 c3 d3 e3<br>
a4 b4 c4 d4 e4]<br>
The forward pass creates 15 values of losses. As far as I understand, we should accumulate the loss from a1 to a2 and a2 to a3 and a3 to a4.<br>
So, instead of computing loss from output and target of size 15, shall I compute the loss for size 3, add them up, do loss.backwards() for all the 5 inputs?</p>
How can I get the gradient of each pixels in GANs?

username: fruit_loops
cooked: <p>Hello.</p>
<p>I have DCGAN model which generate 64x64 image for MNIST.</p>
<p>I would like to calculate d x_{i,j} / d theta.</p>
<p>So, try to calculate each pixels’ gradient.</p>
<p>What I tried was,</p>
<pre><code class="lang-auto">for image channel, width, height:
#Hooking only for Conv2d layers
                for name, param in netG.main.named_parameters():
                    if ('0' in name and '10' not in name or '3' in name or '6' in name or '9' in name or '12' in name):
                        param.register_hook(lambda grad: grad_list.append(grad.view(-1)))

#Try to calculate gradient
                fake[0,channel,j,k].backward(retain_graph = True)

netG.zero_grad()
</code></pre>
<p>This code made an error. CUDA out of memory.</p>
<p>How can I get this?</p>
Get linear combination of matrix entries

username: Borel
cooked: <p>Hi there, i would like to get a linear combination of a matrix rows weighted by scalars on other matrix row. My setup is the following</p>
<pre><code class="lang-auto">S: Tensor B x M x N
W: Tensor B x M x M
</code></pre>
<p>Now suppose <code>S[0] = [[1,2,1,0],[-1,3,0,2],[1,2,2,-1]]</code> and<br>
<code>W[0] = [[0.2,0.3,0.5], [0.1,0.1,0.8], [0.2,0.2,0.6]]</code>. What i want to get is a matrix with entries</p>
<pre><code class="lang-auto">a = 0.2[1,2,1,0] + 0.3[-1,3,0,2] + 0.5[1,2,2,-1] = [0.4, 2.3, 1.2, 0.1]
b = 0.1[1,2,1,0] + 0.1[-1,3,0,2] + 0.8[1,2,2,-1] = [0.8, 2.1, 1.7, -0.6]
c = 0.2[1,2,1,0] + 0.2[-1,3,0,2] + 0.6[1,2,2,-1] = [0.6, 2.2, 1.4, -0.2]
</code></pre>
<p>And that’s for each pair of chanels <code>S[i], W[i]</code>. I came up with a solution but implies <code>repeat</code> and <code>view</code>  several times and it looks super ugly to me. Any help is welcome! Ty</p>
Accessing model's parameters by params key

username: Aqiqi
cooked: <p>Hello,<br>
I created an optimizer like so</p>
<pre><code class="lang-auto">model = Sequential(
        Linear(in_features, 64),
        ReLU(),
        Dropout(p=.2),
        Linear(64, 32),
        ReLU(),
        Dropout(p=.2),
        Linear(32, 3),
        ReLU(),
        Linear(3, 1))

opt = torch.optim.SGD(
    [
        {'params': model[0].parameters(), 'lr': 0.1},
        {'params': model[1:3].parameters(), 'lr': 0.01},
        {'params': model[4:7].parameters(), 'lr': 0.001},
    ])
</code></pre>
<p>After I serialize <code>opt.state_dict()</code></p>
<pre><code class="lang-auto">{'state': {},
 'param_groups': [{'lr': 0.1,
   'momentum': 0,
   'dampening': 0,
   'weight_decay': 0,
   'nesterov': False,
   'params': [139865343712712, 139865343712928]},
  {'lr': 0.01,
   'momentum': 0,
   'dampening': 0,
   'weight_decay': 0,
   'nesterov': False,
   'params': [139865343712640, 139865343713072]},
  {'lr': 0.001,
   'momentum': 0,
   'dampening': 0,
   'weight_decay': 0,
   'nesterov': False,
   'params': [139865343713144,
    139865343713216,
    139865343709616,
    139865343710048]}]}
</code></pre>
<p>Which arguments should I pass SGD constructor to recreate the optimizer given <code>model</code> and the above Json? I think if I was able to access the model’s parameters given the keys in params like <code>139865343712712</code> , I could recreate the initial input, but I haven’t been able to do that.</p>
<p>I’m really having a hard time understanding the Optimizer class source code, any help would be greatly appreciated.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>I’m not sure to understand the use case completely.<br>
If you want to restore the <code>optimizer</code>, you could store its <code>state_dict</code> and load it afterwards.<br>
To get the <code>id</code> of a parameter, you could use <code>print(id(model.layer.weight))</code>, however I’m not sure what you would like to do with it.</p>

username: Aqiqi
cooked: <p>Thanks for your answer. The context around this is I’m serializing the model and optimizer and deserializing in another context where I don’t know the architecture of <code>model</code> and the initial call to SGD’s constructor</p>
<p>Since I’m already saving <code>model.state_dict()</code> entirely, it seems that I do not need to save <code>optimizer.state_dict()</code>, but only the <code>param_groups</code>.<br>
For example, when there is only one param_group, <code>SGD([*model.parameters()],**opt.state_dict()['param_groups'][0])</code> reconstructs the inital optimizer nicely, and I am trying to generalize this to more than one param_groups.</p>
<p>A <code>Sequential</code> model does not have a layer attribute, but not sure what you meant even after trying different calls. but yes, I would need a function that takes an id of a parameter that I would call. Then I would be able to call the optimizer’s constructor in the same way as I initially do, with a list of dictionaries.</p>
To reverse a tensor in some dimension with masked info

username: Stalin_Varanasi
cooked: <p>Hi,</p>
<p>Is there a better way to reverse a tensor with a mask in some dimension? Currently, I do this:</p>
<pre><code class="lang-auto">def masked_reverse(x, pad=0.):
    mask = (x != pad).float()
    upper_tri = torch.triu(torch.ones(x.size(1), x.size(1)))
    lower_tri = upper_tri.transpose(0, 1) 
    ones = torch.ones(x.size())
    ones = ones.matmul(upper_tri) - 1
    ones = ones * (1 - mask)
    # obtain re-arranged tensor for reversing the indices
    rev_indices = (mask.matmul(lower_tri) - 1) * mask # reverse indices start with 0
    rearrange = rev_indices + ones
    rearrange_mat = torch.zeros(rearrange.size(0), rearrange.size(1), rearrange.size(1))
    rearrange_mat = rearrange_mat.scatter(2, rearrange.long().unsqueeze(2), 1) 
    xr = torch.bmm(rearrange_mat.long(), x.unsqueeze(2)).squeeze(-1)
    return xr
</code></pre>
<pre><code class="lang-auto"># Example input with 0's as padded tokens
inp=torch.LongTensor(4,10).random_(1,10)
inp[0][8:]=0 # input masked from position 8-9
inp[1][6:]=0 # input masked from position 6-9
inp[2][9:]=0 # input masked at position 9


&gt;&gt;&gt; inp
tensor([[6, 5, 4, 4, 6, 4, 8, 5, 0, 0],
        [3, 9, 7, 4, 4, 7, 0, 0, 0, 0],
        [4, 9, 8, 1, 7, 3, 8, 3, 4, 0],
        [4, 5, 7, 9, 3, 1, 1, 5, 8, 9]])

#Reversed
&gt;&gt;&gt; masked_reverse(inp)
tensor([[5, 8, 4, 6, 4, 4, 5, 6, 0, 0],
        [7, 4, 4, 7, 9, 3, 0, 0, 0, 0],
        [4, 3, 8, 3, 7, 1, 8, 9, 4, 0],
        [9, 8, 5, 1, 1, 3, 9, 7, 5, 4]])
</code></pre>

username: Allan_Jie
cooked: <p>Probably this is the only way to do it right now. But it is memory inefficient.</p>

username: HAORAN_LI
cooked: <p>zhen qiao!!!</p>

username: Allan_Jie
cooked: <p>You can use the masked_filp function in the AllenNLP package now.</p>
Find actual gpu execution time

username: Navdeep_Kumar
cooked: <p>Suppose I have a model and it has only one layer, say the nn.linear layer. I am interested in finding out the acutal time of the operation i.e. matmul and bias adddition excluding the time to move the data around that is calls such as cudamalloc() or cudamallocmanaged().<br>
What I plan of doing is finding out the cuda/cublas calls that nn.linear does and the instrument that call to get the time.<br>
I have this idea but dont know how to execute this.<br>
I have a NLP model(which I want to instrument) inside a container, which has pytorch built from nvidia ngc pytorch image. Any suggestions on how to do this?</p>

username: ptrblck
cooked: <p>You could use <code>nsys</code> to profile your code and visualize specific parts of your model using Nsight Systems.<br>
Since you are using the NGC container, <code>nsys</code> will already be installed.<br>
Have a look at the <a href="https://pytorch.org/docs/stable/cuda.html#nvidia-tools-extension-nvtx">nvtx docs</a> to see how to set range markers.</p>

username: Navdeep_Kumar
cooked: <p>Ok thanks for the reply. i was abel to get the overall statistics. Now suppose that I have many layers is there any way to get profiling results at the granularity level of the layers?</p>
Custom Class Torch nn.Module Error

username: Kiran_Ikram
cooked: <p>Hi, In implementing class BertAdapter(nn.Module): , I am getting the following error:</p>
<p>File “/Users/ikram/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 576, in <strong>getattr</strong><br>
type(self).<strong>name</strong>, name))<br>
AttributeError: ‘BertAdapter’ object has no attribute ‘module’</p>
<p>Is this due to the fact that I’m trying to run this on my CPU, and I’m not connected to a GPU?</p>

username: ptrblck
cooked: <p>Could you post the class definition of <code>BertAdapter</code>, which raises this error, please?</p>

username: Kiran_Ikram
cooked: <p>The error is being thrown in line 106 of <a href="https://github.com/facebookresearch/EmpatheticDialogues/blob/master/retrieval_train.py" rel="nofollow noopener">https://github.com/facebookresearch/EmpatheticDialogues/blob/master/retrieval_train.py</a> “dtype = model.module.opt.dataset_name”</p>
<p>BertAdapter is defined in <a href="https://github.com/facebookresearch/EmpatheticDialogues/blob/master/empchat/bert_local.py" rel="nofollow noopener">https://github.com/facebookresearch/EmpatheticDialogues/blob/master/empchat/bert_local.py</a><br>
class BertAdapter(nn.Module):<br>
def <strong>init</strong>(self, opt, dictionary):<br>
from parlai.agents.bert_ranker.helpers import BertWrapper</p>
<pre><code>    try:
        from pytorch_pretrained_bert import BertModel
    except ImportError:
        raise Exception(
            "BERT rankers needs pytorch-pretrained-BERT installed. "
            "\npip install pytorch-pretrained-bert"
        )
    super().__init__()
    self.opt = opt
    self.pad_idx = dictionary[PAD_TOKEN]
    self.ctx_bert = BertWrapper(
        bert_model=BertModel.from_pretrained(BERT_ID),
        output_dim=opt.bert_dim,
        add_transformer_layer=opt.bert_add_transformer_layer,
    )
    self.cand_bert = BertWrapper(
        bert_model=BertModel.from_pretrained(BERT_ID),
        output_dim=opt.bert_dim,
        add_transformer_layer=opt.bert_add_transformer_layer,
    )
</code></pre>
<p>…</p>

username: ptrblck
cooked: <p>Thanks for the line of code!<br>
The <code>.module</code> attribute is added, when you wrap your model into e.g. <code>nn.DataParallel</code> as seen in <a href="https://github.com/facebookresearch/EmpatheticDialogues/blob/8122f7979eb0ba86b6fc196a64a779e1a98999a0/retrieval_train.py#L182">this line of code</a>.</p>
<p>However, based on the condition it seems this line will only be called, if you are using the GPU (or rather if <code>opt_.cuda</code> is <code>True</code>).<br>
Is this the case or are you using the CPU?<br>
In the latter case, just remove the <code>.module</code> attribute and use <code>dtype = model.opt.dataset_name</code> directly.</p>

username: Kiran_Ikram
cooked: <p>Thank you so much! Your response was super helpful!</p>
Model be worthier than it's previous

username: Diaa_elsayed_ziada
cooked: <p>I’m working on 3d landmark detection model<br>
the training process works very well for el first 2 epochs and at the middle of the  epoch 3 the loss starts to increase in an insane way and model results get much worth than it’s previous<br>
what could be the reason of this</p>

username: ptrblck
cooked: <p>Is the training loss or the validation loss going up?<br>
Could you post the loss curves for both losses?</p>

username: Diaa_elsayed_ziada
cooked: <p>for frist two epochs<br>
trainloss inf -&gt; 0.1 -&gt; 0.03<br>
validationloss inf-&gt;0.8-&gt;0.021<br>
when this issue happens in the first 30% of epoch 3 i stop train to check the results in real images<br>
the results was very bad compared to the previous model</p>
What is the difference between passing through a linear layer and using `nn.Parameter`?

username: seankala
cooked: <p>Hello. This may be a bit of an elementary question, but I was having trouble figuring out the nuts and bolts of things.</p>
<p>I’m currently trying to implement a neural network model, and in the original paper there is something about performing matrix multiplication with a layer-specific weight matrix.</p>
<p>Until now, when I perform that operation I used <code>torch.nn.Linear</code>, but I noticed that the code implementation used <code>torch.nn.Parameter</code> instead and performed matrix multiplication explicitly.</p>
<p>What is the difference between these two modules? I initially thought they were the same thing, but something’s telling me that’s not the case. Thanks.</p>

username: G.M
cooked: <p>By default, <code>nn.Linear</code> has a bias, which means it also performs a matrix addition after the matrix multiplication(y=Wx+B). U can turn the “bias” off by specifying <code>bias=False</code>; this makes <code>nn.Linear</code> equivalent to the matrix multiplication(y=Wx).</p>

username: seankala
cooked: <p>Hi, thanks for the reply. So if I set the <code>bias</code> argument to <code>False</code> and run data through a <code>nn.Linear</code> layer, that would be the equivalent of performing matrix multiplication of the data with an <code>nn.Parameter</code>?</p>

username: G.M
cooked: <p>Yes. In fact, if u print the code of a TouchScript involving a Linear layer, you’ll actually see that in TouchScript, <code>nn.Linear</code> is expressed as <code>torch.addbmm</code>.</p>
Runtime Error: Function MulBackward0 returned an invalid gradient at index 1 - expected type torch.FloatTensor but got torch.cuda.FloatTensor

username: MariosOreo
cooked: <p>Hi there,</p>
<p>In the training phase, then I run <code>loss.backward()</code>, it raised this error:</p>
<pre><code class="lang-auto">  File "/home/jingweipeng/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/jingweipeng/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function MulBackward0 returned an invalid gradient at index 1 - expected type torch.FloatTensor but got torch.cuda.FloatTensor
</code></pre>
<p>I have checked device attribute of all parameters, using the code snippet like <a href="https://discuss.pytorch.org/t/unction-cudnnconvolutionbackward-returned-an-invalid-gradient-at-index-1-expected-type-torch-floattensor-but-got-torch-cuda-floattensor/51845/2">this</a>.<br>
The Loss Function is simple <code>nn.CrossEntropyLoss()</code>.<br>
How can I trace such an error, using <code>hooks</code>? Could you give me some hints or advice? Thanks in advance!</p>

username: MariosOreo
cooked: <p>I think there must be some tensors on CPU and some tensors on GPU, right?</p>
<p>How can I find such invalid gradient <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

username: MariosOreo
cooked: <p>I have solved this error. Some variables initialized in <code>forward</code> were not put deployed on cuda.</p>

username: Hari_Krishnan
cooked: <p>Hi,</p>
<p>I’m getting the same error, can you share how you found out which variables were not deployed on cuda. I’m using a large model , so it is difficult for me to print out the type of every model and check. Is there any other way to go about it?</p>
How to train networks with loss feedback from other places?

username: uduse
cooked: <p>Say I have a NN that generates images. The input of the NN is random noise and the category of the image I want to generate. Now, I have a different program that tells what category of the image actually gets generated or how far the generated image is from the desired category. Can I use this information to train the NN in some way?</p>
<p>(It’s not GAN. It’s different in two ways. There are no “real” inputs compared to generated ones. The discriminator (or evaluator) is not a NN and has nothing to do with tensors.)</p>

username: ptrblck
cooked: <p>You could try to use the output of your “program” e.g. as a target for an autoencoder-like architecture, where the label might be fed to a side branch coming from the latent tensor, while the generator output is trained with e.g. <code>nn.MSELoss</code>.</p>
<p>I assume your program cannot generate valid gradients, which could be fed to the generator?</p>

username: uduse
cooked: <p>Thank you for your reply!</p>
<p>Yes, my outside program cannot generate valid gradients. Instead, it only spits out numbers. I decided to connect a second NN after the first NN to approximate my outside program. If the approximation is good enough, then I can train my original NN with the gradients provided by the second NN.</p>
Sudden Increase in Training Accuracy for Resnet-50 Binary Classification

username: Aka
cooked: <p>I’m applying transfer learning to the resnet50 from torchvision models, i.e. replaced the last fc layer with 2 neurons for binary classification problem. All other layers are frozen. I started training the network with learning rate 1e-2 and reducing it by a factor of 0.1 after 5 epochs if validation loss is not decreased. However, when the learning rate decreases from 1e-2 to 1e-3 for the first time, there is a suddent and very big increase in training accuracy (therefore decrease in the loss). Training accuracy jumps from around 54% to 96%. I attach the loss and accuracy plots. I use cross entropy loss. What might be the reason of this sudden big increase in the training accuracy?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c5ed4bbcadb3fabdd4f2cd532a77fe332a52d1d.png" alt="acc" data-base62-sha1="fsGJXr3C7sCfz9qhvGEWtUGzOzj" width="313" height="235"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/6/1/615707d6d3a2eb0a0fa12b854241b2964e80ee17.png" alt="loss" data-base62-sha1="dT6Makq9laqV7bzuM2B41vStJDp" width="313" height="235"></p>

username: ptrblck
cooked: <p>Your initial learning rate might be too high. Lowering the learning rate might “start” the training.<br>
If your dataset is quite easy (based on the step size it seems you are using very few samples), the model might instantly overfit the training data.</p>

username: Aka
cooked: <p>Actually you’re right. My dataset size is not much. I suspected learning rate to be high, therefore started training with learning rate 1e-3. In that case the model doesn’t fully overfit (training accuracy doesn’t reach around 100%), meaning I didn’t observe such dramatic increases. However, validation accuracy also doesn’t increase accordingly. This is also overfit, I guess, because the model can convey what it seems to learn from training set to validation set.</p>
<p>What made me surprised is that after jumping around global minima with high learning rate, one epoch would be enough to reach almost 100%. It didn’t make sense to me. Thanks for your response.</p>
Join two tensors based on a value along a dimension (similar to SQL join)

username: mohsenkiskani
cooked: <p>Is it possible to do an operation similar to a SQL join or SQL merge in Pytorch? For instance, I have 2 tensors A and B (both two dimensional) and I want to join them along dimension 0. In other words, I want to grab the row B[i, 1:] from tensor B and concatenate it to row A[j, :] if A[j, 0] == B[i, 0] and I want to do this for all rows of A (equivalent to SQL left join). We can assume that for each row j in A there exists only one such row i in B. Can we do this without a for loop so that we don’t slow down the gradient flow?</p>
Faster R-CNN Image size in training and inference

username: sigma_x
cooked: <p>For Faster/Mask RCNN, before training the model, min and max size of the input images is fixed, which is determined to a great extent by the CUDA memory capacity. This is OK.</p>
<p>What I found at inference time, is that accuracy of the model strongly depends on the size of the input image. By default, <code>min_size=800, max_size=1333</code>, but as I varied both hyperparameters, I got either better of worse results. I think the best results are obtained when the <code>min_size</code> and <code>max_size</code> are close to the input image’s size, but I’m not sure.</p>
<p>So this is my question: is there a confident way to find optimal input image size?</p>

username: ptrblck
cooked: <p>I would assume the model should work the best during validation using images with shapes and other properties as close as possible to the training data.<br>
The shape limits are most likely added due to the architecture (but I haven’t looked into the source code to verify it).</p>
How to make two tensors to be of the same shape

username: Ameen_Ali
cooked: <p>Hello,<br>
I have two tensors :<br>
first = [A , B , 2048].<br>
second = [C , D 2048]<br>
Where A &gt; C and B &gt; D.<br>
how can I padd the second tensor with zeros so it match the first tensor dimensions, and in generally does padding affectu accuracy ?<br>
Thanks</p>

username: ptrblck
cooked: <p>You could use <code>F.pad</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">A, B, C, D = 5, 4, 3, 2
a = torch.randn(A, B, 2)
b = torch.randn(C, D, 2)

out = F.pad(b, (0, 0, (B-D)//2, (B-D)//2, (A-C)//2, (A-C)//2))
print(out.shape == a.shape)
</code></pre>
<p>Note that <code>F.pad</code> pads starting from the last dimension.</p>
<aside class="quote no-group" data-username="Ameen_Ali" data-post="1" data-topic="72339">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ameen_ali/48/15528_2.png" class="avatar"> Ameen_Ali:</div>
<blockquote>
<p>and in generally does padding affectu accuracy ?</p>
</blockquote>
</aside>
<p>This might be the case, but I think as long as you pad your tensors in a reasonable way, it shouldn’t matter.</p>