akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Inplace error in LSTM backward pass

username: Alper_YILDIZ
cooked: <p>Hi everyone,</p>
<p>I am trying to run LSTM network, it may be used after convolutional layer so I reshape incoming tensor from [ batch_size, in_channels, window_length ] to [ batch_size, window_length, in_channels ]  but I got inplace error as given below.</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [800, 201]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
<p>I pass layer parameters and batch size to the LST object. I see what is the problem, but couldn’t find any way to make it work.</p>
<pre><code class="lang-auto">class LST(nn.Module):
    def __init__(self, param, BS ):
        super().__init__()
        self.batch_size = int(BS)
        self.init_hidden_states(param)
        self.layer = nn.LSTM( *param )


    def init_hidden_states(self,param):
        if param[6] == True:
            NUM_OF_DIRECTIONS = 2
        else:
            NUM_OF_DIRECTIONS = 1
        self.hidden = (torch.randn(param[2] * NUM_OF_DIRECTIONS, self.batch_size, param[1]).to('cuda'),
                       torch.randn(param[2] * NUM_OF_DIRECTIONS, self.batch_size, param[1]).to('cuda'))      


    def forward(self, x ):            
        
        batchsize , features , windowlength = x.shape
        y = x.contiguous().view(batchsize, windowlength, features)
        z, self.hidden = self.layer( y, self.hidden )
        batchsize, windowlength, features = z.shape
        t = z.contiguous().view(batchsize, features, windowlength)
        return t
</code></pre>

username: Alper_YILDIZ
cooked: <p>I see the problem here is hidden states that inplaced with layer operation, can anyone suggest where to detach hidden states and how to push them back for next batch operation?</p>
I am trying to get the values of __getitem__ function

username: MUHAMMAD_JAHANZAIB_K
cooked: <p>I have  no idea how to get the return value of  the class function<br>
i am new in the python kindly answer me i am tired…</p>
<pre><code class="lang-auto">class Dataset(torch.utils.data.Dataset):
    

    def __init__(self, csv_file, root_dir, transform=None):
        
        self.csv = pd.read_csv(csv_file)
        self.root   = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.csv)
    
    def __getitem__(self,idx):
        
        img_name=self.csv.iloc[idx,0]
        landmarks =self.csv.iloc[idx, 4:].as_matrix()
        

        return img_name,landmarks

dataset = Dataset(csv_file='Dataset/loclization.csv', root_dir='Dataset/local/')
</code></pre>
<p>i need to get img_name and landmarks</p>

username: futscdav
cooked: <p><code>__getitem__(self, idx)</code> is what gets called when you do the index operator ([idx]).</p>
<p>So <code>dataset[idx]</code> actually calls <code>dataset.__getitem__(idx)</code>. You can of course also call <code>__getitem__</code> directly, but it’s meant to be accessed through the <code>[]</code> operator.</p>
Multiply dimension of Tensor with scalar in an Onnx compatible way

username: Julien_Maille
cooked: <p>Dear experts,<br>
I’m trying to multiply my <code>[B, 2, X, Y]</code> tensor with a (constant_1, constant_2) vector of scalars.<br>
The constraint is that it should export to onnx (opset 9)<br>
I tried</p>
<pre><code class="lang-python"># this won't export to onnx
def forward(self, x):
  x[:,0,:,:] *= const_1
  x[:,1,:,:] *= const_2
  return x
</code></pre>
<pre><code class="lang-python"># this does export fine but obviously not doing what I want
def forward(self, x):
  x.mul_(const_1)
  return x
</code></pre>
<pre><code class="lang-python"># this actually doesn't do anything
def forward(self, x):
  x[:,0,:,:].mul_(const_1)
  x[:,1,:,:].mul_(const_2)
  return x
</code></pre>
<p>How can I achieve this using torch operators?</p>
Use DataLoader with a tuple index

username: Luis_Costa
cooked: <p>Hello, I am trying to use my DataLoader with a tuple index dataset. I need this because each index refers to a specific coordinate in a huge image, therefore I keep it as the index. However, when I am trying to load each image from the the Dataloader, I get the following error:</p>
<blockquote>
<p>error list indices must be integers or slices, not tuple (37, 71)</p>
</blockquote>
<pre><code class="lang-auto">with torch.no_grad():
        for batch in data_loader:
            data, _, idx = batch
            logits = model(data.to(device))
            probabilities = F.softmax(logits, dim=1)
</code></pre>
<p>This is how, I create the dataloader:</p>
<pre><code class="lang-auto">pool_loader = DataLoader(dataset, batch_size=batch_size, num_workers=num_workers,
                                         sampler=SubsetRandomSampler(pool_idx))
</code></pre>
<p>Is there any way to use the Dataloader using a tuple index?</p>
Phased LSTMs implementation in Pytorch

username: Mina_soltan
cooked: <p>Has anyone implemented phased LSTM in Pytorch? I can’t find anything useful online.</p>

username: marchinidavide
cooked: <p>I searched the same  months ago and no one did. Maybe we should just do it ourself!<br>
Any interest in team up?</p>

username: Mina_soltan
cooked: <p>I can contribute but I have very limited hours I can put in.</p>

username: Ginsunuva
cooked: <p>A bit late, but I made an implementation in a couple hours here if anyone else still needs a Torch version:</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/AAnoosheh/c6877583ee5c46d6adf81b5bc355379d" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/AAnoosheh/c6877583ee5c46d6adf81b5bc355379d" target="_blank" rel="nofollow noopener">https://gist.github.com/AAnoosheh/c6877583ee5c46d6adf81b5bc355379d</a></h4>
<h5>phased_lstm.py</h5>
<pre><code class="Python">import math

import torch
import torch.nn as nn
import torch.nn.init as init
from torch.nn import functional as F


class PhasedLSTMCell(nn.Module):
    """Phased LSTM recurrent network cell.</code></pre>
This file has been truncated. <a href="https://gist.github.com/AAnoosheh/c6877583ee5c46d6adf81b5bc355379d" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: cstsunfu
cooked: <p>Hi, I am learning the PhasedLSTM now, and I want to refer to your torch version code. And the link couldn’t found now, do you have the backup of this code?</p>

username: Ginsunuva
cooked: <p>Hi,<br>
Sorry don’t know why it’s gone.</p>
<p>Here’s a re-upload of the gist, slightly modified:<br>
</p><aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/AAnoosheh/b3cce037ae24a86d89505907ec098d62" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/AAnoosheh/b3cce037ae24a86d89505907ec098d62" target="_blank" rel="nofollow noopener">https://gist.github.com/AAnoosheh/b3cce037ae24a86d89505907ec098d62</a></h4>
<h5>phased_lstm.py</h5>
<pre><code class="Python">import math

import torch
import torch.nn as nn



class PhasedLSTMCell(nn.Module):
    """Phased LSTM recurrent network cell.
    https://arxiv.org/pdf/1610.09513v1.pdf</code></pre>
This file has been truncated. <a href="https://gist.github.com/AAnoosheh/b3cce037ae24a86d89505907ec098d62" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I just modified it from the version in my codebase, deleting unnecessary code.<br>
Haven’t tried running it for now, but should be fine. Hopefully no typos.</p>
Difference between nn.Linear and nn.Conv1d

username: Xuyang_Bai
cooked: <p>I think Linear and Conv1d these two layers performs similarly when doing pointwise convolution, so what is the difference between these two layers and how can I choose which one to use ?</p>

username: klory
cooked: <p>when num_channel=1, kernel=1, strider=1, padding=1 then they are the same</p>

username: Ashish_Gupta1
cooked: <p>As far i know …when you have to preserve the spatial information like in the semantic segmentation…use convolution. And when you need not to do anything related to spatial information like in basic classification(mnist,cat dog classifier) go with Linear layers.</p>

username: orena1
cooked: <p>Do note that Linear is faster…<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/55576314/conv1d-with-kernel-size-1-interpretation" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/5692012/robinfrcd" target="_blank" rel="nofollow noopener">
    <img alt="RobinFrcd" src="https://i.stack.imgur.com/IByBi.png?s=128&amp;g=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/55576314/conv1d-with-kernel-size-1-interpretation" target="_blank" rel="nofollow noopener">Conv1D with kernel_size=1 interpretation</a>
</h4>

<div class="tags">
  <strong>python, conv-neural-network, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/5692012/robinfrcd" target="_blank" rel="nofollow noopener">
    RobinFrcd
  </a>
  on <a href="https://stackoverflow.com/questions/55576314/conv1d-with-kernel-size-1-interpretation" target="_blank" rel="nofollow noopener">02:58PM - 08 Apr 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How to permute output neurons

username: mustgoplay
cooked: <p>I am getting outputs from an unsupervised layer with 10 outputs for MNIST.  The output neurons are permuted though since we don’t assign them directly.  I.e. the 10 outputs are actually: [9., 6., 5., 8., 2., 1., 4., 7., 0., 3.] instead of 0-9.</p>
<p>The shape of the output data is [bs, 10].  How can I permute the 10 so that the layer output is in the right order and maintain the gradient please?  Permute seems to only want to permute bs and 10.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>You could index dim1 with your indices as seen here:</p>
<pre><code class="lang-python">x = torch.arange(5 * 10).view(5, 10)
idx = torch.randperm(10)
x = x[:, idx]
print(x)
&gt; tensor([[ 9,  8,  0,  7,  2,  3,  1,  6,  5,  4],
        [19, 18, 10, 17, 12, 13, 11, 16, 15, 14],
        [29, 28, 20, 27, 22, 23, 21, 26, 25, 24],
        [39, 38, 30, 37, 32, 33, 31, 36, 35, 34],
        [49, 48, 40, 47, 42, 43, 41, 46, 45, 44]])
</code></pre>
<p>Let me know, if that would work for you.</p>

username: mustgoplay
cooked: <p>That should work.  Thanks!</p>
How to get the output at the 2nd block of a resnet-50?

username: HaziqRazali
cooked: <p>How can I get the output at the 2nd block of a resnet i.e. the layer that puts out a tensor with 128 channels?</p>
<p>The code below converts the model into a list then uses torch.nn.Sequential which destroys the residual connections in the resnet.</p>
<pre><code class="lang-auto">        self.encoder = models.resnet18(pretrained=True)
        modules = list(self.encoder.children())
        self.encoder = torch.nn.Sequential(*(list(modules)[:-4]))
</code></pre>

username: ptrblck
cooked: <p>The residual connections won’t be destroyed, as they are implemented in the <code>BasicBlock</code> and <code>Bottleneck</code> layers. However, while wrapping the model in an <code>nn.Sequential</code> container might work for your use case, I would recommend to use forward hooks and return the desired activation.<br>
<a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">Here</a> is an example of how to use forward hooks.</p>

username: HaziqRazali
cooked: <p>Thank you. But that would mean that memory is still being used for the later layers during the forward pass right ? I am training a model that uses the output of resnet’s 2nd block and would not want to waste memory.</p>

username: ptrblck
cooked: <p>Yes, that’s right.<br>
If you don’t need the following layers, your approach should work.<br>
As a quick test you could use your <code>nn.Sequential</code> approach and compare it to the activation from the forward hook of the original model.</p>

username: HaziqRazali
cooked: <p>Thank you ! I have verified that the outputs are equal.</p>
How to get gender information as label from CelebA dataset

username: lepoeme20
cooked: <p>Hi,</p>
<p>I’d like to get label which has only gender information from torchvision.datasets.CelebA</p>
<p>My code snippet is as follows:</p>
<pre><code class="lang-auto">trainset = torchvision.datasets.CelebA(..., download=True, split='train', target_type='attr', ...)
trainloader = torch.utils.data.DataLoader(trainset, ...)

for idx, (imgs, labels) in enumerate(trainlader):
    imgs = imgs.to(device)
    labels = labels.to(device)
</code></pre>
<p>What I really want to do is get labels only have gender information.</p>
<p>In this case, what should I do?</p>

username: ptrblck
cooked: <p>If I’m not mistaken, the default <code>target_type="attr"</code> should yield various attributes including the <code>male</code> attribute, so you could slice the target for this attribute only.</p>
Parent Struct Attribute Always Overrides Child Module Class

username: episodeyang
cooked: <p>I have ran into this multiple times now.</p>
<p>this does not work with class attributes.</p>
<pre><code class="lang-python">import torch
from torch import nn as nn
from torch_utils import View, Λ


class Metric(nn.Module):
    latent_dim = None
    embed = None
    kernel = None


# regular conv, migrated from ConvLargeL2.
class Conv(Metric):
    def __init__(self, input_dim, latent_dim):
        super().__init__()
        self.latent_dim = latent_dim
        self.embed = nn.Sequential(
            nn.Conv2d(input_dim, 32, kernel_size=4, stride=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=4, stride=2),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=4, stride=2),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.Conv2d(64, 32, kernel_size=4, stride=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            View(128),
            nn.Linear(128, latent_dim),
        )
        self.kernel = Λ(lambda a, b: (a - b).norm(2, dim=-1))

    def forward(self, x, x_prime):
        *b, C, H, W = x.shape
        *b_, C, H, W = x_prime.shape
        z_1, z_2 = torch.broadcast_tensors(
            self.embed(x.reshape(-1, C, H, W)).reshape(*b, self.latent_dim),
            self.embed(x_prime.reshape(-1, C, H, W)).reshape(*b_, self.latent_dim))
        *b, W = z_1.shape
        return self.kernel(z_1, z_2).reshape(*b, 1)

main__:
   net = Conv(...)
   print(net.embed)
</code></pre>
<p>gives ‘None’.</p>

username: ptrblck
cooked: <p>Try to initialize <code>Metric</code> as a module via:</p>
<pre><code class="lang-python">class Metric(nn.Module):
    def __init__(self):
        super(Metric, self).__init__()
        latent_dim = None
        embed = None
        kernel = None
</code></pre>
<p>and it should work.</p>
Very small class weights

username: nameless
cooked: <p>I am solving token classification problem. There is a huge class imbalance in data. Some of the classes have a million samples, others several hundred thousands. I saw many recommended to assign class weights to  do 1/amount of samples per class. But this way gives me very small numbers. <img src="https://discuss.pytorch.org/uploads/default/original/3X/3/9/394eff9df3a5775edccee2f3905b57734c1c75ee.png" alt="Снимок" data-base62-sha1="8aYt66o81A5MofEGRTS5JNZEAhg" width="632" height="214"><br>
Does it make sense or I have to multiply this on smth like 10^n?</p>

username: ptrblck
cooked: <p>You could multiply the weights, if you are suspecting rounding errors or any other numerical issues.<br>
The weights are used relatively, so you should be able to add a constant offset to the tensor.</p>
How to use different learning rates of Adam for continue training

username: Wei_Wong
cooked: <p>After training the model for a while, I save a checkpoint for model and optimizer, what should I do to change the learning rate of Adam for continue training?</p>

username: ptrblck
cooked: <p>You could try to access the <code>lr</code> attribute of each <code>param_group</code> and change the learning rate:</p>
<pre><code class="lang-python">optimizer.param_groups[0]['lr'] = new_lr
</code></pre>
Computing constrained gradients

username: murph213
cooked: <p>Here, computing gradients is an end in itself, not a means to the end of minimizing some function.</p>
<p>I have a vector <code>x</code> whose elements must always sum to 1.  I want to know how much <code>f(x)</code> changes for <strong>valid</strong> infinitesimal changes in <code>x</code>.  That is, how much does <code>f</code> vary over small changes in the regime where <code>x</code> sums to 1.</p>
<p>Using <code>.backward()</code> and traditional gradients naively would consider infinitesimal changes along the coordinate axes in the superspace of <code>x</code>. <code>x+delta</code> may not be a valid vector anymore, so this strategy cannot be used for my purpose.</p>
<p>So:</p>
<blockquote>
<p>How can I measure the gradient taken only in valid directions?</p>
</blockquote>

username: KFrank
cooked: <p>Hi Ryan!</p>
<aside class="quote no-group quote-modified" data-username="murph213" data-post="1" data-topic="79881" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/97f17d/48.png" class="avatar"> murph213:</div>
<blockquote>
<p>I have a vector <code>x</code> whose elements must always sum to 1.  I want to know how much <code>f(x)</code> changes for <strong>valid</strong> infinitesimal changes in <code>x</code>.  That is, how much does <code>f</code> vary over small changes in the regime where <code>x</code> sums to 1.</p>
<blockquote>
<p>How can I measure the gradient taken only in valid directions?</p>
</blockquote>
</blockquote>
</aside>
<p>You have a vector, <code>x</code>, that satisfies a constraint, <code>g (x) = 0</code> (where,<br>
in your case, <code>g (x) = x.sum() - 1</code>).</p>
<p>Your constraint defines a hypersurface in “<code>x</code>” space (in your specific<br>
case, a hyperplane), and you only want to consider infinitesimal<br>
changes to <code>x</code> that lie in this hypersurface.</p>
<p>The gradient of your constraint function, <code>g</code>, is perpendicular to your<br>
constraint hypersurface, so you want to subtract off (“project away”)<br>
any component of your gradient of <code>f</code> that is perpendicular to the<br>
constraint surface (that is, any component that is parallel to the<br>
normal (the vector perpendicular) to your constraint surface).</p>
<p>So you could use pytorch’s autograd and <code>backward()</code> to separately<br>
compute <code>grad_f</code> and <code>grad_g</code>.  Then the constrained gradient you<br>
want is:</p>
<pre data-code-wrap="python"><code class="lang-python">grad_f_constrained = grad_f - ((grad_f.dot (grad_g) / (grad_g**2).sum()) * grad_g
</code></pre>
<p>Good luck.</p>
<p>K. Frank</p>

username: murph213
cooked: <p>This makes sense, I’ll circle back if I have questions during the implementation.  Thanks!</p>
When i try to make predictions after i have trained my network RuntimeError: shape '[-1, 12544]' is invalid for input of size 6400

username: AnxheloD
cooked: <pre><code class="lang-auto">class SimpleCNN(torch.nn.Module):
  def __init__(self, input_ch, output):
    super(SimpleCNN, self).__init__()
    self.cnn1 = torch.nn.Conv2d(in_channels=input_ch, out_channels=16, kernel_size=[3,3],stride=1, padding=1)
    self.cnn2 = torch.nn.Conv2d(in_channels=16, out_channels=32, kernel_size=[3,3],stride=1, padding=1)
    self.cnn3 = torch.nn.Conv2d(in_channels=32, out_channels=32, kernel_size=[3,3], padding=1)
    self.cnn4 = torch.nn.Conv2d(in_channels=32, out_channels=64, kernel_size=[3,3], padding=1)
    self.cnn5 = torch.nn.Conv2d(in_channels=64, out_channels=128, kernel_size=[3,3], padding=1)
    self.cnn6 = torch.nn.Conv2d(in_channels=128, out_channels=256, kernel_size=[3,3], padding=1)
    self.pool = torch.nn.MaxPool2d(kernel_size=[3,3])
    self.Bnorm = torch.nn.BatchNorm2d(num_features=256)
    self.drop = torch.nn.Dropout2d(p=0.3)
    self.lin1 = torch.nn.Linear(256*7*7, out_features=512)
    self.lin2 = torch.nn.Linear(512, output)

  def forward(self, x):
    x = F.relu(self.cnn1(x))
    x = self.pool(F.relu(self.cnn2(x)))
    x = F.relu(self.cnn3(x))
    x = self.pool(F.relu(self.cnn4(x)))
    x = F.relu(self.cnn5(x))
    x = self.pool(F.relu(self.cnn6(x)))
    x = self.Bnorm(x)
    x = self.drop(x)
    x = x.view(-1, 256*7*7)
    x = F.relu(self.lin1(x))
    x = self.lin2(x)
    return x
    
model = SimpleCNN(3, 29)
</code></pre>
<p>My prediction (does this problem only for single image prediction)</p>
<pre><code class="lang-auto">#validation mode on torch, also there is a mode to turn the model into val() mode
correct = 0
total = 0
with torch.no_grad():
    for data in test_dataLoader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print('Accuracy of the network on the 20 test images: %d %%' % (
    100 * correct / total))
</code></pre>

username: ptrblck
cooked: <p>Could you replace this line:</p>
<pre><code class="lang-python">x = x.view(-1, 256*7*7)
</code></pre>
<p>with</p>
<pre><code class="lang-python">x = x.view(x.size(0), -1)
print(x.shape)
</code></pre>
<p>I guess you might not be using the same preprocessing pipeline and thus your images might have a different spatial shape during evaluation.<br>
Let me know, if that’s the case or if you still see an error.</p>

username: AnxheloD
cooked: <p>Hello, thanks for the reply, i solved the issue that same day, the problem was i changed my input image size and i had to reconfigure my linear laryer again. Thank you again</p>
Variable length matrix multiply leads to slow backward pass

username: Philipp_Hager
cooked: <p>Hey all,<br>
I try to perform a matrix multiplication of a batch of word vectors and specific attention weights. So I have a batch of e.g., 30 document, with 200 words and 512 embedding dimensions. Additionally, I have a matrix of attention weights for 16 different heads (4 in this example), which I want to multiply with every word in my document. The twist is that I want attention head (1-4) to only apply to specific dimensions of my word vectors, e.g., attention head 1 to dimensions 0-127, head 2 to dimensions 128-191 etc…</p>
<pre><code class="lang-auto">x = torch.randn(30, 200, 512)                   # Batch x Words x Embedding
attention = torch.randn(30, 4, 200, 200)        # Batch x Heads x Words Weights x Word Weights
</code></pre>
<p>The code I already came up with works and is reasonably fast in a forward pass, but incredibly slow in a backward pass.</p>
<pre><code class="lang-auto">start_dim = 0
num_heads = 4
head_dimensions = torch.tensor([128, 64, 32, 32])

for i in range(num_heads):
    end_dim = (start_dim + head_dimensions[i]).item()

    x[:, :, start_dim:end_dim] = torch.matmul(attention[:, i, :, :], x[:, :, start_dim:end_dim])

    start_dim = end_dim
</code></pre>
<p>Anyone an idea, why this is slow in backward propagation and how it could be solved more elegantly?</p>
Over fitting in ResNet 34

username: Praveen_G_B
cooked: <p>Hi Everyone,</p>
<p>I am training ResNet34 from scratch for three class classification on 23823 patches. I have tried using adam optimizer with different learning rates and reduce on LR plateau scheduler, but the training loss decreases, whereas the validation loss decreases for a while and then increases. I have used data augmentation such as random rotate and horizontal flip since I work on gray scale images, I cannot use more data augmentation operations. I have tried using weight decay also and as well as SGD optimizer, but the ResNet34 model overfits.</p>
<p>I have tried batch sizes of 4,16 and 32 also.<br>
I have attached the training and validation loss curve. Please suggest where I might be going wrong.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad5ad00db12011245970f415725673942a5ad566.png" data-download-href="https://discuss.pytorch.org/uploads/default/ad5ad00db12011245970f415725673942a5ad566" title="ResNet_apex_classification_mosaic_channel_complete"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad5ad00db12011245970f415725673942a5ad566_2_690x303.png" alt="ResNet_apex_classification_mosaic_channel_complete" data-base62-sha1="oJzaoIbCRnR5FrNpHZTdmJSTRiK" width="690" height="303" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad5ad00db12011245970f415725673942a5ad566_2_690x303.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad5ad00db12011245970f415725673942a5ad566_2_1035x454.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad5ad00db12011245970f415725673942a5ad566_2_1380x606.png 2x" data-dominant-color="FCFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ResNet_apex_classification_mosaic_channel_complete</span><span class="informations">1620×712 89.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: chetan06
cooked: <p>I am facing the same problem with resnet50 model. I am using the 32 batch size and had done some similar types of augmentations.</p>
Img should be PIL Image. Got <class 'torch.Tensor'>

username: chetan06
cooked: <pre><code class="lang-auto">test_transform = transforms.Compose({
    transforms.Resize(320),
    transforms.CenterCrop(250),
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.ToTensor(),
    normalize
})
</code></pre>
<pre><code class="lang-auto">testdata = DogsDataset(sub, '/content/drive/My Drive/Kaggle competitions/dog_breed dataset/test/',test_transform)
testloader = DataLoader(testdata,batch_size=24)
</code></pre>
<pre><code class="lang-auto">def test_submission(model):
    since = time.time()
    sub_output = []
    model.train(False)
    for inputs, labels in testloader:
        inputs = inputs.to(device)
        outputs = model(inputs)
        sub_output.append(outputs.data.cpu().numpy())
    sub_output = np.concatenate(sub_output)
    for idx,row in enumerate(sub_output.astype('float')):
        sub_output[idx] = np.exp(row)/np.sum(np.exp(row))
    output.loc[:,1:] = sub_output
    print()
    time_elapsed = time.time() - since
    print('Run complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
</code></pre>
<pre><code class="lang-auto">if torch.cuda.is_available():
    device = 'cuda'
else:
    device = 'cpu'
model = model.to(device)
test_submission(model)
</code></pre>
<p>I am implementing this test function for dog breed identification.</p>
<p>On running this, I am getting the following error and can’t get where I am wrong</p>
<pre><code class="lang-auto">
TypeError: img should be PIL Image. Got &lt;class 'torch.Tensor'&gt;
</code></pre>

username: cskarthik7
cooked: <p>Because you are using torch.Tensor() in transforms. Can you share your Dataoader function.</p>

username: chetan06
cooked: <p>I have not used custom dataloader</p>

username: chetan06
cooked: <p>I think I got my mistake. Actually I have done some problem with cpu and gpu.</p>
Profiling backward for inplace operations

username: seliad
cooked: <p>We want to profile backward time for in place operations, e g:<br>
a *= b<br>
For this we need to detach a, and set requires grad to True.<br>
However then it’s a leaf variable modified inplace which autograd doesn’t allow.</p>
<p>Is there any workaround?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m not sure to understand your goal?<br>
You want to benchmark the runtime of a single op ?</p>
<p>If so, you can do:</p>
<pre><code class="lang-auto">base = torch.rand(10, requires_grad=True)
a = base + 1 # The backward of this is an identity so as fast as it can get

a *= b

a.backward(a)
</code></pre>
<p>But note that you’re mostly going to measure the overhead of the autograd here.<br>
You can use the autograd profiler though to get a more precise idea without the autograd overhead.</p>

username: seliad
cooked: <p>Yes we want to profile runtime of single ops, when it makes sense.<br>
(and for everything in the computation graph, actually).</p>
<p>I assume that gradients for <code>mul</code> are already created in autograd and its just a matter of accumulating it (==copy? or something similar) in a leaf or not.<br>
Maybe for div() (or some other other in-place ops) its more notable.</p>
How can i introduce weights to my final loss

username: Mahmoud_Ghorbel
cooked: <p>hello,<br>
how can i introduce weights to my final loss and how can i learn the best weight?</p>
<pre><code>def criterion(triplet_y, softmax_y, labels):
    losses = [embedding_criterion(output, labels)[0] for output in triplet_y] + \
                 [xent_criterion(output, labels) for output in softmax_y] 
    loss = sum(losses)
    return loss</code></pre>

username: ptrblck
cooked: <p>If you use trainable weights on your last loss, the training would push them to a negative value, as it’s yielding the minimal loss, wouldn’t it?</p>
<p>Anyway, you could define a new parameter as:</p>
<pre><code class="lang-python">weight = nn.Parameter(torch.tensor(1))
</code></pre>
<p>and push it to the optimizer with all other parameters:</p>
<pre><code class="lang-python">optimizer = torch.optim.SGD(list(model.parameters()) + [weight], lr=1e-3)
</code></pre>

username: Mahmoud_Ghorbel
cooked: <p>maybe i did not explain well what i want to do<br>
im mean that i wanna have a weight for every loss like this :</p>
<pre><code class="lang-auto"> losses = W1 *  [embedding_criterion(output, labels)[0] for output in triplet_y] + \
                W2 *  [xent_criterion(output, labels) for output in softmax_y] 
</code></pre>
<p>but this is not working, it generates an error</p>

username: ptrblck
cooked: <p>What kind of error is it generating?</p>

username: Mahmoud_Ghorbel
cooked: <p>i did</p>
<pre><code class="lang-auto">0.2 *  [xent_criterion(output, labels) for output in softmax_y] + \
[xent_criterion(output, labels) for output in softmax_y] 
</code></pre>
<p>the error is<br>
[xent_criterion(output, labels) for output in softmax_y] + <br>
TypeError: can’t multiply sequence by non-int of type ‘float’</p>
<p>but i think that the problem is solved, i must do[ 0.2 * xent_criterion(output, labels) for output in softmax_y] + \</p>
<p>but still dont know how to learn the best valur of the weight, i just take 0.2 randomly</p>

username: ptrblck
cooked: <p>Ah OK, yes the error is thrown as you cannot multiply a scalar with a <code>list</code> directly.</p>
<aside class="quote no-group" data-username="Mahmoud_Ghorbel" data-post="5" data-topic="80452">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahmoud_ghorbel/48/23304_2.png" class="avatar"> Mahmoud_Ghorbel:</div>
<blockquote>
<p>but still dont know how to learn the best valur of the weight, i just take 0.2 randomly</p>
</blockquote>
</aside>
<p>I’m still unsure, if “learning” the weight would work, since it would just push it to a negative value.<br>
However, if you want to use the weight as a trainable parameter, refer to my code snippet from my previous post.</p>

username: Mahmoud_Ghorbel
cooked: <p>thank you for your help</p>
How to build this "simple" RNN architecture

username: pao.fanello
cooked: <p>Hi all,<br>
I’m not an expert of Pytorch and I’m trying to figure out how I can build this specific type of Neural Network Architecture.<br>
It can be identified as a Jordan Network, or maybe even as a Recursive Neural Network.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2d71a034cd922a4ed35f2f3304639fa54c60ab64.png" data-download-href="https://discuss.pytorch.org/uploads/default/2d71a034cd922a4ed35f2f3304639fa54c60ab64" title="The-Jordan-network-Output-activation-values-are-fed-back-to-the-input-layer-to-a-s-e"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2d71a034cd922a4ed35f2f3304639fa54c60ab64.png" alt="The-Jordan-network-Output-activation-values-are-fed-back-to-the-input-layer-to-a-s-e" data-base62-sha1="6u0VywvzZVtT8Om6KmfzCgKfrmY" width="511" height="500" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">The-Jordan-network-Output-activation-values-are-fed-back-to-the-input-layer-to-a-s-e</span><span class="informations">683×668 24 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
So, as you can see, as input layer we have an input vector and a state vector (concatenated) that is actually taken from the NN output of the previous time step. This means that for training I should provide only the starting state Xo and then a time series of inputs and the related time series of output to compare the predicitons.<br>
I would like it to be of arbitrary depth, as arbitrary should be the number of time steps to consider.<br>
The loss should be computed as a sum of the errors of each time step.<br>
So, concerning the training, is an RNN composed by a hidden number of multiple dense layers and a very specific state vector form, while, once trainded, for inference I will use it as a simple forward network.<br>
I don’t understand if can I use the Pytorch implementations of RNN, because in there the “state” is not of a specific form imposed by me, but depends on the NN architecture, and I cannot build a multilayer dense NN hidden inside the recurrence that in my case will happen only from output to input.<br>
Considering that there’s no “state” passing from time step to time step, but only a feedback of the output as input I don’t know if I can define a simple forward network and then during the training run it several times (once for each time step) considering the complete time series for the loss function.<br>
Thank you for your help.</p>
How to modify the dropout rate of pretained model pytorch

username: Wei_Wong
cooked: <p>how to modify the dropout rate of pretained model pytorch, for example mobilenetV2?<br>
Thank you in advance.</p>

username: ptrblck
cooked: <p>You could directly access the attributes of the dropout layer, e.g. via:</p>
<pre><code class="lang-python">model.classifier[0].p = 0.5
</code></pre>
Normalize Gradients of loss function w.r.t input before backpropagating

username: Deepayan137
cooked: <p>Hi,<br>
I have been trying to implement this <a href="https://arxiv.org/pdf/1903.00277.pdf" rel="nofollow noopener">paper</a> for the adversarial generation of handwritten images from a textual string.</p>
<p>My model comprises of two networks</p>
<ol>
<li>A CGAN architecture comprising of an encoder, Generator and a Discriminator. The encoder encodes the raw textual string into a high-dimensional feature vector while the decoder which is actually the generator takes the encoded string and a noise vector and tries to output fake word images. On the other hand, the discriminator is trained to tell real and fake images apart  which is the norm in a standard GAN architecture.</li>
<li>Besides, the above components my network also comprises of a pre-trained recognizer (OCR) module which takes an input image and outputs a string. The OCR is an attention based encoder-decoder network and the loss function used is NLL. The reason for providing a recognizer in addition to the discriminator is that I want the content of the image to be a word. The Discriminator alone will not work since it can only tell whether an image looks real or fake.</li>
</ol>
<p>Since there are two different architectures and two different loss functions, the gradients of each loss function w.r.t input vary in magnitude. As mentioned in the paper (section3.4) the norm of the gradient of NLL loss w.r.t to input is 10^2 time the gradient of discriminative loss w.r.t to the input. Thus I am not able to train the network properly.</p>
<p>The paper mentions a method for balancing the gradients of the recognizer w.r.t to the discriminator by multiplying it with a scalar such that both the gradients lie in the same range before back propagating it to the generator.</p>
<p>However, I am not able to understand as to how do I do this at an implementation level.</p>
<h2>Code</h2>
<pre><code class="lang-auto">   self.gen_opt.zero_grad()
   fake_images = self.gen.decode(decoder_input)
   loss_recog = self.nll_criterion(fake_images, targets)
   loss_gen = self.disc.calc_gen_loss(fake_images)
   loss_gen_total = loss_gen + los_recog
   loss_gen_total.backward()
   self.gen_opt.step()
</code></pre>
<p>I have tried to modify the gradients w.r.t to fake image by using the command <code>fake_images.grad()</code>. However it returns <code>None</code>. This is after I have invoked  <code>fake_images.requires_grad = True</code> and also `fake_images.retain_grad = True.</p>
<p>Any suggestions on how to modify the gradients of one loss w.r.t. to other will be highly appreciated.</p>
Pytorch:IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

username: sgaur
cooked: <p>It is discuss on many platforms but I am still not getting full understanding of this code.<br>
Below are the inputs.</p>
<p><strong>Number of classes = 5 (0,1,2,3,4)</strong><br>
<strong>Input Size  =  torch.Size([8, 3, 512, 512]) ## 8 is a batch-size</strong><br>
**Actual Labels size = torch.Size([8]) **<br>
<strong>predicted labels = torch.Size([8, 5])</strong></p>
<blockquote>
<p>epoch = 1<br>
from torch.autograd import Variable<br>
criteria = nn.CrossEntropyLoss()<br>
optimizer = torch.optim.Adam(params = model.parameters() , lr = .001)<br>
for n in range(epoch):<br>
for index,(images,label) in enumerate(train_loader):<br>
images = Variable(images.to(device))<br>
labels = Variable(labels.to(device))<br>
y_pred = model(images)<br>
print(‘images’,images.shape,‘labels’,labels.shape,‘y_pred’,y_pred.shape)<br>
loss = criteria(labels,y_pred)<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()<br>
<span class="hashtag-raw">#print</span>(images[0])<br>
<span class="hashtag-raw">#print</span>(labels[0])<br>
break;</p>
</blockquote>
<p>Please let me know if anyone need more Info.</p>

username: ptrblck
cooked: <p>Try to swap the <code>labels</code> and <code>y_pred</code> tensors when passing them to your criterion.<br>
If that doesn’t help, could you post the complete stack trace?</p>
<p>Also, <code>Variables</code> are deprecated since PyTorch <code>0.4</code>, so you should use tensors now. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: sgaur
cooked: <p>Wow.!! Thanks Bro it’s works.</p>
How do I use nn.CrossEntropyLoss() for seq2seq where my predication is of size (BS, seq_len, vocab_size) and truth of size (BS, seq_len)

username: KenW
cooked: <p>How do I use nn.CrossEntropyLoss() for seq2seq where my predication is of size (BS, seq_len, vocab_size) and truth of size (BS, seq_len), for example</p>
<pre><code class="lang-auto">predication = torch.randn(2, 3, 5, requires_grad=True) # (BS, seq_len, vocab_size)
target = torch.empty(2, 3, dtype=torch.long).random_(5) # (BS, seq_len)
</code></pre>
<pre><code class="lang-auto">predication: # size = (2, 3, 5)
tensor([[[-1.3824, -1.4598, -0.3210, -0.2991,  0.2965],
         [ 0.2591, -0.5094, -0.7029,  0.2963, -1.8912],
         [ 2.0020, -1.1158,  1.1687, -0.5815, -0.4416]],

        [[ 2.9818,  0.4093,  1.9568,  0.0664, -0.3604],
         [-0.6369, -0.3365, -1.3922, -0.6929, -0.1229],
         [ 0.6589, -1.3124, -2.0313, -1.4866, -1.8163]]], requires_grad=True)
</code></pre>
<pre><code class="lang-auto">target: # size = (2, 3)
tensor([[4, 3, 3],
        [3, 2, 1]])
</code></pre>
<p>i.e. [-1.3824, -1.4598, -0.3210, -0.2991,  0.2965] is the probabilities of words in my vocabulary for the first word in first batch first sequence, which predicts word with label(or index)=4 in the vocabulary and that is the same as the ground truth.</p>

username: ptrblck
cooked: <p>Try to permute the dimensions in your <code>predication</code> tensor to <code>[batch_size, nb_classes, seq_len]</code>, i.e. <code>[2, 5, 3]</code> and it should work.</p>
Will weights initialized code be executed again during training after being initialized?

username: Curimeowcat
cooked: <p>In the Actor Critic framework, suppose I have an actor like this(fan_in_initializer() is just a self-defined method),<br>
When I create an actor instance, line 10, 13 and 16 will be executed once, then if I train the model, will these lines of code be executed again?</p>
<pre><code class="lang-auto">class Actor(nn.Module):

    def __init__(self, s_dim, a_dim):
        super(Actor, self).__init__()

        self.s_dim = s_dim
        self.a_dim = a_dim

        self.fc1 = nn.Linear(s_dim, 512)
        self.fc1.weight.data = fan_in_initializer(self.fc1.weight.data.size()) # line 10

        self.fc2 = nn.Linear(512, 256)
        self.fc2.weight.data = fan_in_initializer(self.fc2.weight.data.size()) # line 13

        self.fc3 = nn.Linear(256, a_dim)
        self.fc3.weight.data.uniform_(-EPS,EPS) # line 16

    def forward(self, s):
        x = F.relu(self.fc1(s))
        x = F.relu(self.fc2(x))
        a = F.tanh(self.fc3(x))
        return a
</code></pre>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>No, the <code>__init__</code> method will only be called, when you create an instance of the model via:</p>
<pre><code class="lang-python">model = Actor(1, 1)
</code></pre>
<p>During training the <code>forward</code> method will be called.</p>
RNN working on CPU, shows loss but cannot send to GPU

username: bulb
cooked: <p>I’m new to NLP. I’m trying to use the Torch’s nn.Embeddings and nn.RNN().<br>
I noticed a strange thing while doing this. For some reason, my model trains on CPU. But I cannot send it to GPU for training.<br>
Here is the Model architecture:</p>
<p><span class="hashtag">#single</span> layer RNN.<br>
import torch.nn as nn<br>
import torch</p>
<p>class RNN(nn.Module):<br>
def <strong>init</strong>(self, ip_dim, emb_dim, hidden_dim, op_dim):<br>
super().<strong>init</strong>()<br>
self.embedding = nn.Embedding(ip_dim, emb_dim, padding_idx=0)<br>
self.rnn = nn.RNN(emb_dim, hidden_dim)<br>
self.fc = nn.Linear(hidden_dim, op_dim)<br>
def forward(self, text):<br>
text = torch.transpose(text, 0,1)<br>
emb = self.embedding(text)<br>
op, hidden = self.rnn(emb)<br>
out = self.fc(hidden)<br>
return out</p>
<p>Here is me instantiating the model:</p>
<p>emb_dim_len = 100<br>
hdim = 256<br>
net = RNN(ip_dim = len(v2i), # v2i  = vocab to index. dictionary with elements like: {“ice” : 1}<br>
emb_dim = emb_dim_len, hidden_dim = hdim,<br>
op_dim = len(Classes))</p>
<p>net.to(torch.device(‘cuda:0’))<br>
<span class="hashtag">#Throws</span> the following error:<br>
return self._apply(lambda t: t.cuda(device))</p>
<p><span class="hashtag">#RuntimeError:</span> CUDA error: device-side assert triggered</p>

username: Stacie_Davis
cooked: <p>Please use <em>‘preformatted text Ctrl-Shift-C’</em>  for posting codes.</p>
<p>Also, seems like <code>def init</code> might be first issue to resolve, make it dunder.</p>

username: ptrblck
cooked: <p>Were you able to use the GPU in the past?<br>
If so, are you able to create a simple tensor on the device via: <code>torch.randn(10, device='cuda')</code>?</p>
ValueError: not enough values to unpack (expected 2, got 0)

username: arezoo_moradi
cooked: <p>I want to load the NSL_KDD data set in python using this link:</p>
<p><a href="https://github.com/smellslikeml/deepIDS/blob/master/deep_IDS.ipynb%C2%A0" rel="nofollow noopener">https://github.com/smellslikeml/deepIDS/blob/master/deep_IDS.ipynb </a></p>
<p>But when I run this command, attack_map = {k:v for (k,v) in attack_map} I get this error:</p>
<p>ValueError: not enough values to unpack (expected 2, got 0)</p>
<p>hence I’m a newbie to Python, I don’t know how to fix this error. could you anyone help me?</p>
<p>with open(‘G:/RUN_PYTHON/kddcup.names.txt’, ‘r’) as infile:</p>
<p>kdd_names = infile.readlines() kdd_cols = [x.split(’:’)[0] for x in kdd_names[1:]]</p>
<p>kdd_cols += [‘class’, ‘difficulty’]</p>
<p>kdd = pd.read_csv(‘G:/RUN_PYTHON/KDDTrain+.txt’, names=kdd_cols)</p>
<p>kdd_t = pd.read_csv(‘G:/RUN_PYTHON/KDDTest+.txt’, names=kdd_cols)</p>
<p>kdd_cols = [kdd.columns[0]] + sorted(list(set(kdd.protocol_type.values))) + sorted(list(set(kdd.service.values))) + sorted(list(set(kdd.flag.values))) + kdd.columns[4:].tolist()</p>
<p>attack_map = [x.strip().split() for x in open(‘G:/RUN_PYTHON/training_attack_types.txt’, ‘r’)]</p>
<p>attack_map = {k:v for (k,v) in attack_map}</p>
<p>I know attack_map has empty elements.</p>
<p>when I printed the attack_map, print(attack_map),</p>
<p>out: [[‘back’, ‘dos’], [‘buffer_overflow’, ‘u2r’], [‘ftp_write’, ‘r2l’], [‘guess_passwd’, ‘r2l’], [‘imap’, ‘r2l’], [‘ipsweep’, ‘probe’], [‘land’, ‘dos’], [‘loadmodule’, ‘u2r’], [‘multihop’, ‘r2l’], [‘neptune’, ‘dos’], [‘nmap’, ‘probe’], [‘perl’, ‘u2r’], [‘phf’, ‘r2l’], [‘pod’, ‘dos’], [‘portsweep’, ‘probe’], [‘rootkit’, ‘u2r’], [‘satan’, ‘probe’], [‘smurf’, ‘dos’], [‘spy’, ‘r2l’], [‘teardrop’, ‘dos’], [‘warezclient’, ‘r2l’], [‘warezmaster’, ‘r2l’], []]</p>
<p>it has a [] element but I don’t how to can solved this problem.</p>
<p>best regards</p>

username: ptrblck
cooked: <p>You could filter out all empty values via:</p>
<pre><code class="lang-python">tmp = [['back', 'dos'],
       ['buffer_overflow', 'u2r'],
       ['ftp_write', 'r2l'],
       ['guess_passwd', 'r2l'],
       ['imap', 'r2l'],
       ['ipsweep', 'probe'],
       ['land', 'dos'],
       ['loadmodule', 'u2r'],
       ['multihop', 'r2l'],
       ['neptune', 'dos'],
       ['nmap', 'probe'],
       ['perl', 'u2r'],
       ['phf', 'r2l'],
       ['pod', 'dos'],
       ['portsweep', 'probe'],
       ['rootkit', 'u2r'],
       ['satan', 'probe'],
       ['smurf', 'dos'],
       ['spy', 'r2l'],
       ['teardrop', 'dos'],
       ['warezclient', 'r2l'],
       ['warezmaster', 'r2l'],
       []]
# filter out empty lists
tmp = [a for a in tmp if a != []]
attack_map = {k:v for (k, v) in tmp}
</code></pre>

username: arezoo_moradi
cooked: <p>@ ptrblck, thanks to your response</p>
What are the effects of backproping on a truncated tagret/output?

username: Ge0rges
cooked: <p>In my training loop what would the affects of truncating the model output and target vectors as so:</p>
<pre><code class="lang-auto">action_output = model(inputs)
         
# The effects of the below two lines
action_output = action_output[:, tasks]   # Tasks is an array of 1*"target size"
action_target = action_target[:, tasks]

action_loss = criterion(action_output, action_target)
        
action_loss.backward()
</code></pre>
<p>What I believe it is doing is that it is truncating the computation graph and backproping only on weights that affect that output node.</p>

username: ptrblck
cooked: <p>Yes, that should be the case, which would create a zero gradient for the last linear layer:</p>
<pre><code class="lang-python">model = models.resnet18()
data = torch.randn(2, 3, 224 , 224)
target = torch.randn(2, 1000)
criterion = nn.MSELoss()

output = model(data)
output = output[:, :100]
target = target[:, :100]
loss = criterion(output, target)
loss.backward()

print(model.fc.weight.grad[100:].abs().sum())
&gt; tensor(0.)
</code></pre>
<p>Note that the preceding layers would have valid gradients, as they are not directly connected to the output.</p>