akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CUDA memory error if I don't call .backward()

Dishank_Bansal: <p>I am getting CUDA out of memory error running following.</p>
<pre><code class="lang-auto">def train(self, epochs, saving_freq):
        assert self.model.train
        while self.epoch &lt;= epochs:
            running_loss = {}

            for idx, batch_sample in enumerate(self.train_loader):
                print("Iteration:", idx)
                loss_dict = self.train_step(batch_sample)
                
                if idx%10==0:
                     with torch.no_grad():
                    #----------- Logging and Printing ----------#
                    print("{:&lt;8d} {:&lt;9d} {:&lt;7.4f}".format(self.epoch, idx, loss_dict['tot_loss'].item()))
                    for loss_name, value in loss_dict.items():
                        self.tb_writer.add_scalar('Loss/'+loss_name, value.item())
</code></pre>
<p>where <code>train_step</code> is defined as:</p>
<pre><code class="lang-auto">    def train_step(self, batch_sample):
        in_images = batch_sample['image'].to(self.device)
        target = [x.to(self.device) for x in batch_sample['target']]

        rpn_proposals, instances, rpn_losses, detection_losses = self.model(in_images, target, self.is_training)
        print([len(x) for x in instances])
        
        loss_dict = {}
        loss_dict.update(rpn_losses)
        loss_dict.update(detection_losses)
        
        loss = 0.0
        for k, v in loss_dict.items():
            loss += v
        loss_dict.update({'tot_loss':loss})

        self.optimizer.zero_grad()
        # loss.backward()
        self.optimizer.step()

        return loss_dict
</code></pre>
<p>Weird thing is that if I do any of the following 3, the memory error goes away:</p>
<ol>
<li>uncomment loss.backward() - I understand that this frees memory graph, but even in my case, at each iteration, previous memory allocation should be freed itself.</li>
<li>I change if <code>idx%10==0</code>: <code>to idx%1==0</code> in function <code>train()</code> . ie instead of doing it every iteration intead of every 10 iteration</li>
<li>If I comment this line:</li>
</ol>
<pre><code class="lang-auto">
for loss_name, value in loss_dict.items():
        self.tb_writer.add_scalar('Loss/'+loss_name, value.item())
</code></pre>
<p>So I guess above line is not allowing to free the memory. BTW, this line is just logging to tensorboard.</p>
<p>Let me know if you need an error log.</p>
How to debug on macOS

alanzhai219: <p>On Ubuntu, I can run “gdb --args python my_script.py” and set breakpoints in C++ ops. And then it works well.<br>
On macOS, I have built a debug version on macOS successful<br>
ly.<br>
export DEBUG=1<br>
MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install<br>
But, I don’t know how to set bps in cpp codes. Anyone can help me?<br>
Thanks.</p>
Register_hook lead to cuda out of memory

MrBeandou: <p>when I use register_hook to try to print some layer’s info of gradient, when I do not use torch.nn.DataParallel(), no gpu memory leak happen ,however when I try to use torch.nn.DataParallel(). The cuda:0 card’s memory is increasing after running register_hook function and making inference the network, what is the reason of it and how to solve it ,can anyone help?<br>
pytorch edition 1.2.0  python 2.7 cuda 10.0</p>
<p>class CaffeModel(nn.Module):<br>
def <strong>init</strong>(self, basemodel_protofile, height=384, width=128):<br>
super(CaffeModel, self).<strong>init</strong>()</p>
<pre><code>    self.base = CaffeNet(basemodel_protofile, width=width, height=height, omit_data_layer=True, phase='TRAIN')


def forward(self, x, key_, gradkeylt = [], normalize = True, test = False):
    x = self.base(x)
    y = x[key_]
    if normalize:
        y = F.normalize(y, p=2, dim=1)
    if not test:
        y = y * 200
    for gradkey in gradkeylt:
        if gradkey in x.keys() and x[gradkey].device.index == 0:
            ipdb.set_trace()
            print(gradkey)
            x[gradkey].register_hook(self.bh)
    return y

def load_checkpoint_for_basemodel(self, checkpoint):
    print("Loading checkpoint ....")
    model_dict = self.base.state_dict()
    pretrained_dict = torch.load(checkpoint)
    new_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict.keys()}
    model_dict.update(new_dict)
    print('Total : {}, update: {}'.format(len(pretrained_dict), len(new_dict)))
    self.base.load_state_dict(model_dict)
    print("loaded finished!")

def bh(self, grad_):
    grad_.detach_()
    print("max:%.8f, min:%.8f, std:%.8f, mean:%.8f" % (grad_.abs().max().item(), grad_.abs().min().item(), grad_.abs().std().item(), grad_.abs().mean().item()))
    del grad_</code></pre>

Umair_Javaid: <p>running into same problem. Any solutions?</p>
What is a difference between Loss Function and Evaluation Matrix

sgaur: <p>What is a difference between Loss Function and Evaluation Matrix also why we cannot use evaluation matrix as a Loss functions.</p>
<p>I know it is a pretty  noob question.</p>
<p>Thanks.</p>

ptrblck: <p>A loss function is used to train the model and has usually “good” properties to calculate the gradients.<br>
E.g. you could use <code>torch.argmax</code> on out model output, which would give you the predicted label.<br>
However, you wouldn’t be able to train your model with it, as <code>argmax</code> has almost everywhere a zero gradient. That’s why you would fall back to a soft argmax method (<code>softmax</code>) to use this as a proxy loss function.</p>
How to calculate class accuracy in multi label problem

netaglazer: <p>hi,<br>
i have a multi label problem, and i want to calculate the accuracy for each class<br>
i have a cnn model<br>
how can i do it with pytorch?</p>

ptrblck: <p>You could calculate the confusion matrix and calculate the per-class accuracy and other metrics.<br>
Once you have the predictions and targets, you could use e.g. <code>sklearn.metrics</code> for it or implement it from scratch (we have some posts in this forum on how to calculate a confusion matrix and other metrics based on it).</p>
Is it possible to go from layer to layer in tranformers Bert

margemi: <p>I was training to understand how the code of Bert from transformers library works so I compared the output of the following codes:</p>
<pre><code class="lang-auto">self.bert = BertModel.from_pretrained('bert-base-uncased')
</code></pre>
<p><strong>code1</strong></p>
<pre><code class="lang-auto"> _ , pooled_output = self.bert(input_ids, token_type_ids, attention_mask)
</code></pre>
<p><strong>code2</strong></p>
<pre><code class="lang-auto">x = self.bert.embeddings(input_ids, token_type_ids, attention_mask)
    for i in range(0, 12):
        x = self.bert.encoder.layer[i](x)[0]
    pooled_output = self.bert.pooler(x)
</code></pre>
<p>However, the output of the two methods is not the same. (I used model.eval() so the dropout layer will not affect the output)</p>
<p>Finally, I should mention that I tried to fix the seed so I will get the same results by</p>
<pre><code class="lang-auto">seed_val = 42
random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)
</code></pre>
<p>Did I miss something?</p>

ptrblck: <p>I assume you are using <code>BertModel</code> from the Huggingface repository?<br>
If so, then you could compare all intermediate outputs to the <a href="https://huggingface.co/transformers/_modules/transformers/modeling_bert.html#BertModel.forward">original forward</a> method and check, where the difference occurs in your manual calculation.</p>
Understanding AdderNet model construction coding

promach: <p>What is the purpose of the <a href="https://github.com/huawei-noah/AdderNet/blob/master/resnet20.py#L65-L79" rel="nofollow noopener">following code snippet</a> in <a href="https://arxiv.org/pdf/1912.13200.pdf#page=3" rel="nofollow noopener">AdderNet</a> ?</p>
<pre><code class="lang-auto">    def _make_layer(self, block, planes, blocks, stride=1):
        downsample = None
        if stride != 1 or self.inplanes != planes * block.expansion:
            downsample = nn.Sequential(
                adder.adder2d(self.inplanes, planes * block.expansion, kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(planes * block.expansion)
            )

        layers = []
        layers.append(block(inplanes = self.inplanes, planes = planes, stride = stride, downsample = downsample))
        self.inplanes = planes * block.expansion
        for _ in range(1, blocks):
            layers.append(block(inplanes = self.inplanes, planes = planes))

        return nn.Sequential(*layers)
</code></pre>

ptrblck: <p>This code snippet seems to be similar to the <a href="https://github.com/pytorch/vision/blob/7b60f4db9707d7afdbb87fd4e8ef6906ca014720/torchvision/models/resnet.py#L177-L199"><code>_make_layer</code></a> method used in the resnet creation.</p>
<p>It’s accepting a few arguments, e.g. the desired <code>block</code> to use in these layers, and creates an <code>nn.Sequential</code> container out of these arguments.</p>
<p>In a torchvision resnet, you could print one of these blocks e.g. via <code>print(model.layer4)</code> to see which submodules were created.</p>
How can I get .grad ? .grad is None

Yuki_Takezawa: <p>I want to get d(model(x))/dx, but following program returns None.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3) # 28x28x32 -&gt; 26x26x32
        self.conv2 = nn.Conv2d(32, 64, 3) # 26x26x64 -&gt; 24x24x64 
        self.pool = nn.MaxPool2d(2, 2) # 24x24x64 -&gt; 12x12x64
        self.dropout1 = nn.Dropout2d()
        self.fc1 = nn.Linear(12 * 12 * 64, 128)
        self.dropout2 = nn.Dropout2d()
        self.fc2 = nn.Linear(128, 1)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.dropout1(x)
        x = x.view(-1, 12 * 12 * 64)
        x = F.relu(self.fc1(x))
        x = self.dropout2(x)
        x = self.fc2(x)
        return x
</code></pre>
<pre><code class="lang-auto">model = Net()
input = torch.randn((1, 1, 28, 28), requires_grad=True)
y = model(input)
y.backward()
print(x.grad) # This is None.
</code></pre>
<p>How can I get this grad ?</p>

kshitij: <p>Should that <code>print(x.grad)</code> be <code>print(input.grad)</code>?</p>
<p>I tried the following and it worked.</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
from torch.nn import functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3) # 28x28x32 -&gt; 26x26x32
        self.conv2 = nn.Conv2d(32, 64, 3) # 26x26x64 -&gt; 24x24x64 
        self.pool = nn.MaxPool2d(2, 2) # 24x24x64 -&gt; 12x12x64
        self.dropout1 = nn.Dropout2d()
        self.fc1 = nn.Linear(12 * 12 * 64, 128)
        self.dropout2 = nn.Dropout2d()
        self.fc2 = nn.Linear(128, 1)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.dropout1(x)
        x = x.view(-1, 12 * 12 * 64)
        x = F.relu(self.fc1(x))
        x = self.dropout2(x)
        x = self.fc2(x)
        return x

model = Net()
input = torch.randn((1, 1, 28, 28), requires_grad=True)
y = model(input)
y.backward()
print(input.grad)
</code></pre>

Yuki_Takezawa: <p>Sorry, It’s stupid mistake.<br>
Thank you.</p>
Model update after a complete loss.backward()

Bruce_zhuang: <p>I would like to ask a question that I thought Pytorch has taken care of, but it has not.</p>
<p>Well, the main loop for training a network is like</p>
<pre><code class="lang-auto">outputs = model(images)
loss = F.cross_entropy(outputs, labels)
optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>I thought that the parameter update process (<code>optimizer.step()</code>) was happening during the calculation of the gradients (<code>loss.backward()</code>). Give a 3-layer network for example. When the gradients in layer 3 have been calculated, the update could happen right after that, which is in parallel with the calculation of the gradients in layer 2. Instead, <code>optimizer.step()</code> happens after the completion of <code>loss.backward()</code>. This leads to some efficiency drop. As a matter of fact, when I remove <code>optimizer.step()</code>, the GPU utilization is improved (say 70%-&gt;80%). My theory is that <code>optimizer.step()</code> keep the network training idle for quite some time, and the GPU becomes hungry.</p>
<p>So, is the parallel process of updating parameter and calculating the gradients a difficult thing, or Pytorch just chooses not to do this for some reason? Thank you!</p>

ptrblck: <p>I think the main design decision for this behavior is, because the gradient calculation and the update steps are independent from each other.<br>
E.g. you can easily accumulate gradients with multiple forward/backward passes, if a larger batch doesn’t fit in your memory.<br>
Also, you might work with multiple losses, which will use their own <code>.backward()</code> call.</p>
<p>Forcing the <code>optimizer.step()</code> method to kick in right after the first gradient was calculated seems like a huge limiting factor.</p>

Bruce_zhuang: <p>Yes. seems like a bit too wrapped up in that way. But we can still improve the efficiency a bit by manually doing the simultaneous update right (in the simplest case in my question)?</p>
Why Testing a CNN is taking a lot of memory?

nas: <p>Hi,</p>
<p>I am training a very simple CNN. While training everything goes well, but when it comes to testing I am getting <em>Runtime error</em> for not having enough memory! I am new in using PyTorch! I am posting the testing portion of the code below, any help will be highly appreciated!</p>
<pre><code class="lang-auto">loss_function = nn.MSELoss()
net.eval()
Variable_store_3 = np.empty((0,1)) 
Variable_store_1 = []  
for epoch in range(1):
    running_loss = 0
    Variable_store_2 = []
    for batch in test_dataloader:
        
        x, y = batch
        outputs = net(x)
        Variable_store_1.extend(outputs)   #for plotting purposes
        Variable_store_2 .extend(y)            #for plotting purposes
        outputs_np = outputs.detach().numpy()
        Variable_store_3 = np.append(Variable_store_3, outputs_np, axis = 0)
        loss = loss_function(outputs, y)
        running_loss += loss.item() * 128  #Batch_size
        
    final_loss = math.sqrt((running_loss / len(test_dataset)))
    print(f"{epoch+1} epoch | testing loss = {final_loss}") 
</code></pre>

ptrblck: <p>Could you wrap your testing code into a <code>with torch.no_grad()</code> block?<br>
This would avoid storing the intermediate tensors, which would be needed to calculate the gradients in the backward pass.</p>
<p>Also, you might want to define a training and validation function, if that’s not already the case.<br>
Since Python uses function scoping, some variables from the training might be freed additionally.</p>

simaiden: <p>Hoy much size is your data? I think that’s happen because you are loading all the outputs to <code>Variable_store_3</code>, and all this data can’t fit in memory. Also try using torch.no_grad(),  this means taht you are not saving the gradients to use backprop (because you are not training)</p>
<pre><code class="lang-auto">for batch in test_dataloader:
    with torch.no_grad():
        x, y = batch
        outputs = net(x)
        Variable_store_1.extend(outputs)   #for plotting purposes
        Variable_store_2 .extend(y)            #for plotting purposes
        outputs_np = outputs.detach().numpy()
        Variable_store_3 = np.append(Variable_store_3, outputs_np, axis = 0)
        loss = loss_function(outputs, y)
        running_loss += loss.item() * 128  #Batch_size
        
</code></pre>

nas: <p>Thank you very much for replying and for the solution.</p>
<p>As you have advised, I have wrapped my testing code with <code>torch.no_grad()</code>. I just did a small training and with that the testing. And it seems the problem is solved! But, now I will be doing longer training and then testing and will post the update here.</p>
<p>Again, thank you, very much</p>

nas: <p>Thank you for your reply and for the solution.</p>
<p>The size of my testing data was 6080000.<br>
Then downsampled into 23850.<br>
and torch.Size([23750, 100, 1]) was this.</p>
Delete model.pt from asset folder ("Hello World Android App")

juvegimmy: <p>I tried to run the “Hello World” Android application and everything goes correctly.<br>
I deleted the model.pt file from the assets folder and the application continues to run correctly. How is it possible that the model was found if it was deleted?<br>
Sorry for the stupid question and for my bad English</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/601f97e0da77b21c95cddcca6b09eed84cd11e5c.png" data-download-href="https://discuss.pytorch.org/uploads/default/601f97e0da77b21c95cddcca6b09eed84cd11e5c" title="Schermata 2020-04-18 alle 23.43.57"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/601f97e0da77b21c95cddcca6b09eed84cd11e5c_2_690x265.png" alt="Schermata 2020-04-18 alle 23.43.57" data-base62-sha1="dIlwzNyvtFEG8rIwu4M3gaYgJWs" width="690" height="265" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/601f97e0da77b21c95cddcca6b09eed84cd11e5c_2_690x265.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/601f97e0da77b21c95cddcca6b09eed84cd11e5c_2_1035x397.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/601f97e0da77b21c95cddcca6b09eed84cd11e5c_2_1380x530.png 2x" data-dominant-color="353637"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Schermata 2020-04-18 alle 23.43.57</span><span class="informations">2502×962 334 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

simaiden: <p>Cause the java app has builded before you deleted the file</p>

juvegimmy: <p>so if I want to change the model what should I do to delete the old one?</p>

simaiden: <p>Not only re-run the app but also rebuild it</p>

juvegimmy: <p>I’ve already done it and it doesn’t work. I even tried to delete the app and download it again, immediately delete the model.pt and run it for the first time, incredibly it works. I am desperate</p>
Lstm - minimal example issue

joekid: <p>Hi folks,<br>
After reading some tutorials, I did a minimal example aiming to classify (binary) an input sequence:</p>
<pre><code class="lang-auto">class LSTM_Seq(nn.Module):
    def __init__(self, input_size, hidden_size, n_layers, output_size):
        super().__init__()
        
        self.hidden_size = hidden_size
        self.n_layers = n_layers                
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers=n_layers, bidirectional=False)

        self.linear = nn.Linear(hidden_size, output_size)
                
    def init_hidden(self, n_layers, batch_size, hidden_size):
        return (torch.zeros(n_layers, batch_size, hidden_size),   
                torch.zeros(n_layers, batch_size, hidden_size) )
    
    def forward(self, x):        
        # input should be (sequence, batch, number_of_features)
        x = x.permute(1, 0, 2)
        mini_batch = x.size(1)
        self.hidden = self.init_hidden(self.n_layers, mini_batch, self.hidden_size)
        out, self.hidden = self.lstm(x, self.hidden)        
        out = self.linear(out[-1])        
        out = F.log_softmax(out, dim=1)
        return out

model = LSTM_Seq(input_size=2, hidden_size=100, n_layers=1, output_size=2)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

# %% 
torch.manual_seed(123)

# 10000 samples of  sequence dims= 500, number of features=2
x_train = torch.rand(10000, 500, 2).float()
y_train = (torch.rand(10000) &gt; 0.4).long()

batch_size = 32
train_dataset = torch.utils.data.TensorDataset(x_train, y_train) 
train_dataloader = torch.utils.data.DataLoader(train_dataset, 
                                               batch_size=batch_size, shuffle=True) # create your dataloader

total_train_batch = len(train_dataloader)
# %%  
for epoch in range(1):
    running_loss = 0.0
    
    iter_train_dataset = iter(train_dataloader)    
    for k in range(total_train_batch):    
        seqs, labels = next(iter_train_dataset)
        
        optimizer.zero_grad()        
        outputs = model(seqs)
        loss = F.nll_loss(outputs, labels)
        loss.backward()
        optimizer.step()
    
        running_loss += loss.item()
        if k % 50 == 0:    # print every  mini-batches
            print('[%d, %3d] loss: %.3f' %
                      (epoch + 1, k + 1, running_loss / 50))
            running_loss = 0.0
</code></pre>
<p>However, the model seems not learning, it shows:</p>
<pre><code class="lang-auto">[1,   1] loss: 0.014
[1,  51] loss: 0.673
[1, 101] loss: 0.668
[1, 151] loss: 0.665
[1, 201] loss: 0.677
[1, 251] loss: 0.674
[1, 301] loss: 0.680
</code></pre>
<ol>
<li>
<p>I verified the <code>loss</code> has a gradient function, the loss should be reduced in some sense.</p>
</li>
<li>
<p>How the hidden states works, how can I modify my code in order to get</p>
</li>
</ol>
<blockquote>
<p>transferred between timestep to timestep</p>
</blockquote>
<p>as said by <a href="https://discuss.pytorch.org/u/smth">smth</a> and <a href="https://discuss.pytorch.org/u/Danya">Danya</a> in another <a href="https://discuss.pytorch.org/t/when-to-initialize-lstm-hidden-state/2323/7">forum</a>.</p>
Minibatch implementation in pytorch for object detection

kbtorcher: <p>I am using a Faster R-CNN model with Resnet-50 and FPN backbone. These are the parameters relevant to this question:</p>
<ol>
<li>Mini-batch size = 6 RGB images</li>
<li>Number of GPUs = 1</li>
<li>Input mini-batch tensor size = torch.Size([6, 3, 768, 1184])</li>
</ol>
<p>According to the Resnet-50 architecture, the first convolution layer is defined as follows: <code>.conv1 = Conv2d( 3, 64, kernel_size=7, stride=2, padding=3, bias=False, norm=get_norm(norm, out_channels), )</code></p>
<p>After the model runs this first convolution, conv1 tensor size is torch.Size([6, 64, 384, 592]).</p>
<p>I have four questions:</p>
<ol>
<li>
<p>As per conv1 tensor size, do we have 64 feature maps for each image or are they averaged and shared over the 6 images?</p>
</li>
<li>
<p>How does the convolution process takes place between the input and conv1, given that we have 6 images?</p>
</li>
<li>
<p>Do each of the 64 filters apply over RGB channels of an image? Or do they apply directly over the colored pixel values without splitting into respective channels?</p>
</li>
<li>
<p>I understand that at the end of a forward pass, the loss and gradients are averaged by mini_batch_size. Are the weights updated for each image iteratively?</p>
</li>
</ol>

ptrblck: <ol>
<li>
<p>For each image 64 output channels will be created. The activation output shape of the conv layer will be <code>[batch_size, out_channels, height, width]</code>.</p>
</li>
<li>
<p>The majority of PyTorch operations work on batched data, which will apply the desired operation on each sample in a vectorized way.</p>
</li>
<li>
<p>If you are using the default setup with <code>groups=1</code>, each filter kernel will use all input channels. Have a look at <a href="https://cs231n.github.io/convolutional-networks/#conv">CS231n - Convolutions</a> for a good explanation. I’m not sure, what the <code>norm</code> argument does and it seems to be a custom implementation.</p>
</li>
<li>
<p>The <code>backward()</code> call only calculates the gradients. The parameter updates will be performed by <code>optimizer.step()</code>, which uses the calculated <code>.grad</code> attributes in all parameters to update them.</p>
</li>
</ol>
How to sample feature maps?

wayne: <p>I want to sample feature maps in oder to select some ‘important’ channels. But this ‘sample’ operation(e.g. dropout some channels of feature maps, shown as figure below) is not differentiable and when backpropagation it will be wrong.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da5e5f62c502a19d2e4630296996e385a8f3766b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/da5e5f62c502a19d2e4630296996e385a8f3766b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da5e5f62c502a19d2e4630296996e385a8f3766b_2_690x219.jpeg" alt="image" data-base62-sha1="v9MhpRKx1wgX3Gma2OIFfojCzaP" width="690" height="219" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da5e5f62c502a19d2e4630296996e385a8f3766b_2_690x219.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da5e5f62c502a19d2e4630296996e385a8f3766b_2_1035x328.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da5e5f62c502a19d2e4630296996e385a8f3766b_2_1380x438.jpeg 2x" data-dominant-color="EAEFEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1422×452 87.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Which operation are you using at the moment and why is it not differentiable?</p>
How can the computation graph be Immutable and Dynamic

murph213: <p>I am reading the (pre-print) paper on autograd <a href="https://openreview.net/pdf?id=BJJsrmfCZ" rel="nofollow noopener">[Paszke et. al.]</a>.  In that paper, both these claims are made: (1) PyTorch creates a <em>dynamic</em> computation graph and (2) the graph is <em>immutable</em>.</p>
<p>I cannot fully reconcile how these two statements can both be true.  Could someone please elaborate and help me understand?</p>
<p>I have a basic understanding of what mutability means – at least in the context of basic Python.  I also have a basic understanding that PyTorch’s graph is define-by-run, as opposed to TensorFlow etc.</p>

googlebot: <p>“dynamic” refers to creation (emphasizing <em>runtime</em> creation), “immutable” - to further modifications (immutable nodes may be implied). And this graph is almost invisible to user code btw.</p>

murph213: <p>OK, I think this makes sense.  What I conclude is that: (1) each sequence of forward steps (say at different epochs) dynamically builds up a computation graph, and (2) the nodes of such a graph are immutable.  In other words, multiple graphs may be built up throughout program execution, but once a node is entered in such a graph, that node is treated as a mutable object.</p>
<p>This also fits with their comment that in-place objects are handled by “rebasing” pointers within the computation graph (which is different from editing the node itself).</p>
Custom Differentiation of Loss function with Autograd

jmaronas: <p>Hello.</p>
<p>As far as I now <code>loss.backward()</code> only works if <code>loss</code> is a single scalar value. I am considering a loss function that is not implemented in pytorch <a href="https://arxiv.org/pdf/1905.09670.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1905.09670.pdf</a> (equation 5) and I want to avoid numerical problems.</p>
<p>I know that a way to stabilize the softmax + log score loss is to compute the derivative directly (as Bishop’s book states) and then backpropagate that derivative, instead of defining the loss using elementary functions and then use automatic differentiation.</p>
<p>I would like to do something similar in this problem. However, the problem is that if I manually compute the derivative of this loss w.r.t the logits, then I end up having a vecor/matrix instead of a single scalar value, so I can no use <code>.backward()</code> anymore. How should I proceed in this case, i.e, is there an automatic way to backpropagate gradients of a vector w.r.t to all the inputs to the graph?</p>
<p>Thanks, Juan</p>

ptrblck: <p><code>backward()</code> accepts a gradient as its argument.<br>
If you don’t pass anything to this method, a scalar 1 is automatically used.<br>
If you already have the gradient w.r.t the tensor, you could thus use <code>loss.backward(gradient)</code>.</p>

jmaronas: <p>I have reviewed the documentation and have one more question. So lets say that <code>gradient</code> has my craft derivatives of the probability vector w.r.t logits that I have to backpropagate. I guess the correct call now would be:</p>
<pre><code class="lang-auto">gradients.backward(torch.ones_like(gradients))
</code></pre>
<p>Is this equivalent to do <code>gradients.sum().backward()</code>? I guess it is.</p>

ptrblck: <p>This should create the same gradients.<br>
I’ve used this simple dummy snippet to verify it:</p>
<pre><code class="lang-python"># Setup
torch.manual_seed(2809)

model = nn.Sequential(
    nn.Conv2d(3, 6, 3, 1, 1),
    nn.ReLU(),
    nn.Conv2d(6, 1, 3, 1, 1),
    nn.ReLU(),
    nn.Flatten(),
    nn.Linear(1*24*24, 10)
)

x = torch.randn(1, 3, 24, 24)

out = model(x)

# Switch here
out.backward(torch.ones_like(out))
#out.sum().backward()

for p in model.parameters():
    print(p.grad.abs().sum())
</code></pre>
Why my Validation accuracy changes each time when I execute. Any problem with the DataLoader(I am using a Windows Machine)

Abishek_Kamal: <pre><code class="lang-auto">import glob
from PIL import Image
import torch

import torch.nn as nn
from torchvision import datasets, transforms

classes=('with_mask','without_mask')

class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.cnn_model = nn.Sequential(

            nn.Conv2d(3, 6, 5),  # (N, 3, 224, 224) -&gt; (N,  6, 220, 220)
            nn.Tanh(),
            nn.AvgPool2d(2, stride=2),  # (N, 6, 220, 220) -&gt; (N,  6, 110, 110)
            nn.Conv2d(6, 16, 5),  # (N, 6, 110, 110) -&gt; (N, 16, 106, 106)
            #nn.BatchNorm2d(16),
            nn.Tanh(),
            nn.AvgPool2d(2, stride=2)  # (N,16, 106, 106) -&gt; (N, 16, 53, 53)
        )
        self.fc_model = nn.Sequential(
            nn.Linear(44944, 120),  # (N, 400) -&gt; (N, 120)
            #nn.Dropout(0.2),
            nn.Tanh(),
            nn.Linear(120, 84),  # (N, 120) -&gt; (N, 84)
            nn.Tanh(),
            nn.Linear(84, 2)  # (N, 84)  -&gt; (N, 2)
        )

    def forward(self, x):
        x = self.cnn_model(x)
        x = x.view(x.size(0), -1)  # -1, in_features x.size(0), -1
        x = self.fc_model(x)
        return x

model=torch.load("models/mask_model_lenet_20epoch.pth")
model=model.eval()



def evaluation(dataloader):
    total, correct = 0, 0
    for data in dataloader:
        inputs, labels = data
        outputs = model(inputs)
        _, pred = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (pred == labels).sum().item()
    return 100 * correct / total


test_dir = 'data/Test/'
test_transform = transforms.Compose([
        #transforms.Resize(256),
        transforms.RandomSizedCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225])])

testset = datasets.ImageFolder(test_dir,transform=test_transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=32, shuffle=False)

print(evaluation(testloader))
</code></pre>

Aditya_Kumar: <p>t should not change a lot. But a little bit of change is due to <code>transforms.RandomSizedCrop</code>.</p>
Is there a way to make a 2 steps classification training?

AndreTeixeira: <p>I’m training a model to classify 14 different plants using resnet101, but 3 of them are decreasing the accuracy of my model. I’ve trained a model especific for classify these 3 plants and then i got a good precision on it. Is there a way to use this new “3 problematic plants model” in the original 14 plants classification model in order to get a bigger accuracy?</p>

ptrblck: <p>You could try to create a staged classifier, where the first level would classify each sample into the “easy” and “hard” classes, and call the corresponding model.<br>
However, you would of course have to retrain this model and would have to compare it to the standard approach of a single model.</p>
<p>I would guess that a single model with some tweaks (regularization, more aggressive data augmentation) would work better.</p>

AndreTeixeira: <p>Thanks! I’ll try the first solution and i will continue to do my best in the process of regularization and data augmentation, even though i still haven’t gotten the results i expected.</p>
Loss function that is weighted by the label

luciolis: <p>Hi !</p>
<p>I want to make a loss function that depends on the label (some dimensions are somehow co-linear). For example, I would like to weight the loss of feature 2 by the norm of feature 1. The issue here is that autograd says that it cannot compute the gradient</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [20, 224, 224]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>for the code</p>
<pre><code class="lang-auto">    outputs[:, 1] *=  labels[:,0]
    labels[:, 1] *= labels[:,0]

    loss_parameters = F.smooth_l1_loss(outputs, labels)
</code></pre>
<p>Do you have any idea on how to make that happen ?</p>
<p>Thanks<br>
Adrian</p>

tom: <p>You need to avoid the inplace operation.<br>
If the size along the second dimension is 2, you could use</p>
<pre><code class="lang-python">mult = torch.stack([torch.ones_like(labels[:,0]), labels[:, 0]], dim=1)
outputs = outputs * mult
labels = labels * mult
</code></pre>
<p>or so.</p>
<p>Best regards</p>
<p>Thomas</p>

luciolis: <p>Great ! I could copy labels[:,0] to mult to make it not requiring the gradient. Thanks</p>
<p>Finally I did this:</p>
<pre><code class="lang-auto">    loss_parameters = F.smooth_l1_loss(outputs, labels, reduction='none')
    loss_parameters[:, 1, :, :] *= labels[:, 0, :, :]
</code></pre>
<p>It’s slightly different, but since it’s a L1 norm, it’s almost the same.</p>
Softmax and Cross Entropy vs Log Softmax and NLLLoss

No_No: <p>Hi, I am observing some weird behaviour.</p>
<p>I have made a classifier and I have tried two different output and loss combinations ; 1) Softmax and Cross Entropy and 2) Log Softmax and NLLLoss</p>
<p>When I run them both, they will both have an initial loss of 1.38, but the loss of the logsoftmax and nllloss will continue all the way down to 0.25 where as in softmax and crossentropy it will stop around 0.9.</p>
<p>Even weirder is that they both perform the same on the test set.</p>
<p>Am I right in thinking they are roughly similar so shouldn’t the log softmax and nll loss perform better better on the test set given it has a lower loss?</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects the raw logits as the model outputs, so you would have to remove the <code>softmax</code> for the first case.<br>
You should get identical values afterwards, since <code>nn.CrossEntropyLoss</code> uses <code>F.log_softmax</code> and <code>nn.NLLLoss</code> internally.</p>
Smaller torch saved model

nicolalandro: <p>I have a trained NTS-NET that uses 108 MB on file storage. In my server I do not have enough space but is only some MB. So I compress “state_dict” using “tar.gz” and I arrive to 100 MB. It i just enought. So to load the model I use the funcion</p>
<pre><code class="lang-auto">import pickle
import tarfile

from torch.serialization import _load, _open_zipfile_reader


def torch_load_targz(filep_ath):
    tar = tarfile.open(filep_ath, "r:gz")
    member = tar.getmembers()[0]
    with tar.extractfile(member) as untar:
        with _open_zipfile_reader(untar) as zipfile:
            torch_loaded = _load(zipfile, None, pickle)
    return torch_loaded


if __name__ == '__main__':
    torch_load_targz("../models/nts_net_state.tar.gz")
    # equivalet for torch.load("../models/nts_net_state.pt")  for .tar.gz 
</code></pre>
<p>So at the end I read the torch model from tar.gz directly. But in this way the prediction are too slow.<br>
Exist some better solution at this problem?</p>
<p>(I’m using torch-1.4.0, and python 3.6)</p>

ptrblck: <p>Is the loading of the <code>state_dict</code> slow or the model predictions?<br>
The latter shouldn’t be influenced by how the <code>state_dict</code> is loaded or are you reloading it in every iteration?</p>

nicolalandro: <p>The slow part is the extract tar.gz and it save only 8MB into the state_dict stored file.<br>
I’m speaking about only predictions (of small data) so I do not have iterations inside a single call.</p>

ptrblck: <p>It is expected that unzipping a file will take longer than e.g. reading a binary file.<br>
What kind of system are you using that you are running out of memory for 108MB?</p>

nicolalandro: <p>No. I fill my server memory with:</p>
<ul>
<li>with torch and torchvision and other libraries</li>
<li>and 108MB of trained model</li>
</ul>
<p>For example I see that transform a tensorflow model using tensorflow-lite the size in MB of the model can be reduced a lot. I was wondering for something like that using pytorch. If exist some other way may be it is less slow.</p>

ptrblck: <p>You could try to <a href="https://pytorch.org/docs/stable/quantization.html">quantize</a> your model to reduce the size.<br>
However, I’m not sure, how experimental this feature is at the moment,</p>

nicolalandro: <p>Thank you! I try this way! And I wondering also for other.</p>
How to add tensors of variable length to feed to Linear layer?

neerajshukla1911: <p>I trying to add tensors of different length and feed to Linear Layer</p>
<pre><code class="lang-auto">input1 = torch.randn(4, 40, 56, 56).view(-1, 4)

input2 = torch.randn(4, 40, 55, 55).view(-1, 4)

input3 = torch.randn(4, 40, 54, 54).view(-1, 4)

output = input1 + input2 + input3

fc = torch.nn.Linear(62720, 3)

output = fc(output1)
</code></pre>
<p>But getting the following error</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (125440) must match the size of tensor b (121000) at non-singleton dimension 0
</code></pre>

googlebot: <p>simple way is fc(torch.cat((input1,input2,input3),-1)), memory efficient way is emulating nn.Linear layer: input1.mm(weight1.t()) + input2.mm(weight2.t()) + … + bias, where weights may be slices of nn.Parameter(output_size, total_input_size) in second dimension</p>

neerajshukla1911: <p>For fc(torch.cat((input1,input2,input3),-1)) still getting error</p>
<pre><code class="lang-auto">RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 125440 and 121000 in dimension 0 at /pytorch/aten/src/TH/generic/THTensor.cpp:612
</code></pre>

googlebot: <p>Oh, linear layer works on last dimension, thus you can concatenate on last dimension and proceed. But in your code, tensor lengths vary before last dimension, thus you can’t do this. You can use .view(4,-1) or .view(4*40,-1) on inputs, but varying first dimension (treated as batch(=independent) dimension) makes no sense. Maybe you need permute(). Also, I’m not sure where 62720 comes from, replace it with concatenated length.</p>
Initializing a tensor from a list of tensors?

dcoukos: <p>I am trying to make a tensor from a list of tensors, which is throwing a <strong>ValueError: only one element tensors can be converted to Python scalars</strong>.</p>
<p>my setup is like so:</p>
<pre><code class="lang-auto">I have calculated three tensors n0, n1, n2
n0 = tensor([-0.4360, -1.1935, -0.8568])
n1 = tensor([ 0.8926, -0.0698,  0.5011])
n2 = tensor([-1.3286, -1.1237, -1.3579])

b = torch.tensor([[n2-n1],
                            [n0-n2],
                            [n1-n0]])
</code></pre>
<p>produces the error<br>
I am having trouble understanding:</p>
<ul>
<li>Why this way of making a tensor would be undesirable</li>
<li>How to do this in a more pytorch friendly way</li>
</ul>
<p>Thanks for any help!</p>

raj-rishav: <p>I ain’t sure about the 1st question. But I would use below code for the variable <code>b</code>.</p>
<pre><code class="lang-auto">b = torch.cat(((n2 - n1).unsqueeze(0), (n0 - n2).unsqueeze(0), (n1 - n0).unsqueeze(0)), 0)
</code></pre>

G.M: <p>For your first question, I think it’s because it is hard to decide whether you want to leave the input tensors as a dimension or you only want the values in it. It’ll be quite confusing.</p>

saty_101: <p>Have you found the answer to your question? I can’t wrap my head around to get the desired result and want to know why error is coming up.</p>
How to make debug layers to be ignored during pre-trained weight assignment?

csailnadi: <p>I modified VGG network by injecting debug-purpose layers that simply save the output of a layer. Clearly such layer does not have <code>.weight</code>. Now I am trying to load state weights to restart the model. Hover I get errors:<br>
<code>Missing key(s) in state_dict: "features.save_layer1_1.weight", "features.save_layer1_1.bias"...</code></p>
<p>How do I force pytorch to ignore such debug layers and load weights only to layers with weights?</p>
<p>Here is new VGG model</p>
<p>The way model is loaded now:</p>
<pre><code class="lang-auto">def load_vgg16():
    model = VGG16(num_classes=365)

    model.features = torch.nn.Sequential(collections.OrderedDict(zip([
        'conv1_1', 'relu1_1', 'save_layer1_1',
        'conv1_2', 'relu1_2', 'save_layer1_2',
        'pool1',
        'conv2_1', 'relu2_1', 'save_layer2_1',
        'conv2_2', 'relu2_2', 'save_layer2_2',
        'pool2',
        'conv3_1', 'relu3_1', 'save_layer3_1',
        'conv3_2', 'relu3_2', 'save_layer3_2',
        'conv3_3', 'relu3_3', 'save_layer3_3',
        'pool3',
        'conv4_1', 'relu4_1', 'save_layer4_1',
        'conv4_2', 'relu4_2', 'save_layer4_2',
        'conv4_3', 'relu4_3', 'save_layer4_3',
        'pool4',
        'conv5_1', 'relu5_1', 'save_layer5_1',
        'conv5_2', 'relu5_2', 'save_layer5_2',
        'conv5_3', 'relu5_3', 'save_layer5_3',
        'pool5'],
        model.features)))
    model.classifier = torch.nn.Sequential(collections.OrderedDict(zip([
        'fc6', 'relu6', 'save_layer6'
        'drop6',
        'fc7', 'relu7', 'save_layer7'
        'drop7',
        'fc8a'],
        model.classifier)))

    sd = torch.hub.load_state_dict_from_url(url) 
    model.load_state_dict(sd)
    model.eval()
    return model
</code></pre>
<p>‘save_layer*’ layers should be ignored during loading.</p>

ptrblck: <p>You could try to pass <code>strict=False</code> while loading the <code>state_dict</code>.</p>

csailnadi: <p>this problem seems to be solved for me by modifying the state dictionary’s <code>_metadata</code></p>
Is it possible to vectorize MSE for multiple outputs

maxmatical: <p>I have a regression model with multiple outputs, so I have a custom mse function</p>
<pre><code class="lang-auto">def custom_mse(predicted, target):
    total_mse = 0
    for i in range(target.shape[1]):
        total_mse+=nn.MSELoss()(predicted[i], target[i])
    return total_mse
</code></pre>
<p>the purpose is to add up all the mse for each individual output, and return the total as the loss. This works well enough, but is there a way to get rid of the for loop and vectorize the <code>total_mse</code> calculation?</p>

ptrblck: <p><code>nn.MSELoss</code> accepts batches of your model output and targets.</p>
<p>I’m a bit confused about the indexing. You are iterating using <code>target.shape[1]</code>, but are indexing both tensors in dim0. Is this a typo?</p>

maxmatical: <p>The data has <code>target.shape</code> = <code>(n_samples, k)</code> where k &gt; 1. I’m trying to have a model predict the soft pseudolabels generated by a teacher model, so the targets are the predicted logits for each class, and I’m using the sum of MSE for each class as the loss function</p>
<blockquote>
<p>I’m a bit confused about the indexing. You are iterating using <code>target.shape[1]</code> , but are indexing both tensors in dim0. Is this a typo?</p>
</blockquote>
<p>Yes, that appears to be an error in the code, it should be indexing <code>[:,i]</code></p>
<p>I did some further digging, it looks like I had an error in my original implementation of <code>nn.MSELoss</code>. Fixing that seemed to give me the results I’m looking for.</p>
PyTorch: TypeError: only integer tensors of a single element can be converted to an index

shamoons: <p>I have:</p>
<pre><code class="lang-auto">print(f"inp: {inp.size()}\tleft_end_indices: {left_end_indices.size()}\tself.side_length: {self.side_length}")
left_remain_input = inp[:,left_end_indices + 1:left_end_indices + 1 + self.side_length]
</code></pre>
<p>which has the sizes:</p>
<pre><code class="lang-auto">inp: torch.Size([16, 1544, 161])        left_end_indices: torch.Size([16])      self.side_length: 48
</code></pre>
<p>But the error I get:</p>
<pre><code class="lang-auto">TypeError: only integer tensors of a single element can be converted to an index
</code></pre>
<p>Basically, I want to set <code>left_remain_input</code> to be portions of <code>inp</code> based on where the <code>left_end_indices</code> are in the second dimension. How can I accomplish this with PyTorch?</p>
Load model incompatible keys

Alex5: <p>Hi,<br>
I’m trying to load a model that I saved with fastai (using resnet34 base architecture and save function).</p>
<pre><code class="lang-auto">model_path = '../input/resnet34-ship-clf/Resnet34_clf_256x256_1ep.pth'
model = nn.Sequential(models.resnet34())
pretrained_weights = torch.load(model_path)
model.load_state_dict(pretrained_weights, strict=False)
</code></pre>
<p>I’ve got the error message : <code>_IncompatibleKeys(missing_keys=['0.conv1.weight' ...</code></p>
<p>Pre-trained weight names are in the form : <code>'x.x.x.conv1.weight'</code><br>
What should I do during loading to get same names ?<br>
Thanks</p>

ptrblck: <p>Could you try to load the <code>state_dict</code> into the <code>resnet34</code> directly instead of wrapping it into an <code>nn.Sequential</code> container?</p>

Alex5: <p>It returns <code>_IncompatibleKeys(missing_keys=['conv1.weight'...</code><br>
and now pretrained weights are in the form <code>'x.x.conv1.weight'</code></p>

ptrblck: <p>I’m not sure how the <code>state_dict</code> was saved, but you could iterate the <code>state_dict</code>, remove the unnecessary two numbers at the beginning, and create a new <code>OrderedDict</code> to load in your model.</p>
Pass convLayer output 4D tensor to ConvLstm Layer input (5D tensor)

Peter_Bardawil: <p>Hello everyone,</p>
<p>I am adding an ConvLstm layer after conv13 layer of mobilenet which outputs a 4d tensor(batch_size,nb_of_features,width,height) and I want to pass this tensor to a ConvLstm layer that takes tensor of  dimensions(batch_size,seq_length,nb_of_features,width,height) but I dont get how I should do that, how to convert this 4d tensor to 5d tensor and how to fill it correctly, there is something that isnt really clear for me.<br>
Thanks for your help !</p>