akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to do backward for the distributed joint training?

Zhuolin_Yang: <p>Hi all, I was trying to distribute several models (say 8) into 8 gpu devices. For each model models[i], I calculated the sum of output as follows:</p>
<pre><code class="lang-auto">for i in range(args.num_models):
    input_var = input_var.to("cuda:%d" % (i))
    target_var = target_var.to("cuda:%d" % (i))
    criterion = nn.CrossEntropyLoss().to("cuda:%d" % (i))
    models[i] = models[i].to("cuda:%d" % (i))
    loss += criterion(models[i](input_var), target_var)

loss.backward()
</code></pre>
<p>But some error occurred:</p>
<pre><code class="lang-auto">RuntimeError: Function AddBackward0 returned an invalid gradient at index 1 - expected device cuda:1 but got cuda:0
</code></pre>
<p>Any suggestions? Thanks!</p>

ptrblck: <p>Try to push the losses to the same device before accumulating them.<br>
Also, which PyTorch version are you using, as I cannot reproduce this issue locally (and also thought it would be fixed by now).</p>

Zhuolin_Yang: <p>Thank you so much! Problem solved <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Slow quantized inference on Cortex-A72

MaxS1996: <p>Hi,<br>
I compiled a TFLite Flatbuffer file into a bundle executable and got a major increase in inference time.<br>
What could be the reason for such behavior?</p>
<ul>
<li>Device: Raspberry Pi 4 (Cortex-A72)</li>
<li>Inference Time: (min, mean, max)<br>
**  Float: 1.41ms; 1.80ms; 12.05ms;<br>
**  Int8  : 3.14ms; 3.84ms; 18.66ms;</li>
</ul>
<p>How can I fix that?</p>

jfix: <p>Hi <a class="mention" href="/u/maxs1996">@MaxS1996</a>, I’d suggest reading this issue – it will give you a better understanding of why using quantization doesn’t necessarily mean that performance gets better: <a href="https://github.com/pytorch/glow/issues/4505" rel="nofollow noopener">https://github.com/pytorch/glow/issues/4505</a></p>
RTX 2080Ti and RTX 2060 Data Parallel

a.arbabian: <p>Hello,</p>
<p>I am upgrading my machine from an RTX 2060 to RTX 2080Ti. I was wondering if I can leave the 2060 in there and use both GPUs with DataParallel(). I’ve used DataParallel() before but both the cards were 1080Ti’s. So my questions are:</p>
<ol>
<li>Can DataParallel() utilize the VRAM from both these cards despite them having very different specs?</li>
<li>Would there be any complications with AutoMixedPrecision from APEX? Both the cards are RTX so they should (in theory) benefit from amp.</li>
</ol>
<p>Thanks!</p>

JuanFMontesinos: <p>You can but it would be a mess.<br>
They have way different amount of memory. The effective memory per gpu would be min(2060,2080). Therefore you would be wasting most of the 2080’s memory.</p>
<p>Same with computational power. 2060s will bottleneck the other one.</p>
How can I reduce size of my input

harshini_sewani: <p>I have an FMRI dataset containing a matrix of (392)<em>(392) (after applying correlation on the 1D file of FMRI). I have to apply Autoencoder on it. How can I reduce the size of my input? because my model will contain first line like this: self.fc1=nn.Linear(392</em>392,__) this is huge to send it to Autoencoder.</p>
How to I load part of a model weights?

soulslicer: <p>I have modified a model by removing some stuff at the end of it. But of course when I try to load the weights, it complains that certain keys dont exist in the model.</p>
<p>I tried to remove the keys as follows:</p>
<pre><code class="lang-auto">            # Remove keys contaning second stage
            keys_to_remove = []
            for key in ckpt_state.keys():
                if "rroi" in key:
                    keys_to_remove.append(key)
            for key in keys_to_remove:
                del ckpt_state[key]

</code></pre>
<p>And the models load state dict worked, but the optimizer load_state_dict crashes with</p>
<pre><code class="lang-auto">ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group
</code></pre>
<p>How to I remove the right stuff from the optimizer weights?</p>
Gaussian Tensor Generation

MUnique: <p>Hi,<br>
How can I have an (n,1) gaussian tensor in PyTorch with the largest value in the middle (mean of the gaussian) and all entries normalized to be between (0,1)?</p>

ptrblck: <p>If you would like to create a Gaussian window (not sample from the Gaussian distribution), you could use e.g. <a href="https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.signal.gaussian.html"><code>scipy.signal.gaussian</code></a> to create the values and transform it to a tensor via <code>torch.from_numpy(signal)</code>.</p>

MUnique: <p>Thanks for the reply. Actually, I have a tensor like (B,C,H,W) and I want to multiply the last two dimensions with a gaussian (H and W are the spatial dimensions. So it’s like an attention on the middle pixels). I want it to be fully implemented in Pytorch to be able to do backpropagation.</p>

KFrank: <p>Hi M!</p>
<aside class="quote no-group" data-username="MUnique" data-post="3" data-topic="89397" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/d78d45/48.png" class="avatar"> MUnique:</div>
<blockquote>
<p>I want it to be fully implemented in Pytorch to be able to do backpropagation.</p>
</blockquote>
</aside>
<p>If your two-dimensional (HxW) Gaussian window is going to be fixed,<br>
i.e., something like <code>[1.0, 2.0, 1.0]</code> (in 1-d) that doesn’t depend on the<br>
values in the tensor you want to multiply it with, then where those fixed<br>
values came from won’t affect back-propagation.  So <code>from_numpy (...)</code><br>
will be fine.</p>
<p>The point is that in back-propagation you won’t be differentiating<br>
with respect to the constants in your Gaussian window; you’ll be<br>
differentiating with respect to variables (elements of tensors that<br>
have <code>requires_grad = True</code>) that those constants multiply.</p>
<p>Best.</p>
<p>K. Frank</p>
Should I define my custom loss function as a class?

Amr_Mustafa: <p>I need to implement a custom loss function and looking for best practices on doing so. In the PyTorch documentation, it is clear that the predefined loss functions are implemented as classes. My question is, should I also define my custom loss function as a class or a normal Python function would do? I also would like to know why PyTorch implements losses as classes. Thanks in advance!</p>

JuanFMontesinos: <p>It’s not really necessary.<br>
It’s all about how pytorch is structured. Classes calls in the enf functionals which are functions with no kwargs but only args whose signature is c++.</p>
<p>So they just follow that convention.</p>
<p>Another reason is that some class may have weighting schemes and if they are defined as classes (nn.Modules) they can be allocated on cuda easily and recursively.</p>
Torch.topk with k=0 leads to errors when used for indexing another tensor

Aayush-Ankit: <p>The indices (empty tensor) returned by topk for k=0, leads to <code>RuntimeError: CUDA error: device-side assert triggered</code> when being used for selecting indexes within another tensor. However, when using an empty tensor (not from <code>topk.indices</code>) for selecting indexes to the another tensor works fine. Is this an expected behavior ? Note that the behavior with <code>device="cpu"</code> is correct.</p>
<p><strong>Pytorch version 1.5.1</strong>. The code runs fine on older version (1.2.0)</p>
<p>Code to reproduce the above:</p>
<pre><code class="lang-auto">import torch
#device = "cpu"
device = "cuda:0"
x = torch.arange(1,10).float().to(device)

# Case - 1: use an empty cuda tensor for index
y = []
y_t = torch.LongTensor(y).to(device)

# Case - 2: use an empty cuda tensor from topk.indices for index
topk = torch.topk (x, k=0, largest=False)

print(y_t, topk.indices)
x[y_t] = 0.0
print(x)
x[topk.indices] = 0.0
print(x) 
</code></pre>

ptrblck: <p>Thanks for reporting this issue! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Would you mind creating an <a href="https://github.com/pytorch/pytorch/issues">issue</a> so that we could track and fix it, please?</p>

Aayush-Ankit: <p>Sure, I’ll create an issue with the details I have obtained. Thanks.</p>
Symeig gives repeated eigen values for a covariance matrix

Vandana_Rajan: <p>Hi,</p>
<p>I have a covariance matrix (symmetric positive definite) and hence I expected that symeig would give me unique eigen values. But it is giving repeated eigen values and this creates backward pass error in my custom loss function. I tried increasing the precision by changing from float32 to float64. However, the problem still persists. Can you please let me know how to solve the issue?</p>

albanD: <p>Hi,</p>
<p>Out of curiosity, how do you ensure that the eigenvalues are unique? The positive definitiveness only ensure that they’re non-zero right?</p>

Vandana_Rajan: <p>Well. I am not an expert in linear algebra but I read that covariance matrices are diagonalisable and hence have unique eigen values. I might be wrong. But then again, even if the eigen values are not unique, why symeig should give such an error? Also, eig does not have a backward function.</p>

albanD: <p>What is the error you get from symeig? It should work fine for repeated eigenvalues.</p>

Vandana_Rajan: <p>I am using the autograd anomaly detector and it gave the following error “Function ‘SymeigBackward’ returned nan values in its 0th output.” Looking at the symeig function description, I saw that such nan values in the backward pass can happen when there are non-unique eigen values. " <strong>Extra care needs to be taken when backward through outputs. Such operation is really only stable when all eigenvalues are distinct. Otherwise, <code>NaN</code> can appear as the gradients are not properly defined</strong>."</p>

albanD: <p>Ho thanks for the reference, I didn’t knew that.<br>
I guess you’ll need to do some extra processing of your input matrix to ensure all the eigenvalues are distinct. But I’m not sure how to do that <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Save model as png

Nilakshan_Kunanantha: <p>Can we store PyTorch deep learning model as a png image(Like Keras does)?</p>

ebarsoum: <p>If you want to visualize the model, you can use netron to load PyTorch model: <a href="https://github.com/lutzroeder/netron" rel="nofollow noopener">https://github.com/lutzroeder/netron</a></p>

Nilakshan_Kunanantha: <p>Really helped me a lot<br>
Thanks !!!</p>
Cam grad CNN+LSTM

Zrufy: <p>I’m tryng to create a cam grand from my model CNN+LSTM.<br>
I take my first part of the  model and the second to pass in this function</p>
<pre><code class="lang-auto">def GradCAM(img, c, features_fn, classifier_fn):

    feats = features_fn(img.cuda()) # [1, 2048, 7, 7]

    _, N, H, W = feats.size()

    out = features_fn(feats)

    c_score = out[0, c] # output value of class c

    grads = torch.autograd.grad(c_score, feats) # get gradient map (grads[0][0])

                                                # [2048, 7, 7]

    w = grads[0][0].mean(-1).mean(-1) # GAP of grads

                                      # [2048]

    sal = torch.matmul(w, feats.view(N, H*W)) # feats.view(N, H*W) -&gt; [2048, 49]

    sal = sal.view(H, W).cpu().detach().numpy()

    sal = np.maximum(sal, 0)

    return sal
</code></pre>
<p>But when i go to run the model i have this error</p>
<blockquote>
<p>RuntimeError: Given groups=1, weight of size 24 1 5 5, expected input[1, 512, 1, 26] to have 1 channels, but got 512 channels instead</p>
</blockquote>

ptrblck: <p>Is your model working generally without splitting it?<br>
The error points towards a shape mismatch in a conv layer. If the model is working in the original form, make sure to apply all functional API calls, such as flattening the activation tensors etc.</p>

Zrufy: <p>for an inference I don’t need to split the model. I wanted to see where the convolutive part was concentrated with a cam-grad. So I wanted to take the first part of the model and see the results.</p>
What's the best way to export gradient which is still part of the computational graph?

Mactarvish: <p>Suppose I have a model F(x)=y, which means x is the input and y is the output, what’s the best way to get the gradient ∂y / ∂x ? Note that the gradient still needs to be part of its computational graph (so that any operations on the gradient can affect other variables in the computational graph, which is exactly what the “gradient penalty” (GP) shown in paper “Improved Training of Wasserstein GANs” does).</p>
<p>I’ve found there is a way to do that:</p>
<pre><code class="lang-auto">        prob =self.D(input_image)
        # calculate ∂D(input_image) / ∂input_image
        grad = torch.autograd.grad(outputs=prob , inputs=input_image,
                                               grad_outputs=torch.ones(prob .size()).cuda(),
                                               create_graph=True, retain_graph=True)[0]

</code></pre>
<p>But this code is too old and not elegant enough, and I doubt its correctness in current PyTorch. Is there a better way to do this? Thanks in advance.</p>
Loading dataset

lil: <p>Hi, I have custom dataset that loads data from a subfolder root_x that contains 625 training images and a subfolder root_y that contains 625 matrices. My issue is that when I call the function len() after the loading, the returned value is 67 instead of 625.  Can someone explains me why and what I should change to get the correct size when I call len().</p>
<p><span class="hashtag">#custom</span> class<br>
class MappingDataset(Dataset):</p>
<pre><code>def __init__(self, root_x, root_y, transform=None):
    self.root_x = root_x
    self.root_y = root_y
   
def __len__(self):
    return len(self.root_x)

def __getitem__(self, idx):
    img_name = self.root_x + str(idx) + '.png'
    image = io.imread(img_name)
    wpmap_name = self.root_y + str(idx) + '.txt'
    wpmap = np.loadtxt(wpmap_name, np.int)
    sample = {'image': image, 'wpmap': wpmap} 
    return sample
</code></pre>
<p><span class="hashtag">#main</span><br>
training_set = MappingDataset(root_x, root_y)<br>
print(len(training_set) = 67 <span class="hashtag">#instead</span> of 625</p>
<p>I note that I can iterate over the training_set when I explicitly give the range of the for loop.<br>
for i in range(1, 626):<br>
sample = training_set[i]<br>
print(i, sample[‘image’].shape, sample[‘wpmap’].shape)</p>

lil: <p>I corrected myself. That was the issue.<br>
def <strong>len</strong>(self):<br>
return len(os.listdir(self.root_x))</p>
How to tune BERT for words if they could belong to many (or even all) our our classes?

Mona_Jalal: <p>Essentially, we can tune BERT for sentences for our classes. But how can we tune BERT for words specially if they could even belong to many or all of our classes (e.g. a set of 10 classes)?</p>
<p>Not tuning the BERT seems to yield unfavorable results.</p>
<p>A question by me and <a class="mention" href="/u/lais823">@lais823</a></p>

macwiatrak: <p>You can use <a href="https://huggingface.co/transformers/model_doc/bert.html#bertfortokenclassification" rel="nofollow noopener">this</a> for finetuning BERT for words (specifically, tokens). I think the default there is multi-output classification, not multi-label, but you can change it to multi-label relatively easy.</p>
Imbalanced Classification

Angelina_Robert: <p>Hi, just wondering if anyone could help to generate imbalanced CIFAR10 and CIFAR100 datasets in python.</p>

ptrblck: <p>For the sake of completeness:<br>
<a href="https://github.com/pytorch/tutorials/pull/236">here</a> is an old tutorial, which creates an artificially imbalanced CIFAR10 dataset. Could you reuse it or is something missing for your use case?</p>

Angelina_Robert: <p>Yeh, that tutorial is a very gud guide, however, using its snippet for imbalanced dataset  creation in my code causes the following error:<br>
for count, prop in zip(train_class_counts, imbal_class_prop)<br>
TypeError: only size-1 arrays can be converted to Python scalars</p>
<p>In original tutorial it is line number 272.</p>

ptrblck: <p>Thanks for reporting this issue.<br>
After fixing some legacy code, such as:</p>
<pre><code class="lang-python">targets, class_counts = get_labels_and_class_counts(
                self.dataset.targets)
</code></pre>
<p>I can run the code without the issue in line272.<br>
Are you seeing any other issues before this failure?</p>

Angelina_Robert: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="89399">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">targets, class_counts = get_labels_and_class_counts(
                self.dataset.targets)
</code></pre>
</blockquote>
</aside>
<p>I have also fixed that issue, however problem is occuring when I inject that piece of code in my code. Moreover, could you please guide where (line number) have you made the above change.</p>
Inefficiency in metric calculation

pytorchnewbie: <p>Suppose I have two metrics, mse and correlation. Every 2 epochs of the training, I want to run the evaluator and collect the MSE metric. Also, at the end of training (800 epochs later), I want to run the evaluator and collect and the correlation metric. However, the metrics are attached to the evaluator engine. Therefore, every time evaluator is run, both metrics are calculated. This seems inefficient, since I only need to calculate correlation on the evaluator run at the end of the 800 training epochs, not 400 times (800/2) during training. Thoughts?</p>

vfdev-5: <p>You can attach/detach metrics, so I think you can just attach it at the end of the training:</p>
<pre><code class="lang-python">@trainer.on(Events.EPOCH_COMPLETED(every=2))
def compute_mse():
    evaluator.run(validation)

@trainer.on(Events.COMPLETED)
def compute_correlation():
    correlation_metric.attach(evaluator, "corr")
    evaluator.run(validation)
    # optionally
    correlation_metric.detach(evaluator)
</code></pre>

pytorchnewbie: <p>Beautiful! This is exactly what I wanted to know. Thanks!</p>
TypeError: pic should be PIL Image or ndarray. Got <class 'bool'>

alameer: <p>I would like to split images using Open CV then feed to the by PyTorch model to count object in each image. I am getting the following error message when running the code: TypeError: pic should be PIL Image or ndarray. Got &lt;class ‘bool’&gt;</p>
<pre><code class="lang-auto">import os

import numpy as np
import torch
from PIL import Image
from torch.utils.data import Dataset
from torchvision import transforms, utils
#from torchvision.transforms import Grayscalei
import pandas as pd
import pdb
import cv2

class CellsDataset(Dataset):
    # a very simple dataset

    def __init__(self, root_dir, transform=None, return_filenames=False):
        self.root = root_dir
        self.transform = transform
        self.return_filenames = return_filenames
        self.files = [os.path.join(self.root,filename) for filename in os.listdir(self.root)]
        self.files = [path for path in self.files
                      if os.path.isfile(path) and os.path.splitext(path)[1]=='.png']

    def __len__(self):
        return len(self.files)

    def __getitem__(self, idx):
        path = self.files[idx]
        sample = Image.open(path)        

        sample = cv2.imread(path)
        b,g,r=cv2.split(sample)
        sample=cv2.imwrite('sample.png', g)


        #transform3 = Grayscale(num_output_channels=3)
        #sample = transform3(sample) # convert to a 3 channel grayscale, as it needs to be 3 channel.
        if self.transform:
            sample = self.transform(sample)

        if self.return_filenames:
            return sample, path
        else:
            return sample</code></pre>

ptrblck: <p><code>cv2.imwrite</code> will return a <code>bool</code> value to indicate, if the write process was successful or not, which you are reassigning to <code>sample</code> before passing to <code>self.transform</code>.<br>
Remove the assignment and rerun the code.</p>

alameer: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Many Thanks for your help. That worked okay now.</p>
RuntimeError: Expected isFloatingType(grads[i].scalar_type()) to be true, but got false

Surya_Lohia: <blockquote>
<p>inputs = Variable(inputs,requires_grad=False)<br>
labels = Variable(labels,requires_grad=False)</p>
<p>logits = model.forward(inputs)<br>
outputs= loss.forward(softmax(logits).argmax(axis=1).float(),labels.float())<br>
optimizer.zero_grad()<br>
outputs.backward()<br>
optimizer.step()</p>
</blockquote>
<p>I have <strong>images</strong> as an input and corresponding binary labels to the CNN but I am facing the error as</p>
<p><strong>RuntimeError: Expected isFloatingType(grads[i].scalar_type()) to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.) (validate_outputs at …\torch\csrc\autograd\engine.cpp:476)</strong><br>
<strong>(no backtrace available)</strong></p>
<p>I have used <strong>BCEloss</strong> function and <strong>Adam</strong> as optimizer. Used <strong>Softmax</strong> for the output of BCEloss<br>
Any idea to fix it?</p>

ptrblck: <p>There are some issues in the code:</p>
<ul>
<li>
<code>Variables</code> are deprecated since PyTorch <code>0.4</code>, so you can use tensors now.</li>
<li>Don’t call <code>.forward</code>, but the module directly via: <code>model(inputs)</code>.</li>
<li>
<code>nn.BCELoss</code> is used for a binary classification (or multi-label classification) and expects a <code>sigmoid</code> as the last non-linearity. We generally recommend to use raw logits + <code>nn.BCEWithLogitsLoss</code> for better numerical stability.</li>
<li>
<code>argmax</code> will create an output of <code>LongTensors</code>, which will break the computation graph, since <code>LongTensors</code> cannot have gradients. It’s also not needed to call <code>argmax</code> on the output before passing it to the criterion.</li>
</ul>
Simple Web App for NLP DL Model

Raphy-N: <p>I’ve almost completed my DL models for an NLP project. Now I want to make a <strong>web app</strong>.</p>
<p>I created models in <strong>PyTorch</strong> to detect propaganda text fragments in news articles. The models have poor performance. But given a news article, they can return offsets &amp; labels for each offset (in a text file), identifying the type of propaganda in each span of text.</p>
<p>Now I want to build a prototype Web app to demo this project with following minimal requirements:</p>
<ol>
<li>User provides articles (upload text file or provide link for online blog/article)</li>
<li>System highlights all detected fragments. On hovering over each, a tooltip displays the label (propaganda technique name classifier gives a label to each span).</li>
<li>Display bar or pie charts to show frequency of each propaganda type in the provided articles. And the propaganda percentage found in each article.<br>
Can you please suggest a python framework with which I can build a minimal prototype in the shortest possible time? Something that I can learn in a few days. I’m thinking of trying it out on <a href="https://www.streamlit.io/" rel="nofollow noopener">StreamLit</a> Can anybody guide me whether Flask, StreamLit or Django will be better, coz I have a deadline &amp; need to build a simple web app by learning any new framework.</li>
</ol>

vdw: <p>That’s not a PyTorch question, so you might won’t get the right answers here.</p>
<p>Having said that, you can expose the functionality of your models via an API. For example, I use <a href="https://falconframework.org" rel="nofollow noopener">Falcon</a> for that. Your API has to offer at least one endpoint to upload the content (e.g., <code>/upload</code>) that</p>
<ul>
<li>listens to uploaded text files or links</li>
<li>processes the content (e.g., making a prediction using your trained models)</li>
<li>returns all the data needed to visualize the results</li>
</ul>
<p>The you can build any kinde of Web frontend that uses the API. But again, I’m not a Web developer, and this is the wrong forum for this type of question :).</p>
How to calculates differences between adjacent elements

seyeeet: <p>I want to calculate the differences between adjacent elements of a tensor <code>X</code> in 1 D.<br>
e.g if my tensor has a dimension of <code>BxC</code> where <code>B</code> is batch size and <code>C</code> is a channel.<br>
function <code>diff(X)</code> should outputt <code>[X(2)-X(1) X(3)-X(2) ... X(c)-X(c-1)]</code>.<br>
How can I do it in pytorch?</p>

ptrblck: <p>This code should work to compute the difference in the <code>C</code> dimension:</p>
<pre><code class="lang-python">B, C = 10, 5
x = torch.arange(B * C).view(B, C)
result = x[:, 1:]  - x[:, :-1]
</code></pre>
From where does the backward() method come in custom loss functions

nowyouseeme: <p>Continuing the discussion from <a href="https://discuss.pytorch.org/t/custom-loss-functions/29387/2">Custom loss functions</a>: In this solution, <a class="mention" href="/u/ptrblck">@ptrblck</a> demonstated custom loss function. However, the function <code>my_loss</code> returns a torch tensor. Then how come running <code>.backward()</code> on this tensor works in pytorch?</p>
<p>In my case, my loss function looks like this :</p>
<pre><code class="lang-auto">def my_loss(out, tar):
    loss = torch.sum(out * tar)
    return loss
</code></pre>
<p>Now this returned loss does not have any parameter or a method named <code>backward()</code>, then how can I run <code>.backward()</code> on the torch tensor returned by <code>loss = my_loss(out, tar)</code> ?</p>

ptrblck: <p><code>backward</code> is not only a method of e.g. <code>autograd.Function</code>, but also a tensor method defined as <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.backward"><code>torch.Tensor.backward</code></a>, which is why you can call <code>.backward()</code> directly on tensors.</p>
Assertion error; Pysyft + Pytorch

SandPhoenix: <p>Hello everyone,<br>
I am stuck here in a strange problem.<br>
So, I trained my model normally but when I try to test it out using using pysyft, I am facing a problem that does not let the testing even start.<br>
When I tried debugging, I got that this is the main error causing this<br>
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 64 3 3 3, but got 1-dimensional input of size [0] instead<br>
I need some help ASAP and thank you all</p>

ptrblck: <p>Based on the error message it seems you are trying to pass a scalar input to the model, while a 4-dimensional input is needed.<br>
Without more information I would recommend to check the input shape and try to debug, why it’s a scalar value during the test.</p>

SandPhoenix: <p>Thx, actually the problem was with the shares generation thing <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
thanx for the reply</p>
Shouldn't ReduceLROnPlateau() super optimizer in its init?

bfeeny: <p>Regarding ReduceLROnPlateau():</p>
<pre><code class="lang-auto">optim = torch.optim.AdamW(model.parameters(), lr=param['lr'], amsgrad=True) 
scheduler = ReduceLROnPlateau(optimizer=optim, mode='max', patience=1, verbose=True, factor=0.5)
val_roc = torch.tensor(15.0, device=device)
scheduler.step(val_roc)
scheduler.get_last_lr()
</code></pre>
<pre><code class="lang-auto">---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
&lt;ipython-input-39-f8112ad93830&gt; in &lt;module&gt;
----&gt; 1 scheduler.get_last_lr()

AttributeError: 'ReduceLROnPlateau' object has no attribute 'get_last_lr'
</code></pre>
<p>Inside def step() we do:<br>
<code>        self._last_lr = [group['lr'] for group in self.optimizer.param_groups]</code></p>
<p>Yet there is no way to pull that out.</p>
<p>Class <code>class _LRScheduler(object):</code> has</p>
<pre><code class="lang-auto">    def get_last_lr(self):
        """ Return last computed learning rate by current scheduler.
        """
        return self._last_lr
</code></pre>
<p>But inside of ReduceLROnPlateau()'s init we do not have:</p>
<p><code>        super(ReduceLROnPlateau, self).__init__(optimizer, last_epoch)</code></p>
<p>Im just trying to track the LR as my model builds, and encountered this.  I think someone forgot to super<br>
_LRScheduler?</p>

ptrblck: <p>You could create a feature request for <code>get_last_lr</code> for this scheduler and meanwhile use the <code>'lr'</code> attribute of the parameter group of the optimizer directly:</p>
<pre><code class="lang-python">print(optimizer.param_groups[0]['lr'])
</code></pre>
Get top k indices/values of all rows

rachel: <p>Hi, I’m looking to get the topk gradients of all rows, not topk of each row. For example, if I have a conv layer of shape [64, 64, 3, 3] and k=2, I only want 2 top values and their corresponding indices returned. Ultimately, I want a new tensor with a shape matching the dimensions of the original weight, with all elements zeroed out except the top k gradients.</p>
<p>Here is what I have now, which is quite with the mapping from flattened indices to expanded:</p>
<pre><code class="lang-auto">def get_topk_idx(grad): 
  grad_list =  grad.detach().cpu().tolist()
  stack = [enumerate(grad_list)]
  path = [None]
  while stack:
      for path[-1], x in stack[-1]:
          if isinstance(x, list):
              stack.append(enumerate(x))
              path.append(None)
          else:
              yield x, tuple(path)
          break
      else:
          stack.pop()
          path.pop()

def get_topk_grads(grad, num_gradients):
  idx_map = [] # flat_idx --&gt; (val, idx)
  topk_idx = get_topk_idx(grad)
  for i in topk_idx:
    idx_map.append(i)

  grad_flat = torch.abs(grad).flatten()
  topk_values, topk_indices = grad_flat.topk(num_weights)
  topk_tensor = torch.cuda.FloatTensor(grad.size()).fill_(0)

  for i in range(len(topk_indices)):
    idx_expanded = idx_map[topk_indices[i]][1]
    topk_tensor[idx_expanded[0], idx_expanded[1], idx_expanded[2], idx_expanded[3]] = grad[idx_expanded[0], idx_expanded[1], idx_expanded[2], idx_expanded[3]]

  return topk_tensor
</code></pre>
<p>Thanks in advance for the help!</p>

ptrblck: <p>You could calculate the <code>topk</code> of the flattened tensor and use <a href="https://discuss.pytorch.org/t/how-to-do-a-unravel-index-in-pytorch-just-like-in-numpy/12987/3">this implementation</a> to unravel the indices:</p>
<pre><code class="lang-python">def unravel_index(index, shape):
    out = []
    for dim in reversed(shape):
        out.append(index % dim)
        index = index // dim
    return tuple(reversed(out))


x = torch.randn(2, 3, 4, 5)
res = torch.topk(x.view(-1), k=3)

idx = unravel_index(res.indices, x.size())
print(x[idx] == res.values)
&gt; tensor([True, True, True])
</code></pre>
IoU score for Muilticlass segmentation

said_rasidin: <p>I bit confuse how to calculate IoU score and dice for multi class segmentation, is my code valid for multi-class?</p>
<pre><code class="lang-auto">def IoU_score(inputs, targets, num_classes=23, smooth=1e-5):
    with torch.no_grad():
        #soft = nn.Softmax2d()
        inputs = F.softmax(inputs, dim=1) #convert into probabilites 0-1
        targets = F.one_hot(targets, num_classes = n_classes).permute(0,3,1,2).contiguous()#convert target into one-hot


        inputs = inputs.contiguous().view(-1)
        targets = targets.view(-1)

        intersection = (inputs * targets).sum()
        total = (inputs + targets).sum()
        union = total - intersection 

        IoU = (intersection + smooth)/(union + smooth)
        
        return IoU.item()

</code></pre>

ptrblck: <p>Based on <a class="mention" href="/u/tom">@tom</a>’s <a href="https://discuss.pytorch.org/t/iou-pytorch-implementation/21473/3">post</a> the usual way would be to calculate a “class agnostic” loss and it seems he also liked an implementation to it, so you might compare your current approach to it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How do I dynamically swich on/off weight_decay

Yinda_XU: <p>TL:DR</p>
<p>How do I manually “mask out” some weights (element-level) from being penalized by weight_decay in PyTorch optimizer (e.g. optim.SGD)? For example, the weights that are not involved in the forward computation.</p>
<p>Here is a minimum runnable example (MRE) to explain what I wish to do:</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch import optim
# parameter
weight = nn.Parameter(torch.ones((256, 256, 3, 3)))
bias = nn.Parameter(torch.ones((256)))
print("before weight_decay:", weight[128, 128, :, :].detach().numpy())
# build optimizer
optimizer = optim.SGD([weight, bias], lr=1e-2, weight_decay=1e-2)  # large wd for demo.
optimizer.zero_grad()
# cropped from parameter (weight/bias)
#   Note that weight[128, 128, :, :] is not involed in the forward computation
weight_ = weight[:128, :128, :, :]
bias_ = bias[:128]
# one training step
x = torch.ones((1, 128, 14, 14))
y = nn.functional.conv2d(
            x, weight_, bias_, stride=1, padding=1, dilation=1, groups=1)
l = y.sum()
l.backward()
optimizer.step()
# TODO: prevent this part from being penalized by weight_decay since they are not used for computation
print("after weight_decay:", weight[128, 128, :, :].detach().numpy())
</code></pre>

ptrblck: <p>I think the easier approach would be to calculate the weight decay for the “used” parameters manually instead of trying to disable it for “unused” parameters.<br>
<a href="https://discuss.pytorch.org/t/simple-l2-regularization/139/2">Here</a> is a simple example of adding a custom regularization.</p>
Dataloader slow between epochs

wombat: <p>I have a large audio dataset with about 1000 speakers and dozens of utterances per speaker. For the model I’m training I need to sample a largish batch (64) of speakers and then randomly sample 10 utterances per speaker.</p>
<p>I’ve created a <code>Dataset</code> which indexes over the speakers with the <code>__getitem__</code> method lazily returning the 10 utterances. This works well but is slow between epochs when the dataloader needs to spawn new processes, etc. Since there are only 16 batches per epoch this quickly becomes a large amount of wasted time.</p>
<p>Any recommendations would be appreciated. Thanks!</p>

ptrblck: <p>Do you load anything in your <code>Dataset</code>'s <code>__init__</code> method?<br>
The spawning of the multiple workers shouldn’t create a bottleneck, if you lazily load the data in <code>__getitem__</code>.</p>

wombat: <p>I just load a list of files in the <code>__init__</code>. I don’t think it is process spawning that’s causing the bottleneck. The call to <code>__getitem__</code> has to do a lot of IO so I think when the dataloader starts and none of the workers have prefetched any data it takes a while to start. In the middle of an epoch this isn’t a problem because some of the worker have already fetched data.</p>

zylo117: <p>I am also suffering this issue. When I set num_workers to 0, this issue goes away. But with larger num_workers, the time between epochs is longer, especially when batchsize is larger like 256.<br>
Did dataloader repawn processes everytime it runs out of data like when epoch ends?<br>
Its spawning overhead takes more time than the train itself. I think it should be optimized.</p>

kinwai_cheuk: <p>Do you know what is the cause? I am also facing the same problem on linux. When num_workers is 0, it goes right to the next epoch. But when num_workers &gt; 0, it takes few seconds to go to the next epoch. The larger the num_worker value, the slower the between epoch transition is.</p>

ptrblck: <p>Have a look at <a href="https://discuss.pytorch.org/t/enumerate-dataloader-slow/87778">this topic</a> for a potential explanation of this effect and a workaround. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to avoid reloading Pytorch ML model in Flask REST API for every sequential resquests

rajarajan: <p>Hello there, I have a build a Flask Rest API to classify the images using RESNET50 in PyTorch, my problem is, model and weights are loaded for each request, is there any way to load only once for the overall life cycle of the API.</p>

ptrblck: <p>I assume your Flask application would have some kind of initialization method, where you could setup your model and just create the prediction for each request.<br>
Based on your description it seems that you are recreating the model for each request, is that correct?</p>
Exchange the gradients of two networks

oasjd7: <p>Hi, all.<br>
I want to exchange the gradients of two networks.<br>
then this code works fine?</p>
<pre><code class="lang-auto">loss.backward()
for p,q in zip(model1.parameters(), model2.parameters()):
    p.grad = q.grad
    q.gard = p.grad
optimizer.step()
</code></pre>

ptrblck: <p>Yes, this should be possible (besides the minor typo in <code>q.gard</code> <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).<br>
I’m not sure, what the second assignment should do, as the first one would already assign the gradients from <code>model2</code> to <code>model1</code>.<br>
The optimizer will just use the <code>.grad</code> attributes of all parameters without any knowledge how these gradients were created (at least in the default use cases).</p>
Applying transformations for a train and test data

vishnu_vardhan1: <p>Hi, I am using Cats and Dogs dataset by microsoft. The dataset consists of 25,000 images where all the images of cats contain in cats folder and all the images of dogs contain in the Dogs folder.<br>
However, I split the data into train,valid and test data as given below:</p>
<blockquote>
<p>data_transf = transforms.Compose([<br>
transforms.CenterCrop(233),<br>
transforms.ToTensor(),<br>
transforms.Normalize(mean =  [0.485, 0.456, 0.406],<br>
std  =  [0.229, 0.224, 0.225])])</p>
<p>train_data = datasets.ImageFolder(root= ‘/content/PetImages’, transform = data_transf)<br>
valid_size = 0.2<br>
test_size  = 0.1</p>
<p>num_train = len(train_data)<br>
indices = list(range(num_train))<br>
np.random.shuffle(indices)<br>
split = int(np.floor(valid_size * num_train))<br>
train_idx, valid_idx = indices[split:], indices[:split]</p>
<h1>define samplers for obtaining training and validation batches</h1>
<p>train_sampler = SubsetRandomSampler(train_idx)<br>
valid_sampler = SubsetRandomSampler(valid_idx)</p>
<p>num_train = len(train_sampler)<br>
indices = list(range(num_train)<br>
np.random.shuffle(indices)<br>
split = int(np.floor(test_size * num_train))<br>
train_idx, test_idx = indices[split:], indices[:split]<br>
train_sampler = SubsetRandomSampler(train_idx)<br>
test_sampler = SubsetRandomSampler(test_idx)</p>
<h1>prepare data loaders</h1>
<p>trainloader = DataLoader(dataset = train_data, batch_size=32, drop_last=True,sampler=train_sampler)<br>
validloader = DataLoader(dataset = train_data, batch_size=32,drop_last=True,sampler=valid_sampler)<br>
testloader = DataLoader(dataset = train_data, batch_size=32,drop_last=True,sampler=test_sampler)</p>
</blockquote>
<p>Can anyone help me in applying data augmentation techniques to train data and valid data  without applying it on test data?</p>

ptrblck: <p>You could create multiple <code>ImageFolders</code> for training, validation, and testing, and pass the corresponding transformation to each of these.<br>
Once these datasets are created, you could reuse the <code>SubsetRandomSampler</code> approach and create the <code>DataLoaders</code>.<br>
Since <code>ImageFolder</code> will load the data lazily, you shouldn’t see a slowdown or any other performance hit.</p>
LSTM CPU memory leak on specific batchsize and hidden size

pol1: <p>Hello, I’ve encountered a memory leak on a LSTM model and condensed the issue into the following code.</p>
<p>I’ve tried the following solutions:</p>
<ol>
<li>Detach hidden state using repackage_hidden() <a href="https://discuss.pytorch.org/t/help-clarifying-repackage-hidden-in-word-language-model/226/8">https://discuss.pytorch.org/t/help-clarifying-repackage-hidden-in-word-language-model/226/8</a>
</li>
<li>gc.collect()</li>
<li>torch.nograd()</li>
</ol>
<p>However, problem still persists using pytorch 1.5.1, and my machine has 64GB of ram. Any help is appreciated!</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import psutil, os, gc

def repackage_hidden(h):
    """Wraps hidden states in new Tensors, to detach them from their history."""

    if isinstance(h, torch.Tensor):
        return h.detach()
    else:
        return tuple(repackage_hidden(v) for v in h)

# Doesn't leak memory
# batch = 3
# hidden_size = 256

# batch = 3
# hidden_size = 512

# batch = 6
# hidden_size = 256


# Leaks memory
batch = 6
hidden_size = 512

rnn = nn.LSTM(320, hidden_size, num_layers=5, bidirectional=True)
x = torch.randn(5, batch, 320)
h0 = torch.randn(10, batch, hidden_size)
c0 = torch.randn(10, batch, hidden_size)
with torch.no_grad():
    for i in range(1000):
        print(i, psutil.Process(os.getpid()).memory_info().rss)    
        output, hidden = rnn(x, (h0, c0))
        hidden = repackage_hidden(hidden)
        gc.collect()
</code></pre>

ptrblck: <p>For the sake of completeness: issues seems to be solved <a href="https://discuss.pytorch.org/t/lstm-on-cpu-wont-release-memory-when-all-refs-deleted/89026/16">here</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Regarding precision

Ganga: <p>my batch_size is 256(default),i was evaluating the model by checking its precission value,for inputs more than the batch_size i am getting good precission,but for input images less than 256 i am getting very poor precission value<br>
my precision is calculated like below…<br>
it will compare the prediction with the target if it matches it will result in “1”,such that it will count the number of resulted “1s” and stored in ‘correct_k’ and formula is…(correct_k/batch_size)*100</p>
<p>Another problem is ,for input &gt;256 target is starting from [0,1,2…255],and predicted value has the index starting  same as the target …so while comparing i am getting the good precision value…BUT for input&lt;256 target is starting from [1,2,…256],but prediction is happening correct ,so while comparing prediction with target i am getting very less precision</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>  can you please help in how target is calculated<br>
here is the code…<br>
“”“prec1, prec5 = accuracy(output.data, target, topk=(1, 5))<br>
def accuracy(output, target, topk=(1,)):<br>
“”“Computes the precision@k for the specified values of k””"<br>
maxk = max(topk)<br>
batch_size = target.size(0)</p>
<pre><code>_, pred = output.topk(maxk, 1, True, True)
pred = pred.t()
correct = pred.eq(target.view(1, -1).expand_as(pred))

res = []
for k in topk:
    correct_k = correct[:k].view(-1).float().sum(0, keepdim=True)
    res.append(correct_k.mul_(100.0 / batch_size))
return res"""</code></pre>
PyTorch separate image by red and green channel

alameer: <p>I have a set of images that contains green and red cells. I would like to be able to count the green cells and red cells. The current code calculate both type of cells. I would like to separate them. Any advice would appreciated. Thanks</p>
<pre><code class="lang-auto">class CellsDataset(Dataset):
    # a very simple dataset

    def __init__(self, root_dir, transform=None, return_filenames=False):
        self.root = root_dir
        self.transform = transform
        self.return_filenames = return_filenames
        self.files = [os.path.join(self.root,filename) for filename in os.listdir(self.root)]
        self.files = [path for path in self.files
                      if os.path.isfile(path) and os.path.splitext(path)[1]=='.png']

    def __len__(self):
        return len(self.files)

    def __getitem__(self, idx):
        path = self.files[idx]
        sample = Image.open(path)
        #transform3 = Grayscale(num_output_channels=3)
        #sample = transform3(sample) # convert to a 3 channel grayscale, as it needs to be 3 channel.
        if self.transform:
            sample = self.transform(sample)

        if self.return_filenames:
            return sample, path
        else:
            return sample
</code></pre>

ptrblck: <p>How is the current code counting these cells?<br>
Based on the code snippet you shared, it seems you are only loading the data.</p>