akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
IoU saturates regardless the backbone used

Beinan_Wang: <p>This is more of a deep learning question than a PyTorch question. I am observing a weird phenomenon when training semantic segmentation networks. In short, my training program gives (roughly) the same final IoU scores no matter what backbone I use. I have tried ResNet (34, 50, 101), ResNeXt-50, WideResNet-50, WiderResNet-38, Xception, etc. This is uncommon because a stronger backbone should result in a better performance when used for transfer learning, right? I have observed that in the first few epochs stronger backbones indeed lead to faster performance improvement. However, they also reach a plateau faster so that weaker backbones eventually catch up. To give more details, all models are trained with the exact same configuration. I’m trying to figure out want hyperparameter is causing this saturation phenomenon. The one I’m suspecting the most is the batch size. Due to budget reasons, I can only use a batch size of 2. I know that most SoTA semantic segmentation networks are trained with a batch size of 8 or greater. However, a batch size of 2 has been leading to better than SoTA results for low complexity backbones (ResNet18, ResNet34). For ResNet34, the IoU score is only increased by 0.2 when increasing the batch size from 2 to 4. I can’t test how batch size affects stronger backbones because of insufficient memory. The loss function I use is the sum of categorical cross entropy and log jaccard.</p>
<p>Edit 1: I forgot to mention I had to cast models to half() because of insufficient memory.</p>
Rewriting spdiags & spsolve?

nickthoukas: <p>I’m trying to rewrite some legacy code and switch from scipy to pytorch and I’m having difficulties re-writing the spdiags and spsolve.</p>
<p>How could I perform these operations on GPUs?</p>
<pre><code class="lang-auto">A = spdiags(B.T,d,k,k)

A = A + A.T + spdiags(D.T, 0, k, k)

return spsolve(A, input)
</code></pre>

albanD: <p>Hi,</p>
<p>I’m afraid we don’t have support for these functions on sparse Tensors.<br>
You can check the <a href="https://pytorch.org/docs/stable/sparse.html?highlight=sparse">sparse doc</a> to see what is available.</p>
<p>You can use the Dense version of these methods though if this is not too memory intensive.<br>
Also sparse operations will most likely be slower than dense ones on GPU (even though they use less memory).</p>
How to channel-wise multiply an matrix and a vector

ginobilinie: <p>Hi all,</p>
<p>I’d like to implement a function like the squeeze-excitation attention, for example, we have a matrix BxCxHxW, and we also have an C-dim vector (both are in the form of tensor). I’d like to channel-wise multiply the matrix and vector. How can I implement it?</p>
<p>Previously, in senet, we just do it by: mat*camap, but I have tested it on pytorch 1.2, it shows<br>
where mat: 3x16x16, camat: 3-dim vector</p>
<blockquote>
<blockquote>
<blockquote>
<p>mat*camap</p>
</blockquote>
</blockquote>
</blockquote>
<p>Traceback (most recent call last):</p>
<p>File “”, line 1, in </p>
<p>RuntimeError: The size of tensor a (16) must match the size of tensor b (3) at non-singleton dimension 2</p>

Sunshine352: <p>You can do like this:<br>
mat * camap.unsqueeze(dim=-1). unsqueeze(dim=-1)</p>

ginobilinie: <p>Thanks. I’ll try. One more question, if we have the batch dimension in mat, we also have to expand the batch dimension for the camap or not?</p>

Sunshine352: <p>Yes, it is necessary.</p>

ginobilinie: <p><a class="mention" href="/u/sunshine352">@Sunshine352</a>, Thanks so much.</p>
Out of place sigmoid function

avinash_m: <p>Hi all,</p>
<p>I am getting the following error:<br>
one of the variables needed for gradient computation has been modified by an inplace operation.</p>
<p>When I used pytorch anamoly detection it showed following error:<br>
File “”, line 36, in mask<br>
temp2 = torch.sigmoid(temp2)</p>
<p>How should I perform out of place operation on sigmoid function??</p>
<p>Thank you</p>

albanD: <p>Hi,</p>
<p>This is already out of place.<br>
This stack shows you the function that needs the value of its output. But that output was modified inplace.<br>
So you need to look for the lines below where <code>temp2</code> is modified inplace, and make sure it is not anymore.<br>
If you’re not sure, a simple fix is to replace this line with: <code>temp2 = torch.sigmoid(temp2).clone()</code> so that <code>temp2</code> does not point to the output of the sigmoid anymore but a fresh Tensor.</p>
Generating ideal input from a backpropagation of a pre-trained model

Chame_call: <p>Is that possible to generate an input which seems ideal w.r.t. model by backpropagation of the model?</p>
<p>Here I have such arch for classification sequences into 2 categories:</p>
<pre><code class="lang-auto">class SDNet(nn.Module):
    def __init__(self, input_size=2, hidden_layer_size=10, output_size=1):
        super().__init__()
        self.hidden_layer_size = hidden_layer_size

        self.lstm = nn.LSTM(input_size, hidden_layer_size)

        self.linear = nn.Linear(hidden_layer_size, output_size)
        self.sigmoid = nn.Sigmoid()

        self.hidden_cell = None

    def forward(self, input_seq):

        lstm_out, self.hidden_cell = self.lstm(input_seq.view(len(input_seq) ,1, -1), self.hidden_cell)
        out = self.linear(lstm_out.view(len(input_seq), -1))[-1]
        out = self.sigmoid(out)
        return out
    
    def init_hidden(self):
        self.hidden_cell = (torch.zeros(1,1,self.hidden_layer_size),
                            torch.zeros(1,1,self.hidden_layer_size))

    def evaluate(self, seq):
	    #rnn.eval()
	    self.init_hidden()
	    seq = torch.tensor(seq, dtype = torch.float)
	    with torch.no_grad():
	            y_pred = self(seq)
	    return y_pred.item()
</code></pre>
<p>Can I take an output like for example 1 or 0 (or near it cause an activation is sigmoid) and generate an input for the model which will give such a prediction???</p>

ptrblck: <p>You could probably set <code>requires_grad=True</code> in your input, calculate the gradients using your desired target, and optimize the input instead of the model parameters.</p>

Chame_call: <pre><code class="lang-auto">input_tensor = torch.tensor(two_line_seq, dtype = torch.float, requires_grad=True)

?.backward(torch.tensor([1.]))
input_tensor += input_tensor.grad
</code></pre>
<p>But from which tensor I must make backpropagation?</p>
<p>Suppose I have to do it with last output of the net then we have</p>
<blockquote>
<p>input_tensor = torch.tensor(two_line_seq, dtype = torch.float, requires_grad=True)<br>
pred = rnn(input_tensor)<br>
single_loss = loss(pred, label)<br>
single_loss.backward(torch.tensor([1.]))<br>
input_tensor += input_tensor.grad</p>
</blockquote>
<p>but <code>torch.tensor([1.])</code> is the default argument for backward so I’m confused…</p>

Chame_call: <p>Can you tell on which tensor I have to work backwards?<br>
Is that have to be model’s loss?</p>

ptrblck: <p>You could pick a specific class of your pretrained model and use the standard training routine.<br>
However, this should result in some pattern-like input image, but I assume that’s what you are looking for.</p>
<p>Also, <a href="https://distill.pub/2017/feature-visualization/">this blogpost</a> might be interesting to read for some ideas.</p>
Can't i change the activation value of a model before loss.backward operation?

111189: <p>Hi. While training with CIFAR-10 dataset and ResNet-20, I want to change the activation value in backpropagation, but I got the following error.</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 16, 32, 32]], which is output 0 of ReluBackward1, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
<p>Can’t I change the activation value saved in the forward path during the backward path?</p>
<p>Thank you.</p>

albanD: <p>You can change it. But if you change it inplace and something else needed the original value (the relu in this case), then the relu cannot compute its backward anymore. Hence the error.</p>

111189: <p>Thank you for your reply. Then, is there no way to use the changed activation instead of the existing activation in calculating the gradient?</p>

albanD: <p>But then you won’t compute gradients anymore. That is fine?</p>

111189: <p>I want to see the code how to calculate the gradient from an existing built-in function(for example, torch.nn.Conv2d). If that’s possible, Maybe i can change the activation value to whatever i want in calculating the gradient</p>

albanD: <p>There is no unified way to compute these.<br>
But even if you do this, you won’t computes gradients anymore.</p>

111189: <p>That’s too bad. Okay. Thank you for your answer.</p>
Apply a nn.Module function to data in batches

maxmatical: <p>For example, I have data with features <code>features</code> in an <code>n x m1 x m2</code> tensor, and a modelthat I’m using to transform the input <code>features</code> by putting it through the model</p>
<pre><code class="lang-auto">

class model(nn.Module):
    def __init__(self):
        super(model, self).__init__()
        ...

    def forward(self, x):
        ...
        return x

model = model()
</code></pre>
<p>However due to the size of the data, I get a cuda out of memory error if I just do</p>
<pre><code class="lang-auto">new_features = model(features)
</code></pre>
<p>So I want to process the data in batches using a <code>DataLoader</code>, but what should I do after making the batch predictions to get the <code>new_features</code> as an <code>n x m1 x m2</code> tensor?</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">Data loading tutorial</a> might be a good starter to familiarize yourself with the <code>Dataset</code> and <code>DataLoader</code>.<br>
If you have already loaded all samples before, you could pass them to a <code>TensorDataset</code> (alternatively you could also write your own custom <code>Dataset</code>) and just wrap it in a <code>DataLoader</code>.</p>

maxmatical: <p>From the documentation, to create a <code>TensorDataset</code> I just need the <code>x</code> and <code>y</code> values, and pass that into the <code>TensorDataset</code>? Does it require the <code>y</code> values be in a one-hot encoded tensor (so <code>n x n_classes)</code> or can it be just an <code>n x 1</code> tensor?</p>

ptrblck: <p>The shape of the target depends on your use case.<br>
<code>TensorDataset</code> will take whatever tensors you pass to it.</p>
<p>However, if you are dealing with a multi-class classification (and use e.g. <code>nn.CrossEntropyLoss</code>), your targets should be <code>LongTensors</code> containing the class indices.<br>
For a model output of <code>[batch_size, nb_classes]</code>, your targets would be <code>[batch_size]</code> with values in <code>[0, nb_classes-1]</code>.</p>
Batchnorm fine tuning

Dan_Erez: <p>Hi,<br>
tl;dr : Does batchnorm use “learned” mean/variance to normalize data when in eval() mode? If so how do I get to mean/variance of batchnorm ( Not the affine transformation after normalization ).</p>
<p>Longer version:<br>
I have a trained network. I have some new unlabeled data with the same categories as before but a slightly different domain. For simplicity let’s say I trained a “dog type” classifier on images taken during the day and now I want to fine-tune on new images that were taken at night.</p>
<p>I want to update the normalization factors in group norm to reflect the new data’s statistics. To the best of my understanding group norm during inference = 1) normalization with learned mean/std + 2) a learned affine transformed.</p>
<p>I only see the parameters of the affine transform. Is there a way to get to the mean/std and change it.</p>
<p>I tried to bypass this by training the network with a constant loss of zero ( since the mean/var are not dependent on the loss ). This did not do anything…</p>
<p>Thanks,<br>
Dan</p>

Dan_Erez: <p>according to <a href="https://github.com/pytorch/pytorch/issues/5406" rel="nofollow noopener">this</a> there should be some parameters such as running_mean , running_std.<br>
How do I get to them and can I change them- preferably by running a bunch of unlabeled input examples so that the running mean is slowly adjusted toward a new mean.</p>

Dan_Erez: <aside class="quote" data-post="1" data-topic="43959">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sudarshan_vb/48/11291_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/does-group-norm-maintain-an-running-average-of-mean-and-variance/43959">Does group norm maintain an running average of mean and variance?</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Does group norm maintain an running average of mean and variance ?
  </blockquote>
</aside>
3d CAM or visualization of activations for 3d data

nabsabs: <p>Hi all, I am looking for a way to visualize 3D activations from a classification based network. If I try to use CAM, I would get <em>one</em> attention map for for every one of the slices of my input tensor (3d medical data) but that’s not really indicative of which slice and which spatial co-ordinates activated the classification label. This may be an open research question but I’m wondering if there is any way to visualize this that is common in the community - any help appreciated!</p>
<p>PS - I checked out the most relevant topic here in <a href="https://discuss.pytorch.org/t/understanding-deep-network-visualize-weights/2060/10">this</a> post which only tried to visualize filters but thats not my exact use case.</p>
A invalid link in pytorch homepage

simiao_lai: <p>In this website:<a href="https://pytorch.org/get-started/previous-versions/" class="inline-onebox" rel="noopener nofollow ugc">Previous PyTorch Versions | PyTorch</a>, I want to download local whl with cuda8.0 build in <a href="https://download.pytorch.org/whl/cu80/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu80/torch_stable.html</a>, but I can not open this.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d2496a8332b9a126e6e90e39096fbc53777233f.png" data-download-href="https://discuss.pytorch.org/uploads/default/9d2496a8332b9a126e6e90e39096fbc53777233f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d2496a8332b9a126e6e90e39096fbc53777233f.png" alt="image" data-base62-sha1="mq9lUlGMKliBKqoBHugmAl9GZb1" width="690" height="319" data-dominant-color="FAF3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">730×338 31.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
other links like cuda90 and cuda100 is ok.</p>

smth: <p>hi <a class="mention" href="/u/simiao_lai">@simiao_lai</a></p>
<p>It just looks like the links exist, but they are hidden. If you View Source of the page, you will see the links.</p>
<p>We will work on fixing this.</p>
PyTorch community members are invited to take part in the Express 2019 F/OSS-Firms Survey!

moneil: <p>Hi everyone</p>
<p>We are looking for participants to help us with a study on the sustainability of free and open source software, ‘Mapping the co-production of digital infrastructure by peer projects and firms’, which is funded by a Sloan and Ford Foundations grant.</p>
<p>We are trying to learn about how commercial firm engagement with F/OSS, for example in the form of paying developers to develop F/OSS, impacts volunteer projects such as PyTorch.</p>
<p>Our quantitative survey of GitHub shows that PyTorch is one of the top recipients of commits by people with a professional email address (eg <span class="mention">@microsoft.com</span>), which is why we are reaching out to members of the PyTorch community.</p>
<p>Participation in the survey requires about five minutes of your time (there are only seven questions). If you are involved in PyTorch as a developer, maintainer or contributor, please follow the link below to answer our questions:</p>
<p><a href="https://unicanberra.limequery.com/626395?lang=en" class="onebox" target="_blank" rel="nofollow noopener">https://unicanberra.limequery.com/626395?lang=en</a></p>
<p>The research team: Mathieu O’Neil, University of Canberra; Xiaolan Cai, University of Canberra; Laure Muselli, Telecom Paris; Mahin Raissi, University of Canberra; Stefano Zacchiroli, Universite Paris Diderot / INRIA &amp; former Debian Project Leader</p>
<p>Duration: The survey will run for 2 weeks, from November 25 to December 9 2019.</p>
<p>More information? Please visit our project webpage: <a href="https://www.canberra.edu.au/research/faculty-research-centres/nmrc/research/mapping-the-co-production-of-digital-infrastructure-by-peer-projects-and-firms" rel="nofollow noopener">https://www.canberra.edu.au/research/faculty-research-centres/nmrc/research/mapping-the-co-production-of-digital-infrastructure-by-peer-projects-and-firms</a></p>
<p>Any questions? Email: <a href="mailto:mathieu.oneil@canberra.edu.au">mathieu[dot]oneil[at]canberra[dot]edu[dot]au</a></p>
<p>This research has been approved by the Human Research Ethics Committee at the University of Canberra (Project no. 1865).</p>
Policy Gradient for NLP

andrei_97: <p>As a beginner in RL, I am totally at a loss on how to implement a policy gradient for NLP tasks (such as NMT).</p>
<p>More specifically, I am trying to fine-tune a pre-trained Seq2Seq model via a policy gradient that gets rewards dependent on BLEU scores.</p>
<p>However, I didn’t manage to find any resources (at least not fit for my experience?) on how to do this?</p>
<p>From what I understand I need to create a loss based on the vocabulary distributions of each time step (of the decoder) and the reward received by a prediction weighted down by a base line. However I am confused how would this look since I am only generating a reward for the complete translation. Also what is the baseline supposed to be (a reward from a non-RL trained model ? ) ?</p>
<p><code>loss = - torch.sum(torch.log( policy(state) * (reward - baseline)))</code><br>
<code>loss.backward()</code></p>
<p>I know this is probably incredibly simple for some of you, but it goes way over my head <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> Would appreciate some help for a noob.</p>
How to handle variable input lenghts for tansformer encoder?

hadaev8: <p>I have variable text lenghts and want to use encoder like in this example<br>
<a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a><br>
For RNNs I should use pack_padded_sequence, what should I do for transformer?<br>
Or it is ok for transformer to get zeros in inputs?</p>
Converting log_softmax layer into ONNX format

Icwhatudidthr: <p>I want to convert a network into ONNX format, and bumped into this problem.</p>
<p>The conversion of log_softmax layer is not supported. This code:</p>
<pre><code class="lang-auto">import torch
import torch.nn
import torch.nn.functional

img_size, num_inputs = 64, 3

class SimpleModel(torch.nn.Module):
    def __init__(self):
        super(SimpleModel, self).__init__()
        self.layer = torch.nn.AvgPool2d(int(64/4))
    def forward(self, x):
        return torch.nn.functional.log_softmax(x, dim=1)

if __name__ == "__main__":
    print("Torch version:", torch.__version__)
    model = SimpleModel()
    dummy_input = torch.randn(10, num_inputs, img_size, img_size, device='cpu')
    torch.onnx.export(model, dummy_input, "simple.onnx", verbose=True)
</code></pre>
<p>generates the following error output:</p>
<pre><code class="lang-auto">Torch version: 1.0.1.post2

Traceback (most recent call last):
  File "test.py", line 18, in &lt;module&gt;
    torch.onnx.export(model, dummy_input, "simple.onnx", verbose=True)
  File "/usr3/ams/radio/rantonio/.local/lib/python3.5/site-packages/torch/onnx/__init__.py", line 27, in export
    return utils.export(*args, **kwargs)
  File "/usr3/ams/radio/rantonio/.local/lib/python3.5/site-packages/torch/onnx/utils.py", line 104, in export
    operator_export_type=operator_export_type)
  File "/usr3/ams/radio/rantonio/.local/lib/python3.5/site-packages/torch/onnx/utils.py", line 287, in _export
    proto, export_map = graph._export_onnx(params, _onnx_opset_version, defer_weight_export, operator_export_type)
RuntimeError: ONNX export failed: Couldn't export operator aten::log_softmax
</code></pre>
<p>Is there a particular reason why porting log_softmax to ONNX is not supported by Pytorch?</p>
<p>Thanks,</p>

davebulaval: <p>Did you found a solution ?</p>

vadimkantorov: <p>The relevant bug report is probably this one: <a href="https://github.com/pytorch/pytorch/issues/20643" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/20643</a></p>
_pickle.UnpicklingError: pickle data was truncated

Aisen: <p>I’ve installed pytorch 1.0 on ubuntu18.04.<br>
When I try to use webcam demo provided by  maskrcnn-benchmark. An error occured:<br>
Traceback (most recent call last):<br>
File “webcam.py”, line 80, in <br>
main()<br>
File “webcam.py”, line 64, in main<br>
min_image_size=args.min_image_size,<br>
File “/home/aisen/github/maskrcnn-benchmark/demo/predictor.py”, line 149, in <strong>init</strong><br>
_ = checkpointer.load(cfg.MODEL.WEIGHT)<br>
File “/home/aisen/github/maskrcnn-benchmark/maskrcnn_benchmark/utils/checkpoint.py”, line 61, in load<br>
checkpoint = self._load_file(f)<br>
File “/home/aisen/github/maskrcnn-benchmark/maskrcnn_benchmark/utils/checkpoint.py”, line 134, in _load_file<br>
return load_c2_format(self.cfg, f)<br>
File “/home/aisen/github/maskrcnn-benchmark/maskrcnn_benchmark/utils/c2_model_loading.py”, line 206, in load_c2_format<br>
return C2_FORMAT_LOADER[cfg.MODEL.BACKBONE.CONV_BODY](cfg, f)<br>
File “/home/aisen/github/maskrcnn-benchmark/maskrcnn_benchmark/utils/c2_model_loading.py”, line 192, in load_resnet_c2_format<br>
state_dict = _load_c2_pickled_weights(f)<br>
File “/home/aisen/github/maskrcnn-benchmark/maskrcnn_benchmark/utils/c2_model_loading.py”, line 136, in _load_c2_pickled_weights<br>
data = pickle.load(f, encoding=“latin1”)<br>
_pickle.UnpicklingError: pickle data was truncated</p>

anantguptadbl: <p>Hello <a class="mention" href="/u/aisen">@Aisen</a>, can you post the code which dumped the pickle file which you are trying to load</p>
Non linear regression methods

Aditya_Raj_P15VSS201: <p>Hi every one,</p>
<p>I want to predict continuous values from a CNN  model as output by giving an image as input:<br>
below is the code</p>
<p>class MultiLabelNN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(MultiLabelNN, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3,64, 5)<br>
self.pool = nn.MaxPool2d(2,2)<br>
self.conv2 = nn.Conv2d(64, 128, 5)<br>
self.conv3 = nn.Conv2d(128, 256, 5)<br>
self.conv4 = nn.Conv2d(256,320,5)<br>
self.fc1 = nn.Linear(250880,2048)<br>
self.fc2 = nn.Linear(2048, 1024)<br>
self.fc3 = nn.Linear(1024, 512)<br>
self.fc4 = nn.Linear(512, 6)<br>
def forward(self, x):<br>
x = self.conv1(x)<br>
x = nn.ReLU(x)<br>
x = self.pool(x)<br>
x = self.conv2(x)<br>
x = nn.ReLU(x)<br>
x = self.pool(x)<br>
x = self.conv3(x)<br>
x = nn.ReLU(x)<br>
x = self.pool(x)<br>
x = self.conv4(x)<br>
x = nn.ReLU(x)<br>
x = self.pool(x)<br>
x = x.view(-1, 250880)<br>
x = self.fc1(x)<br>
x = self.fc2(x)<br>
x = self.fc3(x)<br>
x = self.fc4(x)<br>
return x</p>
<p>I want to do support vector regression instead of linear regression in my last fully connected layer self.fc4 = nn.Linear(512, 6)…</p>
<p>my question is that is there any option to do this ??</p>

anantguptadbl: <p>Hello <a class="mention" href="/u/aditya_raj_p15vss201">@Aditya_Raj_P15VSS201</a></p>
<p>In order to perform <strong>Support Vector Regression</strong>, you will have to add a <strong>HingeLoss</strong> element to your code</p>
<p>A sample script would be the following</p>
<pre><code class="lang-auto">
# HINGE LOSS
import os
import numpy as np
import torch
from torch import nn
from torch.autograd import Variable

torch.manual_seed(42)
class SVR(nn.Module):
    def __init__(self):
        super(SVR,self).__init__()
        self.linearModel=nn.Linear(100,1)
        
    def forward(self,x):
        x = self.linearModel(x)
        return x
    
model=SVR()
def hingeLoss(outputVal,dataOutput,model):
    loss1=torch.sum(torch.clamp(1 - torch.matmul(outputVal.t(),dataOutput),min=0))
    loss2=torch.sum(model.linearModel.weight ** 2)  # l2 penalty
    totalLoss=loss1+loss2
    return(totalLoss)

optimizer=torch.optim.SGD(model.parameters(),lr=0.001)

X=np.random.rand(1000,100).astype(np.float32)
Y=np.random.randint(2,size=(1000)).reshape(1000,1).astype(np.float32)

for epoch in range(10000):
    inputVal=Variable(torch.from_numpy(X))
    outputVal=Variable(torch.from_numpy(Y))
    optimizer.zero_grad()
    modelOutput = model(inputVal)
    totalLoss=hingeLoss(outputVal,modelOutput,model)
    totalLoss.backward()
    optimizer.step()
    if epoch % 100 == 0:
        print('epoch [{}/{}], loss:{:.4f}'.format(epoch + 1, 100, totalLoss))
</code></pre>

Aditya_Raj_P15VSS201: <p>Please accept my sincere thanks for your valuable suggestion and code</p>
Save each individual image in grid image

creatives07: <p>I want to save each individual image in grid image. How should I modify this code? Or is there an answer that I would like to refer to?</p>
<pre><code>class Generator(nn.Module):
def __init__(self):
    super(Generator, self).__init__()

    self.label_emb = nn.Embedding(opt.n_classes, opt.latent_dim)

    self.init_size = opt.img_size // 4  # Initial size before upsampling
    self.l1 = nn.Sequential(nn.Linear(opt.latent_dim, 128 * self.init_size ** 2))

    self.conv_blocks = nn.Sequential(
        nn.BatchNorm2d(128),
        nn.Upsample(scale_factor=2),
        nn.Conv2d(128, 128, 3, stride=1, padding=1),
        nn.BatchNorm2d(128, 0.8),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Upsample(scale_factor=2),
        nn.Conv2d(128, 64, 3, stride=1, padding=1),
        nn.BatchNorm2d(64, 0.8),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Conv2d(64, opt.channels, 3, stride=1, padding=1),
        nn.Tanh(),
    )
</code></pre>
<p>…</p>
<pre><code>generator = Generator()
</code></pre>
<p>…</p>
<pre><code>def sample_image(n_row, batches_done):
    """Saves a grid of generated digits ranging from 0 to n_classes"""

    # Sample noise
    z = Variable(FloatTensor(np.random.normal(0, 1, (n_row ** 2, opt.latent_dim))))

    # Get labels ranging from 0 to n_classes for n rows
    labels = np.array([num for _ in range(n_row) for num in range(n_row)])
    labels = Variable(LongTensor(labels))
    gen_imgs = generator(z, labels)
    save_image(gen_imgs.data, "images/%d.png" % batches_done, nrow=1, normalize=True)</code></pre>
Confusion matrix liver lesion 4 classes

GK092: <p>i combined cnn(squeezenet)+lstm for liver classification i got 91% as train accuracy and 0.276 for loss…but i want to know why the result of my conf max is 0 everywhere, please help!<br>
classes = [‘Hcc’, ‘Meta’, ‘Hema’, ‘Heal’]<br>
out_dir = “./”<br>
set_name = "Liver Lesion’’<br>
model = combine()<br>
model. load_state_dict(torch.load(’/gk4_net.pth’))<br>
y_pred, y_true, y_prob = get_output(model, testloader)<br>
cnf_matrix = confusion_matrix(y_pred, y_true, labels=np.arange(4))<br>
show_confMat(cnf_matrix, classes, set_name, out_dir)</p>
How to check whether model loading is successful and how to release model in version 1.2 or 1.3

cyanM: <p>According to the release note, there are same different between nightly version  and old version to load a model.</p>
<p>In the old version, the code is like this:</p>
<ul>
<li>std::shared_ptr&lt;<a>torch::jit::script::Module</a>&gt; model  = torch::jit::load(“model.py”);</li>
</ul>
<p>and now it change into :</p>
<ul>
<li>torch::jit::script::Module model  = torch::jit::load(“model.py”);</li>
</ul>
<p>and  in the old version, i can use nullptr to check whether loading is successful:</p>
<ul>
<li>if (model == nullptr) std::cout &lt;&lt; “failed to load model” &lt;&lt; std::endl;</li>
</ul>
<p>but how can i do this in the version after 1.2？</p>
<p>there have same problem when to release loaded model to load another one.</p>
<p>Can someone tell me how to do these, thanks.</p>
<p><a href="https://discuss.pytorch.org/t/how-to-check-whether-model-loading-is-successful-in-version-1-2/54389">related issue</a></p>
Custom Dataloader/dataset (one sample for each class)

crazatos: <p>How do I go about creating a imageFolderDataset that has one sample for each class?</p>

ptrblck: <p>The approach depends a bit on the type of data you are working with.<br>
E.g. if each sample is an image, you could move each image to a separate folder and let <code>ImageFolder</code> create the labels for it.</p>
<p>On the other hand, if you are working with another type of data, you would most likely create the labels manually using some logic, and pass it to <code>TensorDataset</code> or write a custom <code>Dataset</code> to fit your needs.</p>
How optimizer changes weights

dato_nefaridze: <p>take look at my code:</p>
<pre><code class="lang-auto">optimizer=optim.Adam(net.parameters(),lr=0.001)
 optimizer.step()
</code></pre>
<p>how does optimizer changes net parameters? it is passed once and then they don’t have any connection.is there any pointer stuff there?or i missed something while learning pytorch\python.</p>

ptrblck: <p>The optimizer will use the references to all passed parameters and update them in the <code>step</code> function.<br>
You could check the <code>id</code> of each parameter in e.g. <code>optimizer.param_groups[0]['params']</code> and will see that the ids match the parameter ids in the model.</p>
I want numpy define loss function. I need use np.linalg.det()

Mrfangfang: <p>Signature of method ‘loss.forward()’ does not match signature of base method in class ‘Function’ less… (⌘F1)<br>
Inspection info: This inspection detects inconsistencies in overriding method signatures.</p>
<p>class loss(Function):</p>
<pre><code>def forward(x,INPUT):

    batch_size = x.shape[0]

    X = x.numpy()
    input = INPUT.numpy()
    Loss = 0
    for i in range(batch_size):
        R_r = input[i,0:4]
        R_i = input[i,4:8]
        H_r = input[i,8:12]
        H_i = input[i,12:16]
        R = np.concatenate((R_r, R_i), axis=1)
        H = np.concatenate((H_r, H_i), axis=1)

        T_r = input[i,16:32]
        T_i = input[i,32:48]
        temp_t1 = np.concatenate((T_r,T_i),axis=1)
        temp_t2 = np.concatenate((-T_i,T_r),axis=1)
        T = np.concatenate((temp_t1,temp_t2),axis=0)

        phi_r = np.diagonal(X[i,0:4])
        phi_i = 1 - np.power(phi_r,2)
        temp_phi1 = np.concatenate((phi_r,phi_i),axis=1)
        temp_phi2 = np.concatenate((-phi_i, phi_r), axis=1)
        phi = np.concatenate((temp_phi1,temp_phi2),axis=0)

        temp1 = np.matmul(R,phi)
        temp2 = np.matmul(temp1,T)
        H_hat = H + temp2

        Q_r = np.zeros((4,4))
        Q_r[np.triu_indices(4,1)] = X[i,4:10]
        Q_r = Q_r + Q_r.T
        row,col = np.diag_indices(4)
        Q_r[row,col] = X[i,10:14]
        Q_i = np.zeros((4,4))
        Q_i[np.triu_indices(4,1)] = X[i,14:20]
        Q_i = Q_i - Q_i.T

        temp_Q1 = np.concatenate((Q_r,Q_i),axis=1)
        temp_Q2 = np.concatenate((-Q_i,Q_r),axis=1)
        Q = np.concatenate((temp_Q1,temp_Q2),axis=0)

        H_hat_r = H_hat[0:4]
        H_hat_i = H_hat[4:8]

        temp_H1 = np.concatenate((-H_hat_i.T,H_hat_r.T),axis=0)
        H_hat_H = np.concatenate((H_hat.T,temp_H1),axis=1)
        temp_result1 = np.matmul(H_hat,Q)
        temp_result2 = np.matmul(temp_result1,H_hat_H)


        Loss += np.linalg.det(1+temp_result2)
    Loss = t.from_numpy(Loss / batch_size)
    return Loss

def backward(grad_output):
    return grad_output</code></pre>

ptrblck: <p>Could you try to adapt your custom <code>Function</code> based on <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">this tutorial</a>?<br>
I assume this error is raised, as your functions are defined in an unexpected way.</p>

Mrfangfang: <p>Thank you very much. I will try it.<br>
My main question is that I do many array operations in numpy like concatenate.<br>
Whether it would do harm to the backward.</p>

ptrblck: <p>As long as you define the <code>backward</code> manually, the numpy operations will work, as you are responsible to calculate the right gradients.</p>
<p>However, if you could swap all numpy methods to the PyTorch equivalents, Autograd will automatically create the backward pass for you.</p>
Lr_scheduler help

usa: <p>Could someone help me understanding this code line by line? Thank you</p>
<p>import math<br>
import numpy as np</p>
<p>from torch.optim.optimizer import Optimizer</p>
<p>class AdjustLR(object):<br>
def <strong>init</strong>(self, optimizer, init_lr, sleep_epochs=5, half=5, verbose=0):<br>
super(AdjustLR, self).<strong>init</strong>()<br>
self.optimizer = optimizer<br>
self.sleep_epochs = sleep_epochs<br>
self.half = half<br>
self.init_lr = init_lr<br>
self.verbose = verbose</p>
<pre><code>def step(self, epoch):
    if epoch &gt;= self.sleep_epochs:
        for idx, param_group in enumerate(self.optimizer.param_groups):
            new_lr = self.init_lr[idx] * math.pow(0.5, (epoch-self.sleep_epochs+1)/float(self.half))
            param_group['lr'] = new_lr
        if self.verbose:
            print('&gt;&gt;&gt; reduce learning rate &lt;&lt;&lt;')</code></pre>

ptrblck: <p>Do you have any questions to a particular line of code?<br>
Could you post, what you’ve understood so far and let us know, what’s confusing?</p>

usa: <p>i have difficulties in understanding how to move it to matlab and which functions i need to change. Is there a toolbox maybe to do that?</p>

ptrblck: <p>Sorry, I’m not familiar with deep learning tools in MATLAB, as I’ve just used it for signal processing in the past.<br>
I’m not sure if this is the best place to ask. Maybe you could get better help in the MATLAB forum, if you explain your use case (i.e. what you would like to achieve).</p>
Why data.random_split produce different distribution of dataset

Jaeju_An: <p>I am currently working with pytorch 0.4.1<br>
In my dataset, there are 10,736 instances that have 9 different classes.<br>
I tried to train my models with same training set and testset in the dataset.<br>
After applying torch.manual_seed(), I splited train and test dataset with data.random_split function. each size is 0.8 and 0.2<br>
The code is here<br>
1st.</p>
<blockquote>
<pre><code>torch.manual_seed(10)
train_dataset, test_dataset_1gram = data.random_split(full_dataset_1gram, [train_size, test_size])
train_loader = data.DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle = False)
test_loader_1gram = data.DataLoader(test_dataset_1gram, batch_size=BATCH_SIZE, shuffle=False)
</code></pre>
</blockquote>
<p>2nd.</p>
<blockquote>
<pre><code># spliting
torch.manual_seed(10)
train_dataset, test_dataset_2gram = data.random_split(full_dataset_2gram, [train_size, test_size])
train_loader = data.DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle = False)
test_loader_2gram = data.DataLoader(test_dataset_2gram, batch_size=BATCH_SIZE, shuffle=False)
</code></pre>
</blockquote>
<p>… last</p>
<blockquote>
<pre><code># spliting
torch.manual_seed(10)
train_dataset, test_dataset_addition = data.random_split(full_dataset_addition, [train_size, test_size])
train_loader = data.DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle = False)
test_loader_addition = data.DataLoader(test_dataset_addition, batch_size=BATCH_SIZE, shuffle=False)
</code></pre>
</blockquote>
<p>the distribution of test datasets is here<br>
1st<br>
[(0, 302),<br>
(1, 467),<br>
(2, 595),<br>
(3, 101),<br>
(4, 6),<br>
(5, 160),<br>
(6, 72),<br>
(7, 244),<br>
(8, 201)]</p>
<p>2nd<br>
[(0, 302),<br>
(1, 467),<br>
(2, 595),<br>
(3, 101),<br>
(4, 6),<br>
(5, 160),<br>
(6, 72),<br>
(7, 244),<br>
(8, 201)]</p>
<p>3rd<br>
[(0, 302),<br>
(1, 467),<br>
(2, 595),<br>
(3, 101),<br>
(4, 6),<br>
(5, 160),<br>
(6, 72),<br>
(7, 244),<br>
(8, 201)]</p>
<p>4th<br>
[(0, 306),<br>
(1, 466),<br>
(2, 596),<br>
(3, 99),<br>
(4, 5),<br>
(5, 157),<br>
(6, 75),<br>
(7, 242),<br>
(8, 202)]</p>
<p>BOOM? why the distribution suddenely different?<br>
the overall dataset would be this<br>
[(0, 1532),<br>
(1, 2470),<br>
(2, 2937),<br>
(3, 447),<br>
(4, 39),<br>
(5, 732),<br>
(6, 387),<br>
(7, 1179),<br>
(8, 1013)]<br>
and I checked all of datasets I have same distribution.</p>
<p>what is the problem?</p>

ptrblck: <p>If you are working with seeds to get the exact same results, you would have to make sure no additional calls to the random number generator are called before the actual split call.</p>
<p>I would recommend to create the split indices once, store them, and reuse for each dataset (e.g. as a <code>Subset</code>), which would make it easier.</p>
<p>On the other hand, if you want to use the seeding approach, you would have to debug your code and make sure that no other calls were made while creating your 4th dataset, which might be cumbersome.</p>
Computing gradient with respect to the learning rate

snxeno: <p>In the code I try to do something like this:</p>
<pre><code class="lang-auto">lr = [nn.Parameter(torch.ones(1)*0.5) for _ in range(n)] #I have a list of learning rates parameters that I want to learn

grad = autograd.grad(train_loss, model.parameters, create_graph=True)

for param_indx, param in enumerate(model.parameters()):
  param.data.add_(-lr[0]*grad[param_indx]) #manual SGD on parameters

model_loss = loss(model(input), y)
autograd.grad(model_loss, lr[0]) # compute loss gradient with respect to the learning rate

</code></pre>
<p>It should theoretically be able to compute that gradient, doesn’t it?</p>
<p>But I get an error saying:</p>
<p><code>RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</code></p>

snxeno: <p>for a longer code version:</p>
<pre><code class="lang-auto">lr = [[nn.Parameter(torch.ones(1)*0.5) for _ in range(N)] for _ in range(parameter_length)] # a parameter_length x N list of learning rates
test_losses = []
for epoch in range(epochs):
    
    for j in range(N):
        train_loss = loss(model(data), y)
        grad = autograd.grad(train_loss, model.parameters())

        for param_indx, param in enumerate(model.parameters()):
        param.data.add_(-lr[param_indx][j] * grad[param_indx])

        test_loss = (model(data_test), y_test)
        test_losses.append(test_loss)
        
    
    loss_wrt_lr = []
    for j in range(N):
        for param_indx, _ in enumerate(model.parameters()):
            lr_grad = autograd.grad(test_losses[j], lr[param_indx][lr]) # this is not computing the gradient
            loss_wrt_lr.append(lr_grad)
</code></pre>
LSTM: Why does enforce_sorted change the size of the expected hidden state?

matthias.l: <p>Hi!<br>
I have several questions about the argument <code>enforce_sorted</code> in the function <code>nn.utils.rnn.pack_padded_sequence</code></p>
<ol>
<li>Depending on this argument the expected hidden state tensors given to lstm are exepeted to be either of dimension <code>batch_size, sequence length, hidden_size</code> or <code>batch_size, input_size, hidden_size</code>.<br>
See <a href="https://github.com/pytorch/pytorch/blob/681b610f35968761435b28a6bc564e17a7f553d0/torch/nn/modules/rnn.py#L518" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/681b610f35968761435b28a6bc564e17a7f553d0/torch/nn/modules/rnn.py#L518</a>.<br>
I don’t understand when the first dimension is useful and what it has to do if the packed sequences are sorted by descending length.</li>
<li>Why is the information about the actual length of the sequence relevant, they are padded to the same length already?</li>
<li>Why are unsorted sequences not ONNX compatible?</li>
</ol>
<p>Thank you very much!</p>
<p>Best,<br>
Matthias</p>
Imagenet meta file not found - while using torchbench

barakb: <p>Hey I’m trying to work with torchbench in order to benchmark on imagent.<br>
I’m getting an error that looks a pytorch problem and not torchbench issue.<br>
The error is:<br>
<code>RuntimeError: ('Meta file not found or corrupted.', 'You can use download=True to create it.')</code><br>
I dived into the source code and I couldn’t understand the why is the meta fiel, whatever it is, missing.</p>
<p>This is how I called torchbench:<br>
ImageNet.benchmark(<br>
model=model,<br>
paper_model_name=‘FixResNeXt-101-32x48d’,<br>
paper_arxiv_id=‘1906.06423’,<br>
input_transform=input_transform,<br>
data_root=“imagenet_dataset”,<br>
batch_size=32,<br>
num_gpu=1<br>
)</p>
<p>Any idea?</p>

barakb: <p>Any luck someone, I still have no clue?</p>
Cvtransforms help

usa: <p>Could someone help me understanding this code line by line? Thank you<br>
import random<br>
import cv2<br>
import numpy as np</p>
<p>def CenterCrop(batch_img, size):<br>
w, h = batch_img[0][0].shape[1], batch_img[0][0].shape[0]<br>
th, tw = size<br>
img = np.zeros((len(batch_img), len(batch_img[0]), th, tw))<br>
for i in xrange(len(batch_img)):<br>
x1 = int(round((w - tw))/2.)<br>
y1 = int(round((h - th))/2.)<br>
img[i] = batch_img[i, :, y1:y1+th, x1:x1+tw]<br>
return img</p>
<p>def RandomCrop(batch_img, size):<br>
w, h = batch_img[0][0].shape[1], batch_img[0][0].shape[0]<br>
th, tw = size<br>
img = np.zeros((len(batch_img), len(batch_img[0]), th, tw))<br>
for i in xrange(len(batch_img)):<br>
x1 = random.randint(0, 8)<br>
y1 = random.randint(0, 8)<br>
img[i] = batch_img[i, :, y1:y1+th, x1:x1+tw]<br>
return img</p>
<p>def HorizontalFlip(batch_img):<br>
for i in xrange(len(batch_img)):<br>
if random.random() &gt; 0.5:<br>
for j in xrange(len(batch_img[i])):<br>
batch_img[i][j] = cv2.flip(batch_img[i][j], 1)<br>
return batch_img</p>
<p>def ColorNormalize(batch_img):<br>
mean = 0.413621<br>
std = 0.1700239<br>
batch_img = (batch_img - mean) / std<br>
return batch_img</p>

ptrblck: <p>Do you have any questions to a particular line of code?<br>
Could you post, what you’ve understood so far and let us know, what’s confusing?</p>
Step iteration in step

slavavs: <p>I have a large dataset. How can I make a selection from this set in order in steps? I use Dataloader, it allows you to do a bat by random sampling or in order. I want the batch to be configured in order, but in increments. For example, 1, 10, 20, 30, 40, 50. This is the first batch. Then, 2, 11, 21, 31, 41, 51 … So can I do this?</p>

ptrblck: <p>You could write your custom <code>sampler</code> and pass it to the <code>DataLoader</code>, which should return the indices as you need them.<br>
Alternatively, you could reduce the length of your <code>Dataset</code> by the factor of your batch size, and create the batch in <code>__getitem__</code> manually using <code>batch_size=1</code> in your <code>DataLoader</code>.<br>
While this approach could be simpler to implement, each batch will be created by a single worker, which might slowdown your code, if the bottleneck is in data loading.</p>
How to use DataParallel correctly?

Hitori940101: <p>I tested the code on <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#sphx-glr-beginner-blitz-data-parallel-tutorial-py" class="inline-onebox" rel="noopener nofollow ugc">Optional: Data Parallelism — PyTorch Tutorials 2.2.0+cu121 documentation</a>. It works well.<br>
But when I changed the code</p>
<p>class Model(nn.Module):<br>
# Our model</p>
<pre><code>def __init__(self, input_size, output_size):
    super(Model, self).__init__()
    self.fc = nn.Linear(input_size, output_size)

def forward(self, input):
    output = self.fc(input)
    print("\tIn Model: input size", input.size(),
          "output size", output.size())

    return output
</code></pre>
<p>model = Model(input_size, output_size)</p>
<p>to</p>
<p>model = nn.Linear(input_size, output_size)</p>
<p>,the result became to <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/2/12c9307a47c71ab2db2363f43cb6b6a621c20403.png" data-download-href="https://discuss.pytorch.org/uploads/default/12c9307a47c71ab2db2363f43cb6b6a621c20403" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/2/12c9307a47c71ab2db2363f43cb6b6a621c20403.png" alt="image" data-base62-sha1="2GbDIPU4KO2WhP5bP104SJFdG1l" width="690" height="147" data-dominant-color="0E0E0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">718×154 3.34 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> . WHY?</p>

ptrblck: <p>Did you wrap your linear layer in <code>nn.DataParallel</code> as well?<br>
Since you are just using an <code>nn.Linear</code> module, the input sizes won’t be printed, which is defined in the original <code>forward</code> method.</p>
<p>Let me know, if I misunderstood your question or what your confusion is about.</p>
Module 'torchaudio.transforms' has no attribute 'DownmixMono'

arminarj: <p>Hi,<br>
I’m using <strong>torchaudio</strong> v0.3.1 and I’m getting  this error while I’m running torchaudio official example in google colab,</p>
<blockquote>
<p><code>module 'torchaudio.transforms' has no attribute 'DownmixMono'</code></p>
</blockquote>
<p><a href="https://pytorch.org/tutorials/beginner/audio_classifier_tutorial.html?highlight=audio" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/audio_classifier_tutorial.html?highlight=audio</a><br>
<a href="https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/audio_classifier_tutorial.ipynb" rel="noopener nofollow ugc">https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/audio_classifier_tutorial.ipynb</a></p>

vincentqb: <p><code>DownmixMono</code> has been deprecated, see <a href="https://github.com/pytorch/audio/releases" rel="nofollow noopener">breaking changes</a>.</p>

huckiyang: <p>If you open to degrade, this works to me.</p>
<p><code>conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=10.0 torchaudio=0.2.0 -c pytorch</code></p>
RuntimeError: Function 'StdBackward0' returned nan values in its 0th output

saransh: <p>Hey, I am trying to implement the following loss function.</p>
<pre><code class="lang-auto">def calculate_asn_loss_t_test_f(original_tensor, reconstructed_tensor, beta = 0.8, lambda_p = 1, lambda_n = 0.01):
    residual_tensor = torch.abs(reconstructed_tensor - original_tensor).float().cuda()
    residual_tensor = residual_tensor.sum(dim = [1, 2, 3]).float().cuda()
    residual_l   = original_tensor.size(0)
    pos_residual = residual_tensor[: residual_l//2].float().cuda()
    neg_residual = residual_tensor[residual_l//2 :].float().cuda()
    pos_mean = pos_residual.mean().cuda().float()
    neg_mean = neg_residual.mean().cuda().float()
    pos_std  = pos_residual.std().cuda()
    neg_std  = neg_residual.std().cuda().float()
    
    a = beta - pos_mean.float()
    b = neg_mean + lambda_p * (pos_std ** 2).float() + lambda_n * (neg_std ** 2).float()
    if a &gt; b.mean():
        print("RETURNING loss a: ", a)
        return a + epsilon
    else:
        print("RETURNING loss b: ", b)
        return b.mean() + epsilon
</code></pre>
<p>But it is giving me the following error:<br>
RuntimeError: Function ‘StdBackward0’ returned nan values in its 0th output.<br>
Any help?</p>

tom: <p>If you have a constant pos_residual or neg_risidual, the std will be 0 and the gradient of the std will be NaN.</p>
<p>Best regards</p>
<p>Thomas</p>