akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Backward pass is outrageously slow

TeaWaterSleep: <p>Hello,<br>
I’m trying to implement text generation RNN-based with sequence of different length with padding and masked crossed-entropy loss. Here a snippet of critical code.<br>
Each backward step of the loss takes 30s against less than 2 for all above.</p>
<p>Thanks in advance for any suggestion !</p>
<pre><code class="lang-auto">import csv
import numpy as np
import logging
import time
import string
from itertools import chain

import torch
import torch.nn as nn
import torch.optim
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
from torch.utils.tensorboard import SummaryWriter
from pathlib import Path
from textloader import *
# from generate import *
import logging
logging.basicConfig(level=logging.INFO)


def maskedCrossEntropy(output, target, padcar):
    mask = target != padcar
    loss = torch.nn.CrossEntropyLoss(reduction="none")(output.permute(0,2,1), target.long()) * mask
    return loss.sum() / mask.sum()

class RNN(nn.Module):
    def __init__(self, latent, dim, out):
        super().__init__()
        self.latent = latent
        self.dim = dim
        self.out = out
        self.hidden_state = torch.tensor(latent)
        self.lin_hs = nn.Linear(latent, latent)
        self.lin_ft = nn.Linear(dim, latent)
        self.lin_dec = nn.Linear(latent, out) 

    def decode(self, hs):

        d = self.lin_dec(hs)
        return d

    def forward(self, batch, hs):
        l = []
        for i in range(batch.shape[0]):
            hs = self.one_step(batch[i,:], hs)
            l.append(hs)
        return torch.stack(l)
    
    def one_step(self, batch, hs):
        return torch.tanh(self.lin_hs(hs) + self.lin_ft(batch.clone()))

speech = ""
with open('data/full_speech.txt') as f:
    while True:
        c = f.read(1)
        speech += c
        if not c:
            break

LR = 10e-3
SEQ_LEN = 100
PRED_LEN = 10
LATENT_DIM = 50
BATCH_SIZE = 500
EPOCH_RANGE = 5


embedding = nn.Embedding(len(id2lettre),50)
speech_dataset = TextDataset(speech)
speech_dataloader = DataLoader(speech_dataset, BATCH_SIZE, shuffle=True, drop_last=True, collate_fn=collate_fn)
model = RNN(LATENT_DIM, len(id2lettre), len(id2lettre))
optim = torch.optim.Adam(model.parameters(), lr = LR)
loss = torch.nn.CrossEntropyLoss()
hs = torch.zeros(BATCH_SIZE, LATENT_DIM)

for epoch in range(EPOCH_RANGE):
    print(epoch)
    i = 0
    for x in speech_dataloader:

        optim.zero_grad()
        hst = model(embedding(x.long()), hs)
        hst = model.decode(hst)
        l = maskedCrossEntropy(hst, x, PAD_IX)
        l.backward()
        optim.step()
</code></pre>
How to define a static parameter in a model?

tueboesen: <p>I’m trying to define a static tensor in a pytorch model. Initially I just created the variable as a tensor and then saved it in my model. However that leads to problem when I use, model.to(device), since the tensor does not get shifted to cuda devices when I do this.</p>
<p>I can make the parameter shift by defining it as a nn.parameter(), but I do not want to alter or train this parameter.</p>
<p>So what exactly is the best way to tell pytorch to ensure that this static tensor gets shifted between cuda and cpu, while not messing up anything with the optimizer?</p>

googlebot: <p>module.register_buffer(“x”, tensor(…)), but for scalars a simple attribute (module.pi = 3.14) is sometimes better, both are auto-transfered to gpu.</p>
Pytorch concat Torch.Tensor of different depth dimension

Scott_Hoang: <p>Hi everyone, I need help.<br>
For a class assignment, I have to reimplement a paper called “3D Convolutional Neural Networks for Human Action Recognition”. The paper proposed this convolutional network<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/38b17be4a50663ff5659d5452a308801e7223b29.png" data-download-href="https://discuss.pytorch.org/uploads/default/38b17be4a50663ff5659d5452a308801e7223b29" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/38b17be4a50663ff5659d5452a308801e7223b29.png" alt="image" data-base62-sha1="85wZKvFyI6YbS3z4PV6GQbKHH1n" width="690" height="278" data-dominant-color="E9E9E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1399×565 23.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Currently, I’m stuck on the hardwired layers which are composed of 5 distinct channels: gray, gradient-x, gradient-y, optflow-x, optflow-y. The gray channel contains the gray pixel values of the seven input frames. The gradient-x and gradient-y are obtained by calculating on the horz and vertical directions, respectively, on each seven input frames. Finally, the optical flow x and optical flow y channels contain the optical flow fields along the horizontal and vertical directions, computed from adjacent input frames.<br>
grey, gradient-y, and gradient-x has the dimension of (7,60,40) (DxHxW) while opt-x and y have the dimension of (6,60,40). I computed gradient using np.gradient() and computed optical flow using cv2.calcOpticalFlowFarneback(). In order to push everything into CUDA, I computed this in my DataLoader; However, due to the size mismatch, I can’t concatenate them into a single cube. Do you guys have any suggestion?</p>

Kushaj: <p><code>torch.stack(a, b, dim=0)</code></p>

Arya_Singh: <p>Hi, I am doing the same implementation for a course assignment. Were you able to solve the size mismatch issue? If yes, can you please tell me as well?</p>
Work around for large broadcasting cause CUDA out of memory

bsun0802: <p>Hi, I have two tensors <strong>x</strong> with shape [B,B,C, 1]  and <strong>y</strong> with shape [1, B, C, N].</p>
<p>I want to do <code>torch.mean(x*y, dim=1)</code> which will output a tensor with shape [B,C,N].</p>
<p>But the broadcasting result <code>x*y</code> with shape [B,B,C,N] is too large (12.76GB) due to large B and C. I tried with <code>torch.einsum</code> and intend to avoid the broadcasting product, but it seems <code>torch.einsum</code> needs to generate the intermediate <code>x*y</code> implicitly and results in the same OOM error.</p>
<p>Does anybody have a suggestion or work-around?</p>
<p>Thanks a lot.</p>

googlebot: <p>perhaps batched matmul: (C, B1, B2) @ (C, B2, N) = (C,B1,N)</p>
.data in pytorch

ADONAI_TZEVAOT: <p>What does <code>.data</code> in pytorch mean?<br>
<code>z_x</code> is the output of a linear layer<br>
<code>z_x_nu = z_x.data</code></p>
<p>So what is z_x.data here? Please explain</p>

Ali_Haider: <aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/51743214/is-data-still-useful-in-pytorch" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/7850499/maybe" target="_blank" rel="noopener nofollow ugc">
    <img alt="Maybe" src="https://i.stack.imgur.com/9kovQ.jpg?s=128&amp;g=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/51743214/is-data-still-useful-in-pytorch" target="_blank" rel="noopener nofollow ugc">Is .data still useful in pytorch?</a>
</h4>

<div class="tags">
  <strong>python, version, pytorch, tensor</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/7850499/maybe" target="_blank" rel="noopener nofollow ugc">
    Maybe
  </a>
  on <a href="https://stackoverflow.com/questions/51743214/is-data-still-useful-in-pytorch" target="_blank" rel="noopener nofollow ugc">09:31AM - 08 Aug 18 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Have a look.
Slicing tensor breaks backprop

new2pytorch: <p>I have an input <code>X</code> that goes into an MLP which outputs a tensor <code>xyz</code> of shape <code>torch.Size([4, 64, 1024])</code>. I want to backprop to the input using only the first 3 columns of the last dimension. But when I slice <code>xyz</code> using <code>xyz[:, :, 0:3]</code> what happens is that autograd doesn’t backprop to the input <code>X</code> anymore. I also tried creating a linear layer and assign its weights manually from a tensor so it only selects the columns I want but Pytorch gives an error and it won’t work. Any ideas on what I should do?</p>

JuanFMontesinos: <p>Can you provide an script which shows that? It should work.</p>
Train loss curve understanding

Harshini_Sewani1: <p>Hello Everyone,<br>
I am very new to ML field. I have used PyTorch to train my model and test.</p>
<p>I have plotted the curve as shown in the fig.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fcec33bc3ce54671f03ff10b1431e0b6d76dbebb.png" data-download-href="https://discuss.pytorch.org/uploads/default/fcec33bc3ce54671f03ff10b1431e0b6d76dbebb" title="page"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fcec33bc3ce54671f03ff10b1431e0b6d76dbebb_2_690x288.png" alt="page" data-base62-sha1="A5snL17b7xOyuiIAXbDemCxHDQf" width="690" height="288" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fcec33bc3ce54671f03ff10b1431e0b6d76dbebb_2_690x288.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fcec33bc3ce54671f03ff10b1431e0b6d76dbebb_2_1035x432.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/c/fcec33bc3ce54671f03ff10b1431e0b6d76dbebb.png 2x" data-dominant-color="F3F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">page</span><span class="informations">1358×567 61 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can someone help me to give basic interpretation to the curve?</p>
<p>Thank You So Much.</p>

Harshini_Sewani1: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Please help me out . I will really appreciate your help</p>

Ali_Haider: <p>The model curve tell us two things<br>
Accuracy - Correct predictions<br>
Loss        -  We reduce a loss using some kind of optimization.</p>
<p>Training accuracy is the one telling about correct samples predicted from training set<br>
Validation acc tells about the predictions on validations set.</p>
<p>Seeing your model I assume it to be trained well as it’s validation and training accuracies are comparable as well loss curves.</p>

Harshini_Sewani1: <p><a class="mention" href="/u/ali_haider">@Ali_Haider</a> Thank you so much for your explanation <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> Really appreciate it.</p>

Ali_Haider: <p>Still you can further train it as your model might has some room for improvement. Models representational capacity is limited by optimization so, It depends if your model have achieved some local minima or like good minima. To see it better is to further train but save your current model.</p>
Question about the use of zero_grad() in DDPG Implementation

trcn: <p>Hi,</p>
<p>I am looking at the DDPG implementation in <a href="https://github.com/sfujim/TD3/blob/master/DDPG.py" rel="noopener nofollow ugc">this repo.</a> And the updates are done by</p>
<pre><code class="lang-auto">        # Compute critic loss
		critic_loss = F.mse_loss(current_Q, target_Q)

		# Optimize the critic
		self.critic_optimizer.zero_grad()
		critic_loss.backward()
		self.critic_optimizer.step()

		# Compute actor loss
		actor_loss = -self.critic(state, self.actor(state)).mean()
		
		# Optimize the actor 
		self.actor_optimizer.zero_grad()
		actor_loss.backward()
		self.actor_optimizer.step()
</code></pre>
<p>My question is: It doesn’t zero the gradients of critic before <code>actor_loss.backward()</code>. This would mean gradients are accumulated for variables of critic, right? So, they are the sum of gradients for <code>critic_loss</code> and <code>actor_loss</code> .</p>
<p>If yes, why the gradients for variables of actor is not affected by this when “applying chain rule”?</p>
<p>My thinking was, if we had just one variable for each network:A: Actor and C: Critic,</p>
<p>After <code>critic_loss.backward()</code>:</p>
<ul>
<li>
<code>C.grad</code> is populated with <code>d(critic_loss)/dC</code>
</li>
<li>
<code>A.grad</code> is 0, as it is not a part of the <code>critic_loss</code> computation.</li>
</ul>
<p>And after <code>actor_loss.backward()</code>:</p>
<ul>
<li>
<code>C.grad</code> is now   <code>d(critic_loss)/dC + d(actor_loss)/dC</code>
</li>
<li>
<code>A.grad</code> is <code>C.grad x dC/dA = (d(critic_loss)/dC + d(actor_loss)/dC) x  dC/dA</code>
</li>
</ul>
<p>But apparently, it is not true. I guess it is because my understanding of autograd is wrong.<br>
Could you explain what I am missing here? Is it just that <code>A.grad</code> wouldn’t look at <code>C.grad</code>  but <code>d(actor_loss)/dC</code>?</p>
<p>Thanks in advance. Sorry for the long question.</p>
Distributed Data Parallel repeated pickling

Rizhiy: <p>I’m trying to change from plain DP to DDP. My code works, but is very slow.<br>
I narrowed down the problem to a weird behaviour: the dataset, which I pass as an argument in <code>mp.spawn</code>, is being pickled and unpickled every time I create new iterator from loader.</p>
<p>Why is this the case? Surely all arguments should be pickled and unpickled once when processes are created.</p>
<p>Rough code:</p>
<pre><code class="lang-python">def train(rank, dataset):
    setup_ddp(rank)
    model = create_model()
    loader = create_loader(dataset)

    train_iter = iter(loader)

    for i in range(NUM_ITERS):
        try:
            batch, targets = next(train_iter)
        except StopIteration:
            train_iter = iter(loader)
            # The dataset is being pickled and unpickled every time next is called on new iterator
            batch, targets = next(train_iter)

        step(model, batch, targets)

    cleanup_ddp(rank)


dataset = create_dataset()
mp.spawn(train, args=(dataset,), nprocs=WORLD_SIZE)
</code></pre>

Rizhiy: <p>I figured out that the problem was in loader instead of DDP. I had four workers for each loader, which were created repeatedly each time a new <code>iter</code> was created, with 8 processes it meant that 32 workers were created each time.<br>
Reducing the number of workers solved the problem.</p>
Loss Function Weight and Single Output

localh: <p>Let’s say we have a logistic regression model that takes <code>num_features</code> and outputs <code>num_classes</code>. Traditionally, <code>num_classes</code> is equal to one.</p>
<p>Now if I want to weight my loss function, I would make something like this:<br>
<code>F.binary_cross_entropy(probas, target, weight=torch.tensor([2]))</code></p>
<p>Where my weights were calculated by:</p>
<pre><code class="lang-auto"># n samples / n_classes * bincount
((183473+47987) / (2 * np.array([183473, 47987])))
[0.63074419 2.41213088] # [0, 1]
</code></pre>
<p>My question is, the weight in the loss function, as I can only give it one value, which class is it for, 0 or 1?</p>
<p><code>target</code>, in our data set, contains either zeros or ones, like a traditional binary regression problem.</p>

ptrblck: <p><code>nn.BCEWithLogitsLoss</code> allows you to pass a <code>weight</code> as well as a <code>pos_weight</code> argument.<br>
The former tensor should have the shape <code>[batch_size]</code> and will be applied to each sample, while the latter should have the shape <code>[nb_classes]</code> and will be applied to the positive examples as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html?highlight=bcewithlogitsloss#torch.nn.BCEWithLogitsLoss">docs</a>.<br>
Based on your code snippet I assume you would like to use <code>pos_weight</code>.</p>

localh: <p>Thanks so much. It’s silly that I overlooked this.</p>
Implementing weighted least square with torch.nn

vrvrv: <p>Hi,</p>
<p>This is my first uploading in this forum. I’m now trying to implement simple linear regression task, but it is slightly different from common ols problem.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/867e30e33911259eca684ec5648faac3aae55c9d.png" data-download-href="https://discuss.pytorch.org/uploads/default/867e30e33911259eca684ec5648faac3aae55c9d" title="wls"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/867e30e33911259eca684ec5648faac3aae55c9d_2_517x147.png" alt="wls" data-base62-sha1="jbMlk6CGjkRTIZVg9gDFNMXHd5r" width="517" height="147" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/867e30e33911259eca684ec5648faac3aae55c9d_2_517x147.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/867e30e33911259eca684ec5648faac3aae55c9d_2_775x220.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/867e30e33911259eca684ec5648faac3aae55c9d_2_1034x294.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">wls</span><span class="informations">1420×404 28.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I need a help to implement first term efficiently. Is there any module that help computing weighted least square?<br>
My trial is</p>
<pre><code class="lang-auto">output = W.matmul(X) # k * D matrix
l = torch.diagonal((output-y).T.matmul(Sigma).matmul(output-y)).sum()
</code></pre>
<p>Thank you in advance,</p>
Incorporate additional ground truth information to segmentation net

stadtgiraffe: <p>Hello,</p>
<p>I am looking for ways to incorporate anatomy information in CT Data (at what body height a slice is positioned) to my multi organ segmentation net.<br>
I have checked the ways described here <a href="http://motion.pratt.duke.edu/sidechannel/pdf/icra2017incorporating.pdf" rel="noopener nofollow ugc">Incorporating Side-Channel Information into Convolutional Neural Networks for Robotic Tasks</a> but I am wondering if there is a best practice to do this or if there are other ways. I do not have a fully connected layer (its a UNet like structure), so simply adding channels to the FCN input is not possible.</p>
<p>Thanks for the help!</p>

JuanFMontesinos: <p>One simple way is using FiLM (Feature-wise layer modulation)<br>
</p><aside class="onebox pdf">
  <header class="source">
      <a href="https://arxiv.org/pdf/1709.07871.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>
  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1709.07871.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://arxiv.org/pdf/1709.07871.pdf" target="_blank" rel="noopener nofollow ugc">1709.07871.pdf</a></h3>

<p class="filesize">6.12 MB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Sooo if you are using a U-Net this has already been done for audio<br>
Here you have code.<a href="https://github.com/gabolsgabs/cunet" rel="noopener nofollow ugc">https://github.com/gabolsgabs/cunet</a><br>
This way you can simply condition inner layers of U-Net with a one-hot vector (or any tensor if you have some spatial info)
<p>Anyway you have the new visual transformers, they just encode the position of the image patches before feeding them into the nn.<br>
You can try and approach like this too.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://whoisnnamdi.com/favicon.ico" class="site-icon" width="48" height="48">
      <a href="https://whoisnnamdi.com/transformers-image-recognition/" target="_blank" rel="noopener nofollow ugc" title="11:17PM - 04 October 2020">Who is Nnamdi? – 4 Oct 20</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:644/396;"><img src="https://whoisnnamdi.com/content/images/2020/10/M_MKtM3ruT.png" class="thumbnail" width="644" height="396"></div>

<h3><a href="https://whoisnnamdi.com/transformers-image-recognition/" target="_blank" rel="noopener nofollow ugc">(Paper Explained) An Image is Worth 16x16 Words: Transformers for Image...</a></h3>

<p>Large Transformer trained on large datasets outperform CNN-based architectures and achieve state of the art results on image recognition tasks</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
3D CT images - Resizing and Resampling

moreshud: <p>I loaded 3D CT images as .npy files to 2D UNet with a spatial dimension 512, 512. How can I ensure the information is preserved when resizing to 256, 256 - maybe the choice of interpolation and others when saving as .npy files</p>

moreshud: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/vfdev-5">@vfdev-5</a> Your contributions will be appreciated</p>

vfdev-5: <p><a class="mention" href="/u/moreshud">@moreshud</a> could you please detail what you mean by</p>
<blockquote>
<p>How can I ensure the information is preserved when resizing to 256, 256</p>
</blockquote>

moreshud: <p>I would reframe the statement. Does resizing changes the pixel value for a label with <code>0, 1</code> or an image in general? If yes, which of the interpolation can be used to keep the pixel value the same?</p>

vfdev-5: <p>Yes, resizing in general changes the pixel values. For resizing targets (e.g. segmentation masks) we usually apply “nearest” interpolation mode. In this mode, resizing is approx equivalent to removing columns and lines =&gt; that new pixels are exactly the same to the source. For other modes, new pixels are interpolated =&gt; new pixel will have float values between 0 and 1. For resizing images we can apply bilinear or cubic interpolation mode.</p>

moreshud: <p>Thanks.</p>
<p>The resizing can be done with PyTorch or cv2. How can I perform resampling on slices from 3D CT images?</p>

vfdev-5: <p>Yes, both can work. As for torchvision, it would be better to produce a 3D tensor from 2D using unsqueeze before calling <code>transforms.Resize((new_h, new_w))</code>. For example,</p>
<pre><code class="lang-python">image_slice_2d = torch.rand(32, 32)
# shape is (32, 32), dtype = float32
image_slice_3d = image_slice_2d.unsqueeze(dim=0)
# shape is (1, 32, 32), dtype = float32

t = transforms.Resize((20, 20), interpolation=2)  # interpolation is bilinear here
res = t(image_slice_3d)
# res.shape = (1, 20, 20) and dtype = float32
res.shape, res.dtype
</code></pre>

moreshud: <p>Does torchvision have the resampling inbuilt function?</p>

vfdev-5: <p>Internally it uses either <code>PIL.Image.transform</code> if input is PIL.Image or <code>torch.nn.functional.interpolate</code> for tensor inputs.<br>
Please, see here for more info: <a href="https://pytorch.org/docs/stable/torchvision/transforms.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/torchvision/transforms.html</a></p>

moreshud: <p>Thanks a lot for your help!</p>
Purpose of image augmentation

cosmosaa: <p>I can think of 2 possible reasons for doing image augmentation. I’m not sure if either one is more correct or both are correct.</p>
<ol>
<li>Produce variations on the training dataset to produce “realistic” images that artificially grow the dataset size.</li>
<li>Make it more difficult for the network to “memorize” the training data, by creating random “noise”. Some examples of transformations including blurring and changes in perspective would create images that would not be representative of “in the wild” images, but might cause the network to “have to think more” and not “memorize” the training set.</li>
</ol>
<p>I’m wondering which of the 2 reasons or both reasons are generally understood to be the reason for using image augmentation. I ask because some transformations are interesting, such as blurring, changes in brightness and so on. But in my use case blurred images for example would not be found “in the wild”, and so I’m wondering if it’s appropriate to use it.</p>

ptrblck: <p>I always assumed the data augmentation should fill the “real data domain” with randomly transformed samples, i.e. the transformations should stick close to what you would expect to see in the wild (but I might not be up-to-date on this topic).</p>
Custom Loss Function in a Differentiable Manner

bedrick: <p>Hello,</p>
<p>I try to create a custom loss function in a differentiable manner. For example I have two tensors:</p>
<p>a = [2, 8, 9, 10, 2] and b = [1, 2, 6, 7, 8] and my loss function should penalize a - b when a - b is positive.</p>
<p>if I write loss function as follows, I think it is not differentiable, because of “&lt;” operation.</p>
<pre><code>   diff = a - b
   diff[ diff &lt; 0 ] = 0
   return diff
</code></pre>
<p>How can I create this loss function in a differentiable manner?</p>
<p>Thanks in advance.</p>

ptrblck: <p>It should work and pass a zero gradient to the masked elements:</p>
<pre><code class="lang-python">a = torch.randn(2, 2, requires_grad=True)
b = torch.randn(2, 2)

diff = a - b
diff[ diff &lt; 0 ] = 0
print(diff)
&gt; tensor([[0.6480, 0.0000],
          [0.0000, 0.0272]], grad_fn=&lt;IndexPutBackward&gt;)

diff.mean().backward()
print(a.grad)
&gt; tensor([[0.2500, 0.0000],
          [0.0000, 0.2500]])
</code></pre>
How to rerun model with same results?

THEblob: <p>Hello,<br>
I initialize my model and save the state-dict of it together with the optimizer.<br>
I load this model for every rerun and it gives me similar results but not equal ones.<br>
Is it even possible to rerun a model with the exact same results as before?</p>

ptrblck: <p>It should be possible as long as you are not using non-deterministic operations as explained <a href="https://pytorch.org/docs/stable/notes/randomness.html">here</a>.</p>
Simple autograd fail

kingstar101: <p>Hi guys, very new to PyTorch but have been using successfully on a few small projects until I hit a problem which is summarised in the MWE below. Should this not be returning 6? I am missing something here but am unsure why, and I am certain it is simple.</p>
<pre><code class="lang-auto">import torch
p = torch.tensor([1.], requires_grad=True)
class Square(torch.autograd.Function):
    @staticmethod
    def forward(ctx, target):
        ir = torch.pow(target, 2.)
        ctx.save_for_backward(target)
        return ir

    @staticmethod
    def backward(ctx, u):
        target, = ctx.saved_tensors
        return target * 2.

sq = Square.apply(p)
six = sq.pow(3.)
six.backward()
p.grad
</code></pre>
<p>This returns tensor([2.]), which is clearly the derivative from square, so why is it not also applying the cube part?</p>
<p>Thank you!!</p>

Nikronic: <p>Hi,<br>
Because you are not using any gradient value from any operation after you custom autograd function.</p>
<aside class="quote no-group" data-username="kingstar101" data-post="1" data-topic="104088">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ebca7d/48.png" class="avatar"> kingstar101:</div>
<blockquote>
<p><code>return target * 2.</code></p>
</blockquote>
</aside>
<p>If you check your <code>backward</code> method, you have a argument <code>u</code> which contains all grads from operations after <code>Square</code>. Because you are not using it, no matter what you do, it just breaks the all backwards edges right there.</p>
<p>What you are computing:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26bca3357d9a49c3879877de14c53221530a1f73.png" alt="image" data-base62-sha1="5wGitftNP8ckbXimJqGlkDcbih5" width="110" height="102"></p>
<p>What you should compute:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8126b4365ebce07a32e57fd351318ce3b8ccd29.png" alt="image" data-base62-sha1="zoxSgetatzN7O76ypv3WrMB4agN" width="306" height="196"></p>
<p>So overally:</p>
<p>To fix this, just incorporate <code>u</code> into your computation of backward:</p>
<pre><code class="lang-auto">return u * (target * 2.)
</code></pre>
<p>Bests</p>

kingstar101: <p>Of course! Thank you, and  nice explanation</p>
Accessing weights scripted module

Joe_Harrison: <p>I’m trying to access the weights of a scripted module. I load the module as follows:</p>
<p><code>torch::jit::script::Module model; model = torch::jit::load(argv[1]);</code><br>
Where the first argument is a .pt file.</p>
<p>I can then print the parameters by iterating over them:<br>
<code>for (const auto&amp; params : model.parameters()) { std::cout &lt;&lt; params &lt;&lt; std::endl; }</code></p>
<p>But how can I then alter this params tensor? Say I want to add ones to each value in the tensor. I can’t alter the params variable as it is a constant. But if I remove const I get an error that this construction of iterating doesn’t work without consts.</p>

ptrblck: <p>You could use the <code>.copy_</code> operation as in this example:</p>
<pre><code class="lang-auto">torch::NoGradGuard guard;
for (auto&amp; p : model-&gt;parameters()) {
  auto sz = p.view(-1).size(0);
  p.copy_(torch::cos(torch::arange(0, sz, tensor_options).view(p.sizes())));
}
</code></pre>
Hybrid 2d Conv+ConvTranspose layer

jasha: <p>Let’s say I have a 4-dimensional tensor (batch x channel x time x space). I’d like to downsample the space axis and upsample the time axis. I can achieve this sequentially as follows:</p>
<pre><code class="lang-auto">tnsr = torch.randn(2, 3, 4, 5, requires_grad=True)  # batch x channel x time x space
space_downsampler = nn.Conv2d(3, 3, (1, 3), stride=(1, 2), padding=(0, 1))
tnsr = space_downsampler(tnsr)  # downsample space dimension
assert tnsr.shape == (2, 3, 4, 3)
time_upsampler = nn.ConvTranspose2d(3, 3, (3, 1), stride=(2, 1), padding=(1, 0))
tnsr = time_upsampler(tnsr)  # upsample time dimension
assert tnsr.shape == (2, 3, 7, 3)
</code></pre>
<p>This solution uses a 1x3 filter with stride (1,2) followed by a 3x1 filter with fractional stride (0.5, 1). But I would prefer to use a single 3x3 filter that has stride (0.5, 2). This would mean that the 3x3 filter is applied with fractional striding along one axis and with normal 2x striding along the other axis.</p>
<p>How might I implement this? One idea that comes to mind would be to preprocess the tensor by “injecting zeros” into the time axis, so that a poor-man’s fractional stride can be achieved using a normal convolution module:</p>
<pre><code class="lang-auto">tnsr = torch.randn(2, 3, 4, 5, requires_grad=True)  # batch x channel x time x space
tnsr_expanded = torch.zeros(2, 3, 7, 5)
tnsr_expanded[:, :, ::2] = tnsr
hybrid_conv = nn.Conv2d(3, 3, (3, 3), stride=(1, 2), padding=(1, 1))
tnsr = hybrid_conv(tnsr_expanded)
assert tnsr.shape == (2, 3, 7, 3)
</code></pre>
<p>But there is a problem with the above implementation: one cannot backpropagate gradients smoothly through the <code>tnsr_expanded[:, :, ::2] = tnsr</code> assignment (as far as I know). Here is my clumsy alternative, which does support backpropagation:</p>
<pre><code class="lang-auto">tnsr = torch.randn(2, 3, 4, 5, requires_grad=True)  # batch x channel x time x space
zeros = torch.zeros(2,3,5)
tnsr_expanded = torch.stack((tnsr[:,:,0],zs,tnsr[:,:,1],zs,tnsr[:,:,2],zs,tnsr[:,:,3]),dim=2)
hybrid_conv = nn.Conv2d(3, 3, (3, 3), stride=(1, 2), padding=(1, 1))
tnsr = hybrid_conv(tnsr_expanded)
assert tnsr.shape == (2, 3, 7, 3)
</code></pre>
<p>Any ideas for a more elegant solution?</p>
<p>EDIT:<br>
Nevermind, it appears that I was mistaken about backpropagating gradients through the <code>tnsr_expanded[:, :, ::2] = tnsr</code> assignment: backpropagation works as expected.</p>
Separate LSTM networks with different sequence length

andy29: <p>Hi.<br>
I am quite new to this, and still trying to figure things out. Any help is appreciated.<br>
I am trying to create two separate lstm networks, which would take in different sequence lengths. So, for example lstm1 would take in sequence length of 6 and lstm2 would take sequence length of 12. Afterwards, the outputs of both networks should be put together to produce an output of sequence length 1.</p>
<p>Can this be done?<br>
I have read about pack_padded sequence… But not sure if it applies to hear.</p>
Dilated Convolution

sharjeel: <p>I’ve been studying about dilated convolution and that it is used to increase the receptive field of the layer.<br>
My question is that why do we need dilated convolutions to increase the receptive field when a bigger kernel can also increase the receptive filed of the layers?</p>

phan_phan: <p>Dilation doesn’t increase the number of parameters of a convolution.<br>
But the number of parameter increases with the square of the size of the kernel.</p>
How can I do a mutual concatenation

Lingfei: <p>Hi guys,</p>
<p>I’ve confronted with a problem that I cannot solve. Say I have a tensor A with shape (batch_A, hidden_dim) and tensor B with shape (batch_B, hidden_dim). Now I would like to do a reshape for both tensor into shape (batch_A, h, hidden_dim/h), (batch_B, h, hidden_dim/h),the mutual concatenation into (batch_A x batch_B, 2h, hidden_dim/h) and then perform Conv on it. What would be the best way to do such operation. As far as I know, I could use this:</p>
<pre><code class="lang-auto">A = A.reshape(-1, h, hidden_dim/h)
B = B.reshape(-1, h, hidden_dim/h)
A = A.unsqueeze(0).repeat(batch_B,1, 1, 1)
B = B.unsqueeze(1).repeat(1, batch_A, 1, 1)
concat_tensor = torch.cat([A, B], dim=-2)
res = conv(concat_tensor)
</code></pre>
<p>Is there any better way to do this? As <code>repeat()</code> will consume a lot of memory. This would not ideal for me because one of my batch size would be as large as 15,000.</p>
<p>Regards,</p>
Mypy does not recognize torch.cuda.amp

Rafi: <p>I could not post all the code but basically the <code>mypy</code> does not pass because it does not recognize the amp.autocast() function and throw an error</p>
<p><strong>error: Module has no attribute "autocast"</strong></p>
<pre><code class="lang-auto">from torch.cuda import amp 
 with amp.autocast():
     # some code
</code></pre>
<p>here is my code to run <code>mypy</code> :</p>
<p><code>python -m mypy --ignore-missing-imports . || exit 1</code></p>
<p>Is it possible to deactivate this error</p>
Why is bias=False in Conv for Quantization-aware training?

icyhearts: <p>in this tutorial: <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#quantization-aware-training" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#quantization-aware-training</a><br>
bias=False is used in Conv.<br>
I am wondering why we disable <code>bias</code> in Conv Quantization-aware training?<br>
Many thanks.</p>
Recommended Loss function for Imbalanced multiclass Semantic Segmentation problem

sleepyo: <p>Hi Guys,<br>
Do you have any recommendations for loss function in the Imbalanced multiclass Semantic Segmentation problem? The weight difference in each class is pretty huge actually, like [0.1, 0.1, 0.4, 0.4]<br>
currently, I’m using LovaszBCELoss, it’s good for accuracy, but the f1-score is pretty bad, 0.9 acc but 0.7 F1 (weighted F1).</p>
DenseNet, Sizes of tensors must match

Celmar: <p>Hello,<br>
would you know how I can adapt this code so that sizes of tensors must match because I have this error: <code>x = torch.cat([x1,x2],1) RuntimeError: Sizes of tensors must match except in dimension 0. Got 32 and 1 (The offending index is 0)</code>.</p>
<p>My images are size 416x416.</p>
<p>Thank you in advance for your help,</p>
<pre><code class="lang-auto">num_classes = 20
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
                
        self.inc = models.inception_v3(pretrained=True)
        self.inc.aux_logits = False

        for child in list(self.inc.children())[:-5]:
            for param in child.parameters():
                param.requires_grad = False

        self.inc.fc = nn.Sequential()
                    
        self.dens121 = models.densenet121(pretrained=True)

        for child in list(self.dens121.children())[:-6]:
            for param in child.parameters():
                param.requires_grad = False

        self.dens121 = nn.Sequential(*list(self.dens121.children())[:-1])
           
        self.SiLU = nn.SiLU()      
        self.linear = nn.Linear(4096, num_classes)
        self.dropout = nn.Dropout(0.2)
        
    def forward(self, x):
        x1 = self.SiLU(self.dens121(x))
        x1 = x1.view(-1, 2048)
        
        x2 = self.inc(x).view(-1, 2048)
        x = torch.cat([x1,x2],1)

        return self.linear(self.dropout(x))
</code></pre>

ptrblck: <p>I cannot reproduce this issue by using an input tensor of <code>[batch_size, 3, 416, 416]</code> and am running into a shape mismatch error before.<br>
Changing <code>x.view(-1, 2048)</code> to <code>x.view(x.size(0), -1)</code> and adapting the expected <code>in_features</code> in <code>self.linear</code> works:</p>
<pre><code class="lang-python">num_classes = 20
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
                
        self.inc = models.inception_v3(pretrained=True)
        self.inc.aux_logits = False

        for child in list(self.inc.children())[:-5]:
            for param in child.parameters():
                param.requires_grad = False

        self.inc.fc = nn.Sequential()
                    
        self.dens121 = models.densenet121(pretrained=True)

        for child in list(self.dens121.children())[:-6]:
            for param in child.parameters():
                param.requires_grad = False

        self.dens121 = nn.Sequential(*list(self.dens121.children())[:-1])
           
        self.SiLU = nn.SiLU()      
        self.linear = nn.Linear(175104, num_classes)
        self.dropout = nn.Dropout(0.2)
        
    def forward(self, x):
        x1 = self.SiLU(self.dens121(x))
        x1 = x1.view(x.size(0), -1)
        
        x2 = self.inc(x).view(x.size(0), -1)
        x = torch.cat([x1,x2],1)

        return self.linear(self.dropout(x))

model = Net()
x = torch.randn(2, 3, 416, 416)
out = model(x)
</code></pre>
When should a tensors gradients be detached?

Yemi: <p>I understand how detaching gradients from tensors work. However, I do not know when to do this. When looking at a research paper, how do you know which variables to let gradients flow through? an example from an RL paper(say DDPG <a href="https://arxiv.org/pdf/1509.02971.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1509.02971.pdf</a> ) might be helpful.</p>
Torch.tanh vs torch.nn.functional.tanh

vadimkantorov: <p>A little confusing that both exist. Is any of these deprecated?</p>

smth: <p>we should now deprecate torch.nn.functional.tanh on master, as tensors and variables are now merged.</p>

jpeg729: <p>If I use nn.Tanh I have to declare it in my_model.<strong>init</strong>(), e.g.</p>
<pre><code class="lang-auto">self.tanh = nn.Tanh()
</code></pre>
<p>Whereas I can use nn.functional.tanh directly in my_model.forward(), e.g.</p>
<pre><code class="lang-auto">output = nn.functional.tanh(input)
</code></pre>
<p>If you deprecate nn.functional.tanh I could do</p>
<pre><code class="lang-auto">output = nn.Tanh()(input)
</code></pre>
<p>where ever I need the functional form, but it would be slower because of the class instantiation.</p>
<p>What has this to do with Variables and Tensors? I am confused.</p>

alexis-jacq: <aside class="quote no-group" data-username="jpeg729" data-post="4" data-topic="15897">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpeg729/48/3464_2.png" class="avatar"> jpeg729:</div>
<blockquote>
<p>If you deprecate nn.functional.tanh I could do</p>
<p>output = nn.Tanh()(input)</p>
</blockquote>
</aside>
<p>Or you could still use torch.tanh():</p>
<pre><code class="lang-auto">output = input.tanh()
</code></pre>

banikr: <p>I am expecting output having both positive and negative values. So,</p>
<p><code>output = torch.tanh(model(input))</code><br>
in the final output of network should be fine?</p>
<p>Or are any other variants available?</p>

mathematics: <p>There are only few activations like torch.tanh in torch.<br>
But for torch.nn.functional , i found alot of functional activations.<br>
Why is that?</p>

yrluo: <p>Same confusion here <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>
[C++] [TorchScript] torchvision::nms not in TorchScript

zetyquickly: <p>Hello, this topic is relevant to (<a href="https://discuss.pytorch.org/t/torchvision-android-torchvision-ops-when-will-they-be-available-on-mobile/61360" class="inline-onebox">[Torchvision] [Android] Torchvision.ops when will they be available on Mobile</a>)</p>
<p>I’ve tried to load a model exported using torchscript with <code>C++ API</code>  (<a href="https://download.pytorch.org/libtorch/nightly/cpu/libtorch-cxx11-abi-shared-with-deps-latest.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/nightly/cpu/libtorch-cxx11-abi-shared-with-deps-latest.zip</a>) and it produces errors the same as <code>JNI</code> does</p>
<pre><code class="lang-auto">root@Callisto:~/libtorch_examples/build# ./example-app /root/libtorch_examples/scripted_rpn.pt 
terminate called after throwing an instance of 'torch::jit::script::ErrorReport'
  what():  
Unknown builtin op: torchvision::nms.
Could not find any similar ops to torchvision::nms. This op may not exist or may not be currently supported in TorchScript.
:
Serialized at code/__torch__/torchvision/ops/boxes.py:25:7
    scores: Tensor,
    iou_threshold: Tensor) -&gt; Tensor:
  _7 = ops.torchvision.nms(boxes, scores, annotate(float, iou_threshold))
       ~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
</code></pre>
<p>How to make this operation available from <code>C++</code>?</p>

hookzeng: <p>Hi, I meet same problem as you. Do you find the way to solve it out? Thx~</p>

zetyquickly: <p>I’ve implemented slow python analogue of C++ operation and it’s worked well</p>
Compute hessian vector product

Hoang_Tran: <p>Hi, I’m trying to write an optimizer using pytorch that uses the information from Hessian Vector Product. Usually, I can easily compute Hessian Vector product in the training loop by calling torch.autograd.grad and set the attribute create_graph to True. However, since I want to compute it inside an optimizer, I have no access to loss and can’t change create_graph to True. Therefore, I’ve been getting this error: “element 0 of tensors does not require grad and does not have a grad_fn”. May I ask if there’s any way that I can get around this? Thank you!</p>

Yaroslav_Bulatov: <p>You can pass loss into your optimizer by using a closure. Couple of examples: look at the “step” function:</p>
<ul>
<li><a href="https://github.com/fmeirinhos/pytorch-hessianfree/blob/master/hessianfree.py" rel="noopener nofollow ugc">https://github.com/fmeirinhos/pytorch-hessianfree/blob/master/hessianfree.py</a></li>
<li><a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/lbfgs.py" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/optim/lbfgs.py</a></li>
</ul>
GAT implementation in pytorch geometric

Bixqu: <p>I am trying to understand the code of the <a href="https://pytorch-geometric.readthedocs.io/en/latest/_modules/torch_geometric/nn/conv/gat_conv.html#GATConv" rel="noopener nofollow ugc">Graph Attention Network</a> implementation, but I am stuck at the following chunk of code:</p>
<pre><code class="lang-auto">if isinstance(in_channels, int):
            self.lin_l = Linear(in_channels, heads * out_channels, bias=False)
            self.lin_r = self.lin_l
        else:
            self.lin_l = Linear(in_channels[0], heads * out_channels, False)
            self.lin_r = Linear(in_channels[1], heads * out_channels, False)
</code></pre>
<p>and from the documentation we understand that:</p>
<blockquote>
<p>in_channels (int or tuple): Size of each input sample. A tuple<br>
corresponds to the sizes of source and target dimensionalities.</p>
</blockquote>
<p>But the attention coefficient is between two graph nodes that have equal feature dimensionality, so what exactly is the _l and _r needed for? Why can’t you compute just one (as in the first part of the if ) ?</p>
Pytorch Geometric's softmax function seems to break the GAT layer

Bixqu: <p>I have copied the <a href="https://pytorch-geometric.readthedocs.io/en/latest/_modules/torch_geometric/nn/conv/gat_conv.html" rel="noopener nofollow ugc">GAT</a> layer to a separate file. I want to first make it work, then experiment with it a bit. I am, however, having a problem with making it run. The problem seems to come from this line:</p>
<pre><code>    alpha = softmax(alpha, index, ptr, size_i)
</code></pre>
<p>in the <code>message</code> function. It seems to be required for the <a href="https://github.com/rusty1s/pytorch_geometric/blob/master/torch_geometric/utils/softmax.py" rel="noopener nofollow ugc">Softmax function</a> as:</p>
<blockquote>
<p>ptr (LongTensor, optional): If given, computes the softmax based on<br>
sorted inputs in CSR representation. (default: :obj:<code>None</code>)</p>
</blockquote>
<p>However, if I include a copy of the GAT layer (local copy on my drive) into a model and try to run it for node classification, I get:</p>
<pre><code class="lang-auto">~/my_test/custom_layers/layers.py in forward(self, x, edge_index, size, return_attention_weights)
    135 
    136         # propagate_type: (x: OptPairTensor, alpha: OptPairTensor)
--&gt; 137         out = self.propagate(edge_index, x=(x_l, x_r),
    138                              alpha=(alpha_l, alpha_r), size=size)
    139 

~/anaconda3/envs/py38/lib/python3.8/site-packages/torch_geometric/nn/conv/message_passing.py in propagate(self, edge_index, size, **kwargs)
    255         # Otherwise, run both functions in separation.
    256         if mp_type == 'edge_index' or self.__fuse__ is False:
--&gt; 257             msg_kwargs = self.__distribute__(self.__msg_params__, kwargs)
    258             out = self.message(**msg_kwargs)
    259 

~/anaconda3/envs/py38/lib/python3.8/site-packages/torch_geometric/nn/conv/message_passing.py in __distribute__(self, params, kwargs)
    178             if data is inspect.Parameter.empty:
    179                 if param.default is inspect.Parameter.empty:
--&gt; 180                     raise TypeError(f'Required parameter {key} is empty.')
    181                 data = param.default
    182             out[key] = data

TypeError: Required parameter ptr_i is empty.
</code></pre>
<p>Don’t understand exactly what’s going on.</p>
<p><strong>EDIT</strong>: It works if I delete the parameter from the <code>message</code> function, and use just:</p>
<pre><code class="lang-auto">    def message(self, x_j: Tensor, alpha_j: Tensor, alpha_i: OptTensor,
                index: Tensor, size_i: Optional[int]) -&gt; Tensor:
        
        alpha = alpha_j if alpha_i is None else alpha_j + alpha_i
        alpha = F.leaky_relu(alpha, self.negative_slope)
        alpha = softmax(alpha, index, size_i)
        self._alpha = alpha
        alpha = F.dropout(alpha, p=self.dropout, training=self.training)
        return x_j * alpha.unsqueeze(-1)
</code></pre>
<p>However I am not sure if this is what I should do, since I am deleting something I don’t understand.</p>