akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Calling backward twice error

Jake_Sokol: <p>I am trying to train my model with the following code, and I am getting a runtime error: “RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.”</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">	criterion = nn.MSELoss(reduction='mean')
	optimizer = torch.optim.SGD(model.parameters(), lr=.1, momentum=.9, weight_decay=1e-4)
	for i in range(100):
		for space in train_spaces:

			optimizer.zero_grad()

			input_ = model.space_to_input(space)
			input_2 = input_.repeat(len(input_), 1)
			
			predictions = model.forward(input_)
			predictions = predictions.masked_fill(input_2 == -1, -1)

			loss = criterion(predictions, input_2)			
				
			loss.backward()
		
			optimizer.step()
</code></pre>

ptrblck: <p>Could you post an executable code snippet using random input shapes, so that we could reproduce and debug this issue, please?</p>
Moving tensors to GPU is super slow

pavel1860: <p>hi, I’m pretty new to pytorch and I am trying to fine tune a BERT model for my purposes.<br>
the problem is that the <code>.to(device)</code> function is super slow. moving the transformer to the gpu takes 20 minutes.</p>
<p>I found some test code on pytorch github repo</p>
<pre><code class="lang-auto">import torch

import torch.nn as nn

import timeit

print("Beginning..")

t0 = timeit.default_timer()

if torch.cuda.is_available():

    torch.cuda.manual_seed(2809)

    torch.backends.cudnn.deterministic = True

    device = torch.device('cuda:0')

    ngpus = torch.cuda.device_count()

    print("Using {} GPU(s)...".format(ngpus))

print("Setup takes {:.2f}".format(timeit.default_timer()-t0))

t1 = timeit.default_timer()

model = nn.Sequential(

    nn.Conv2d(3, 6, 3, 1, 1),

    nn.ReLU(),

    nn.Conv2d(6, 1, 3, 1, 1)

)

print("Model init takes {:.2f}".format(timeit.default_timer()-t1))

if torch.cuda.is_available():

    t2 = timeit.default_timer()

    model = model.to(device)

print("Model to device takes {:.2f}".format(timeit.default_timer()-t2))

t3 = timeit.default_timer()

torch.cuda.synchronize()

print("Cuda Synch takes {:.2f}".format(timeit.default_timer()-t3))

print('done')
</code></pre>
<p>the output is:</p>
<pre><code class="lang-auto">
import torch...
Beginning..
Using 1 GPU(s)...
Setup takes 0.00
Model init takes 0.00
Model to device takes 952.94
Cuda Synch takes 0.00
done
</code></pre>
<p>this is my environment:</p>
<pre><code class="lang-auto">Pytorch version is: 1.7.0
Cuda version is: 10.1
cuDNN version is : 7604
Arch version is : sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37
</code></pre>
<p>system information:<br>
os: Windows 10<br>
graphics card: NVIDIA GeForce RTX 3090<br>
processor:	AMD Ryzen 9 5900X 12-Core Processor, 3693 Mhz<br>
motherboard:  ROG STRIX B550-F GAMING (WI-FI)<br>
memory: 16GB</p>

ptrblck: <p>You are most likely running into the JIT kernel compilation, since you are not using sm_80 or sm_86 in your binaries due to CUDA10.1.<br>
Use the CUDA11.0 binaries and the startup time should be gone.</p>
Slow gpu to cpu transfer

sanchit2843: <p>I am working on a research project and comparing semantic segmentation, object detection algorithms in terms of speed. The semantic segmentation model which I am using works around 10 fps faster than the object detection model. But when I add code to move tensor from GPU to CPU, the semantic model gets slower(probably because the tensor is of big size) and the fps difference is reduced to 2 fps.</p>
<pre><code class="lang-auto">prediction = prediction.cpu()
</code></pre>
<p>prediction is a 1*2*1920*1080 tensor.<br>
I want to find contours of prediction once it is converted to NumPy array. Is there any way I can speed up the process either by speeding up conversion or performing the contour detection process on GPU(though I couldn’t find any GPU implementation.)<br>
Any resource or ideas which can help in this situation would be of great help.<br>
Best.</p>

ptrblck: <p>Transferring a tensor from the GPU to the CPU will synchronize the code and might thus reduce the overall performance of your script.<br>
The transfer itself might not take a lot of time, but the code would have to wait until the GPU workload is finished to be able to transfer the result back to the CPU.</p>
Training loss suddenly explodes

jaewoo_song: <p>Hi, I’m trying to fine-tune BERT for entity recognition task.</p>
<p>While training, the train loss decreases steadily and evaluation scores increase as I expected, but suddenly at a certain point, the train loss becomes large and all evaluation scores become 0.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44673cf5ec62cee84775910227069e496e3afca1.png" data-download-href="https://discuss.pytorch.org/uploads/default/44673cf5ec62cee84775910227069e496e3afca1" title="tempsnip"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44673cf5ec62cee84775910227069e496e3afca1.png" alt="tempsnip" data-base62-sha1="9L7EAvtfH2quR6CfBpowcnrnQfn" width="576" height="500" data-dominant-color="302D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">tempsnip</span><span class="informations">962×834 16.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I think this might be vanishing/exploding gradient, but I don’t know how to fix this.</p>
<p>I added gradient clipping and learning rate scheduling which reduce the learning rate if the validation score does not decrease.</p>
<p>And I also initialized the weights of output layer for token-level classification into the xavier-uniform.</p>
<p>Here are specific details for my training.</p>
<ul>
<li>encoder: bert-base-uncased</li>
<li>initialization of output layer: xavier uniform</li>
<li>batch size: 8</li>
<li>starting learning rate: 5e-5</li>
<li>scheduler: ReduceLROnPlateau
<ul>
<li>factor: 0.1</li>
<li>patience: 3 epochs</li>
<li>threshold: 1e-3</li>
</ul>
</li>
<li>optimizer: AdamW</li>
<li>loss: CrossEntropy</li>
<li>gradient clipping: 1.0</li>
</ul>
<p>Thank you.</p>
H5netcfd/xarray onto pytorch

escorciav: <p>Anyone with (any) experience gluing <a href="https://github.com/h5netcdf/h5netcdf" rel="noopener nofollow ugc">h5netcfd</a> onto pytorch?<br>
I recently came across it as I was reading about <a href="http://xarray.pydata.org/en/stable/" rel="noopener nofollow ugc">xarray</a>.</p>
<p>I was wondering if someone has experience with it. I prefer to use standard stuff rather than reinvent the wheel DIY.</p>
<p>Cheers!</p>
Configure a convolutional layer so the padding can have 4 parameters/dimensions

woselwosel: <p>I’d like to create a Conv2d with different values for the 4 dimensions of the padding. For example, I’d like to do something like this:</p>
<p><code> m = nn.Conv2d(16, 33, (3, 5), stride=(1, 1), padding=(0,0, 1,1))</code></p>
<p>According to the official documentation that’s not currently possible, but I’d like to know if there’s any way or workaround to achieve this.</p>
<p>Thanks in advance.</p>

Nikronic: <p>Hi,</p>
<p>The simplest solution I can think of is that you create a wrapper class around available <code>nn.Conv2d</code> but instead of passing any paddings to <code>nn.Conv2d</code> object, you use explicit padding by using <code>torch.nn.functional.pad</code>.</p>
<p>Following post explains how to use explicit padding and wrapping it into another class that contains <code>nn.Conv2d</code> would pretty much easy.</p><aside class="quote quote-modified" data-post="2" data-topic="85121">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/same-padding-equivalent-in-pytorch/85121/2">Same padding equivalent in Pytorch</a> 
  </div>
  <blockquote>
    Hi, 
PyTorch does not support same padding the way Keras does, but still you can manage it easily using explicit padding before passing the tensor to convolution layer. Here, symmetric padding is not possible so by padding only one side, in your case, top bottom of tensor, we can achieve same padding. 
import torch
import torch.nn.functional as F

x = torch.randn(64, 32, 100, 20)
x = F.pad(x, (0, 0, 2, 1))  # [left, right, top, bot]
nn.Conv2d(32, 32, (4, 1))(x).shape


Please see these post abou…
  </blockquote>
</aside>

<p>Bests</p>
Tensor indexing from image feature batch using tensor of indices

shubhendu_jena: <p>Hi everyone. I have an image feature tensor of shape (16, 500, 500, 64) and a tensor of indices of shape (16, 6890, 2) which are essentially pixel locations I am meant to slice. How do I do this in PyTorch without using for loops (since I am doing this inside the model of my network and loops being slow doesn’t seem like a great idea). Any help is appreciated. Thanks in advance!</p>
<p>Just to make things clear, the slicing is supposed to give an output of size (16, 6890, 64), i.e. feature slices corresponding to pixel locations.</p>

shubhendu_jena: <p>Found a possible solution here : <a href="https://discuss.pytorch.org/t/how-to-do-the-tf-gather-nd-in-pytorch/6445/18" class="inline-onebox">How to do the tf.gather_nd in pytorch? - #18 by Cogito2012</a>. On applying this, I do get an output of desired shape. However, this error keeps triggered for the following operations ( <a href="https://discuss.pytorch.org/t/runtimeerror-cuda-error-device-side-assert-triggered/34213">RuntimeError: CUDA error: device-side assert triggered</a>)</p>
<p>Edit : This works! Turns out there was something wrong with my indices.</p>
Crossenthropy loss dimension out of range

Jake_Eum: <p>IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)</p>
<p>target-&gt; tensor([1., 0., 0., 1., 0., 1., 1., 0., 1., 1., 1., 1., 1., 0., 1., 1., 0., 0.,<br>
0., 0., 1., 0., 0., 0., 1., 0.])<br>
output-&gt; tensor([-3.2691, -3.1722, -3.2648, -3.3598, -3.2569, -3.4140, -3.2098, -3.2161,<br>
-3.4087, -3.2803, -3.1042, -3.1663, -3.3718, -3.3803, -3.1643, -3.2461,<br>
-3.2690, -3.3987, -3.3615, -3.1216, -3.3198, -3.4017, -3.1110, -3.2741,<br>
-3.1383, -3.1585], grad_fn=)</p>
<pre><code class="lang-auto">def train(epoch):
  model.train()
  for batch_idx, (data, target) in enumerate(train_loader):
    optimizer.zero_grad()
    output = model(data)
    print('target-&gt;',target[0])
    print('output-&gt;',output[0])
    criterion = nn.CrossEntropyLoss()
    loss = criterion(output[0],target[0])
    loss.backward()
    optimizer.step()
    if batch_idx % log_interval == 0:
      print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
        epoch, batch_idx * len(data), len(train_loader.dataset),
        100. * batch_idx / len(train_loader), loss.item()))
      train_losses.append(loss.item())
      train_counter.append(
        (batch_idx*64) + ((epoch-1)*len(train_loader.dataset)))
      torch.save(model.state_dict(), './result/model.pth')
      torch.save(optimizer.state_dict(), './result/optimizer.pth')
</code></pre>

KFrank: <p>Hi Jake!</p>
<aside class="quote no-group" data-username="Jake_Eum" data-post="1" data-topic="111259" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jake_eum/48/30524_2.png" class="avatar"> Jake_Eum:</div>
<blockquote>
<pre><code class="lang-auto">    criterion = nn.CrossEntropyLoss()
    loss = criterion(output[0],target[0])
</code></pre>
</blockquote>
</aside>
<p>I see two issues:</p>
<p>First, pytorch (models, losses, etc.) works with <em>batches</em> of samples.<br>
<code>criterion (output[0], target[0])</code> looks like you are passing<br>
a single sample from a batch to <code>criterion</code>.  This will break things.</p>
<p>Second, the shapes of <code>output</code> and <code>target</code> don’t match properly for<br>
use with <code>CrossEntropyLoss</code>.  <code>CrossEntropyLoss</code> expects your<br>
<code>output</code> to have shape <code>[nBatch, nClass]</code> and <code>target</code> to have<br>
shape <code>[nBatch]</code> (no class dimension).  <code>target</code> should consist of<br>
integer class labels that run from <code>0</code> to <code>nClass - 1</code>.</p>
<p>One final comment:  Your <code>target</code> looks like it is only <code>0</code>s and <code>1</code>s, so<br>
it looks like you are working with a binary classification problem.<br>
You can treat a binary problem as a two-class multi-class problem.<br>
Doing so is fine, and makes perfect sense, but you might prefer<br>
treating this explicitly as a binary problem and use something like<br>
<code>BCEWithLogitsLoss</code> as your loss criterion.</p>
<p>Best.</p>
<p>K. Frank</p>
Does Calling model.train() inside a function reinitialize the model weights each time

ParthShah412: <pre><code class="lang-auto">## First Approach
def train1(epoch):
    model.train()
    for idx, data in enumerate(train_dataloader):
       ....
for i in range(1,5):
    train1(i)
## Second approach
def train2():
    for idx, data in enumerate(train_dataloader):
       ....
model.train()
for i in range(1,5):
    train2()
</code></pre>
<p>Can calling model.train() in every epoch actually reinitialize the model weights? I am seeing no difference in model weights of epoch 1 Vs epoch n using first approach. Can this be the issue or the issue is something else?</p>

Dwight_Foster: <p>model.train() just tells the model it is learning things from the data. It has nothing to do with initializing weights or anything like that. It is not necessarily critical to training a model. Model.eval() is the opposite and that is used to tell the model nothing needs to be learnt and it also makes sure there isn’t an error passed with the batchnorm functions when passing a image with a batch_size of 1.</p>
Noise Segmentation In Unet

Danesh-A: <p>I am using Monai in pytorch for 2D segmentation. The output of the algorithm is showing a dice score accuracy of over 80 percent yet the images look like plain noise.</p>
<p>The parameters for the Unet:<br>
monai.networks.nets.UNet(2, 1, 2, channels= (32, 64, 128, 256, 512), strides=(2, 2, 2, 2))</p>
<p>I am not sure what is causing this issue.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8490a20589972781112929e2496173a2f84e01e.png" alt="image" data-base62-sha1="szNYNutLSHXusDY99dLCgDmjuIm" width="512" height="128"></p>
Distributed Data Paralell stucks after completing one epoch

Liana_Baghdasaryan: <p>I have a problem running multigpu training on one node with 2 GPU s.<br>
Currently the program stucks on   these lines<br>
imgs = imgs.to(self.device)<br>
true_masks = true_masks.to(self.device)</p>
<p>Can anyone help me?</p>

osalpekar: <p>Do you mind providing a script to reproduce your issue, the environment (HW setup, torch version, etc.) that you ran this in, and the logs you are seeing? It would help debugging the issue!</p>
torchvision.transforms.Normalize

nitaifingerhut: <p>Hi,<br>
I have a 3d tensor of size:<br>
(batch_size, 3, num_points)<br>
each sample represents 3d coordinates (num_points, each point is x,y,z).</p>
<p>I want to use torchvision.transforms.Normalize to normalize each axis<br>
meaning: the mean of x coordinate in each sample will be zero, with variance = 1 (same for y,z).<br>
the normalization should apply to each sample individually, regardless of other samples in batch.</p>
<p>thanks for your help!</p>