akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to make my model converge as the loss is not going below 2

username: aulakh83
cooked: <p>I am building a medical report generation model which is a variant of encoder decoder model equipped with both visual and semantic attention. It is somewhat similar to [1711.08195] On the Automatic Generation of Medical Imaging Reports. i have also calculated the loss as specified in the paper  batch_tag_loss = criterion_tag(tags, Variable(labels,requires_grad=False)).sum()</p>
<p>topics, ps = sentLSTM(vis_enc_output, mlc_output, captions, args.device)</p>
<pre><code>    loss_sent = criterion_stop (ps.view(-1, 2), prob.view(-1))

    loss_word = torch.tensor([0.0]).to(args.device)

    for j in range(captions.shape[1]):
        word_outputs = wordLSTM(topics[:, j, :], captions[:, j, :])

        loss_word += criterion_words(word_outputs.contiguous().view(-1, vocab_size), captions[:, j, :].contiguous().view(-1))

    loss = args.lambda_tag * batch_tag_loss+ args.lambda_sent * loss_sent + args.lambda_word * loss_word
</code></pre>
<p>trying to minimize the loss but even after 150 epochs the loss doesnot go below 2. What can be the reason behind it? I might be doing something wrong but cannot figure out. Can anyone please help</p>
How to learn the index of tensor?

username: lidehui
cooked: <p>Hi, how can I use PyTorch to learn index?<br>
As the code follow,</p>
<p><em>input=torch.randn((3,3))</em><br>
<em>index=torch.Tensor([1,2])</em><br>
<em>input = nn.Parameter(input)</em><br>
<em>index = nn.Parameter(index)</em><br>
<em>index=index.long()</em><br>
<em>output=input[index,:]</em><br>
<em>output_sum=output.sum()</em><br>
<em>output_sum.backward()</em><br>
<em>print(“input_grad”, input.grad)</em><br>
<em>print(“index_grad”, index.grad)</em></p>
<p>I got output,<br>
input_grad tensor([[0., 0., 0.],<br>
[1., 1., 1.],<br>
[1., 1., 1.]])<br>
index_grad None</p>
<p>Obviously, index has no gradient as it’s  turned to integer.<br>
How can I set the index to learnable? Thanks!</p>

username: huahuanZ
cooked: <p>You can’t backward the gradient to <code>index</code>, since <code>output_sum</code> is non-derivable to <code>index</code>.<br>
Can you explain why you want <code>index</code> to be learnable ?</p>

username: lidehui
cooked: <p>Maybe my network is strange, but learnable index is necessary to complete it. Do you know any alternative method to learn the index？</p>
Simulation of a paper

username: erfanm
cooked: <p>Hi, PyTorch community.<br>
I was looking to hire someone to simulate a paper for me. It’s mainly related to matrix completion and prediction with a neural network.<br>
I’m somehow short on time.( I have max 10 days)<br>
so anyone can help please msg me and we can talk about costs and payment methods.<br>
Here’s the paper link: <a href="https://ieeexplore.ieee.org/document/9214506" class="inline-onebox" rel="noopener nofollow ugc">Deep Learning-Enabled Sparse Industrial Crowdsensing and Prediction | IEEE Journals &amp; Magazine | IEEE Xplore</a><br>
And Here’s the Dataset link: <a href="https://zenodo.org/record/2654726#.YUMXH7gzZPY" class="inline-onebox" rel="noopener nofollow ugc">Sensorscope Data | Zenodo</a></p>
<p>Best, Erfan</p>

username: Sofia_Grace
cooked: <p>Hello <a class="mention" href="/u/erfanm">@erfanm</a></p>
<p>I would love to assist you with your requirement to simulate a paper.</p>
<p>Please reach me at my personal mail or skype to discuss further in details.</p>
<p>Looking forward to discuss with you .</p>
<p>Best Regards,</p>
<p>Sofia</p>
<p>Email : sofia at cisinlabs dot com / Skype :live:sofia_6831</p>
Filter all silent segment, including front, mid, and back of a complete audio using T.vad()

username: lllxw
cooked: <p>I have an audio file containing too much silent segment, and i want to filter the silent  segments with <code>torchaudio.functional.vad()</code>. But the funciton can only trim the front silent part of audio, it still remains silent segment in the mid and back. Can someone tell me why? Thanks!</p>
<p>Besides, I want to know the meaning of each parameter in <code>vad()</code>. Beacuse I try to split the audio into many segments with shorter duration, like 0.03s, process each with vad, and concat the process segment together finally, but i don’t know how to set the parameters.</p>
<p>Thanks a lot !</p>
RuntimeError: Function returned nan values in its 0th output

username: Siddharth_Tandon
cooked: <p>Hi,</p>
<p>I am getting below error when trying to train the network “Function ‘’ returned nan values in its 0th output”. Along with this I have enabled “torch.autograd.set_detect_anomaly(True)”</p>
<p><strong>but if I use grouped convolution, the network works completely fine</strong>. Btw, I am leveraging single GPU to train the network, and as per my understanding this should have similar effect with or without grouped convolution on single GPU (please correct my understanding)</p>

username: ptrblck
cooked: <p>I don’t quite understand the issue and don’t know how grouped convolutions fit into the use case.<br>
Did you check why the invalid outputs are created? I.e. is your model training exploding?</p>
Inplace Broadcast Reduction Operation

username: eduardo4jesus
cooked: <p>I have an operations like this:</p>
<pre><code class="lang-auto">x = torch.randint(16, size=(128, 1, 3, 32, 32), dtype=torch.float32)
h = torch.randint(16, size=(1, 6, 3, 32, 32), dtype=torch.float32)
z = (x*h).sum(2) # output shape (128, 6, 32, 32)
</code></pre>
<p>Is there a way to avoid the memory allocation of the intermediary step <code>x*h</code>? A mix of broadcast, inplace and reduction on a single call?</p>

username: eduardo4jesus
cooked: <p>Wondering if anyone would have a solution for this.</p>
<p>Also, the question isn’t quite clear. The issue is that the operation with broadcast would require more memory than available. So, the goal is not to avoid memory allocation, but to reduce instead of crashing.</p>

username: ptrblck
cooked: <p>You could replace the broadcasting with a loop iterating e.g. <code>x</code>, which would trade compute for memory.</p>
Torch.Tensor.softmax missing from documentation

username: Kaykay
cooked: <p>Tensors have a <code>softmax()</code> method, but it’s not displayed here: “torch.Tensor — PyTorch master documentation” <a href="https://pytorch.org/docs/master/tensors.html" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor — PyTorch master documentation</a></p>
<p>Is there a reason for this or was it overlooked?</p>

username: arya47
cooked: <p>Are you referring to <a href="https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html" class="inline-onebox" rel="noopener nofollow ugc">Softmax — PyTorch 1.9.0 documentation</a> ?</p>

username: Kaykay
cooked: <p>Yes, it’s the same function but as a method for a Tensor.</p>
<p>These 2 calls are identical:</p>
<pre><code class="lang-auto">torch.nn.Softmax(1)(tensor)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">tensor.softmax(1)
</code></pre>
<p>but the Tensor variant isn’t listed on the documentation anywhere (at least not that I could find).</p>

username: ptrblck
cooked: <p>Thanks for raising this issue. Would you mind creating an issue on <a href="https://github.com/pytorch/pytorch">GitHub</a> so that we can track and fix it? Would you also be interested in fixing the missing docs?</p>
RuntimeError: expected scalar type Long but found Float with Custom Data

username: NeuralFoX
cooked: <p>Hello, I am a beginner in PyTorch.<br>
I am trying to learn by building a toy example with custom data. Here is my code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
data={};
for iter in range(10):
 data[iter]=[torch.tensor([1]).long().type(torch.LongTensor),torch.tensor([2]).long().type(torch.LongTensor)]
model=nn.Linear(1,1);
loss=nn.BCELoss();
adam = optim.Adam(model.parameters(), lr=1e-1);
for epoch in range(10):
 adam.zero_grad();
 x=data[epoch][0];
 y=data[epoch][1];
 x,y=x.type(torch.LongTensor),y.type(torch.LongTensor)
 pred=model(x.type(torch.LongTensor));
 l=loss(pred,y);
 print(f'Epoch {epoch}: training loss: {l}');
 loss.backward();
 adam.step();
</code></pre>
<p>When I run this script this is the error I get:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "c", line 15, in &lt;module&gt;
    pred=model(x.type(torch.LongTensor));
  File "/home/samarth/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/samarth/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 96, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/samarth/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1847, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: expected scalar type Long but found Float
</code></pre>
<p>I have told PyTorch to convert the data into LongTensor multiple times, but it still won’t work.<br>
Can anyone help me?<br>
Thank you.</p>

username: ptrblck
cooked: <p>PyTorch assumes that the first argument to a method has the “right” <code>dtype</code> and thus claims that <code>Long</code> is expected in the model parameter.<br>
However, you should use <code>FloatTensor</code>s to train the model in order to be able to calculate gradients (which are defined for floating point types).<br>
Transform the data and target thus via <code>float()</code> and this error should be resolved.</p>
<p>However, your code will still not be working, since <code>nn.BCELoss</code> expects the model output to be in <code>[0, 1]</code> so apply a <code>sigmoid</code> on the model output or use <code>nn.BCEWithLogitsLoss</code> instead, which also increases the numerical stability.</p>
<p>Also, <code>loss.backward()</code> won’t work and you should replace it with <code>l.backward()</code>.</p>

username: NeuralFoX
cooked: <p>Thanks a lot. I will try this and get back to you.</p>

username: NeuralFoX
cooked: <p>Thank you for helping me. My code is working now.</p>
TRANSFER LEARNING FOR COMPUTER VISION TUTORIAL Question

username: andronikmk
cooked: <p>Hi everyone! I’m new to Pytorch and was going over a great tutorial. (<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 1.9.0+cu102 documentation</a>). But I’m having trouble figuring out how to make a simple inference. For example, all I want to do after the model is done training I want to input a test image and get back a result with some probabilities. Thanks!!</p>

username: arya47
cooked: <p>Lets assume you have loaded you model as ‘model’</p>
<ul>
<li>Load you image to the kernel ( I recomment using PIL)</li>
</ul>
<pre><code class="lang-auto">from PIL import Image
image = Image.open('path')
</code></pre>
<ul>
<li>Perform image transforms</li>
</ul>
<pre><code class="lang-auto">import torchvision.transforms as tf
toTensor = tf.Compose([
    tf.ToTensor()
])
image_input = toTensor(image)
image_input = image_input.resize(1,3,1500,1500)
image_input.shape
</code></pre>
<ul>
<li>Predict</li>
</ul>
<pre><code class="lang-auto">with torch.no_grad():
            preds = torch.sigmoid(model(image_input))
            preds = (preds &gt; 0.19).float()
</code></pre>
<ul>
<li>Plot the results</li>
</ul>
<p>I’ve done similar stuff in a project maybe this might help</p><aside class="onebox githubblob" data-onebox-src="https://github.com/aryaman4152/Satellite-images-semantic-segmentation/blob/main/Prediction.ipynb">
  <header class="source">

      <a href="https://github.com/aryaman4152/Satellite-images-semantic-segmentation/blob/main/Prediction.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/aryaman4152/Satellite-images-semantic-segmentation/blob/main/Prediction.ipynb" target="_blank" rel="noopener nofollow ugc">aryaman4152/Satellite-images-semantic-segmentation/blob/main/Prediction.ipynb</a></h4>


    <pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7de940cc",
   "metadata": {},
   "source": [
    "# Load model and make predictions (find roads)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb09a229",
   "metadata": {},
   "source": [
    "### Redefining model "
   ]
  },
  {
   "cell_type": "code",
</code></pre>


  This file has been truncated. <a href="https://github.com/aryaman4152/Satellite-images-semantic-segmentation/blob/main/Prediction.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Can I use the Adam optimizer in C language?

username: geunseok_choi
cooked: <p>Can I use the Adam optimizer of pytorch or tensorflow library in C language?</p>
<p>Here I want to optimize a function.</p>
<p>I can deliver a function, gradient of that function.</p>

username: arya47
cooked: <p>You can use adam in the c++ pytorch api<br>
Some links that might be usefull:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/src/optim/adam.cpp">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/src/optim/adam.cpp" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/src/optim/adam.cpp" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/torch/csrc/api/src/optim/adam.cpp</a></h4>


    <pre><code class="lang-cpp">#include &lt;torch/optim/adam.h&gt;

#include &lt;torch/csrc/autograd/variable.h&gt;
#include &lt;torch/nn/module.h&gt;
#include &lt;torch/serialize/archive.h&gt;
#include &lt;torch/utils.h&gt;

#include &lt;ATen/ATen.h&gt;
#include &lt;c10/util/irange.h&gt;

#include &lt;cmath&gt;
#include &lt;functional&gt;

namespace torch {
namespace optim {

AdamOptions::AdamOptions(double lr) : lr_(lr) {}

bool operator==(const AdamOptions&amp; lhs, const AdamOptions&amp; rhs) {
  return (lhs.lr() == rhs.lr()) &amp;&amp;
</code></pre>


  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/src/optim/adam.cpp" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p><a href="https://caffe2.ai/doxygen-c/html/classtorch_1_1optim_1_1_adam.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://caffe2.ai/doxygen-c/html/classtorch_1_1optim_1_1_adam.html</a></p>
An error of cudnn

username: sunshine1
cooked: <p>When I used pytorch to train the model, sometimes I got an error  “RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED”. I don’t know what causes this error. When I downgrade the pytorch version, this error can diminish. I want to know the reason.</p>

username: arya47
cooked: <p>There are many reasons why the error is thrown, can’t help without your code and complete error information</p>
Enabling out-of-order batching in a multiprocess DataLoader?

username: hgarrereyn
cooked: <p>I am working with a dataset where samples take uneven amounts of time to load. Most samples load on the order of 20ms while some samples take much longer (i.e. 10 seconds). I have a custom <code>Dataset</code> and I’m using a <code>DataLoader</code> to parallelize the loading process.</p>
<p>I’ve noticed that even when I set <code>shuffle=True</code>, the data loader will block waiting for certain workers. In an optimal situation, you would expect all available workers to be processing an “expensive” sample at any given moment but what I observe is that the workers are underutilized. A worker that processed a fast sample may be blocked on a worker that is processing a slow sample.</p>
<p>I can implement my own solution using <code>multiprocessing.Pool.imap_unordered</code> but I wanted to see if there is a canonical PyTorch way to solve this problem.</p>

username: hgarrereyn
cooked: <p>I have some example code that shows the speed difference between the existing torch <code>DataLoader</code> and a custom <code>imap_unordered</code> solution:</p>
<pre><code class="lang-auto">import time
import multiprocessing
import numpy as np

from torch.utils.data import DataLoader, Dataset

class DemoDataset(object):
    def __len__(self):
        return 1000
    
    def __getitem__(self, idx):
        if idx % 10 == 0:
            print('wait', idx)
            time.sleep(0.05)
            print('done', idx)
        
        return idx

class FastLoader(object):
    def __init__(self, ds, num_workers, batch_size, shuffle):
        self.ds = ds
        self.num_workers = num_workers
        self.batch_size = batch_size
        self.shuffle = shuffle

    def _process(self, batch):
        return [self.ds[x] for x in batch]

    def __iter__(self):
        work = list(range(len(self.ds)))
        if self.shuffle:
            np.random.shuffle(work)
        batches = []
        for i in range(0, len(work), self.batch_size):
            batches.append(work[i:i+self.batch_size])

        with multiprocessing.Pool(self.num_workers) as p:
            for item in p.imap_unordered(self._process, batches):
                yield item

def test(cls):
    a = time.time()
    ds = DemoDataset()
    data = cls(ds, num_workers=4, batch_size=4, shuffle=True)

    for item in iter(data):
        print(item)

    b = time.time()

    return b - a

if __name__=='__main__':
    fast = test(FastLoader)
    base = test(DataLoader)
    print('fast', fast)
    print('base', base)
</code></pre>
<p>In this example, the “fast” loader is nearly twice as fast since most of the time you have all available workers fully utilized on the slow samples.</p>
Apex AMP with torch.cuda.amp

username: Kaykay
cooked: <p>What are the downsides to using both mixed precision packages together? Currently, my script is something like this to choose one or the other:</p>
<pre><code class="lang-auto">with autocast(enabled=device.type == 'cuda' and apex is None):
	logps = model(images)
	loss = criterion(logps, labels)
	if apex is None:
		scaler.scale(loss).backward()
		scaler.step(optimizer)
		scaler.update()
	else:
		with amp.scale_loss(loss, optimizer) as scaled_loss: # type: torch.FloatTensor
			scaled_loss.backward()
		optimizer.step()
</code></pre>
<p>Is this conditional split necessary? Is there a specific reason not to mix them together, or is it relatively benign? It would make my code cleaner to remove the <code>apex is None</code> checks if unneeded but I’m not an expert on mixed precision so I don’t know the potential issues this may cause. I am considering something like this:</p>
<pre><code class="lang-auto">with autocast(enabled=device.type == 'cuda'):
	logps = model(images)
	loss = criterion(logps, labels)
	with amp.scale_loss(loss, optimizer) as scaled_loss:
		scaler.scale(scaled_loss).backward()
	scaler.step(optimizer)
	scaler.update()
</code></pre>

username: ptrblck
cooked: <p>I’m not sure why you would like to mix them, as <code>torch.cuda.amp</code> is also providing the <code>GradScaler</code> class so there is no need to use the deprecated <code>apex.amp</code> one.</p>

username: Kaykay
cooked: <p>I was under the impression that they were slightly different, is that not the case? I’ve been going back and forth to see if I found more favorable results using one or other, and that’s why my program has those conditional checks to make it easier to alternate instead of commenting out portions over and over again</p>

username: ptrblck
cooked: <p><code>apex.amp</code> is deprecated so you should stick to the native implementation. <a href="https://discuss.pytorch.org/t/torch-cuda-amp-vs-nvidia-apex/74994/3">This post</a> explains it in more detail.</p>
In-balanced number of batches in distributed training

username: HuangLED
cooked: <p>Hey, folks,</p>
<p>Wonder what pytorch’s expected behavior is when data partition is not balanced, then as a result each training process has a drastically different number of batches.   As a concrete example,  two training machines, each does 10 epoches on its local partition of data, then b/c data is not balanced between partitions, one partition has 100  batches and the other partition has 10 batches, how to make pytorch work in this scenario? Can we have the remaining 90 batches proceed without doing all-reduce?</p>
<p>One idea is probably to let each partition has different batch_size, but for some reason, it is hard to have that information as well.</p>

username: rvarm1
cooked: <p>We have built uneven input support in DDP for this purpose. It is designed to allow users to train DDP models when there are a different # of inputs across ranks (if model has no communication other than those done by DDP), as well as throw an exception that can be caught and recovered from if the model does have custom user communication.</p>
<p>Here are the docs: <a href="https://pytorch.org/docs/stable/_modules/torch/nn/parallel/distributed.html#DistributedDataParallel.join" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.parallel.distributed — PyTorch 1.9.0 documentation</a></p>
<p>and tutorial <a href="https://pytorch.org/tutorials/advanced/generic_join.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed Training with Uneven Inputs Using the Join Context Manager — PyTorch Tutorials 1.9.0+cu102 documentation</a>. Feel free to follow up with anymore questions!</p>
Best way to handle throw_on_early_termination exception for models with sync_bn?

username: orchidmajumder
cooked: <p>When trying to train a model in DDP which has a <code>sync_batch_norm</code> layer with uneven inputs, the suggestion is to use the parameter <code>throw_on_early_termination</code> as True within the<code> join()</code> context manager of DDP. I wanted to know what is the best way to handle the exception raised by the <code> join()</code> context manager in case one rank exhausts all its input and we just want the training to continue normally for next iterations? Should we just swallow the exception and move forward? Should we put a CUDA synchronize call as part of the exception handling? Anything else?</p>

username: rvarm1
cooked: <blockquote>
<p>we just want the training to continue normally for next iterations</p>
</blockquote>
<p>Assuming you mean continue training with DDP, this exception basically indicates this is not possible - one rank has finished all its inputs while others have not. As a result, if you continue training DDP will hang (probably at SyncBN step) and the exception is designed to indicate this.</p>
<p>The main use case of the exception is to use it as a “signal” to finish the training process. i.e. all processes will raise this exception, and application code can catch it and terminate the main training loop, saving/evaluating the trained model appropriately. Usually inputs are only off by a few examples across ranks, so it should be fine to terminate training at this stage - if this isn’t true then you may want to look into how to better balance the dataset across ranks.</p>

username: orchidmajumder
cooked: <p>Got it, thanks for your reply!</p>
About stochastic gradient descent

username: ljh
cooked: <p>Graph attention network normally dose not support input to be a batch, I want to know that whether I can implement stochastic gradient descent by feed one data at one time, accumulate the loss and finally divide the loss by the batch_size that I define myself? Does it reach the same goal as input as a batch？<br>
It should be like this:</p>
<pre><code class="lang-auto">### method1
batch_size = 64
loss_batch = 0
for i in range(batch_size):
    output = model(data)  # data.shape(224,224,3)
    loss = calculate the loss for the output 
    loss_batch = loss_batch + loss
loss_batch = loss_batch / batch_size
loss_batch.backward()

### method2
output = model(data_batch) # data_batch.shape(batch_size, 224,224,3)
loss_batch = calculate the loss for the output 
loss_batch.backward()

</code></pre>
<p>Dose the two methods reach the same goal?</p>

username: googlebot
cooked: <p>Yes, though your method1 is bad, it is better to accumulate gradients with loss.backward() in a loop, this releases “backward graphs”. But you should use method2, unless there are obstacles with some unimplemented batched operations.</p>

username: ljh
cooked: <p>Thanks for your reply.<br>
I still have one question though.<br>
If I accumulate the gradients with loss.backward() for n times and update the parameters with one opt.step(). Is it equivalent to n times of loss.backward() and n times of opt.step() with batch_size = 1. Or one loss.backward() and opt.step() with batch_size=n?</p>

username: googlebot
cooked: <p>if you do N optimizer ‘steps’, your forward() will use N different sets of parameter values, it is not an accumulation.</p>
<p>You also have to scale loss or gradients or LR if doing accumulation, to have an equivalent effect with batches.</p>
Error installing torch(1.9.0) with Cuda 11.1

username: s_t
cooked: <p>I have Nvidia A6000. Itried to run command pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a> but i got following errors.</p>
<p>pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f torch-1.9.0+cu111-cp38-cp38-linux_x86_64.whl<br>
Defaulting to user installation because normal site-packages is not writeable<br>
Looking in links: torch-1.9.0+cu111-cp38-cp38-linux_x86_64.whl<br>
Processing ./torch-1.9.0+cu111-cp38-cp38-linux_x86_64.whl<br>
ERROR: Could not find a version that satisfies the requirement torchvision==0.10.0+cu111 (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3, 0.5.0, 0.6.0, 0.6.1, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.9.1, 0.10.0)<br>
ERROR: No matching distribution found for torchvision==0.10.0+cu111</p>

username: Samuel_Bachorik
cooked: <p>Hi, try to install Anaconda,create env and then run this command</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia
</code></pre>
<p><strong>NOTE:</strong> ‘nvidia’ channel is required for cudatoolkit 11.1</p>
Why does torch.max use so much GPU memory?

username: clavichord93
cooked: <p>I tried to use <code>torch.max</code> on a <code>[1000, 1000, 3, 256]</code> tensor along the <code>dim=2</code>, but got the CUDA out of memory error. Why does <code>torch.max</code> use so much GPU memory?</p>
<p>Code:</p>
<pre><code class="lang-auto">import torch
a = torch.rand(1000, 1000, 3, 256).cuda()
b = a.max(dim=2)[0]
</code></pre>
<p>And the error message:</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 15.26 GiB (GPU 0; 23.70 GiB total capacity; 5.72 GiB already allocated; 13.82 GiB free; 5.72 GiB reserved in total by PyTorch)
</code></pre>
<p>The tensor <code>a</code> uses 2.8G memory, but <code>torch.max</code> requires 15.26GB memory.</p>

username: eqy
cooked: <aside class="quote no-group" data-username="clavichord93" data-post="1" data-topic="129280">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/a5b964/48.png" class="avatar"> clavichord93:</div>
<blockquote>
<pre><code class="lang-auto">b = a.max(dim=2)[0]
</code></pre>
</blockquote>
</aside>
<p>Very interesting, I can reproduce this on 3080 and a recent build. I’ll see if I can find more about the underlying cause.</p>

username: eqy
cooked: <p>We’ve opened an issue for this:<br>
<a href="https://github.com/pytorch/pytorch/issues/63869" rel="noopener nofollow ugc"><code>min</code>/<code>max</code> require a huge allocation on GPU · Issue #63869 · pytorch/pytorch (github.com)</a></p>
<p>For now, if you don’t need to use the indices, a workaround would be to use <code>amax</code> rather than max here:</p>
<pre><code class="lang-auto">import torch
a = torch.rand(1000, 1000, 3, 256).cuda()
b = a.amax(dim=2)
</code></pre>

username: clavichord93
cooked: <p>Yes, <code>amax</code> uses less memory, but it is much slower during training.</p>
How to calculate step_size_up in Cyclic Learning rate

username: talhaanwarch
cooked: <p>I want to know, is there any formula to calculate cylic learning rate <strong>step_size_up</strong> .<br>
I know the number of images, let say 10,000, batch size 50, epoch 10. What should be my <strong>step_size_up</strong> ?</p>
Incorrect dtype conversion from PIL

username: Latope2-150
cooked: <p>Hi,</p>
<p>I am experiencing a surprising behavior when converting a binary PIL image to BoolTensor through numpy.</p>
<pre><code class="lang-python">import numpy as np
import torch
from PIL import Image, ImageDraw

mask = Image.new(mode='1', size=(6, 6), color=False)
draw = ImageDraw.Draw(mask)
draw.polygon([(2, 2), (4, 2), (4, 4), (2, 4)], fill=True)

np_mask = np.array(mask)
print(np_mask.dtype, np.unique(np_mask))

tensor_mask = torch.from_numpy(np_mask)
print(tensor_mask.dtype, torch.unique(tensor_mask))

float_mask = tensor_mask.float()
print(float_mask.dtype, torch.unique(float_mask))
</code></pre>
<p>I would expect this code to print 0 and 1 as unique values of the <code>float_mask</code>. However, the unique values found are 0 and 255.</p>
<p>Where does this behavior come from? It does not happen for numpy as the code shows.</p>

username: ptrblck
cooked: <p>This seems to be related to <a href="https://github.com/pytorch/pytorch/issues/54789">this issue</a> as it seems that the <code>BoolTensor</code> is transformed to a <code>ByteTensor</code> where <code>True</code> represents <code>255</code>. Feel free to add your use case to this issue.</p>

username: Latope2-150
cooked: <p>Indeed this seems to be the same issue. For some reason I did not find it before posting. Thanks!</p>
Indices of the batch where I use SubsetRandomSampler

username: ElQadasi
cooked: <p>I want to print the indices of the batch where I use SubsetRandomSampler.</p>
<p>train_dataset = datasets.MNIST(root=’./train’, train=True,download=True,transform=transform)<br>
train_loader = DataLoader(train_dataset, batch_size=32,sampler=SubsetRandomSampler(range(len(train_dataset))))</p>
<p>for train_x, train_label in train_loader:<br>
# I need to print the indices of each batch which should be a random number between 0-59999</p>

username: ptrblck
cooked: <p>You could create a custom <code>Dataset</code> and return the index with the data and target tensors in the <code>__getitem__</code> method. To do so, take a look at <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a> and or override the <code>MNIST</code> dataset with your custom <code>__getitem__</code> implementation.</p>

username: ptrblck
cooked: <p>I think the best approach depends on your actual use case.<br>
If you want to get the index used to load the current sample, I would write a custom <code>Dataset</code> and just return it. This would avoid any issues with using another sampler or any kind of shuffling etc. since the index will be directly returned. On the other hand, if you are more interested in the sampler indices for a particular reason, you might want to check its implementation or write a custom one etc. (in that case I’m not sure I understand the question correctly).</p>

username: ElQadasi
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>. Yes, I customized the dataset and I could get the indices throught <strong>getitem</strong>.</p>
<p>I made things complicated at the beginning. Thank you again for your help.</p>
[Bug?]Two tensors have the same data_ptr

username: fangwei123456
cooked: <h1>
<a name="name-version-build-channel-1" class="anchor" href="#name-version-build-channel-1"></a>Name                    Version                   Build  Channel</h1>
<p>pytorch                   1.7.1           py3.8_cuda110_cudnn8_0    pytorch</p>
<p>Codes:</p>
<pre><code class="lang-auto">import torch

device = 'cuda:0'
x = torch.rand([2, 3, 4], device=device).transpose(0, 2)
y = torch.zeros_like(x)
args_list = [x, y]
ret_list = []
for item in args_list:
    if isinstance(item, torch.Tensor):
        item = item.contiguous()
        ret_list.append(item.data_ptr())
print(ret_list)
</code></pre>
<p>Outpus:</p>
<p><code>[47284487168, 47284487168]</code></p>
<p><code>x</code> and <code>y</code> are two different tensors, but they have the same data_ptr.</p>

username: ptrblck
cooked: <p>You are creating new tensors in:</p>
<pre><code class="lang-python">item = item.contiguous()
</code></pre>
<p>and replace them directly in each iteration, which could (and in your case will) reuse the <code>data_ptr</code>.<br>
If you print <code>item.data_ptr()</code> of <code>x</code> and <code>y</code> or <code>item</code> before the <code>contiguous()</code> call you would see that they are different.</p>

username: fangwei123456
cooked: <aside class="quote no-group" data-username="fangwei123456" data-post="1" data-topic="132104">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fangwei123456/48/12650_2.png" class="avatar"> fangwei123456:</div>
<blockquote>
<p><code>contiguous</code></p>
</blockquote>
</aside>
<p>Thanks! I have another question, which is how to set contiguous for some tensors? For example:</p>
<pre><code class="lang-auto">
import torch

def set_contiguous(args_list: list):
    for i in range(args_list.__len__()):
        args_list[i] = args_list[i].contiguous()

device = 'cuda:0'
x = torch.rand([2, 3, 4], device=device).transpose(0, 2)
y = torch.zeros_like(x)
args_list = [x, y]
set_contiguous(args_list)
print(x.is_contiguous())
</code></pre>
<p>The outputs are:<br>
<code>False</code></p>
<p>As you said, <code>args_list[i] = args_list[i].contiguous()</code> creates new tensor, and <code>x, y</code> are not set to contiguous tensors. How can I define a correct <code>set_contiguous</code> function?</p>

username: ptrblck
cooked: <p>The entries in <code>args_list</code> are contiguous, since you are assigning the new tensor to it:</p>
<pre><code class="lang-python">print(args_list[0].is_contiguous())
&gt; True
</code></pre>

username: fangwei123456
cooked: <p>Thanks! It seems hard to find a real ‘inplace’ function to make ‘x, y’ contiguous. Now I use the outputs to overlap inputs, which behaves like ‘inplace’:</p>
<pre><code class="lang-auto">import torch

def set_contiguous(*args):
    ret = []
    for item in args:
        ret.append(item.contiguous())
    return ret
device = 'cuda:0'
x = torch.rand([2, 3, 4], device=device).transpose(0, 2)
y = torch.zeros_like(x)
x, y = set_contiguous(x, y)
print(x.is_contiguous())
</code></pre>
Caching Memory Allocator

username: Woosung
cooked: <p>Hello,</p>
<p>I found the document that PyTorch utilizes the caching memory allocator for fast memory deallocation without device synchronization.<br>
<a href="https://pytorch.org/docs/stable/notes/cuda.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/cuda.html</a></p>
<p>Is it a general concept or PyTorch specified memory management technique?<br>
Are there any detailed documents or information about caching memory allocator?<br>
I want to understand how caching memory allocator works.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>It’s a common pattern used on top of CUDA in different frameworks to avoid the expensive direct allocations. You could take a code into the <a href="https://github.com/pytorch/pytorch/blob/635d864b26b740d2c88966738111c41b2aedd2ca/c10/cuda/CUDACachingAllocator.cpp">implementation</a> for detailed information.</p>

username: Woosung
cooked: <p>Thanks!!<br>
That was fast <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
For example，there are 3 tensor A, B, C, where C = A*B. When I set B "detach()"，why A.grad is None?

username: sea_Fire
cooked: <p>like the picture below：<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/2/3214353378ab11f7ddc36992f259ada10c47dad7.png" alt="question" data-base62-sha1="791arYVVT7QBpirHVVMmPQR02Oz" width="430" height="490"></p>
Images are blur[Solved the issue]

username: 44_Pashikanti_Pranit
cooked: <p>I am working on a binary classification problem when i am displaying my image its blurred this is my <a href="https://colab.research.google.com/drive/1dsj34GTnLzIAe48e4Iv3wvTIGqQka4L6#scrollTo=IER6dxWs545P" rel="noopener nofollow ugc">code</a><br>
which affecting my model performance.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2f10c71243f781d68614d3ee17dd1bfd2152f29.png" data-download-href="https://discuss.pytorch.org/uploads/default/d2f10c71243f781d68614d3ee17dd1bfd2152f29" title="Screenshot (375)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2f10c71243f781d68614d3ee17dd1bfd2152f29_2_690x388.png" alt="Screenshot (375)" data-base62-sha1="u64IcVDitcJKcw8JbS2jKGAtH9T" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2f10c71243f781d68614d3ee17dd1bfd2152f29_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2f10c71243f781d68614d3ee17dd1bfd2152f29_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2f10c71243f781d68614d3ee17dd1bfd2152f29_2_1380x776.png 2x" data-dominant-color="41423F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (375)</span><span class="informations">1920×1080 239 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Your notebook is unfortunately not public, but based on the posted images I would guess that the image resolution is just really low?</p>

username: 44_Pashikanti_Pranit
cooked: <p>Next time ill remember it, Can you help me with the resources for model architecture using pytorch</p>

username: ptrblck
cooked: <p>Feel free to create a new topic with your question and I’m sure I’ll take a look at it as well as other users. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Dataset partition

username: lxm-001
cooked: <p>i want to question if the data augmentation photo can be used in the validation?</p>

username: ptrblck
cooked: <p>I’m not quite understanding the question. Could you explain your use case and issue a bit more, please?</p>

username: gphilip
cooked: <p>I am not <a class="mention" href="/u/lxm-001">@lxm-001</a> , but here is how I parsed their question:</p>
<blockquote>
<p>There are some images that I got by applying various augmentation steps to images which are present in the training data. Is it OK to use such images (which I got by augmenting the “real” training data) as inputs to my validation step as well, or should validation be done entirely on “real” training data?</p>
</blockquote>
<p>Perhaps they meant to ask something else, though!</p>

username: ptrblck
cooked: <p>Ah OK. In case that’s the question: I would try to keep the validation dataset as close to the test dataset (which one should only use once the model training is done) so that the validation dataset can indeed act as a proxy for the test set (i.e. the unseen samples).<br>
In that case, no I would not add random data augmentation to the validation data, but would make sure to use the same (static) preprocessing such as resizing, normalization etc.</p>
The speed and accuray of tiny model

username: sunshine1
cooked: <p>The tiny model such as yolov3-tiny or yolov4-tiny is designed for limited computing resource.Although their speed  increases much, the accuracy decrease obviously. If there is a good way to improve speed and accuracy jointly?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sunshine1" data-post="1" data-topic="132066">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/9de053/48.png" class="avatar"> sunshine1:</div>
<blockquote>
<p>If there is a good way to improve speed and accuracy jointly?</p>
</blockquote>
</aside>
<p>This sounds like the goal of a lot of research projects as it’s a win-win. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"><br>
Besides general good research ideas and clever model architectures I don’t know if there are other recommendations.</p>
RuntimeError: Expected 5-dimensional input for 5-dimensional weight 45 3 1 7 7, but got 4-dimensional input of size [117, 720, 1280, 3] instead

username: asha97
cooked: <p>I am trying to work on video dataset , so when I read the video using the function <code>torchvision.io.read_video</code><br>
I am getting the ouput in form of a tuple. (I only want to work with video frames).<br>
If i try to pass elements from tuple one by one to <code>r2plus1d=models.video.r2plus1d_18(pretrained=False, progress=True)</code><br>
I am getting this error.</p>
<p>Is this not a correct way to work with videos, if it is correct how I should modify this?</p>

username: ptrblck
cooked: <p>Based on the error message a 5D input tensors is expected. As described in the <a href="https://pytorch.org/vision/0.8/models.html#video-classification">docs</a> the tensors should have the shape <code>[batch_size, 3, T, H, W]</code>, where <code>T</code> represents the number of frames while <code>H</code> and <code>W</code> were set to <code>112</code> in the pretrained models.</p>
When get the intermediate features, it's get the OOM error

username: nick_zhang
cooked: <p>I want to get the intermediate features for calculating the perceptual loss, but get the OOM error, blow is my code:</p>
<pre><code class="lang-auto">class Vgg19(torch.nn.Module):
    def __init__(self):
        
        super(Vgg19, self).__init__()
        features = list(vgg19(pretrained=True).features)
        self.features = torch.nn.ModuleList(features).eval()
        
    def forward(self, x):
        
        results = []
        for ii, model in enumerate(self.features):
            x = model(x)
            
            if ii in {1, 6, 11, 20, 29}:
                results.append(x)
        return results
</code></pre>
<pre><code class="lang-auto">vgg = Vgg().to(torch.device("cuda"))
features_list = vgg(x)
</code></pre>
<p>when I get the intermediate features by above-mentioned code, I get the OOM error? why?<br>
I train this model with 4 Nvidia v100-16G cards and use distributedataparallel mode.</p>

username: ptrblck
cooked: <p>You are storing 5 potentially large tensors and the memory requirement depends on all operations you are performing on this data or in your training script in general.<br>
Could you describe your expectation, i.e. why is an OOM error not expected? Was the core working in another form, i.e. are you seeing the OOM only by storing these 5 tensors? If so, how are you processing them? Did you try to check the memory usage before the changes and afterwards?</p>
Error When Reading Data

username: will2610
cooked: <p>Im am converting images into torch tensors for a segmentation model, bit new to using torch, and when running my model I get this error.</p>
<p>MetaImage: M_ReadElementsData:data not read completely ideal = 5734400 : actual = 2700286</p>
<p>No indication of where this in the code, was wondering if anyone has had this issue before, sorry if this ends up not being torch realted.</p>

username: ptrblck
cooked: <p>The <code>MetaImage</code> errors/warning are usually raised by ITK. Are you working on a medical use case with ITK? If so, check the data loading pipeline and try to narrow down if reading a dicom or so fails.</p>
[Beginner Question] Trouble using view()

username: machina
cooked: <p>“real” is a mini-batch and the goal is to make a copy of that batch called “adv_ex” and add small amounts of noise (gen(item)) to each image in adv_ex, however, it throws an error that indicates that I’m using a view incorrectly. Any help interpreting this error? Thanks</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de02f705d4e9a6bcdfb2ee5c34d92447973ed7e4.png" alt="image" data-base62-sha1="vG0lKop4bNsNNLGFzW4cWzFikuM" width="560" height="216"></p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_24355/3111319459.py in &lt;module&gt;
     14         for idx,item in enumerate(adv_ex):
     15             #item.size() = 784
---&gt; 16             purturbation = gen(item)
     17             adv_ex[idx] = adv_ex[idx] + purturbation #item.size() = 784
     18 

RuntimeError: A view was created in no_grad mode and its base or another view of its base has been modified inplace with grad mode enabled. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one.
</code></pre>

username: ptrblck
cooked: <p>Could you post a minimal, executable code snippet to reproduce the issue, please?<br>
Based on the error message and the posted code snippet it’s a bit hard to give a proper debugging advice besides repeating the error message.</p>
Adam optimizer performance difference

username: Sudu
cooked: <p>I am training a model with 2 Adam optimizers (in separate code), but one of the optimizers performs more superior than the other. The model I am using can be found here <a href="https://github.com/cydonia999/Learning_to_See_in_the_Dark_PyTorch/blob/470a6a8e9c6367d8fa88ee6d1dea211dd9fb1f81/models/lsid.py#L45" rel="noopener nofollow ugc">Model</a>, and the first optimizer is what is being used by the repository of the above model is</p>
<pre><code class="lang-auto">optim = torch.optim.Adam(
            [
                {'params': get_parameters(model, bias=False)},
                {'params': get_parameters(model, bias=True), 'lr': cfg['lr'] * 2, 'weight_decay': 0},
            ],
            lr=cfg['lr'],
            weight_decay=cfg['weight_decay'])
</code></pre>
<p>The get_parameters function can be found here <a href="https://github.com/cydonia999/Learning_to_See_in_the_Dark_PyTorch/blob/470a6a8e9c6367d8fa88ee6d1dea211dd9fb1f81/demo.py#L29" rel="noopener nofollow ugc">get_parameters</a>.<br>
The configurations dictionary is</p>
<pre><code class="lang-auto">configurations = {
    1: dict(
        max_iteration=1000000,
        lr=1e-4,
        momentum=0.9,
        weight_decay=0.0,
        gamma=0.25,
        step_size=32300, # "lr_policy: step"
        interval_validate=1000,
    ),
}
cfg = configurations[1]
</code></pre>
<p>The above optimizer achieves performance of 28.6dB in just 50k iterations. I then used the more common Adam optimizer definition</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model.parameters(),
            lr=args.lr,
            weight_decay=args.weight_decay)
</code></pre>
<p>The learning rate I used is <code>1e-4</code> and weight decay <code>1e-4</code>. This optimizer gives results of only about 27.5dB even after 100k iterations. Also the rest of the code is identical for both the optimizers. So why is this optimizer worse than the previous one?</p>