akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
With torch.nograd() - context manager

GGPYTORCH000: <p>It is common knowledge that<br>
with fp.open()<br>
the fp.close() will happen automatically after indented lines afterward with opening statement finished executing.</p>
<p>I came across:<br>
with torch.nograd(), does it mean at the end, it will re-enable back gradient back? How does context manager know to turn back on if so?</p>

ptrblck: <p>Yes, gradient calculation will be enabled when this context is left assuming it was allowed before.<br>
During its <code>__enter__</code> call it will save the current status of <code>torch.is_grad_enabled()</code> and will reset it in its <code>__exit__</code> method as seen <a href="https://github.com/pytorch/pytorch/blob/ad782ff7df950dfda64e271dd4ee3c6128971103/torch/autograd/grad_mode.py#L133-L143">here</a>.</p>
Which clone method should I use to create a deep copy of tensor

alisonqiu: <p>How to create a copy of <code>tensorInputs</code> with the same values but different addresses for each data point in the following code? a search of the <a href="https://pytorch.org/cppdocs/search.html?q=clone&amp;check_keywords=yes&amp;area=default#" rel="noopener nofollow ugc">documentation</a> brought me to many possible <code>clone</code> methods, and none of them have detail documentation about how to use them.</p>
<p>The reason for clone is because I want to run the following code in a loop and I need to avoid data racing</p>
<pre><code class="lang-auto">            at::Tensor tensorInputs = torch::from_blob((void*)wavBuffer, {1, bufLength}, at::kFloat);


            float* floatInput = tensorInputs.data_ptr&lt;float&gt;();
            if (!floatInput) {
                return nil;
            }
   
            c10::InferenceMode guard;
            CFTimeInterval startTime = CACurrentMediaTime();
            auto result = _impl.forward({ tensorInputs }).toStringRef();
            CFTimeInterval elapsedTime = CACurrentMediaTime() - startTime;
            NSLog(@"inference time:%f", elapsedTime);
                
            return [NSString stringWithCString:result.c_str() encoding:[NSString defaultCStringEncoding]];
</code></pre>
<p>I tried the following from this <a href="https://discuss.pytorch.org/t/initializing-a-tensor-from-a-tensor-c/50058">post</a> and it doesn’t work:<br>
tensorCopy = tensorInputs.detach().clone(LEGACY_CONTIGUOUS_MEMORY_FORMAT);</p>

ptrblck: <p>Calling <code>clone</code> on the tensor sounds like the right approach. Could you describe the issues you are seeing using it?</p>

alisonqiu: <p>data race still happened because I’m cloning tensorInputs while modifying it in the _impl.forward function. I got the error:</p>
<p><code>Thread 17: EXC_BAD_ACCESS (code=1, address=0x14a585000)</code></p>

ptrblck: <p>If you <code>clone</code> <code>tensorInputs</code> via:</p>
<pre><code class="lang-auto">at::Tensor tensorInputs = torch::from_blob((void*)wavBuffer, {1, bufLength}, at::kFloat).clone();
</code></pre>
<p>the object will stay alive until it gets out of scope.<br>
I don’t understand how you are modifying it inside the <code>forward</code> method but maybe you want to <code>clone</code> it there so the object stays alive?</p>
Does PyTorch's PixelShuffle use Sugawara initialization?

carbocation: <p>In <a href="https://arxiv.org/abs/1806.02658" rel="noopener nofollow ugc"> Super-Resolution using Convolutional Neural Networks without Any Checkerboard Artifacts</a>, Sugawara <em>et al</em> suggest a solution to the <a href="https://distill.pub/2016/deconv-checkerboard/" rel="noopener nofollow ugc">checkerboard problem noted by Odena, Dumoulin, and Olah</a>. Interestingly, the solution has to do with initialization.</p>
<p>It’s not super clear to me what the default initialization is for PyTorch’s PixelShuffle. Does it use Sugawara initialization? If not, does it at least use the earlier <a href="https://arxiv.org/abs/1707.02937" rel="noopener nofollow ugc">ICNR</a> method of normalization?</p>

carbocation: <p>Looking at how <a href="https://github.com/fastai/fastai/blob/422fd1588e84704a15be02d99e8d64aab236a25f/fastai/layers.py#L364-L388" rel="noopener nofollow ugc">FastAI initializes PixelShuffle</a>, it gives the impression that PyTorch is not doing any special initialization.</p>

ptrblck: <p>PyTorch’s <code>nn.PixelShuffle</code> layer does not contain any trainable parameters and thus does not initialize them with a specific method.<br>
Based on the linked code snippets you shared it seems that at least FastAI implements a <code>PixelShuffle_ICNR</code> layer, which contains a <code>ConvLayer</code> as well as a <code>PixelShuffle</code> layer.<br>
The former then seems to be initialized with a specific method.</p>

carbocation: <p>Very good point, thanks for your reply. The fact that PixelShuffle requires external / surrounding code to be specially initialized for optimal performance is a bit unusual. But it seems like something best solved by a library rather than within PyTorch itself.</p>
ModuleNotFoundError: No module named 'torch.utils.model_zoo'

delip: <p>I am on the latest PyTorch and <code>torchaudio</code> and I get this error when I import torchaudio. Any ideas?</p>
<pre><code class="lang-auto">1 # Imports used through the rest of the notebook.
      2 import torch
----&gt; 3 import torchaudio
      4 import torch.nn as nn
      5 import torch.nn.functional as F

File ~/.python/current/lib/python3.10/site-packages/torchaudio/__init__.py:1
----&gt; 1 from torchaudio import (  # noqa: F401
      2     _extension,
      3     compliance,
      4     datasets,
      5     functional,
      6     io,
      7     kaldi_io,
      8     models,
      9     pipelines,
     10     sox_effects,
     11     transforms,
     12     utils,
     13 )
     14 from torchaudio.backend import get_audio_backend, list_audio_backends, set_audio_backend
     16 try:
...
   (...)
     25         progress_bar (bool, optional): Display a progress bar (Default: ``True``).
     26     """

ModuleNotFoundError: No module named 'torch.utils.model_zoo'
</code></pre>
<p>Details on my environment is here:</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.13.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.10.4 (main, Nov 29 2022, 20:00:25) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-1094-azure-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla V100-PCIE-16GB
Nvidia driver version: 520.61.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] rotary-embedding-torch==0.2.1
[pip3] torch==1.13.1
[pip3] torchaudio==0.13.1
[conda] No relevant packages
</code></pre>

ptrblck: <p>I cannot reproduce the issue using the same versions:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.1+cu117'
&gt;&gt;&gt; import torchaudio
&gt;&gt;&gt; torchaudio.__version__
'0.13.1+cu117'
</code></pre>
<p>Are you using multiple virtual environments or could another PyTorch version have been installed into the base env?</p>
Bug in Pytorch GPU memory handling?

FaultyBagnose: <p>I’m running a test modelled from this Stack Exchange post: <a href="https://stackoverflow.com/questions/57496285/why-is-the-memory-in-gpu-still-in-use-after-clearing-the-object" rel="noopener nofollow ugc">https://stackoverflow.com/questions/57496285/why-is-the-memory-in-gpu-still-in-use-after-clearing-the-object</a></p>
<pre><code class="lang-auto">
import torch
import GPUtil

def memory_test(device):
    x = torch.rand(10000,300,200).cuda(device)

memory_test(1)
GPUtil.showUtilization()
torch.cuda.empty_cache()
GPUtil.showUtilization()
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">| ID | GPU  | MEM |
-------------------
|  0 |   0% |  0% |
|  1 |   0% | 26% |
| ID | GPU  | MEM |
-------------------
|  0 |   0% |  6% |
|  1 |  43% |  6% |
| ID | GPU  | MEM |
</code></pre>
<p>When torch.cuda.empty_cache() cleans up, it is well-known that it keeps a small piece (6%) for cache overhead. But it seems to allocate this overhead also for GPU 0 (which has not been in use). Is this a bug?</p>
<p>FaultyBagnose</p>

FaultyBagnose: <p>Has anyone tried to reproduce this potential bug?</p>

ptrblck: <p>Yes, and I cannot reproduce it so maybe your PyTorch version is old and this issue might have already been fixed:</p>
<pre><code class="lang-python">| ID | GPU | MEM |
------------------
|  0 |  0% |  0% |
|  1 | 38% |  7% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |
|  4 |  0% |  0% |
|  5 |  0% |  0% |
|  6 |  0% |  0% |
|  7 |  0% |  0% |
| ID | GPU | MEM |
------------------
|  0 |  0% |  0% |
|  1 |  5% |  1% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |
|  4 |  0% |  0% |
|  5 |  0% |  0% |
|  6 |  0% |  0% |
|  7 |  0% |  0% |
</code></pre>
Collaborative Filtering using Pytorch

arkh: <p>Hi!<br>
I am trying to create a Netflix-like recommendation system using the movielens dataset. I have trained a model with the following definition:</p>
<pre><code class="lang-auto">class RecSysModel(nn.Module):
    def __init__(self, n_users, n_movies, n_factors=50):
        super(RecSysModel, self).__init__()
        self.user_factors = nn.Embedding(n_users, n_factors)
        self.movie_factors = nn.Embedding(n_movies, n_factors)
        self.output = nn.Linear(n_factors*2, 1) # n_factors = 100, 1
        
    def forward(self, user_id, movie_id):
        user_factors = self.user_factors(user_id)
        movie_factors = self.movie_factors(movie_id)
        out = torch.cat([user_factors, movie_factors], dim=1)
        out = self.output(out)
        return out
</code></pre>
<p>Now I want to use it in production for new users that are not in the dataset, so how should I take in a list of movies a new user has liked and recommend movies based on that?<br>
Thanks!</p>

arkh: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Any suggestions on this topic?<br>
Thanks!</p>

ptrblck: <p>Sorry, but I don’t know which approach would work for your use case.</p>

arkh: <p>No problem :).<br>
Thanks!</p>
MNIST image size

jun_suk_ha: <pre><code class="lang-auto">import torch
import torchvision
# !pip install torchvision
from torchvision import datasets, transforms
mnist2 =  torchvision.datasets.MNIST('mnist_data', train=True, download=True,
                        transform=transforms.Compose([
                            torchvision.transforms.Pad(18, fill=0, padding_mode='constant'),
                            transforms.ToTensor()
                        ]))
</code></pre>
<p>The following code block gives [28,28] for img2 shape.</p>
<pre><code class="lang-auto">import matplotlib.pyplot as plt
img2= mnist2.data[0]
print(img2.shape)
plt.imshow(img2)
plt.show()
</code></pre>
<p>The following one gives [1, 64, 64] for img shape.  Why are they different?</p>
<pre><code class="lang-auto">img, digit = next(iter(mnist2))
print(img.shape)
plt.imshow(img.squeeze())
plt.show()
</code></pre>

ptrblck: <p>The first code snippet uses the internal raw data which has a spatial shape of <code>28x28</code>.<br>
The second code snippet uses the <code>DataLoader</code> which adds the <code>Pad</code> transformation to each sample and thus increases the spatial shape to <code>28 + 18 + 18 = 64</code>.</p>
Deep double descent with ADAMW

mom333: <p>I hope this is an appropriate place to bring this up. I am using pytorch but it is more of a question related to the training methodology. Like I mentioned in the title I am trying to make a double descent network and I am using ADAMW to optimize my weights.</p>
<p>I managed to get to near 0 loss with average loss oscillating around 1e-4 / 1e-5. And I am somewhat stuck. Stuck because I keep cosine annealing with diminishing returns to the weight norm of all my weights and my network is still outputting garbage. Somewhat because I noticed that I can keep increasing weight decay above 1 and I am still having the loss reconverge towards 0 instead of network completely breaking down.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c696d09b25f513916e737e4aa5b40bc819684d0c.png" data-download-href="https://discuss.pytorch.org/uploads/default/c696d09b25f513916e737e4aa5b40bc819684d0c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c696d09b25f513916e737e4aa5b40bc819684d0c_2_690x372.png" alt="image" data-base62-sha1="skNEPjRlpm9UcGFut7K5AtwoZkw" width="690" height="372" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c696d09b25f513916e737e4aa5b40bc819684d0c_2_690x372.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c696d09b25f513916e737e4aa5b40bc819684d0c_2_1035x558.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c696d09b25f513916e737e4aa5b40bc819684d0c_2_1380x744.png 2x" data-dominant-color="F5F6F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1713×925 119 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I just wanted to confirm it, if I am understanding it correctly. I am thinking that loss convergence still being possible with weight decay above 1 is because at very small weight magnitude the denomiator of moving averages of my gradient has become very small and now moving averages part of ADAM algorithm dominates the update step? It is like the moving averages can still update and compensate instead of completely blowing up, which happens when the denomiator tends to 0, but they are also updating weights much more than current gradient update and weight decay.</p>
<p>That would also mean that for ADAMW the “optimal” weight decay for normalization at small weight magnitude is constantly moving? With that in mind I did try switching to pure SGD, but there even with a reasonable weight decay (at least compared to what I am doing now) of 0.3 my loss starts diverging instantly. Am I reasoning correctly here?</p>
CNN-LSTM performance identical to LSTM

jurc: <p>I am trying to recreate the models from a study in which CNN-LSTM outperformed LSTM, but my CNN-LSTM produces nearly identical results to the LSTM. So it seems like the addition of the convolutional layers is not doing anything. The study describes the CNN-LSTM model like this:</p>
<blockquote>
<p>The model is constructed by a single LSTM layer and two CNN layers. To form the CNN part, two 1D convolutional neural networks are stacked without any pooling layer. The second CNN layer is followed by a Rectified Linear Unit (ReLU) activation function. Each of the flattened output of the CNN’s ReLU layer and the LSTM layer is projected to the same dimension using a fully connected layer. Finally, a dropout layer is placed before the output layer.</p>
</blockquote>
<p>Did I make a mistake in the implementation? The results of my CNN-LSTM are almost exactly the same as when I use the LSTM on its own. The LSTM on its own is the exact same code as below, just without the two conv1d layers and without the ReLu activation function.</p>
<pre><code class="lang-auto">class CNN_LSTM(nn.Module):
    def __init__(self, input_size, seq_len, params, output_size):
        super(CNN_LSTM, self).__init__()
        self.n_hidden = params['lstm_hidden']        # neurons in each lstm layer
        self.seq_len = seq_len                       # length of the input sequence
        self.n_layers = 1                            # nr of recurrent layers in the lstm
        self.n_filters = params['n_filters']         # size of filter in cnn
        self.c1 = nn.Conv1d(in_channels=1, out_channels=params['n_filters'], kernel_size=1, stride=1) 
        self.c2 = nn.Conv1d(in_channels=params['n_filters'], out_channels=1, kernel_size=1, stride=1)
        self.lstm = nn.LSTM(
            input_size=input_size,         # nr of input features
            hidden_size=params['lstm_hidden'],       
            num_layers=1
        )
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(in_features=seq_len*params['lstm_hidden'],  out_features=params['dense_hidden'])
        self.dropout = nn.Dropout(p=.4)
        self.fc2 = nn.Linear(in_features=params['dense_hidden'], out_features=output_size) # output_size = nr of output features  

    def reset_hidden_state(self):
        self.hidden = (
            torch.zeros(self.n_layers, self.seq_len, self.n_hidden).to(device=device),
            torch.zeros(self.n_layers, self.seq_len, self.n_hidden).to(device=device),
        )

    def forward(self, sequences):
        out = self.c1(sequences.view(len(sequences), 1, -1))
        out = self.c2(out.view(len(out), self.n_filters, -1))
        out = F.relu(out)
        out, self.hidden = self.lstm(
            out.view(len(out), self.seq_len, -1),
            self.hidden
        )
        out = self.flatten(out)
        out = self.fc1(out)
        out = self.dropout(out)
        out = self.fc2(out)
        return out
</code></pre>
<p><a href="https://www.mdpi.com/1424-8220/22/3/1060/htm#B24-sensors-22-01060" rel="noopener nofollow ugc">Source for the study I am using.</a></p>
Is there a way to do multi-label classification with CLIP?

Johnny_Cash: <p>The concrete use case is a as following. I have the classes baby, child, teen, adult. My idea was to use similarity between text and image features (for text features I used the prompt ‘there is at least one (c) in the photo’, c being one of the 4 classes).</p>
<p>I went through quite a lot of examples, but I am running into the issue that the similarity scores are often very different for a fixed class or/and classes that appear might have a very similar threshold (like baby and child). For similarity scores I use the cosine similarity multiplied by 2.5 to stretch the score into the interval [0, 1] as is done in the CLIP Score paper.</p>
<p>Setting a threshold in that sense doesn’t seem possible.</p>
<p>Does anyone have an idea for that? I feel quite stuck here, how I should proceed.</p>
Not sure if this layer is DP-friendly

ege_b: <p>Hi, I do have an implementation of PG-GANs in hand and the discriminator of this model has a so-called Minibatch-STD layer. Opacus seems to validate the layer w/o any problems. However, what this layer actually does is that it introduces a new statistic as an extra dimension to every sample and this statistic is dependent on other samples on the batch.<br>
Ex.: when inputted tensor([[1,2,3], [3,4,5]]) it outputs tensor([[1,2,3, 1.4142], [3,4,5, 1.4142]]).</p>
<p>The layer seems like this:</p>
<pre><code class="lang-python">class Minibatch_std(nn.Module):
	def __init__(self):
		super().__init__()
	def forward(self, x):
		size = list(x.size())
		size[1] = 1
		
		std = torch.std(x, dim=0)
		mean = torch.mean(std)
		return torch.cat((x, mean.repeat(size)),dim=1)
</code></pre>
<p>and is used here:</p>
<pre><code class="lang-python">class D_Block(nn.Module):
	def __init__(self, in_ch, out_ch, initial_block=False):
		super().__init__()

		if initial_block:
			self.minibatchstd = Minibatch_std()
			self.conv1 = EqualizedLR_Conv2d(in_ch+1, out_ch, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
			self.conv2 = EqualizedLR_Conv2d(out_ch, out_ch, kernel_size=(4, 4), stride=(1, 1))
			self.outlayer = nn.Sequential(
									nn.Flatten(),
									nn.Linear(out_ch, 1)
									)
		else:			
			self.minibatchstd = None
			self.conv1 = EqualizedLR_Conv2d(in_ch, out_ch, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
			self.conv2 = EqualizedLR_Conv2d(out_ch, out_ch, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
			self.outlayer = nn.AvgPool2d(kernel_size=(2, 2), stride=(2, 2))

		self.relu = nn.LeakyReLU(0.2)
		nn.init.normal_(self.conv1.weight)
		nn.init.normal_(self.conv2.weight)
		nn.init.zeros_(self.conv1.bias)
		nn.init.zeros_(self.conv2.bias)
	
	def forward(self, x):
		if self.minibatchstd is not None:
			x = self.minibatchstd(x)
		
		x = self.conv1(x)
		x = self.relu(x)
		x = self.conv2(x)
		x = self.relu(x)
		x = self.outlayer(x)
		return x
</code></pre>
<p>Is this layer DP-friendly?</p>
<p>Note: the implementation is taken from <a href="https://github.com/Maggiking/PGGAN-PyTorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Maggiking/PGGAN-PyTorch: A pytorch implementation of Progressive Growing GAN.</a></p>

alexandresablayrolle: <p>Thanks for reporting this. I believe your concern is valid. When dealing with layers for which per-sample gradient is not implemented, Opacus computes it based on “best-effort” basis. Mixing information across a batch is definitely not DP-friendly unfortunately.</p>
How to solve my problem of max_step parameter?

test_project: <p>I’m trying to train a model.<br>
n_batch=2<br>
n_dataset=5000</p>
<pre><code class="lang-auto">class mymodel(pl.LightningModule):
  def __init__(self, config , learning_rate = 1e-4, max_steps = 100000//2):
    super(mymodel, self).__init__()
    
    self.config = config
    self.save_hyperparameters()
    self.training_losses = []
    self.validation_losses = []
    self.max_steps = max_steps

  def configure_optimizers(self):
    return torch.optim.AdamW(self.parameters(), lr = self.hparams['learning_rate'])


  def forward(self, batch_dict):
    return answer_vector

  def calculate_metrics(self, prediction, labels):
      batch_size = len(prediction)
      ac_score = 0

      for (pred, gt) in zip(prediction, labels):
        ac_score+= calculate_acc_score(pred.detach().cpu(), gt.detach().cpu())
      ac_score = ac_score/batch_size
      return ac_score

  def training_step(self, batch, batch_idx):
    answer_vector = self.forward(batch)
    loss = nn.CrossEntropyLoss()(answer_vector.reshape(-1,self.config['classes']), batch['answer'].reshape(-1))
    _, preds = torch.max(answer_vector, dim = -1)

    train_acc = self.calculate_metrics(preds, batch['answer'])
    train_acc = torch.tensor(train_acc)

    return loss

  def validation_step(self, batch, batch_idx):
    logits = self.forward(batch)
    loss = nn.CrossEntropyLoss()(logits.reshape(-1,self.config['classes']), batch['answer'].reshape(-1))
    _, preds = torch.max(logits, dim = -1)

    ## Validation Accuracy
    val_acc = self.calculate_metrics(preds.cpu(), batch['answer'].cpu())
    val_acc = torch.tensor(val_acc)

    ## Logging
    self.log('val_ce_loss', loss, prog_bar = True)
    self.log('val_acc', val_acc, prog_bar = True)
    
    return {'val_loss': loss, 'val_acc': val_acc}

  def optimizer_step(self, epoch_nb, batch_nb, optimizer, optimizer_i, opt_closure = None, on_tpu=False,
    using_native_amp=False, using_lbfgs=False):

        ## Warmup for 1000 steps
        if self.trainer.global_step &lt; 1000:
            lr_scale = min(1., float(self.trainer.global_step + 1) / 1000.)
            for pg in optimizer.param_groups:
                pg['lr'] = lr_scale * self.hparams.learning_rate

        ## Linear Decay
        else:
            for pg in optimizer.param_groups:
                pg['lr'] = polynomial(self.hparams.learning_rate, self.trainer.global_step, max_iter = self.max_steps)

        optimizer.step(opt_closure)
        optimizer.zero_grad()
</code></pre>
<p>In 5’th epoch (maybe less or more) I encountered that stop training. so I increase max_step . But when I increase max_step(max_step==100K) I have problem with loss and acc that loss&gt;100 &amp;&amp; acc==0 .I attach screen of this problem.</p>
Recommended way to move between CPU and GPU

aakash: <p>Moving model and data between cpu and gpu with <code>.to(device)</code> or <code>.cpu()</code> or <code>.cuda()</code> could make the code messy if not done properly.<br>
Are there any recommend guideline/standard practice to go about it?</p>
<p>What about moving data to gpu in the <code>forward(self, ...)</code> function of a network and returning results after moving them back to cpu?<br>
something along the lines of</p>
<pre><code class="lang-auto">    def forward(self ,x):
        x = x.cuda()
        x = self.fc(x)
        x = x.cpu()
        return x
</code></pre>
<p>Is this a good idea or are there reasons to not go in this direction?<br>
Thanks.</p>

ptrblck: <p>Moving or creating data inside the <code>forward</code> via <code>cuda()</code> might yield errors, if you are using <code>nn.DataParallel</code> or <code>DistributedDataParallel</code>, since the model will be copied to each specified device, while your <code>cuda()</code> call will move the data to the default device.<br>
Of course you could specify the device id by reading it from the attribute of an internal parameter e.g. via <code>x = x.to(self.param.device)</code>, but I would generally push the data to the device in the <code>DataLoader</code> loop.</p>
<p>I’m not sure, if you need to result on the CPU, but note that this call will be synchronizing your code.<br>
If you are training and thus need the output for the further loss calculation, I would just leave the output on the device.</p>

aakash: <p>Fair enough, that makes sense. Thanks!<br>
For my current situation I am playing around Reinforcement Learning algorithms and the inputs to the network could come from a variety of places including the gym and my replaybuffer…<br>
Also right now I am doing some numpy operations after the forward pass and before the loss calculation …so maybe changing operations to run without grad on torch tensors, instead of using numpy will be more elegant in this case …</p>

Pranjal_Agarwal: <p>I too need to run some numpy operations (specifically scipy operations) after forward pass, before computing loss. I did not understand " <em>so maybe changing operations to run without grad on torch tensors, instead of using numpy will be more elegant in this case</em> . Please explain.</p>
Correct way of retaining intermediate tensors with gradients?

conv8d: <p>What’s the correct way of doing the following loop?</p>
<pre><code class="lang-auto"># assume gradient is enabled for all tensors
b1_list, b2_list = [], []
for i in range(n):
    a1, a2 = some_function()
    b1, b2 = some_neural_net(a1, a2)
    b1_list.append(b1) # or b1_list.append(b1.clone()) ? or something else?
    b2_list.append(b2) # or b2_list.append(b2.clone()) ? or something else?
b1_tensor = torch.stack(b1_list)
b2_tensor = torch.stack(b2_list)
</code></pre>

eqy: <p>It doesn’t look like <code>b1</code> or <code>b2</code> are reused or here, so simply holding on to the references without the clone seems fine, unless there is another part of the code that is being omitted. The autograd graph should be kept around until <code>backward()</code> is called without calling <code>clone</code>.</p>

conv8d: <p>But <code>b1</code>, <code>b2</code> get updated in the next loop iteration. Is this an issue for gradients?<br>
Also <code>b1_tensor</code> and <code>b2_tensor</code> are used later to compute losses.</p>

eqy: <p>No, assignment of references in Python should affect the gradients e.g.,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.randn(10, requires_grad=True)
&gt;&gt;&gt; l = list()
&gt;&gt;&gt; a.prod().backward()
&gt;&gt;&gt; a.grad
tensor([ 0.0006, -0.0028,  0.0090,  0.0011, -0.0022,  0.0013, -0.0103,  0.0014,
        -0.0030, -0.0010])
&gt;&gt;&gt; a.data_ptr()
34399296
&gt;&gt;&gt; l.append(a)
&gt;&gt;&gt; a = torch.randn(10, requires_grad=True)
&gt;&gt;&gt; a.sum().backward()
&gt;&gt;&gt; a.grad
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
&gt;&gt;&gt; a.data_ptr()
76175680
&gt;&gt;&gt; l[0].data_ptr()
34399296
&gt;&gt;&gt; l[0].grad
tensor([ 0.0006, -0.0028,  0.0090,  0.0011, -0.0022,  0.0013, -0.0103,  0.0014,
        -0.0030, -0.0010])
&gt;&gt;&gt;
</code></pre>
The doc of `TORCH.TENSOR.SCATTER_` does not have this argument `value`

otakutyrant: <p>See <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.scatter_.html#torch-tensor-scatter" rel="noopener nofollow ugc">the doc</a> that lacks the note about <code>value</code> and <a href="https://pytorch.org/tutorials/beginner/basics/transforms_tutorial.html#lambda-transforms" rel="noopener nofollow ugc">this tutorial</a> where <code>value</code> appeared.</p>

ptrblck: <p>Thanks for raising the discrepancy in the docs. Would you be interested in fixing this issue?</p>

otakutyrant: <p>Okay. I’ll try to fix it.</p>
Why is "default_collate" not the DataLoader's default argument?

otakutyrant: <p><a href="https://stackoverflow.com/a/72867757/12080580" rel="noopener nofollow ugc">This answer</a> said DataLoader will use <code>default_collate</code> to handle Dataset’s return value implicitly when <code>collate_fn</code> is undefined, maybe. But <a href="https://stackoverflow.com/a/72867757/12080580" rel="noopener nofollow ugc">DataLoader</a>’s default argument for <code>collate_fn</code> is actually <code>None</code>, inducing developer hard to find out that DataLoader will use <code>default_collate</code> implicitly when <code>collate_fn</code> is undefined.</p>

ptrblck: <p>The <code>collate_fn</code> is either set to <code>detault_collate</code> or <code>default_convert</code> based on the internal attribute <code>self._auto_collation</code> as seen <a href="https://github.com/pytorch/pytorch/blob/39d49dbe4546c9f4d153ec748f62371e4546158a/torch/utils/data/dataloader.py#L366-L370">here</a> which is most likely the reason it’s set to <code>None</code>.</p>
Assert scalar.squeeze().ndim == 0, "scalar should be 0D" AssertionError: scalar should be 0D

Nora22: <p>Hi, I’m training a GNN model (encoder)  in Pytorch geometric with a custom loss function and I’m facing an issue while doing the training iteration for my model. The code is as below:</p>
<pre><code class="lang-auto">def train(dataset, epochs, criterion, writer):
    train_loader = DataLoader(dataset, batch_size=1, collate_fn=dataset.collate_fn, shuffle=False)

    hidden_features = 16
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    # build the model
    model = GNN(dataset.num_features, hidden_features=hidden_features)
    model = model.to(device)
    opt = torch.optim.Adam(model.parameters(), lr=0.0001)

    # train
    for epoch in range(1, epochs + 1):
        total_loss = 0
        model.train()
        for batch in train_loader:
            for data in batch:  
                data.cuda()

                # Zero your gradients for every batch!
                opt.zero_grad()

                output = model(x, edge_index)

                loss = criterion.forward(output, y)
                loss.sum().backward()

                opt.step()


                total_loss += loss
                writer.add_scalar("loss", total_loss, epoch)


            print("Epoch {}. Loss: {:.4f}".format(
                epoch, loss))


    return model
</code></pre>
<p>can you please tell me how to fix this and if the training process is correct?</p>

ptrblck: <p>I guess the error is raised in <code>writer.add_scalar("loss", total_loss, epoch)</code> which expects a scalar input while <code>total_loss</code> seems to contain multiple values.</p>

Nora22: <p>thanks, can you please tell me if the training process correct?</p>

ptrblck: <p>The code looks generally alright besides a few minor issues:</p>
<ul>
<li>During training you would usually shuffle the dataset while you are specifying <code>shuffle=False</code> in the <code>DataLoader</code>.</li>
<li>
<code>.cuda()</code> is not an inplace operation on tensors so you might need to use <code>data = data.cuda()</code> or <code>data = data.to(device)</code> in case <code>data</code> is a tensor.</li>
<li>You should not call the <code>forward</code> method directly but the object, so you might want to change <code>criterion.forward(...)</code> to <code>criterion(...)</code>.</li>
<li>You are storing the entire computation graph in <code>total_loss += loss</code> since <code>loss</code> is still attached to it. Use <code>total_loss += loss.detach()</code> to avoid increasing the memory in each iteration.</li>
</ul>
ViT Regression + parallel CNN embedding

sashapiv: <p>Hi everyone,</p>
<p>I need your help, I am trying to implement shared weights CNN instead of linear embedding and change from classification to regression model.</p>
<p>If you have any piece of advice I will appreciate it!!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b58c8ec7e0c583096916e64e0c56a6df44a731a1.png" data-download-href="https://discuss.pytorch.org/uploads/default/b58c8ec7e0c583096916e64e0c56a6df44a731a1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b58c8ec7e0c583096916e64e0c56a6df44a731a1_2_690x360.png" alt="image" data-base62-sha1="pU3z4BFPOxorphdrjmlQnxcSgvL" width="690" height="360" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b58c8ec7e0c583096916e64e0c56a6df44a731a1_2_690x360.png, https://discuss.pytorch.org/uploads/default/original/3X/b/5/b58c8ec7e0c583096916e64e0c56a6df44a731a1.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/5/b58c8ec7e0c583096916e64e0c56a6df44a731a1.png 2x" data-dominant-color="ECEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">934×488 88.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How does monobeast style multiprocessing work exactly? How should I integrate jit compilation?

Ryan_Keathley1: <p>I’m doing policy gradient with a modified monobeast. That means the actors are torch.mp spawned processes and the target function they run receives the actor net as an argument. Before this, <code>actor_net.share_memory()</code> is called.</p>
<p>I don’t understand what is really happening. I assume the actor’s weights only exist in one location in memory, but each worker process is mostly self contained and has its own python object representing the actor.<br>
So then multiple CPU threads are able to do inference in parallel? When performing inference, the threads are all reading from the exact same memory for the model’s params? Could this perhaps be a bottleneck depending on the number of workers. Also, the main process will occasionally update the actor’s params to match the learners <code>actor_model.load_state_dict(model.state_dict())</code> and it does so without any locking. How is this possible?<br>
Within each of my worker processes, I have multiple threads working that use the actor net. I feel safe knowing that the threads cannot try to access the net at the same time because of the GIL. However this is not true in the multiprocessing case, right.</p>
<p>Secondly, I tried <code>torch.jit.trace</code> on my custom net which uses some python iteration and I got a 33% speedup. I would like to use a compiled actor net. Would it be as simple as compiling, calling shared memory on the compiled net, and using that instead?</p>
<p>Edit: load_state_dict()'ing a compiled net and passing one (with torch.mp not mp) doesnt throw errors. It remains to be seen if it works identically</p>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [7616, 3, 3]], which is output 0 of AsStridedBackward0, is at version 12; expected version 11 instead

sb_romanm: <p>I’m running into issues during backprop on a function that calculates skeleton joint positions in 3D space from rotation matrices. This function computes the global position of one joint at a time.<br>
Implementation:</p>
<pre><code class="lang-auto">def torch_mat_to_pos(mat_rotations: torch.Tensor, parents: List[int], offsets: torch.Tensor) -&gt; torch.Tensor:
    """
    Convert rotation matrices to joint positions.

    Args:
        mat_rotations: Rotation matrices as tensor of shape (frames, joints, 3, 3).
        parents: Parents of each joint as tensor of shape (joints,).
        offsets: Offsets of each joint as tensor of shape (joints, 3).

    Returns:
        Joint positions as tensor of shape (frames, joints, 3).
    """
    global_rot = torch.clone(mat_rotations)
    global_pos = torch.zeros((mat_rotations.shape[0], mat_rotations.shape[1], 3),
                             dtype=mat_rotations.dtype, device=mat_rotations.device)
    for i, parent in enumerate(parents):
        if parent == -1:
            continue

        # multiply this joint's rotmat to the rotmat of its parent
        global_rot[:, i] = mat_rotations[:, i] @ global_rot[:, parent]

        k = offsets[i].repeat((mat_rotations.shape[0], 1, 1))
        # multiply the offsets by the parent's rotmat
        q = k @ global_rot[:, parent]
        # add the offsets to the parent's position
        global_pos[:, i] = global_pos[:, i] + torch.squeeze(q, 1)
    return global_pos
</code></pre>
<p>I’ve enabled anomaly detection, and it seems like the issue is in this line:<br>
<code> global_rot[:, i] = mat_rotations[:, i] @ global_rot[:, parent]</code></p>
<p>I tried creating global_rot with torch.zero_like and ended up with the same result.<br>
Also, note that I’m getting this error already at index 1 , right after this line (when testing backprop on  global_rot[:, I]).</p>
<p>Any suggestions / ideas on how to solve this? Thanks in advance!</p>

sb_romanm: <p>I think I found a solution that doesn’t require assignment.<br>
I had to create a separate function that calculates a joint’s global rotation matrix and call it each time I need a parent’s global rotation.<br>
calculating <code>q</code> would look like this:</p>
<pre><code class="lang-auto">parent_rot = _get_joint_global_rot_mat(parent, mat_rotations, parents)
q = k @ parent_rot
</code></pre>
<p>This way I’m not getting the in place modification error I was getting earlier. The caveat is that I’m recumputing the same multiplications multiple times, so it’s less efficient. Since I’m running this with torchscript it’s less of an issue.</p>
<p>I guess I didn’t realize how much more limited I am with the options I have to implement anything that runs through backprop.</p>
CUDA error from trying to load dataset with tqdm?

Sam_Lerman: <p><code>tqdm</code> is just a visual progress bar helper.</p>
<pre><code class="lang-auto">for i, (x, y) in enumerate(tqdm(self.mnist_dataset)):
       ...
</code></pre>
<p>That <code>enumerate tqdm</code> portion triggers this error:</p>
<pre><code class="lang-auto">File "/u/me/conda/envs/env/lib/python3.8/site-packages/torch/utils/data/_utils/pin_memory.py", line 34, in _pin_memory_loop
    data = pin_memory(data, device)                                            
  File "/u/me/conda/envs/env/lib/python3.8/site-packages/torch/utils/data/_utils/pin_memory.py", line 65, in pin_memory                                                                            
    return type(data)([pin_memory(sample, device) for sample in data])  # type: ignore[call-arg]                                                                                                       
  File "/u/me/conda/envs/env/lib/python3.8/site-packages/torch/utils/data/_utils/pin_memory.py", line 65, in &lt;listcomp&gt;         
    return type(data)([pin_memory(sample, device) for sample in data])  # type: ignore[call-arg]
  File "/u/me/conda/envs/env/lib/python3.8/site-packages/torch/utils/data/_utils/pin_memory.py", line 50, in pin_memory                                                                             
    return data.pin_memory(device)                                                                                                                                                                     
RuntimeError: CUDA error: out of memory
</code></pre>

ptrblck: <p>Is your code working fine without the usage of <code>tqdm</code>? If so, could you post a minimal, executable code snippet to reproduce the issue, please?</p>

Sam_Lerman: <p>Extremely minimal example:</p>
<pre><code class="lang-auto">from torch.utils.data import DataLoader
from torchvision.datasets.mnist import MNIST
from torchvision.transforms import ToTensor


dataset = MNIST('./', download=True, transform=ToTensor())

dataset = DataLoader(dataset=dataset,
                     pin_memory=True)  # pin_memory triggers CUDA error

for _ in dataset:
    continue
</code></pre>
<p>It seems the <code>pin_memory=true</code> is responsible. Running on school’s A6000 GPUs.</p>

ptrblck: <p>Thanks for the code snippet. I cannot reproduce it using <code>1.14.0.dev20221207+cu117</code> so you might need to update to  recent version in case you are using an older release.</p>
Gradient is not zero at position where input is masked

LILAKKU: <p>During DDP training, I prepend my input sequence with 23 pad tokens, and set their attention mask to 0. However, when I print the gradient of the hidden state of the 22th position (using zero-based indexing), it does not return 0, while gradients of the preceding 22 positions are all 0. Why is that?</p>
<p>Also, I noticed that the gradient of the hidden state corresponding to the last input token is also always 0, is there a reason for that?</p>
torch.nn.Parameter has None gradient

J_B_28: <p>Hello,</p>
<p>I am making a simple test on MLP to replace the normal weight with a customized residual version weight, I used torch.nn.Parameter to achieve that:</p>
<pre><code class="lang-auto">for i, lin in enumerate(self.lins[:-1]):
      if self.residual_weight and lin.weight.size(-1) == args.hidden:
         print("before", lin.weight.grad)
         lin.weight = torch.nn.Parameter(args.phi*torch.eye(args.hidden).to(device) + (1- 
                            args.phi)*lin.weight.clone(), requires_grad=True)
         print("after", lin.weight.grad)
      x = lin(x)
      x = self.bns[i](x)
      x = F.relu(x)
      x = F.dropout(x, p=args.dropout, training=self.training)
x = self.lins[-1](x)
</code></pre>
<p>The first print (“before”) statement will output the gradient normally, but the second one (“after”) will always output None, even I have set requires_grad=True, I am wondering what I should do to solve this problem to let the backward prop perform well. Thanks.</p>

ptrblck: <p>It seems you are recreating the parameter in the <code>forward</code> method without any gradient history so the <code>None</code> <code>.grad</code> attribute is expected in the <code>"after"</code> line of code.<br>
The <code>.grad</code> attribute will be populated in the <code>.backward</code> call of the loss.<br>
Note however, that your optimizer might be update this parameter unless you pass it as a new param group to it.</p>
How to retain leaf node status of a tensor when using .to(memory_format=torch.channels_last)

rinkujadhav2013: <p>I’m trying to create a random tensor with <code>memory_format=torch.channels_last</code>. I’m doing it as follows:</p>
<pre><code class="lang-auto">_X = torch.rand(3, 3, 70, 70, requires_grad=True, dtype=torch.float32, pin_memory=True)
X = _X.to(memory_format=torch.channels_last)
</code></pre>
<p>But I still get the <code>The .grad attribute of a Tensor that is not a leaf Tensor is being accessed.</code> warning and the <code>X.grad</code> is None after the backward pass. If I remove the 2nd line (and of course replace <code>_X</code> with <code>X</code> in the 1st one) then everything works as expected.</p>
<p>It’s same if I switch <code>_X</code> and <code>X</code> in the code above.</p>

ptrblck: <p>You can <code>.detach()</code> the tensor and call <code>.requires_grad_()</code> on the detached version to create a new leaf tensor:</p>
<pre><code class="lang-python">_X = torch.rand(3, 3, 70, 70, requires_grad=True, dtype=torch.float32, pin_memory=True)
X = _X.to(memory_format=torch.channels_last).detach()
X.requires_grad_()
print(X.is_leaf)
# True
</code></pre>
Custom rocm hip and c++ extensions

salykovaa: <p>Dear PyTorch developers and community,</p>
<p>We have nice tutorial <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">cpp_extension</a> on custom cuda extensions written by Peter Goldsborough. I’m wondering if the same can be done but on AMD GPUs with kernels written using rocm HIP. I mean the following: call custom forward+backward hip kernel from pytorch and include it in deep learning pipeline. Is it currently supported and are there any limitations?</p>
<p>Does somebody have experience of writing custom hip/c++ kernels and using them in pytorch?</p>
Very precise optimiser

wiiiktor: <p>I have an output vector with logits, like [ 0.123, 2.111, 3.222, -0.234, 0.234 ] and I want to push the 3.222 value down to 2.222, for the same input (so, the next output vector could look like [ 0.023, 1.345, 2.222, -0.234, -0.123 ]; other values can be any, but 2.222 must be there, ok, maybe with some slight error, like 2.224). What method can I use?</p>

J_Johnson: <p>Like this?</p>
<pre><code class="lang-auto">model_ouputs=model(data_batch)

targets=model_outputs.detach()
targets[:,2:3]=targets[:,2:3]-1

loss=optimizer(model_outputs, targets)

</code></pre>
<p>(Assumes dim=0 is batch dimension.)</p>

Viktor_Todosijevic: <p>Is this what you mean?</p>
<pre><code class="lang-auto">loss = MSELoss()

for input in dataloader:
    optimizer.zero_grad()
    your_logits_vector = model(input)
    l = loss(your_logits_vector, [0, 0, 2.222, 0, 0])
    l.backward()
    optimizer.step()

</code></pre>
How do we add "opset" to graph context

Srikant_Panda: <p>There are many references for opset in “torch/onnx/symbolic_helper.py”. For example in <strong>_slice_helper</strong> with <strong>g.opset &lt;= 9</strong>.</p>
<p>During ONNX conversion actual graph context that is received here is missing opset reference, so we get an error: <em>AttributeError: ‘torch._C.Graph’ object has no attribute ‘opset’</em>.</p>
<p>So is there any way to add “opset” to graph context.</p>
<p>Thanks in advance.</p>

ptrblck: <aside class="quote no-group" data-username="Srikant_Panda" data-post="1" data-topic="169277">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/srikant_panda/48/56048_2.png" class="avatar"> Srikant_Panda:</div>
<blockquote>
<p>so we get an error: <em>AttributeError: ‘torch._C.Graph’ object has no attribute ‘opset’</em>.</p>
</blockquote>
</aside>
<p><code>.opset</code> seems to be an attribute of <code>GraphContext</code>, not <code>Graph</code> as seen <a href="https://github.com/pytorch/pytorch/blob/62713636d8a65f313e47594ba686cc018d07cec4/torch/onnx/symbolic_helper.py#L684">here</a> and <a href="https://github.com/pytorch/pytorch/blob/02eb0bdbc1e48c43a224cdb5b57f4f7835983588/torch/onnx/_internal/jit_utils.py#L22-L43">here</a>. Internally the <code>Graph</code> is another attribute as described <a href="https://github.com/pytorch/pytorch/blob/c42a408baa65b37b459d538b440313f67c7c1cb7/torch/onnx/README.md#extra-context-for-symbolic-functions">here</a>.</p>

Srikant_Panda: <p>aha. I got that. I realized this was with using MMCV 1.7. And they also have an update for this issue with the upcoming version. Thank you for the clarification.</p>