akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to get a list of every tensor in the autograd tree?

omorrison1: <p>Hello,<br>
I am trying to train a rather large pytorch model. Training runs just fine if I use a batch_size of 1, but when using larger batches, the back propagation step fails (see error message below). Unfortunately, I have found it difficult to track down which tensor is causing the problem. My attempts to track down this tensor have been unsuccessful, so far. I have checked every tensor I can find in the code base and I have yet to even find a tensor that has 16 elements.</p>
<p>Is there a more systematic approach that I can take here? Surely, the autograd system must store some sort of reference to the relevant tensors? If I could view a record of all the tensors in the autograd tree, then perhaps I could narrow down my search based on the values in found in the relevant tensor?</p>
<p>Or perhaps there is some other approach that I should take?</p>
<p>Any help would be strongly appreciated.</p>
<pre><code class="lang-auto"> -------ERROR--------  
  one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.DoubleTensor [16]] is at version 11; expected version 10 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!  
  --------------------------- 

</code></pre>

AlphaBetaGamma96: <p>Have you use the <code>torch.autograd.set_detect_anomaly</code> function? <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.set_detect_anomaly" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 2.3 documentation</a></p>
<p>Can you share the model?</p>
<aside class="quote no-group" data-username="omorrison1" data-post="1" data-topic="204240">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/omorrison1/48/70586_2.png" class="avatar"> omorrison1:</div>
<blockquote>
<p>Surely, the autograd system must store some sort of reference to the relevant tensors? If I could view a record of all the tensors in the autograd tree</p>
</blockquote>
</aside>
<p>You could try plotting out your network with <code>torchviz</code>? <a href="https://github.com/szagoruyko/pytorchviz" class="inline-onebox" rel="noopener nofollow ugc">GitHub - szagoruyko/pytorchviz: A small package to create visualizations of PyTorch execution graphs</a></p>

omorrison1: <p>Problem solved:</p>
<p>I was resetting parameters via the overall model (DDP), instead of the individual modules (model.module).</p>
<p>Consequently, the parameters were being reset by each GPU. This resulted in the version number of the tensors being incremented more than expected.</p>
<p>Not sure how to mark this thread as solved.</p>
<p>Thank you for the advice!</p>
Pytorch: Matrix multiplication (@) with max instead of sum

yclin: <p>Is there a way in Pytorch to do the following (or is there a general mathematical term for this):<br>
Assume normal matrix multiplication (torch.mm):<br>
M3[i,k] = sum_j(M1[i,j] * M2[j,k])    size: M1: a×b; M2 b× c<br>
Now I would like to replace the sum by max :<br>
M3[i,k] = max_j(M1[i,j] * M2[j,k])<br>
As you can see it is completely parallel to the above, just we take  <code>max</code>  over all  <code>j</code>  and not the sum.<br>
NOTE: the size of M1 and M2 can be extremely large (e.g. M1 of size: 128×10000 or in this form (2×64)×(100×100)). In this case,  a 'for-loop is not acceptable because of running time. Considering the memory issue, we can NOT do (M1.unsqueeze(2)*M2.unsuqeeze(0)).max(dim=1) either, a method mentioned in the ref link.<br>
Ref link: <a href="https://stackoverflow.com/questions/41164305/numpy-dot-product-with-max-instead-of-sum" rel="nofollow noopener">https://stackoverflow.com/questions/41164305/numpy-dot-product-with-max-instead-of-sum</a></p>
<p>Thanks in advance!</p>

tom: <p>There are many examples for “matmul in cuda” (e.g. from NVidia but also others), you could write your own kernel. To get the derivative, I’d recommend writing a function that keeps the indices of the maxima (similar to <code>torch.max</code> with dimension) and use the corresponding lookup in the backward (you might be able to use “native” PyTorch for that.</p>
<p>Best regards</p>
<p>Thomas</p>

Collin_Indeed: <p><a class="mention" href="/u/yclin">@yclin</a></p>
<p>Did you find a solution to this in pytorch?</p>
F.conv2D_dtype problem

Reginald_Li: <p>Problem: F.conv2d(), input dtype is bfloat16, dtype of weight and bias are also bfloat16, but the output is float16<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bcba4d5197f3daf2d5e7178785a7cb0b133cc6d.png" data-download-href="https://discuss.pytorch.org/uploads/default/1bcba4d5197f3daf2d5e7178785a7cb0b133cc6d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bcba4d5197f3daf2d5e7178785a7cb0b133cc6d.png" alt="image" data-base62-sha1="3XTcc64suS62699pkq0vp7UA80d" width="690" height="57" data-dominant-color="1B1B1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1909×160 9.93 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>code:<br>
def _conv_forward(self, input: Tensor, weight: Tensor, bias: Optional[Tensor]):<br>
# print(f’pytorch forward 2 = {input}‘)<br>
print(f’padding mode = {self.padding_mode}’)<br>
if self.padding_mode != ‘zeros’:<br>
return F.conv2d(F.pad(input, self._reversed_padding_repeated_twice, mode=self.padding_mode),<br>
weight, bias, self.stride,<br>
_pair(0), self.dilation, self.groups)</p>
<pre><code>    print(f'---------------------================================')
    # print(f'input = {input}, weight = {weight}, bias = {bias}, stride = {self.stride}, padding = {self.padding}, dilation = {self.dilation}, groups = {self.groups}')
    print(f'stride = {self.stride}, padding = {self.padding}, dilation = {self.dilation}, groups = {self.groups}')
    if torch.isnan(input).any() or torch.isinf(input).any():
        raise ValueError("input contain NaN or Inf values")
    if torch.isnan(weight).any() or torch.isinf(weight).any():
        raise ValueError("weight contain NaN or Inf values")
    print(f'[$$ debug dtype]: input_dtype_shape = {input.dtype}_{input.shape}, \
          weight_dtype_shape = {weight.dtype}_{weight.shape}, \
          bias_dtype_shape = {bias.dtype}_{bias.shape}')
    return F.conv2d(input, weight, bias, self.stride,
                    self.padding, self.dilation, self.groups)

def forward(self, input: Tensor) -&gt; Tensor:
    temp_v = self._conv_forward(input, self.weight, self.bias)
    print(f'[pytorch debut --- temp_v.dtype] = {temp_v.dtype}_{temp_v.shape}') # float16
    return temp_v
</code></pre>
<p>torch version: torch 2.3.0+cu121<br>
device: Nivida4090</p>

ptrblck: <p>I cannot reproduce the issue using <code>torch==2.3.1+cu121</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.3.1+cu121'
&gt;&gt;&gt; x = torch.randn(1, 3, 224, 224, device="cuda", dtype=torch.bfloat16)
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; conv = nn.Conv2d(3, 3, 3, 1, 1).cuda().bfloat16()
&gt;&gt;&gt; out = conv(x)
&gt;&gt;&gt; print(out.dtype)
torch.bfloat16
</code></pre>
Inconsistent outputs between PyTorch versions

tpjb: <p>Hello!</p>
<p>I ran into an interesting inconsistency a few days ago. While preparing my environment to switch from torch 1.13 + cuda 11.7 to torch 2.1.2 + cuda 12.1, I double checked the outputs of a default resnet34 with IMAGENET1K_V1 weights in both PyTorch versions. On a torch.zeros(1, 3, 224, 224) as input, the exact same models produce slightly different values for all 1000 outputs. This is also the case for models that will be subdued to a PyTorch version switch in a production setting.</p>
<p>Now, I am wondering: Is this expected behaviour?</p>

soulitzer: <p>This should be expected yes. There’s no guarantee of bitwise consistency between versions. Slightly different values can happen if implementation details such as the order of operations is changed, etc.</p>
Best architecture for clustering

Neskelogth: <p>I am currently working on a task in which I have to perform unsupervised clustering using a neural network.</p>
<p>My architecture involves an autoencoder to derive a latent space representation for the input classes and a clustering module that should cluster the latent representations obtained with the autoencoder. I was thinking that it should be quite easy to have a clustering module based on Neural Networks and optimized end-to-end but I did not manage to find any implementation of something like this, so how can I approach this problem?</p>
<p>If this is a stupid question I’m sorry but I’m new to unsupervised learning and clustering</p>
Large dataset (HDF5) with concurrent reads - and buffer?

slvnwhrl: <p>Dear all,</p>
<p>I’m currently building a large textual dataset which will grow to tens of millions of text objects (i.e., rows in my dataset) pretty soon. My goal, among other things, is to apply neural topic modelling. I have access to GPUs, however, the whole dataset won’t fit into memory… so I need to come up with an efficient and effective solution for training.</p>
<p>My current plan:</p>
<ul>
<li>
<p>Store the data in single HDF5 file.</p>
</li>
<li>
<p>HDF5 allows concurrent reads so I can use PyTorch’s DataLoader with multiple workers to split the workload.</p>
</li>
<li>
<p>I’d also want to load random batches from the dataset which should be possible with HDF5… will still have to evaluate reading speed implications, though.</p>
</li>
<li>
<p>There has been some discussion in this forum around this topic (e.g., <a href="https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643">DataLoader, when num_worker &gt;0, there is bug</a>) which is already pretty helpful!</p>
</li>
<li>
<p>However, what I haven’t really found out so far how I could possibly combine this with some sort of buffer: In principal, my idea would be to load x number of batches (with random samples from the dataset) into memory (the buffer) to minimize reading from the file. I think that the <a href="https://pytorch.org/data/main/generated/torchdata.datapipes.iter.Sampler.html#torchdata.datapipes.iter.Sampler" rel="noopener nofollow ugc">Sampler DataPipe</a> from PyTorch goes into this direction… but I don’t really need an iterative solution (I want to be able to store random random samples while making sure that I don’t sample the same random samples across workers…).</p>
</li>
</ul>
<p>Has anyone ever implemented something like that? Possibly with public code? Any thoughts are appreciated, thanks!</p>

wait_for_it: <p>Hey hey, I have a similar problem, did you find a solution to your problem.</p>
<p>Regards</p>
Can I use libtorch1.6 in cuda9.0

zhuohengX: <p>I need to used libtorch in cuda9.0,and but didn’t find official version.And I try <a href="https://download" rel="noopener nofollow ugc">https://download</a>.<a href="https://so.csdn.net/so/search?q=pytorch&amp;spm=1001.2101.3001.7020" rel="noopener nofollow ugc">pytorch</a>.org/libtorch/cu92/libtorch-cxx11-abi-shared-with-deps-1.4.0%2Bcu92.zip.But it didn’t work.</p>
Self-adaptive loss weights

Joeyoung: <p>I want to implement self-adaptive loss weights (let’s say lambda) instead of constant lambda for all loss data.<br>
i.e.<br>
Instead of Total_loss = lambda1 * sigma( loss_i )_dataset1 + lambda2 * sigma( loss_i )_dataset2,<br>
Total_loss = sigma( lambda1_i * loss_i )_dataset1 + sigma( lambda2_i * loss_i )_dataset2</p>
<p>Is there anyway to use the built-in Adam optimizer to optimize lambda to maximize total loss?<br>
I implemented it by adopting Adam algorithm and it worked well for single batch or single GPU card. However, it cannot handle the multi batches or multi GPU cards.</p>
Optimizing PyTorch Inference Pipeline with TensorRT: Parallel Processing and Memory Management Questions

Heun: <p>Hi,</p>
<p>I’m trying to optimize a PyTorch inference pipeline for a segmentation model running in TensorRT. The general setup involves multiple producers reading images from Redis, preprocessing them, and feeding them to a TensorRT model. Then, multiple consumers will postprocess the output of the TensorRT model.</p>
<p>To maximize parallelism in the pre- and post-processing stages, I created multiple workers using the <code>torch.multiprocessing</code> module. My question now is how to optimize the data transfers from preprocessing to the TensorRT model, and from the model to postprocessing.</p>
<p>Here is the general setup with multiple pre- and post-processing workers, but only one inference worker:</p>
<pre><code class="lang-auto">import torch
import redis
import tensorrt as trt

input_queue = torch.multiprocessing.Queue()
output_queue = torch.multiprocessing.Queue()

def preprocessing_worker(input_queue):
    stream = torch.cuda.Stream()
    while True:
        with torch.cuda.stream(stream):
            image = redis.get()  # Fetch image from Redis
            preprocessed_image = preprocess(image).pin_memory()  # Preprocess and pin memory
            cuda_preprocessed_image = preprocessed_image.cuda(non_blocking=True)  # Copy to device
            stream.synchronize()
            input_queue.put(cuda_preprocessed_image)

def inference_worker(input_queue, output_queue):
    stream = torch.cuda.Stream()
    trtContext = ...  # Initialize TensorRT context
    while True:
        with torch.cuda.stream(stream), trtContext:
            input_image = input_queue.get()
            output = trtContext.execute_async_v2(input_image)  # Execute model
            stream.synchronize()
            output_queue.put(output)

def postprocessing_worker(output_queue):
    stream = torch.cuda.Stream()
    while True:
        with torch.cuda.stream(stream):
            output = output_queue.get()
            output = output.cpu()  # Copy to host
            postprocess_output(output)  # Postprocess output
</code></pre>
<p>My questions are:</p>
<ol>
<li>To enable parallel HtoD (Host to Device), DtoH (Device to Host) memory copies, and kernel executions, I have created separate CUDA streams for each worker. Is this necessary?</li>
<li>If separate streams are needed, is it mandatory to synchronize the stream after a <code>.cuda()</code> call before putting the data into the queue?</li>
<li>Does this method generate memory leaks if I don’t manually delete the data from the queues?</li>
</ol>
<p>Thank you in advance for your help!</p>
After torch.utils.data.random_split, len function and shape attribute gives different values

waliston: <p>I used</p>
<pre><code class="lang-auto">train_dataset, val_dataset = torch.utils.data.random_split(dataset=train_val_dataset, lengths=[train_size, val_size])
</code></pre>
<p>Then I verified:</p>
<pre><code class="lang-auto">len(train_dataset)
</code></pre>
<p>and it gives 54000.</p>
<p>But when I verified:</p>
<pre><code class="lang-auto">train_dataset.dataset.data.shape
</code></pre>
<p>it gives 60000.</p>
<p>Why would it happen?</p>
<p>thanks</p>

ptrblck: <p><code>random_split</code> creates <code>Subset</code>s, which use their internal indices to sample from the original dataset as seen <a href="https://github.com/pytorch/pytorch/blob/f8d60e0e0a4420def0cf6b3bc0a0c0d46c93de1c/torch/utils/data/dataset.py#L393">here</a>. You are accessing the internal and original dataset, which was not manipulated.</p>
Python Version - Backwards Version Compatibility Thought

BryRNCloudGuy: <p>Apologize as a nube and Not sure what category this belongs under but my first exposure torch has been a burdensome one . . . Regardless as complaining is pointless on the quest to a solution. . .</p>
<blockquote>
<p>Blockquote</p>
</blockquote>
<p>Can some help advise as to the best route forward in addressing this ideally one that doesn’t require me to install an older version of python ?</p>
<p>I am getting this error:</p>
<p>ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python &gt;=3.7,&lt;3.11; 1.21.3 Requires-Python &gt;=3.7,&lt;3.11; 1.21.4 Requires-Python &gt;=3.7,&lt;3.11; 1.21.5 Requires-Python &gt;=3.7,&lt;3.11; 1.21.6 Requires-Python &gt;=3.7,&lt;3.11<br>
ERROR: Could not find a version that satisfies the requirement torch&lt;2,&gt;=1.11.0 (from piper-train) (from versions: 2.2.0, 2.2.1, 2.2.2)<br>
ERROR: No matching distribution found for torch&lt;2,&gt;=1.11.0</p>

ptrblck: <p>I don’t know which Python version you are using but the binaries should are available for Python 3.8 - 3.12 (at least for Linux). If you are using an older or newer Python version you could try to build PyTorch from source using it as it seems you don’t want to change your Python version for some reason.</p>
How to combine additional features into word embeddings

tnji: <p>This familiar with my ideas.</p><aside class="onebox stackexchange" data-onebox-src="https://datascience.stackexchange.com/questions/22177/how-to-use-embedding-with-3d-tensor-in-keras">
  <header class="source">

      <a href="https://datascience.stackexchange.com/questions/22177/how-to-use-embedding-with-3d-tensor-in-keras" target="_blank" rel="noopener nofollow ugc">datascience.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://datascience.stackexchange.com/users/37814/offchan" target="_blank" rel="noopener nofollow ugc">
    <img alt="offchan" src="https://www.gravatar.com/avatar/3d12195be3c7acaadb23e9685d56e0d0?s=256&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://datascience.stackexchange.com/questions/22177/how-to-use-embedding-with-3d-tensor-in-keras" target="_blank" rel="noopener nofollow ugc">How to use Embedding() with 3D tensor in Keras?</a>
</h4>

<div class="tags">
  <strong>python, tensorflow, keras, rnn</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://datascience.stackexchange.com/users/37814/offchan" target="_blank" rel="noopener nofollow ugc">
    offchan
  </a>
  on <a href="https://datascience.stackexchange.com/questions/22177/how-to-use-embedding-with-3d-tensor-in-keras" target="_blank" rel="noopener nofollow ugc">11:20AM - 11 Aug 17 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I’m re-implementing some table-to-text papers using RNN-based seq2seq (like this one <a href="https://arxiv.org/pdf/1603.07771v3" rel="noopener nofollow ugc">https://arxiv.org/pdf/1603.07771v3</a>)</p>
<p>But I get trouble in understanding the representation of the input. The inside is for each word token embedding (slot values) concatenate with additional information of the field name. But the field name has several tokens (e.g day of birth) so it would be 3D input pass to Embedding layer. but word token is 2D input.</p>
<p>For example:</p>
<p><code>"George Mikell .... 4 April..."</code> let say 10 tokens are slot values</p>
<p>and a sequence of corresponding field name (actually i’m not sure this is the meaning of the authors):</p>
<p><code>name, name, ..., day of birth, day of birth...</code>   Okay this is confused part. As the field name has several tokens. If I express it it would be (10, l) which l is max token for a filed, let say 3. → (10, 3)</p>
<p>Okay in the forward phase, let say batch size = 1.</p>
<p>So slot values: (1,10)<br>
field values: (1,10,3)</p>
<p>then after embedding slot embedding → (1, 10, 64)<br>
field embeddings → (1, 10, 3, 32)</p>
<p>Then how can I combine those feature?</p>
<p>Thank you</p>
Dispatch ops flow and dtensor

schinmayee: <p>Can someone please explain how the control flow for ops works: from python to how it gets lowered into C++/cuda and back? How does this work for dtensors? Also, where are the ops for DTensors defined and is there documentation for it?</p>

soulitzer: <p>This might be helpful <a href="http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/" class="inline-onebox" rel="noopener nofollow ugc">Let’s talk about the PyTorch dispatcher : ezyang’s blog</a></p>
<p>Subclasses like DTensor plug into the PyTorch dispatcher at the Python key.</p>
<p>For DTensor specifically, further dispatching is defined in <code>torch/distributed/_tensor/dispatch.py</code>.</p>
How to do epoch dependent calculation in loss function for CUDAGraph

Joeyoung: <p>Hi,</p>
<p>I have a loss function to calculate loss dependent on epoch, e.g.</p>
<p>optimizer.zero_grad()<br>
forward_func()<br>
loss = loss_func( epoch )<br>
loss.backward()</p>
<p>When trying to use CUDAGraph, epoch is never changed during replaying after stream capture.<br>
(No use of CUDAGraph works well.)<br>
Any idea about how to pass epoch during CUDAGraph replaying?</p>
<p>Thanks,<br>
Joe</p>

ptrblck: <p>Try to pass it as a tensor.</p>

Joeyoung: <p>Thank you. It works.<br>
I realized that the argument of the function in captured stream shouldn’t be changed during replaying. So, the static address like torch tensor (during replaying) must be given.</p>
CPU torch.randn for fp16 is slower than casting from fp32

koskoakos: <p>(Only on CPU)<br>
Initializing a tensor directly as float16 or bfloat16 with normal distribution is slower than converting from float32:</p>
<pre><code class="lang-auto">python -m timeit -s "import torch" "torch.randn(10, 100, 100, 1000, dtype=torch.float32)"
1 loop, best of 5: 680 msec per loop

python -m timeit -s "import torch" "torch.randn(10, 100, 100, 1000, dtype=torch.bfloat16)"
1 loop, best of 5: 1.57 sec per loop

python -m timeit -s "import torch" "torch.randn(10, 100, 100, 1000, dtype=torch.float32).to(torch.bfloat16)"
1 loop, best of 5: 744 msec per loop
</code></pre>
<p>Is there a convenient way to profile it deeper? Also, not sure what exactly is happening <a href="https://github.com/pytorch/pytorch/blob/6079c5091091d872b8dafbaa4e31a5b6194647ad/aten/src/ATen/native/Distributions.cpp#L272" rel="noopener nofollow ugc">in Distribution.cpp</a><br>
Same for init.xavier_normal_, but uniform is fine.<br>
Or maybe should I just not init models in fp16 or bfloat16 anyways?</p>
CUDA runs out of memory after some steps

ThibaultDECO: <p>I have the following code, but CUDA keeps running out of memory after a few steps (can be 3 steps, can be 50 steps). Is there something I’m doing wrong?</p>
<pre><code class="lang-auto">import torch
from torch import Tensor
from torch.nn import BCELoss
from torch.optim import Optimizer
from torch.utils.data import DataLoader

def _evaluate_single_batch(self, batch: tuple[Tensor]) -&gt; Tensor:
    input_ids = batch[0]
    attention_mask = batch[1]
    number_of_chunks = [len(x) for x in input_ids]

    input_ids_combined = []
    for x in input_ids:
        input_ids_combined.extend(x.tolist())
    input_ids_combined_tensors = torch.stack([torch.tensor(x) for x in input_ids_combined]).to(self.device)

    attention_mask_combined = []
    for x in attention_mask:
        attention_mask_combined.extend(x.tolist())
    attention_mask_combined_tensors = torch.stack([torch.tensor(x) for x in attention_mask_combined]).to(self.device)

    preds_out = self.neural_network(input_ids_combined_tensors, attention_mask_combined_tensors)
    preds = preds_out.flatten().cpu()
    del input_ids_combined_tensors, attention_mask_combined_tensors, preds_out
    torch.cuda.empty_cache()

    preds_split = preds.split(number_of_chunks)
    pooled_preds = torch.cat([torch.mean(x).reshape(1) for x in preds_split])

    return pooled_preds

def _train_single_epoch(self, dataloader: DataLoader, optimizer: Optimizer) -&gt; None:
    self.neural_network.train()
    cross_entropy = BCELoss()

    for step, batch in enumerate(dataloader):

        labels = batch[-1].float().cpu()
        predictions = self._evaluate_single_batch(batch)
        loss = cross_entropy(predictions, labels) / self.accumulation_steps
        loss.backward()

        if ((step + 1) % self.accumulation_steps == 0) or (step + 1 == len(dataloader)):
            optimizer.step()
            optimizer.zero_grad()
            torch.cuda.empty_cache()

        del batch, labels, predictions, loss
</code></pre>

ptrblck: <p>You could check the shape of all input batches and see if the OOM error is raised for a specifically large input.</p>
Theoretical equivalency of weights between Single and Multi-GPUs

shahnawaz: <p>If we split a batch of data into multiple gpus, will it be mathematically equivalent (in terms of learned weights) to using whole batch in single gpu?</p>
<p>Following are the assumptions for both experiments</p>
<ol>
<li>using same seed</li>
<li>no batch norm</li>
<li>exactly same data in the batch</li>
<li>for the sake of simplicity, let assume we are doing just single iteration step</li>
<li>keeping the learning rate constant to 1</li>
</ol>
<p>I read multiple places that they should be mathematically equal. But from my calculation below, I am not able to proof it</p>
<p>Let the model be, F(X) = w * X<br>
Loss(X, F) = sum(F(X) - Const)<br>
and aggregation function for multiple gpus synchronization is Mean function</p>
<p>Experiment 1: [batch_size=2, num_of_gpu=1]<br>
Input: X = [x1, x2]<br>
loss = w*(x1 + x2) - 2*Const<br>
grad_single_gpu = grad(loss, w) = d(loss)/dw = x1+x2</p>
<p>Experiment 2: [batch_size=1, num_of_gpu=2]<br>
GPU_0<br>
Input: X = [x1]<br>
loss = (w * x1) - Const<br>
grad_gpu_0 = grad(loss, w) = d(loss)/dw = x1</p>
<p>GPU_1<br>
Input: X = [x2]<br>
loss = (w * x2) - Const<br>
grad_gpu_1 = grad(loss, w) = d(loss)/dw = x2</p>
<p>Now, the gradient synchronization will average the gradients from both gpus, so effective gradient = (grad_gpu_0 + grad_gpu_1) / 2 = (x1+x2)/2, which is not equivalent to the experiment 1’s (x1+x2).</p>
<p>Am I missing something?</p>
<p>Moreover, in real training, the batches are selected at random, models are non-linear, learning rate is cosine-annealing. Wouldn’t these factors create even higher differences?</p>

ptrblck: <aside class="quote no-group" data-username="shahnawaz" data-post="1" data-topic="204612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shahnawaz/48/16458_2.png" class="avatar"> shahnawaz:</div>
<blockquote>
<p>Am I missing something?</p>
</blockquote>
</aside>
<p>Yes, the default loss reduction is computing the mean by summing the gradient and dividing by the batch size. Assuming the batch size is equal on all GPUs, the result will be the same.</p>

shahnawaz: <p>Got it. Thanks. Does the same applies to multi-node training as-well?</p>
Random quantization

chumpro: <p>I have come up with something I call NonScalarFilter. It is basically random quantization.<br>
It is not used for the purpose of making the model smaller. It is used in a similar way as noising.</p>
<pre><code>def nonscalarfilter( tensor, r1 = 0.0, r2 = 100.0 ) :

	precision = random.uniform( r1, r2 )

	return tensor.mul( precision ).round().div( precision )
</code></pre>
<p>NonScalarFilter is a mechnism similar to noising in that removes information.</p>
<p>The idea is that by removing information during training of a neural network, one forces the neural network to learn to recreate that information.<br>
Noising, however, has a local minimum; avaraging, blurring, the pixels of a noisy image is a reasonable denoising mechaism.</p>
<p>NonScalarFilter has no such local minimum. When using NonScalarFilter during training of a neural network, one can therefore improve it much further.</p>
<p>Upscalers, autoencoders and denoisers are examples of neural networks that can benefit from NonScalarFilter.<br>
It is only used during training and it is placed in the information flow as follows:</p>
<pre><code>input_image -&gt; nonscalarfilter -&gt; encoder -&gt; decoder -&gt; output_image
</code></pre>
<p>or</p>
<pre><code>input_image -&gt; encoder -&gt; nonscalarfilter -&gt; decoder -&gt; output_image
</code></pre>
<p>or even</p>
<pre><code>input_image -&gt; nonscalarfilter -&gt; encoder -&gt; nonscalarfilter -&gt; decoder -&gt; output_image
</code></pre>
<p>The resulting neural network will then learn to create information without having to deal with the local minimum of a denoiser blurring the image.</p>

chumpro: <p>I just wanted to add:</p>
<p>NonScalarFilter forces the neural network to look further and take detail information with a pinch of salt. As a result, the network learns to look at the big picture and produce details.</p>
<p>I have used this years ago to improve DeepFill2.</p>
SequenceParallel sharding is wrong?

Diagnosta_23: <p>According to the documentation <code>torch.distributed.tensor.parallel.SequenceParallel</code> is supposed to shard on the sequence dimension i.e. <code>[B, T, C] -&gt; [B, T//_world_size, C]</code> but it seems to be <em>tiling</em> instead i.e. <code>[B, T, C] -&gt; [B, T*_world_size, C]</code>. What am I missing?</p>
Does torch_tensorrt / tensorrt support bfloat16?

Vedant_Roy: <p>I’m getting the following error:</p>
<pre><code class="lang-auto">AttributeError: module 'tensorrt' has no attribute 'bf16'

While executing %clone_1 : [num_users=1] = call_function[target=torch.ops.aten.clone.default](args = (%expand,), kwargs = {memory_format: torch.contiguous_format, _itensor_to_tensor_meta: {&lt;tensorrt_bindings.tensorrt.ITensor object at 0x1544e7d99af0&gt;: ((6, 37, 192, 336, 15), torch.bfloat16, False, (35804160, 64512, 336, 1, 2386944), None, False, {}), &lt;tensorrt_bindings.tensorrt.ITensor object at 0x1544e7d996b0&gt;: ((6, 37, 192, 336, 15), torch.bfloat16, False, (35804160, 64512, 336, 1, 2386944), None, False, {})}})
</code></pre>
<p>When using <code>torch.compile</code> with the tensorrt backend:</p>
<pre><code class="lang-auto">        import torch_tensorrt
        my_model = torch.compile(my_model, mode="reduce-overhead", fullgraph=true, backend="tensorrt")
</code></pre>
How to Add New Classes to a Model that Already Has a Certain Number of Classes

Vadym_Danylchuk: <p>I have a SequenceForClassification model, and after training a pre-trained BERT model on 44 classes, I want to add 20 more classes and train it again. I freeze all the layers except the classification layer before training. However, when I change the classification layer to accommodate the new number of classes, I always get the following error:<br>
I am new to data science, so I might be using some terms incorrectly. Here is the piece of code I am currently using:</p>
<pre><code class="lang-auto">model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased', 44)

model.classifier = nn.Linear(model.bert.config.hidden_size, 66)

with torch.no_grad():
    outputs = model(input_ids = b["input_ids"].to(device),
          attention_mask = b["attention_mask"].to(device),
          labels = b["labels"].to(device))
</code></pre>
<p><code>RuntimeError: shape '[-1, 43]' is invalid for input of size 660</code></p>
<p>How can I fix this issue?</p>

soulitzer: <p>The stack trace should tell you what line/function is failing, then look at the other inputs of that function to see where the original output size is still being used.</p>
Region Proposal Network Question

Timx: <p>Good day guys,</p>
<p>I did read about RPN and looked at the source code and found these 2 fnc that I didn’t understand, what purpose do they serve? And why we should ever consider using them? Another question is that batch_size_per_image; is it meant to be the number of the anchors used on the image or the batch size of my data?</p>
<pre><code class="lang-auto">        batch_size_per_image (int): number of anchors that are sampled during training of the RPN
                    for computing the loss
        pre_nms_top_n (Dict[str, int]): number of proposals to keep before applying NMS. It should
            contain two fields: training and testing, to allow for different values depending
            on training or evaluation
        post_nms_top_n (Dict[str, int]): number of proposals to keep after applying NMS. It should
            contain two fields: training and testing, to allow for different values depending
            on training or evaluation
</code></pre>
<p>I appreciate your time. Thanks!</p>
How can I correctly load and use the EfficientNetB3 classifier through PyTorch?

Joe_Yabuki: <p>Hi guys. I’ve used pyTorch to implement an EfficientNetB3 classifier that I have saved as ‘efficientNet_b3.pt’.<br>
Now I would like to use this classsifier ad so what I do is:</p>
<ul>
<li>load the classifier in this way</li>
</ul>
<pre><code class="lang-auto">classifier = torch.load(cls_path, map_location=torch.device('cpu'))
</code></pre>
<ul>
<li>then, reading a video in input, when I detect something that has to be classified, I take the Region of interest (ROI), I process that, and then I call the clasifier on it:</li>
</ul>
<pre><code class="lang-auto">        roi_resized = cv2.resize(roi, (300, 300))  # Right dimension for efficientNetB3
        roi_resized = roi_resized / 255.0  # Normalization phase
        roi_resized = roi_resized.reshape(1, 300, 300, 3)

        # Classify
        pred = classifier(torch.tensor(roi_resized, dtype=torch.float32))
</code></pre>
<p>But in this way, when there is something that needs to be classified, I get:</p>
<pre><code class="lang-auto">pred = classifier(torch.tensor(roi_resized, dtype=torch.float32))
TypeError: 'collections.OrderedDict' object is not callable
</code></pre>
<p>How can I resolve?</p>

soulitzer: <p>How did you save your model?</p>
PyTorch fasterrcnn-resnet50_fpn detects only one class objects

MarcinSz: <p>I posted this because maybe it’ll help to someone. In my case the problem was in annotations data (COCO json).</p>
<p><strong>Description</strong></p>
<p>I want a model for detecting features on custom images. I used officially available code from <a href="https://github.com/pytorch/vision/tree/7e9e7844e7652bf7b90eef2e9b9397bd20c00e4c/references/detection" rel="noopener nofollow ugc">here</a> (changed and adapted for my purposes but it turned out that changes in code wasn’t a problem).</p>
<p><strong>Issue</strong></p>
<p>Prediction detects only one class of objects. However, dataset images are the same size, have only two colors, presents up to four two well-distinctive shapes (for ex. circle and cross). Distribution of objects is uniformed (nearly 50% of objects of each class). Many different settings of parameters (learning rates, optimizers, schedulers, moments, learning steps, number of batches, number of epochs, weights decays, etc.) didn’t help.</p>
<p><strong>Cause of the problem and solution</strong></p>
<p>In my case I use annotations in COCO format (as JSON file). When “categories” dictionary contained only classes for detection, ex:</p>
<pre><code class="lang-auto">"categories": [{"id": 0, "name": "CROSS"}, {"id": 1, "name": "CIRCLE"}]
</code></pre>
<p>only circles have been detected on images. When there is an additional class for background, ex:</p>
<pre><code class="lang-auto">"categories": [{"id": 0, "name": "BACKGROUND"}, {"id": 1, "name": "CROSS"}, {"id": 2, "name": "CIRCLE"}]
</code></pre>
<p>There’s no problem. Category 0 is never used in this file but it seems to exist. Of course, annotations should have proper mapping now to IDs + 1. If this rule is described somewhere, please someone to put the explanation and link here. I found mentioned that the “0” category is needed for background but on different site.</p>
# NOTE: [Computing jacobian with vmap and grad for multiple outputs]

Nilsgoe: <p>I found this code block in the pytorch source code for jacobians.<br>
I am not really sure how to get it working and I think it is quite cryptic.<br>
I didn’t find a proper implementation of this. Is there one? And if not could someone help me how i would construct it.</p>
<pre><code class="lang-auto"># NOTE: [Computing jacobian with vmap and grad for multiple outputs]
            #
            # Let's consider f(x) = (x**2, x.sum()) and let x = torch.randn(3).
            # It turns out we can compute the jacobian of this function with a single
            # call to autograd.grad by using vmap over the correct grad_outputs.
            #
            # Firstly, one way to compute the jacobian is to stack x**2 and x.sum()
            # into a 4D vector. E.g., use g(x) = torch.stack([x**2, x.sum()])
            #
            # To get the first row of the jacobian, we call
            # &gt;&gt;&gt; autograd.grad(g(x), x, grad_outputs=torch.tensor([1, 0, 0, 0]))
            # To get the 2nd row of the jacobian, we call
            # &gt;&gt;&gt; autograd.grad(g(x), x, grad_outputs=torch.tensor([0, 1, 0, 0]))
            # and so on.
            #
            # Using vmap, we can vectorize all 4 of these computations into one by
            # passing the standard basis for R^4 as the grad_output.
            # vmap(partial(autograd.grad, g(x), x))(torch.eye(4)).
            #
            # Now, how do we compute the jacobian *without stacking the output*?
            # We can just split the standard basis across the outputs. So to
            # compute the jacobian of f(x), we'd use
            # &gt;&gt;&gt; autograd.grad(f(x), x, grad_outputs=_construct_standard_basis_for(...))
            # The grad_outputs looks like the following:
            # ( torch.tensor([[1, 0, 0],
            #                 [0, 1, 0],
            #                 [0, 0, 1],
            #                 [0, 0, 0]]),
            #   torch.tensor([[0],
            #                 [0],
            #                 [0],
            #                 [1]]) )
            #
            # But we're not done yet!
            # &gt;&gt;&gt; vmap(partial(autograd.grad(f(x), x, grad_outputs=...)))
            # returns a Tensor of shape [4, 3]. We have to remember to split the
            # jacobian of shape [4, 3] into two:
            # - one of shape [3, 3] for the first output
            # - one of shape [   3] for the second output
</code></pre>

soulitzer: <p>What issues are you running into with running it?</p>

Nilsgoe: <p>In this not the line</p>
<pre><code class="lang-auto">vmap(partial(autograd.grad(f(x), x, grad_outputs=...)))
</code></pre>
<p>is proposed but I am not really understand how this could work for the above described.</p>
Unbalanced data in multiple object detection

mapostig: <p>Hello,</p>
<p>I am working in object detection with my own dataset.</p>
<p>The objects present in my dataset are very umbalanced, and I was wondering if it is possible to apply something as a weighted random sampler (I have already employed this for image classification) but for object detection.</p>
<p>In my images several structures are present.</p>
<p>Thankyou</p>

ptrblck: <p>Balancing a mutli-label dataset is a bit tricky, as a simple weighted sampling might even increase the imbalance, e.g. if the majority classes are often associated with the oversampled minority classes.</p>
<p>There are a few approaches, such as <a href="https://arxiv.org/pdf/1802.05033.pdf">SCUMBLE, REMEDIAL</a>, <a href="https://ecmlpkdd2019.org/downloads/paper/624.pdf">MLSOL</a>, MLSMOTE etc.</p>
<p>I don’t know, what the current state of the art is, but I’m sure <a class="mention" href="/u/rasbt">@rasbt</a> would know. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

mapostig: <p>Thankyou so much <a class="mention" href="/u/ptrblck">@ptrblck</a>, I will check the approaches you have given me.<br>
I’ll provide I new answer if I reach a solution!</p>

duddal: <p><a class="mention" href="/u/mapostig">@mapostig</a><br>
Did you find a solution? Thanks.</p>

ankita.buntolia: <p>Hi, could you please share the approach which worked for you?</p>

MTB: <aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/issamemari/pytorch-multilabel-balanced-sampler">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/issamemari/pytorch-multilabel-balanced-sampler" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/a4e0dd3009a565321c635c3f9cdcb7b52c78ce10f9a81302cbc64fb5f133db4e/issamemari/pytorch-multilabel-balanced-sampler" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/issamemari/pytorch-multilabel-balanced-sampler" target="_blank" rel="noopener nofollow ugc">GitHub - issamemari/pytorch-multilabel-balanced-sampler: PyTorch samplers...</a></h3>

  <p>PyTorch samplers that output roughly balanced batches with support for multilabel datasets - issamemari/pytorch-multilabel-balanced-sampler</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I found something!!<br>
I think least sampled class sampler could help.</p>
Correct way to set cross entropy weights in an umbalanced dataset?

HugoT: <p>In a classification problem, if I have an unbalanced train dataset, let’s say 40 elements of class 1 and 60 elements of class 2, what should I put in the weight parameter of nn.CrossEntropyLoss so this imbalance is considered so it emulates a balanced dataset?</p>
<p>Is weights = [1/40, 1/60] the correct answer?</p>

KFrank: <p>Hi Hugo!</p>
<aside class="quote no-group" data-username="HugoT" data-post="1" data-topic="204512" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hugot/48/70684_2.png" class="avatar"> HugoT:</div>
<blockquote>
<p>Is weights = [1/40, 1/60] the correct answer?</p>
</blockquote>
</aside>
<p>Yes, this would be a fine starting point.</p>
<p>A couple of comments:</p>
<p>40 / 60 is not very imbalanced.  I probably wouldn’t bother with class<br>
weights.</p>
<p>Your results should be largely insensitive to the exact values of your class<br>
weights – if they’re not, you should consider the possibility that their might<br>
be something unstable about your model or training procedure.</p>
<p>If you only have two classes, you <em>might</em> choose to structure your problem<br>
as a binary problem (rather than a multiclass problem that happens to<br>
have two classes) and use <code>BCEWithLogitsLoss</code> and <code>pos_weight</code>, rather<br>
than <code>CrossEntropyLoss</code> and <code>weight</code>.  It doesn’t really matter much – the<br>
two approaches are very similar.</p>
<p>Best.</p>
<p>K. Frank</p>
Evaluation never finishes

Zhreyu: <p>I am trying to train a Large model, the training goes fine, but when the evaluation starts, per iteration/step time increases over time and gets worse. It eventually stops at some point where there the GPU usage drops to zero but there are still some memory allocated in GPU. There is no I/O operations happening during this time and other hardware resources are used minimally. The issue only seems to appear when I try to increase the dataset size. When I have a smaller dataset the evaluation goes through completing and the training finishes though the patter of increasing per iteration/step time is still noticeable. Increasing batchsize does help in this case but the evalution bar stops moving over time. Is there any possible way to debug this?</p>

ptrblck: <p>Did you check if your host RAM is increasing and if it’s eventually using the <code>swap</code>?</p>

Zhreyu: <p>Thanks a lot for quick response. I ran into some issues with my system. I will check again soon and let you know. What I noticed before is the memory usage was not very high there was a good size of free RAM available (around 800gb).</p>

Zhreyu: <p>Thank you for the suggestion. I’ve checked the host RAM and swap usage during the evaluation phase using detailed resource monitoring logs. It shows that both RAM and swap usage remain relatively stable throughout the training and evaluation periods, with no significant spikes or excessive consumption that would suggest memory constraints or heavy swapping activities.</p>

ptrblck: <p>So it seems you are still seeing a slowdown after a while while, which also seems to be related to the dataset size, but neither I/O ops are visible nor the usage of swap space. This is indeed really weird and you might need to use a visual profiler to check which part of the code is causing the slowdown. You could also check the thermal readings to see if the frequencies are reduced due to overheating.</p>

Zhreyu: <p>Thank you for your suggestions. I’ve used profiler to watch over the operations during the evaluation process. The profiler’s outputs shows that certain operations, particularly <code>aten::copy_</code> and <code>aten::cudnn_convolution</code> , are heavily utilized and might be contributing to the slowdown.</p>

ptrblck: <p>Thanks for checking the profile! In this case I don’t understand how the size of the dataset could be related and would guess your GPU might run into thermal issues.<br>
Monitor the clocks via <code>nvidia-smi dmon</code> to see if the frequencies drop.</p>

Zhreyu: <p>I was able to fix this issue by decreasing num-workers parameter to zero, Thanks for the help <a class="mention" href="/u/ptrblck">@ptrblck</a>. Though I still don’t understand the exact reason for the error the evaluation sped up when I increased number of workers.</p>