akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Reproducing results across different machines (CPU) - Dropout layers

pedro.mat94: <p>I was trying to reproduce the results for a CNN in two different machines (both running on CPU) but I ended up with different results (setting the same python and torch seed in advance). After checking the CNN layers individually, I noticed that the difference is caused in the Dropout layer specifically. I couldn’t find the exact implementation for Dropout, so I was wondering if there’s any idea on what may be behind this?</p>
<p>Some additional details:</p>
<pre><code class="lang-auto">import random
import torch
random.seed(2)
torch.manual_seed(3)
a = torch.randn(4, 4)
dp1 = torch.nn.Dropout(p=0.3)
dp1(data)
# Result in one of the machines:
# tensor([[-1.0361, -3.5810, -0.0000,  1.8477],
#         [-2.0117, -0.0000, -0.0785,  1.7217],
#         [-0.0000, -2.3690, -0.0955, -0.7392],
#         [-0.0000,  1.5011,  1.8367,  2.3154]])
# Result for the other machine:
# tensor([[-1.0361, -0.0000, -1.1379,  1.8477],
#         [-2.0117, -1.2687, -0.0000,  1.7217],
#         [-1.4952, -2.3690, -0.0000, -0.7392],
#         [-0.4844,  1.5011,  1.8367,  2.3154]])
</code></pre>
<ul>
<li>I also tried the tips from the reproducibility page - using deterministic algorithms, … However, the results are still different.</li>
<li>I used a docker image, so library versions are exactly the same (Python 3.9/torchvision 0.11.1/torch 1.10.0)</li>
</ul>
<p>Any tips/ideas are welcome, thanks.</p>

ptrblck: <p>As described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility docs</a> there is no guarantee to receive exactly the same random values between different platforms, CPUs, GPUs etc.</p>

pedro.mat94: <p>Indeed, I thought there could be a specific reason given that it happened only in a specific layer. Thanks.<br>
(In the meantime, I tested it out in a AWS machine with an ARM architecture and I was able to get the same results. I didn’t use AWS for my original experiments and my docker image was built for x86 so I still have to understand what happened but in any case hope this may help someone).</p>
GPU memory bandwidth calculation for Llama model inference

Arunima_Ghosh: <p>Hi,<br>
I am trying to calculate GPU memory bandwidth for Huggingface Llama model inference. I want to calculate GPU memory bandwidth for this line of code:</p>
<pre><code>**outputs = model.generate(input_ids, generation_config=generation_config)**
</code></pre>
<p>I am using these two functions to measure bandwidth but the memory bandwidth I am getting is very low.<br>
Can you please guide me what might be wrong in my calculation? is the max_memory or the bandwidth calculated correctly?</p>
<p>Thanks</p>
<p>def inference():<br>
with torch.no_grad():<br>
outputs = model.generate(input_ids, generation_config=generation_config)<br>
return outputs</p>
<p>def measure_bandwidth(func):</p>
<pre><code>torch.cuda.reset_peak_memory_stats()
torch.cuda.synchronize()

start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

start.record()
result = func()
end.record()

torch.cuda.synchronize()
elapsed_time = start.elapsed_time(end) / 1000  # Convert to seconds
max_memory = torch.cuda.max_memory_allocated() / (1024*1024*1024) 
bandwidth = max_memory / elapsed_time  # GB/s
return result, bandwidth
</code></pre>
<p>outputs, bandwidth = measure_bandwidth(inference)<br>
print(f"Estimated GPU memory bandwidth usage: {bandwidth:.2f} GB/s")</p>
Tensor Parallel examples for custom modules

rk53: <p>Hi,</p>
<p>Tensor Parallel currently supports only</p>
<ul>
<li>nn.Linear and nn.Embedding via RowwiseParallel and ColumnwiseParallel</li>
<li>nn.LayerNorm and nn.Dropout via SequenceParallel</li>
</ul>
<p>I need to implement tensor parallelization for custom modules and other standard pytorch modules such as nn.Conv1d. However, this currently raises NotImplementedError.</p>
<p>Are there any examples or guidelines for how to implement ColumnwiseParallel and RowwiseParallel for other modules such as nn.Conv1d?</p>
<p>Thanks!</p>

weifengpy: <p>Good question. Do you mind sharing the full error log of <code>NotImplementedError</code> and the code to reproduce? This can be a feature request in <a href="https://github.com/pytorch/pytorch/issues/new/choose" class="inline-onebox" rel="noopener nofollow ugc">Sign in to GitHub · GitHub</a><br>
TP/SP are implemented for matmul / nn.Linear right now. The behavior for <code>nn.Conv1d</code> is not well defined</p>

rk53: <p>I am referring to this NotImplementedError for anything other than nn.Linear and nn.Embedding: <a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/tensor/parallel/style.py#L144" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/distributed/tensor/parallel/style.py at main · pytorch/pytorch · GitHub</a></p>
<p>I have created an issue regarding common modules such as convolutions and groupnorm here: <a href="https://github.com/pytorch/pytorch/issues/133221" class="inline-onebox" rel="noopener nofollow ugc">Tensor parallel for convolutions and groupnorm · Issue #133221 · pytorch/pytorch · GitHub</a></p>
<p>The implementation of tensor parallel covers really just a tiny fraction of common modules, which makes tensor parallel currently not applicable to most models other than llama. An alternative is  to use the Megatron library, but I would prefer to directly use Pytorch distributed.</p>

weifengpy: <p>I see all the related POCs are tagged in the issue. Thanks for explaining the context clearly</p>
Sequence Parallel examples

edrt: <p>Hi, this link provides an example about tensor parallel together with sp on norm layers and FSDP:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/examples/blob/main/distributed/tensor_parallelism/fsdp_tp_example.py">
  <header class="source">

      <a href="https://github.com/pytorch/examples/blob/main/distributed/tensor_parallelism/fsdp_tp_example.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/main/distributed/tensor_parallelism/fsdp_tp_example.py" target="_blank" rel="noopener nofollow ugc">pytorch/examples/blob/main/distributed/tensor_parallelism/fsdp_tp_example.py</a></h4>


      <pre><code class="lang-py">import sys
import os
import torch
import torch.distributed as dist
import torch.nn as nn
import torch.nn.functional as F

from log_utils import rank_log, get_logger, verify_min_gpu_count

# ---- GPU check ------------
_min_gpu_count = 4

if not verify_min_gpu_count(min_gpus=_min_gpu_count):
    print(f"Unable to locate sufficient {_min_gpu_count} gpus to run this example. Exiting.")
    sys.exit()
# ---------------------------

from llama2_model import Transformer, ModelArgs

from torch.distributed.device_mesh import init_device_mesh
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/examples/blob/main/distributed/tensor_parallelism/fsdp_tp_example.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
However, this parallel is a mainly Megatron style TP. I just wanna know how to write simple sequence parallel style codes only on sequence dim with the same llama model(not the other quite simple sp <a href="https://github.com/pytorch/examples/blob/main/distributed/tensor_parallelism/sequence_parallel_example.py" rel="noopener nofollow ugc">example</a>), like what happend in DeepSpeed-Ulyses. So can any provide minimal code changes, e.g. the specific kind of <code>parallelize_plan</code>? I would appreciate a lot.</p>

weifengpy: <p>good question. We do not have an out of the box example for SP-only. We had a tutorial for SP but SP is always on top of TP <a href="https://pytorch.org/tutorials/intermediate/TP_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Large Scale Transformer model training with Tensor Parallel (TP) — PyTorch Tutorials 2.4.0+cu121 documentation</a></p>

edrt: <p>Thanks, I’ve seen this doc and it’s same with my provided link. After a day, I realized sequence-only parallel might can be down but need to convert model weight to DTensor. However, SP-only is meaningless, a Ulysses-like parallel is meaningful.<br>
Thus here I wanna know things in detail: Since Ulysses handles SP with all-to-all matrix transpose and similar attention-head parallism toTP, I think my question is can we done this with DTensor now? or in detail, can DTensor automatically call all-to-all communication?</p>
`autograd.Function` backwards not retaining grad

Jerome_Ku: <p>I have a custom <code>autograd.Function</code> that calls the following in <code>forward</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">def torch_conv(x, weight):
    _, d_model, L = x.shape
    kernel_size = weight.shape[-1]
    print(f"torch_conv: {x.requires_grad=}, {weight.requires_grad=}")
    y = F.conv1d(
        x,
        weight,
        bias=None,
        stride=1,
        padding=kernel_size - 1,
        groups=d_model,
    )
    print(f"conv out: {y.requires_grad=}")
    y = y[..., :L]
    print(f"conv out sliced: {y.requires_grad=}")
    
    return y
</code></pre>
<p>When I run this function outside of the context of the autograd.Function:</p>
<pre data-code-wrap="python"><code class="lang-python">bs = 1
np = 1
hn = 768
seqlen = 8192
dtype = torch.float32
device = "cuda"

x = torch.randn(bs, np * hn, seqlen, dtype=dtype, device=device).requires_grad_()
w = torch.randn(d, 1, hl, dtype=dtype, device=device).requires_grad_()
out = torch_conv(x, w)
</code></pre>
<p>The print messages are expected: inputs and outputs show <code>requires_grad = True</code> as expected.</p>
<p>However, when running this snippet within <code>autograd.Function</code>, the inputs <code>require_grad = True</code> as expected, but <code>conv_out</code> prints <code>requires_grad = False</code>, which causes downstream problems, as I need to backprop through <code>out</code>.</p>
<p>Not sure what is going on here.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/jerome_ku">@Jerome_Ku</a>,</p>
<p>The <code>.grad</code> attribute of a tensor is only created when you compute a loss and backprop. As in, you call <code>loss.backward()</code> and the <code>.grad</code> attribute will be there.</p>
<p>If you’re using it as a custom <code>torch.autograd.Function</code> object, you will need to define the backward formula yourself. If you haven’t defined it, then it won’t exist by definition.</p>
How Can I Utilize PyTorch for Real-Time Image Classification While Leveraging the 'What Country Am I In' Navigation Tool?

JovitaYoshio: <p>I’ve been exploring PyTorch for building and deploying deep learning models, and I’m currently interested in creating a real-time image classification system. Specifically, I want to use PyTorch’s capabilities to classify images captured on the fly from a mobile device or a webcam. However, my curiosity has led me to wonder how I could also integrate geographic data into the classification process, particularly using the ‘What Country Am I In’ navigation tool.</p>
<p>The ‘<a href="https://onlinecompass.net/what-country-am-i-in" rel="noopener nofollow ugc">what country am in</a>’ tool is a navigation utility that helps users determine their current country based on their GPS location. My question revolves around how I can leverage this tool in conjunction with a PyTorch model. For instance, could I use the geographic information from the tool to add context to the image classification results? If so, how might I structure the data pipeline in PyTorch to include location data from this tool along with image data for improved predictions?</p>
<p>I’m also interested in understanding the potential challenges that could arise when combining these two types of data. For example, how would I handle discrepancies between the predicted image classification and the geographic context provided by the ‘What Country Am I In’ tool? Are there any specific PyTorch libraries or extensions that might facilitate this kind of data integration, or would I need to preprocess the location data separately before feeding it into the model?</p>
<p>Furthermore, I would appreciate any insights into optimizing the model’s performance when dealing with both image and location data. Since real-time processing is crucial, what are some best practices for ensuring that the PyTorch model can handle these two inputs efficiently without significant latency? Are there specific architectural considerations or model designs that might be more suited to this type of task?</p>
<p>Lastly, I’m curious if anyone in the community has experience with similar projects where PyTorch was used alongside navigation or geographic tools. Any examples, case studies, or relevant experiences would be incredibly helpful. I’m eager to learn how I can apply PyTorch in this innovative way and look forward to any guidance or suggestions from the community.</p>
Torch sparse tensor unsupported at mps

Rami_Nasser: <p>I have sparse tensor for handling graph representation, however it can’t be processed by ‘mps’, I wonder if there is a workaround ?<br>
Error I get:<br>
NotImplementedError: Unsupported device type for sparse layout: mps</p>

albanD: <p>Hi,</p>
<p>Sparse support is not planned yet I’m afraid.<br>
You should be able to run that part on the CPU and then transfer to do dense ops on the MPS side.</p>

David_Venus: <p>Hello,</p>
<p>I am facing the same problem. Still no support for spare tensors on MPS planned?</p>
<p>Right now I am doing the spare matrix multiplication on the CPU, but this is really slow because my tensors are quite big.</p>
<p>Any way to fix this?</p>
Fancy idexing memory footprint

Konsthr: <p>Hi! Is it possible to use the fancy indexing in PyTorch without returning a copy of the indexed tensor?</p>
<p>For example I have a large 3D tensor X [100,100,100] and want to extract multiple (say 10) [50,50,50] slices to get the resulting tensor X_sliced [500,50,50,50]. Currently, I use fancy indexing  as  X_sliced  = X[mask_0+borders_0, mask_1+borders_1, mask2+borders_2], where masks and borders are indexing int64 tensors. However instead of view with zero additional memory footprint this method returns copied slices significantly increasing the memory footprint.</p>
<p>Is there a way to do this kind of indexing without using additional memory?</p>
Error when export model to ONNX through dynamo_export

cokespace2: <p>As the latest version of PyTorch has that api <a href="https://github.com/pytorch/pytorch/blob/main/torch/onnx/__init__.py#L52" rel="noopener nofollow ugc">dynamo_export</a>, and it seems designed to export ONNX model from dynamo, so I have 2 questions below:</p>
<ul>
<li>
<ol>
<li>What’s the difference between <code>torch.onnx.export</code> and <code>torch.onnx.dynamo_export</code> ? They both process normal <code>nn.Modules</code> and then generate a ONNX model.</li>
</ol>
</li>
<li>
<ol start="2">
<li>I even can not export a resnet50 from <code>mmpretrain</code> through <code>dynamo_export</code>, and I don’t find any document for it, so is it in early stage?</li>
</ol>
</li>
</ul>

marksaroufim: <p>They differ in how you graph capture. Regular export uses JIT and Dynamo export uses Dynamo</p>
<p>Dynamo export is still in early stages but will likely become the default in the future given that torchscript development was paused</p>

cokespace2: <p>Thanks mark!</p>
<p>Btw, as this <code>dyamo_export</code> is taking a normal <code>nn.Module</code> in and outputs an ONNX model, can we make a api that enable us to put a <strong>traced</strong> <code>fx.Graph</code> into it then output an ONNX model?</p>
<p>Since I found that PyTorch is also working on another quantization tool <code>pt2e</code>, so this api can be usefule that we quantize, optimize models in fx.Graph format, then export to ONNX model for serving.</p>

marksaroufim: <p>cc <a class="mention" href="/u/bowenbao">@BowenBao</a> who is maybe already looking into this</p>

Gurkirt: <p>I would like to know where are we on this. I have fx.graphmodule generated form pt2e after quantization, I am not able to export such a graphmodule to onnx. I am really interested in this because I got the best quantization result using pt2e compare dot eager mode.</p>
Using pre allocated CUDA buffers when calling module forward

Gonen_Raveh: <p>When the loaded PyTorch Module returns a “Tensor” it works great. Here is the code:</p>
<pre><code class="lang-auto">&gt; // allocate your cuda buffer
&gt; cudaMalloc(cuda_output_ptr,....)
&gt; // make it a proper Torch Tensor
&gt; at::Tensor predTensor = torch::from_blob((void*)cuda_output_ptr, c10::IntArrayRef(myTensorShape), tensor_options);                
&gt; // call module foreward and use std::move() for the result variable
&gt; std::move(predTensor) = mModule.forward(std::move(inputs)).toTensor();
</code></pre>
<p>But, that does not work for me when the Module outputs a List[Tensor] or TensorList<br>
did anyone had success with that? Here is my code:</p>
<pre><code class="lang-auto">&gt; c10::List&lt;at::Tensor&gt; predTensorList;
&gt; // add cuda allocated output buffers...
&gt; for (int i=0; i&lt;xxx; ++i) {
&gt;     at::Tensor predTensor = torch::from_blob((void*)output_ptr, 
&gt;     c10::IntArrayRef(pytorchTensorShapeRefPtr), tensor_options);
&gt;     predTensorList.push_back(std::move(predTensor));
&gt; }
&gt; std::move(predTensorList) = mModule.forward(std::move(inputs)).toTensorList();
</code></pre>
<p>The result is that the output cuda buffers stay empty or with their previous values.</p>
Data Transfer slow from gpu to cpu

Anya: <p>I tested the data transfer speeds between CPU to GPU and GPU to CPU, and there was a significant difference between the two.<br>
PCIe: Speed 8GT/s, Width x8<br>
result：<br>
CPU-&gt;GPU:5.30 GB/s<br>
GPU-&gt;CPU:0.93 GB/s<br>
What’s the problem? How to accelerate data transfer time from GPU to CPU?<br>
My test code is here:</p>
<pre><code class="lang-auto">import torch
import time

tensor_size = 1024 * 32  # 1GB
dtype = torch.float32

tensor = torch.randn((tensor_size, tensor_size), dtype=dtype, device='cuda')
targer_tensor = torch.zeros((tensor_size, tensor_size), dtype=dtype, device='cpu')
torch.cuda.synchronize()

start_time = time.time()

num_iterations = 10
for _ in range(num_iterations):
    tensor.cpu()

torch.cuda.synchronize()

end_time = time.time()

total_data_transferred_gb = (num_iterations * tensor_size * tensor_size * tensor.element_size()) / (1024**3)

average_bandwidth_gb_per_s = total_data_transferred_gb / (end_time - start_time)

print(f"Average bandwidth: {average_bandwidth_gb_per_s:.2f} GB/s")
</code></pre>

Anya: <p>OK,I tested pinned memory in CPU,which acclerated CPU&lt;-&gt;GPU transfer to 11GB/s. But I wonder why it is so slow when using normal tensor.cpu(). Is it because it needs CPU memory allocation?</p>
oneDNN error "Not support device type cuda"

TFW: <p>I’m following the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html" rel="noopener nofollow ugc">guide</a>, after adding the line <code>torch.jit.enable_onednn_fusion(True)</code><br>
it creates the following error:<br>
<code>RuntimeError: The following operation failed in the TorchScript interpreter. Traceback of TorchScript (most recent call last): RuntimeError: Not support device type cuda</code></p>
<p>My model is compiled with torch.jit.script, stack trace shows basically nothing<br>
Is oneDNN cpu only or did I not install something required?</p>

TFW: <p>Oh it was listed under <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html#cpu-specific-optimizations" rel="noopener nofollow ugc">CPU specific optimizations</a> didn’t see the category</p>
How Do I Get the Intermediate Gradients During Backpropagation?

Chen_Zhang: <p>Now assume I have a module denoted as function $f, g$ which takes input $x$ and parameters $\theta_f, \theta_g$.  The loss is $L = g(z, \theta_g)$ and $z = f(x, \theta_f)$.<br>
When the backpropagation starts, we have $\frac{\partial L}{\partial \theta_g}$ and $\frac{\partial L}{\partial z}$. Then we use the second partial derivative to calculate $\frac{\partial L}{\partial \theta_f}$.<br>
When we call the backward(), the autograd will do the above operation and then give us $\frac{\partial L}{\partial \theta_g}$ and $\frac{\partial L}{\partial \theta_f}$, but is there a way that I can get $\frac{\partial L}{\partial z}$?</p>

Chen_Zhang: <p>btw, the display of equations seems went rong</p>
Results on Android and PC are different with the same model

NazaRik: <p>Hi, I’ve trained EfficientNet model from timm and converted it into .ptl for android. Here is conversion code:</p>
<pre><code class="lang-auto">dummy_input = torch.randn(1,3,256,256)
traced_script_module = torch.jit.trace(model, dummy_input)
traced_script_module_optimized = optimize_for_mobile(traced_script_module)
traced_script_module_optimized._save_for_lite_interpreter("./models/classifier_best.ptl")
</code></pre>
<p>Then I try to forward Image from assets to my model but results are very different<br>
Maybe I’m doing something wrong? Here is my Android code:</p>
<pre><code class="lang-auto">try {
    module = Module.load(assetFilePath(TestModelActivity.this, "classifier_best.ptl"));
} catch (IOException e) {
    throw new RuntimeException(e);
}

try {
    bitmap = BitmapFactory.decodeStream(getAssets().open("solara_cropped.jpg"));
} catch (IOException e) {
    throw new RuntimeException(e);
}
Tensor inputTensor = TensorImageUtils.bitmapToFloat32Tensor(bitmap,
        TensorImageUtils.TORCHVISION_NORM_MEAN_RGB, 
        TensorImageUtils.TORCHVISION_NORM_STD_RGB,
        MemoryFormat.CHANNELS_LAST);

Tensor outputTensor = module.forward(IValue.from(inputTensor)).toTensor();
float[] scores = outputTensor.getDataAsFloatArray();
</code></pre>
Need to run torch 1.9.0 >= 1.60 on RTX4090

Ryan_Marr: <p>Need to run torch 1.9.0 &gt;= 1.6.0 on RTX4090 because these are the requirements for Unity mlagents<br>
What I need to figure out is what version of Cuda to install to satisfy these requirements on Ubuntu 20.04<br>
I’ve tried Cuda Version 12.7 and 11.6 with torch 1.6.0 but I still see the same error:</p>
<pre><code class="lang-auto">NVIDIA GeForce RTX 4090 with CUDA capability sm_89 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Ryan_Marr" data-post="1" data-topic="208225">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ryan_marr/48/57227_2.png" class="avatar"> Ryan_Marr:</div>
<blockquote>
<p>Cuda Version 12.7 and 11.6</p>
</blockquote>
</aside>
<p>CUDA 12.7 is not a thing and your locally installed CUDA toolkit won’t be used (unless you build PyTorch from source or a custom CUDA extension). Based on the error message you have installed a PyTorch binary with CUDA 10.2 runtime dependencies and you can verify it via <code>print(torch.version.cuda)</code>. If so, install a PyTorch binary with CUDA 11+ runtime dependencies.</p>

Ryan_Marr: <p>This worked thank you for the help</p>
Why Doesn’t My Tensor Offloading Strategy Reduce GPU Memory Usage in Forward Pass?

zbtrs: <p>I am training a model using PyTorch and would like to offload tensors to CPU memory during the forward pass as soon as they are no longer needed, and then reload them to GPU memory during the backward pass if required. This can be achieved using <code>torch.autograd.graph.save_on_cpu()</code>, but why doesn’t the following code work as expected?</p>
<pre><code class="lang-auto">class DeepModel(nn.Module):
    def __init__(self, layer_num=40, layer_size=4096):
        super(DeepModel, self).__init__()
        self.layers = nn.ModuleList([nn.Linear(layer_size, layer_size) for _ in range(layer_num)])
        self.relu = nn.ReLU()

    def forward(self, x, offload=False):
        if offload:
            prev_tensor = x
            for idx, layer in enumerate(self.layers):
                x = self.relu(layer(prev_tensor))
                prev_tensor.to('cpu', non_blocking=True)
                prev_tensor = x
        return x
</code></pre>
<p>Here is the test output:</p>
<pre><code class="lang-auto">Testing without offloading:
Memory allocated before forward pass: 2576.62 MB
Memory allocated after forward pass: 3224.75 MB
Max memory allocated during forward pass: 3240.75 MB

Testing with offloading:
Memory allocated before forward pass: 2584.75 MB
Memory allocated after forward pass: 3224.75 MB
Max memory allocated during forward pass: 3240.75 MB

</code></pre>
<p>There’s no significant reduction in GPU memory usage. However, with the following approach using <code>torch.autograd.graph.save_on_cpu()</code>, I do observe a noticeable decrease in memory usage:</p>
<pre><code class="lang-auto">class DeepModel(nn.Module):
    def __init__(self, layer_num=40, layer_size=4096):
        super(DeepModel, self).__init__()
        self.layers = nn.ModuleList([nn.Linear(layer_size, layer_size) for _ in range(layer_num)])
        self.relu = nn.ReLU()

    def forward(self, x, offload=False):
        if offload:
            with torch.autograd.graph.save_on_cpu(pin_memory=True):
                prev_tensor = x
                for idx, layer in enumerate(self.layers):
                    x = self.relu(layer(prev_tensor))
                    prev_tensor.to('cpu', non_blocking=True)
                    prev_tensor = x
        return x
</code></pre>
<p>Why doesn’t the first method reduce memory usage as expected? What could be causing this difference?</p>
Performance of SMOTE on CIFAR10 dataset

ARNAB_BANERJEE: <p>Hello, guys,</p>
<p>I was working on implementing SMOTE on the CIFAR10 dataset. I artificially introduced imbalance by making the first class a minority class(555 training examples) and kept all the training examples of other classes intact. I then applied SMOTE to the minority class to remove any imbalance. I was mainly concerned about the class accuracy of the minority class. After running the code, I got the following results:</p>
<p>Accuracy(Recall) of minority class(first class) when using the entire CIFAR10 dataset: 87.9%</p>
<p>Accuracy(Recall) of minority class(first class) when using the imbalanced CIFAR10 dataset: 50.9%</p>
<p>Accuracy(Recall) of minority class(first class) when using the SMOTE-balanced CIFAR10 dataset: 40.6%</p>
<p>The accuracy of other classes does not change much in all the above scenarios.</p>
<p>While training with the SMOTE-balanced dataset, I noticed that the validation accuracy of my validation set is very high (almost 1) whereas the test accuracy is as low as 40%. It indicates that the model must be overfitting this class.</p>
<p>I wonder if SMOTE is a good technique for the CIFAR dataset or any image dataset in general. Also, if someone else has tried SMOTE on CIFAR and got better results than just training with an imbalanced dataset please let me know how can I improve the class accuracies?</p>

Baraa: <p>Were you able to implement SMOTE on CIFAR-10?</p>
Type of the forward method of RvNNs in C++

jparsert: <p>Many tutorials/samples of RvNNs (Recursive/Tree Neural Networks) in torch found on the internet use Python’s torch API. This allows them to implement a forward method with the signature</p>
<pre><code class="lang-auto">def forward(tree):
....
</code></pre>
<p>However, the type of the argument (i.e. <code>tree</code>) is a tree data structure that we traverse.<br>
When trying to reimplement this in C++ i ran into the problem that to overload the <code>forward</code> method, the argument needs to be of type <code>torch::Tensor</code>.</p>
<p>Hence, I am wondering what the standard way of implementing RvNNs is for C++.<br>
One way would probably be to store the tree in the RvNN type and basically use that to traverse the tree. But I suspect that that would make batching etc. impossible.</p>
Discrepancy in BatchNorm2d Gradient Calculation Between TensorFlow and PyTorch

ElLoboLoco: <p>I’ve compared the gradient calculations of a <code>BatchNorm2d</code> layer across different deep learning frameworks, specifically TensorFlow and PyTorch. While doing so, I’ve encountered a significant discrepancy in the gradients computed by the two frameworks. More specifically, the gradient computed by PyTorch is near zero, while TensorFlow returns something close to one.</p>
<p>Here’s the code I used for comparison:</p>
<pre><code class="lang-auto">import numpy as np

# TensorFlow/Keras
import tensorflow as tf

# PyTorch
import torch
import torch.nn as nn


# Set a common random seed for reproducibility
seed = 0
np.random.seed(seed)
tf.random.set_seed(seed)
torch.manual_seed(seed)

# Create a random input tensor with the same shape for all frameworks
input_shape = (4, 3, 5, 5)
x_np = np.random.randn(*input_shape).astype(np.float32)

# TensorFlow/Keras BatchNorm2d
class TFModel(tf.keras.Model):
    def __init__(self):
        super(TFModel, self).__init__()
        self.bn = tf.keras.layers.BatchNormalization(axis=1, epsilon=1e-05, momentum=0.1)

    def call(self, x):
        return self.bn(x)

# Instantiate the model
tf_model = TFModel()

# Convert the numpy array to a tensor and ensure it's being watched
x_tf = tf.convert_to_tensor(x_np)
x_tf = tf.Variable(x_tf) 

with tf.GradientTape() as tape:
    y_tf = tf_model(x_tf)
    y_tf_sum = tf.reduce_sum(y_tf)

# Compute the gradient of the output with respect to the input
grad_tf = tape.gradient(y_tf_sum, x_tf)

# Convert the gradient to a numpy array for comparison
grad_tf = grad_tf.numpy()

# PyTorch BatchNorm2d
class TorchModel(nn.Module):
    def __init__(self):
        super(TorchModel, self).__init__()
        self.bn = nn.BatchNorm2d(3)

    def forward(self, x):
        return self.bn(x)

torch_model = TorchModel()
x_torch = torch.tensor(x_np, requires_grad=True)
y_torch = torch_model(x_torch)
y_torch.sum().backward()
grad_torch = x_torch.grad.detach().numpy()


# Calculate the difference between TensorFlow and PyTorch gradients
diff_tf_torch = np.mean(np.abs(grad_tf - grad_torch))

# Print the differences
print(f"Difference between TensorFlow and PyTorch gradients: {diff_tf_torch:.6f}")
print("grad pytorch : ", grad_torch[0])
print("grad tf : ", grad_tf[0])
</code></pre>
<h3><a name="outputs-1" class="anchor" href="#outputs-1"></a>Outputs:</h3>
<ul>
<li>Difference between TensorFlow and PyTorch gradients: 0.999500 (or similar)</li>
<li><code>grad_pytorch</code>: Shows a gradient of all zeros (10^-10).</li>
<li><code>grad_tf</code>: Shows non-zero gradients.</li>
</ul>
<p>I tried to compare both under same conditions. Furthermore, zero gradient for the input seems weird to me. Could there be an issue with how the layers are initialized, or how the inputs are being processed in PyTorch that might lead to this discrepancy? Any insights or explanations would be greatly appreciated</p>
Can i fine tune "silero_stt" model on my custom dataset?

Nima_Chelongar: <p>i load model like:</p>
<pre><code class="lang-auto">model, decoder, utils = torch.hub.load(repo_or_dir='snakers4/silero-models', model='silero_stt', language='en', device='cuda')
</code></pre>
<p>and try to fine tune it with :</p>
<pre><code class="lang-auto">for epoch in range(100):
    model.train()
    for batch in data_loader:
        optimizer.zero_grad()
        input, targets = batch
        output = model(input)
        loss = criterion(output, targets)
        loss.backward()
        optimizer.step()
    print(f"Epoch {epoch}: Loss {loss.item()}")
</code></pre>
Implementation of SWISH : a self-gated activation function

GunHo_Choi: <p>A new activation function named “swish” came out and I tried to make a custom layer according to this(<a href="http://pytorch.org/docs/master/notes/extending.html#extending-torch-autograd" rel="nofollow noopener">http://pytorch.org/docs/master/notes/extending.html#extending-torch-autograd</a>) example and the paper(<a href="https://arxiv.org/pdf/1710.05941.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1710.05941.pdf</a>).</p>
<p>Is this a proper way of making a custom activation function?</p>
<pre><code>Class Swish(Function):
    @staticmethod
    def forward(ctx, i):
        result = i*i.sigmoid()
        ctx.save_for_backward(result,i)
        return result

    @staticmethod
    def backward(ctx, grad_output):
        result,i = ctx.saved_variables
        sigmoid_x = i.sigmoid()
        return grad_output * (result+sigmoid_x*(1-result))

swish= Swish.apply

class Swish_module(nn.Module):
    def forward(self,x):
        return swish(x)
    
swish_layer = Swish_module()</code></pre>

greaber: <p>I find it simplest to use activation functions in a functional way.  Then the code can be</p>
<pre><code class="lang-auto">def swish(x):
    return x * F.sigmoid(x)
</code></pre>

kvrd18: <p>I doubt if it’s the most memory efficient implementation present right now.</p>

Md_Mahfujur_Rahman_0: <p>What is for F… what module that have to be imported as F. Can you please tell here?</p>

smth: <p><a class="mention" href="/u/md_mahfujur_rahman_0">@Md_Mahfujur_Rahman_0</a> look at some code samples in <a href="https://github.com/pytorch/examples/">https://github.com/pytorch/examples/</a> and you will find <code>F</code></p>

yao-ying: <p>Actually，there is another learnable Activation function in the paper：Swish-β=x · σ(βx)。Coud you please respective implementation it in：channel-shared，channel-wise，element-wise forms,I found it difficult to implementation.thank you!</p>

rahul_kanojia: <p>F stands for torch.nn.functional</p>

shalom_p: <p><a class="mention" href="/u/yao-ying">@yao-ying</a> Going by your comment I think the implementation would be something like this.</p>
<pre><code class="lang-auto">import torch.nn as nn
class learnableSwish(nn.Module):
    def __init__(self):
        super(learnableSwish, self).__init__()
        self.beta = nn.Parameter(torch.as_tensor(0))

    def forward(self,x):
        x = x*nn.functional.sigmoid(self.beta*x)
        return x
</code></pre>
<p>please let me know if you face some issues.</p>
GPU Utilization Tutorial/Troubleshooting

Jeff_Cochran: <p>I’m running someone else’s model locally, and I noticed that, as written, the GPU utilization is really low, and the training is slow. My initial reaction was to increase the batch size two orders of magnitude, however the training was still slow and the utilization was &lt; 5%.</p>
<p>So, I have two questions:</p>
<ul>
<li>has anyone produced a nice tutorial on how to improve GPU utilization? This seems like an important topic for anyone creating their own models.</li>
<li>is there a set of “usual suspects” in model/dataset/dataloader code that, when overlooked, kill GPU performance?</li>
</ul>

Eduardo_Lawson: <p>This can be a very naive answer. But did you check if the model and data were sent to the device?</p>

Soumya_Kundu: <p><a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html</a></p>
<p>Maybe this?</p>
How many lines of code (LOC) in pytorch?

shaoyu_young: <p>Hi, guys.</p>
<p>I want to calculate <strong>how many lines of python code (I mean the code in .py file)</strong> in a specific pytorch version.</p>
<p>(<strong>Exclude blank lines, comments, docstrings, etc. that will not be executed</strong>)<br>
Is there any way to count easily?</p>
<p>Thx in advance <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>You could clone the repository, filter for the needed files, and coins the lines in them as described <a href="https://stackoverflow.com/a/4822516">here</a>.</p>

shaoyu_young: <p>Got it, really thank u <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
How to Convert pytorch qat model to tensorrt

AnnaTrainingG: <p>I find that the converted qat model in pytorch can’t use GPU Kernel, But I don’t find the function or ways to convert to tensorrt. How to Convert pytorch qat model to tensorrt?</p>

narendasan: <p>Try using this tool if you are looking to use QAT with TensorRT <a href="https://nvidia.github.io/TensorRT-Model-Optimizer/guides/_pytorch_quantization.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Quantization — Model Optimizer 0.15.0</a></p>
Torch_tensorrt compile for a range of sizes with steps

ahmedius2: <p>Hi,</p>
<p>I need to compile a model for a range of input sizes with a step size. For example, the model should be compiled for the following 4 sizes where step size is 12.<br>
1x256x192x12<br>
1x256x192x24<br>
1x256x192x36<br>
1x256x192x48</p>
<p>If I try to compile with min and max shapes (1x256x192x12 and 1x256x192x48), it doesn’t work because the model fails to make the inference for some sizes due to the torch.cat operation. Is there a way to resolve this issue without changing the model?</p>
<p>Also, when compiling a model with torch_tensorrt like the following, does the value of torch.backends.cudnn.benchmark matter?<br>
torch_tensorrt.compile(self.backbone_2d, ir=‘dynamo’, inputs=inp)</p>
<p>Making it False causes the following warning:</p>
<p>WARNING:py.warnings:/home/humble/.local/lib/python3.10/site-packages/torch/nn/modules/conv.py:456: UserWarning:<br>
Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at /opt/pytorch/aten/src/ATen/nat<br>
ive/cudnn/Conv_v8.cpp:919.)<br>
return F.conv2d(input, weight, bias, self.stride,</p>

narendasan: <p>For dynamic shapes, follow this guide: <a href="https://pytorch.org/TensorRT/user_guide/dynamic_shapes.html" class="inline-onebox" rel="noopener nofollow ugc">Dynamic shapes with Torch-TensorRT — Torch-TensorRT v2.5.0.dev0+4d8a94a documentation</a></p>
<p>torch.backends.cudnn.benchmark should not matter</p>
Loss function for object detection

dannepai: <p>Hello!</p>
<p>I’m training a small network to detect and locate a certain type of snail in an image. The network consists of a few Conv2d layers, a couple of Linear layers and the final output is a vector <em>(x, y, width, height, has_snail)</em>. These five elements goes through a sigmoid activation.</p>
<p>Training the network with ~80 augmented images resized to 512*384 pixels and MSE loss results in a decent performance.</p>
<p>However, I’d like to use BCE loss for <em>has_snail</em> and MSE for the everything else, but the model seems to get much worse by doing that. It’s unstable, it doesn’t predict the locations as well as before, and the “has_snail” is always very close to 1 (the dataset is imbalanced).</p>
<p>My loss function is as follows:</p>
<pre><code class="lang-auto">import torch.nn.functional as F
import torch.nn as nn

class MSE_BCE_loss(nn.Module):
    def __init__(self):
        super().__init__()
    
    def forward(self, predictions, target):
        mse = F.mse_loss(predictions[:,:4], target[:,:4])
        bce = F.binary_cross_entropy(predictions[:,4:], target[:,4:])

        return 0.5 * mse + 0.5 * bce
</code></pre>
<p>What am I doing wrong? Is it something in the code or is it just wrong to add two different types of losses together like this? I’ve tried adjusting the coefficients of the two losses without any apparent effect.</p>
Check out this PyTorch optimization sample to improve workload efficiency on the Tiber cloud

sonyai: <p><a href="https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Harness-PyTorch-and-TensorFlow-for-AI-on-Intel-Tiber-Developer/post/1622234" rel="noopener nofollow ugc">https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Harness-PyTorch-and-TensorFlow-for-AI-on-Intel-Tiber-Developer/post/1622234</a></p>
<p>Check out this code sample optimization for PyTorch and how you can try it out and run it for free on the Intel Tiber Developer Cloud</p>
Ticket Giveaway for PyTorch Conference

PerforatedAI: <p>Hello, this is Rorry Brenner, the founder of <a href="https://www.perforatedai.com/free-trial" rel="noopener nofollow ugc">Perforated AI</a>.  We’re one of the sponsors for the upcoming <a href="https://events.linuxfoundation.org/pytorch-conference/sponsor/" rel="noopener nofollow ugc">PyTorch conference</a>.  As a bronze sponsor they gave us 4 tickets but we’ll only be bringing 3 people.  Right now the startup is in a phase where we’re just looking for folks to do free trials and see how they like our optimization system.  We’d love to give that ticket to someone willing to try things out.  Open to industry folks or academics.  If you’re interested just message me through our website above with a link to your linkedin and I’ll be in touch.  Trial will require about an hour of your time then and re-running your training pipeline.</p>
Cuda is available but nccl is not : RuntimeError: Distributed package doesn't have NCCL built in

silvermet999: <p>I am trying to perform DDP on my model. cuda is available and device count outputs 2, world size outputs 2 and rank outputs 1. But when I run</p>
<pre><code class="lang-auto">dist.init_process_group("nccl", rank=rank, world_size=world_size)
RuntimeError: Distributed package doesn't have NCCL built in
</code></pre>
<p>and</p>
<pre><code class="lang-auto">torch.cuda.nccl.version()
AttributeError: module 'torch._C' has no attribute '_nccl_version'
</code></pre>
<p>I had Cuda 12.4 but because Pytorch recently made it available I downgraded to 12.1. But I still have the same problem.</p>

ptrblck: <p>Which PyTorch binary have you installed? If you are on Windows, then no NCCL support is available and you would need to use Gloo.</p>

silvermet999: <p>it is  <strong>torch 2.4.0+cu121</strong> torchaudio 2.4.0+cu121 torchvision 0.19.0+cu121 installed using the pytoch <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">documentation</a> . Yes, I am using windows so it was solved with gloo, thank you for your reply. I wanted to use nccl because I thought gloo worked only with cpu. Is there any information to when nccl support would be available on Windows please?</p>

ptrblck: <p>No NCCL support for Windows is planned.</p>
NER Bert Random behavior

Muhammad-T: <p>Hello fellows ,I was working on Customer NER problem by fine-tuning base bert yet for the 4th time the model wasn’t learning any thing with total random loss values yet constant over epochs for each try , eg :<br>
Epoch 0: Train Loss = 0.2952, Val Loss = 0.2944<br>
Epoch 10: Train Loss = 0.2950, Val Loss = 0.2944</p>
<p>here is the Model class :<br>
class NERBertModel(nn.Module):</p>
<pre><code>def __init__(self, num_tag):
    super(NERBertModel, self).__init__()
    self.num_tag = num_tag
    self.bert = BertModel.from_pretrained('bert-base-uncased')
    self.bert_drop = nn.Dropout(0.3)
    self.out_tag = nn.Linear(768,self.num_tag)
    
def forward(self, ids, mask, token_type_ids, target_tags):
    output, _ = self.bert(ids, attention_mask=mask, token_type_ids=token_type_ids, return_dict=False)
    bert_out = self.bert_drop(output) 
    tag = self.out_tag(bert_out)

    #Calculate the loss
    Critirion_Loss = nn.CrossEntropyLoss(ignore_index=-100)
    active_loss = mask.view(-1) == 1
    active_logits = tag.view(-1, self.num_tag)
    active_labels = torch.where(active_loss, target_tags.view(-1), torch.tensor(Critirion_Loss.ignore_index).type_as(target_tags))
    loss = Critirion_Loss(active_logits, active_labels)
    return tag, loss
</code></pre>
<p>hyper params :<br>
<span class="hashtag-raw">#Function</span> for getparameters<br>
def get_hyperparameters(model, ff):</p>
<pre><code># ff: full_finetuning
if ff:
    param_optimizer = list(model.named_parameters())
    no_decay = ["bias", "gamma", "beta"]
    optimizer_grouped_parameters = [
        {
            "params": [
                p for n, p in param_optimizer if not any(nd in n for nd in no_decay)
            ],
            "weight_decay_rate": 0.01,
        },
        {
            "params": [
                p for n, p in param_optimizer if any(nd in n for nd in no_decay)
            ],
            "weight_decay_rate": 0.0,
        },
    ]
else:
    param_optimizer = list(model.out_tag.named_parameters())
    optimizer_grouped_parameters = [{"params": [p for n, p in param_optimizer]}]

return optimizer_grouped_parameters
</code></pre>
<p>train\val_functions :</p>
<p>def train_fn(train_data_loader, model, optimizer, device, scheduler):<br>
<span class="hashtag-raw">#Train</span> the Model<br>
model.train()<br>
loss_ = 0<br>
for data in tqdm(train_data_loader, total = len(train_data_loader)):<br>
for i, j in data.items():<br>
data[i] = j.to(device)</p>
<pre><code>    #Backward Propagation
    optimizer.zero_grad()
    _, loss = model(**data)
    loss.backward()
    optimizer.step()
    scheduler.step()
    loss_ += loss.item()
return model, loss_ / len(train_data_loader)
</code></pre>
<p>def val_fn(val_data_loader, model, optimizer, device, scheduler):<br>
model.eval()<br>
loss_ = 0<br>
for data in tqdm(val_data_loader, total = len(val_data_loader)):<br>
for i, j in data.items():<br>
data[i] = j.to(device)<br>
<em>, loss = model(**data)<br>
loss</em> += loss.item()<br>
return loss_ / len(val_data_loader)</p>
<p>PS :  the dataset is so inbalanced with the target class hardly being 10% of the labels totality</p>