akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Distributed recv function removes grad_fn of communicated tensor

randomuserT: <p>I am using pytorch distributed api with gloo as backend (device=cpu) to send activations between a client process to a server process.</p>
<p>However, I find that somehow the grad_fn property of the activations tensor is no longer present when the tensor is received on the server, while it is present at the client-side at the time of sending.<br>
I was under the impression that I can use pytorch distributed API for this very use case of also including the grad_fn when sending tensors back and forth, is this not possible?</p>
<p>Client process;</p>
<pre data-code-wrap="python"><code class="lang-python">inputs, labels = data
inputs, labels = inputs.to(device), labels.to(device)

client_1_optimizer.zero_grad()
client_1_activations = client_1_model(inputs)
client_1_activations.retain_grad()
client_1_activations = client_1_activations.to(device)
# client_1_activations.grad_fn is &lt;MaxPool2DWithIndicesBackward0 object at 0x00000158932CC7C0&gt; here
dist.send(tensor=client_1_activations, dst=0)
</code></pre>
<p>Server process;</p>
<pre data-code-wrap="python"><code class="lang-python"># Reserving memory for the incoming activations tensor
client_1_activations = torch.zeros((4, 6, 14, 14), requires_grad=True)
client_1_activations = client_1_activations.to(device)
dist.recv(tensor=client_1_activations, src=1)

# Somehow client_1_activations .grad_fn is None here?
</code></pre>
<p>However, when following the example that can be found at <a href="https://medium.com/@esaliya/pytorch-distributed-with-mpi-acb84b3ae5fd" rel="noopener nofollow ugc">url</a>, the received tensor does in fact still contain the grad_fn, as I was expecting as well;</p>
<pre><code class="lang-auto">import os

import torch
import torch.distributed as dist
from torch.multiprocessing import Process


def run(rank):
    print(f"I am {rank}")

    device = torch.device("cpu")

    tensor = torch.zeros(1, requires_grad=True)
    tensor = tensor.mean()
    tensor = tensor.to(device)

    if rank == 0:
        # Send the tensor to process 1
        print("sending: {}".format(tensor.grad_fn))
        dist.send(tensor=tensor, dst=1)
        print("sent")
    else:
        # Receive tensor from process 0
        dist.recv(tensor=tensor, src=0)
        print("received tensor {}".format(tensor.grad_fn))

def init_process(rank, size, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)
    run(rank)

if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        p = Process(target=init_process, args=(rank, size))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
</code></pre>
<p>What am I doing wrong here? Any help would be much appreciated</p>
Torchvision::nms not found

squareroot: <p>I have compiled example from torchvision(0.17.2) library (hello-world)<br>
and add  strings to main function.</p>
<pre><code class="lang-auto">  auto&amp; ops = torch::jit::getAllOperatorsFor(torch::jit::Symbol::fromQualString("torchvision::nms"));
  std::cout &lt;&lt; "ops size: " &lt;&lt; ops.size() &lt;&lt; std::endl;
  std::cout &lt;&lt; "-----------------------------" &lt;&lt; std::endl;
</code></pre>
<p>But application return zero size vector.<br>
Where I’m wrong ?</p>
<p>Full code is below:</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#include &lt;torchvision/vision.h&gt;

int main() {
  torch::DeviceType device_type;
  device_type = torch::kCPU;
  auto&amp; ops = torch::jit::getAllOperatorsFor(torch::jit::Symbol::fromQualString("torchvision::nms"));
  std::cout &lt;&lt; "ops size: " &lt;&lt; ops.size() &lt;&lt; std::endl;
  std::cout &lt;&lt; "-----------------------------" &lt;&lt; std::endl;
  return 0;
  torch::jit::script::Module model;
  try {
    std::cout &lt;&lt; "Loading model\n";
    // Deserialize the ScriptModule from a file using torch::jit::load().
    model = torch::jit::load("resnet18.pt");
    std::cout &lt;&lt; "Model loaded\n";
  } catch (const torch::Error&amp; e) {
    std::cout &lt;&lt; "error loading the model\n";
    return -1;
  } catch (const std::exception&amp; e) {
    std::cout &lt;&lt; "Other error: " &lt;&lt; e.what() &lt;&lt; "\n";
    return -1;
  }

  // TorchScript models require a List[IValue] as input
  std::vector&lt;torch::jit::IValue&gt; inputs;

  // Create a random input tensor and run it through the model.
  inputs.push_back(torch::rand({1, 3, 10, 10}));
  auto out = model.forward(inputs);
  std::cout &lt;&lt; out &lt;&lt; "\n";

  if (torch::cuda::is_available()) {
    // Move model and inputs to GPU
    model.to(torch::kCUDA);

    // Add GPU inputs
    inputs.clear();
    torch::TensorOptions options = torch::TensorOptions{torch::kCUDA};
    inputs.push_back(torch::rand({1, 3, 10, 10}, options));

    auto gpu_out = model.forward(inputs);
    std::cout &lt;&lt; gpu_out &lt;&lt; "\n";
  }
}
</code></pre>

squareroot: <p>I have found solution for cmake here: <a href="https://github.com/facebookresearch/detectron2/blob/main/tools/deploy/CMakeLists.txt" class="inline-onebox" rel="noopener nofollow ugc">detectron2/tools/deploy/CMakeLists.txt at main · facebookresearch/detectron2 · GitHub</a><br>
But how to solve problem in qmake ?</p>
NAdamW and Demon optimizers

bavshehata: <p>Hey everyone,</p>
<p>While looking deeper into optimizers, I wanted to train my model using NAdamW ( a mix between NAdam and AdamW but I could not find any pytorch implementations, only a <a href="https://github.com/OverLordGoldDragon/keras-adamw" rel="noopener nofollow ugc">keras implementation</a>. I’d appreciate any help to implement it.</p>
<p>As for DEMON (Decaying Momentum), I did find a <a href="https://github.com/JRC1995/DemonRangerOptimizer" rel="noopener nofollow ugc">pytorch implementation</a> that looks promising, but I could not find any results or comparisons. Anyone tried using it before or can point me to another implementation?</p>

yermandy: <p>See <code>decoupled_weight_decay</code> in <a href="https://pytorch.org/docs/stable/generated/torch.optim.NAdam.html" class="inline-onebox" rel="noopener nofollow ugc">NAdam — PyTorch 2.2 documentation</a></p>
<pre><code class="lang-auto">decoupled_weight_decay (bool, optional) – whether to use decoupled weight decay as in AdamW to obtain NAdamW (default: False)
</code></pre>
When to use dist.all_gather()

Shuwei_Li: <p>Hi,</p>
<p>Thanks for reading this post. Recently I got access to multiple gpus so I started to use distributed. I got a few questions regarding to all_gather() function:</p>
<ol>
<li>
<p>If I do not use it, does it mean that my models will be updated separately on different gpus?</p>
</li>
<li>
<p>If I need to use it, can I just simply use it on the computed loss? In this way, I can get a conclusion version of loss which can help me update all the models on different gpus in the same way?</p>
</li>
<li>
<p>What about the BN layer? If I only use it on the loss, will the BN layer or other normalization layer updated correctly?</p>
</li>
</ol>
<p>Thanks for your time!</p>

ptrblck: <p>It seems you want to manually apply a distributed workflow using distributed operations.<br>
I’m unsure why and would recommend starting with e.g. <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html"><code>DistributedDataParallel</code></a> first as it would take care of the gradient reduction etc. for you.</p>
What is the difference between register_parameter (requires_grad=False) and register_buffer in PyTorch?

mamon: <p>During deep learning Training, the update of declared values with this variable is done under with torch.no _grad().</p>
<p>When learning with ddp, it’s the process of obtaining the average for each batch sample at each gpu, so no synchronization is required. Is there a difference between the two? register_buffer performs worse. It’s not like gradient is flowing, so I don’t know why the performance difference is significant.</p>
<p>example in model init</p>
<ol>
<li>self.matrix = nn.Parameter(torch.zeros(197, 192).cuda(), requires_grad=False)</li>
<li>self.matrix = torch.zeros(197, 192).cuda()</li>
<li>self.register_buffer(‘matrix’, torch.zeros(197, 192))</li>
</ol>
<p>1 and 3 are same performence, but 2 is different</p>

ptrblck: <p>Cross-post from <a href="https://discuss.pytorch.org/t/nn-register-buffer-vs-torch-declaration-performance-difference/200691">here</a> whithout a followup.</p>
FSDP multi-node comms overhead

gbarello: <p>I am trying to use FSDP HYBRID_SHARD for multi-node training and I am seeing unexpectedly large comms overheads. I am currently trying to diagnose and hoping someone here can help understand what needs to be changed.</p>
<p>Not sure what the best way to share my pytorch profiler trace is… here is a screenshot. You can see there is a long, green, block (stream 24) of NCCL AllReduce_Sum_bf16 ops that is the bottleneck - I believe these are the cross-node gradient all-reduce operations? So I guess, a few questions:</p>
<ol>
<li>I thought in FSDP the gradient all-reduce was executed as a reduce-scatter followed by an all-gather. I see these in smaller blocks throughout the backwards pass, I assume these are the intra-node gradient syncs. Why does cross-node not use the same scheme?</li>
<li>We are definitely not seeing the bandwidth utilized the way we expect. This is an 11B T5 model on a cluster that should have ~1Tb inter-node bandwidth (5x 200Gbps NICs). Using <a href="https://github.com/NVIDIA/nccl-tests" rel="noopener nofollow ugc">nccl-tests</a> we measured ~600 Gb of useable bandwidth - plenty to make this workload GPU limited, but if I convert the time these all-reduces take into a Gbps number its about 170 Gbps - maybe we’re only using one NIC for some reason? Why would this be the case in our torch code but not when running the nccl tests?</li>
</ol>
<p>Anyway, I know there are a lot of environment variables and other configs that could be causing the bottleneck, but if anyone here has ideas that would be amazing - looking forward to being able to fully utilize our resources!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1cf0da75d3ba8ee2ba1c6587700eb9c30a8a2a7b.png" data-download-href="https://discuss.pytorch.org/uploads/default/1cf0da75d3ba8ee2ba1c6587700eb9c30a8a2a7b" title="Screenshot 2024-04-15 at 8.00.36 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1cf0da75d3ba8ee2ba1c6587700eb9c30a8a2a7b_2_690x178.png" alt="Screenshot 2024-04-15 at 8.00.36 AM" data-base62-sha1="481ouPwAoKMppN4DkxEIO39rYQr" width="690" height="178" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1cf0da75d3ba8ee2ba1c6587700eb9c30a8a2a7b_2_690x178.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1cf0da75d3ba8ee2ba1c6587700eb9c30a8a2a7b_2_1035x267.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1cf0da75d3ba8ee2ba1c6587700eb9c30a8a2a7b_2_1380x356.png 2x" data-dominant-color="E9ECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-04-15 at 8.00.36 AM</span><span class="informations">2838×734 249 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Obscure error message when trying to export a compiled function

thomasryck: <p>I am trying to compile and save a function which is using Pytorch as a way to do some matrix computation on GPU.</p>
<pre><code class="lang-auto">def grad(self,input : torch.Tensor, matmul_result : torch.Tensor = None) -&gt; torch.Tensor:
        import os
        if matmul_result is not None:
            inputs = [matmul_result]
            if not os.path.isfile("trt_model.ts"):
                self.grad_aux_compile = torch.compile(self.get_grad_aux_to_compile, backend="torch_tensorrt")
                trt_exp_program = torch.export.export(self.get_grad_aux_compile, args=tuple(inputs))
                torch.export.save(trt_exp_program, "trt_model.ep")
            elif self.grad_aux_compile is None:
                self.grad_aux_compile = torch.jit.load("trt_model.ep")   
            tmp = self.grad_aux_compile(*inputs)
        else:
            tmp = self.get_grad_aux(input, matmul_result)
        gradient_size = tmp.size()
        if (len(gradient_size) &gt; 1):
            n = gradient_size[1]
        else:
            n = gradient_size[0]
        return self.add_dual_pred_tensor(tmp, None, 1.0/n)
</code></pre>
<p>When the code is running, the following line of code is causing, the next error:</p>
<blockquote>
<p>trt_exp_program = torch.export.export(self.get_grad_aux_compile, args=tuple(inputs))</p>
</blockquote>
<blockquote>
<p>AssertionError: graph-captured input <span class="hashtag-raw">#2</span>, of type &lt;class ‘torch.Tensor’&gt;, is not among original inputs of types: (&lt;class ‘torch.Tensor’&gt;)</p>
</blockquote>
<p>I do not know where to start to fix this. I use the same input as I use to run the command so I am quite lost. Is there a way to see the graph ? I would like to understand what is the expected input.</p>
<p>Can it be because the method is using a class attribute and is not static ? The method can be runned compiled.</p>
<p>I also tried torch.jit which did not improve my performance.<br>
I also saw that there are other methods if you compile module and not function.</p>
<p>Is there another way to export the compiled function?</p>
How to set "Target" for captum for multi-label classification

Abdulfatai_Lawal: <p>Hello Everyone,</p>
<p>Most of the tutorials (<a href="https://colab.research.google.com/drive/1fLJbFPz0yMCQg81DdCP5I8jXw9LoggKO?usp=sharing#scrollTo=PmpNLlsC3GKG" rel="noopener nofollow ugc">here</a> and <a href="https://github.com/shenhao-stu/dgl_captum" rel="noopener nofollow ugc">here</a> I have seen with regards to graph explainability using captum pretty much have the target set to 0. I understand this is a feature of the dataset.</p>
<p>I need help to understand what would be an appropriate target for a multi-label classification model ( each example can belong to multiple classes simultaneously. The model predicts the probability of each class independently, and multiple classes can have non-zero probabilities for a single example).</p>
<p>My aim in using captum is to understand why the model makes such predictions, so I am not quite sure exactly how to set the target.</p>
<p>I have gone through the note added to the website, but not quite sure how to proceed.</p>
<p>Please let me know if there is anything else I need to provide to better explain the question.</p>
<p>Thank you in anticipation of your response.</p>
Infinite recursion when wrapping torch.nn.Module via `__getattr__`

schridde: <p>I wrote a simple wrapper to add special methods to a given neural network.<br>
While the below implementation works well for general objects e.g. strings, lists etc. I get a RecursionError when applying it to a <code>torch.nn.Module</code>. It seems that in the latter case the call <code>self.instance</code> inside the <code>__getattr__</code> method is unsuccessful which hence falls back on <code>__getattr__</code> again leading to the infinite loop (I also tried <code>self.__dict__['instane']</code> without luck).</p>
<p>What is going on here? What is the correct way of doing what I want?</p>
<pre><code class="lang-auto">import torch

class MyWrapper(torch.nn.Module):
    def __init__(self, instance):
        super().__init__()
        self.instance = instance

    def __getattr__(self, name):
        print("trace", name)
        return getattr(self.instance, name)

# Working example
obj = "test string"
obj_wrapped = MyWrapper(obj)
print(obj_wrapped.split(" ")) # trace split\n ['test', 'string']

# Failing example
net = torch.nn.Linear(12, 12)
net.test_attribute = "hello world"
b = MyWrapper(net)

print(b.test_attribute) # RecursionError: maximum recursion depth exceeded
b.instance # RecursionError: maximum recursion depth exceeded
</code></pre>
Torch::where doesn`t work

fakemade: <p>Hi, i have the follow code:</p>
<p>torch::Tensor a = torch::tensor({1, 2, 3, 6});<br>
torch::Tensor b = torch::tensor({0, 2, 3, 4});<br>
auto result = torch::cat({a,b});<br>
auto resultU = at::_unique2(result,true,true, true);<br>
auto acat = std::get&lt;0&gt;(resultU);<br>
auto cond = std::get&lt;2&gt;(resultU);<br>
auto wh = torch::where(cond &gt; 1);<br>
auto intersection = acat.index({1});<br>
std::cout &lt;&lt; “acat\n” &lt;&lt; acat &lt;&lt; “\ncond\n” &lt;&lt; cond &lt;&lt; “\nwh\n” &lt;&lt; wh &lt;&lt; “\ninter\n” &lt;&lt; intersection &lt;&lt; std::endl;</p>
<p>and torch::where doesn’t work. I expect at::Tensor as return value, but get vector of at::Tensor . If i use torch::where like <code>auto wh = torch::where(cond &gt; 1, cond,1);</code> i just get the copy of tensor cond. What did i do wrong?</p>
Does pytorch 2.2.2 support cuda 12.4?

Pranjal.Pok7: <p>I was trying to download  pytorch 2.2.2 with cuda 12.1 but ran with an issues with nvidia drivers.  Does pytorch 2.2.2 supports cuda 12.4 if it is build from source or I have to stick with cuda 12.2(heard it was compatible with pytorch 2.2.2 if it was built from source)</p>

ptrblck: <p>Yes, you can build the current PyTorch release from source with CUDA 12.4 while the binaries are being worked on.</p>
It seems hard to get a visual view of batch data in whole dataloader

hitbuyi: <p>The bebow is code snippnet,  for every epoch, get every batch from datasetloader.</p>
<pre><code class="lang-auto">        for i, data_batch in enumerate(self.data_loader):
            self._inner_iter = i
            self.call_hook('before_train_iter')
            self.run_iter(data_batch, train_mode=True, **kwargs)
</code></pre>
<p>self.data_loader should be viewed by IDE such vscode, pycharm, and  data_batch should be appeared in self.data_loader visually, BUT I can’t  see data_batch in self.data_loader in pycharm, why?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/361c8baa8337a918032b2a59e612e5e00955d7a8.png" data-download-href="https://discuss.pytorch.org/uploads/default/361c8baa8337a918032b2a59e612e5e00955d7a8" title="Screenshot from 2024-04-15 16-58-43"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/361c8baa8337a918032b2a59e612e5e00955d7a8_2_690x312.png" alt="Screenshot from 2024-04-15 16-58-43" data-base62-sha1="7IGWp4FMNRYcgVgQ6KdeFnGFCKQ" width="690" height="312" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/361c8baa8337a918032b2a59e612e5e00955d7a8_2_690x312.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/361c8baa8337a918032b2a59e612e5e00955d7a8_2_1035x468.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/361c8baa8337a918032b2a59e612e5e00955d7a8_2_1380x624.png 2x" data-dominant-color="3D444D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-04-15 16-58-43</span><span class="informations">1776×804 196 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
RBF kernel for Conv2D

Filippo_Portera: <p>Good morning,<br>
we would like to implement a kernel/filter operation that uses a Radial Basis Function dot product <a href="https://en.wikipedia.org/wiki/Radial_basis_function_kernel" rel="noopener nofollow ugc">RBF kernel</a>, to plug it in the Conv2d function.<br>
We don’t how where to start since the .cpp/.cu code is quite complex.<br>
At present we tried to compile torch-2.4.0a0+git3c09c6b from scratch, following the README.md, but we had a problem with ninja:</p>
<p>filippo@filippoPC:~/Downloads/pytorch-main/pytorch$ python3 setup.py develop<br>
– Building version 2.4.0a0+git3c09c6b<br>
cmake --build . --target install --config Release – -j 12<br>
ninja: error: loading ‘build.ninja’: No such file or directory</p>
<p>Many thanks for your time and best regards,<br>
Filippo Portera</p>

Filippo_Portera: <aside class="quote no-group" data-username="Filippo_Portera" data-post="1" data-topic="200853">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/filippo_portera/48/11874_2.png" class="avatar"> Filippo_Portera:</div>
<blockquote>
<p>python3 setup.py develop</p>
</blockquote>
</aside>
<p>I solved the issue following another tutorial.</p>
Pytorch Docker Installation issue

Awais_Muhammad: <p>I am trying to install pytorch 1.12.1 in a docker container.<br>
My host system has the following configurations<br>
Sun Apr 14 19:51:17 2024<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 470.82.01    Driver Version: 470.82.01    CUDA Version: 11.4     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|                               |                      |               MIG M. |<br>
|===============================+======================+======================|<br>
|   0  NVIDIA RTX A5000    On   | 00000000:01:00.0 Off |                  Off |<br>
| 30%   46C    P8    22W / 230W |     13MiB / 24256MiB |      0%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+<br>
|   1  NVIDIA RTX A5000    On   | 00000000:21:00.0 Off |                  Off |<br>
| 30%   37C    P8    16W / 230W |     13MiB / 24256MiB |      0%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+<br>
|   2  NVIDIA TITAN V      On   | 00000000:4C:00.0 Off |                  N/A |<br>
| 35%   51C    P8    29W / 250W |     73MiB / 12057MiB |      0%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+</p>
<p>I am using nvidia/cuda:11.6.1-devel-ubuntu20.04 image and i installed pytorch from pip using<br>
pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url <a href="https://download.pytorch.org/whl/cu116" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu116</a></p>
<p>i run the container with --gpus all<br>
This is the problem now<br>
print(torch.cuda.is_available())<br>
i get this<br>
/usr/local/lib/python3.8/dist-packages/torch/cuda/<strong>init</strong>.py:88: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at …/c10/cuda/CUDAFunctions.cpp:109.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
False</p>
<p>I have tried a bunch of solution but none worked, has anyone faced a similar issue?</p>
Implementing a Keypoint Regression Dataset: Addressing Coordinate Format Errors

SimCan: <p>I would like to implement a dataset for the <em>Keypoint Regression</em> task. The images are single-channel (BW), while the coordinates of the key points are currently in the format of a list of lists: <code>[[x0, y0], [x1, y1]]</code>. This is the implementation of the dataset:</p>
<pre><code class="lang-auto">class PoseDataset(Dataset):
    def __init__(self, image_paths, annotations, transform=None):
        self.image_paths = image_paths
        self.annotations = annotations
        self.transform = transform

    def __len__(self):
        return len(self.image_paths)

    def __getitem__(self, idx):
        image_path = self.image_paths[idx]
        image = Image.open(image_path)

        k_x0 = self.annotations[idx][0]
        k_y0 = self.annotations[idx][1]
        k_x1 = self.annotations[idx][2]
        k_y1 = self.annotations[idx][3]
        
        keypoints = torch.tensor([[k_x0, k_y0], [k_x1, k_y1]]).float()

        if self.transform:
            image, keypoints = self.transform(image, keypoints)

        return image, keypoints

transform = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.RandomVerticalFlip(),
    transforms.ToTensor(),
])

dataset = PoseDataset(image_paths, annotations, transform=transform)
dataloader = DataLoader(dataset, batch_size=BS, shuffle=True)

next(iter(dataloader))
</code></pre>
<p>This, instead, is the resulting error from the aforementioned code.</p>
<pre><code class="lang-auto">Cell In[10], line 31, in PoseDataset.__getitem__(self, idx)
     28 keypoints = [[k_x0, k_y0], [k_x1, k_y1]]
     30 if self.transform:
---&gt; 31     image, keypoints = self.transform(image, keypoints)
     33 return image, keypoints

TypeError: Compose.__call__() takes 2 positional arguments but 3 were given
</code></pre>
<p>I would like to know why I am getting this error.</p>
<ul>
<li>Is it perhaps due to the format of the coordinates?</li>
<li>If so, in what format should they be?</li>
</ul>
Back propagating loss function separately

DHARMENDRA_PRAJAPAT: <p>I have fine tuned a pre-trained model on multiple task.<br>
Optimising the model with multiple loss functions for different tasks</p>
<p>how these approach are different in optimising the model.<br>
Approach 1: → L = L1 + alpha<em>L2 + beta * L3; back propagating weighted total loss<br>
Approach 2: → back propagating losses separately like, L1, alpha</em>L2 and then beta*L3</p>
<p>kindly explain these two way of back propagating losses</p>

ptrblck: <p>Both will yield the same results but the second approach would call <code>backward</code> multiple times and would thus add some runtime overhead.</p>

DHARMENDRA_PRAJAPAT: <p>thank you for the reply.<br>
I have fine tuned the model with both approach compromising the runtime overhead for second approach but i get different results. what could be the possible reason?<br>
I have back propagated loss in approach_2 like this → <br>
loss_1.backward(retain_graph=True)<br>
loss_2.backward(retain_graph=True)<br>
loss_3.backward()</p>

ptrblck: <p>I don’t knot but would recommend to compare the actual loss values first, then the gradients etc. If you get stuck, minimize the code and post an executable code snippet reproducing the issue.</p>
Fine-tune a pretrained model from github

Najeh_Nafti: <p>how can I do finetuning in pytorch of a pretrained model in github on my own dataset? EXample, I need to fine-tune BigGAN on my own dataset</p>

koszmit: <p>Hey, did you manage to do it? I am also trying to fine-tune BigGAN on my own dataset, but I am struggling with it.</p>
Extremely Imbalanced and Gapped Dataset in Regression Problem

abdulfaqihalm: <p>Currently I am working with a biological dataset with a range of 0-to-1 to do a multi-task regression with Deep Learning. However, this dataset has an empty gap in the range 0 to 0.2 (however there are several datapoints which are 0s) which can’t be interpolated due to biological reason.  I also can’t randomly upsampling or downsampling my data with the same reason.  In addition, there is also high tendency of a skewed data around 0.3. Until now I have only tried to train my model with standard HuberLoss and MSELoss and tried several type of models. However, I still don’t get “good” prediction result with correlation of only (~0.65). I came across several articles (such as <a href="https://cdn.aaai.org/Workshops/2000/WS-00-05/WS00-05-001.pdf" rel="noopener nofollow ugc">this one</a>) about weighted MSE but most of them talking about classification problem rather than regression. In addition, I read an <a href="https://towardsdatascience.com/strategies-and-tactics-for-regression-on-imbalanced-data-61eeb0921fca" rel="noopener nofollow ugc">article about imbalanced dataset in regression</a> but seems like they did interpolation for the data.</p>
<p>My question: is it possible to improve my model by weightning the Loss function? If so how could I approach it?<br>
Many thanks in advance!</p>
Calculating the spatial_scale for ROIAlign

Monami_Bhuyan: <p>I want to implement <code>torchvision.ops.RoIAlign</code> to extract the features from a custom CNN model. I need assistance in finding the <code>spatial_scale</code> for ROIAlign.</p>
<p>Specifically, I want to ROI Align on a layer that has a feature map with spatial dimensions of <strong>24 x 24</strong>. I already have my bounding box coordinates. However, they are defined on the scale of the original image which is <strong>1080x1920</strong>.</p>
<p>Additionally, I am not sure if this information is useful but while training my CNN model, I did resize my images to 224 x 224. However, the bounding box coordinates were identified using a different model for object detection. Thus there is no weight sharing between the two models.</p>
<p>Now I am not sure about the <code>spatial_scale</code> since the height and width of the spatial dimensions for the scale in which the bounding box/ROIs were extracted aren’t equal.<br>
The following code demonstrates what I think  I should do. But I am not confident if this is correct:</p>
<pre><code class="lang-auto">feature_map_size = 28
original_frame_height = 1080
original_frame_width = 1920
spatial_scale_height = feature_map_size / original_frame_height
spatial_scale_width = feature_map_size / original_frame_width
spatial_scale = min(spatial_scale_height, spatial_scale_width)
</code></pre>
<p>Any help/input would be much appreciated. Thanks in advance!</p>
Using FSDP for only part of a model

divinho: <p>Does FSDP have support for this? I only want to shard certain parts of the model. I know <code>FullyShardedDataParallel</code> has an <code>ignored_modules</code> arg but I don’t understand how that is supposed to work if one has nn.Linear layers that one sometimes wants to be sharded and sometimes not.</p>
<p>I was wondering if one could manually call <code>torch.distributed.fsdp.wrap</code> on individual parts of the model but some initial tests seem to indicate that approach does not work (seems the root module needs to be of type <code>FullyShardedDataParallel</code>)?</p>
<p>Related to that is the question of what happens to the outer FSDP unit during training. This is referenced in the <a href="https://pytorch.org/tutorials/intermediate/FSDP_adavnced_tutorial.html#transformer-wrapping-policy" rel="noopener nofollow ugc">FSDP advanced tutorial on transfomer wrapping policy</a>. Does this get unloaded while other (child) FSDP units are running? Or is this always kept in unsharded state?</p>

GuWei007: <p>Can you try these two parameters of FSDP?<br>
auto_wrap_policy and ignored_modules</p>
OSError for CohereForAI/c4ai-command-r-v01 LLM Model

ahkarami: <p>Hi,<br>
When I want to use and load the 4-bit version of <code>CohereForAI C4AI Command-R model</code>, I got following error:</p>
<pre><code class="lang-auto">OSError: CohereForAI/c4ai-command-r-v01 does not appear to have a file named configuration_cohere.py. Checkout 'https://huggingface.co/CohereForAI/c4ai-command-r-v01/main' for available files.
</code></pre>
<p>I follow as below:<br>
<a href="https://huggingface.co/CohereForAI/c4ai-command-r-v01-4bit" rel="noopener nofollow ugc">https://huggingface.co/CohereForAI/c4ai-command-r-v01-4bit</a></p>
<p>Would you please help me to address this issue?<br>
<a href="https://huggingface.co/CohereForAI/c4ai-command-r-v01-4bit/discussions/7" rel="noopener nofollow ugc">Reference</a></p>
<p>My code is as below:</p>
<pre><code class="lang-auto"># pip install 'transformers&gt;=4.39.1' bitsandbytes accelerate
import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

# ** prepare input texts:
input_text_english_test = "What is ML?"

# ** Load the model and tokenizer:
checkpoint = "CohereForAI/c4ai-command-r-v01-4bit"
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
max_new_tokens = 256
temperature = 0.3

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    # low_cpu_mem_usage = True,
    bnb_4bit_compute_dtype=torch.bfloat16    
)

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, quantization_config=quantization_config, device_map="auto")

# ***** First Test:
print(" ***** First Test  ***** ")
messages = [{"role": "user", "content": input_text_english_test}]
input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(device)
## &lt;BOS_TOKEN&gt;&lt;|START_OF_TURN_TOKEN|&gt;&lt;|USER_TOKEN|&gt;Hello, how are you?&lt;|END_OF_TURN_TOKEN|&gt;&lt;|START_OF_TURN_TOKEN|&gt;&lt;|CHATBOT_TOKEN|&gt;

gen_tokens = model.generate(
    input_ids, 
    max_new_tokens=max_new_tokens, 
    do_sample=True, 
    temperature=temperature,
    )

gen_text = tokenizer.decode(gen_tokens[0])
print(gen_text)
print(" --------- Print Generated Text --------- ")
output_text = tokenizer.decode(gen_tokens[0], skip_special_tokens=True).strip()

final_result = output_text[len("&lt;|START_OF_TURN_TOKEN|&gt;&lt;|USER_TOKEN|&gt;") + 
                           len(input_text_english_test)+
                           len("&lt;|START_OF_TURN_TOKEN|&gt;&lt;|CHATBOT_TOKEN|&gt;"):].strip()
print(final_result)
print(" -------------------- \n")

</code></pre>

GuWei007: <p>This is not a PyTorch issue; it seems to be a problem with the environment configuration. Please ask in the Hugging Face community for assistance.</p>
Can we use tanh activation function to detect outliers?

laro: <p>Can we use tanh activation function to detect outliers ?</p>
<p>Does the following image is true for dataset with outliers (after training model with tanh activation function) ?</p>
<p><a href="https://i.stack.imgur.com/deYEG.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/deYEG.png" alt="enter image description here" width="490" height="301"></a></p>
The training and validation

dozop: <p>have a minor concern regarding the training and validation phase. Assuming that the loss decreases for N iterations. However, at iteration N+1, the loss increases. Should I use the model parameters from iteration N or N+1 before continuing the training and validation phase.</p>
<pre><code>if val_losses[-1] &lt; best_val_loss:
    best_val_loss = val_losses[-1]
    counter=0
    best_model_wts = copy.deepcopy(model.state_dict())
else:
    counter+=1
    print('Validation loss has not improved since: {:.3f}..'.format(best_val_loss), 'Count: ', str(counter))
    if counter &gt;= early_stop_counter:
        print('Early Stopping Now!!!!')
        model.load_state_dict(best_model_wts)
        break
</code></pre>
Loading traced pytorch model to C++

Zarina_Maks: <p>Will be grateful for any help <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>I’m having a problem with using pytorch 2.2.1+cpu  pretrained model in C++ via TorchLib 2.2.2 (cpu version). I’m also using Visual Studio 2022. I’ve added Torchlib to Cmake and it successfully downloads the model and works until line:</p>
<pre><code class="lang-auto">`at::Tensor output = module.forward(inputs).toTensor();`
</code></pre>
<p>After this line it crushes with the error: <code> INTEL MKL ERROR: The specified module could not be found. mkl_avx2.1.dll. Intel MKL FATAL ERROR: Cannot load mkl_avx2.1.dll or mkl_def.1.dll.  and I didn't figured out what is wrong with TorchLib here.</code></p>
<p>The whole code is here:</p>
<pre><code class="lang-auto">`#include &lt;torch/script.h&gt;
...
    string path = "traced_energynn_model.pt";
    torch::jit::script::Module module;
    try {
        // Deserialize the ScriptModule from a file using torch::jit::load().
        module = torch::jit::load(path);
    }
    catch (const c10::Error&amp; e) {
        std::cerr &lt;&lt; "error loading the model\n";
        return -1;
    }
    std::cout &lt;&lt; "Model traced_energynn_model loaded fine\n";

    // Create a vector of inputs.
    std::vector&lt;torch::jit::IValue&gt; inputs;
    inputs.push_back(torch::randn({ 1, 2 }));

    // Execute the model and turn its output into a tensor.
    at::Tensor output = module.forward(inputs).toTensor();
    std::cout &lt;&lt; output &lt;&lt; "\n";`
</code></pre>
<p>Seems like the problem is with some part of the lib related to python, but it’s strange because the model loads to C++ by LibTorch successfully.</p>

ptrblck: <p>Could you create a <a href="https://github.com/pytorch/pytorch/issues">GitHub issue</a> so the code owners could take a look at it, please?</p>

Zarina_Maks: <p>Sure, here it is <a href="https://github.com/pytorch/pytorch/issues/124009" class="inline-onebox" rel="noopener nofollow ugc">Loading traced pytorch model to C++ · Issue #124009 · pytorch/pytorch · GitHub</a>, thank you!</p>
Performance gap between torch 2.2.2 and 1.12.0

ShadowBbBb: <p>Hi, Pytorch’s team.<br>
I am training models on 4 NVIDIA RTX 3090 GPUs. When I tried to upgrade the existing PyTorch version (1.12.0) to a newer version (2.2.2), the memory usage on each GPU increased by approximately 30% (from 16GB to 21GB), and the training speed also slowed down (from 1.5iters/s to 1.2s/iters).<br>
What aspects should I focus on to solve the above problems?</p>

ptrblck: <p>Profile the code to narrow down parts of the code causing the slowdown.</p>
Defining custom internal activation functions on LSTM

Danibahnani: <p>Hi, I’m trying to implement custom internal activation function for LSTM. Essentially I’m trying to replace the LSTM cell’s tanh function with torch.sin. I’m very new to machine learning and PyTorch, and from what I’ve understood you can implement custom activation functions between layers, but is there any way to use custom function in the cells themselves?</p>

deborah621: <aside class="quote no-group" data-username="Danibahnani" data-post="1" data-topic="200635">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/danibahnani/48/66316_2.png" class="avatar"> Danibahnani:</div>
<blockquote>
<p>I’m trying to implement custom internal activation function for LSTM. Essentially I’m trying to replace the LSTM cell’s tanh function with torch.sin. I’m very new to machine learning and PyTorch, and from what I’ve understood you can implement custom activation functions between layers, but is there any way to use custom function in the cells themselves?</p>
</blockquote>
</aside>
<p>You can create custom activation functions in PyTorch and use them in your LSTM cells. To replace the tanh activation function in LSTM cells with your custom function (e.g., torch.sin), you’ll need to modify the LSTM cell implementation.The LSTM cell in PyTorch has default activations: activation=“tanh” and recurrent_activation=“sigmoid”.</p>
<p>import torch<br>
import torch.nn as nn</p>
<p>class CustomLSTMCell(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden_size):<br>
super().<strong>init</strong>()<br>
self.input_size = input_size<br>
self.hidden_size = hidden_size<br>
self.weight_ih = nn.Parameter(torch.Tensor(4 * hidden_size, input_size))<br>
self.weight_hh = nn.Parameter(torch.Tensor(4 * hidden_size, hidden_size))<br>
self.bias_ih = nn.Parameter(torch.Tensor(4 * hidden_size))<br>
self.bias_hh = nn.Parameter(torch.Tensor(4 * hidden_size))<br>
# Initialize weights and biases here</p>
<pre><code>def forward(self, input, hx):
    # Compute LSTM cell operations using custom activation functions
    gates = torch.matmul(input, self.weight_ih.t()) + self.bias_ih + torch.matmul(hx, self.weight_hh.t()) + self.bias_hh
    ingate, forgetgate, cellgate, outgate = gates.chunk(4, 1)
    cellgate = torch.sin(cellgate)  # Custom activation (replace with torch.sin)
    cy = (forgetgate * hx) + (ingate * cellgate)
    hy = torch.tanh(cy)  # Custom activation (replace with torch.tanh)
    return hy, cy
</code></pre>
How to Pytorch call CUDA kernel and obtain input parameters

Kevin1: <p>Hi,</p>
<p>I would like to know is there a way/suggestions on how to know the CUDA library call/kernel call is invoked in Pytorch? For example, for general matrix matrix multiplication, an automated way to obtain the matrix input dimension and sparsity when the pytorch high level API call to low level API which further translated into library call. Where can I intercept the input information and where is the call to the exact GEMM routine.</p>

ptrblck: <p>The call setup can be found in <a href="https://github.com/pytorch/pytorch/blob/9d37cefcb0c1e867efcfe15c6c0d734ab3b14c84/aten/src/ATen/native/cuda/Blas.cpp"><code>aten/src/ATen/native/cuda/Blas.cpp</code></a> and the actual calls are in <a href="https://github.com/pytorch/pytorch/blob/9d37cefcb0c1e867efcfe15c6c0d734ab3b14c84/aten/src/ATen/cuda/CUDABlas.cpp"><code>aten/src/ATen/src/cuda/CUDABlas.cpp</code></a>.</p>

Kevin1: <p>Thank you! I got a follow-up question. After reading the code, I did not find the handling of sparse matrix dense multiplication. For example, I see the calling to cuSparse library using Nvidia profiling tool. So, which file should I look into to find out the details of handling sparse matrix dense matrix multiplication.</p>

ptrblck: <p>The cuSPARSE calls should be defined in <a href="https://github.com/pytorch/pytorch/blob/82daf981510319281518434c76be198e309a872b/aten/src/ATen/native/sparse/cuda/SparseMatMul.cu"><code>aten/src/ATen/native/sparse/cuda/SparseMatMul.cu</code></a> and <a href="https://github.com/pytorch/pytorch/blob/bacd2ced4f2b914dfab25c8b46a7f9621d5f54b5/aten/src/ATen/native/sparse/cuda/SparseCUDABlas.cpp"><code>aten/src/ATen/native/sparse/cuda/SparseCUDABlas.cpp</code></a>.</p>

Kevin1: <p>Thanks! I read the Aten directory and I do have a follow-up question: How to trace the library invocation (e.g. GEMM) to the Intel MKL on CPU or cublas/cusparse on GPU and record the tensor size of those library call. Reading code base to figure them out seems like an inefficient way. Is there a plug-in/tool to do it automatically and record the tensor size of those library call? I tried the pytorch profiler’s tracing, but it does not give the tensor size and seems does not give the function name.</p>
<p>I also noticed that there are some dispatching mechanism of a kernel from CPU and GPU like here: <a href="https://pytorch.org/tutorials/advanced/dispatcher.html" class="inline-onebox" rel="noopener nofollow ugc">Registering a Dispatched Operator in C++ — PyTorch Tutorials 2.0.0+cu117 documentation</a><br>
Will this be helpful to track kernel invocation and intercept the input matrix size for GEMM call?</p>
<p>Thanks again!</p>

puddingfjz: <p>May I ask which implementation of the csrmm will be called when I run ``W@X’', where W is a csr matrix and X is a dense matrix?</p>
PyTorch L2-norm between 2 tensors of different shapes

grid_world: <p>I have 2 tensors in PyTorch:</p>
<pre><code>a.shape, b.shape
# (torch.Size([1600, 2]), torch.Size([128, 2]))
</code></pre>
<p>I want to compute L2-norm distance between each of the 128 values in ‘b’ having 2-dim values from all 1600 values in ‘a’. Currently, I have an inefficient for loop to do it for each values in b as follows:</p>
<pre><code># Need to compute l2-norm squared dist b/w each b from a-
l2_dist_squared = list()

for bmu in bmu_locs:
    l2_dist_squared.append(torch.norm(input = a.to(torch.float32) - b, p = 2, dim = 1))

l2_dist_squared = torch.stack(l2_dist_squared)

# l2_dist_squared.shape
# torch.Size([128, 1600])
</code></pre>
<p>Is there a better way to do as a one liner?</p>

UMAR_MASUD: <p>Hi, you could use this functionality - <a href="https://pytorch.org/docs/stable/generated/torch.cdist.html" class="inline-onebox" rel="noopener nofollow ugc">torch.cdist — PyTorch 2.2 documentation</a> which calculates the distance between each vector in ‘b’ to each vector in ‘a’.</p>
<p>E.g. <code>torch.cdist(b, a, p=2)</code> will give (128, 1600) shape, where each row represents the L2-norm distances between the corresponding value in ‘b’ and all values in ‘a’.</p>
Backward(inputs=) doesn't work when the model is moved between devices

thanrl: <pre><code class="lang-auto">import torch
import torch.nn as nn

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.weight_mul = nn.Parameter(torch.randn(D,))
        self.weight = nn.Parameter(torch.randn(D,))

    def forward(self, x):
        x = x * self.temp_weight
        return x

torch.manual_seed(0)
D = 5

x = torch.randn(D,).cuda()
model = Model()

model.cuda()
model.temp_weight = model.weight * model.weight_mul

model.cpu(); model.cuda()
output = model(x)
output.sum().backward(inputs=[model.weight, model.weight_mul])

print("model.weight.grad", model.weight.grad)
print("model.weight_mul.grad", model.weight_mul.grad)
</code></pre>
<p>To my surprise, the <code>.grad</code> are None. There are two ways to get the <code>backward()</code> to work. One is to remove the <code>inputs</code> to <code>backward()</code>, the other is to remove <code>model.cpu(); model.cuda()</code>. But why?</p>

thanrl: <pre><code class="lang-auto">torch.manual_seed(0)
D = 5

x = torch.randn(D,).cuda()
weight_mul = torch.randn(D,).requires_grad_()
model = Model()

model.cuda()
model.temp_weight = model.weight * weight_mul.cuda()

model.cpu(); model.cuda()
output = model(x)
                
output.sum().backward(inputs=[model.weight, weight_mul])
# output.sum().backward()

print("model.weight.grad", model.weight.grad)
print("weight_mul.grad", weight_mul.grad)
</code></pre>
<p>In this case, <code>weight_mul</code> gets gradient but <code>model.weight</code> doesn’t. Removing the <code>inputs</code> both’d get gradients. The computation graphs according to <code>torchviz</code> are identical in both cases. So somehow <code>module.cpu()</code> is disconnecting the computation graph but not totally?</p>
<p>May be related: This works</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = x * self.weight
        return x

x = torch.randn(D,).cuda()
model = Model().cuda()
output = model(x)
model.cpu()
output.sum().backward()
</code></pre>
<p>But this gives a device mismatch error in the backward():</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = x * self.temp_weight
        return x

x = torch.randn(D,).cuda()
weight_mul = torch.randn(D,).cuda().requires_grad_()
model = Model().cuda()

model.temp_weight = model.weight * weight_mul
output = model(x)
model.cpu()

output.sum().backward()
</code></pre>
<p>unless the <code>model.cpu()</code> is removed.</p>