akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Problems with Importing PyTorch for Snake ML Algorithm w/ Patrick Loeber

Jamison_Ballou: <p>Hello. I am new to using PyTorch, and am trying to follow the tutorial for creating a snake ML algorithm (<a href="https://www.youtube.com/watch?v=L8ypSXwyBds" rel="noopener nofollow ugc">https://www.youtube.com/watch?v=L8ypSXwyBds</a>). I got the program to work on my Mac, but it simply does not work with my Chromebook running Linux. I am using VS Code to run both programs. The source code can be found here: <a href="https://github.com/patrickloeber/snake-ai-pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - patrickloeber/snake-ai-pytorch</a>. When I run the program, This is what appears in the terminal:<br>
<code>Hello from the pygame community. https://www.pygame.org/contribute.html Traceback (most recent call last):   File "/home/jamballou/PyGames/Snake ML/snake-ai-pytorch-main/agent.py", line 6, in &lt;module&gt;     from model import Linear_QNet, QTrainer   File "/home/jamballou/PyGames/Snake ML/snake-ai-pytorch-main/model.py", line 2, in &lt;module&gt;     import torch.nn as nn   File "/home/jamballou/.local/lib/python3.9/site-packages/torch/nn/__init__.py", line 1, in &lt;module&gt;     from .modules import *  # noqa: F403   File "/home/jamballou/.local/lib/python3.9/site-packages/torch/nn/modules/__init__.py", line 1, in &lt;module&gt;     from .module import Module   File "/home/jamballou/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 8, in &lt;module&gt;     from torch._prims_common import DeviceLikeType ModuleNotFoundError: No module named 'torch._prims_common'</code><br>
I think the problem has to do with Importing the PyTorch modules. I have tried uninstalling and reinstalling all the used packages in this project, and even running the project from a virtual environment using virtualenv. As I said, this program works perfectly on my Mac, so I am baffled as to why it does not work on my Chromebook. If anyone could help, that would be much appreciated.</p>

ptrblck: <p>Your PyTorch version might be too old. Could you check it via <code>torch.__version__</code> and update to the latest stable or nightly version, if possible?</p>

Jamison_Ballou: <p>Thank you for replying. PyTorch is the latest version, 2.2.</p>

ptrblck: <p>I cannot reproduce the issue using the latest <code>2.3.0</code> release and <code>2.2.2</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.3.0+cu121'
&gt;&gt;&gt; from torch._prims_common import DeviceLikeType
&gt;&gt;&gt; DeviceLikeType
typing.Union[str, torch.device, int]
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.2.2+cu121'
&gt;&gt;&gt; from torch._prims_common import DeviceLikeType
&gt;&gt;&gt; DeviceLikeType
typing.Union[str, torch.device, int]
</code></pre>

Jamison_Ballou: <p>Ok, thanks for all your help. Since the program works on my Mac, I won’t worry about fixing it on the Chromebook. Thanks again for your help.</p>
Dynamic number of output labels during the training

Lelouch1: <p>Hi</p>
<p>I am training a model continuously and the user may add new output labels during the training. For sake of clarity, lets say that the base number of labels is 3 and that we use a simple linear model with an input embedding dimension of 5. Hence the weight matrix of the linear layer has shape (3,5).</p>
<p>I wonder what is the best strategy to take into account that current number of labels may be higher in the future.</p>
<ol>
<li>When initializing the model, we use a bigger output dimension, e.g 10 instead of 3 and hope that the user will not add more than 7 labels. Hence we have a matrix of shape (10,5) but only the first 3 rows will really be used at the beginning of the training. I see several issues : a) waste of compute (we do useless computations since the 7 last rows correspond to labels that are currently not present in the dataset) and b) the model may learn to set those 7 last rows as full of zeros since those labels are never predicted. Then when I will add a 4th label and the 4th row will start to be trained, it wont backpropagate anything because of those zeros.</li>
<li>When the user add a new label, I simply extend the previous weight matrix (3,5) into a new matrix (4,5), keep the first 3 rows as the old matrix but randomly initialize the 4th row properly (with kaiming init for instance).</li>
</ol>
<p>To my knowledge the 2) seems the best one but I’ve never really seen anyone doing it so maybe I missed some important issues.</p>
<p>I also thought that it would not work when I use adaptative optimizer like ADAM since they have one learning rate for each parameter, so if I add new parameters adam will not work anymore but maybe I can do some surgery on the optimizer as well similarly to what i’ve done with the model.</p>
<p>Any advice ?</p>
How to dynamically link nvrtcCompileProgram to libtorch_cuda.so

Shivam_Mittal: <p>I built a pytorch 2.0 image using pytorch builder repo. I need an image with no/minimum static linking as possible. I turned off these environment variables for the same:<br>
USE_STATIC_CUDNN=0<br>
export ATEN_STATIC_CUDA=0<br>
export USE_CUDA_STATIC_LINK=0</p>
<p>The resultant image has nvrtcCompileProgram as statically linked in libtorch_cuda.so. I need to dynamically intercept this API but static linking makes it impossible. How can I change the build to make it dynamically linked?</p>

Dee_Joy: <p>Could I inquire whether you’ve found a solution to this issue? I’m facing a similar need but I’m looking to intercept the CUDA Driver API.</p>
Is it safe to set flag for single autograd.Function.backward call

Ventus: <p>I have defined a custom autograd.Function with forward() and backward() overrided.</p>
<p>In each training step, my use case requires two autograd backward calls: one with <code>retain_graph=True</code>, and another is for real loss backward.</p>
<p>The problem is, for the first backward() call I need to set a temporary flag in order to make it have some special behavior. Currently I have implemented it via setting a flag to a python object called <code>native_module</code> that is stored into <code>ctx</code> when calling forward(), and it seems to be working fine. I want to know if this is ensured to run safely.</p>
<p>The generic pseudo code looks like this:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

class fun(torch.autograd.Function):
    @staticmethod
    def forward(ctx, native_module, input):
        ctx.native_module = native_module
        return ctx.native_module.fwd(input)
    @staticmethod
    def backward(ctx, doutput):
        # Check the flag in native module
        some_flag = ctx.native_module.some_flag
        grad = ctx.native_module.bwd(doutput, some_flag)
        return grad

class NativeModule:
    some_flag: bool = False
    def fwd(self, input):
        # Some behavior
        return ...
    def bwd(self, doutput, some_flag: bool):
        if some_flag:
            # Some behavior
            return ...
        else:
            # Some other behavior
            return ...

# Train step
module = NativeModule()
for _ in range(100):
    input = ...
    output = fun.apply(module, input)
    
    # First backward call (some_flag passed to module should be True)
    module.some_flag = True
    grad_in = torch.autograd.grad(output, input, ..., retain_graph=True)[0]
    module.some_flag = False
    loss = some_func(grad_in.data, output)
    
    # Second backward call (some_flag passed to module should be False)
    loss.backward()
    
    # Some other procedure

</code></pre>

soulitzer: <p>Sounds ok, but why does your NativeModule  bwd function take some_flag as input even though it can already access some_flag as a module field?</p>

Ventus: <p>Thanks for your reply! Oh, that comes from an oversimplification of my actual use case. Sorry!</p>
<p>What I am uncertain of the most is that: when we call <code>torch.autograd()</code>, will the static backward() of <code>autograd.Function</code> be called instantly and synchronously? Will there be any asynchronous or delayed mechanisms to the backward() call?</p>

soulitzer: <p>Ah I see, no worries!<br>
Yes, that is correct. I don’t expect there to be any weird interaction with the global here.</p>
Broadcast pytorch array across channels based on another array

eigenvivek: <p>I have two arrays, <code>x</code> and <code>y</code>, with the same shape. <code>x</code> represents data and <code>y</code> represents which class each datapoint in <code>x</code> belongs to. I want to create a new tensor where the data in <code>x</code> are partitioned into channels based on their classes in <code>y</code>.</p>
<p>I can accomplish this if I use a one-hot encoding. However, for large tensors (especially with a large number of classes), PyTorch’s one-hot encoding quickly uses up all memory on the GPU.</p>
<p>Is there a more memory-efficient way to do this broadcasting?</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

B, C, N = 2, 10, 1000

x = torch.randn(B, 1, N)
y = torch.randint(low=0, high=C, size=(B, 1, N))

one_hot = torch.nn.functional.one_hot(y, C)  # B 1 N C
one_hot = one_hot.squeeze().permute(0, -1, 1)  # B C N

z = x * one_hot  # B C N
</code></pre>

eigenvivek: <p>Great answer on stackoverflow for anyone in the future: <a href="https://stackoverflow.com/questions/78323859/broadcast-pytorch-array-across-channels-based-on-another-array" class="inline-onebox" rel="noopener nofollow ugc">python - Broadcast pytorch array across channels based on another array - Stack Overflow</a></p>
Performance on AMD GPUs

Paul_Mullowney: <p>Hi,<br>
I have collected performance data on MI250X (single GCD) and MI300 AMD GPUs. I see a significant slow down in the following kernels compared to MI250X. I am not at all familiar with the PyTorch source. I would like some help understanding the source (i.e. how the specific kernel is launched), so I can better understand the performance issue.</p>
<ol>
<li>
<p>void at::native::_scatter_gather_elementwise_kernel&lt;256, 4, at::native::_cuda_scatter_gather_internal_kernel&lt;true, c10::Half&gt;::operator()<a>at::native::ReduceAdd</a>(at::TensorIterator&amp;, long, long, long, at::native::ReduceAdd const&amp;)::{lambda(int)<span class="hashtag-raw">#1</span>}&gt;(int, at::native::_cuda_scatter_gather_internal_kernel&lt;true, c10::Half&gt;::operator()<a>at::native::ReduceAdd</a>(at::TensorIterator&amp;, long, long, long, at::native::ReduceAdd const&amp;)::{lambda(int)<span class="hashtag-raw">#1</span>})</p>
</li>
<li>
<p>void at::native::indexFuncLargeIndex&lt;c10::Half, long, unsigned int, 2, 2, -2, true, at::native::(anonymous namespace)::ReduceAdd&gt;(at::cuda::detail::TensorInfo&lt;c10::Half, unsigned int&gt;, at::cuda::detail::TensorInfo&lt;c10::Half, unsigned int&gt;, at::cuda::detail::TensorInfo&lt;long, unsigned int&gt;, int, int, unsigned int, unsigned int, long, long, at::native::(anonymous namespace)::ReduceAdd const&amp;, c10::Half)</p>
</li>
</ol>

Paul_Mullowney: <p>I observe the issue with the 2.2 release of ROCm branch of PyTorch, i.e.</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/ROCm/pytorch/tree/release/2.2">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/ROCm/pytorch/tree/release/2.2" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/ROCm/pytorch/tree/release/2.2" target="_blank" rel="noopener nofollow ugc">GitHub - ROCm/pytorch at release/2.2</a></h3>

  <p><a href="https://github.com/ROCm/pytorch/tree/release/2.2" target="_blank" rel="noopener nofollow ugc">release/2.2</a></p>

  <p><span class="label1">Tensors and Dynamic neural networks in Python  with strong GPU acceleration</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Paul_Mullowney: <p>I found the source for code for the scatter_gather. My question, to those familiar, is does this operation absolutely require atomics? Are there alternative implementations (prefix scan based perhaps) that could be used in place?</p>
Gathering results from DDP

TeDataPro: <p>Hello,</p>
<p>I am using DDP to train my model on 2 GPUs. At inference, here’s what I do :</p>
<pre><code class="lang-auto">self.model.eval()
for i, (_, visual, labels) in enumerate(self.test_data):
    visual = visual.to(self.gpu_id)
    labels = labels.to(self.gpu_id)
    loss   = self._run_batch_val(visual, labels)
</code></pre>
<p>And here’s what happends in the <em>_run_batch_val</em> function :</p>
<pre><code class="lang-auto">  def _run_batch_val(self, visual, labels):
      with torch.no_grad():
          predictions, loss = self.model(visual, labels) # (batch_size, num_pathologies)
      self.pred_test = np.vstack((self.pred_test, predictions.detach().cpu().tolist()))
      self.labels_test = np.vstack((self.labels_test, labels.detach().cpu().tolist()))

      return {'loss_val': loss.mean().item()}
</code></pre>
<p>Where <strong>self.pred_test</strong> and <strong>self.labels_test</strong> are numpy arrays of shape (n, 5) since it’s a multilabel classification problem (n samples, 5 labels).</p>
<p>So when computing metrics, I have 2 results : one per GPU. Do you know how to merge predictions and labels from the 2 GPUs to gather it and get one single overall metric ?</p>
<p>I heard about <strong>dist.gather_all</strong> but did not find a solution for my problem.</p>
<p>Hope it’s clear, thank you so much in advance!</p>
Indexing Tensors with a two-dimensional tensor

Wk_Qo: <p>I came across a piece of code while going through PyTorch, and I’m having trouble understanding certain parts of it. Specifically, I encountered confusion regarding the following code snippet:</p>
<pre><code>    batch_x1 = torch.randn(32,88,96)
    bs, stock_num = batch_x1.shape[0], batch_x1.shape[1]
    mask = torch.ones_like(batch_x1)
    rand_indices = torch.rand(bs, stock_num).argsort(dim=-1)
    mask_indices = rand_indices[:, :int(stock_num/2)]
    batch_range = torch.arange(bs)[:, None]
    mask[batch_range, mask_indices, stock_num:] = 0
    enc_inp = mask * batch_x1
</code></pre>
<p>In this code snippet, I understand how <code>rand_indices</code> is used to generate a mask <code>mask</code> that randomly sets half of the values to 0. However, I’m not entirely clear on how <code>mask[batch_range, mask_indices, stock_num:]</code> indexes the tensor <code>mask</code> because both <code>batch_range</code> and <code>mask_indices</code> are two-dimensional tensors instead of the integers or lists that I’m familiar with.<br>
Could someone please explain the indexing operation in this code? Thank you!</p>

ptrblck: <p>Creating a smaller example and printing the intermediates might help understanding the indexing better:</p>
<pre data-code-wrap="python"><code class="lang-python">batch_x1 = torch.arange(2*3*4).view(2, 3, 4)
print(batch_x1)
# tensor([[[ 0,  1,  2,  3],
#          [ 4,  5,  6,  7],
#          [ 8,  9, 10, 11]],

#         [[12, 13, 14, 15],
#          [16, 17, 18, 19],
#          [20, 21, 22, 23]]])


bs, stock_num = batch_x1.shape[0], batch_x1.shape[1]
mask = torch.ones_like(batch_x1)
rand_indices = torch.rand(bs, stock_num).argsort(dim=-1)
print(rand_indices)
# tensor([[0, 2, 1],
#         [2, 0, 1]])

mask_indices = rand_indices[:, :int(stock_num/2)]
print(mask_indices)
# tensor([[0],
#         [2]])

batch_range = torch.arange(bs)[:, None]
print(batch_range)
# tensor([[0],
#         [1]])

mask[batch_range, mask_indices, stock_num:] = 0
print(mask)
# tensor([[[1, 1, 1, 0],
#          [1, 1, 1, 1],
#          [1, 1, 1, 1]],

#         [[1, 1, 1, 1],
#          [1, 1, 1, 1],
#          [1, 1, 1, 0]]])

enc_inp = mask * batch_x1
</code></pre>
<p>Here you are using advanced indexing as described <a href="https://numpy.org/doc/1.21/reference/arrays.indexing.html#integer-array-indexing">here</a> to avoid repeating the indices and use broadcasting instead.<br>
The numpy example indexing the corner values shows the same usage.</p>
<p>In your example you are setting:</p>
<ul>
<li>batch0, row0, columns3:</li>
<li>batch1, row2, columns3:</li>
</ul>
<p>to zero.</p>

Wk_Qo: <p>Thanks a lot sir, your reply did help me and I understand it now!</p>
Files wav labelling automated and processing to create datasets

twistin: <p>Hello everyone<br>
I’m embarking on a project that in principle is a bit intimidating (for me of course) I’m mapping the city where I live with recordings in different landmarks. With them I want to check the biotic and atropogenic levels of each place in order to make an ecoacoustic study.<br>
I need a workflow for audio tagging and processing. This is very important for further data processing, but I lack a solid knowledge of the subject.<br>
I have had some ‘talk’ with my friend ‘Copilot’ but after one morning I abandoned that way for not getting satisfactory results. Anyone who has worked these things around here and is willing to share the experience and way of working?<br>
Attached are some attempts to create a script to do this but they didn’t work<br>
Thank you very much.</p>
<pre><code class="lang-auto">import os
import pandas as pd
from torch.utils.data import Dataset, DataLoader
import torchaudio


annotations_file = 
class AudioDataset(Dataset):
    def __init__(self, annotations_file, audio_dir, transform=None):
        self.annotations = pd.read_csv(annotations_file)
        self.audio_dir = audio_dir
        self.transform = transform

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        audio_sample_path = os.path.join(self.audio_dir, self.annotations.iloc[index, 0])
        label = self.annotations.iloc[index, 1]
        waveform, sample_rate = torchaudio.load(audio_sample_path)
        if self.transform:
            waveform = self.transform(waveform)
        return waveform, label

# Suponiendo que ya tienes un modelo entrenado llamado 'modelo'
# y una función de transformación llamada 'transformacion_audio'

# Cargar el dataset no etiquetado
dataset_no_etiquetado = AudioDataset('path/to/annotations.csv', 'path/to/audio_dir', transform=transformacion_audio)
dataloader = DataLoader(dataset_no_etiquetado, batch_size=1, shuffle=False)

# Etiquetar los nuevos sonidos
predicciones = []
for waveform, _ in dataloader:
    outputs = modelo(waveform)
    _, predicted_labels = torch.max(outputs, 1)
    predicciones.append(predicted_labels.item())

# Guardar las predicciones en un archivo CSV
nuevas_etiquetas = pd.DataFrame(predicciones, columns=['Etiqueta'])
nuevas_etiquetas.to_csv('nuevas_etiquetas.csv', index=False)
'''</code></pre>

ptrblck: <p>I’m unsure where exactly you are stuck in your workflow. Are you looking for a system to tag your audio samples first or did you do it already and would like to create a <code>Dataset</code> now, but are stuck?</p>

twistin: <p>Hi, sorry if I have not been clear in explaining myself. I have recorded soundscapes, I have them unprocessed and unlabeled. I want to create windows with durations of 3 seconds and label them so that I can then process them as data for an eco acoustic job (show the areas of the city with biotic and antepogenic noise levels.</p>
What is the detailed operations of `torchfunc.jvp`

Hongduan_Tian: <p>Hi,</p>
<p>Recently, I am studying about “torchfunc.jvp” function. However, I am confused about how a list of “tangents” and a list of “gradients” obtain an output with the same size as the output of models?</p>
<p>To be specific, the output of the model is [batch_size, dim], and the gradients are a list. How can I get a vector-jacobian with two lists of tensors?</p>

soulitzer: <p>Are you trying to compute the vector-jacobian product or the jacobian-vector product. The jvp function produces the jacobian-vector product.</p>

Hongduan_Tian: <p>Thanks for your reply. I think I misunderstood some parts of the function. Now, it is clear. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
[jit] bad constant exponent (e+38.f) in default_program fused_mul_div_add

sits: <p>Hi - I have been using DJL (<a href="https://github.com/deepjavalibrary/djl" class="inline-onebox" rel="noopener nofollow ugc">GitHub - deepjavalibrary/djl: An Engine-Agnostic Deep Learning Framework in Java</a>) with PyTorch of late, and hit this bug: <a href="https://github.com/pytorch/pytorch/issues/107503" class="inline-onebox" rel="noopener nofollow ugc">[nightly][jit] bad constant exponent (e+38.f) in default_program fused_mul_div_add · Issue #107503 · pytorch/pytorch · GitHub</a>.</p>
<p>This seems to be a regression that was introduced after PyTorch 2.0.1, at least for Linux.  I found for Windows it was already an issue in 2.0.1.</p>
<p>The response on the bug report indicates torchscript is in “maintenance mode” and is not a problem, but I would say a regression should be fixed as a priority even in maintenance mode, as this used to work, and it affects downstream projects like DJL.</p>
<p>Is there a way for this issue to be given priority please?  This effectively blocks DJL users from using more recent versions of PyTorch, and Windows seems to be broken.</p>
<p>It seems to be an issue of how floating point numbers are written out in an incompatible format.</p>
<p>Many thanks in advance for any help provided…</p>
Contextual Bandit with PyTorch instead of TF?

arvnoodle: <p>So I saw there’s a TF agent that can do Linear Thompson Sampling but was hoping for if there’s a similar Pytorch implementation of that? If you guys know anything? Planning to use it for Send Time Optimization</p>

kek3731: <p>I want it too. Is there anyone who can help me?</p>

vmoens: <p>Hey thanks for the suggestion.<br>
We’re working on implementing this in torchrl. Stay tuned!</p>

kek3731: <p>I find pytorch code for thomson sampling.<br>
and it may implement contextual bandit</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/RonyAbecidan/Neural-Thompson-Sampling">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/RonyAbecidan/Neural-Thompson-Sampling" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/0349e855f8e92259be6ebebe00bc2a2d83fb7bc4df863dc0542d007f2994f160/RonyAbecidan/Neural-Thompson-Sampling" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/RonyAbecidan/Neural-Thompson-Sampling" target="_blank" rel="noopener nofollow ugc">GitHub - RonyAbecidan/Neural-Thompson-Sampling: Study of the paper 'Neural...</a></h3>

  <p>Study of the paper 'Neural Thompson Sampling' published in October 2020 - GitHub - RonyAbecidan/Neural-Thompson-Sampling: Study of the paper 'Neural Thompson Sampling' published in ...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>But I didn’t understand something that was implemented.<br>
Can i study this if i join your slack?</p>

cryptolocker: <p>Hey <a class="mention" href="/u/arvnoodle">@arvnoodle</a> just curious how did you go about, what did you end up choosing ?</p>
CUDA Out of Memory Error with jvp() in Large Model Training - Need Insights or Fixes

Manikishan: <p>Hello PyTorch Community,</p>
<p>I am encountering a persistent CUDA out of memory (OOM) error during the training of a large model using BERT for sequence classification. The issue arises specifically around the use of the <code>jvp()</code> function for Jacobian-vector products in my training loop. Below, I’ve provided an overview of the relevant parts of my code and the specific issues I’m facing:<br>
Note: I am noticing the memory footprint is higher than normal and when I try to profile it I am getting OOM error.</p>
<h4><a name="code-snippet-1" class="anchor" href="#code-snippet-1"></a><strong>Code Snippet:</strong></h4>
<p>The outline code to show relevant parts of the implementation.</p>
<pre><code class="lang-auto"># Simplified snippet focusing on the problematic area
for epoch in range(num_epochs):
    for batch in train_loader:
        v_params = tuple([torch.randn_like(p).to(device) for p in params])
        
        # Forward pass and loss computation
        loss = model(batch)

        # Jacobian-vector product calculation
        _, jvp = torch.autograd.functional.jvp(fwd_function, (params,), (v_params,))
        
        # Gradient assignment
        for p, v in zip(params, v_params):
            p.grad = v * jvp

        del v_params
        optimizer.step()
        optimizer.zero_grad()

        torch.cuda.empty_cache()  # Attempt to free up unused memory
</code></pre>
<h4><a name="issue-description-2" class="anchor" href="#issue-description-2"></a><strong>Issue Description:</strong></h4>
<ul>
<li>The training process consistently results in an OOM error when calculating <code>jvp()</code>.</li>
<li>Despite attempts to manage memory by deleting temporary variables and clearing the CUDA cache, the error persists.</li>
<li>The model trains without issues when the <code>jvp()</code> call is omitted, suggesting a significant memory overhead associated with this operation.</li>
</ul>
<h4><a name="attempts-to-debug-3" class="anchor" href="#attempts-to-debug-3"></a><strong>Attempts to Debug:</strong></h4>
<ul>
<li>Memory profiling indicates high memory usage at the <code>jvp()</code> call, but I have not been able to reduce this usage effectively.</li>
</ul>
<h4><a name="questions-4" class="anchor" href="#questions-4"></a><strong>Questions:</strong></h4>
<ol>
<li>Has anyone experienced similar issues with <code>jvp()</code> in large model training scenarios?</li>
<li>Are there known memory optimization strategies or alternative approaches to computing the Jacobian-vector product that might reduce memory consumption?</li>
<li>Any recommendations for profiling tools or techniques that could provide more detailed insights into memory allocation and deallocation within PyTorch?</li>
</ol>
<p>I appreciate any insights or suggestions the community might offer. Thank you for your help!</p>
Mixing pytorch autograd with custom vjp for optimization

Amit_Gupta: <p>Hi, I want to train a my NN on first derivative with respect to the input.<br>
I am computing an intermediate representation using a C++ binding, which is then fed to NN.<br>
ie.</p>
<pre><code class="lang-auto">E = NN(θ,ζ(r))
</code></pre>
<p>Here  θ is the parameters of NN, and r is the input, and ζ is the transform function (descriptor).<br>
Loss function uses derivatives against r,</p>
<pre><code class="lang-auto">F = - ∂E/∂ζ(r) * ∂ζ/∂r
</code></pre>
<p>Here the function ζ is implemented in C++ for performance and several other reasons, and ∂ζ/∂r is computed using Enzyme AD C++ vjp bindings.</p>
<p>Now if I am correct, the optimization would need the derivatives,</p>
<pre><code class="lang-auto">∂^2E/∂ζ∂θ * ∂ζ/∂r
</code></pre>
<p>Currently I am computing it as</p>
<pre data-code-wrap="python"><code class="lang-python">zeta = cpp_zeta_function(r) # non autograd tracked function
E = model(zeta)
dE_dzeta = torch.autograd.grad(..., create_graph=True, retain_graph=True)[0]
F = cpp_zeta_function.gradient(r, dE_dzeta) # non autograd tracked function
forces = torch.from_numpy(F)
force_summed = scatter_add(forces, image, dim=0)
losses = loss(forces_summed, forces_true)
</code></pre>
<p>I do not get any leaf tensor warning so it seems to be working. Is this approach correct? Can Autograd figure this activity flow correctly?</p>
<p>If not is there a way for tell the optimizer to actually use <code>∂^2E/∂ζ∂θ * ∂ζ/∂r</code> gradients ( compute <code>cpp_zeta_function.gradient(r, ∂^2E/∂ζ∂θ)</code> manually)?</p>

JuanFMontesinos: <p>You can implement your own backprop and bind your functions there.<br>
<a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html</a></p>

Amit_Gupta: <p>But will it not run in same issues? I have only implemented the first derivative, and custom autograd function will also fail if autograd could not figure out that it does not need this derivative for second derivatives.</p>
Features extraction and visualisation

Jesus: <p>Hello everyone. I have implemented a custom cnn using libtorch. Here is it:<br>
class ConvnetImpl : public torch::nn::Module<br>
{<br>
private:<br>
torch::nn::Sequential model;</p>
<p>public:<br>
ConvnetImpl()<br>
{<br>
model = torch::nn::Sequential(<br>
torch::nn::Conv2d(torch::nn::Conv2dOptions(/<em>input channel</em>/ 1, /<em>output channel</em>/ 16, /<em>kernel_size=</em>/3)),<br>
torch::nn::ReLU(true),<br>
torch::nn::MaxPool2d(torch::nn::MaxPool2dOptions(/<em>kernel</em>/ {3, 3}).stride({2, 2})),</p>
<pre><code>        torch::nn::Conv2d(torch::nn::Conv2dOptions(/*input channel*/ 16, /*output channel*/ 32, /*kernel_size=*/3)),
        torch::nn::ReLU(true),
        torch::nn::MaxPool2d(torch::nn::MaxPool2dOptions(/*kernel*/ {3, 3}).stride({2, 2})),


        torch::nn::Conv2d(torch::nn::Conv2dOptions(/*input channel*/ 32, /*output channel*/ 64, /*kernel_size=*/3)),
        torch::nn::ReLU(true),
        torch::nn::MaxPool2d(torch::nn::MaxPool2dOptions(/*kernel*/ {3, 3}).stride({2, 2})),


        torch::nn::Conv2d(torch::nn::Conv2dOptions(64, 128, 3)),
        torch::nn::ReLU(true),
        torch::nn::MaxPool2d(torch::nn::MaxPool2dOptions(/*kernel*/ {3, 3}).stride({2, 2}))
    );

   register_module("convnet", model);
};

torch::Tensor forward(torch::Tensor &amp;input_image)
{
    torch::Tensor features = model-&gt;forward(input_image);
    features = features.permute({0, 2, 3, 1}); //(batch_size, height, width, features)
    return features;
}
</code></pre>
<p>};<br>
TORCH_MODULE(Convnet);</p>
<p>How can i extract the output of each conv layers and visualize it ?</p>
How to train CycleGAN of MRI Images and Continuous wavelet transform (CWT)

Sameer_Ahmed1: <p><strong>Hello Everyone, I need some information about how to train the cycleGAN model using MRI Images and a Continuous Wavelet Transformer (CWT).</strong></p>
<p>Dataset: <a href="https://openneuro.org/datasets/ds004505/versions/1.0.4" class="inline-onebox" rel="noopener nofollow ugc">OpenNeuro</a></p>
<p>Basically, In this dataset, we have 25 different subjects (participants) data of both MRI Images and EEG signals consisting of 120 Channels. Now my task is to give this dataset into cycleGAN, In cycleGAN we have two domains (Domain-A, I assign to MRI Images), and (Domain-B, I assign to EEG signals consisting of 120 channels). Now the problem is that cycleGAN typically takes image data not signal data. So I convert EEG signals into Continuous wavelet transform (CWT). But here I face another problem, Normally in CycleGAN we give one image to domain-A and expect to predict the one image to the domain-B or vice-versa. But here because we have 120 different channels of EEG signals, I fuse this (I take the mean of the different channels 120 channels and produce only a single channel signal) Then I give this single channel into Continuous wavelet transform (CWT), and it produces a single image.</p>
<p>Now this task is image-to-image translation, I tried to train this into typical CycleGAN but it is not working because I think in cycleGAN we have to give related data to both domain-A and domain-B but here both are different. Please guide me about that. How can I train this? Thanks in advance.</p>
<p><strong>Note</strong>: I found one <a href="https://ieeexplore.ieee.org/document/9978122" rel="noopener nofollow ugc">Research Paper</a> about this topic but it is not based on cycleGAN it is based on CNN. In this research paper written that, If we use CNN + LSTM layer it gives best result specially for CWT data.</p>
Loop a tokenizer over multiple files

Steven_Duffy: <p>I am currently trying to attempting to tokenize large text, however i have a lot of files in the directory that i want to tokenize as this is very time consuming to do 1 by 1.</p>
<p>from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline<br>
import torch</p>
<p>tokenizer = AutoTokenizer.from_pretrained(“joeddav/distilbert-base-uncased-go-emotions-student”)<br>
model = AutoModelForSequenceClassification.from_pretrained(“joeddav/distilbert-base-uncased-go-emotions-student”)</p>
<p>txt=“…”</p>
<p>words_input_dir = “/content/sample_data/”</p>
<p>for filename in os.listdir(words_input_dir):<br>
if filename.endswith(“.txt”):<br>
with open(filename, “r”) as input_file:<br>
input_tokens = word_tokensize(input_file.read())</p>
<p>tokens = tokenizer.encode_plus(input_file.read(), add_special_tokens = False, return_tensors = ‘pt’)</p>
<p>print(len(tokens))</p>
<p>tokens</p>
<p>before I added the loop the original read</p>
<p>tokens = tokenizer.encode_plus(txt, add_special_tokens = False, return_tensors = ‘pt’)</p>
<p>TYIA.</p>
Guidance with image super-resolution models

pcgmont: <p>I’m looking for Super Resolution models that can run on an embedded platform (type Nvidia Jetson) in real time, while obtaining the highest output image quality possible.</p>
<p>I’ve been looking both to Single Image (SISR) and Multi Frame (MFSR) options. I see mainly four families of models: models that only focus on spatial super-resolution (SISR), models that take advantage of temporal correlations (MFSR), models that try to extract high frequency parts of an image, and vision transformer based models. However, not being a expert in the field, I quickly see that there is an enormous amount of possible models, where many may not be suited to embedded devices. I would like to ask some guidance on the following:</p>
<ol>
<li>First, would you recommend a particular family of models to achieve my goal?</li>
<li>Do you have a particular model in mind?</li>
</ol>
<p>Thank you in advance</p>

pcgmont: <p>I take the liberty of tagging you <a class="mention" href="/u/trypag">@trypag</a> and <a class="mention" href="/u/lornatang">@Lornatang</a> because I saw you were active on super-resolution topics. Hope you don’t mind!</p>
Loss.backward() call not updating loss

Youngseo_Kim: <p>Hi there, I’m training text features recently, but the loss doesn’t updated, and I really don’t know where the problem is</p>
<p>here is the code</p>
<pre><code class="lang-auto">        for iter in tqdm(range(100)):
            total_style_loss = 0
            for j in range(i):
                style_prompts_i = model("style_prompts", i)
                style_prompts_j = model("style_prompts", j)
                sim = cos_sim(style_prompts_i, style_prompts_j)
                total_style_loss = total_style_loss + sim
            loss_style = total_style_loss / i

            optimizer.zero_grad()
            loss_style.backward()
            optimizer.step()
</code></pre>
<p>the result is:</p>
<pre><code class="lang-auto">tensor([0.9084], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
  1%|▊                                                                                   | 1/100 [00:00&lt;00:23,  4.30it/s]tensor([0.9084], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor([0.9084], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor([0.9084], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor([0.9084], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
  5%|████▏                                                                               | 5/100 [00:00&lt;00:05, 17.29it/s]tensor([0.9084], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor([0.9084], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor([0.9084], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor([0.9084], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
  9%|███████▌                                                                            | 9/100 [00:00&lt;00:03, 24.49it/s]tensor([0.9084], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor([0.9084], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor([0.9084], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
tensor([0.9084], device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
 13%|██████████▊                                                                        | 13/100 [00:00&lt;00:03, 28.87it/s
</code></pre>

soulitzer: <p>Can you check:</p>
<ul>
<li>are the .grads being populated before .step()</li>
<li>is style_prompts_i.requires_grad True</li>
<li>are your parameters requires_grad=True</li>
</ul>

Youngseo_Kim: <p>Thank you for your answer <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
Out.backward() and weight access

Am_b_reas: <p>Hey there, Can I run a backward pass only on an intermediate step? I need to access the weights after that step in the forward function without backpropagating through it.</p>

soulitzer: <p>Yes, and if this is part of a module, you can use a module forward hook to save that intermediate to a global, and at the end of the forward, call .backward() on that intermediate.</p>
Backpropagating w.r.t detached tensor

haj: <p>In my code, I have a class with a forward function that takes the output of a conv2d as input <code>x</code>. Thus, the input <code>x</code> will be attached to the computational graph and will require gradient. However, I have to do some calculations in the <code>forward</code> function that inevitably requires <code>x</code> to be transformed to numpy array and thus detached from the computational graph.</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = x.detach().cpu().numpy()
</code></pre>
<p>The problem is that I need to pass the gradient w.r.t. <code>x</code> to the previous conv2d layer. Will this be possible if I make a custom backward path as specified in [Custom Autograd] even though <code>x</code> is detached?(<a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch — PyTorch 2.2 documentation</a>)</p>

soulitzer: <p>Yes, you’d want to use a custom autograd Function here for the part of the computation you are computing with numpy but still want to backward through. <a href="https://pytorch.org/docs/stable/autograd.html#function" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 2.2 documentation</a></p>

haj: <p>Thanks for the answer!</p>
Testing the Binary Segmentation Model and Calculating IoU

jawi289p: <p>Hello, I am new to machine learning coding and I am training a binary segmentation model using Pytorch. I have loaded the trained model. I have around 120 test images. Can anyone please help me is my testing and calculating IoU score correct?</p>
<pre><code class="lang-auto">batch_size = 8
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
# Load the saved model
model = EncoderDecoder(in_channels=1, out_channels=1)  
model.load_state_dict(torch.load('best_model.pth'))
model.eval()  # Set the model to evaluation model

# Define lists to store predicted masks and ground truth labels
predicted_masks = []
ground_truth_labels = []
original_images = []

# Iterate over the test dataset
for inputs, labels in test_loader:
    # Move data to GPU
    inputs, labels = inputs.to(device), labels.to(device)
    original_images.append(inputs)

    # Forward pass
    with torch.no_grad():
        outputs = model(inputs)

    # Convert outputs to binary masks using a threshold 
    predicted_masks.append((outputs &gt; 0.5).float())  # Applying threshold directly and keeping as float tensor
    
    # Append labels directly as tensors
    ground_truth_labels.append(labels)

# Concatenate the ground truth labels
ground_truth_labels = torch.cat(ground_truth_labels, dim=0)

# Concatenate original images
original_images = torch.cat(original_images, dim=0)

# Stack the predicted masks into a single tensor
predicted_masks = torch.cat(predicted_masks, dim=0)

torch.Size([120, 1, 650, 1250]) #shape of predicted_masks

///Calculating IoU Score
def iou_pytorch(inputs: torch.Tensor, targets: torch.Tensor, smooth: float = 1e-6):
    inputs = inputs.contiguous().view(-1)
    targets = targets.view(-1)

    intersection = (inputs * targets).sum().float()
    total = (inputs + targets).sum().float()
    union = total - intersection 

    iou = (intersection + smooth) / (union + smooth)
    
    return iou.item()
</code></pre>

jawi289p: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> waiting…!!!</p>

ptrblck: <p>Please don’t tag specific users in questions, as it will discourage others to post valid answers.<br>
You can compare your implementation with other libs (e.g. I guess <code>torchmetrics</code> or <code>kornia</code> would implement IOU already).</p>
Cuda 12.2 and pytorch 2.2 compatibility

Imen_Mabrouk: <p>Hello everyone,</p>
<p>I am new to PyTorch and am trying to use PyTorch with CUDA 12.2 on a Jetson AGX Orin (jetpack 6.0). How can I fix this error ( PyTorch must user CUDA )?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/3/438375cf4801b43abc787e28b5225f69ce6ed1fa.png" data-download-href="https://discuss.pytorch.org/uploads/default/438375cf4801b43abc787e28b5225f69ce6ed1fa" title="Screenshot from 2024-04-23 11-21-55"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/438375cf4801b43abc787e28b5225f69ce6ed1fa_2_690x201.png" alt="Screenshot from 2024-04-23 11-21-55" data-base62-sha1="9DfDQuj78opIWQgSGVHfy1lAvqy" width="690" height="201" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/438375cf4801b43abc787e28b5225f69ce6ed1fa_2_690x201.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/438375cf4801b43abc787e28b5225f69ce6ed1fa_2_1035x301.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/438375cf4801b43abc787e28b5225f69ce6ed1fa_2_1380x402.png 2x" data-dominant-color="3D1A32"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-04-23 11-21-55</span><span class="informations">1565×457 90.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The default ARM binaries to not support CUDA and you could download compatible binaries as mentioned <a href="https://discuss.pytorch.org/t/torch-cuda-is-available-is-false-for-cuda-version-11-4/156616/6">here</a>.</p>
Advanced Computer Vision

Tornike: <p>Hi, I’m looking for some free computer vision books or courses where i will learn about camera models, calibration and etc… i have searched on the internet but all of them are very expensive.</p>
<p>Any sources where i could learn that ? i need theoretical and practical sources.</p>
<p>Thanks in advance.</p>

jS5t3r: <p>This one is standard book available as pdf: <a href="https://www.robots.ox.ac.uk/~vgg/hzbook/" class="inline-onebox" rel="noopener nofollow ugc">Multiple View Geometry in Computer Vision&lt;BR&gt;Second Edition</a></p>
Setting up source code for development contribution error

duffymitch12: <p>I am currently trying to setup PyTorch on my local machine to develop locally for contribution to the project. I have an Anaconda environment setup and other recommended dependencies but when I run the command “Python3 setup.py develop” it produces too many errors and does not complete. I have provided the end of the snippet from the command line. Help would be greatly appreciated.</p>
<pre><code class="lang-auto">/Users/mitchellduffy/Desktop/eecs-481/torch/pytorch/build/third_party/onnx/onnx/onnx_onnx_torch-ml.pb.h:2041:46: error: non-virtual member function marked 'final' hides virtual member function
  inline StringStringEntryProto* New() const final {
                                             ^
/usr/local/include/google/protobuf/message.h:253:12: note: hidden overloaded virtual function 'google::protobuf::Message::New' declared here: different number of parameters (1 vs 0)
  Message* New(Arena* arena) const override = 0;
           ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
4 warnings and 20 errors generated.
[530/1100] Building CXX object caffe2/...pu/GridSamplerKernel.cpp.DEFAULT.cpp.o
ninja: build stopped: subcommand failed.
</code></pre>

ptrblck: <p>The end of the error message is usually not helpful as the initial error would trigger these, so check the first failure and post it here.</p>

duffymitch12: <p>This is the first error that is generated:</p>
<pre><code class="lang-auto">/Users/mitchellduffy/Desktop/eecs-481/torch/pytorch/build/third_party/onnx/onnx/onnx_onnx_torch-ml.pb.h:17:2: error: This file was generated by an older version of protoc which is
#error This file was generated by an older version of protoc which is
 ^
/Users/mitchellduffy/Desktop/eecs-481/torch/pytorch/build/third_party/onnx/onnx/onnx_onnx_torch-ml.pb.h:18:2: error: incompatible with your Protocol Buffer headers. Please
#error incompatible with your Protocol Buffer headers. Please
 ^
/Users/mitchellduffy/Desktop/eecs-481/torch/pytorch/build/third_party/onnx/onnx/onnx_onnx_torch-ml.pb.h:19:2: error: regenerate this file with a newer version of protoc.
#error regenerate this file with a newer version of protoc.
 ^

</code></pre>

ptrblck: <aside class="quote no-group" data-username="duffymitch12" data-post="3" data-topic="201373">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/duffymitch12/48/69454_2.png" class="avatar"> duffymitch12:</div>
<blockquote>
<p><code>This file was generated by an older version of protoc which is</code></p>
</blockquote>
</aside>
<p>Your system <code>protoc</code> might be incompatible and you could use <code>BUILD_CUSTOM_PROTOBUF=ON</code> to build with the <code>protobuf</code> from <code>third_party/protobuf</code>.</p>