akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is this example correct? `weights_only=True`?

Aknw_Fen: <p>Assuming we reload for training, don’t we need to load more data than <code>weights_only</code> ? (I’m aware though of arbitrary executing code and other caveats.)</p>
<pre data-code-wrap="python"><code class="lang-python">model = TheModelClass(*args, **kwargs)
optimizer = TheOptimizerClass(*args, **kwargs)

checkpoint = torch.load(PATH, weights_only=True)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']

model.eval()
# - or -
model.train()
</code></pre>
<p><strong>Source</strong></p>
<p><a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#load" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/saving_loading_models.html#load</a></p>

soulitzer: <p>if you mean things like optimizer state, weights_only=True does load that for you.</p>
<p>from <a href="https://pytorch.org/docs/stable/generated/torch.load.html" class="inline-onebox" rel="noopener nofollow ugc">torch.load — PyTorch 2.5 documentation</a></p>
<blockquote>
<p>Indicates whether unpickler should be restricted to loading only tensors, primitive types, dictionaries and any types added via <a href="https://pytorch.org/docs/stable/notes/serialization.html#torch.serialization.add_safe_globals" rel="noopener nofollow ugc"><code>torch.serialization.add_safe_globals()</code></a>.</p>
</blockquote>
Contrastive Learning Loss Function

Tonnar_Castellano: <p>I am trying to implement the loss function I am unsure if it is correct. My loss seems to explode into infinity. I was wondering if someone was able to point out an error with my code?</p>
<pre><code class="lang-auto">class ContrastiveLoss(nn.Module):
    def __init__(self, temperature=0.5):
        super(ContrastiveLoss, self).__init__()
        self.temperature = temperature

    def forward(self, projections_1, projections_2):
        z_i = projections_1
        z_j = projections_2
        z_i_norm = F.normalize(z_i, dim=1)
        z_j_norm = F.normalize(z_j, dim=1)
        cosine_num = torch.matmul(z_i, z_j.T)
        cosine_denom = torch.matmul(z_i_norm, z_j_norm.T)
        cosine_similarity = cosine_num / cosine_denom

        numerator = torch.exp(torch.diag(cosine_similarity) / self.temperature)

        denominator = cosine_similarity
        diagonal_indices = torch.arange(denominator.size(0))
        denominator[diagonal_indices, diagonal_indices] = 0
        denominator = torch.exp(torch.sum(cosine_similarity, dim=1))
        loss = -torch.log(numerator / denominator).sum()
        return loss
</code></pre>
Cross-entropy loss ignores labels of a specific index but throws an error

aohenuo: <p>In the definition of <code>torch.nn.module.loss</code>, there is a parameter <code>ignore_index_value</code> that can ignore losses at specific positions. However, when I used <code>-100</code> to pad labels during training, the following is the padding code I used:</p>
<pre data-code-wrap="python"><code class="lang-python">query_inputs["labels"] = torch.cat([torch.full((bs, v_token_length,), -100, device=query_inputs["input_ids"].device), query_inputs["input_ids"]], dim=1)
query_inputs["labels"] = query_inputs["labels"].type(torch.LongTensor)
</code></pre>
<p>But during training, I encountered the following error:</p>
<pre><code class="lang-auto">nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [0,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
</code></pre>
<p>This indicates that the label exceeds the valid range. However, such an error shouldn’t occur because <code>-100</code> is the default ignore index in PyTorch. How can I resolve this issue?</p>

ptrblck: <p>The default <code>ignore_index=-100</code> works for me:</p>
<pre data-code-wrap="python"><code class="lang-python">batch_size = 16
nb_classes = 32
device = "cuda"

x = torch.randn(batch_size, nb_classes, requires_grad=True, device=device)
target = torch.randint(0, nb_classes, (batch_size,), device=device)

criterion = nn.CrossEntropyLoss()

loss = criterion(x, target)
print(loss)

# use the ignored index
target[0] = -100
loss = criterion(x, target)
print(loss)

# use invalid index
target[1] = -101
loss = criterion(x, target)
# /pytorch/aten/src/ATen/native/cuda/Loss.cu:250: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [1,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
print(loss)
# RuntimeError: CUDA error: device-side assert triggered
</code></pre>
Batch normalization - don't see all the parameters

Oren_Zeev_Ben_Mordeh: <p>I’ve saved a batch normalization layer (2d) and I only see the weights and bias but not ‘running_mean’, nor the ‘running_var’. I’ve used: torch.save(model.state_dict()…</p>
<p>I may doing something funny. I’m not loading those with torch, but with regular Python code, as a dict. Is there a way to figure those out (‘running_mean’, nor the ‘running_var’) from weight and bias? I don’t see them explicitly in the dictionary.</p>
<p>Why am I not using PyTorch to load? Don’t ask, Kaggle competition limitation.</p>

Oren_Zeev_Ben_Mordeh: <p>Okay, it is more involved. I print to the “standard output” and then it goes into a Python file. Probably the issue is not PyTorch related. I’ll have to check why it was not included in the output. I believe the issue as raised above (by me) can be close.</p>

Oren_Zeev_Ben_Mordeh: <p>Here is my code that includes the “bug”:</p>
<blockquote>
<pre><code>my_state_dict = {
    name: param.tolist()
    for name, param in model.named_parameters() 
}
</code></pre>
</blockquote>
<p>I’m now adding the running_mean and running_var “manually”.</p>

ptrblck: <p>Running stats are buffers, not parameters, which is why they are not returned via <code>.named_parameters()</code>. Use the <code>state_dict</code> directly or make sure to save <code>.named_buffers()</code> as well.</p>
DistributedSampler is NOT required for FSDP on a single cluster

hH1sG0n3: <p>I understand the need for a <code>DistributedSampler</code> in conjunction with <code>FSDP</code> for multi-cluster training, because the state of the dataset is not shared across instances, and therefore the same data points will inevitably be yielded.</p>
<p>I also understand the need for a <code>DistributedSampler</code> in conjunction with <code>DDP</code>, as its benefits arise from aggregating results on different batches in parallel.</p>
<p>However, I do not think a <code>DistributedSampler</code> is necessary with iterable datasets on a single multi-GPU cluster.</p>
<p>Correct me if I am wrong.</p>

hH1sG0n3: <p>I have performed experiments that prove that a distributed sampler is indeed required for FSDP on a single multi-GPU cluster.</p>
<p>The reason for this investigation is the requirement for an iterable dataset (and not a map style) with which the PyTorch DistributedSampler is not compatible. I have created an iterable dataset that borrows the distributed sampler logic and ensures distributing non-duplicated data across ranks.</p>
DDP without Distributed Sampler

Rakshit_Kothari: <p>What happens when we do not give a distributed sampler? Does it essentially iterate over all samples with as many ranks as we have?</p>

rvarm1: <p>If the data is not sharded across different DDP ranks (i.e. with a distributed sampler or some custom sharding logic that you may have), then yes, DDP will use all samples on all ranks (in your example I guess there’s 2 ranks).</p>
<p>This is why in general you want want to partition your data appropriately across ranks to ensure different model replicas get different data.</p>

Rakshit_Kothari: <p>Thanks Rohan, this cleared it out for me.</p>
Custom autograd doesn't learn as good

sharjeel: <p>i’m using a customized autograd function for convolutional layers as shown in the code snippet</p>
<pre><code class="lang-auto">class conv_autograd(Function):

    def forward(ctx, input, weight, stride, groups, padding, dilation, bias = None, mask = None):
        ctx.save_for_backward(input, weight, bias, mask)
        ctx.stride = stride
        ctx.padding = padding
        ctx.dilation = dilation
        ctx.groups = groups

        if mask == None:
            conv_out = F.conv2d(input, weight, stride = stride, groups = groups, padding = padding, dilation = dilation, bias = None)
        else:
            conv_out = F.conv2d(input, weight*mask, stride = stride, groups = groups, padding = padding, dilation = dilation, bias = None)
        return conv_out
    
    def backward(ctx, grad_output):
        input, weight, bias, mask = ctx.saved_tensors
        stride = ctx.stride
        padding = ctx.padding
        dilation = ctx.dilation
        groups = ctx.groups

        grad_input = grad_weight = grad_bias = None

        if ctx.needs_input_grad[0]:
            grad_input = F.grad.conv2d_input(input.shape, weight, grad_output, stride=stride, padding=padding, dilation=dilation, groups=groups)
        if ctx.needs_input_grad[1]:
            grad_weight = F.grad.conv2d_weight(input, weight.shape, grad_output, stride=stride, padding=padding, dilation=dilation, groups=groups)
            if mask is not None:
                grad_weight *= mask
        if bias is not None and ctx.needs_input_grad[2]:
            grad_bias = grad_output.sum(dim=(0, 2, 3))

        return grad_input, grad_weight, grad_bias, None, None, None, None, None
</code></pre>
<p>And the goal here is to train the weights of the convolutional layer using the fixed input mask. However even when i am not using the mask (mask = None) the model’s accuracy does not cross 38%. While using ResNet50 on CIFAR-100 dataset… What am i doing wrong?</p>

soulitzer: <p>You can use <a href="https://pytorch.org/docs/stable/generated/torch.autograd.gradcheck.gradcheck.html" class="inline-onebox" rel="noopener nofollow ugc">torch.autograd.gradcheck.gradcheck — PyTorch 2.5 documentation</a> to validate that its producing the correct gradients for a given set of inputs.</p>

Soumya_Kundu: <p>I think gradcheck is only for if it is actually producing any gradients at all? How would you validate if it is producing the correct gradients just through gradcheck?</p>

soulitzer: <p>For a given set of inputs + your function, gradcheck will compute the gradients once with your provided backward and once numerically via finite differencing. If the results are the same, then we can be confident that given that the forward is correct, the backward would also be correct.<br>
This should also tell you if one the gradients wrt the inputs is unexpectedly zero.</p>
Faster way to only calculate some derivatives

qq-me: <p>here is my idea for an optimizer - you only calculate some derivatives on each step, other remain the same. So it works as a kind of momentum. But that would be useful if you could somehow calculate only some derivative in a way that is faster than calculating all of the derivatives. Is there a way to do that?</p>

soulitzer: <p>You can specify which inputs you would like to compute derivatives for via the inputs= arguments to .backward( or autograd.grad(.</p>
activation memory

amber: <p>How can I implement custom activation storage in a model, for example, by merging the activation values of two layers, storing them, and ensuring proper backpropagation functionality?</p>

soulitzer: <p>You can use saved tensor hooks to hook onto the activation saving logic</p>
<p><a href="https://pytorch.org/tutorials/intermediate/autograd_saved_tensors_hooks_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/autograd_saved_tensors_hooks_tutorial.html</a></p>
Problem with cuda/pytorch on Fedora

Michal_P: <p>I have a Fedora 41 Linux and i want to use pytorch with cuda. Everywhere I look i see that info that pytorch works witch cuda up to 12.5 but i have cuda 12.7. What can I do? That’s what nvidia-smi returns for me:</p>
<p>Wed Dec 11 20:04:25 2024<br>
+<br>
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |<br>
|<br>
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                        |               MIG M. |</p>
<p>|   0  NVIDIA GeForce GTX 1650        Off |   00000000:01:00.0 Off |                  N/A |<br>
| N/A   36C    P8              3W /   50W |       1MiB /   4096MiB |      0%      Default |<br>
|                                         |                        |                  N/A |<br>
+</p>
<p>It would be best if i could get it for python 3.13 if that’s possible.</p>
<p>I’m quite new to linux and cuda so I would apreciate some help</p>

ptrblck: <aside class="quote no-group" data-username="Michal_P" data-post="1" data-topic="214122">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michal_p/48/74476_2.png" class="avatar"> Michal_P:</div>
<blockquote>
<p>pytorch works witch cuda up to 12.5 but i have cuda 12.7</p>
</blockquote>
</aside>
<p>PyTorch works with all released CUDA toolkits up to 12.6U3, which is the latest CUDA toolkit release, and the PyTorch nightly binaries already ship with it.<br>
The <code>nvidia-smi</code> output shows the driver, which is also supported as it’s new enough.</p>

Michal_P: <p>To my understanding it the output also contains CUDA version: 2.7. Is there any way to downgrade the version to 2.6 so that I can use pytorch?</p>
Dynamic Data Reinsertion in Dataloader

Caquot_Leonard: <p>Hi,</p>
<p>Hi there ! I am training a model requiring recursive predictions. After each step, some data is fully processed, while the rest needs to be reinserted into the dataloader for further iterations.</p>
<p>How can I dynamically reinsert partially predicted data into the dataloader during training ? Are there recommended hooks or approaches for this? Should I work on dataset / dataloader / batch sampler level ?</p>
<p>Thanks in advance!</p>
Pytorch support SM_87 cuda architecture

yajuvendra: <p>Can anyone kindly help me download PyTorch, which supports sm_87 on aarch64? Right now, I am using the version below.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
print(torch.version.cuda)<br>
12.2<br>
print(torch.cuda.get_arch_list())<br>
[‘sm_50’, ‘sm_80’, ‘sm_86’, ‘sm_89’, ‘sm_90’, ‘sm_90a’]</p>
</blockquote>
</blockquote>
</blockquote>

ptrblck: <p>Check <a href="https://discuss.pytorch.org/t/cuda-and-pytorch/209517/2">this post</a>.</p>

yajuvendra: <p>Hi, i tried that and having issue. I have RHEL 9.4</p>
<p>(myenv) [yajuvendra@llmhost pytorch_wheel]$ python -m pip install torch-2.3.0-cp310-cp310-linux_aarch64.whl</p>
<p>ERROR: torch-2.3.0-cp310-cp310-linux_aarch64.whl is not a supported wheel on this platform.</p>
<p>(myenv) [yajuvendra@llmhost pytorch_wheel]$ uname -m</p>
<p>aarch64</p>
<p>(myenv) [yajuvendra@llmhost pytorch_wheel]$ uname -a</p>
<p>Linux llmhost 5.14.0-427.22.1.el9_4.aarch64 <span class="hashtag-raw">#1</span> SMP PREEMPT_DYNAMIC Mon Jun 10 15:57:50 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux</p>
<p>br…<br>
Yaju</p>

ptrblck: <p>Your Python version might be different as the wheel is built for Python 3.10.<br>
If you get stuck, you can use a pre-built container from <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-pytorch">here</a>.</p>

yajuvendra: <p>thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
How to add custom operators to export_for_training?

scott_femto: <p>I am interested in customizing the new torch.export.export_for_training with additional custom operators to add on to the ATen opset.</p>
<p>The docstring indicates that this may be possible <a href="https://github.com/pytorch/pytorch/blob/main/torch/export/__init__.py#L83:" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/export/__init__.py at main · pytorch/pytorch · GitHub</a><br>
“The traced graph (1) produces normalized operators in the all ATen operator set (as well as any user-specified custom operators)”</p>
<p>I have been unable to locate documentation or examples to show how to add my custom operation to the tracing, but am eager to learn more.</p>
<p>Also apologies if I have miscategorized this to torch.compile. I think my question is also relevant to the new pt2e quantization approach.</p>

ptrblck: <p><a href="https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html">This tutorial</a> might be a good starter explaining different paths to add custom operators (the first link shows how to add custom ops for <code>torch.compile</code> and <code>torch.export</code>).</p>
Map a tensor at each index based on its neighbors?

ecoArcGaming: <p>The problem I am trying to solve is a transformation on a 3D tensor X. For each entry X[i][j][k], I want to evaluate a function f(X[i][j][k]) that depends on the 8 neighbors (ignoring edge cases) of X[i][j][k], namely X[i±1][j±1][k±1]. Can this be done efficiently in python PyTorch?</p>

KFrank: <p>Hi Eco!</p>
<aside class="quote no-group" data-username="ecoArcGaming" data-post="1" data-topic="214061" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ecoarcgaming/48/74453_2.png" class="avatar"> ecoArcGaming:</div>
<blockquote>
<p>namely X[i±1][j±1][k±1]. Can this be done efficiently in python PyTorch?</p>
</blockquote>
</aside>
<p>If I understand your use case properly, yes.</p>
<p>Construct a specific kernel for a <code>Conv3d</code> that plucks out the eight shifted<br>
tensors that you want.  So it will be a <code>3x3x3</code> kernel with one input channel<br>
and eight output channels with each channel plane containing a single <code>1.0</code><br>
in the position that corresponds to the desired neighbor.</p>
<p>When you apply that <code>Conv3d</code> to a (single-channel) tensor, its output will be<br>
an eight-channel tensor whose channels are tensors that have been shifted<br>
so that the desired “neighbor” pixels line up with the corresponding pixels in<br>
the input tensor.</p>
<p>Best.</p>
<p>K. Frank</p>
GPU utilization at 5% when using HDF5 dataloader

radon_me: <p>I have an image dataset in HDF5 format. When I load the dataset and begin training, I see &lt;5% GPU utilization, although I see a reasonable 75% memory utilization. I am unable to narrow down the cause for it and I suspect it’s due to the hdf5 data format causing a bottleneck.<br>
Here is the data loader:</p>
<pre><code class="lang-auto">class ImageDataset(Dataset):
    def __init__(self, filename, transform=None):
        self.file_path = filename
        self.archive = None
        self.transform = transform
        with h5py.File(filename, 'r', libver='latest', swmr=True) as f:
           self.length = len(f['dataset'])

    def _get_archive(self):
        if self.archive is None:
            self.archive = h5py.File(self.file_path, 'r', libver='latest', swmr=True)
            assert self.archive.swmr_mode
        return self.archive

    def __len__(self):
        return self.length

    def __getitem__(self, idx):
        archive = self._get_archive()
        indices = archive['dataset'][str(idx)]
        indices = [indices[0].decode('utf-8'), indices[1].decode('utf-8'), int(indices[2].decode('utf-8'))]
        image_bin = archive[indices[0]][indices[1]]['scene_left_0'][indices[2]]
        img = Image.open(BytesIO(image_bin))
        if self.transform:
            img = self.transform(img)
        return img
    
    def close(self):
        self.archive.close()

training_data = ImageDataset('/home/data/data.h5', transform=transform)

train_dataloader = DataLoader(training_data, batch_size=152, shuffle=True, num_workers=12, pin_memory=True)
</code></pre>
<p>I have GPU utilization that looks like the attached image:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be5d516dc841a152218e9fa106b86cfac4176b3a.png" data-download-href="https://discuss.pytorch.org/uploads/default/be5d516dc841a152218e9fa106b86cfac4176b3a" title="Screenshot from 2024-12-11 16-09-47"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/be5d516dc841a152218e9fa106b86cfac4176b3a_2_690x270.png" alt="Screenshot from 2024-12-11 16-09-47" data-base62-sha1="ra2Es1JdLD8QqGwjQuFT9TywsjM" width="690" height="270" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/be5d516dc841a152218e9fa106b86cfac4176b3a_2_690x270.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/be5d516dc841a152218e9fa106b86cfac4176b3a_2_1035x405.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/e/be5d516dc841a152218e9fa106b86cfac4176b3a.png 2x" data-dominant-color="052F39"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-12-11 16-09-47</span><span class="informations">1220×478 23.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Sharing torch compile kernels between layers

divinho: <p>I’m in a situation where compiling does not work for the whole model, so I’m wrapping each layer with <code>torch.compile</code>. Because I have (a fixed number of) variable shapes there is a significant amount of compilation time.</p>
<p>Each layer is just a transformer layer, I’m wondering if the compilation has to be redone for each layer or is there a way to share that across layers? So the compilation only happens when the input shape changes for the first layer (and the rest inherit the new kernel).</p>
<p><a class="mention" href="/u/ezyang">@ezyang</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Caching should already be used and you could check <a href="https://pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html">this doc</a> for more information.</p>
Why the memory usage is higher than expected when loading nvidia/NV-Embed-v2 model with FP16 precision?

Arunima_Ghosh: <p>Hi,</p>
<p>I am using the <code>nvidia/NV-Embed-v2</code> model with <strong>7.85 billion parameters</strong> to generate embeddings. I’m loading the model with <strong>FP16 precision</strong> using the following code:</p>
<p>python</p>
<p>Copy code</p>
<pre><code class="lang-auto">import torch
from transformers import AutoTokenizer, AutoModel

torch.cuda.empty_cache()
device = torch.device('cuda')
model = AutoModel.from_pretrained('nvidia/NV-Embed-v2', trust_remote_code=True).to(device)
model.to(torch.float16)
</code></pre>
<p>I expect the model to take approximately <strong>15.7 GB of memory</strong> (calculated as 7.85×27.85 \times 27.85×2 GB for FP16 precision). However, when I measure the memory usage with this code—without making any inference—the reported memory consumption is about <strong>25 GB</strong>.</p>
<p>Can someone please explain why the memory usage is higher than expected? Is this extra memory usage due to model overhead (e.g., optimizer states, activation storage) or something else?</p>
<p>Thanks in advance!</p>

ptrblck: <p>I don’t know if your memory estimation is correct or not.<br>
However, you are loading the model in the default precision to your GPU first and are then transforming all registered parameters and buffers to <code>float16</code>. I also don’t know how you are measuring the used memory but note that the peak memory will of course be higher of the original model used a wider precision.</p>

Arunima_Ghosh: <p>Hi,</p>
<p>I am using <strong>nvidia-smi</strong> and <strong>nvtop</strong> to monitor GPU memory usage. Are these tools appropriate for measuring GPU memory usage, or are there other methods or tools that might provide a more accurate or comprehensive readout for this specific case?</p>

ptrblck: <p>Yes, <code>nvidia-smi</code> will report the total memory used. If you want to check the allocated and reserved memory in your script, you could use <code>torch.cuda.memory_allocated()/.memory_reserved()./memory_summary()</code>.</p>
<p>The previous point still holds: if you are loading the model in <code>float32</code> first to the GPU before transforming the parameters and buffers to <code>float16</code> later, the peak memory usage will report the memory usage from the <code>float32</code> model.</p>
Callback for Integrated Gradients

blackpearl006: <p>I am utilising PyTorch_lightning for Training and Testing and I have tried to incorporate Integradted Gradients Calculations in Test and as well of predict step. It also throws <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code>, Clueless how to tackle this, I am planning to implement IG extraction using Captum through a Callback function. Is it possible ?</p>

blackpearl006: <p>I am still stuck at the same error.</p>
<pre><code class="lang-auto">  File "/home/neelamlab/anaconda3/envs/torch2.5.1/lib/python3.12/site-packages/torch/autograd/__init__.py", line 496, in grad
    result = _engine_run_backward(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/neelamlab/anaconda3/envs/torch2.5.1/lib/python3.12/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p><strong>CallBack implementation</strong></p>
<pre data-code-wrap="python"><code class="lang-python">class IntegratedGradientsCallback(Callback):
    def __init__(self, output_dir):
        super().__init__()
        self.output_dir = output_dir

    @torch.enable_grad()
    def on_test_batch_end(self, trainer, pl_module, outputs, batch, batch_idx, dataloader_idx=0):
        inputs, labels, label_info = batch  # Unpack batch
        model = pl_module
        regression_model = RegressionHeadWrapper(model)
        outputs = regression_model(inputs)
        print(outputs)
        
        attr_map = getInputAttributions(regression_model, inputs)  # Replace with your IG computation function
        print(attr_map)
</code></pre>
<p>My model is a 2 headed model so I have a custom wrapper on it while getting predictions and performing integradted gradients</p>
<pre data-code-wrap="python3"><code class="lang-python3">class RegressionHeadWrapper(nn.Module):
    def __init__(self, model):
        super(RegressionHeadWrapper, self).__init__()
        self.model = model

    def forward(self, x):
        x_age, _ = self.model(x)  # Extract only the regression output
        return x_age

</code></pre>
<p>IG call:</p>
<pre><code class="lang-auto">def getInputAttributions(model, input_tensor):
    input_tensor.requires_grad_()
    ig = IntegratedGradients(model)
    attr = ig.attribute(input_tensor, n_steps=10, internal_batch_size=config['batch_size'])
    return attr
</code></pre>
`torch.cuda.is_available()` allocates unwanted memory?

mmitjans: <p>I want to read how much total free memory each one of my GPU devices has, so that I can automatically assign the least used device to a new process I’m launching. For this, I’m using this function:</p>
<pre><code class="lang-auto">def get_least_used_gpu():
    """Return the name of the GPU that has the most free memory.

    Returns:
        str: The name of the GPU with the least used memory, or "cpu" if no GPU is available.
    """
    if not torch.cuda.is_available():
        return "cpu"
    free_memory = [torch.cuda.mem_get_info(i)[0] for i in range(torch.cuda.device_count())]
    min_gpu = torch.argmax(torch.tensor(free_memory))
    return f"cuda:{min_gpu}"
</code></pre>
<p>However, I realized that this function was allocating up to additional 1.2 GB of data to all unused GPUs. After debugging, I realized it was the call <code>torch.cuda.is_available()</code> the one responsible for that (previously I thought it was the <code>torch.cuda.mem_get_info()</code> call). If I disable the <code>if</code> statement, only 200 MB of data are allocated to all other GPUs. What’s the reason for that?</p>
<p>Thank you,<br>
Marc</p>

ptrblck: <p>I cannot reproduce the issue using a current source build and see a driver initialization using 3MB on the device. <a href="https://github.com/pytorch/pytorch/blob/3cb7ec312c3f33d2f1494db387b901e9562168e7/torch/cuda/__init__.py#L130-L143"><code>torch.cuda.is_available()</code></a> will also use NVML if it’s available and will thus not call into <code>cuInit</code>. Also, based on the size of your context CUDA’s lazy module loading might not be used so you have either disabled it or are using an old PyTorch release.</p>

Santiago_Castellano: <p>Hi!<br>
I’m experiencing the same problem. Are there any updates on this issue?</p>
<p>Thanks!</p>
Forward Pass through different modules based on input type

nidheeshg11: <p>I want to use a DeIT model with different tokenizers based on the input type. For example, if my dataset has four types of data, I want to use four different tokenizers, one for each type but one single encoder block. Each batch will have a mix of all types of data. If a sample is of type 0, the forward pass should look like tokenizer0 → encoder → classifier, and so on for different types.<br>
Code:</p>
<pre><code class="lang-auto">class DeitWithClassifier(nn.Module):
    def __init__(self):
        super(DeitWithClassifier, self).__init__()
        self.tokenizers = {
            0: DeiTModel.from_pretrained("facebook/deit-tiny-distilled-patch16-224").embeddings,
            1: DeiTModel.from_pretrained("facebook/deit-tiny-distilled-patch16-224").embeddings,  
            2: DeiTModel.from_pretrained("facebook/deit-tiny-distilled-patch16-224").embeddings,  
            3: DeiTModel.from_pretrained("facebook/deit-tiny-distilled-patch16-224").embeddings,  
        }
        deit_model = DeiTModel.from_pretrained("facebook/deit-tiny-distilled-patch16-224")
        self.encoder = deit_model.encoder
        self.layer_norm = deit_model.layernorm
        self.pooler = deit_model.pooler
        self.projector = nn.Linear(192, 512)
        self.classifier = nn.Linear(512, 1)

    def forward(self, batch):
        output, embedding = [], []

        for t in range(len(batch['type'])):
            tokenizer = self.tokenizers[batch['type'][t]]

            tokenized = tokenizer(batch['pixel_values'][t].unsqueeze(0).to(device))
            encoded = self.encoder(tokenized).last_hidden_state
            normalized = self.layer_norm(encoded)

            emb = self.projector(normalized[0][0])
            logits = self.classifier(emb)

            output.append(logits)
            embedding.append(emb)

        output = torch.stack(output)
        embedding = torch.stack(embedding)

        return output, embedding
</code></pre>
<p>Each batch has ‘type’ which indicates the type of the samples and ‘pixel_values’.<br>
Though this code works, I am not sure if the backpropogation is happening like it is supposed to.<br>
Is using a list in the forward pass ok?<br>
Does it mess with the gradients/backpropogation? How else can I implement this?</p>

ptrblck: <aside class="quote no-group" data-username="nidheeshg11" data-post="1" data-topic="214110">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nidheeshg11/48/74189_2.png" class="avatar"> nidheeshg11:</div>
<blockquote>
<p>Is using a list in the forward pass ok?</p>
</blockquote>
</aside>
<p>Yes, using a <code>list</code> to <code>append</code> activations is fine and you won’t break the computation graph if you create a single tensor via <code>torch.cat</code> or <code>torch.stack</code> from this <code>list</code>. You should not create a new tensor via <code>torch.tensor(list_entries)</code> since this will detach the newly created tensor from the computation graph.</p>
How to add an MLP in a feedback loop (not just feedforward as usual in NN) to implement kind of recurrent NN with arbitrary structure?

neyronon: <p>I am trying to implement a neural network in a manner of control systems using feedback loops in order to develop a multichannel control system. But RNN-like layers don’t suit for this purpose. I am a little confused how can we make such kind of thing in PyTorch and how can we train that? Is it possible?</p>

ptrblck: <p>I don’t fully understand your description but PyTorch allows you to use loops and thus also to feed the model output to itself again.</p>

neyronon: <p>I mean something like in this picture. But imagine all blocks are different MLP.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.polytechnichub.com/block-diagram-process-control-system/">
  <header class="source">

      <a href="https://www.polytechnichub.com/block-diagram-process-control-system/" target="_blank" rel="noopener nofollow ugc" title="07:51AM - 29 April 2017">Polytechnic Hub – 29 Apr 17</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:600/286;"><img src="https://www.polytechnichub.com/wp-content/uploads/2017/04/closed_loop.jpg" class="thumbnail" width="600" height="286"></div>

<h3><a href="https://www.polytechnichub.com/block-diagram-process-control-system/" target="_blank" rel="noopener nofollow ugc">Block diagram of process control system - Polytechnic Hub</a></h3>

  <p>The figure shows the block diagram of close loop system or process control system. The process control system consists of process or plant, feedback elemen</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74f1bcfb879d1d523d0f1908c22f3452c586c6f9.png" data-download-href="https://discuss.pytorch.org/uploads/default/74f1bcfb879d1d523d0f1908c22f3452c586c6f9" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74f1bcfb879d1d523d0f1908c22f3452c586c6f9.png" alt="image" data-base62-sha1="gGxj77GKJslZFUZ3kI44TmWKb2x" width="600" height="286"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">600×286 63.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to liberate CUDA Memory succesfully?

Jose_Leyva: <p>Hello,</p>
<p>I am running a Jupyter Notebook with a LLM. When evaluating the model with the dataloader, it throws an error of CUDA out of memory.</p>
<p>I was investigating and try to do the eval by batches and liberate memory on the process.</p>
<p>However, I notice that the server is not releasing the memory of CUDA even after calling<br>
gc.collect() or torch.cuda.empty_cache()</p>
<p>I made a toy example to illustrate this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af1fa59e98fb53eb6f25e7f62b3371828b56f185.png" data-download-href="https://discuss.pytorch.org/uploads/default/af1fa59e98fb53eb6f25e7f62b3371828b56f185" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af1fa59e98fb53eb6f25e7f62b3371828b56f185.png" alt="image" data-base62-sha1="oZdmmbcY2OuGjpRd3NQyZJpSlE1" width="312" height="500" data-dominant-color="222323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">369×590 26.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Also, when re-running the notebook, it allocates more memory instead of overwriting it.<br>
To ensure the CUDA Memory is liberate it i need to turn off the kernel, but i don’t want to do that.</p>
<p>How can i solve this issue? Is there a problem with my CUDA installation or PyTorch?<br>
Why the torch.cuda.empty_cache() is not liberating memory?</p>

ptrblck: <p>Screenshots of code snippets cannot be copied unfortunately so we are are unable to reproduce the issue. Clearing the cache works fine as seen in <a href="https://discuss.pytorch.org/t/free-all-gpu-memory-used-in-between-runs/168202/2">this post</a>  including a minimal and executable code snippet.</p>
When using class weights is bad?

laro: <ul>
<li>I have a DB with 50 different classes.</li>
<li>One of the classes has x10 more data than the other classes.</li>
<li>Each class has ~20K samples and the ‘big’ class has ~200K samples</li>
<li>When training classification model <strong>without</strong> class weights - I’m getting slightly better results than training the model with class weights.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/919896bc260021d3986cd2b5c0d8ab2aa9974ac3.png" data-download-href="https://discuss.pytorch.org/uploads/default/919896bc260021d3986cd2b5c0d8ab2aa9974ac3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/919896bc260021d3986cd2b5c0d8ab2aa9974ac3.png" alt="image" data-base62-sha1="kM09v0RdpUImOBPNogKAAEGAJ6r" width="690" height="101" data-dominant-color="F2F2F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1113×164 4.67 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ul>
<ol>
<li>How can it be ?</li>
<li>According to my results - it seems that if I have an important class which I don’t want to have many mistakes - it is better to get more samples and it will not affect badly on the other classes, am I right ?</li>
</ol>

ptrblck: <p>The small difference in accuracy could probably be explained with the <a href="https://en.m.wikipedia.org/wiki/Accuracy_paradox#:~:text=The%20accuracy%20paradox%20is%20the,too%20crude%20to%20be%20useful.">Accuracy Paradox</a>.</p>

laro: <p>I ran  <strong>classification_report</strong> and checked the F1 for each class.<br>
The results (F1) are very very close (&lt;1% difference)  (for each classes) between with or without class weights.<br>
So, according to my results, is there any logic why do I need to use class weights ?</p>

ptrblck: <p>Class weights are used to add a penalty to the model training e.g. to counter overfitting a majority class. Based on your previous output the model is already overfitting to the majority class.</p>
How to Load Llama-3.3-70B-Instruct Model in Float8 Precision?

Arunima_Ghosh: <p>Hi everyone,</p>
<p>I am currently using the following code snippet to load the Llama-3.3-70B-Instruct model in BF16 precision:</p>
<pre><code class="lang-auto">import transformers
import torch

model_id = "meta-llama/Llama-3.3-70B-Instruct"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

outputs = pipeline(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])
</code></pre>
<p>However, I want to load this model in Float8 (FP8) precision instead of BF16 or FP16 to optimize performance and memory usage.</p>
<p>Could someone guide me on:</p>
<ol>
<li>Whether it’s possible to load the Llama-3.3-70B-Instruct model with FP8 precision using the Hugging Face Transformers library?</li>
<li>Any necessary modifications to the above code or additional libraries required for enabling FP8 precision?</li>
</ol>
<p>Thank you in advance for your help!</p>