akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RedHat, CUDA 12.4 - torch GPU not avaiable

username: Jonny
cooked: <p>Torch version 2.4.0+cu124 is installted,<br>
The nvidia driver 550.90.07 is also installed.<br>
nvcc (release 12.4, V12.4.131) is installed.</p>
<p>But torch.cuda.is_available() returns False.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90cc135b4a8861cc55ab4dfbc3be7b4f997beb52.png" data-download-href="https://discuss.pytorch.org/uploads/default/90cc135b4a8861cc55ab4dfbc3be7b4f997beb52" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90cc135b4a8861cc55ab4dfbc3be7b4f997beb52.png" alt="image" data-base62-sha1="kEVZ78bA1QY9tMJfQA5SHJBQe5k" width="690" height="434" data-dominant-color="151515"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">730×460 9.82 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does someone has an idea to fix this issue.<br>
Why does torch not work on Red Hat Enterprise Linux 8.10 (Ootpa), Linux 4.18.0-553.8.1.el8_10.x86_64 with CUDA?</p>

username: ptrblck
cooked: <p>You’ve enabled <code>MIG</code> so make sure the right MIG setup is used and the desired slice is passed to <code>CUDA_VISIBLE_DEVICES</code>.</p>
Issue with torch.layer_norm Function Returning NaN Values

username: HichTala
cooked: <p>Hi,<br>
I’m encountering an issue with the <code>torch.layer_norm</code> function, where it returns a Tensor containing <code>NaN</code> values under certain inputs. I’m unsure if this behavior indicates a bug or if it’s expected.</p>
<p>Here’s a minimal code snippet that reproduces the problem:</p>
<pre><code class="lang-auto">import torch

features = torch.load('features.pt')
print(torch.isnan(features).any())
print(torch.isnan(torch.layer_norm(features, (64,))).any())
</code></pre>
<p>And <a href="https://colab.research.google.com/drive/1afQyk11s0CCOFogrg1xQFiQQlwEbxpxq?usp=sharing" rel="noopener nofollow ugc">here</a> a colab notebook that reproduces the problem.</p>
<p>Please find the input tensor <a href="https://drive.google.com/file/d/1QEQq_MA_vevJ-NyKyOfOB9xemSBKcFBF/view?usp=sharing" rel="noopener nofollow ugc">here</a>.</p>
<p>Thank you for your help!</p>
Facing inplace operation in NTM similar works

username: yf_chen
cooked: <p>I create a model and it’s similar to NTM(neural turing mathine) .<br>
But when it comes to writing module , it got a <code>inplace operation</code> error.  The error happens in the following code.</p>
<pre><code class="lang-auto">                w_pattern = self.fc2(pattern[i])
                prev_mem = self.mem

                erase = self.get_erase(w_pattern)  # [1, M]
                era_mat = torch.zeros(self.N, self.M)
                era_mat[self.num_patterns, :] = erase

                add = weight[i] * w_pattern # [1, M]
                add_mat = torch.zeros(self.N, self.M)
                add_mat[self.num_patterns, :] = add
                
                # this line below got an error
                self.mem = (prev_mem * (torch.ones(self.N, self.M) - era_mat) + add_mat)

                self.access_order.append(self.num_patterns)
                self.num_patterns = self.num_patterns + 1

</code></pre>
<p>I found this line because I debugged line by line and found that in this line the property self.mem. _version has changed from 1 to 0.<br>
But i don’t know how to solve it ,I’m still trying</p>

username: yf_chen
cooked: <p>I mimicked it from this <a href="https://github.com/loudinthecloud/pytorch-ntm/blob/master/ntm/memory.py" rel="noopener nofollow ugc">code</a>.</p>
<pre><code class="lang-auto">    def write(self, w, e, a):
        """write to memory (according to section 3.2)."""
        self.prev_mem = self.memory
        self.memory = torch.Tensor(self.batch_size, self.N, self.M)
        erase = torch.matmul(w.unsqueeze(-1), e.unsqueeze(1))
        add = torch.matmul(w.unsqueeze(-1), a.unsqueeze(1))
        self.memory = self.prev_mem * (1 - erase) + add

</code></pre>
OpenCL backend for pytorch - now easily installables

username: artyom-beilis
cooked: <p>Hello Dear Pytorch users,</p>
<p>I released a new version of <a href="https://github.com/artyom-beilis/pytorch_dlprim" rel="noopener nofollow ugc">pytorch out of tree OpenCL backend</a> - it allows to train your models on AMD, NVidia and even Intel GPUs on both Windows and Linux - without platform specific mess.</p>
<p>What is new is that I provide now prebuilt <code>whl</code> files for easy installation for pytorch 2.4, See: <a href="https://github.com/artyom-beilis/pytorch_dlprim?tab=readme-ov-file#installation" class="inline-onebox" rel="noopener nofollow ugc">GitHub - artyom-beilis/pytorch_dlprim: DLPrimitives/OpenCL out of tree backend for pytorch</a></p>
<p>And how is the performance? While it isn’t as good as native cuda or rocm, depending on network it still competitive result, especially for ROCM: <a href="https://github.com/artyom-beilis/pytorch_dlprim/blob/master/benchmark.md" class="inline-onebox" rel="noopener nofollow ugc">pytorch_dlprim/benchmark.md at master · artyom-beilis/pytorch_dlprim · GitHub</a></p>
<p>I encourage everybody to test it.</p>
<p>It still has lots of gaps in both functionality and performance - but I think having true open-source and cross platform training backend is very important for the future of deep learing.</p>

username: artyom-beilis
cooked: <p>I <a href="https://github.com/artyom-beilis/pytorch_dlprim/releases/tag/0.2.0" rel="noopener nofollow ugc">released a new version 0.2.0</a> of the OpenCL backend - including binary whl files for pytorch 2.4</p>
<h2><a name="p-453692-in-the-nutshell-1" class="anchor" href="#p-453692-in-the-nutshell-1"></a>In the nutshell</h2>
<ul>
<li>New Intel Arch GPU is now tested and performance improvements added.</li>
<li>Visual transformers are now validated and working.</li>
<li>Many operators implemented and many fixed</li>
</ul>
<p>Benchmarks are updated to <a href="https://github.com/artyom-beilis/pytorch_dlprim/blob/master/benchmarks/v0.2.0/summary.md" rel="noopener nofollow ugc">latest verson</a></p>
<p>In a short summary comparing OpenCL vs native performance (rocm/cuda/xpu)</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Vendor</th>
<th>Device</th>
<th>% Perf Train</th>
<th>% Perf Test</th>
<th>compare to</th>
</tr>
</thead>
<tbody>
<tr>
<td>AMD</td>
<td>rx6600XT</td>
<td>71%</td>
<td>85%</td>
<td>rocm</td>
</tr>
<tr>
<td>NVidia</td>
<td>gtx960</td>
<td>65%</td>
<td>69%</td>
<td>cuda</td>
</tr>
<tr>
<td>Intel</td>
<td>Arc A380</td>
<td>40%</td>
<td>54%</td>
<td>xpu</td>
</tr>
</tbody>
</table>
</div><p>Unfortunatly my code isn’t optimal enough for Intel, but on the good side I think I can integrate oneDNN quite easily - but this is next step</p>
Pass large data into mp.spawn

username: Tigran_Ishkhanov
cooked: <p>Hello,</p>
<p>I have a question about <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.4.0+cu121 documentation</a></p>
<p>There is a large collator and dataset objects that I’m passing to the train function like this:</p>
<p>if world_size &gt; 0:<br>
mp.spawn(train_gpu, args=(world_size, model, collator, dataset),<br>
nprocs=world_size, join=True)</p>
<p>Is it better to create all the large objects inside the function train_gpu() or pass them like above in terms of memory usage?</p>
<p>Thank you!</p>
Why bf16 do not need loss scaling?

username: walle_autoscale
cooked: <p>I read in <a href="https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html" rel="noopener nofollow ugc">this</a> post that when using fp16 mixed precision, we need loss-scaling to <a href="https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html#lossscaling" rel="noopener nofollow ugc">Preserve Small Gradient Magnitudes</a>.</p>
<p>However, bf16 has less fraction bits than fp16, so I think using bf16 will not be able to preserve small gradient values. So it seems that loss scaling is also needed in fp16.</p>
<p>Can you help me figure out this?</p>

username: J_Johnson
cooked: <p>When a computer represents a number, there are certain parts of that memory allocated for exponents and certain parts allocated for precision. The exponents give a wider range of numbers(i.e. -10^32 to 10^32) while precision bits give more incremental precision in steps between two numbers(i.e. 1.1, 1.2, 1.3, … vs. 1.01, 1.02, 1.03, …).</p>
<p>bfloat16 allocates more memory for exponents and less for precision.</p>

username: walle_autoscale
cooked: <p>Thanks, but I still do not understand why bf16 do not need the loss scaling for better precision. since in fp16, we need loss scaling to avoid small gradient values becoming zero. And in fp16, typically we need to amplify the loss to preserve small gradient values.</p>

username: J_Johnson
cooked: <p>bfloat16 can go all the way down to ~10e-38 whereas float16s smallest value is ~6e-8. Does it make sense why that might be beneficial when many model parameters can often be below that threshold?</p>

username: cyk2018
cooked: <p>why we need loss scaling?<br>
because if the loss is &gt; 0 but &lt; smallest data fp16 can be. the data in fp16 will be 0<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08422fd1db7bcf8a18bf6ec2dd8268f7ec229682.png" alt="image" data-base62-sha1="1b3CHSTVzDd5Cb7BPMzeFhOFdGG" width="663" height="370"><br>
Looking this picture, many data is in this interval. So if we take bf16 we can show this without loss scaling.</p>
Setting DTensor OpDispatcher's allow_implicit_replication flag from environment variable for distributed inference of HuggingFace models

username: pgimenes
cooked: <p>When the <a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/_tensor/README.md" rel="noopener nofollow ugc">DTensor</a> dispatcher encounters an op that with a mixture of Tensor and DTensor arguments, the following error is raised:</p>
<pre><code class="lang-auto">  File "/home/.../torch/distributed/_tensor/_dispatch.py", line 354, in try_get_replicate_spec
    raise RuntimeError(
RuntimeError: aten.embedding.default: got mixed torch.Tensor and DTensor, need to convert all torch.Tensor to DTensor before calling distributed operators!
</code></pre>
<p>This is annoying since models from HuggingFace use tensors created at runtime which are not set as module attributes, and hence can’t be casted to DTensor before a forward pass. This behaviour can be avoided by setting the <code>self._allow_implicit_replication</code> to True in the OpDispatcher at <code>torch/distributed/_tensor/_dispatch.py</code>, which currently requires either changing local installation files or forking torch. Can this be changed to something like the following?</p>
<pre><code class="lang-auto">self._allow_implicit_replication = os.environ["TORCH_DTENSOR_ALLOW_IMPLICIT_REPLICATION"]
</code></pre>

username: yf225
cooked: <p>Would you like to create an issue on <a href="https://github.com/pytorch/pytorch/issues?q=sort%3Aupdated-desc+is%3Aissue+is%3Aopen" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a> for DTensor maintainers to discuss this? Thanks!</p>

username: irisz
cooked: <aside class="quote no-group" data-username="pgimenes" data-post="1" data-topic="206326">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pgimenes/48/71361_2.png" class="avatar"> pgimenes:</div>
<blockquote>
<p>When the <a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/_tensor/README.md" rel="noopener nofollow ugc">DTensor</a> dispatcher encounters an op that with a mixture of Tensor and DTensor arguments, the following error is raised:</p>
<pre><code class="lang-auto">
</code></pre>
</blockquote>
</aside>
<p>Could you use the implicit_replicatin() context instead? For example: <a href="https://github.com/pytorch/pytorch/blob/main/test/distributed/_tensor/test_dtensor.py#L782" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test/distributed/_tensor/test_dtensor.py at main · pytorch/pytorch · GitHub</a></p>
How to make same input/output for autoencoder?

username: nim9405
cooked: <p>I want to use autoencoder’s latent vector as feature extractor. For that, I want to put same input/output image for the autoencoder model.</p>
<p>I want to make “input_size” in both encoder &amp; decoder as a same image. Current model’s result is not a fixed. it just assigned the size of the output space.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

# Define the autoencoder architecture
class Autoencoder(nn.Module):
    def __init__(self, input_size, latent_dim):
        super(Autoencoder, self).__init__()

        # Encoder
        self.encoder = nn.Sequential(
            nn.Linear(input_size, 64),
            nn.ReLU(),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Linear(32, latent_dim)
        )

        # Decoder
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 32),
            nn.ReLU(),
            nn.Linear(32, 64),
            nn.ReLU(),
            nn.Linear(64, input_size)
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x
</code></pre>

username: ptrblck
cooked: <p>I don’t fully understand your question as your code already shows the reuse of <code>input_size</code> making sure the output activation has the same shape as the input.</p>

username: nim9405
cooked: <p>Hi, thanks for the reply.</p>
<p>What I want to do is not the fix the shape, but fix the data value.</p>
<p>As you mentioned, “input_size” in decoder only fix the size of the output, not the value.</p>
<p>I want to use exactly same data with input as the output of the model.</p>
<p>Thanks!</p>

username: deepsthecoder
cooked: <aside class="quote no-group" data-username="nim9405" data-post="1" data-topic="209105">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nim9405/48/72533_2.png" class="avatar"> nim9405:</div>
<blockquote>
<p><code>input_size</code></p>
</blockquote>
</aside>
<p>In my understanding, below is your task.</p>
<blockquote>
<p>I want to use autoencoder’s latent vector as feature extractor</p>
</blockquote>
<p>But, the approach that you mention does not seem to be right:</p>
<blockquote>
<p>For that, I want to put same input/output image for the autoencoder model.</p>
</blockquote>
<p>If you need to train your autoencoder to get the latent feature vector from your input image, you need to then train the decoder to reconstruct the original image itself.<br>
For this you need to compare the <code>x</code> output from the decoder to be same as the input image. This is where loss function comes into picture.</p>
<p>Simple approach is to compute L2 loss between the input image and the output from the decoder.</p>
<p>For extraction of latent features you could use the output of the encoder, once you finish training the model.</p>

username: nim9405
cooked: <p>Understood.<br>
Thanks for correcting me.</p>
How to improve training performance with Apex package

username: tjk
cooked: <p>Hello!</p>
<p>I am using NVIDIA Apex package for speeding up training of my CNN model, I compare the performance between using traditional Adam algorithm and using Apex O1 optimization technique with the following code:</p>
<pre><code class="lang-auto">optimizer = optim.Adam(model.parameters(), lr = 1e-3)
model, optimizer = amp.initialize(model, optimizer, opt_level = 'O1')
</code></pre>
<p>The training process is speed up 3-4 times visibly compared with traditional Adam algorithm. But when I check the training model, I find it turns out performance of model trained with Apex in testing sets is degraded compared with just using Adam. Are there any solutions? Because I want to speed up training process and obtain good performances on testing datasets.</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p><code>apex.amp</code> is deprecated and <code>torch.amp</code> utils. should be used instead.</p>

username: tjk
cooked: <p>Thanks for notifying that.</p>

username: tjk
cooked: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a>. I have switched to torch.amp utils, but I find out another problem.  Also iterating 500 epoches, using torch.utils is more prone to overfitting than the direct Adam-only method, how can I solve this problem?</p>

username: ptrblck
cooked: <p>Could you describe what exactly you are using from <code>torch.utils</code> that seems to be responsible for the overfitting you are seeing?</p>

username: tjk
cooked: <p>I suspect the autocast function in torch is responsible for overfitiing</p>
<pre><code class="lang-auto">with autocast():
            outputs = model(inputs)
            loss = criterion(outputs, labels)
</code></pre>

username: ptrblck
cooked: <p>We haven’t seen any overfitting issues using <code>amp</code> and a few examples are given e.g. in <a href="https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision/#accuracy-amp-fp16-fp32">this blog post</a>. It would be great if you could provide more information about the expected target accuracy and the the mean+/-stddev of the achieved accuracy when comparing different approaches.</p>
What build steps are needed to reproduce libtorch release build?

username: ryguy
cooked: <p>Hello, I am interested in building libtorch, specifically in a way that reproduces the binaries that are packaged into this release:<br>
<a href="https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-2.1.1%2Bcpu.zip" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-2.1.1%2Bcpu.zip</a></p>
<p>Does anyone have a build recipe that they can share?</p>
<p>Thanks!<br>
Ryan</p>
Deploying image segmentation YOLO models using torchscript and Next.js

username: raian-ruku
cooked: <p>I have trained YOLOv7, YOLOv8 &amp; YOLOv9 using pytorch, annotated with RoboFlow. I want to build a website using Next.js for online segmentation. I have found a few tutorials on deploying using Flask and ONNX formats. but as I have no idea about Flask and have already trained the models, I really want to use torchscript. is there any way to do so?</p>
Overlapping compute and repeated communication

username: Jacob_Buckman
cooked: <p>I want to overlap the following two operations: (1) a big matmul (2) a distributed cumulative sum, across 8 nodes.</p>
<p>The first way I tried to do this was:</p>
<pre><code class="lang-auto"># A, B, and C are arrays that already exist
future = irecv(previous_acc, RANK-1)
compute_matmul(B, C)
future.wait()
isend(previous_acc + A)
</code></pre>
<p>But this would only overlap the very first communication (from 0 → 1) with the computation, because node 0 will not send to node 1 until it is finished with its matmul.</p>
<p>To try to fix, this I considered a second way:</p>
<pre><code class="lang-auto"># A, B, and C are arrays that already exist
recv(previous_acc, RANK-1)
isend(previous_acc + A)
compute_matmul(B, C)
</code></pre>
<p>But this would mean that node 7 has to wait until all communications (0 → 1 → … → 7) are complete before it starts computing its matmul. So once again, we fail to overlap.</p>
<p>Is there a pattern or tool that solves this? I’ve tried splitting execution into Python threads, and even into CUDA streams, but nothing has worked so far.</p>
How does Pytorch sequence together modules

username: whoab
cooked: <p>I am curious how Pytorch sequences together modules.</p>
<p>The application is, suppose I have a module structure A-&gt;B-&gt;C-&gt;D-E.</p>
<p>Then, I want to inject new modules A-&gt;B-&gt;G-&gt;H-&gt;C-&gt;D-&gt;E in the middle.</p>
<p>In general, I want to take any existing module and splice new modules into it.</p>
<p>I know I can replace modules by modifying the attribute. For example, resnet.fc = TransformerBlock would insert a transformer block for the fully connected layer of resnet.</p>
<p>So under the hood, does Pytorch store a sequential list of attributes to parse over, passing the output from one module as the input to the next?</p>

username: ptrblck
cooked: <p>It depends on the model implementation and while simple models can be created using an <code>nn.Sequential</code> container, the majority of the models use custom <code>nn.Module</code>s defining the forward pass in their <code>forward</code> method as seen <a href="https://github.com/pytorch/vision/blob/d0ebeb55573820df89fa24f6418b9e624683932d/torchvision/models/resnet.py#L266-L285">here</a> for e.g. ResNet models.</p>
Classifier Outputs The Same Probabilities For All Samples

username: JONATHAN_ASH
cooked: <p>Hello,</p>
<p>I have trained a number of models with different hyper-parameters. I save the state dict for each model at every 10 epochs, and load them in a separate script for inference. However, the saved models output the exact same probability distribution for all samples. To clarify, there are 3 possible classes, and the probability for class 0 in sample 1 is exactly the same as the probability for class 0 in all other samples. I have confirmed that the data has been loaded correctly, and that the models have been trained correctly via plotting the loss curve. I am completely puzzled, and desperately need help diagnosing the issue. I have included my inference code below. Any and all help would be greatly appreciated. Thanks!</p>
<pre><code class="lang-auto">import torch
import pandas as pd
import os
import torch.nn as nn
import torch.nn.functional as F
import argparse
from sklearn.model_selection import train_test_split
from sklearn.metrics import balanced_accuracy_score, confusion_matrix
import numpy as np
from LA_model import Net, load_data

parser = argparse.ArgumentParser()
parser.add_argument("--model_name", type=str)
args = parser.parse_args()

device=torch.device('cuda' if torch.cuda.is_available() else 'cpu')
df_train=pd.read_csv('train_set.csv')
df_test=pd.read_csv('test_set.csv')

model = Net()
model.load_state_dict(torch.load('saved_models/'+args.model_name+'.pt',  weights_only=True))
model.to(device=device)
model.eval()
with torch.no_grad():
    train_pred=[]
    train_label=[]
    test_pred=[]
    test_label=[]
    bctr=0
    for i in range(len(df_train)//100):
        cur_x, y = load_data(df_train, bctr, 100, device)
        bctr+=1
        print(model(cur_x))
        y_hat = torch.argmax(model(cur_x),1)
        for a, b in zip(y, y_hat):
            train_pred.append(int(b))
            train_label.append(int(a))
    bctr=0
    for i in range(len(df_test)//100):
        cur_x, y = load_data(df_test, bctr, 100, device)
        bctr+=1
        y_hat = torch.argmax(model(cur_x),1)
        for a, b in zip(y, y_hat):
            test_pred.append(int(b))
            test_label.append(int(a))
    acc = balanced_accuracy_score(test_label, test_pred)
    train_acc = balanced_accuracy_score(train_label, train_pred)
</code></pre>
Feature extraction + fine tuning in resnet50

username: sguerin
cooked: <p>Hi! I’m trying to train resnet50 for binary classification [in a very small dataset (600 MRI images)]. I read <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">this tutorial</a> and I’ve tried very different configurations of learning rate, and custom fc layer. The code bellow is the configuration that gets best results so far.</p>
<p>When doing Feature Extraction with custom FC layer the model gets 75% acc max.<br>
When doing Fine Tuning with custom FC layer the model gets 85% acc max.<br>
When doing FE and then FT It gets 90% max.</p>
<p>I want to try Feature Extraction followed by Fine Tuning, maybe that approach could get better results. I have done the following:</p>
<pre><code class="lang-python">#  Load the model

model = models.resnet50(pretrained=True, progress=True)

#  Freeze all parameters in the base model

for param in model.parameters():
   param.requires_grad = False

# Replace FC layer with custom layer

model.fc = nn.Sequential(nn.Dropout(),
                         nn.Linear(model.fc.in_features, 1),
                         nn.Sigmoid())

# Train, optimize only FC layer
model.train()
criterion = nn.BCELoss()
optimizer = torch.optim.SGD(self.model.fc.parameters(), lr=self.learning_rate)

# Eval and save best model
model.eval()
torch.save(self.model.state_dict(), 'state.pth')
</code></pre>
<p>In the resulting model I perform fine tuning. I’ve try this:</p>
<pre><code class="lang-python"># Load state dict of last model
model = self.model
            model.load_state_dict(torch.load(state))
            model.to(device)

# Freze parameters only in fully conected layer
for param in model.fc.parameters():
    param.requires_grad = False
for param in model.parameters():
    if param != model.fc.parameters():
    param.requires_grad = True

# Train, optimize all layers except FC
criterion = nn.BCELoss()
base_parameters = [self.model.conv1.parameters(),
                   self.model.bn1.parameters(),
                   self.model.layer1.parameters(),
                   self.model.layer2.parameters(),
                   self.model.layer3.parameters(),
                   self.model.layer4.parameters()]
optimizer = torch.optim.SGD(itertools.chain(*base_parameters), lr=0.01)
</code></pre>
<p>I have some questions in my mind:</p>
<ol>
<li>Am I right when I think that FE → FT it’s a better approach but I’m not doing it right?</li>
<li>How can I call parameters in groups? (Base model parameters and Fully connected layer parameters)</li>
<li>What should I try to improve my model?</li>
</ol>
<p><strong>Thanks in advance for the help!</strong></p>

username: ali_hamidi
cooked: <p>Did you find any solution? Let me know the best solution if you found it please.</p>
Torch error The specified module could not be found.

username: Pranav_Harer
cooked: <p>I am trying to install  torch with cuda support specifically <code>torch==2.4.0+cu124</code>  although i get error</p>
<pre><code>Traceback (most recent call last):
  File "C:\Users\\sftprog.py", line 6, in &lt;module&gt;
    import torch
    ^^^^^^^^^^^^
  File "C:\Users\venv\Lib\site-packages\torch\__init__.py", line 148, in &lt;module&gt;
    raise err
OSError: [WinError 126] The specified module could not be found. Error loading "C:\Users\\venv\Lib\site-packages\torch\lib\fbgemm.dll" or one of its dependencies
</code></pre>
<p>I tried installing normal version of torch and recreating venv adn installing again but to no avail,I am not sure what i am doing wrong</p>

username: ptrblck
cooked: <p>This is a known issue described <a href="https://discuss.pytorch.org/t/failed-to-import-pytorch-fbgemm-dll-or-one-of-its-dependencies-is-missing/201969/63">here</a> and the nightly binary as well as <code>2.4.1RC</code> builds should already fix it.</p>
Are there any opensource codes that implement Dynamic Sparse Training in pytorch?

username: Mayank_Kumar
cooked: <p>I was able to find one here: <a href="https://github.com/junjieliu2910/DynamicSparseTraining" class="inline-onebox" rel="noopener nofollow ugc">GitHub - junjieliu2910/DynamicSparseTraining: [ICLR-2020] Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers.</a></p>
Cuda is not availble on 2080 TI

username: Abdelkareem_Elkhateb
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82a884f93628c94bb480f34ee198de02a2728c75.png" data-download-href="https://discuss.pytorch.org/uploads/default/82a884f93628c94bb480f34ee198de02a2728c75" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82a884f93628c94bb480f34ee198de02a2728c75.png" alt="image" data-base62-sha1="iDR7AntWmQRYuPcr0vW95fF5kMZ" width="690" height="212" data-dominant-color="141414"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">971×299 16.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>file:///home/kareem/Pictures/Screenshots/Screenshot%20from%202024-09-03%2009-03-32.png<br>
I have a windows server, i am trying to install pytorch 2 with cuda 11.8 but the Cuda is not availbe any help please.</p>

username: ptrblck
cooked: <p>All of our current binaries work on a 2080 Ti and unfortunately your post does not share sufficient information to be able to help.</p>

username: Abdelkareem_Elkhateb
cooked: <p>What information is needed</p>
Affine transformation with different angles and translations

username: PabloVD
cooked: <p>Hi, I have a batch of images of the form (Batch, Channels, Height, Width) and I would like to perform a batched affine transformation using different translations, angles (and possibly rotation centers) for each element in the batch. Thus, I would have translation and angle arrays with shape (Batch, 2) and (Batch) respectively.<br>
However, it seems that the <code>torchvision.transforms.functional.affine</code> implementation only allows to use one value for the angle and translation, which acts on the full batch.<br>
Is there any way I can perform this transformation in a batched way? Thanks in advance.</p>
Data transferring bettween GPUs

username: degawong
cooked: <pre><code class="lang-auto">import torch
a=torch.tensor([1,2,3,4,5,6,7,8,9,10]).cuda(0)
print(a)
a=a.cuda(1)
print(a)
</code></pre>
<p>When the GPU is 2080, the output is</p>
<pre><code class="lang-auto">tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10], device='cuda:0')
tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10], device='cuda:1')
</code></pre>
<p>which meets my expectations. However, when the GPU is 4090, the output is</p>
<pre><code class="lang-auto">tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10], device='cuda:0')
tensor([ 0,  0,  0,  0,  0,  0,  0,  0,  0, 0], device='cuda:1')
</code></pre>
<p>It seems that the memory allocation was successful, but the data copy operation did not occur. Is it possible that the 40x0 series GPUs require additional configuration steps?</p>

username: ptrblck
cooked: <p>Disable p2p support on your 40xx series or update your NVIDIA driver as an older driver allowed the usage of the unsupported p2p connectivity and could thus yield to data corruption.</p>
New Custom Operators manual for triton server

username: HadiSDev
cooked: <p>After 2.4, torch is chaning the way customer operators are are created in C++. Previously for torchscript I could generate a custom library that could be exported to a triton server like : <a href="https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/custom_operations.html" class="inline-onebox" rel="noopener nofollow ugc">Custom Operations — NVIDIA Triton Inference Server</a>.</p>
<p>How would I be able to do that in the new way? I don’t see that part in the manual.</p>
<p>Thanks in advance</p>
How to get required input shape of traced model on libtorch

username: kohyoung
cooked: <p>Hi, The input size of the model needs to be estimated in libtorch.<br>
Here’s what I tried:</p>
<h2><a name="p-453525-step-1" class="anchor" href="#p-453525-step-1"></a>Step</h2>
<ol>
<li>load PyTorch model on Python</li>
<li>save traced model on Python</li>
<li>load traced model on libtorch (C++)</li>
</ol>
<h3><a name="p-453525-h-1-2-load-and-save-on-python-2" class="anchor" href="#p-453525-h-1-2-load-and-save-on-python-2"></a>1-2, load and save on Python</h3>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torchvision
from torchvision.models import ResNet18_Weights

model = torchvision.models.resnet18(weights=ResNet18_Weights.IMAGENET1K_V1).cuda()
model.eval()
example_in = torch.rand(1, 3, 224, 224).cuda()
traced_model = torch.jit.trace(model, example_in)
torch.jit.save(traced_model, "traced_resnet_model.pt")
</code></pre>
<h3><a name="p-453525-h-3-load-traced-model-on-libtorch-3" class="anchor" href="#p-453525-h-3-load-traced-model-on-libtorch-3"></a>3, load traced model on libtorch</h3>
<pre data-code-wrap="cpp"><code class="lang-cpp">#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt; // One-stop header.

int main(void)
{
	torch::jit::script::Module module;
	try {
		module = torch::jit::load("path/to/traced_resnet_model.pt");
	}
	catch (const c10::Error&amp; e) {
		std::cerr &lt;&lt; "Failed\n";
		return -1;
	}

	auto input_type = module.get_method("forward")
		.function()
		.getSchema()
		.arguments()[1]
		.type()
		-&gt;cast&lt;c10::TensorType&gt;();

	if (input_type) {
		auto sizes = input_type-&gt;sizes();
		if (!sizes.concrete_sizes()) {
			std::cerr &lt;&lt; "Unable to estimate input size information." &lt;&lt; std::endl;
		}
		else {
			std::cout &lt;&lt; "Input tensor shape: ";
			for (const auto&amp; size : *sizes.concrete_sizes()) {
				std::cout &lt;&lt; size &lt;&lt; " ";
			}
			std::cout &lt;&lt; std::endl;
		}
	}
	else {
		std::cerr &lt;&lt; "Unable to get TensorType information." &lt;&lt; std::endl;
	}

	return 0;

</code></pre>
<p>I want to get the same dimension as <code>example_in</code>.<br>
Can anyone help me.<br>
Thank you !</p>
Reference to a runtime error: One of the variables needed to calculate the gradient has been changed using the built-in operation: [torch.cuda.FloatTensor [3, 3]], which is the result of 0 for ReluBackward0, has version 1; version 0 was expected instead

username: Ruziy
cooked: <p>I’m trying to train a model using this code <a href="https://github.com/toandaominh1997/EfficientDet.Pytorch?tab=readme-ov-file#training-efficientdet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - toandaominh1997/EfficientDet.Pytorch: Implementation EfficientDet: Scalable and Efficient Object Detection in PyTorch</a> but such a mistake comes out, what should I do?</p>

username: Soumya_Kundu
cooked: <p>This generally indicates that the backward pass is failing because the way the gradient flow is taking place is breaking down because one of the components is breaking the pass.</p>
<p>I imagine the model itself seems faulty.</p>
shap.DeepExplainer(model, background) method of SHAP is not working to explain classification results of VGG16 model

username: Prajakta_Rathod
cooked: <p>I am using SHAP method to explain the classification results of VGG16 model for three classes. I am using GPU. My code is as follows</p>
<p>code for VGG16 model is:</p>
<p>class CNNModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(CNNModel, self).<strong>init</strong>()<br>
self.vgg16 = models.vgg16(pretrained=True)<br>
in_feats = self.vgg16.classifier[6].in_features<br>
self.vgg16.classifier[6] = nn.Linear(in_feats, 3)</p>
<pre><code>def forward(self, x):
    x = self.vgg16(x)
    return x
</code></pre>
<p>code for SHAP method is:</p>
<p>batch = next(iter(test_loader))<br>
images, _ = batch<br>
background = images[:10].to(device)<br>
test_images = images[1:6].to(device)<br>
e = shap.DeepExplainer(model, background)<br>
shap_values, indexes = e.shap_values(test_images, ranked_outputs=3)</p>
<p>I am getting following runtime error</p>
<p>RuntimeError: Output 0 of BackwardHookFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.</p>
<p>Previously this code was working but now it does not work. Can anyone please help me to solve this error</p>
Dynamic Node assigning in DDP

username: omidnr
cooked: <p>I have few servers with different configurations to run a DDP training job, I want to take into account also the networking aspects so If the one of the servers wasting the time and makes the training not better even worse so I want to do something dynamic to evaluate the impact of a node in the training with respect to its speed and affecting the training time so if it has negative impact it should dynamically removed from the job. is there any way to this automatically?</p>
<ul>
<li>I use torchrun to run my code.</li>
</ul>
Modifying `_modules` is equivalent to modify member variable in a model?

username: jack_S
cooked: <pre><code class="lang-auto">from torch import nn

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self._add_module()

    def _add_module(self):
        self._modules['test'] = nn.Linear(3, 5)

m = MyModel()
print(m.test)
</code></pre>
<p>The result of the code above is exactly</p>
<pre><code class="lang-auto">Linear(in_features=3, out_features=5, bias=True)
</code></pre>
<p>I found this interesting. Why is this happening？</p>
Import Dataloader couldnt be resolved

username: Sash
cooked: <p>I am a new beginner in pytorch.<br>
I try to import these:</p>
<p>import torch<br>
from torch import nn<br>
from torch.utils.data<br>
import DataLoader from torchvision<br>
import datasets from torchvision.transforms import ToTensor</p>
<p>but Studio Code cant resolve it</p>

username: Soumya_Kundu
cooked: <p>Hello!</p>
<p>Your format/syntax of calling the imports seem to be faulty.</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets
from torchvision.transforms import ToTensor
</code></pre>
<p>I think you could try this.</p>
<p><a href="https://www.w3schools.com/python/python_modules.asp" class="inline-onebox" rel="noopener nofollow ugc">Python Modules</a> This is a nice intro to what import actually is!</p>

username: Sash
cooked: <p>oooohhhhhh thaaaaank you.<br>
It was the code from a tutorial, and I was changeing everything on my system.</p>
<p>Thank you.</p>
<p>Thats the whole code.</p>
<p>import torch<br>
from torch import nn<br>
from torch.utils.data import DataLoader<br>
from torchvision import datasets<br>
from torchvision.transforms import ToTensor</p>
<p>training_data = datasets.MNIST(<br>
root=“data”,<br>
train=True,<br>
download=True,<br>
transform=ToTensor(),<br>
)</p>
<p>test_data = datasets.MNIST(<br>
root=“data”,<br>
train=False,<br>
download=True,<br>
transform=ToTensor(),<br>
)</p>
<p>batch_size = 64</p>
<p>train_dataloader = DataLoader(training_data, batch_size=batch_size)<br>
test_dataloader = DataLoader(test_data, batch_size=batch_size)</p>
<p>device = “cuda” if torch.cuda.is_available() else “cpu”<br>
print(f"Using {device} device")</p>
Training with CTC loss for OCR produces same output

username: dudeperf3ct
cooked: <p>I am using the <code>nn.CTCLoss(zero_infinity=True)</code> loss function on <code>CRNN</code> model. The output from training the model after a few epochs on the validation set is shown below.</p>
<p>The batch size used is 8 and max_length for each sequence is 16. All the different outputs can be better viewed in this colab notebook: <a href="https://colab.research.google.com/drive/1xLY8GykmZoWh2nE0-WTbyZCphmfJ-CZF?usp=sharing" rel="noopener nofollow ugc">Link to colab notebook</a></p>
<p>The first line contains <code>input_lengths</code> and <code>target_lengths</code>. The second line contains shape of <code>log_probs</code> and <code>targets</code>. From the third line onwards, first string is the originally predicted output, second string is the processed predicted output and third string is the ground truth.</p>
<pre><code class="lang-auto">tensor([16, 16, 16, 16, 16, 16, 16, 16], dtype=torch.int32) tensor([5, 4, 1, 5, 5, 4, 5, 1])
torch.Size([16, 8, 41]) torch.Size([8, 16])
eeeeeeeeeeeee222 e2 22.44___________
eeeeeeeeeeeee222 e2 8.70____________
eeeeeeeeeeeee222 e2 0_______________
eeeeeeeeeeeee222 e2 12.90___________
eeeeeeeeeeeee222 e2 15.80___________
eeeeeeeeeeeee222 e2 2.80____________
eeeeeeeeeeeee222 e2 11.50___________
eeeeeeeeeeeee222 e2 4_______________

</code></pre>
<p>The problem can be either the arguments passed to the CTC loss function are wrong or something else entirely.</p>

username: mtt
cooked: <p>Hello. Looks like it’s been a while since this was posted but i was wondering if you were able to find the issue. I am working on a plate reading task using a similar architecture and i have been having the same issue as this.</p>
<p>The architecture i am using is made of convolutional layers followed by bidirectional lstm layers followed by a dense layer and i am training it using the ctc loss function.</p>
<p>For every input it outputs 2 characters while the average length should be about 8 characters and the outputs are all made of the same characters.</p>
Leveraging half-precision training in PPO and Transformer-XL

username: leckofunny
cooked: <p>Hi folks!</p>
<p>It seems that training in different precision is an under-explored area in DRL:</p>
<ul>
<li>There is an <a href="https://arxiv.org/abs/2102.13565" rel="noopener nofollow ugc">ICML Paper</a> that examines SAC in low-precision,</li>
<li>and <a href="https://research.google/pubs/quantized-reinforcement-learning-quarl/" rel="noopener nofollow ugc">another one</a> that deals quantization-aware training and post-quantization experiments</li>
<li>It would be great to know if you knew of more papers =)</li>
</ul>
<p>My actual problem is centered around the massive memory consumption of my <a href="https://github.com/MarcoMeter/episodic-transformer-memory-ppo" rel="noopener nofollow ugc">PPO + Transformer-XL (TrXL)</a> implementation. Caching the hidden states, that are output by Transformer-XL, requires lots of memory in the dimensions of <code>(num_episodes, max_episode_steps, num_trxl_layers, trxl_dim)</code>. PPO truncates episodes, so I need to buffer hidden states even across epochs.</p>
<p>I’d like to mitigate this memory overhead by leveraging half-precision for these hidden states (either f16 or bf16). What’s the most convenient approach to do this? Shall I cast the tensors from f32 to f16 manually? Shall I use AMP, where I only set the hidden states tensor to a lower precision? Will AMP take effect on other calculations than those with lower precision data, e.g. will it automatically use lower precision for convolutional layers? So far AMP is a little ambiguous to me on what it automatically casts and what not.</p>
<p>Before trying AMP or manually casting hidden states, I’d love to here some thoughts on this. Thanks in advance =)</p>
Conformer has no positional encoding

username: m_mandel
cooked: <p>Hi All,<br>
I appreciate your help.</p>
<p>Looking at the source code of torchaudio.models.conformer: <a href="https://pytorch.org/audio/2.4.0/_modules/torchaudio/models/conformer.html#Conformer" class="inline-onebox" rel="noopener nofollow ugc">torchaudio.models.conformer — Torchaudio 2.4.0 documentation</a></p>
<p>I didn’t see anywhere in the Multi-head attention module the positional encoding that should be there according to the paper. Couldn’t find it in the source code of torch.nn.modules.activation.MultiheadAttention: <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/activation.html#MultiheadAttention" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.modules.activation — PyTorch 2.4 documentation</a></p>
<p>Should I assume that I should create the relative positional encoding myself?<br>
Seems kind of weird, since the positional encoding should be applied after the layerNorm according to the paper (<a href="https://arxiv.org/pdf/2005.08100" rel="noopener nofollow ugc">https://arxiv.org/pdf/2005.08100</a>)</p>
<p>Thanks.</p>
Linux stops training while windows doesen't

username: forty
cooked: <p>Hi!</p>
<p>I am new to Linux, trying too learn it while practicing some ML.<br>
I am training a model, the size of which you can select before training.<br>
When working in Linux (Mint), if I set the depth of the model higher then 27, the training stops with no alarms/messages. This does not happen on windows.<br>
Also, I create a separate thread for the training so it does not freeze up the UI.</p>
<p>I have latest python and pytorch I believe on both operating systems.</p>
<p>How do I go about debugging this?</p>

username: ptrblck
cooked: <p>What kind of UI are you using?</p>

username: forty
cooked: <p>I built it with tkinter, ttkbootstrep for better looks.</p>

username: forty
cooked: <p>removing the ui fixed the issue, so now I know where to keep poking. Thanks for the pointer!</p>

username: ptrblck
cooked: <p>Good to hear removing the UI helped and thanks for the update.</p>
I encountered a version issue while running PyTorch. The code runs smoothly with PyTorch 1.6, but it doesn't work with PyTorch 1.13.

username: Silence_QI
cooked: <p>The code that is causing the error is as follows，Does anyone know how to resolve this issue without downgrading PyTorch? My GPU is a 4060, and I’ve tried lower versions of PyTorch, but they don’t work correctly.</p>
<p>Traceback (most recent call last):<br>
File “D:\桌面\SA-MASAC-master\train_masac.py”, line 158, in <br>
main(args)<br>
File “D:\桌面\SA-MASAC-master\train_masac.py”, line 152, in main<br>
ddpg_continuous(config)<br>
File “D:\桌面\SA-MASAC-master\train_masac.py”, line 115, in ddpg_continuous<br>
config.mini_batch_size, config.alpha)<br>
File “D:\桌面\SA-MASAC-master\robust_masac.py”, line 576, in <strong>init</strong><br>
self.actor = RobustActorNet(obs_dim, hid_shape, act_dim).to(config.DEVICE)<br>
File “D:\桌面\SA-MASAC-master\robust_masac.py”, line 254, in <strong>init</strong><br>
self.a_net = BoundedModule(self.a_net, (torch.empty(size=(1, state_dim)),), device=Config.DEVICE)<br>
File “D:\桌面\SA-MASAC-master\auto_LiRPA\bound_general.py”, line 35, in <strong>init</strong><br>
self._convert(model, global_input)<br>
File “D:\桌面\SA-MASAC-master\auto_LiRPA\bound_general.py”, line 459, in _convert<br>
nodesOP, nodesIn, nodesOut, template = self._convert_nodes(model, global_input)<br>
File “D:\桌面\SA-MASAC-master\auto_LiRPA\bound_general.py”, line 296, in _convert_nodes<br>
nodesOP, nodesIn, nodesOut, template = parse_module(model, global_input_cpu)<br>
File “D:\桌面\SA-MASAC-master\auto_LiRPA\parse_graph.py”, line 170, in parse_module<br>
trace_graph = torch.onnx.<em>optimize_trace(trace, torch.onnx.OperatorExportTypes.ONNX)<br>
File "D:\anaconda3\envs\PT\lib\site-packages\torch\onnx_<em>init</em></em>.py", line 394, in _optimize_trace<br>
return utils._optimize_graph(graph, operator_export_type)<br>
File “D:\anaconda3\envs\PT\lib\site-packages\torch\onnx\utils.py”, line 278, in _optimize_graph<br>
graph, params_dict, symbolic_helper.is_caffe2_aten_fallback()<br>
TypeError: _jit_pass_onnx_unpack_quantized_weights(): incompatible function arguments. The following argument types are supported:<br>
1. (arg0: torch::jit::Graph, arg1: Dict[str, IValue], arg2: bool) → Dict[str, IValue]</p>

username: ptrblck
cooked: <p><code>TorchScript</code> is in maintenance mode, so remove scripting or tracing from your model and try to use <code>torch.compile</code> instead in the recent PyTorch release.</p>

username: cxy22
cooked: <p>hello</p>
<p>I also encountered the same problem, have you solved it?</p>
<p>TypeError: _jit_pass_onnx_unpack_quantized_weights(): incompatible function arguments. The following argument types are supported:<br>
1. (arg0: torch::jit::Graph, arg1: Dict[str, IValue], arg2: bool) → Dict[str, IValue]</p>

username: yichao_peng
cooked: <p><a class="mention" href="/u/cxy22">@cxy22</a>  have you solved it? also same problems here.</p>
Classification head in model

username: jawi289p
cooked: <p>I am implementing a paper of semantic segmentation task. In it they have mentioned about adding the classification head. What exactly is it and how to add in the model? Segmentation models pytorch library provides classification head but I am unable to figure it out.</p>
<p>“<em>Apart from UNet CNN, we add a classification head to each architecture for up-sampling the complex low-level features, and map the high-level features to output for semantic segmentation</em>”</p>

username: jawi289p
cooked: <p><span class="mention">@anyone</span> can guide about that?</p>