akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Using ``data_ptr<float>`` on network evaluation and passing it to cuda kernel is not training network

username: Ramansh_Sharma
cooked: <p>Hello,</p>
<p>I am passing in an array of network evaluation data pointers to a an operation efficiently in cuda. The general flow of my code is as such:</p>
<pre><code class="lang-auto">torch::Tensor* evals_arr = new torch::Tensor[4];
float **evals_host;

for (int i =0; i &lt; 4; i++) {
    evals_arr[i] = networks[i]-&gt;forward(inputs[i]); // networks and inputs are std vectors of tensors
    // the evaluation evals_arr[i] is already on the GPU, so the pointer points to a location in the GPU
    evals_host[i] = evals_arr[i].data_ptr&lt;float&gt;();
}

// devals has already been cudaMalloced appropriately
cudaMemcpy(devals, evals, 4*sizeof(float*), cudaMemcpyHostToDevice);
</code></pre>
<p>I pass <code>devals</code> to my cuda kernel and use change its values in some ways. But doing this over multiple iterations of optimization does not change the evaluations of the neural networks at all. I tested the same code in C++ loops without cuda and the network evaluations change. Do I have to use accessors in order to retain the graph of the neural network?</p>
<p>How do I retain PyTorch’s computation graph when passing in C++ PyTorch tensors to cuda kernels?</p>
<p>Thank you.</p>

username: Ramansh_Sharma
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>. Do you know if there is a way to retain the computation graph of tensors (such as neural network evaluations) whose data pointers are passed to cuda kernels?</p>
<p>Thank you.</p>
DataLoader on CPU

username: kbwiley
cooked: <p>I’ve been working through this an Udemy course and the following notebook illustrates a <strong>HUGE</strong> performance cost I am experiencing on an M1 Studio Max:<br>
<a href="https://www.learnpytorch.io/04_pytorch_custom_datasets/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://www.learnpytorch.io/04_pytorch_custom_datasets/</a></p>
<p>I’m convinced the problem isn’t training/testing/inference or anything pertaining to processing the neural net. The problem is the DataLoader. Simply iterating over the DataLoader is insufferably slower on the Mac than on Google colab. This has nothing to do with GPU, either the T4 on Colab or the M1 GPU. On Colab I have not even enabled the GPU runtime and on the both computers I am not – for this demonstration of the problem – moving the model to the GPU device.</p>
<p>After build the ImageLoad, the model, the train/test functions, etc. There is then a fairly normal looking PyTorch loop over the epochs. For the purpose of the demo, I have actually disabled all NN functions by putting a ‘continue’ at the top of the batch loops in the train/test functions, like this:</p>
<pre><code class="lang-auto">    for batch, (X, y) in enumerate(dataloader):
        continue  # &lt;=== Just test the dataloader, not any NN functionality
        X, y = X.to(device), y.to(device)
        y_pred = model(X)
        loss = loss_fn(y_pred, y)
        train_loss += loss.item()
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>I then ran the cell that actually runs the epoch loop. As per the course instructor’s intent, it runs quickly on Colab. The total 5-epoch runtime is 3.38s.</p>
<p>But the exact same notebook, downloaded and unmodified, when run on my 10-core M1 Max Mac Studio, takes 302s to run.</p>
<p>The entire dataset this educational notebook utilizes is 16.2MB. Before anyone responds asking if anything would obviously make the data big (image size, whatever), please note my comparison above. It runs in 3 or 4 seconds, as intended, on Colab. So the code itself, the task designed around it by the instructor include the size of the dataset, is all already designed to be small and fast.</p>
<p>This is direct difference between these two environments.</p>
<p>I saw very similar results when I ran on an iMac Pro, although I don’t have the comparison in front of me at the moment.</p>
<p>Does anyone have any idea what could possibly be going on here?</p>
<p>Thank you.</p>
Getting different predictions each time I load model

username: jdilger
cooked: <p>Hi all,</p>
<p>I’m using resnet50 with some added layers. When I go to run my predictions I get significantly different values for the same input image. When I train I’m using <code>nn.DataParallel</code> and when I’m trying to predict Im using the cpu. In sudo code, my evaluation is as such:</p>
<pre><code class="lang-auto">model = resnet_50_spin_pyramid_error()
model = model.to(torch.device("cpu"))
weights = "model/path.pth"
model.load_state_dict(
    torch.load(weights, map_location=torch.device("cpu"))["state_dict"],
    strict=False,
)
with torch.no_grad()
   model.eval()
   img = load(img_path)
   out = model(img)
</code></pre>
<p>Any ideas what might be going on? I am using <code>nn.BatchNormalization</code> but I think that should be turned off by being in eval mode.</p>
<p>Update: Using <code>torch.manual_seed(your_seed_value)</code> I’m able to get a consistent result, but I didn’t have a seed set during training. Is there any way to figure out the random state it was trained in?</p>
<p>Update: I wasn’t loading my weights correctly <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> fixing that it looks like it works.</p>
<p>Part of the model building function:</p>
<pre><code class="lang-auto">class SpinPyramidError(nn.Module):
    def __init__(self, *args, **kwargs) -&gt; None:
        super().__init__(*args, **kwargs)
        self.dgcn_seg1 = spin(planes=32, ratio=2)
        self.dgcn_seg2 = spin(planes=32, ratio=2)
        self.conv2 = nn.Conv2d(32, 32, 3)
        self.fin_conv2 = nn.Conv2d(32, 2, 3)

    def forward(self, x):
        spin_256 = self.dgcn_seg1(x)
        _, c, h, w = spin_256.shape
        x2_128 = nn.MaxPool2d(kernel_size=2, stride=1)(x)
        spin_128 = self.dgcn_seg2(x2_128)
        spin_128_up = nn.functional.interpolate(spin_128, size=(h, w))
        x2 = torch.add(spin_128_up, spin_256)

        x3 = self.conv2(x)
        x4 = nn.ReLU(inplace=True)(x3)
        x5 = self.fin_conv2(x4)
        return x5

def resnet_50_spin_pyramid_error():
    model = deeplabv3_resnet50() # Load net
    model.classifier[4] = torch.nn.Conv2d(
        256, 32, kernel_size=(1, 1), stride=(1, 1)
    )  # Change final layer to 2 classes
    sp = SpinPyramidError()
    model.classifier.add_module("spin_pyramid", sp)
    return model
</code></pre>
Example of PRIVATEUSE1 for custome device

username: hanhaowen-mt
cooked: <p>With the arrival of version 2.1 of Pytorch, Pytorch has defined new interfaces for extending custom hardware, which makes it more convenient to use custom hardware to leverage Pytorch,<br>
<a href="https://github.com/Ascend/pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Ascend/pytorch: Ascend PyTorch adapter (torch_npu). Mirror of https://gitee.com/ascend/pytorch</a> is a good example to use it, but it is too complicated,So does Pytorch have any plans to write a complete routine on how to adapt to custom hardware?</p>

username: jlui275
cooked: <p>If you are still looking for something, this repo was helpful in explaining basic usage for registering a custom device (<a href="https://github.com/bdhirsh/pytorch_open_registration_example" rel="noopener nofollow ugc">bdhirsh/pytorch_open_registration_example: Example of using pytorch’s open device registration API (github.com)</a>).</p>
<p>My only issue with this example is that it doesn’t go into much detail about how to expand upon it. The code is well commented and explains what is being done, however I’m curious about how the developer (<a class="mention" href="/u/bdhirsh">@bdhirsh</a>) knows what to do (ie. it seems like there may be some documentation that I’m failing to find).</p>
Profiling with hugging face accelerate

username: umgbhalla
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3b96344732bf46253bd4141f04b20008621b873.png" data-download-href="https://discuss.pytorch.org/uploads/default/f3b96344732bf46253bd4141f04b20008621b873" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3b96344732bf46253bd4141f04b20008621b873_2_690x212.png" alt="image" data-base62-sha1="yM5drglZe9oMQMQ4GSxvWwML2g3" width="690" height="212" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3b96344732bf46253bd4141f04b20008621b873_2_690x212.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3b96344732bf46253bd4141f04b20008621b873_2_1035x318.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3b96344732bf46253bd4141f04b20008621b873_2_1380x424.png 2x" data-dominant-color="262B2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">4418×1358 470 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
i have added profiling but this is not capturing traces or any information on cuda device, could it be due to huggingface accelerator abstraction ?</p>
Training network error

username: maramreddy
cooked: <p>below is the code and error i am getting can anyone help me on this.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b0ff038dc07237133d33a44a5717459112ae886.png" data-download-href="https://discuss.pytorch.org/uploads/default/9b0ff038dc07237133d33a44a5717459112ae886" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b0ff038dc07237133d33a44a5717459112ae886.png" alt="image" data-base62-sha1="m7K9EzaUYQ0hNN4wjeJf8RxTY5E" width="690" height="255" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1660×615 19.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
##<span class="hashtag-raw">#error</span><br>
RuntimeError                              Traceback (most recent call last)<br>
Cell In[17], line 7<br>
5 if i%10 == 0:<br>
6     print(f’epoch:{i}::loss:{loss.item()}')<br>
----&gt; 7 loss.backward()<br>
8 with torch.no_grad():<br>
9     w1 -= (learningrate*w1.grad)</p>
<p>File /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:487, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)<br>
477 if has_torch_function_unary(self):<br>
478     return handle_torch_function(<br>
479         Tensor.backward,<br>
480         (self,),<br>
(…)<br>
485         inputs=inputs,<br>
486     )<br>
 → 487 torch.autograd.backward(<br>
488     self, gradient, retain_graph, create_graph, inputs=inputs<br>
489 )</p>
<p>File /opt/conda/lib/python3.10/site-packages/torch/autograd/<strong>init</strong>.py:200, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)<br>
195     retain_graph = create_graph<br>
197 # The reason we repeat same the comment below is that<br>
198 # some Python versions print out the first line of a multi-line function<br>
199 # calls in the traceback and some print out the last line<br>
 → 200 Variable.<em>execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
201     tensors, grad_tensors</em>, retain_graph, create_graph, inputs,<br>
202     allow_unreachable=True, accumulate_grad=True)</p>
<p>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
The different CUDA memory and speed between torch-1.7.1+cuda 11.0 and torch-1.8.0 + cuda 11.1

username: Hao_Du
cooked: <p>Hi, Pytorch’s team, now I use my own Pytorch code to train my model with 8 A100 (80G), which contains 14 layers of dilated convolution and 2 layers of graph transformer, but I found that the cuda memory and speed are different when using different Pytorch versions. For the torch-1.7.1 + cuda 11.0, each device can have 6 batch sizes with 55 GB, but it needs 5 hours to finish an epoch, the torch-1.8.0 + cuda 11.1 can only have 4 batch sizes per device with 69 GB, but it only needs 40 mins to finish an epoch. So, Why does this happen? It feels very strange.</p>

username: ptrblck
cooked: <p>Both PyTorch releases are old so use the latest stable or nightly release and compare your workload against it.</p>

username: Hao_Du
cooked: <p>but, when I use the torch-1.10.1 + cuda 11.1 and compare it with the torch-1.7.1 + cuda 11.0, the phenomenon still appears, so do you have any suggestions?</p>

username: ptrblck
cooked: <p>Yes, profile your workload with the current stable release (<code>2.1.2</code>) or a nightly build.</p>

username: Hao_Du
cooked: <p>ok, thanks, I will have a try</p>
PyTorch models pretraining source

username: adghin
cooked: <p>Hi!</p>
<p>Regarding to pytorch models such as ResNet and ViT, are they pre-trained only on ImageNet-1K (so the default weights only comes from that dataset) or they are pre-trained on ImageNet-21K and then finetuned on ImageNet-1K? I’m asking because timm’s library has, for example, the checkpoint of ViT/B-16 with weights that comes from pre-training on the 21K version and then finetuned on the 1K one, so I wanted to know if that is the case of the models that come from pytorch hub.</p>
<p>Regards,<br>
Adrian</p>
How to gather data from all GPUs for DataParallel?

username: avijit_dasgupta
cooked: <p>Hi,</p>
<p>In my forward pass, I am trying to gather the features from a CNN model to compute the class prototypes from the current batch. I can do that for DistributedDataParallel easily using the example given by ‘MOCO’. But I am not sure how to do it for DataParallel. It would be great if someone can give me some pointers for that.</p>
<p>TIA.</p>

username: lucy
cooked: <p>When working with PyTorch’s DataParallel, extracting features from a CNN model can be a bit different compared to using DistributedDataParallel. In DataParallel, the model is replicated across multiple devices (GPUs), and each replica processes a different batch of data and <a href="https://www.globaldatabase.com/data-enrichment" rel="noopener nofollow ugc">b2b contact enrichment</a>. To gather features from these replicas, you need to collect the outputs from each device and concatenate them. Here’s a general outline of how you can modify your forward pass for feature extraction when using DataParallel:<br>
import torch<br>
import torch.nn as nn<br>
from torch.nn.parallel import DataParallel</p>
<h1><a name="your-cnn-model-1" class="anchor" href="#your-cnn-model-1"></a>Your CNN model</h1>
<p>class YourModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(YourModel, self).<strong>init</strong>()<br>
# Define your model layers here</p>
<pre><code>def forward(self, x):
    # Your forward pass logic
    return x  # Adjust this line based on your actual forward pass
</code></pre>
<h1><a name="wrap-your-model-with-dataparallel-2" class="anchor" href="#wrap-your-model-with-dataparallel-2"></a>Wrap your model with DataParallel</h1>
<p>model = YourModel()<br>
model = DataParallel(model)</p>
<h1><a name="assuming-input-tensor-input_data-for-the-forward-pass-3" class="anchor" href="#assuming-input-tensor-input_data-for-the-forward-pass-3"></a>Assuming input tensor ‘input_data’ for the forward pass</h1>
<p>input_data = torch.randn((batch_size, channels, height, width))</p>
<h1><a name="forward-pass-with-dataparallel-4" class="anchor" href="#forward-pass-with-dataparallel-4"></a>Forward pass with DataParallel</h1>
<p>output_data = model(input_data)</p>
<h1><a name="gather-features-from-different-replicas-5" class="anchor" href="#gather-features-from-different-replicas-5"></a>Gather features from different replicas</h1>
<p>gathered_features = output_data.module.feature_extraction_function()</p>
<h1><a name="gathered_features-now-contains-the-features-from-all-replicas-6" class="anchor" href="#gathered_features-now-contains-the-features-from-all-replicas-6"></a>‘gathered_features’ now contains the features from all replicas</h1>
Why are weights (or kernel) floats and not integers in torch.nn.Conv2d?

username: johnwck
cooked: <p>Hi,</p>
<p>I am learning about CNN using PyTorch.</p>
<p>In many online notes, the kernel is always a matrix (or tensor) of integers.</p>
<p>However, when I get the weights (or kernel) from codes below, they are floats.</p>
<pre><code class="lang-auto">a = nn.Conv2d(1, 1, kernel_size=(3,3), padding=1)
a.weight
</code></pre>
<p>I would like to have guidance on why are the weights floats and not integers.<br>
Or have I got my fundamentals wrong.</p>
<p>Many thanks for any help given.</p>
<p>I have tried Googling as follows on this topic but I could not get any results on it.</p>
<ul>
<li>“why weights in PyTorch Conv2d floats and not integers?”</li>
<li>“why kernel in PyTorch Conv2d floats and not integers?”</li>
</ul>

username: astle_dsa
cooked: <p>The weight matrix, when initialised, is made up of floats only. Integers are limiting, as floats can represent a much wider range of values with a greater degree of accuracy. (Answer from google)<br>
The values, even if they were integers, won’t stay integers for long anyways, as while calculating gradients and stuff, they will eventually turn into floats anyways, during the back propagation process.</p>

username: ptrblck
cooked: <p>Integer values are not differentiable in PyTorch as they are discrete. Of course floating point values are also limited by their <code>dtype</code> but we treat them as continuous values.<br>
Examples use Integer values as these are easier to understand in manual computations.</p>

username: johnwck
cooked: <p>Oh I see. Thank you for the explanation.</p>
<p>For a layman like me, it was very confusing as most of the online notes have the weights in integers, not floats.</p>
<p>On this note, I realised that I have made an error in the title of my post. It should be kernel SAME AS filter. Weights are the values inside the kernel or filter.</p>
<p>Thank you once again.</p>

username: johnwck
cooked: <p>Yes. Your explanation is clear. It makes sense now. Using integers to illustrate the calculations will make it easier for a layman like me to visualise and understand.</p>
<p>Thank you very much for your help.</p>
Should I initialize some submodules beforehand and pass them in as parameters to the final model's __init__ function?

username: Nguyen_Duy
cooked: <p>Hello. I have a question as follows:</p>
<p>With a large model, if I initialize all the child modules in the <strong>init</strong> function of a final model I may end up having to pass in many parameters. Can I build a build_model() function like the example below?</p>
<pre><code class="lang-auto">import torch


class A(torch.nn.Module):
    def __init__(
        self,
        idim: int,
        odim: int,
        sub_1: torch.nn.Module,
        sub_2: torch.nn.Module,
    ):
        super().__init__()
        self.idim = idim
        self.odim = odim
        self.sub_1 = sub_1
        self.sub_2 = sub_2
        self.linear_out = torch.nn.Linear(self.idim, self.odim)
    

def build_model(
    idim: int,
    odim: int,
    sub1_param1: int,
    sub1_param2: int,
    sub2_param1: int,
    sub2_param2: int,
) -&gt; A:
    sub_1 = torch.nn.Linear(sub1_param1, sub1_param2)
    sub_2 = torch.nn.Linear(sub2_param1, sub2_param2)
    
    return A(idim, odim, sub_1, sub_2)
</code></pre>
<p>Here is a simple descriptive example. For me there could be more submodules and each submodule has more parameters. Are there any disadvantages to this implementation? Thank you.</p>

username: ptrblck
cooked: <p>Your approach is fine and should not cause any issues. I would claim it comes down to your programming style and the use case.</p>
Why is each epoch taking longer than the last

username: Onon0
cooked: <p>Each epoch takes 5 minute longer than the last. How can I improve my code.</p>
<pre><code class="lang-auto">def train(loader, model, optimizer, loss_fn, scaler):
    loop = tqdm(loader)
    history = []
    for batch_idx, (data, targets) in enumerate(loop):
        data = data.to(device = DEVICE)
        targets = targets.to(device = DEVICE)
        
        #forward
        with torch.cuda.amp.autocast():
            predictions = model(data)
            loss = loss_fn(predictions, targets)
            history.append(loss)
        optimizer.zero_grad()
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()
        loop.set_postfix(loss = loss)
    return history
</code></pre>

username: ptrblck
cooked: <p>You are appending all computation graphs in the <code>history</code> list which will increase the memory usage and eventually run out of memory. Detach the loss tensor before appending it.</p>
Why do we flatten the output in Language Model training?

username: Amir_Rahnama
cooked: <p>I am following this tutorial to create a language model with transformers: <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html#load-and-batch-data" class="inline-onebox" rel="noopener nofollow ugc">Language Modeling with nn.Transformer and torchtext — PyTorch Tutorials 2.2.0+cu121 documentation</a></p>
<p>I have some questions about the way we process the data. So correct me if I am wrong, we do the followings first:</p>
<ul>
<li>read the text and tokenized them in a flat tensor</li>
<li>Then we turn them into batches</li>
<li>and transform them so that they have equal sequence length</li>
</ul>
<p>So far, so good.</p>
<p>Let’s assume our vocab has 4817 elements. In each batch, data has a size: torch.Size([30, 20]) with a sequence length of 30 and batch size of 20. But then we flatten the targetin batch into the Tensor with size torch.Size([600]).</p>
<p>Then we run the cross entropy loss:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
loss = criterion(output_flat, targets)
</code></pre>
<p>After reshaping the tensor, the model output, output_flat, is of size torch.Size([600, 4817]).</p>
<p>My confusion is why we are training the model like this. Why are we flattening all the targets in the batch here and calculating the loss on the entire batch as a flattened-out object? Should we not calculate the loss one by one for each instance in the batch?</p>
TransformerEncoderLayer produces all nan tensor when src_key_padding_mask is an all-true tensor

username: ddenis
cooked: <p>The issue is as simple as explained in the title: I do not understand why the TransformerEncoderLayer module produces a tensor of nan values when provided with a full <code>true</code> mask.</p>
<p><strong>Minimal working examples</strong></p>
<p><em>WITH MASK:</em></p>
<pre><code class="lang-auto">import torch.nn as nn
import torch

model = nn.TransformerEncoderLayer(d_model = 1, nhead = 1)
tensor = torch.tensor([i for i in range(5)], dtype = torch.float32).unsqueeze(1)
mask = torch.tensor([True for _ in range(5)], dtype = torch.bool)
model(tensor, src_key_padding_mask = mask)

&gt;&gt;&gt; tensor([[nan],
            [nan],
            [nan],
            [nan],
            [nan]], grad_fn=&lt;NativeLayerNormBackward0&gt;)
</code></pre>
<p><em>WITHOUT MASK</em>:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch

model = nn.TransformerEncoderLayer(d_model = 1, nhead = 1)
tensor = torch.tensor([i for i in range(5)], dtype = torch.float32).unsqueeze(1)
model(tensor)

&gt;&gt;&gt; tensor([[0.],
            [0.],
            [0.],
            [0.],
            [0.]], grad_fn=&lt;NativeLayerNormBackward0&gt;)
</code></pre>
Pytorch turtlebot problem

username: bengao
cooked: <p>Hello,</p>
<p>I’m trying unsuccessfully to install pytorch on a turtlebot robot.</p>
<p>Is there anyone who can help me?</p>
<p>Thanks a lot!</p>
<p>Be</p>

username: marksaroufim
cooked: <p>What did you try and what error message did you get? You need share a bit more so we can help</p>
nn.Module ignore Module type instance variables if it is also defined as a class variable

username: GuillaumeTong
cooked: <p>Disclaimer: I’m not sure this counts as a bug, as I’m triggering this by doing something slightly unconventional… But this does feel like a non-intended behavior of nn.Module.</p>
<p>When defining the same names as both class and instance variables on an nn.Module, and storing Modules inside, instances will use the class variable instead of the instance one, as it would normally.</p>
<p>The above description might be very confusing, here is a code example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import nn

class A(nn.Module):
    number = 0.0
    module = None

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.number = 1.0
        self.module = nn.Identity()

    def forward(self, x):
        return self.module(x) * self.number

print(f"{A.number=}, {A.module=}")
a = A()
print(f"{a.number=}, {a.module=}")
t = torch.rand(1, 1)
assert a(t) == t
</code></pre>
<p>Running this results in an error:</p>
<pre><code class="lang-auto">A.number=0.0, A.module=None
a.number=1.0, a.module=None
Traceback (most recent call last):
  [...], in forward
    return self.module(x) * self.number
TypeError: 'NoneType' object is not callable
</code></pre>
<p>I would normally expect <code>a.module</code> to be <code>nn.Identity()</code>, not <code>None</code>… This seems pretty counter-intuitive, especially since <code>a.number==1.0</code> as normal.</p>
<p>In my use case, I needed to do this so that the class variable could act as a default value. Models picked before I had written code with the instance variable could be loaded with that working default value.<br>
For my case, I think I can re-factor the code to define a <code>__setstate__()</code> instead to get that effect.</p>
<p>I’ve tested this on <code>torch==2.1.2</code></p>
Can someone teach me how to use Python to write a multi-threaded video inference

username: yinjun1131
cooked: <p>I am currently trying to perform multi-threaded inference on some edge devices, and I have encountered several issues. Can someone please guide me on how to make improvements?<br>
My Question：<br>
1、When using multi-threaded reasoning, it is possible for multiple threads to return reasoning results in a very short amount of time, and it is also possible for multiple threads to return results without a single thread for a long period of time. How can I make my reasoning timeline smoother<br>
2、When I use multi-threaded inference, later threads may actually have faster inference speed than previous threads. If I don’t use these frames, my video fps may decrease. If I use them, my video stream order may become chaotic<br>
3、When I am multithreaded reasoning, the CPU consumption speed sharply increases, but it seems that my average reasoning speed has not improved at the same level. Perhaps I have 3 threads reasoning a video with an FPS of 22, while the reasoning speed of 18 threads can only reach 25. If I observe the thread pool, I can also see that all 18 threads are enabled. How should I handle this problem<br>
4、When I am conducting multi-threaded inference, should I create multiple models and use different models for each thread, or should I use a single model to the end</p>
Transfer of weights between two different models

username: Salman_Abbasi
cooked: <p>Hi</p>
<p>I have a question. I have two different models. I would like to transfer learned weights of model1 to a different model (differnt layers and structure- autoencoder), such that output of this 2nd network first iteration(epoch), is equal to the output of the learned model(model1).</p>
<p>I do not have a code, because I simply do not know how to even begin with. Is it possible to implement this? If yes, I appreciate response.</p>
<p>Thanks<br>
Sal</p>

username: ptrblck
cooked: <p>Assuming the used layers are equal in their parameter setup (i.e. they are using parameters and buffers in the same shape) you could either copy the data manually e.g. via:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.no_grad():
    model2.layer_name2.weight.copy_(model1.layer_name1.weight)
</code></pre>
<p>or could manipulate the <code>key</code>s of the <code>state_dict</code> and load it.</p>

username: Salman_Abbasi
cooked: <p>Thanks. Both models are entirely different. Below is model1, the outptut of which I like to have on the first iteration of my 2nd model.<br>
class Model(torch.nn.Module):<br>
def <strong>init</strong>(self, initial, min_vel, max_vel):<br>
super().<strong>init</strong>()<br>
self.min_vel = min_vel<br>
self.max_vel = max_vel<br>
self.model = torch.nn.Parameter(<br>
torch.logit((initial - min_vel) /<br>
(max_vel - min_vel))<br>
)</p>
<pre><code>def forward(self):
    return (torch.sigmoid(self.model) *
            (self.max_vel - self.min_vel) +
            self.min_vel)
</code></pre>
<p>model = Model(v_init, 1000, 2500).to(device)</p>
<p>and model 2 is a deep convolutional Autoencoder</p>

username: ptrblck
cooked: <p>If your models are “entirely different” I assume their structure and layers do not match. In this case you might not be able to transfer weights between these unless you can create a mapping between layers. E.g. a 1x1 conv layer could be mapped to a linear layer, but of course not every layer is transferrable to any other module.</p>
Captum + vit_b_16 vizualization

username: UnitTest
cooked: <p>Has anyone successfully used LayerGradCam/LRP on vit_b_16 model? I am using</p>
<blockquote>
<p>create_feature_extractor(base_model, return_nodes=[‘encoder’])<br>
to extract the encoder features. Interested to visualize the GradCam output.</p>
</blockquote>
<p>Also, visualizing the gradients of the model (vit_b_16), we observe jpeg artifacts (on both JPEG and non-JPEG). However, previous research infers that compression should not effect model training. Thoughts?</p>
FSDP with Single Node & Multi-GPU

username: ppower1
cooked: <p>What’s missing from the following example code? When I follow this, I get a runtime error: RuntimeError: “Default process group has not been initialized, please make sure to call init_process_group.”</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP<br>
torch.cuda.set_device(device_id)<br>
sharded_module = FSDP(my_module)<br>
optim = torch.optim.Adam(sharded_module.parameters(), lr=0.0001)<br>
x = sharded_module(x, y=3, z=torch.Tensor([1]))<br>
loss = x.sum()<br>
loss.backward()<br>
optim.step()</p>
</blockquote>
</blockquote>
</blockquote>

username: ptrblck
cooked: <p>Based on the <a href="https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html">FSDP tutorial</a> you are missing the distributed setup call.</p>

username: ppower1
cooked: <p>Thanks for the pointer!</p>
Detaching tensors from graph on the fly

username: richieYT-wan
cooked: <p>Hey, I am not sure which is the best practice.</p>
<p>Say I have a standard training loop, and want to save my predictions to a list (that I will concatenate at the end to get some prediction metrics). Should my output go through <code>.detach().cpu()</code> at each batch, or is it OK to do it at the very end when I concatenate them ?</p>
<p>e.g. :</p>
<p>Version A:</p>
<pre><code class="lang-auto">x_list = []
for batch in batches:
   batch = batch.to(device)
   x_out = model(batch)
   loss = criterion(x_out, batch)
   optimizer.zero_grad()
   loss.backward()
   optimizer.step()
   x_list.append(x_out.detach().cpu())
x_list = torch.cat(x_list)
</code></pre>
<p>or<br>
Version B</p>
<pre><code class="lang-auto">x_list=[]
for batch in batches:
   batch = batch.to(device)
   x_out = model(batch)
   loss = criterion(x_out, batch)
   optimizer.zero_grad()
   loss.backward()
   optimizer.step()
   x_list.append(x_out)
x_list = torch.cat(x_list).detach().cpu()
</code></pre>
<p>I assume version A is more memory efficient on a GPU but might slow down my code due to the non stop copy operation ?</p>

username: ptrblck
cooked: <p>You should <code>.detach()</code> the tensor before appending it to a <code>list</code> as otherwise the output could still keep the computation graph alive thus increasing the memory usage. You don’t necessarily need to move the data back to the CPU as I assume <code>x_out</code> is not huge.</p>
Pytorch fails to load cuda

username: msv
cooked: <p>On Ubuntu: 20.04.2 with<br>
“nvcc --version” returning … release 10.1 V10.1.243<br>
“nvidia-smi” returning: NVIDIA-SMI: 470.82.00 Driver Version: 470.82.00 CUDA Version: 11.4…Tesla P100</p>
<p>Using anaconda3, I’m trying to install and environment with with python3.11 and pytorch with gpu support, but torch.cuda.is_available() returns false, and if I try to run:</p>
<p>$ conda install pytorch torchvision torchaudio cudatoolkit=10.1.243 -c pytorch<br>
this conda still proposes to install:<br>
pytorch/linux-64::pytorch-2.1.2-py3.11_cpu_0</p>
<p>Reading up on version compatibilities, I haven’t found any confllicts…</p>
<p>Please help</p>

username: ptrblck
cooked: <p>Use the supported install commands from <a href="https://pytorch.org/get-started/locally/">here</a> as your current install command specifies <code>cudatoolkit=10.1</code> which we are not building against anymore.</p>
How to Make AMD GPU deterministic

username: Jiachen
cooked: <p>Hi, I am running my code with AMD GPU (MI50) and I want my run to be fully deterministic. However, I found no matter how I set the random seed and cudnn part, I cannot make the run fully deterministic. Can you help help with this? (BTW, I use DDP to train)</p>
<p>My current setting is:<br>
seed = args.seed<br>
random.seed(seed)<br>
np.random.seed(seed)<br>
torch.manual_seed(seed)<br>
torch.backends.cudnn.benchmark = False<br>
torch.backends.cudnn.deterministic = True<br>
torch.use_deterministic_algorithms(True)</p>
Using custom attention masks

username: davidireland3
cooked: <p>I would like to use a custom attention mask for each element of my batch. Let’s suppose I have a batch size of 2, with a single query and 3 keys. Then, suppose my custom mask looks like:</p>
<pre><code class="lang-auto">mask = [
        [[-inf, 0, 0]],
        [[0, -inf, 0]]
       ]
</code></pre>
<p>Now, suppose that my MHA layer is using N heads, the documentation says that the custom mask should be of size (2 * N, 1, 3). How should I set up my mask? would I repeat like <code>mask = mask.repeat(N, 1, 1)</code>, or would I need to make it like</p>
<pre><code class="lang-auto">mask = [
        [[-inf, 0, 0]],
        ...
        ...
        ...
        [[-inf, 0, 0]],
        [[0, -inf, 0]],
        ...
        ...
        ...
        [[0, -inf, 0]]
       ]
</code></pre>
Will this inplace assignment propagate gradient correctly?

username: Tristan_Brugere
cooked: <p>Hello,</p>
<p>In my code I need to do something like this</p>
<pre data-code-wrap="python"><code class="lang-python">. . . 
# a (n_a, n_features)
# b (n_b, n_features)
# some_indices (n_a,) long

result = b.clone()
result[some_indices, :] = a
return result
</code></pre>
<p>Will this propagate gradients correctly for <code>a</code> and <code>b</code> ?<br>
(ie the gradient of <code>result</code> for <code>some_indices</code> propagated to <code>a</code>, and for all the other indices, they are propagated to <code>b</code>)<br>
I am a bit scared of the inplace operation, I couldn’t find explanations of where they’re safe or not for the autograd engine.</p>
<p>Usually I would use a <code>torch.where</code>, but <code>a</code> doesn’t have the same shape as b, so that would require me to get <code>a</code> to that same size, which would also require an assignment…</p>
<p>Thank you for the help</p>

username: ptrblck
cooked: <p>I would expect to see a <code>RuntimeError</code> if the inplace operation is disallowed and would result in wrong gradient calculations. If you are not seeing such an error in your code it should be fine.</p>

username: Tristan_Brugere
cooked: <p>Oh so if there is no error, I can expect the gradients to be correct? That is great!<br>
Thanks!</p>
I3D resnet action recognition model - expected input shape

username: LewsTherin
cooked: <p>Hello! I want to fine-tune the I3D model for action recognition from torch hub, which is pre-trained on Kinetics 400 classes, on a custom dataset, where I have 4 possible output classes.</p>
<p>I’m loading the model and modifying the last layer by:</p>
<pre><code class="lang-auto">model = torch.hub.load("facebookresearch/pytorchvideo", "i3d_r50", pretrained=True)
num_classes = 4
model.blocks[6].proj = torch.nn.Linear(2048, num_classes)
</code></pre>
<p>I couldn’t find any indication as to which is the expected output for the model.<br>
Currently, I’m passing my data in the shape:</p>
<pre><code class="lang-auto">[batch_size, num_channels, num_images, width, height]
</code></pre>
<p>Is that correct?</p>
How to share big data in Dataset across processes?

username: dancedpipi
cooked: <p>I have a Dataset:</p>
<pre data-code-wrap="python"><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, data):
        self.data = data

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data[idx]
</code></pre>
<p>The <code>data</code> is a big numpy tensor(almost 27GB), so larger dataloader’s num_workers will cause out of memory.<br>
It is worth noting that the <code>data.dtype</code> is <code>object</code>, so numpy <code>memmap</code> mode cannot be used directly.</p>
<p>In order to share across processes, I modified it to the following Dataset:</p>
<pre data-code-wrap="python"><code class="lang-python">from multiprocessing import shared_memory
from filelock import FileLock

class MyDataset(Dataset):
    def __init__(self, data):
        data_size = data.nbytes
        lock = FileLock("./create_shared_mem.lock")
        with lock:
            try:
                shm = shared_memory.SharedMemory(name="data", create=True, size=data_size)
                shm_data = np.ndarray(data.shape, dtype=data.dtype, buffer=shm.buf)
                shm_data[:] = data[:]
            except FileExistsError as e:
                shm = shared_memory.SharedMemory(name="data")
                shm_data = np.ndarray(data.shape, dtype=data.dtype, buffer=shm.buf)

        self.data = shm_data

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data[idx]
</code></pre>
<p>But such code implementation will encounter some strange problems.<br>
I want to know what is the best practice for multi-process sharing tensor in torch.</p>
How to back-order specific values

username: gengxingze
cooked: <p>An int tensor his dimension is [batch,n,m], I want to move his element of last dim  whose element is not -1 forward and -1 back.Is it possible to achieve the same result without using a for loop, as a for loop would be very time consuming? Here is a 2D example</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
tensor = torch.tensor([[1, -1, 2, -1],
                       [7, -1, 5, 6],
                       [0, 7, -1, 8]])

for i in range(tensor.shape[0]):
    tensor[i] = torch.cat((tensor[i][tensor[i] != -1], tensor[i][tensor[i] == -1]))
print(tensor)
#tensor([[ 1,  2, -1, -1],
#       [ 7,  5,  6, -1],
#      [ 0,  7,  8, -1]])

</code></pre>

username: J_Johnson
cooked: <p>Anytime you want to do a batch logic argument on a tensor, you can create a boolean mask. For example:</p>
<p>“whose element is not -1” could be determined elementwise with:</p>
<pre><code class="lang-auto">mask = tensor!=-1
</code></pre>
<p>And then we can reorder those with:</p>
<pre><code class="lang-auto">x = torch.stack([torch.cat([tensor[_,:][mask[_,:]], tensor[_,:][~mask[_,:]]], dim=0) for _ in range(tensor.size(0))])
</code></pre>
<p>In this case, the for loop gets processed in parallel for each row.</p>
Loss.backward raises error with make_private()

username: Mohammadreza_Ebrahim
cooked: <p>I am following the tutorial: <a href="https://opacus.ai/tutorials/building_image_classifier" rel="noopener nofollow ugc">https://opacus.ai/tutorials/building_image_classifier</a>. The only difference is that I have a smaller network than resnet and I want to use make_private function instead of make_private_with_epsilon since I do not have the epsilon and delta before hand, but I do know the noise multiplier.</p>
<pre><code class="lang-auto">optimizer = optim.SGD(netS.parameters(), lr=opt.lr)
    
privacy_engine = PrivacyEngine()

model, optimizer, train_loader = privacy_engine.make_private(
    module=netS,
    optimizer=optimizer,
    data_loader=trainloader,
    noise_multiplier= NOISE_MULTIPLIER,
    max_grad_norm=MAX_GRAD_NORM,
)
</code></pre>
<p>When I call loss.backward() in the train function, I get the following error:<br>
RuntimeError: grad can be implicitly created only for scalar outputs</p>
<p>In non-DP training, it makes sense to get this error when we do not do loss.mean(). But I was under the impression that Opacus will handle it in the per-sample gradient setting. Is there anything I am missing? Thanks in advance.</p>
How to use the Inception model for transfer learning?

username: maflobra
cooked: <p>I have created a <code>torchvision</code> model for transfer learning, using the pre-built <em>ResNet50</em> base model, like this:</p>
<pre data-code-wrap="python"><code class="lang-python">        # Create base model from torchvision.models
        model = resnet50(pretrained=True)
        num_features = model.fc.in_features

        # Define the network head and attach it to the model
        model_head = nn.Sequential(
            nn.Linear(num_features, 512),
            nn.ReLU(),
            nn.Dropout(0.25),
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(256, num_classes),
        )
        model.fc = model_head
</code></pre>
<p>Now I wanted to use the <em>Ineception v3</em> model instead as base, so I switched from <code>resnet50()</code> above to <code>inception_v3()</code>, the rest stayed as is. However, during training I get the following error:</p>
<blockquote>
<p>TypeError: cross_entropy_loss(): argument ‘input’ (position 1) must be Tensor, not InceptionOutputs</p>
</blockquote>
<p>So how can one use the <em>Inception v3</em> model from <code>torchvision.models</code> as base model for transfer learning?</p>

username: maflobra
cooked: <p>For the records:<br>
<a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html#inception-v3" rel="noopener nofollow ugc">I found the solution here</a>.</p>

username: dp_unil
cooked: <p>Seems like the link here has been updated and no longer has information on how to train the inceptionv3 model. I am running into a similar issue where I have pipelines set up for transfer learning using resnet but they do not work with the inceptionv3 model. Can you post your solution please?</p>
What is error in this code when train yolo?

username: AishaMohamed2022
cooked: <p>model =YOLO(“/content/runs/segment/train/weights/best.pt”)<br>
result = results[0]</p>
<p>masks = result.masks<br>
len(masks),masks.shape</p>
<p>def instance_segmentation(image_path,masks):<br>
mask=masks<br>
image = Image.open(image_path)<br>
image=torchvision.transforms.Resize(int(masks.shape[1]),int(masks.shape[2]))<br>
image_tensor = torchvision.transforms.ToTensor()(image)<br>
image_tensor=image_tensor.unsqueeze(0)<br>
results = model(image_tensor)<br>
boxes = results.pandas().xyxy[0]  # Bounding boxes<br>
labels = results.xyxyn[0][:, -1].int()  # Class labels<br>
masks = results.pred[0][‘masks’]  # Instance segmentation masks</p>
<pre><code># Plot the segmented instances
fig, ax = plt.subplots(figsize=(8, 8))
ax.imshow(image)
print(image)
for i, (box, label) in enumerate(zip(boxes, labels)):
    xmin, ymin, xmax, ymax = box
    w, h = xmax - xmin, ymax - ymin
    mask = masks[i, 0].mul(255).byte().numpy()
    instance_image = Image.fromarray(mask)
    ax.imshow(instance_image, alpha=0.5, extent=(xmin, xmax, ymax, ymin))
    ax.text(xmin, ymin, str(label.item()), color='r')
</code></pre>
<p>plt.show()</p>
<p>instance_segmentation(“ZH.jpg”,masks)</p>
<p>this error was appear<br>
return inverse_modes_mapping[i]<br>
KeyError: 480</p>

username: AishaMohamed2022
cooked: <p>i want to detect object mask and find is coordinate x and y (position mask with respect to  the image )<br>
and display it. how can i do that?</p>
How to get trapezoidal matrix?

username: boywithacoin
cooked: <p>how do you get upper/lower trapezoidal part of the matrix? i see functions like triu and tril but nothing related to trapazoid.</p>

username: stijnvdl
cooked: <p><code>torch.tril()</code> and <code>torch.triu()</code> support non-square matrices.</p>
<p>If you want to have the diagonal entries set to zero, you could subtract <code>torch.diag()</code> from either <code>torch.tril()</code> or <code>torch.triu()</code></p>
Convolution element wise with pytorch

username: dthanoon
cooked: <p>I have a tensor A of size [904,145] and B of size [904,1234]. Both tensor have 904 elements. I am looking to convolve element wise each element of A with their respective element of B. So basically convolve A[i,:] with B[i,:] for all i. How can I do that in pytorch, I tried multiple combination but everytime I end up with a size of [904,904,1234] (I assume it does each element to all other elements) when I expect a final size of [904,1234]</p>

username: J_Johnson
cooked: <p>Hello and welcome!</p>
<p>I’m not sure what you mean by convolve in this case. Can you write a code example in NumPy of what you want to accomplish(can include loops)?</p>

username: dthanoon
cooked: <p>Something like this.</p>
<pre><code class="lang-auto">import numpy as np
a = np.zeros((904, 1234))
b = np.zeros((904, 145))
c = np.zeros(a.shape)
for i in range(a.shape[0]):
    c[i,:] = np.convolve(a[i,:],b[i,:], mode = 'same')
</code></pre>
<p>But I would like to do it in torch in one line since 904 dimension will play the dimension of my batch size.</p>

username: J_Johnson
cooked: <p>I think you may need <code>torchaudio</code> installed. Then you can do the following:</p>
<pre><code class="lang-auto">import torch
import torchaudio.functional as AF

a = torch.rand((904, 1234))
b = torch.rand((904, 145))
c = AF.convolve(a, b, "same")
print(c.size())
</code></pre>

username: dthanoon
cooked: <aside class="quote no-group" data-username="J_Johnson" data-post="4" data-topic="193829">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar"> J_Johnson:</div>
<blockquote>
<pre><code class="lang-auto">import torch
import torchaudio.functional as AF

a = torch.rand((904, 1234))
b = torch.rand((904, 145))
c = AF.convolve(a, b, "same")
print(c.size())
</code></pre>
</blockquote>
</aside>
<p>thanks it worked! I never heard of torchaudio before that :-o</p>
Windowed attention

username: mariaalfaroc
cooked: <p>Hi everyone,</p>
<p>I want to limit the amount of past tokens the attention mechanism can attend to. I want to use the PyTorch Transformer implementation and I was thinking that one way to achieve that it is by modifying the tgt_mask to take that into account. However, I’m having some doubts:</p>
<ul>
<li>Should I also modify the tgt_key_padding_mask  by telling that there is padding in the positions I don’t want it to attend?</li>
<li>Moreover could I just modify tgt_key_padding_mask?</li>
</ul>
<p>This is the code that I have for creating the tgt_mask.</p>
<pre><code class="lang-auto">def create_variable_window_mask(
    size, window_size, dtype=torch.float32, device=torch.device("cpu")
):
    """
    Creates a mask for the target sequence with a variable window size.
    
    Args:
    size (int): The size of the target sequence.
    window_size (int): The size of the window to focus on the last X tokens.
    
    Returns:
    torch.Tensor: The generated mask.
    """
    mask = torch.full((size, size), float("-inf"), dtype=dtype, device=device)
    for i in range(size):
        if window_size &lt; size:
            start = max(0, i - window_size)
            mask[i, start : i + 1] = 0
        else:
            mask[i, : i + 1] = 0
    return mask
</code></pre>
<p>Thank so much in advance!</p>