akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Continuous label for transfer learning

username: dp_unil
cooked: <p>Hi!</p>
<p>tl;dr = How can I set up a dataset that contains multiple continuous target variables that I want to predict upon one-by-one?</p>
<p>I am trying to predict <em>continuous</em> biological traits (e.g., age, bmi, blood pressure, etc.) from retinal fungus images. I have a pretrained transformer model that I would like to fine tune over many of these continuous labels. The dataset I am using to train, test, and validate on is from ~200,000 individuals. One of the issues I am running into is that it looks like: 1) most of the tutorials I find online are for classifying based on binary or small multi-class labels; or 2) the recommended method of creating a dataset is to save the train/test and control/condition into multiple folders. As I am looking at continuous targets and am using a dataset that is more than 0.5TB, I was thinking I should try to generate a pipeline that would digest a tabular dataframe that contains columns that represent the continuous target variables (e.g., age, bmi, blood pressure, etc.), a column that indicates whether the data is test/train data, and a column that holds the information regarding the path to the photo. Regardless if this tabular metadata is a good way to go about setting up a custom dataset for PyTorch transfer learning, does anyone have a good way to organize their custom image-based datasets to predict continuous target variables? Thank you in advance for any help <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
Accuracy Drop in ViT with Patch Embedding: Investigating the Impact of Added Convolutional Layers

username: External_happy
cooked: <p>Hello, I’m currently working on incorporating a patch embedding layer into my Vision Transformer (ViT). I’ve defined this layer using 3 2D convolutional and initialized it with a normal distribution. The remaining layers of the model use pre-trained weights. However, during inference without training, I noticed a significant accuracy drop to 0%. Initially, the model achieved 45% accuracy with only one 2D convolutional layer (ViT uses one conv as patch embedding).</p>

username: eqy
cooked: <p>Are you finetuning the additional layers or are you evaluating the accuracy with the randomly initialized weights? I’m not sure acceptable accuracy would be expected in this case.</p>

username: External_happy
cooked: <p>I am assessing the precision using randomly initialized weights. Unexpectedly, the addition of a single layer led to a significant decrease in accuracy, contrary to what was suggested in a “Vision Transformer for Contrastive Clustering” that claimed it would enhance accuracy.</p>

username: eqy
cooked: <p>After briefly looking at the paper I couldn’t see where it was indicated that randomly initialized weights were used, could you point to where exactly this is said in the paper?</p>
<p>In general I am having trouble understanding how accuracy could be improved or even maintained if a randomly initialized layer is added to the model without finetuning or weight updates.</p>
Is there any torch == 1.3.1 whl available for the Python 3.10 version?

username: i_amRifat
cooked: <p>Is there any <code>torch == 1.3.1</code> whl available for the <code>Python 3.10</code> version? I am using my university server so I can’t downgrade the Python version, and the admin of the server has already mentioned that they are not going to downgrade Python because of some issues.</p>
<p>So, Can anyone help to find another way to solve the issue?</p>

username: ptrblck
cooked: <p>No, since the old <code>torch==1.3.1</code> version was released on Nov 2019 while Python 3.10 came out October 2021.</p>
Hows is the optimizer and the loss works together

username: preshel
cooked: <p>I read two posts here about this subject but I still could not figure out the “software design” that makes the loss function to have any impact on the model or the optimizer.</p>
<pre><code class="lang-auto">model = ...
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.0001)
predictedY = model(xTrain)
loss = criterion(predictedY, yTrain)
loss.backward() # how does the output of "backward()" impacting the model, if the loss does not have any reference / "pointer" to the model?
optimizer.step()
optimizer.zero_grad()  
</code></pre>
<ul>
<li>the optimizer is getting model.parameters() in the c’tor, so that makes him “Aware” of the model. I get that.</li>
<li>criterion is not related to the model or to the optimizer</li>
<li>loss is not related to the model or to the optimizer</li>
</ul>
<p>So how does the output of the loss getting propagated into the model?</p>

username: ptrblck
cooked: <p>PyTorch’s Autograd tracks all operations performed with trainable parameters and uses this computation graph to calculate the gradients in the <code>backward</code> call. The loss function is treated in the same way as any differentiable operation.<br>
The <code>optimizer</code> holds references to the passed parameters and uses their <code>.grad</code> attribute to update these.</p>

username: preshel
cooked: <p>“Autograd tracks all operations performed with trainable”<br>
How does it tracks? I am trying to technically undertand it the implicit mechanism.<br>
I mean, I would expect the code to look something like this:</p>
<pre><code class="lang-auto">model = ...
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.0001)
predictedY = model(xTrain)
loss = criterion(predictedY, yTrain)
loss.backward(model) # **now it would be obvious** 
optimizer.step()
optimizer.zero_grad()  
</code></pre>

username: ptrblck
cooked: <p><a href="https://pytorch.org/docs/stable/notes/autograd.html">These docs</a> might be helpful.</p>
Will it be ok calling loss.backward() within forward() method?

username: kyuu
cooked: <p>For example, if we design forward() like that:</p>
<pre><code class="lang-auto">def forward(self, x, y):
    y_pred = self.w * x
    loss = mse_loss(y_pred, y)
    loss.backward()
    return y_pred
</code></pre>
<p>and</p>
<p>do</p>
<pre><code class="lang-auto">for epoch in range(10):
    for x, y in data_loader:
        optimizer.zero_grad()
        y_pred = model(x, y)
        optimizer.step()
</code></pre>
<p>in training loop.</p>

username: tom
cooked: <p>Will this technically work? Likely.</p>
<p>Is it a good idea? Maybe not if other people are surprised by it.</p>
<p>Best regards</p>
<p>Thomas</p>

username: ptrblck
cooked: <p>Additionally to the already mentioned surprised users, I would also double check if some utils., such as <code>DDP</code>, would still work as these often depend on “standard” training loops. I.e. in this case I wouldn’t know if <code>DDP</code> would synchronize the gradients or not.</p>

username: kyuu
cooked: <p>Yap, it looks monstrous and maybe idiotic. And I was wondering if DDP will synchronize as well.</p>
<p>But as for some cases:</p>
<pre><code class="lang-auto">def forward(self, x, y):
    x = self.in_layer(x)
    loss = 0
    for _ in range(self.n):
        x = self.net(x)
        loss += self.loss_fn(x, y)
    return x, loss
</code></pre>
<p>and call loss.backward() outside</p>
<p>I wonder if there is another way to cumulate the grad of those losses than maintain a large computation graph, in order to reduce the memory consumption.</p>
<p>If it’s just better to implement a custom function to avoid calling it in forward(), or even avoid .backward() just autograd and synchronize yourself in it for the eyes of somebody or "aesthetics”. Fine, I will take it.</p>
Budget machine for tinkering with LLMs? ROCm a good alternative?

username: vdw
cooked: <p>tl;dr: I’m considering building a budget machine for tinkering with LLMs, but I’m not sure if this is a good idea and how to go about it.</p>
<p>For context: I work in a university department. I currently have access to a 2080 Ti on a shared machine, and we’re in the process of acquiring a small server with 2 L40 cards. So for any larger experiments, I will be able to use this shared machine.</p>
<p>However, I think I would like to have my own small machine for tinkering: trying different models and techniques, and just playing around, and preparing larger experiments to be run on the server. My focus is on teaching and education not on state-of-the-art research.</p>
<p>With aiming for a good amount of VRAM, the 4060 Ti 16GB seems to be the most obvious choice; I also like the low power requirements (regarding energy and cooling). But this card seems to have a poor reputation overall. I’m also not sure what currently the sweet spot w.r.t. the CPU and memory is – I completely lost track of Intel’s and AMD’s generations over the last years.</p>
<p>Regarding getting the most VRAM for the money, an AMD 7900 XTX 24GB would also be an interesting alternative. I know that PyTorch supports ROCm, but there still seems to be the consensus to stick with Nvidia for ML/AI, at least for the time being. Or what are the community’s opinion and experiences with ROCm in practice?</p>

username: nlgranger
cooked: <blockquote>
<p>Or what are the community’s opinion and experiences with ROCm in practice?</p>
</blockquote>
<p>Mine is not very positive. If you have access to an AMD-based cluster, it may be worth investing time to deal with the bugs and workarounds, if it’s your own money you are spending, stay with NVidia.</p>
<p>My preferred hardware setup is:</p>
<ul>
<li>Any GPU that can run my model with batch size one on my dev computer, I just use it to debug my code.</li>
<li>Switch immediately to a server/cluster. If you are a student or academic, you have access to PyCharm pro license, so you can even develop and debug directly on a remote server via ssh. Otherwise, use jupyter if you need to interactively update and run your code.</li>
</ul>
Pytorch numerical error for same inputs in multiple transformer layers

username: Nikos_Ellinas
cooked: <p>Hello,<br>
I have implemented a transformer encoder-decoder architecture for regression, where I give the encoded sequence to the transformer decoder and the decoder runs step-by-step, i.e. it starts with [encoded,zeros] and predicts the first token, then continues with [encoded,zeros,token1_pred] and predicts the second token and so on.<br>
When examining the outputs, I have noticed that even with a single multi head attention layer as the decoder I have numerical errors for the same inputs, e.g. the decoder always outputs the full sequence of tokens but we only use the last one, BUT the previous ones must be the same, as the inputs are exactly the same. By examining these this is not the case, and these errors start from 10e-8 but as they are propagated in the layers of the transformer and the timesteps of the decoding the error becomes so large that the outputs does not make any sense.<br>
I noticed the same behavior in Karpathy’s nanoGPT repo where the same thing happens with the logits which are the continuous values, but since a softmax is applied there, this is never reflected on the outputs because the indices remain the same.</p>

username: tom
cooked: <p>The results of floating-point operations are subject to numerical precision errors even if they mathematically are the same.<br>
This can be a problem (I’ve seen this e.g. with things like Llama’s ROPE and bf16 vs. fp16) but there is only so much that you can do about it since the deviations up to numerical precision are an implementation detail of the operations.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Nikos_Ellinas
cooked: <p>Yes, I noticed that in my case it cannot be avoided as it seems. It is weird because I hadn’t encountered this behavior with LSTMs because they keep their state, but transformers are very different and I notice some huge deviations.</p>

username: tom
cooked: <p>Part of it might be that LSTMs were typically operated with 32bit precision and now we have 16 bit or tf32 a lot. Also, the classification of operators for “fp16-safety” in automatic multi-precision hints towards the softmax in the attention and the LayerNorms being potentially sensitive while LSTMCell is safe (<a href="https://pytorch.org/docs/stable/amp.html#cuda-ops-that-can-autocast-to-float16" class="inline-onebox" rel="noopener nofollow ugc">Automatic Mixed Precision package - torch.amp — PyTorch 2.1 documentation</a>).</p>
<p>Best regards</p>
<p>Thomas</p>

username: Nikos_Ellinas
cooked: <p>Yes, I only play with 32bit, but I intended to start using torch.amp so this is a good thing to keep in mind thanks!</p>
Exceptional time overhead occurs during prefix sum computation

username: BearBiscuit05
cooked: <p>When performing the prefix sum operation, it appears that there is an abnormal time overhead. Subsequent operations on the calculated results of the prefix sum incur significantly high additional time costs. I am unsure about the reason of this thing and how to resolve the issue.<br>
code:</p>
<pre><code class="lang-auto">binAns = torch.bincount(dstList)
cumTime = time.time()
ptrcum = torch.cumsum(binAns, dim=0)
print("dstList:" ,dstList.shape)
print("binAns:" ,binAns.shape)
print(f"using time : {time.time()-cumTime:.4f}s...")
cumTime = time.time()
zeroblock=torch.zeros(1,device=binAns.device)
print(f"zeroblock using time : {time.time()-cumTime:.4f}s...")
cumTime = time.time()
inptr = torch.cat([zeroblock,ptrcum]).to(torch.int32)   
print(f"cat using time : {time.time()-cumTime:.4f}s...")
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">loading data time : 0.4238s
dstList: torch.Size([148151720])
binAns: torch.Size([9805926])
using time : 0.0002s...
zeroblock using time : 0.0000s...
cat using time : 0.0001s...
</code></pre>
<p>If I include a print operation:</p>
<pre><code class="lang-auto">binAns = torch.bincount(dstList)
cumTime = time.time()
ptrcum = torch.cumsum(binAns, dim=0)
print("dstList:" ,dstList.shape)
print("binAns:" ,binAns.shape)
print(f"using time : {time.time()-cumTime:.4f}s...")
cumTime = time.time()
zeroblock=torch.zeros(1,device=binAns.device)
print(f"zeroblock using time : {time.time()-cumTime:.4f}s...")
cumTime = time.time()
inptr = torch.cat([zeroblock,ptrcum]).to(torch.int32)   
print(inptr)  # additional print operation
print(f"cat using time : {time.time()-cumTime:.4f}s...")
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">loading data time : 0.4285s
dstList: torch.Size([148151720])
binAns: torch.Size([9805926])
using time : 0.0002s...
zeroblock using time : 0.0000s...
tensor([        0,         2,         4,  ..., 148151712, 148151712,
        148151712], device='cuda:0', dtype=torch.int32)
cat using time : 140.7660s...
</code></pre>

username: tom
cooked: <p>You want to call <code>torch.cuda.synchronize()</code> before all timetaking in order to have the GPU finish all queued work. The print (or .cpu(), .item(), .tolist()) does implicitly do such a sync.<br>
Python’s documentation suggests using <code>time.perf_counter()</code> instead of <code>time.time()</code> for benchmarking.</p>
<p>Best regards</p>
<p>Thomas</p>
Print() behaves differently when using formatted string

username: preshel
cooked: <p>I have a pytorch tensor array named yTensor.<br>
I was created from Pandas DF as so<br>
yTensor = torch.tensor(dfY.values, dtype=torch.float32).to(device)</p>
<p>When I print an item such as<br>
print(yTensor[0]), it prints “tensor(0., device=‘cuda:0’)”<br>
But when I print it in formatted form<br>
print(f’yTensor[0]={yTensor[0]}'), it prints “0.0”</p>
<p>Is that by design or some sort of bug?</p>

username: tom
cooked: <p>It’s more or less by design and the different ways Python creates representatrions: printing a tensor directly uses the <code>__str__</code> method under the hood while <code>f"{tensor}"</code> will use the <code>__format__</code> method (if it exists). PyTorch implements both and the output is slightly different for scalar tensors.</p>
<p>Best regards</p>
<p>Thomas</p>
Torch.detach() behavior

username: Anshuman_Senapati
cooked: <p>Hi all,</p>
<p>Using torch.detach().cpu().numpy() makes 1.0000 (torch.float32) as 1.0000001 (float32). Is there an explanation for this? Is this behavior specific to MPS? And is the only way of mitigating this is to clip the np vector after detaching?</p>
<p>action from actor tensor([[1.0000, 0.2277]], device=‘mps:0’, grad_fn=&lt;AddBackward0&gt;)<br>
action after detach [[1.0000001  0.22771475]]</p>

username: JuanFMontesinos
cooked: <p>Comparing str of a tensor is not a proper way of comparing two tensor.<br>
You are showing a different number of digits in your comparison. You should proof they are different beyond fp32 numerical error.</p>

username: huahuanZ
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79b1fe9df784e734e8b8bdab0ccd163efd75d72b.png" alt="Screenshot 2023-11-27 at 14.46.56" data-base62-sha1="hmz7MLYzy0UJC4KO9WuoTZLxKL9" width="370" height="120"></p>
<p>Nothing related to detach or mps device.<br>
Pytorch only displays a limited range of precision at printing tensor out, so here “1.0000” is indeed not “1”</p>
Gloo won't connect between AWS ec2 instances

username: ericauld
cooked: <p>Hi, noobie here, I have two ec2 instances which won’t connect with gloo.</p>
<p>Things I’ve tried:</p>
<ul>
<li>The script works when I run it on two shells from the same machine (the rank 0 machine)</li>
<li>I can create a simple connection between the two instances with netcat</li>
<li>I’ve set the GLOO_SOCKET_IFNAME to ‘ens5’ after consulting ifconfig(1)</li>
<li>I’ve confirmed both the rank 0 and rank 1 machine are stuck on <code>init_process_group</code></li>
</ul>
<p>Here’s the code: <a href="https://gist.github.com/ericauld/85512c70c665e7f62844d0d928677ddb" class="inline-onebox" rel="noopener nofollow ugc">distributed_training.py · GitHub</a></p>

username: ericauld
cooked: <p>Some more information:</p>
<p>Turning verbosity up showed that the connection is succeeding, but is hanging after the connection is established.</p>
<p>Rank 0: [I socket.cpp:297] [c10d - debug] The server socket on [::]:23456 has accepted a connection from [ip-….us-west-2.compute.internal]:5740</p>
<p>Rank 1: [I TCPStore.cpp:261] [c10d - debug] TCP client connected to host …:23456</p>
<p>I’m using ipv4.</p>

username: ericauld
cooked: <p>Posted to Stack Exchange:</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/77553886/pytorch-distributed-from-two-ec2-instances-hangs">
  <header class="source">

      <a href="https://stackoverflow.com/questions/77553886/pytorch-distributed-from-two-ec2-instances-hangs" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/8755792/eric-auld" target="_blank" rel="noopener nofollow ugc">
    <img alt="Eric Auld" src="https://i.stack.imgur.com/YA7Ku.jpg?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/77553886/pytorch-distributed-from-two-ec2-instances-hangs" target="_blank" rel="noopener nofollow ugc">PyTorch distributed from two ec2 instances hangs</a>
</h4>

<div class="tags">
  <strong>python, pytorch, distributed-computing</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/8755792/eric-auld" target="_blank" rel="noopener nofollow ugc">
    Eric Auld
  </a>
  on <a href="https://stackoverflow.com/questions/77553886/pytorch-distributed-from-two-ec2-instances-hangs" target="_blank" rel="noopener nofollow ugc">11:32PM - 26 Nov 23 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Fx cannot trace some BERT models

username: Zhihao_Zhang
cooked: <p>I met the <code>ValueError: You cannot specify both input_ids and inputs_embeds at the same time</code> error when I tried to quantize the BERT model using fx quantization. So I looked into the fx trace code and found that the forward args are automatically filled by proxies. However, some of the BERT models require that we cannot specify both args at the same time which results in the error. How could I solve it?</p>

username: michaelklachko
cooked: <p><a class="mention" href="/u/zhihao_zhang">@Zhihao_Zhang</a> have you solved this issue?</p>

username: xuann6
cooked: <p>Hi,</p>
<p>Also stuck in this problem, is there any suggestion?</p>

username: jrt-20
cooked: <p>have you solved this issue?</p>
Functional.conv2d slow when used in model on GPU

username: weiskohlmoe
cooked: <p>In a current project of mine I need to project the weights of my convolutions. To do that I used my own module that uses <code>torch.nn.functional.conv2d</code> instead of <code>torch.nn.Conv2d</code><br>
However I realized that it is very slow when I use the model on a GPU instead of a CPU.<br>
Even more surprising, when I just use conv2d outside of the module (with the same weights) it is also a lot faster.<br>
Here is the minimal code I used to produce this problem:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import time

class PytorchConvMatrixModule(nn.Module):
    def __init__(self,kernel_size):
        super().__init__()
        self.conv_weights = nn.Parameter(torch.randn((1, 1, kernel_size, kernel_size)))

    def forward(self, x, to_transpose):        
        if(to_transpose):
            return F.conv_transpose2d(x, self.conv_weights)

        else:
            return F.conv2d(x, self.conv_weights)
        
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
conv_module_gpu = PytorchConvMatrixModule(7)
conv_module_cpu = PytorchConvMatrixModule(7)
conv_module_cpu.conv_weights = nn.Parameter(conv_module_gpu.conv_weights.clone())
conv_module_gpu.to(device)

x = (torch.rand([24,1,48,48])*2 - 1).to(device)

start = time.time()
y_gpu = conv_module_gpu(x, to_transpose=False)
print("Time for convolution on GPU:", time.time() - start)
start = time.time()
y_cpu = conv_module_cpu(x.cpu(), to_transpose=False)
print("Time for convolution on CPU:", time.time() - start)
weights = conv_module_gpu.conv_weights
start = time.time()
y_direct = F.conv2d(x, weights)
print("Time for convolution on GPU without model: ", time.time() - start)
</code></pre>
<p>I then get the following output:<br>
Time for convolution on GPU: 1.6394927501678467<br>
Time for convolution on CPU: 0.002363443374633789<br>
Time for convolution on GPU without model:  0.0002646446228027344<br>
I’m really surprised by this since I (at least in my opinion) did not change anything too significant to get such a decrease in speed. The input tensor is just uniformly distributed between -1 and 1.<br>
I used pytorch version 2.0.1 and my GPU is a titan RTX.</p>

username: ptrblck
cooked: <p>CUDA operations are executed asynchronously and you would need to synchronize the code manually before starting and stopping the host timers. Alternatively, you could also use <code>torch.utils.benchmark</code>. Your current profiling is thus most likely invalid.</p>

username: weiskohlmoe
cooked: <p>You were totally right. I ran it again using <code>torch.utils.benchmark</code> and got the expected results namely (averaged over 100 runs):<br>
conv_module_gpu(x, to_transpose=False): 48.06 us<br>
conv_module_cpu(x.cpu(), to_transpose=False):  372.21 us<br>
F.conv2d(x, weights): 55.16 us<br>
As explained in the corresponding website only the loading of cuBLAS was timed so it showed the most time for whatever GPU operation I used first.<br>
Thank you for showing me how to time code correctly</p>
Confusing result in torch.matmul()

username: zhenxz0121
cooked: <p>A confusing result about basic torch.matmul().</p>
<pre><code class="lang-python">    all_sequence_output = torch.rand(12,512)
    all_visual_output = torch.rand(1,512)
    retrieve_logits3 = torch.matmul(all_sequence_output, all_visual_output.t())
    retrieve_logits4 = torch.matmul(all_sequence_output[0], all_visual_output.t())
    print(torch.equal(retrieve_logits3[0], retrieve_logits4))
</code></pre>
<p>result:</p>
<pre><code class="lang-auto">False
tensor([127.57218933105468750000000000000000]) tensor([127.57218170166015625000000000000000])
</code></pre>

username: ptrblck
cooked: <p>This is expected behavior caused by the limited floating point precision and a potentially different order of operations.</p>
Combine Tensor Parallelism and Data Parallelism

username: secrettoad
cooked: <p>Hello,</p>
<p>I have a question that I am very much hoping is an easy answer, although I myself cannot find it documented anywhere to save my life. Thank you in advance for any help.</p>
<p>I need to combine that capabilities of tensor parallelism and data parallelism, in order to train a model with a very large layer in relatively little time. I would have a cluster of pytorch workers/nodes, each with multiple GPUs available, onto which i would distribute partitions of the training set, which would then be sharded along with the model parameters onto multiple GPUs within each worker before being fit.</p>
<p>I cannot find an example or even confirmation that this is a supported pattern. If anyone from the dev team is able to let me know if this is something i completely missed or is perhaps planned soon, I would very much appreciate it. Thank you.</p>

username: ptrblck
cooked: <p>You could check <a href="https://github.com/NVIDIA/Megatron-LM">Megatron-LM</a> which implements different parallelization strategies and is the base for a lot of other implementations from various teams.</p>
Different learning rates for different parameters

username: smu226
cooked: <p>Hello! How can I specify a different learning rate for each parameter of my model. In my case I have something of the form:</p>
<pre><code class="lang-auto">x_index = torch.tensor([14.,15.,8.,9.,10.,11.,13.,12.,5.,4.,3.,2.,0.,1.])
wavenumber_vals = torch.tensor([284829.132,284829.132,284840.730,284840.730,284854.732,284854.132,284864.420,284864.920,284884.170,284884.620,284898.434,284898.814,284910.218,284910.218],dtype=torch.float64)

params = torch.tensor([B_0, D_0, gamma_0, b1_0, c1_0, b2_0,eqQ_0],dtype=torch.float64, requires_grad=True)

opt = optim.Adam([params], lr=1e-3)
for i in range(10):
  opt.zero_grad()
  loss = my_loss(func(x_index,params),wavenumber_vals)
  loss.backward()
  opt.step()
</code></pre>
<p>I would like to have a different learning rate for each of the 7 parameters of the model. Thank you!</p>
<p>EDIT:</p>
<p>I tried to do it manually:</p>
<pre><code class="lang-auto">learn_rate = torch.tensor([1e-5,1e-10,1e-3,1e-3,1e-3,1e-3,1e-3], dtype=torch.float64)
for i in range(5):
    loss = my_loss(func(x_index,params),wavenumber_vals)
    loss.backward()
    params = params - learn_rate*params.grad
</code></pre>
<p>but after the first pass in the for loop I am getting this error:</p>
<pre><code class="lang-auto">UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/core/TensorBody.h:494.)
  params = params - learn_rate*params.grad
Traceback (most recent call last):
TypeError: unsupported operand type(s) for *: 'Tensor' and 'NoneType'
</code></pre>
<p>which seems to be related to the fact that after the first pass in the for loop <code>params</code> is not a leaf parameter anymore. How can I fix that?</p>

username: ptrblck
cooked: <p>You could pass each parameter separately and specify the learning rate per-parameter as seen <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options">here</a>.</p>
Parameters disappear after setting require_grad = True

username: Secret_Wang
cooked: <p>Hello researchers and engineers, I was working on some stable diffusion related task, and got this problem. The relavent code is as below.</p>
<pre><code class="lang-auto">trainable_params = text_encoder.get_input_embeddings().parameters()
print('num of parameter list1:', len(list(trainable_params)))
for param in trainable_params:
    param.requires_grad_(True)
print('num of parameter list2:', len(list(trainable_params)))
optimizer = torch.optim.AdamW(trainable_params, 
                                lr=learning_rate, betas=(adam_beta1, adam_beta2),
                                 weight_decay=adam_weight_decay, eps=adam_epsilon)
</code></pre>
<p>At first I directly put <code>text_encoder.get_input_embeddings().parameters()</code> into the optimizer, and got <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code>. Then I iterated the trainable_params and set requires_grad_ of every param to True and got the strange behavior. For the two <code>print()</code> functions, the output of the first one is 1 and the output of the second one is 0.</p>
<p>Why is this happening? Would anyone help me solve it or explain the reason? thanks a lot</p>

username: ptrblck
cooked: <p>Isn’t this expected behavior in Python for all <code>iterator</code>s? I.e. once you iterate then they are exhausted and you would need to recreate the <code>iterator</code> again? I’m not in front of my workstation to try it out.</p>

username: ptrblck
cooked: <p>Yes, seems to be the expected behavior:</p>
<pre><code class="lang-python">model models.resnet18()
trainable_params = model.parameters()
print('num of parameter list1:', len(list(trainable_params)))
# num of parameter list1: 62
print('num of parameter list1:', len(list(trainable_params)))
# num of parameter list1: 0

a = [0, 1, 2]
a_iter = iter(a)
print(len(list(a_iter)))
# 3
print(len(list(a_iter)))
# 0
</code></pre>
What should I do if the onnx model I converted is too large

username: yinjun1131
cooked: <p>I am trying to learn how to convert pytorch to onnx and tflite, but the converted onnx model is too large. I am using yolov8n pose, and the converted model is as large as 100mb, compared to others who only have 10mb, and the running speed is ten times slower than others. Have I forgotten anything?</p>
<p>this is my code</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fdfac5a0292b71e36f0a572efab709258d2b43b8.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fdfac5a0292b71e36f0a572efab709258d2b43b8" title="IMG_20231126_191146"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fdfac5a0292b71e36f0a572efab709258d2b43b8_2_690x310.jpeg" alt="IMG_20231126_191146" data-base62-sha1="AeO4K4F2rFa7leC9X47VokyUcSk" width="690" height="310" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fdfac5a0292b71e36f0a572efab709258d2b43b8_2_690x310.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fdfac5a0292b71e36f0a572efab709258d2b43b8_2_1035x465.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fdfac5a0292b71e36f0a572efab709258d2b43b8_2_1380x620.jpeg 2x" data-dominant-color="6E6E66"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20231126_191146</span><span class="informations">1920×864 107 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Pytorch Geometric ptr

username: Jayden9912
cooked: <p>Hi, Pytorch Community.</p>
<p>May I know what is ptr in this output?</p>
<pre><code class="lang-auto">from torch_geometric.loader import DataLoader
training_generator = DataLoader(train_dataset, 1, True,pin_memory = True, drop_last = True)
for batch_id, input in enumerate(training_generator):
  print(batch_id)
  print(input)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6758d104b2ad0b6e13e55d45273d76ad8276d4a.png" data-download-href="https://discuss.pytorch.org/uploads/default/a6758d104b2ad0b6e13e55d45273d76ad8276d4a" title="pytorch_Q"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6758d104b2ad0b6e13e55d45273d76ad8276d4a.png" alt="pytorch_Q" data-base62-sha1="nKz6WAc4IXWnGu5xIIa4uGQ3uPg" width="690" height="160" data-dominant-color="434343"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_Q</span><span class="informations">756×176 7.21 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: falmasri
cooked: <p>If I’m not wrong I think it is the selected batch number</p>

username: Fang_Wu
cooked: <p>A monotonically increasing pointer tensor that refers to the boundaries of segments such that <code>ptr[0] = 0</code> and <code>ptr[-1] = src.size(0)</code>. Hope this helps.</p>
How does torch.rand() sample from uniform distribution?

username: R.Keelan
cooked: <p>I am writing an ML framework in Rust and I would like it to produce the same random numbers as PyTorch.</p>
<p>Specifically, I would like to produce the same values as <code>torch.rand()</code>, provided that both the PyTorch generator are my RNG are seeded with the same value. I have verified that my RNG produces the same raw values as PyTorch’s internal CPU RNG (CPUGeneratorImpl).</p>
<p>From what I can tell, <code>torch.rand()</code> uses the generator to sample from a <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/TensorFactories.cpp#L893" rel="noopener nofollow ugc">uniform distribution</a> in the range [0.0, 1.0). When I tried finding the implementation of that function call, though, I got lost in dispatchers (I have not been able to build PyTorch from source, so I wasn’t able to trace with the debugger).</p>
<p>I have gotten my RNG, NumPy’s RNG, Python’s built-in RNG, and Rust’s mersenne twister RNG to all produce the same values when seeded appropriately and sampling uniformly from [0.0, 1.0), so it seems like there’s something I’m missing when it comes to PyTorch.</p>

username: R.Keelan
cooked: <p>After poking around in the commit history, I found this code in an old version (a3442f62bc2d01b64e554e925e9e0efa89c5e4c3) of PyTorch:</p>
<pre><code class="lang-c++">// floats have 23 bits of mantissa (fractional part)
static uint32_t FLOAT_MASK = (1 &lt;&lt; 24) - 1;
static float FLOAT_DIVISOR = 1.0f / (1 &lt;&lt; 24);

/* generates a random number on [0,1)-double-interval */
static float uniform_float(THGenerator *_generator)
{
  uint32_t x = (uint32_t)THRandom_random(_generator);
  return (x &amp; FLOAT_MASK) * FLOAT_DIVISOR;
}
</code></pre>
<p>I don’t know that this exact code exists in the current version of PyTorch, but that code does let me produce the same values as <code>torch.rand()</code>.</p>
Failed to access this forum by DiscourseHub

username: hwhsu1231
cooked: <p>Hello, PyTorch.</p>
<p>Recently, I tried to use <a href="https://play.google.com/store/apps/details?id=com.discourse" rel="noopener nofollow ugc">DiscourseHub</a> to access this forum, but it failed with the following error message:</p>
<blockquote>
<p>Authorize application access</p>
<p>Sorry, you do not have the required trust level to access the user API</p>
</blockquote>
<p>Since I haven’t met this error when accessing other Discourse-based forums, I wonder what did I miss? how to fix it?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d85ae84f085bb28daf99446b46678b869a9ee318.png" data-download-href="https://discuss.pytorch.org/uploads/default/d85ae84f085bb28daf99446b46678b869a9ee318" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d85ae84f085bb28daf99446b46678b869a9ee318_2_168x375.png" alt="image" data-base62-sha1="uRXTV4i9HI0dIWF5PoSsgbMG1Ek" width="168" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d85ae84f085bb28daf99446b46678b869a9ee318_2_168x375.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d85ae84f085bb28daf99446b46678b869a9ee318_2_252x562.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d85ae84f085bb28daf99446b46678b869a9ee318_2_336x750.png 2x" data-dominant-color="F4F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1080×2400 196 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Pysyft integration with pytorch

username: saqrthabet
cooked: <p>I am using the pysyft library 0.8.2 to implement Federated Learning experiment. At one part of the code I am supposed to send the tensor information over clients established using pysyft commands. But the attributes send() and get() are no longer supported by Pytorch 2.1.0.</p>
<p>import torch<br>
import pysyft as sy</p>
<p>bob = sy.Worker(name=‘bob’)<br>
print(bob)<br>
print(bob.client_cache)<br>
x = torch.tensor([1,2,3])<br>
print(x)<br>
x.send(bob)   # error message: ‘Tensor’ objects has no attributes ‘send’<br>
print(x)</p>
<p>How to overcome this dilemma and send tensor over the clients?</p>

username: ptrblck
cooked: <p>I don’t think <code>send</code> and <code>get</code> were ever supported tensor functions so I assume these were defined in custom tensor classes?</p>

username: saqrthabet
cooked: <p>As the following link indicates: <a href="https://stackoverflow.com/questions/64050391/how-data-and-target-are-choosen-in-a-federated-learning-pysyft" rel="noopener nofollow ugc">Link</a> using hook = sy.TorchHook(torch) all torch tensors will get additional functionality - methods like .send() and get() . The issue now is that in new Pysyft library sy.TorchHook(troch) is no longer needed, therefore, how to extend the torch functionality to include send() and get(). Or what is the replacement for such commands to perform the same action?</p>
Can't find torch._transformer_encoder_layer_fwd in Pytorch document or in Pytorch public code on github

username: hzpro1221
cooked: <p>I saw this method is called in torch.nn.modules.transformer’s code but I couldn’t find out any information about this method ::</p>

username: ptrblck
cooked: <p>Methods starting with an underscore are private and thus not documented. If you want to look into the forward implementation you could search for the method in the repository.</p>

username: braunagn
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> - for the life of me, I can’t find <code>torch._transformer_encoder_layer_fwd</code> in the github repository.  This is all that comes back: <a href="https://github.com/search?q=repo%3Apytorch%2Fpytorch%20_transformer_encoder_layer_fwd&amp;type=code" rel="noopener nofollow ugc">https://github.com/search?q=repo%3Apytorch%2Fpytorch%20_transformer_encoder_layer_fwd&amp;type=code</a></p>

username: ptrblck
cooked: <p>You can find its dispatching in <a href="https://github.com/pytorch/pytorch/blob/0f5e24bda9450a89ba56d2fdd471f56d97fe4546/aten/src/ATen/native/native_functions.yaml#L14437-L14442"><code>aten/src/ATen/native/native_funcitons.yaml</code></a>, which will then call into <a href="https://github.com/pytorch/pytorch/blob/0f5e24bda9450a89ba56d2fdd471f56d97fe4546/aten/src/ATen/native/transformers/transformer.cpp#L75"><code>transformer_encoder_layer_forward</code></a>.</p>
NCCL hangs in pytorch

username: bbjhjhj
cooked: <pre><code class="lang-auto">import torch
import torch.distributed as dist
import os

def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    dist.init_process_group("nccl", rank=rank, world_size=world_size)

def cleanup():
    dist.destroy_process_group()

def nccl_allgather(rank, world_size):
    setup(rank, world_size)

    tensor = torch.full((10,), rank, dtype=torch.float32).cuda(rank)
    print(f"Rank {rank} has data {tensor}")

    gathered_tensor = [torch.zeros_like(tensor) for _ in range(world_size)]

    dist.all_gather(gathered_tensor, tensor, async_op=True)

    gathered_tensor = torch.cat(gathered_tensor, dim=0)
    print(f"Gathered on rank {rank}: {gathered_tensor}")

    cleanup()

def main():
    world_size = torch.cuda.device_count()
    torch.multiprocessing.spawn(nccl_allgather, args=(world_size,), nprocs=world_size, join=True)

if __name__ == "__main__":
    i = 0
    import os 
    import time
    print(os.getpid())
    os.environ["NCCL_DEBUG"] = "INFO"
    os.environ["NCCL_DEBUG_SUBSYS"] = "GRAPH"

    os.sched_setscheduler(0, os.SCHED_RR, os.sched_param(10))



main()

</code></pre>
<p>I have a node with four gpus. When i execute the above code and allocate 4 CPU cores, the code runs perfectly. However, when i execute the above code with 3 CPU cores, it hangs.<br>
This does not happen when the scheduling class is set to CFS.</p>
Understanding CTCLoss

username: aatg
cooked: <p>I have read several blog articles to get an idea of how CTCLoss works algorithmically, and the PyTorch documentation seems straightforward. All the examples I have seen online conform to my understanding, but I am having trouble getting it to work in practice. Here’s a minimal working example where the losses should be close to zero, because the inputs match the targets.</p>
<p>The unreduced loss when I run this is <code>tensor([5.9605e-07,        inf])</code> instead of both terms being close to zero. An explanation of why the second loss is infinite, and how to correctly use CTCLoss so that both terms are close to zero would be appreciated.</p>
<pre><code class="lang-auto">import torch

e = 1e-7
targets = torch.LongTensor([
    [1, 2, 1],
    [1, 1, 1]
])
logprobs = torch.Tensor([
    [[e, 1-2*e, e], [e, e, 1-2*e], [e, 1-2*e, e], [1-2*e, e, e]],
    [[e, 1-2*e, e], [e, 1-2*e, e], [e, 1-2*e, e], [1-2*e, e, e]]
]) # easier to enter in shape (N=2, T=4, C=2+1)
logprobs = torch.log(logprobs)
logprobs = torch.transpose(logprobs, 0, 1) # get to correct shape (T, N, C)

input_lengths = torch.LongTensor([4,4])
target_lengths = torch.LongTensor([3,3])

loss = torch.nn.CTCLoss(blank = 0, reduction='none')
loss(logprobs, targets, input_lengths, target_lengths)
</code></pre>

username: tom
cooked: <p>A loss of inf (likelihood of 0) means the input sequence cannot produce the target. Here, you would need to predict at least <code>1-blank-1-blank-1</code>, which needs T&gt;=5.</p>
<p>Best regards</p>
<p>Thomas</p>

username: aatg
cooked: <p>Thanks for the clarification. Indeed, the following code, where I encode the predicted sequence by inserting blanks between any two consecutive identical characters, gives the expected result of close to zero error: <code>tensor([9.5367e-07, 9.5367e-07, 1.0729e-06])</code></p>
<pre><code class="lang-auto">import torch

e = 1e-7
targets = torch.LongTensor([
    [1, 2, 1, 1],
    [1, 2, 1, 1],
    [2, 2, 2, 1],
])
logprobs = torch.Tensor([ 
    # encoded using blanks 1 2 1 0 1 0
    [[e, 1-2*e, e], [e, e, 1-2*e], [e, 1-2*e, e], [1-2*e, e, e], [e, 1-2*e, e], [1-2*e, e, e]],
    # encoded using blanks 1 2 1 0 1 0
    [[e, 1-2*e, e], [e, e, 1-2*e], [e, 1-2*e, e], [1-2*e, e, e], [e, 1-2*e, e], [1-2*e, e, e]],
    # encoded using blanks 2 0 2 0 2 1
    [[e, e, 1-2*e], [1-2*e, e, e], [e, e, 1-2*e], [1-2*e, e, e], [e, e, 1-2*e], [e, 1-2*e, e]]
])
logprobs = torch.log(logprobs)
logprobs = torch.transpose(logprobs, 0, 1) # get to correct shape (T, N, C)

input_lengths = torch.LongTensor([6,6,6])
target_lengths = torch.LongTensor([4,4,4])

loss = torch.nn.CTCLoss(blank = 0, reduction='none')
</code></pre>
Torch.cuda.is_available() is not GPU

username: Bahran_Jamshidi
cooked: <p>Hi,</p>
<p>I am unable to use the gpu on my gpu server despite creating a conda environment with pytorch-cuda. Could you please help me solve this issue?<br>
When I run:<br>
torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’) I get ‘cpu’<br>
torch.<strong>version</strong> is 2.1.1<br>
torch.version.cuda is 12.1</p>
<p>I am using an NVIDIA A100 40GBGPU through slurm<br>
When I run nvidia-smi I get:<br>
Driver Version: 535.104.12<br>
CUDA Version: 12.2</p>

username: ptrblck
cooked: <p>Of PyTorch does not detect your GPUs you might need to check your drivers as I can properly use the latest binaries in servers with A100s.</p>
Relevant notes for Executorch + QNN (Qualcomm hwd)

username: escorciav
cooked: <p><strong>Kudos to both teams PyTorch &amp; Qualcomm <img src="https://discuss.pytorch.org/images/emoji/apple/clap.png?v=12" title=":clap:" class="emoji" alt=":clap:" loading="lazy" width="20" height="20"></strong></p>
<p>But, I regret to communicate that I prefer to stay with my previous pipeline when porting stuff onto Qualcomm devices (S8G2/3/4, RB5, etc.) in the time being.</p>
<p>Why?<br>
After some hours of playing with ExecuTorch, I feel that there are still rough edges.</p>
<p>Nevertheless, I encourage anyone who has never used QNN (or SNPE) to go for ExecuTorch.<br>
Indeed, I’m very much looking forward for the next releases after things are ironed out properly.</p>
<p><em>Hopefully then, the PyTorch community offers as much as Tensorflow:TFLite.</em></p>
<p>Relevant links &amp; notes for anyone interested in Qualcomm devices after you’re done with the basic tutorial.</p>
<ul>
<li>Understand what <a href="https://pytorch.org/executorch/stable/runtime-build-and-cross-compilation.html" rel="noopener nofollow ugc">the DeepLab script do under-the-hood</a>.</li>
<li><a href="https://github.com/pytorch/executorch/issues?q=is%3Aissue+is%3Aopen+label%3A%22partner%3A+qualcomm%22" rel="noopener nofollow ugc">Issues about ExecuTorch &amp; QNN</a>
<ul>
<li><a href="https://github.com/pytorch/executorch/pull/658" rel="noopener nofollow ugc">MobileBERT</a></li>
<li><a href="https://github.com/pytorch/executorch/issues/1183" rel="noopener nofollow ugc">InceptionV3</a></li>
<li><a href="https://github.com/pytorch/executorch/issues/1182" rel="noopener nofollow ugc">ViT</a></li>
</ul>
</li>
</ul>
Type casting logic?

username: boywithacoin
cooked: <p><code>torch.tensor([-1.0, -2.0]).to(torch.uint8)</code></p>
<p>result: [255, 254]</p>
<p>I’m testing the above casting operation on machines that are ARM and Intel.<br>
Generally, fp32 → uint8 is an undefined behavior.</p>
<p>How is the above cast in torch consistent on all machines? Are there type casting rules somewhere?</p>
<p>Is the above cast taking a path with intermediate hidden operation to cast to uint8?</p>
<p>Additionally, <code>torch.can_cast(torch.float32, torch.uint8)</code> returns false.</p>

username: boywithacoin
cooked: <p>bumping this. anyone?</p>