akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Bi-LSTM acc doesn't improve over 65%

guyhan735: <p>I’m working using Bi-LSTM for fake news detection, however no matter what I do, my training acc is not improving over 65%, In the pre-processing I have removed stopwords, used lemenization and 4-grams, but nothing seems to improve the model, what can I do to improve the training acc</p>
<pre><code class="lang-auto">class RnnBiLSTM(nn.Module):
  def __init__(self, vocab_size, emb_size, hidden_size, drop_prob=0.5):
    super(RnnLSTM, self).__init__()
    self.hidden_size = hidden_size
    self.emb_size = emb_size
    self.emb = nn.Embedding(vocab_size, emb_size)
    self.lstm1 = nn.LSTM(emb_size, hidden_size, num_layers = 3,bidirectional= True 
    dropout=drop_prob,batch_first=True)
    self.dropout = nn.Dropout(0.3)
    self.lin1 = nn.Linear(hidden_size*2, hidden_size//2) 
    self.lin2 = nn.Linear(hidden_size//2, 1)   
    
  def forward(self, x):
      x_size = x.shape[0]
      embeds = self.emb(x) 
      (h1, c1) = (torch.zeros(6, x_size, self.hidden_size), torch.zeros(6,  x_size, self.hidden_size)) 
      lstm_out, _ = self.lstm1(embeds,(h1, c1))
      out = self.dropout(lstm_out)
      out = torch.relu(self.lin1(out))
      out = torch.sigmoid(self.lin2(out))
      return out
</code></pre>

vdw: <p>I can’t really be sure what the issue might be, but some things to note after looking at your code</p>
<ul>
<li>
<p>You use a Bi-LSTM but use <code>lstm_out</code> for further processing. In case of a Bi-LSTM the last step of the forward pass is in <code>lstm_out[:,-1]</code> while the last step of the backward pass is in <code>lstm_out[:,0]</code>; see <a href="https://discuss.pytorch.org/t/rnn-output-vs-hidden-state-dont-match-up-my-misunderstanding/43280">here</a>. Not sure if that’s what you want. For a basic classification task using <code>lstm_out, hidden = self.lstm1(embeds,(h1, c1))</code> and then using <code>hidden</code> is more convenient.</p>
</li>
<li>
<p>I’m actually not quite sure how or even why your current setup works. The shape of <code>lstm_out</code> is <code>(batch_size, seq_len, num_directions*hidden_dim)</code> which you then give to the linear layers. Usually, the linear layers should get <code>(batch_size, hidden_dim)</code>. I can see why it’s <code>hidden_size*2</code> since for you, <code>num_directions=2</code>. But you still have the <code>seq_len</code> dimension. What is the shape of <code>out</code> before you return it?</p>
</li>
<li>
<p>Fake news detection is not a trivial task. In fact, your network will never learn what TRUE or FALSE is but was fake new most likely looks like. So don’t expect a accuracy of close to 100%.</p>
</li>
<li>
<p>Can you overfit your model – that is can get down the training loss down to 0 using a very small training dataset (e.g., just 5 genuine news and only 5 fake news). This is some kind of basic sanity check if your model is setup correctly.</p>
</li>
</ul>

iman.abduljaleel: <p>I am looking for any code regarding detecting multimodal fake news</p>
Load .pt file from a git link

Praskand: <p>It gives ValueError: The provided filename <a href="https://github.com/k.../blob/master/yolovXn.pt" rel="noopener nofollow ugc">https://github.com/k.../blob/master/yolovXn.pt</a> does not exist.</p>

ptrblck: <p>Your URL is invalid as is contains the placeholder <code>...</code> in it.</p>
An efficient implementation of the following using tensor

GoldenalCheese: <p>I am looking for an efficient implementation which can remove the <code>for</code> loop in the following code. From my previous posts, I understand this can be done by <code>torch.compile</code>. But I feel for this specific case, we don’t need <code>torch.compile</code> and built-in pytorch functionalities should be sufficient. Any solution?</p>
<pre><code class="lang-auto">n,c,w = 4,5,7
d0, d1 = 10, 3 # in real case, there might be more dimensions, d0,d1,...,dD-1
transfer = torch.rand(n,c,w)
signal = torch.rand(d0,d1,n,c,w)
results = torch.zeros((*signal.shape[:-2], w) # shape (d0,d1,n,w)

# Efficient implementation to remove the for loop?
for i in range(n):
    results[...,i,:] = torch.sum(signal[...,i,:,:] * transfer[i], dim = -2) # (d0,d1,w)
</code></pre>
Padding (mode) in convolution

71M073J: <p>1.) I am optimising for performance, so i’d like to know how is padding (mode) implemented in the C++ engine - is padding in convolution all implicit, with indexing for “reflect” or “same”, or is every pad performed first, and then convolved upon?</p>
<p>2.) in the same vein, as i assumed that for efficiency in not copying potentially large tensors it is implemented with indexing, is there a “padding_mode” variant of the functional call?<br>
or would the performance be about the same explicitly first calling f.pad, and then f.conv2d?</p>
Validation when using DDP

mariosconsta: <p>Simple question but I did not find any relevant information about it. I am currently using DDP and I am training on 4 cards, every 10 epochs I have a function that runs on the validation set like this:</p>
<pre><code class="lang-auto">mae, mse = self.validate()
</code></pre>
<p>My question is, should I have it run only on the main GPU, like this?</p>
<pre><code class="lang-auto">if self.local_rank == 0:
    mae, mse = self.validate()
</code></pre>
<p>I feel its a bit pointless to have all cards running validation but I haven’t been able to find info that state otherwise.</p>
<p>Thank you for reading <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

divinho: <p>It makes sense to have all cards running validation as you can complete it sooner. (which means you can start training again sooner)</p>
<p>note for the correct result (combined from all cards) you would need to run <code>torch.distributed.all_reduce()</code></p>

mariosconsta: <p>I have never used this method or seen it in any of the tutorials. Do I use this for validation or training as well?</p>

divinho: <p>It’s going to be difficult for you to make progress if you don’t understand the tools you are using. I suggest understanding how DDP works at a high level.</p>
<p>You should not do it for training, that would be bad as it would slow things down for very little benefit.</p>

mariosconsta: <aside class="quote no-group" data-username="divinho" data-post="4" data-topic="200434">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/9dc877/48.png" class="avatar"> divinho:</div>
<blockquote>
<p>It’s going to be difficult for you to make progress if you don’t understand the tools you are using. I suggest understanding how DDP works at a high level.</p>
</blockquote>
</aside>
<p>This is why I am here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">  Thank you for your input, have a lovely day!</p>

divinho: <p>To add an answer to your original question. You could do what you suggested</p>
<pre><code class="lang-auto">if self.local_rank == 0:
    mae, mse = self.validate()
</code></pre>
<p>and that would work (and be the simplest solution)!</p>
<p>Your overall training would be slower because of the validation being slower. But maybe that’s fine for you.</p>
ValueError: invalid literal for int() with base 10: '-3.21588'

Cian: <p>Sorry, I am kind of a total beginner at this stuff, so please bear with me.</p>
<p>I’m trying to train an NMT model FR-EN but I keep getting this error.</p>
<p>Code:</p>
<blockquote>
<p>!onmt_train -config config.yaml</p>
</blockquote>
<p>Error:</p>
<blockquote>
<p>File “/usr/local/lib/python3.10/dist-packages/onmt/inputters/inputter.py”, line 282, in _load_vocab<br>
counters[name][token] = int(count)<br>
ValueError: invalid literal for int() with base 10: ‘-3.21588’</p>
</blockquote>
<p>I looked through the files and the -3.21588 comes from the <code>source.model</code> file. I understand that the error is basically because -3.21588 is not an integer, and what is there should be an integer. Looking through the files, I assume what it should be reading is the… <code>run.vocab.src</code>? How do I get it to read that?</p>
<p>Note: As I said before, I am a beginner. I used a tutorial on NMT and tried to just replace the sample corpora with my own. Perhaps I didn’t set the config right?</p>

ptrblck: <p>The issue is caused by passing a <code>string</code> to <code>int()</code> instead of a <code>float</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">int('-3.21588')
# ValueError: invalid literal for int() with base 10: '-3.21588'

int(-3.21588)
# -3
</code></pre>
Cosine similarity between tensors of different shapes

SimTho: <p>Hi everyone,<br>
I’m working on implementing a paper and I need to calculate the cosine similarity between a tensor of size [L,C] and a tensor of size [B,C,H,W], where B is the batch size, C the number of channels, H and W the dimension of the image. L is a parameter usually set to around 50<br>
I have the code extract that kind of do the job but is really slow since I use for loops.</p>
<pre><code class="lang-auto">L=50

x = torch.randn(16,128,32,32) # [b,c,h,w]
b,c,h,w=x.size()
keys = torch.randn(L, 128) # 
values=torch.randn(L,128)
x=x.view(x.size(0),x.size(1),-1).permute(0,2,1)

keys = keys.unsqueeze(0).expand(x.size(0), -1, -1)
similarity_scores = torch.zeros(16, L, 1024)


for i in range(16): 
    for j in range(L):
        for k in range(1024):
            similarity_scores[i, j, k] = F.cosine_similarity(keys[i, j], x[i, k], dim=0)
</code></pre>
<p>It is possible to do this operation without for loops ? I cannot find a way on my own.<br>
Thanks a lot in advance</p>
In Multihead attention layers, geting zero MACs(multiply accumulate operations)

TrupsT: <p>I am trying to calculate number of MACs in multihead attention layer, using thop tool. Below is the code used. I am gettings macs = 0! Shouldn’t this number be non zero?<br>
Can someone help me understand why macs=0? And how I can get the exact count?</p>
<pre><code class="lang-auto">#MultiheadAttention
import torch
m3=torch.nn.MultiheadAttention(1024, 8)
query = torch.randn(1,1024);
key = torch.randn(1,1024);
value = torch.randn(1,1024);
o3,mask=m3(query, key, value)
print(o3)
print(o3.shape)

output : 
tensor([[ 0.5605, -0.1514, -0.5640,  ..., -0.0213, -0.4984, -0.2785]],
       grad_fn=&lt;SqueezeBackward1&gt;)
torch.Size([1, 1024])
</code></pre>
<p>‘thop’ prints number of macs as 0!</p>
<pre><code class="lang-auto">from thop import profile
macs, params = profile(m3, inputs=(query, key, value))
macs

output: 
0.0
</code></pre>
Torch.compile cache_size_limit best practice

rickyx: <p>Hey, I am getting the below error in my testing which I have various shapes of inputs run through my model.</p>
<pre><code class="lang-auto">msg = 'cache_size_limit reached'

    def unimplemented(msg: str) -&gt; NoReturn:
        assert msg != os.environ.get("BREAK", False)
&gt;       raise Unsupported(msg)
E       torch._dynamo.exc.Unsupported: cache_size_limit reached

../../anaconda3/lib/python3.9/site-packages/torch/_dynamo/exc.py:193: Unsupported
</code></pre>
<p>I understand this happens when recompilation happens. I could get around it by setting a higher value.</p>
<p>But my questions are:</p>
<ol>
<li>What’s the expected best practice if I am hosting in a production env? Should I just set this value to a high value?</li>
</ol>

marksaroufim: <p>Quite the opposite my suggestion would be to figure out why you’re getting so many recompilations in the first place with <code>TORCH_LOGS="recompiles" python your_script.py</code></p>
Torch profiler data use

AidanGoldfarb: <p>I am using the torch.autograd.profiler and I am not clear how to best work with the data it produces.</p>
<p>My best solution has been adding a <code>build_df()</code> method to the <a href="https://github.com/pytorch/pytorch/blob/main/torch/autograd/profiler_util.py" rel="noopener nofollow ugc">profiler</a>, which returns a pandas dataframe representation of the data, as opposed to an EventList or string, which is returned by <code>key_averages()</code> and <code>table</code> respectively. This allows me to perform the common statistical tests and data manipulation I need in a <a href="https://pandas.pydata.org/docs/reference/" rel="noopener nofollow ugc">well documented</a> manor.</p>
<p>I am curious what solutions others have used to work with the data that the profiler provides, perhaps there is an API which I missed.</p>
<p>Thanks!</p>
Memory cost and running time of @torch.no_grad()

RX28666: <p>Hello,</p>
<p>I’m curious about the potential differences in memory usage and runtime when employing <code>torch.no_grad()</code> for 10 forward passes (no gradient needed) followed by a normal forward-backward update versus using gradient accumulation for 10 forward passes (gradient needed) followed by backward update using accumulated gradients to update the model. I think the runtime should be smaller if using <code>torch.no_grad()</code> as no gradient computation, how about the memory cost between two cases? Thanks.</p>
Pytorch conv2D groups

hapa: <p>Hi everyone,</p>
<p>I was wondering about pytorch’s Conv2D (also 1D, ND…) argument ‘groups’.</p>
<p>My previous layer returns 8 channels, and for each of those, I’d like to learn 2 new ones (i.e., in_channels=8, out_channels=16, groups=8), resulting in a total of 16 channels after the current layer.</p>
<p>My issue now is that the weights of that second layer then have the channel dimension of 16 (not a more comprehensible (8,2) clearly assigning 2 output channels to 1 input channel), but I’d like to trace EXACTLY which new kernel weights have been learned for each previous channel output, i.e., that an input at channel m of the previous layer results in outputs at channels n,k in the following layer.</p>
<p>So does anyone know how the output channels are structured? For example, do channels 1-8 of the output correspond to the first learned filter for each input channel, and channels 9-16 of the output correspond to the second?</p>
<p>Thanks for the help!</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/visualizing-the-outputs-of-kernels-instead-of-the-outputs-of-filters/137266/4">This post</a> might be helpful.</p>

hapa: <p>Awesome, thanks a lot!</p>
LLM pretraining on distributed training

ori: <p>I have seen that when training llm model on multiple GPUs. Model will be deployed on one first and then other GPUs with data. In this case GPU:0 will have memory utilised while other not to do extend. So is there a right way to distribute around all gpu’s and train model. I know their is DDP and FSDP but how can i wrap this to trainier API and train model. For simple models i could do it but not for trainer api.</p>

Goldname: <p>What trainer api? Huggingface trainer? And whcih distributed training technique are you using?</p>

ori: <p>Am trying to use fsdp with hugging face trainer api. But there isn’t any information how could i do it</p>
Non-Deterministic Behaviour on same GPU and same Docker Image?

Kennard_Ng: <p>Hi!</p>
<p>I am trying to achieve deterministic behavior on the same GPU and same Docker image across different compute instance for the same torchscript file. However, the issue is that I am getting different results depending on the server instance? Note that the Docker image and GPU is the same and the underlying OS is the same. I have turned off all possible sources of determinism:</p>
<pre><code>    at::globalContext().setDeterministicAlgorithms(true);
    at::globalContext().setDeterministicCuDNN(true);
    at::globalContext().setBenchmarkCuDNN(false);
    at::manual_seed(0);
    torch::jit::setGraphExecutorOptimize(false);
</code></pre>
<p>and even all runtime optimizations. I am getting consistent results on the same compute instance but different results on different computer instances. Note that the docker image and CUDA hardware is the same.</p>
<p>Thanks</p>
How to make node in a computational graph accumulate, but not continue backpropagation?

Juuso_Korhonen: <p>Hi,</p>
<p>I have quite a specific problem. How could I make one node wait for gradient arriving from 2 separate computational graphs (that share the “same” node)? Following code exemplifies my problem:</p>
<pre><code class="lang-auto"># Create a tensor and set requires_grad=True to track computation
x = torch.tensor([1.0], requires_grad=True)

# Start a computational graph
y = x * 2
y.retain_grad() # to be able to check gradient later

# detach creates another computational graph
residual = y.detach()
residual.requires_grad_(True)

z = y ** 2

k = z + residual

# Compute gradients
k.backward()

print("y grad")
print(y.grad)  # should be 5, but prints 4 (when using detach())

print("x grad")
print(x.grad)  # should be 10, but prints 8 (when using detach())
</code></pre>
<p>So I would specifically want that y to accumulate the output gradient coming from both the residual and the z, and only after that continue backpropagation (to x).</p>
<p>I understand that for this specific example it would work if I simply would not use detach on residual = y.detach(). But this accumulation logic is needed on another more complicated scenario, where that z path actually calls a separate backward(). I would want the backpropagation to get blocked on y until it receives gradients from both computational graphs connected to it. Maybe DDP (distributed data parallel) could help?</p>
Unreasonable memory consumption by torch.masked.sum with broadcasted tensors

GuillaumeTong: <p>Consider the following code, which throws <code>OutOfMemoryError</code>:</p>
<pre><code class="lang-auto">import torch

a = torch.randn(1000000, 1, device="cuda")
mask = torch.randint(0, 2, (1, 1000000), device="cuda", dtype=torch.bool)
a_, mask_ = torch.broadcast_tensors(a, mask)
b = torch.masked.sum(input=a_, mask=mask_, dim=1)
# b = torch.masked.masked_tensor(data=a_, mask=mask_).sum(1)
# b = torch.masked.as_masked_tensor(data=a_, mask=mask_).sum(1)
</code></pre>
<p>This throws the following error:</p>
<pre><code class="lang-auto">File [...], line 6, in &lt;module&gt;
    b = torch.masked.sum(input=a_, mask=mask_, dim=1)
[...]
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3725.29 GiB
[...]
</code></pre>
<p>I expect these to be the memory usage:</p>
<ol>
<li><code>a</code>: 4,000,000B → 4MB</li>
<li><code>mask</code>: 1MB</li>
<li>The (manually) broadcasted tensors take up no additional memory</li>
<li><code>b</code> is of same shape as <code>a</code>, so takes 4MB too</li>
<li>If cloned, <code>a_</code> would take up 4 * 1000000 ** 2B → 3725.29 GiB</li>
</ol>
<p>It seems that <code>torch.masked.sum</code> internally clones <code>a_</code>, leading to that catastrophic vRAM request.<br>
As you may see, I’ve formulated similar expressions with masked tensors, but it seems their construction process currently requires a cloning that fully expands the memory usage…</p>
<p>Is there any way to avoid a memory explosion for such a “masked sum” operation?</p>
Load the program only one time and later give it some images

Sylvain_Ard: <p>Good night,<br>
for my prediction in yolov9 instance segmentation I see that the model and the program load in 27 seconds then it takes 1.5 seconds per image. Would it be possible to load the program once and then give it images as you go and have it process them afterwards? I’m on Windows 10.<br>
Please help me, it’s quite urgent.<br>
Best regards</p>

ptrblck: <p>Yes, serving solutions, such as <a href="https://developer.nvidia.com/triton-inference-server">NVIDIA/Triton Inference Server</a> or <a href="https://pytorch.org/serve/"><code>torchserve</code></a> might be helpful.</p>

Sylvain_Ard: <p>OK thank you but <a href="https://developer.nvidia.com/triton-inference-server" rel="noopener nofollow ugc">NVIDIA/Triton Inference Server</a> works only with NVIDIA GPUs installed on the PC isn’t it and <a href="https://pytorch.org/serve/" rel="noopener nofollow ugc"><code>torchserve</code></a> is for Windows Server, it is not guaranteed to run in Windows 10, and it is huge, other solutions please ?</p>

Sylvain_Ard: <p>could you give me a tutorial to install torchserve, provide it with the model and make yolov9 instance segmentation predictions on the fly please?</p>
Load Weights Before Multiprocessing Inference Will Cause DeadLock?

xtzd: <p>Here is the minimal code case. The code will stuck at Checkpoint 1. Comment that line will erase the stuck, it seems the load weight cause the deadlock. But if change hidden_size from 128 into 6, then stuck wont happen forever! So what’s wrong on earth? Please Help!   torch version 1.12.1+cu116</p>
<pre><code class="lang-auto">import multiprocessing
import torch 
import torch.nn as nn 
 
# Change the hidden from 128 to 6 will erase the stuck?
strange_hidden = 128

lstm = nn.LSTM(input_size=2, hidden_size=strange_hidden, num_layers=1, batch_first=True)
torch.save(lstm.state_dict(),"./plain.pt") 

def run(inputs): 
    model = nn.LSTM(input_size=2, hidden_size=strange_hidden, num_layers=1, batch_first=True)
    model.load_state_dict(torch.load("./plain.pt")) 
    x=model(inputs) 
    return x 

state=torch.rand(1,1,2) 
print("CHECKPOINT 0 PASS")

# Comment the line below will erase the stuck
lstm.load_state_dict(torch.load("./plain.pt"))
# Comment the line above will erase the stuck
# What is weird, if I change the strange hidden_size from 128 to 6, it would never stuck!


x = lstm(state)
print(x[0].shape)
print("CHECKPOINT 1 PASS")

pool = multiprocessing.Pool(5)
for i in range(5):
    pool.apply_async(func=run,args=(state,))
pool.close()
pool.join()
print("CHECKPOINT 2 PASS")
</code></pre>
Help! RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

chbw: <p>I met a problem in finetuning a model,I would appreciate your any help.<br>
here is my code:</p>
<pre><code class="lang-auto">def calculate(decoded):
    count_ones_per_column = torch.sum(decoded&gt;0, dim=0)
    total_rows = decoded.size(0)
    probability_per_column = count_ones_per_column / total_rows
    return probability_per_column

def train_model(train_dataloader,model, optimizer,keys, num_epochs=5):
    criterion = lambda decoded,keys : F.l1_loss(decoded,keys)
    model.train()
    for epoch in range(num_epochs):
        running_loss = 0.0
        for ii , imgs in enumerate(tqdm.tqdm(train_dataloader)):
            imgs=imgs.to(device)            
            decoded=model(imgs) #8*48
            # print(decoded)
            
            prob_1=calculate(decoded)
            optimizer.zero_grad()
            loss = criterion(prob_1, keys)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()
        print(f"Epoch {epoch+1}, Loss: {running_loss/len(train_dataloader)}")

def main():
    train_loader=get_dataloader('data',transform,4,num_workers=0,collate_fn=None)
    keys_05=torch.tensor(0.5).to(device).repeat(48)
    pretrained_model = get_hidden_decoder(48,1,8,64)
    ckpt=get_hidden_decoder_ckpt('other_dec_48b.pth')
    pretrained_model.load_state_dict(ckpt,strict=False)

    pretrained_model.to(device)
    for param in pretrained_model.parameters():
        param.requires_grad_(True)
    optimizer = optim.SGD(pretrained_model.parameters(), lr=0.001, momentum=0.9)
    train_model(train_loader,pretrained_model, optimizer,keys_05)
</code></pre>
<p>I added <code>loss.requires_grad_(True)</code> before <code>loss.backward()</code>,but the loss didn’t change during training.<br>
it seems that <code>decoded&gt;0</code> detaches the tensor, so the <code>grad_fn</code> becomes none after it. I wonder if there is any solutions.Thanks!</p>
Best practice for using data_ptr<Type>

MTMD: <p>When developing custom CUDA kernels, it is common to get a pointer to the GPU memory and pass it to the kernel as shown below.</p>
<pre><code class="lang-auto">kernel&lt;&lt;&lt;num_block, num_threads&gt;&gt;&gt;(tensor.contiguous().data_ptr&lt;float&gt;())
</code></pre>
<p>Where <code>tensor</code> is a tensor that is passed from Python side. The problem is, GPU kernels are asynchronous. So, how do we know that the pointer that is passed to the kernel is still valid when the kernel is actually launched?<br>
More specifically, the call to this kernel returns immediately. So, how does PyTorch know that the Tensor is still needed and shouldn’t be freed?</p>

ptrblck: <p>The <code>CUDACachingAllocator</code> checks if the block requested to be freed (or rather moved to the cache) is in use and if so will record an event as seen <a href="https://github.com/pytorch/pytorch/blob/f688d7a2f7dcb6caf5ded0e1eabedd5a7a6dc43b/c10/cuda/CUDACachingAllocator.cpp#L1296-L1308">here</a>.</p>