akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to use DataLoader pin memory when data is a mix of cpu and gpu

jj0mst: <p>Hello,</p>
<p>I am writing a multi-worker data loading pipeline. My input is given by high resolution images that I am decoding directly into the GPU, while my target is given by common labels that I load in CPU memory and move them to GPU only when accessed by the main (trainer) process.</p>
<p>My problem is that I cannot use pin_memory with my DataLoader, since it tries to also pin the GPU data instead of skipping it, and throws this error:<br>
RuntimeError: cannot pin ‘torch.cuda.ByteTensor’ only dense CPU tensors can be pinned</p>
<p>Does anyone know any workaround for this? I would like to exploit pin memory to speedup the CPU-GPU data transfer, but still have not found any trivial solution to this.</p>

ptrblck: <p>You could explicitly create the target tensor using pinned memory or move it as described in <a href="https://pytorch.org/tutorials/intermediate/pinmem_nonblock.html">this tutorial</a>.</p>
Fine tuning the trained model, the initial loss is quite high

weitaoliu: <p>Hi,</p>
<p>I am trying to fine-tune my previously trained model. What I did was load the model using <code>load_state_dict</code> and then I changed the initial learning rate to a smaller value. One thing that confused me is that the initial training loss when conducting fine-tuning is quite high, and the value is quite different from the loss I got in previous training. If they have similar model coefficients, how can they have very different losses? or if I misconducted something?</p>
<p>More details are given here, it is a fully connected MLP model, and the loading is as follows:</p>
<pre data-code-wrap="Python"><code class="lang-Python">if model[tag].fineTuneSwitch == True:
                model[tag].load_state_dict(torch.load(model[tag].annModelPath))
</code></pre>
<p>the loss records are attached.<br>
the previous training case (initial learning rate: 1e-3):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/3/630ec3387807ac2fe25226a19c07493b785e2560.png" alt="Screenshot from 2024-04-08 17-32-49" data-base62-sha1="e8iTGTOvX7YhcR2iqewpkDHT7b2" width="591" height="296"><br>
the fine-tuning case (initial learning rate: 4e-6):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e37a82f76a54b287e0515210b91309c0540d51d0.png" alt="Screenshot from 2024-04-08 17-33-02" data-base62-sha1="wsmRFAzTdUSOg4wvKDdEhQ7lrR6" width="591" height="296"></p>

weitaoliu: <p>The problem was addressed. The way of loading is right for model fine-tuning.</p>

Shilaj_Baral: <p>Hi <a class="mention" href="/u/weitaoliu">@weitaoliu</a>. Can I please know how did you address your problem?</p>

weitaoliu: <p>Hey, I had the problem due to not correctly using the inputs for fine-tuning. The new inputs should have the same or similar meaning as the original inputs (at least in my case). To check your implementation, if you do not have dropout, you can change the learning rate to zero and check the loss values during fine-tuning. If implementation is fine, the loss values should not be large.</p>
Monitor optimizer step - Adam

Johannes_Vogt: <p>Hey,</p>
<p>I am trying to monitor the optimizer steps that means $\Delta W$ after it came out of the optimizer and before it is applied to the weights.</p>
<p>I started with</p>
<pre><code class="lang-auto">for param_group in optimizer.params_groups:
    for param in param_group["params"]:
        state = optimizer.state[param]
</code></pre>
<p>For an Adam optimizer, each state has the keys max_exp_avg_sq,exp_avg_sq,exp_avg.</p>
<p>What do they mean and which of them is used as $\Delta W$?</p>

qq-me: <p>The states track first momentum, second momentum and maximal values of second momentum, because they are used in the update rule. Most optimizers don’t explicitly store delta W.</p>
<p>You can however clone and store parameters before optimizer step, perform a step, and take the difference between parameters before and after step.</p>
Torch.fx for torch.nn.Transformer

MartinZhang: <p>Hi all,</p>
<p>There is a error from torch.fx</p>
<pre><code class="lang-auto">import torch

model = nn.Transformer(nhead=16, num_encoder_layers=12)

gm = torch.fx.symbolic_trace(model)
</code></pre>
<blockquote>
<p>torch.fx.proxy.TraceError: symbolically traced variable cannot be used as inputs to control flow.</p>
</blockquote>
<p>How to deal it?</p>

sirgogo: <p>Hi, I realize this is 3 years late, but have you had any success or workarounds for this? This still doesn’t work.</p>
5080 sm120 support

planesmart: <p>doesn’t support RTX 5080 (<code>sm_120</code>)</p>
<ul>
<li><img src="https://discuss.pytorch.org/images/emoji/apple/three_o_clock.png?v=14" title=":three_o_clock:" class="emoji" alt=":three_o_clock:" loading="lazy" width="20" height="20"> As soon as PyTorch adds support for <code>sm_120</code>, we can <strong>remove <code>--use-cpu all --no-half</code></strong> and enable full GPU acceleration. Do you know when support for sm 120 will be done?</li>
</ul>

ptrblck: <p>The latest stable PyTorch 2.7.0 as well as the nightly binaries built with CUDA 12.8 support Blackwell architectures (sm_100, sm_120).</p>
Scaling Torch profiling to longer computations

LeanderK: <p>I have a numerical scheme that takes quite some time to execute (14 seconds for one scenario, but it can take way longer). It’s using both pytorch and scipy operations. I want to profile this code to find out how long these operations take and how they scale in relation to each other. I tried to use the the pytorch profiler but quickly get OOM-errors when scaling to longer computations. I can only scale it to an computation that takes around 8 seconds. Is there a way to restrict the resolution/profiled modules in order to slim it down? Maybe compress it in some way?</p>
Improved PyTorch Models in Minutes with Perforated Backpropagation

PerforatedAI: <p>I’ve developed a new optimization technique which brings an update to the core artificial neuron of neural networks. Based on the modern neuroscience understanding of how biological dendrites work, this new method empowers artificial neurons with artificial dendrites that can be used for both increased accuracy and more efficient models with fewer parameters but equal accuracy. Currently looking for beta testers who would like to try it out on their PyTorch projects. I’m sharing here now because I just wrote a <a href="https://medium.com/p/42a502e6369a" rel="noopener nofollow ugc">step-by-step guide</a> to show how simple the process is to improve your current pipelines and see a significant improvement on your next training run.</p>
Batch variable length input then return to original length for forward pass

Mole_Turner: <p>I’m dealing with audio data where the length of each sequence can vary by orders of magnitude at sample level and need a way of allowing the model to process the original sequences.</p>
<p>I have written a simple <code>collate_fn</code> which pads the audio sequences for the purposes of creating the batches, however, I then want to undo the padding so the <code>forward()</code> function of my model operates on the original sequences. The simple reason for this is that training on audio where some examples may contain more zero padding than original data will have ramifications to the learning process.</p>
<p>Is there a way of doing this without resorting to having a pretty inefficient for loop within my forward function where I’m forced to process a single example at a time.</p>
AMD ROCm GPU and Windows Support

barista: <p>What’s the current state of OS support for Windows?</p>

gandoeldpk: <p>Really desperate to wait for these too.</p>

fasterinnerlooper: <p>This link might be useful: <a href="https://community.amd.com/t5/ai/how-to-run-a-large-language-model-llm-on-your-amd-ryzen-ai-pc-or/ba-p/670709" class="inline-onebox" rel="noopener nofollow ugc">How to run a Large Language Model (LLM) on your AM... - AMD Community</a></p>

Utorque: <p>Bumping this question as ROCm on windows seems to be maturing !</p>

alphaleadership: <p>Rebump to up the problem</p>
How to use triton kernels with pytorch?

jinooooo: <p>I want to use custom triton kernels with pytorch for training, how can i do this with minimal code change? What i mean is i dont want to define a custom class and do a apply on the particular kernel. Is there a different way to do this?</p>

jinooooo: <p>thanks, but i was wondering if theres another way of doing this, without using torch autograd function, basically the class/function youd mentioned here</p>
QAT model drops accuracy after converting with torch.ao.quantization.convert

du_tran_ngoc: <p>Hello everyone.</p>
<p>I am implementing QAT model yolov8 in 4bit mode for weight and 8bit for activation by setting quant_min, quant_max in config. The model when training and eval gives quite good results, however when I convert using torch.ao.quantization.convert method, the model gives very bad evaluation results. Does anyone know how to solve this problem?</p>

jerryzh168: <p>can you provide full code? I don’t think 4 bit quant is supported with <code>torch.ao.quantization.convert</code></p>
<p>also this is going to be deprecated, we recommend to use <a href="https://pytorch.org/docs/main/quantization.html#prototype-pytorch-2-export-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch main documentation</a> instead</p>
Permute / Unpermute Gradient

Jerome_Ku: <p>Trying to understand how gradients when permuting / unpermuting tensors are calculated.</p>
<p>Here is a minimal repro:</p>
<pre><code class="lang-auto">import torch


def permute(X, gather_indices, topk):
    return X[gather_indices // topk]


def unpermute(X_permuted, gather_indices):
    X = torch.empty_like(X_permuted)
    X.index_copy_(0, gather_indices, X_permuted)
    return X


def check_permute(X, gather_indices, topk):
    X_permuted = permute(X, gather_indices, topk)
    X_unpermuted = unpermute(X_permuted, gather_indices)
    for i in range(topk):
        x = X_unpermuted[i::topk]
        assert x.shape == X.shape
        assert torch.equal(x, X)


def calculate_permute_grad(grad_output, gather_indices, seq_len, topk):
    assert grad_output.shape[0] == seq_len * topk
    dX = unpermute(grad_output, gather_indices)
    dX = dX.reshape(seq_len, topk, -1)
    dX = dX.sum(dim=1)
    return dX


if __name__ == "__main__":
    num_experts = 4
    seq_len = 32
    hidden_size = 64
    device = "cuda"
    torch.manual_seed(0)
    for dtype in [torch.float32, torch.float16, torch.bfloat16]:
        print(f"dtype: {dtype}")
        for topk in [1, 4, 8]:
            X = torch.randn(seq_len, hidden_size, dtype=dtype, device=device, requires_grad=True)
            X.retain_grad()

            selected_experts = torch.randint(0, num_experts, (seq_len, topk), device=device).view(-1)
            gather_indices = selected_experts.argsort(dim=0)
            assert gather_indices.shape[0] == seq_len * topk
            
            # Check permute
            check_permute(X, gather_indices, topk)

            X_permuted = permute(X, gather_indices, topk)
            grad_out = torch.randn_like(X_permuted)
            X_permuted.backward(grad_out)
            assert X.grad is not None
        
            # Calculate grad manually
            dX = calculate_permute_grad(grad_out, gather_indices, seq_len, topk)
            assert dX.shape == X.grad.shape
            diff = (X.grad - dX).abs().max()
            print(f"topk: {topk}, diff: {diff.item():.6f}")
</code></pre>
<p>Here are the outputs for the following dtypes</p>
<pre><code class="lang-auto">dtype: torch.float32
topk: 1, diff: 0.000000
topk: 4, diff: 0.000000
topk: 8, diff: 0.000001

dtype: torch.float16
topk: 1, diff: 0.000000
topk: 4, diff: 0.003906
topk: 8, diff: 0.007812

dtype: torch.bfloat16
topk: 1, diff: 0.000000
topk: 4, diff: 0.062500
topk: 8, diff: 0.062500
</code></pre>
<p>The small diffs for <code>float32</code> leads me to think that my manual gradient calculation is mathematically correct but the larger diffs for <code>{b}float16</code> implies that the way I’m calculating the gradient is not numerically equivalent to the autograd implementation.</p>
<p>The reason I need to calculate the permutation grad manually is for a custom <code>autograd.Function</code>.</p>
<p>Any recommendations for how to better align the gradients when using lower precision data types (<code>bfloat16</code>)?</p>
<p>Thanks!</p>
Does dynamic indexing cause slow performance?

thebrilliot: <p>I have a project in tensor flow that I want to translate to pytorch. The tensorflow code creates an index out of the output of one layer and uses that index to slice a tensor, i.e. <code>index = tf.reduce_sum((output &lt; num).cast(tf.int32), axis=0); return a @ b[index];</code> more or less. Would this work in pytorch? Would it cause performance issues? Thanks in advance!</p>
Using MPI with CppExtension

ettu: <p>Hi, I’m trying to use the CppExtension to import my C++ MPI code as a library. It works when I take out the MPI components, but with it, I get the “undefined symbol: ompi_mpi_int” error. Here’s what my CppExtension object looks like:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/747e75a112e64a9b15b5fab88ddecf87ab799526.png" data-download-href="https://discuss.pytorch.org/uploads/default/747e75a112e64a9b15b5fab88ddecf87ab799526" title="61b2e2025161c76895060e2674dcff3f6f8bc730_2_1035x217"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/747e75a112e64a9b15b5fab88ddecf87ab799526_2_690x144.png" alt="61b2e2025161c76895060e2674dcff3f6f8bc730_2_1035x217" data-base62-sha1="gCyk8rE136XGidnHRoxrPb2OOJ8" width="690" height="144" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/747e75a112e64a9b15b5fab88ddecf87ab799526_2_690x144.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/747e75a112e64a9b15b5fab88ddecf87ab799526_2_1035x216.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/4/747e75a112e64a9b15b5fab88ddecf87ab799526.png 2x" data-dominant-color="222426"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">61b2e2025161c76895060e2674dcff3f6f8bc730_2_1035x217</span><span class="informations">1035×217 36.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does anything seem blatantly set up incorrectly? I’ve also tried setting the environment variables of my openmpi install manually:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b475ff109fe19a0b32b8e5f7ba732dc0be52e836.png" data-download-href="https://discuss.pytorch.org/uploads/default/b475ff109fe19a0b32b8e5f7ba732dc0be52e836" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b475ff109fe19a0b32b8e5f7ba732dc0be52e836.png" alt="image" data-base62-sha1="pKqKxEgzlU8A1BGteOesP1uj4lE" width="690" height="303" data-dominant-color="403F3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">807×355 6.54 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’ve been stuck on this for quite a while, and honestly any tips would help now. Is there a more effective way of debugging this ompi_mpi error? I’ve already tried a dozen solutions online relating to this error, with no change. If I can pinpoint the cause more precisely, it would be a huge help.</p>
Does torchscript support custom autograd function?

acdc: <p>does torchscript support custom autograd function?</p>
<p>I implemented a custom function followed this link:<br>
<a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html</a></p>
<p>Does torchsceipt support this kind of function? I followed this torchscript tutorial:<a href="https://pytorch.org/docs/stable/jit.html#creating-torchscript-code" rel="nofollow noopener">TorchScript</a> and an <a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py" rel="nofollow noopener">example</a></p>
<p>However, I got error “ValueError: Compiled functions can’t take variable number of arguments or use keyword-only arguments with defaults”, and it’s my custom autograd function caused this error. I assumed that the compiler doesn’t know about the argument type of the custom autograd function. I cannot find how to make custom autograd work with torchscript.</p>

driazati: <p>We do not currently support custom autograd functions, but it is something on our radar that we would like to do in the future. You can find more context in <a href="https://github.com/pytorch/pytorch/issues/22329" rel="nofollow noopener">this issue</a>.</p>
<p>It is also possible to replicate most of the behavior in custom autograd functions now via custom C++ operators.</p>

axoflo: <p>Hi, Any updates on torchscript for autograd function ?</p>

ASFaust: <p>i have found a workaround for the case that you want to compute a custom gradient:</p>
<pre><code class="lang-auto">fx = f(x).detach()
gx = g(x).detach() * x
result = fx + gx - gx.detach()
</code></pre>
<p>where <code>f(x)</code> is your forward pass, and <code>g(x)</code> is your custom gradient.</p>
<p>this lets you use torch jit script with custom gradients.</p>
<p>in the forward pass, it evaluates to <code>f(x)</code>, and in the backward pass, the gradient of this expression is <code>g(x)</code>.</p>
<p>it’s not very memory or compute friendly, but at least you can run your model in jit script.</p>
In a portable hard drive, the model running speed of the transformer is very slow

yinjun1131: <p>When I tried to run the model on my computer, I found that if the resources were placed in a mobile SSD, the output token/s of the model was very slow, even reaching 0.2 token/s. When placed in the computer’s hard drive itself, the running speed would return to normal. When observing the task manager, the mobile SSD only occupied a huge amount of resources during the model loading process. When Model Inference 1 started, I also tried to unplug the mobile hard drive, but there were no errors. Where should I start troubleshooting? Do you have any good suggestions regarding this issue?</p>
Flex Attention for full score_mod matrix

glassices: <p>Hi, I am building a project where the attention matrix is sparse and I got a huge performance boost from using FlexAttention by specifying block_mask.</p>
<p>However, my project also requires a highly customized add_score matrix to be added to the score matrix. The add_score matrix of shape [batch][num_heads][seq_len][seq_len] is fully pre-computed by the dataloader and its value cannot easily determined by simple element-wise operator on arguments b, h, q_idx, and kv_idx.</p>
<p>I tried using the following the it won’t compile</p>
<pre><code class="lang-auto">def score_mod(score, b, h, q_idx, kv_idx):
    return score_add[b, h, q_idx, kv_idx]
</code></pre>
<p>Do we have any good solutions for this?</p>
(image, mask) pair do not correspond to each other

bohare: <p>I am writing a simple custom DataLoader (which I will add more features to later) for a segmentation dataset but the (image, mask) pair I return using <code>__getitem()__</code> method are different; the returned mask belongs to a different image than the one which is returned. My directory structure is <code>/home/bohare/data/images</code> and <code>/home/bohare/data/masks</code> .</p>
<p>Following is the code I have:</p>
<pre><code class="lang-auto">import torch
from torch.utils.data.dataset import Dataset
from PIL import Image
import glob
import os
import matplotlib.pyplot as plt

class CustomDataset(Dataset):
    def __init__(self, folder_path):
        
        self.img_files = glob.glob(os.path.join(folder_path,'images','*.png'))
        self.mask_files = glob.glob(os.path.join(folder_path,'masks','*.png'))
    
    def __getitem__(self, index):
        
        image = Image.open(self.img_files[index])
        mask = Image.open(self.mask_files[index])
        
        return image, mask
    
    def __len__(self):
        return len(self.img_files)
</code></pre>
<pre><code class="lang-auto">data = CustomDataset(folder_path = '/home/bohare/data')
len(data)
</code></pre>
<p>This code correctly gives out the total size of the dataset.</p>
<p>But when I use:<br>
<code>img, msk = data.__getitem__(n)</code> where n is the index of any (image, mask) pair and I plot the image and mask, they do not correspond to one another.</p>
<p>How can I modify/what can I add to the code to make sure the (image, mask) pair are returned correctly? Thanks for the help.</p>

bohare: <p>I found the solution thanks to the <a href="https://stackoverflow.com/questions/64458201/image-mask-pair-do-not-match-one-another-in-a-semantic-segmentation-task?noredirect=1#64458299" rel="noopener nofollow ugc">user David S</a>. Apparently, <code>glob.glob</code> returns the list in arbitrary order so I put <code>sorted</code> in front of both images and masks, like so:</p>
<pre><code class="lang-auto">self.img_files = sorted(glob.glob(os.path.join(folder_path,'images','*.png')))
self.mask_files = sorted(glob.glob(os.path.join(folder_path,'masks','*.png')))
</code></pre>
<p>And now it works.</p>

akulkarni3472: <aside class="quote no-group" data-username="bohare" data-post="1" data-topic="100100">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bohare/48/2374_2.png" class="avatar"> bohare:</div>
<blockquote>
<p><code>import glob</code></p>
</blockquote>
</aside>
<p>I’m running into a similar issue, except I still see images + masks that don’t correspond</p>
<p>i.e.</p>
<pre><code class="lang-auto">image_path:  /Brain_Tumor_2D_Dataset/Meningioma/images/Meningioma_159.tif
mask_path:  /Brain_Tumor_2D_Dataset/Meningioma/masks/Meningioma_160_mask.tif
</code></pre>
<p>instead of</p>
<pre><code class="lang-auto">/Brain_Tumor_2D_Dataset/Glioblastoma/images/Glioblastoma_t2_010.tif
/Brain_Tumor_2D_Dataset/Glioblastoma/masks/Glioblastoma_t2_010_mask.tif
</code></pre>
Loss becomes constant afterwarmup?

ecoArcGaming: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6075778b201d499c33037d473674dbaf8d2eec7.png" data-download-href="https://discuss.pytorch.org/uploads/default/d6075778b201d499c33037d473674dbaf8d2eec7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6075778b201d499c33037d473674dbaf8d2eec7_2_690x216.png" alt="image" data-base62-sha1="uxnUiZ2Bu2O09L6RyAiUN6VgNa7" width="690" height="216" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6075778b201d499c33037d473674dbaf8d2eec7_2_690x216.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6075778b201d499c33037d473674dbaf8d2eec7_2_1035x324.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6075778b201d499c33037d473674dbaf8d2eec7_2_1380x432.png 2x" data-dominant-color="F9FAFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1600×501 49.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am training a standard CLIP model with a pretrained resnet. The loss becomes constant after the warmup steps, does anyone have any suggestions as to how to fix this? Thanks!</p>

KFrank: <p>Hi Eco!</p>
<aside class="quote no-group" data-username="ecoArcGaming" data-post="1" data-topic="219517" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ecoarcgaming/48/74453_2.png" class="avatar"> ecoArcGaming:</div>
<blockquote>
<p>The loss becomes constant after the warmup steps</p>
</blockquote>
</aside>
<p>This seems improbably coincidental that your loss flattens out almost exactly when<br>
your learning rate becomes constant at 0.0005.  I suspect that something else is also<br>
happening when you switch learning rates.</p>
<p>What happens if you warm up for 500 or 2000 steps instead of 1000 or if you have your<br>
constant learning-rate value be 0.0003 or 0.0007 instead of 0.0005?</p>
<p>Best.</p>
<p>K. Frank</p>
Request to Join Slack

zhangfeiv0: <p>Hello everyone,</p>
<p>We are a team from ISCAS, and we aim to optimize PyTorch ATen operators on CPUs using RVV. Please refer to the link:  <a href="https://github.com/pytorch/pytorch/issues/147513" rel="noopener nofollow ugc">#147513</a>.</p>
<p>Meanwhile, the RISE organization is also working on optimizing PyTorch ATen operators for RISC-V:<br>
<a href="https://lf-rise.atlassian.net/wiki/spaces/HOME/pages/453148705/Project+RP013+Optimizing+PyTorch+ATen+Operators+for+High-Performance+RISC-V+Hardware" rel="noopener nofollow ugc">Optimizing PyTorch ATen Operators for High Performance RISC-V Hardware</a>.</p>
<p>In the near future, more PRs will be submitted upstream. To facilitate subsequent development, we would like to apply to join the PyTorch Slack channel. If possible, please add my domain, <code>zhangfei@iscas.ac.cn</code>, to the PyTorch Slack. Thank you very much!</p>
rtx5070running out of memory

lizhi_bi: <p>My device is Ubuntu 24 and RTX 5070, I’m using the latest PyTorch 2.7 with CUDA 12.8. PyCharm keeps showing ‘running out of memory’ warnings. I’ve adjusted the XMX setting to a larger value, but PyCharm’s memory usage keeps increasing, reaching up to 8GB before it freezes. I use a Windows device to remotely access Ubuntu’s PyTorch through SSH, but still encounter this issue. When using an environment without this version installed, the problem doesn’t occur</p>

ptrblck: <p>I assume you are running out of memory on the host so would recommend trying to isolate if this memory usage is expected. I also don’t understand the last statement entirely since you won’t be able to use your GPU at all without a PyTorch build with CUDA 12.8.</p>

lizhi_bi: <p>What I mean is that when running the same code on my Windows laptop GPU, this issue doesn’t occur. My laptop GPU is a GTX 1650, which only requires 2.5GB of memory to run smoothly.However, when using PyTorch 2.7 version on Ubuntu, the memory usage rapidly increases. Even when not running code, just editing code causes memory to grow. The CPU usage also remains consistently high, as if it’s indexing a large number of files. I’ve examined the log files and found that many dynamic link references have failed. I’ve also tried disabling all plugins, but it had no effect.</p>

ptrblck: <p>This sounds like a issues caused in your IDE. Do you see any of these issues when you are executing your script in a terminal?</p>

lizhi_bi: <p>No, I don’t have this issue when using the terminal. But I don’t know what’s wrong with the IDE.</p>
Download models for sport event detection

brainartfu1010: <p>Hello everyone, I hope you’re doing well!</p>
<p>I’m currently working on a Python (Flask) project where I need to implement <strong>AI-based event detection</strong> for sports videos (specifically <strong>soccer</strong> and <strong>basketball</strong>).</p>
<p>The main goal is to <strong>detect events</strong> such as:</p>
<ul>
<li>Goal</li>
<li>Yellow Card (YC)</li>
<li>Red Card (RC)</li>
<li>Pass</li>
<li>Dribble</li>
<li>(and other key events)</li>
</ul>
<p>Most of the project backend is already implemented.<br>
Now I am <strong>ready to integrate</strong> the pretrained model into the Python code to finish the project as soon as possible.</p>
<p>If anyone has:</p>
<ul>
<li>Pretrained models</li>
<li>Good recommendations</li>
<li>Sample code or resources</li>
</ul>
<p>Please <strong>get in touch</strong> — I would really appreciate your help!</p>
<p>Thank you so much in advance <img src="https://discuss.pytorch.org/images/emoji/apple/folded_hands.png?v=14" title=":folded_hands:" class="emoji" alt=":folded_hands:" loading="lazy" width="20" height="20"></p>
NVIDIA RTX 5070 Oobabooga Error

MikeRo128: <p>I can load a model, but as soon as I ask a question this is the error</p>
<p>RuntimeError: CUDA error: no kernel image is available for execution on the device<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>Name: torch<br>
Version: 2.6.0.dev20241112+cu121<br>
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration<br>
Home-page: <a href="https://pytorch.org/" rel="noopener nofollow ugc">https://pytorch.org/</a><br>
Author: PyTorch Team<br>
Author-email: <a href="mailto:packages@pytorch.org">packages@pytorch.org</a><br>
License: BSD-3-Clause</p>

ptrblck: <p>Install a PyTorch binary with CUDA 12.8 and it will work.</p>

MikeRo128: <p>Thank you for your help. Unfortunately I am getting the same error   Here is what I installed  pip install --pre torch==2.8.0.dev20250424+cu128 torchvision==0.22.0.dev20250425+cu128 torchaudio==2.6.0dev20250425+cu128 --index-url <a href="https://download.pytorch.org/whl/nightly/cu128" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu128</a></p>
<p>I also dowloaded <a href="https://download.pytorch.org/whl/nightly/cu128/torch-2.8.0.dev20250425%2Bcu128-cp312-cp312-win_amd64.whl" rel="noopener nofollow ugc">torch-2.8.0.dev20250425+cu128-cp312-cp312-win_amd64.whl</a> and installed and it did not work as well</p>

ptrblck: <p>In that case another 3rd party library might not support your Blackwell GPU and you would need to isolate which call fails.</p>
torch.load with weights_only=True RCE

south-ocean: <p>A security vulnerability has been discovered in PyTorch where <code>torch.load</code> with <code>weights_only=True</code> can still lead to remote code execution (RCE). It is stated that this issue has been fixed in version 2.6 and above.</p>
<p>I’d like to ask whether this fix was applied automatically or through a specific commit. If it was fixed via a commit, could you point me to the exact one? Additionally, is it possible for us to backport this fix to PyTorch 2.5 or 2.4?</p>
<p>Looking forward to your response. Thank you!</p>

ptrblck: <p>Assuming a corresponding CVE was created it should also point to the commit fixing the issue. Do you have the CVE ID and can post it?</p>

approxim8r: <p>The 2.6 changelog links to a forum post where they mention the change that addresses this vulnerability, which links to <a href="https://github.com/pytorch/pytorch/pull/137602" rel="noopener nofollow ugc">this PR</a>.</p>
<p>On first glance the change “flips the default on <code>weights_only</code>” (paraphrasing the PR title) for some cases and in other cases seems to force users to explicitly allow loading serialized objects via <code>safe_globals</code> (do verify this yourself). Assuming you’re talking about <a href="https://github.com/pytorch/pytorch/security/advisories/GHSA-53q9-r3pm-6pq6" rel="noopener nofollow ugc">this vulnerability</a>, I don’t know yet how it was fixed since the above says that RCEs can occur <em>despite</em> setting <code>weights_only=True</code>.</p>

approxim8r: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="219375">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>CVE</p>
</blockquote>
</aside>
<p>OP could be talking about this vulnerability from the PyTorch repo’s Security tab <a href="https://github.com/pytorch/pytorch/security/advisories/GHSA-53q9-r3pm-6pq6" rel="noopener nofollow ugc">here</a> from a week ago. The listed CVE ID is CVE-2025-32434. <a href="https://www.cve.org/CVERecord?id=CVE-2025-32434" rel="noopener nofollow ugc">Here</a> it is.</p>

ptrblck: <p>Thanks! The CVE links to <a href="https://github.com/pytorch/pytorch/commit/8d4b8a920a2172523deb95bf20e8e52d50649c04">this PR</a> which seems to fix this. CC <a class="mention" href="/u/south-ocean">@south-ocean</a></p>

approxim8r: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks! Could you please tell me where you found the PR corresponding to it? I couldn’t find it on the CVE record page.</p>
Different results for the same training model trained on different devices

Omaralmaqtari: <p>We are training a network on medical images for a segmentation task. We ran the same training on different devices and got different results. One device gave us 83%, and the other gave us 91.5%, a difference of 8.5%. Details about the devices’ specs and environments are below:</p>
<p>First device:<br>
System: Intel Core i9-7920X CPU, NVIDIA Titan V GPU, Linux-x86_64, Ubuntu 18.0.4.6 LTS, NVIDIA Driver Version 525.105.17.<br>
Libs: spyder 5.5.1, python 3.12.7, pytorch 2.5.1, pytorch-cuda 12.1, torchvision 0.20.1, monai 1.4.0, numpy 1.26.4, wandb 0.19.4, einops 0.8.0, matplotlib 3.9.2, albumentations 0.0.10, scipy 1.13.1, scikit-learn 1.5.1, timm 1.0.14</p>
<p>Second device:<br>
System: i7-14700K CPU, RTX 4070 TI super GPU, Windows 11 pro, Version: 10.0.22631.</p>
<p>Libs: albumentations 2.0.5, einops 0.8.1, logging 0.4.9.6, matplotlib 3.10.1, monai 1.4.0, numpy 1.26.4, python 3.11.7, scikit-learn 1.6.1, scipy 1.15.2, timm 1.0.15, torch 2.6.0+cu118, torchvision 0.21.0, wandb 0.19.10</p>
Torchcodec stich new frames

yassineAlouini: <p>Let’s say I have a pytorch model that produces a tensor of size (T, C, H, W) where T is the number of frames in a video clip.<br>
Is there an efficient way to paste back this tensor into the original video? Let’s say I want to stitch it back at a given position that might change over the frames T?<br>
Any suggestions are welcome, thanks!</p>
Custom mnist-like dataset

dmitriys: <p>Hello!<br>
I want to train LeNet in libtorch with using c++ api.<br>
I created my own train-images-idx3-ubyte and idx1 from printed digits.<br>
I load it as following:</p>
<pre><code class="lang-auto">    auto train_data_set = torch::data::datasets::MNIST(mnist_dataset_path, torch::data::datasets::MNIST::Mode::kTrain)
                              .map(torch::data::transforms::Normalize&lt;&gt;(0.5, 0.5))
                              .map(torch::data::transforms::Stack&lt;&gt;());
</code></pre>
<p>Here, when i try to load it, it says:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'c10::Error'
  what():  Expected to read number 60000 but found 1930 instead
Exception raised from expect_int32 at /root/pytorch/torch/csrc/api/src/data/datasets/mnist.cpp:49 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0x6c (0x7c49350c29ac in /lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;) + 0xfa (0x7c4935089008 in /lib/libc10.so)
frame #2: &lt;unknown function&gt; + 0x4e13a99 (0x7c4931a13a99 in /lib/libtorch_cpu.so)
frame #3: &lt;unknown function&gt; + 0x4e13e82 (0x7c4931a13e82 in /lib/libtorch_cpu.so)
frame #4: torch::data::datasets::MNIST::MNIST(std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, torch::data::datasets::MNIST::Mode) + 0x4a (0x7c4931a159ca in /lib/libtorch_cpu.so)
frame #5: main + 0x54a (0x5b947b41f2f9 in /home/dmitriys/projects/ray/MNIST-cls-cpp/build/train)
frame #6: &lt;unknown function&gt; + 0x2a1ca (0x7c492c42a1ca in /lib/x86_64-linux-gnu/libc.so.6)
frame #7: __libc_start_main + 0x8b (0x7c492c42a28b in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: _start + 0x25 (0x5b947b41ea95 in /home/dmitriys/projects/ray/MNIST-cls-cpp/build/train)
</code></pre>
<p>How can I use my own dataset which has 28x28 images (printed digits), MNIST like?</p>

Fitanium: <p>The predefined MNIST dataset class in libtorch uses the full MNIST dataset, i.e. 60000 samples in training mode, which is hard-coded in the libtorch library. I think Libtorch just provides a reference example to allow CPP programmers to define their own datasets according to their own problems. <a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/src/data/datasets/mnist.cpp" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/src/data/datasets/mnist.cpp</a><br>
If you want to use your own custom dataset, I think the best way is to define a class similar to the MNIST class. Or read the file directly and convert the data into tensors, then use the data loader and blah, blah。</p>
Qint8 Activations in PyTorch

xiechengyan: <p>Help please!<br>
I am currently working on a project where I need to deploy a PyTorch-based neural network model onto an FPGA for inference. My goal is to quantize the model so that both the activations and weights are within the range of -128 to 127 (8-bit precision)</p>
<p>So far, I have experimented with both <code>fbgemm</code> and <code>qnnpack</code> for quantization:</p>
<ol>
<li><strong>Activation Quantization</strong>: I encountered an issue where I was unable to convert the activations to integer values in the range of -128 to 127. When I doing the inference on my computer, it say the <code>conv</code> layers do not support <code>qint8</code> as input.</li>
<li><strong>QNNPACK Compatibility</strong>: I also tried using QNNPACK for inference, but faced a limitation where QNNPACK does not support the <code>conv3d_unpack</code> operation.</li>
</ol>
<p>Does PyTorch itself offer any features that could support the qint8 activations?<br>
Or would I need to explore other frameworks or toolchains?</p>

jerryzh168: <ol>
<li>yeah activation only supports quint8</li>
<li>I see, it’s possible that qnnpack does not support conv3d, but we are moving away from it</li>
</ol>
<p>here is our new stack: <a href="https://pytorch.org/docs/main/quantization.html#prototype-pytorch-2-export-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch main documentation</a> please take a look at the tutorials attached in the end, this allows for extensions to any backends. available in 2.4 and later I think, we are also moving this to torchao: <a href="https://github.com/pytorch/ao" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/ao: PyTorch native quantization and sparsity for training and inference</a> our single repository for pytorch native quantization and sparsity</p>
Can Anyone help me with this!(Weighted Loss)

Abas1: <p>Im trying to use weighted classes within the CrossEntropy loss function and there is no clear cut method to use in pytorch. Currently im using method 1 however this penalizes the loss by a greater amount than the second method. The class weights for method 1 are a lot higher for minority cases which means parameters get a more aggressive update and this potentially distorts the pretrained weights I assume . which is the standard in pytorch</p>
<p>Method 1: total/(samples_per_class × num_classes)</p>
<p>Method 2 : 1/samples per class</p>
<pre><code class="lang-auto">class_counts = train_df['level'].value_counts().sort_index().tolist()

total = len(train_df)
weights = [total/(len(class_counts)*c) for c in class_counts]
weights = torch.tensor(weights, dtype=torch.float32)
weights = weights.to(device)

"""
below is the current training distribution for prod, which we use to generate class weights
0    18067
2     3704
1     1710
3      611
4      496

"""
print(weights)
</code></pre>
<p>and I get a tensor of tensor([0.2722, 2.8754, 1.3271, 8.0607, 9.9333], device=‘cuda:0’)</p>

KFrank: <p>Hi Abas!</p>
<aside class="quote no-group" data-username="Abas1" data-post="1" data-topic="219457" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/f9ae1b/48.png" class="avatar"> Abas1:</div>
<blockquote>
<p>Method 1: total/(samples_per_class × num_classes)</p>
<p>Method 2 : 1/samples per class</p>
</blockquote>
</aside>
<p>As I read this, “total” and “num_classes” don’t depend on the class.  Therefore the<br>
two sets of weights differ only by a class-independent multiplicative factor and the<br>
<em>relative</em> weights of the classes will be the same.</p>
<p>For common optimizers, this overall factor can be compensated for by dividing the<br>
learning rate by that multiplicative factor – which is a fancy way of saying that the<br>
particular choice doesn’t really matter.</p>
<p>As an aside, I would typically use “total / samples_per_class.”  Using this version<br>
prevents the overall scale factor of the weights from changing should you change<br>
the size of your training set.</p>
<p>Best.</p>
<p>K. Frank</p>
Advice for training deeper networks on very small datasets

adrinorosario: <p>I am trying to build a model with fewer parameters that generalizes well on the flower102 dataset, a very small one in today’s standards. I aim to arrive at an architecture that is smaller, efficient, with fewer parameters, without overfitting. I have read a few papers, such as the EfficientNet, SqueezeNet, and GoogleNet papers. Any advice on how I can approach this research problem?</p>

Eduardo_Lawson: <p>Maybe pre-training in bigger datasets can do the trick, and add several data augmentation to your goal dataset.</p>

adrinorosario: <p>I am currently trying to pretrain the model on the Cifar100 dataset. But then the images in Cifar100 are 32x32 but the flower102 has images of 224x224. Should the goal dataset resolutions also be decreased to 32x32?</p>

Eduardo_Lawson: <p>I don’t se a problem, you just have to be certain that your model can handle the different image size.</p>
Compiled matmul is slower than vanilla matmul

Meng_Zhu: <p>Hi, I wonder why, in my test, compiled torch.matmul is much slower than vanilla torch matmul (&gt;25% difference)? I understand torch.matmul uses cublas and torch compile matmul uses triton via inductor. But I would expect the vanilla cubals is a candidate of the autotune? And hence compiled matmul should at least as fast as the vanilla matmul. Is my understanding correct?</p>
<pre data-code-wrap="python"><code class="lang-python">            time_taken = timeit(
                torch.matmul, A, B, out=C
            )
</code></pre>
<p>vs</p>
<pre data-code-wrap="python"><code class="lang-python">            def matmul_wrapper(A, B, out):
                return torch.matmul(A, B, out=out)
            compiled_matmul = torch.compile(matmul_wrapper, fullgraph=True, mode="max-autotune-no-cudagraphs")
            # warmup
            compiled_matmul(A, B, out=C)
            time_taken = timeit(
                compiled_matmul, A, B, out=C
            )
</code></pre>
<p>env: torch 2.7 on cuda 12.8, B200</p>

ptrblck: <aside class="quote no-group" data-username="Meng_Zhu" data-post="1" data-topic="219405">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/meng_zhu/48/76765_2.png" class="avatar"> Meng_Zhu:</div>
<blockquote>
<p>But I would expect the vanilla cubals is a candidate of the autotune? And hence compiled matmul should at least as fast as the vanilla matmul. Is my understanding correct?</p>
</blockquote>
</aside>
<p>I don’t think the default <code>aten</code> backend (which calls into cuBLAS) will be used unless no choices are available and <code>aten</code> is used as the fallback as seen <a href="https://github.com/pytorch/pytorch/blob/5e320eea665f773b78f6d3bfdbb1898b8e09e051/torch/_inductor/kernel/mm.py#L711C8-L712">here</a> and <a href="https://github.com/pytorch/pytorch/blob/5e320eea665f773b78f6d3bfdbb1898b8e09e051/torch/_inductor/kernel/mm_common.py#L21-L37">here</a>.</p>
<p>CC <a class="mention" href="/u/yf225">@yf225</a> to correct me</p>
Handle important gaps in TimeSeriesDataSet

Eyzzle: <p>Hello everyone,</p>
<p>I am currently working on time series forecasting. I have several dataset that I can work with and I would like to combine them into one whole dataset to improve my model training and therefore my predictions. The issue is that my <strong>datasets are not continuous in time</strong>:</p>
<p>My datasets consist of 1-day values over a period of several years. When joining datasets, I could therefore have values ending in 2014 for one dataset and then starting again in 2018 for another dataset. Obviously <strong>I can not fill the data in this gap for my whole dataset to be continuous</strong>.<br>
Additionally, in some of the dataset, I have some large missing values (for several weeks) which again, can not be filled automatically.</p>
<p>I would like to take advantage of the TimeSeriesDataSet class but <strong>I need to allow gaps in my data</strong>. The main advantage of this class is that the handling of multiple covariates is already implemented, aswell as the <code>from_dataset()</code> method when building a NN architecture.</p>
<p>From what I’ve seen <a href="https://github.com/sktime/pytorch-forecasting/issues/1244" rel="noopener nofollow ugc">in this github discussion</a>, I could set weight to 0, but I am not sure if this is ideal.<br>
Ideally, I would like to be able to override the method to select the time windows: my prediction for the next day is based on the last 7 days, therefore I would like to select windows with continuous data for 7+1 = 8 days. Is there any “easy” way to do this ?</p>
<p>Apart from these two ideas, is there something more appropriate for my issue ? Sorry I am new to pytorch forecasting and could not find solutions when browsing for my problem.</p>
<p>Thank you!</p>
Switch loss function causes "RuntimeError: Found dtype Double but expected Float"

legendu: <p>I tried to fine tune the ResNe18 model on GPU. The training code runs OK. However, I decided to further train the produced model using a different loss function (switched from L1 loss to L2 loss) but got the the error below. No other code or data changed.</p>
<pre><code class="lang-auto">RuntimeError: Found dtype Double but expected Float
</code></pre>
<p>Any idea what might have caused the issue and how to fix it?</p>
<p>Below is the information of my environment.</p>
<pre><code class="lang-auto">OS:                     Ubuntu 20.04 in Docker
python                 3.8.10
torch                   1.9.1+cu111
torchaudio              0.9.1
torchvision             0.10.1+cu111
</code></pre>

legendu: <p>It doesn’t work even if call <code>loss = loss.to(torch.float32)</code> to convert loss to torch.float32 before backward propagation.</p>

isalirezag: <p>you shouldnt do it that way, you need to change the target/label to float, here is an example:</p>
<pre><code class="lang-auto">loss = nn.MSELoss()
input = torch.randn(3, 5, requires_grad=True)
target = torch.randn(3, 5)
output = loss(input, target.float())  # the floating can happen here 
output.backward()
</code></pre>

legendu: <p>What I’m confused is why does <code>l1_loss</code> work while <code>mse_loss</code> doesn’t work?</p>

isalirezag: <p>Im not sure, one reason can be probably because the backprop in the mse cannot handle double format since it is squared L2 norm, but for l1 it is not an issue.</p>

ptrblck: <p>It should not raise an issue in the backward, if the forward was successfully executed.<br>
As mentioned in the other thread: could you post an executable code snippet which we could use to reproduce and debug this issue, please?</p>

Joey_Huang: <p>I meet the same issue when I relace <code>nn.L1Loss</code> with <code>nn.MESLoss</code> in AMP training environment.</p>