akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.distributed.barrier occupies additional CUDA memory

yuan_wentao: <p>I am conducting a DDP parallel training task, and I only want to perform validation in the main process. I used <code>torch.distributed.barrier</code> to prevent non-main processes to avoid NCCL timeout. However, I found that torch.distributed.barrier consumes an additional approximately 4GB of GPU memory (from 93GB to 97GB). Can someone help explain why this happens?</p>
Good performance during evaluation, but poor performance during inference

Rafi_Darmawan: <p>Hi, newbie here.</p>
<p>I tried to pre-train and implement <a href="https://arxiv.org/pdf/2408.08459" rel="noopener nofollow ugc">JPEG-LM</a> from scratch using Llama-2. The code can be found <a href="https://www.kaggle.com/code/drmwnnrafi/pixelart-jpeg-lm-from-scratch?scriptVersionId=204976760" rel="noopener nofollow ugc">here</a>. During evaluation and training, it returns high accuracy, which might indicate overfitting. However, when I perform inference on the training data, the performance is poor (you can check in the code). Did I misimplement the Llama-2 code or the inference code?</p>
<p>You can check the WandB reports <a href="https://api.wandb.ai/links/justateam/mmhtrpdr" rel="noopener nofollow ugc">here</a>. I’m confused about what is happening.</p>
"RTX 2000 Ada Generation Laptop GPU" vs. 3000 version

KFrank: <p>Hello Forum!</p>
<p>What performance difference might I expect between the “RTX 2000 Ada<br>
Generation Laptop GPU” and the “RTX 3000 Ada Generation Laptop GPU”<br>
when training garden-variety pytorch models?</p>
<p>I am considering getting a laptop* and these two gpus are among the choices.<br>
(The rtx 500 and rtx 1000 are also in the mix, if people have thoughts on those.)</p>
<p>The 2000 and 3000 both have the same 8GB of memory as well as the same<br>
base clock speed.  The main relevant difference seems to be that the 3000<br>
has about 50% more “CUDA Processing Cores” (4608 vs. 3072).</p>
<p>More cores sounds good, but would they really get used?  How about thermal<br>
considerations?  Would one be able to saturate most of the cores on a laptop<br>
gpu for a long training run without getting thermally throttled?</p>
<p>I haven’t found any comparisons that address machine learning or pytorch.<br>
The comparisons I’ve seen concern rendering and gaming frame rates, a<br>
rather different use case.</p>
<p>Thanks for any information or advice.</p>
<p>*)  Specifically, I’m looking at the Lenovo ThinkPad P16v Gen 1 (AMD) or<br>
the Gen 2 (Intel).  Aside from the gen1 not supporting the rtx 3000, the main<br>
difference seems to be the gen 1 is based on the AMD “Ryzen 5 / 7 / 9 Pro”<br>
series of cpus, while the gen 2 is based on the Intel “Core Ultra 7 / 9” cpus.<br>
Regarding that, my thinking is that the specific cpu won’t have much effect<br>
on the speed of gpu-accelerated pytorch computations.  Is that fair?  Any<br>
more general input on the cpu differences would be appreciated.</p>
<p>Best.</p>
<p>K. Frank</p>
How to use the multiple local network connected NVIDIA GPUs for the image processing

Joseph_Dingess: <p>Hi, Pytorch community advisors and users!</p>
<p>First of all, I am really grateful to all of you who have read my topic.<br>
I am developing one project to upscale the image using new model, but the process time is about 0.2s using my NVIDIA GeoForce RTX 4060 Laptop GPUs.</p>
<p>The ideal processing time is less than 0.03s, so the current state is insufficient on my side.</p>
<p>Fortunately, we have multiple devices which is using same NVIDIA GPUs on the local network, but I don’t know how to use these devices.</p>
<p>Is it possible? I am using the torch library.</p>
<p>If someone has related this field, let me know how to do that.</p>
<p>Best regards.</p>
<p>Joseph</p>
Random in colab does not work

hatted: <p>I am using colab.</p>
<p>I set the following:</p>
<pre><code class="lang-auto">seed = 1

torch.manual_seed(seed)

torch.cuda.manual_seed(seed)

torch.cuda.manual_seed_all(seed) # if you are using multi-GPU.

np.random.seed(seed) # Numpy module.

random.seed(seed) # Python random module.

torch.manual_seed(seed)

torch.backends.cudnn.benchmark = False

torch.backends.cudnn.deterministic = True
</code></pre>
<p>If i random the second time, it gives me different result.</p>
<pre><code class="lang-auto">tensor_rand_un = torch.rand(3,5)
print("tensor_rand_un :", tensor_rand_un)

tensor_rand_un = torch.rand(3,5)
print("tensor_rand_un :", tensor_rand_un)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor_rand_un : tensor([[0.7576, 0.2793, 0.4031, 0.7347, 0.0293],
        [0.7999, 0.3971, 0.7544, 0.5695, 0.4388],
        [0.6387, 0.5247, 0.6826, 0.3051, 0.4635]])
tensor_rand_un : tensor([[0.4550, 0.5725, 0.4980, 0.9371, 0.6556],
        [0.3138, 0.1980, 0.4162, 0.2843, 0.3398],
        [0.5239, 0.7981, 0.7718, 0.0112, 0.8100]])
</code></pre>
<p>However, if I restart random again, then it gives me the same random again.</p>
<pre><code class="lang-auto">seed = 1
torch.manual_seed(seed)
tensor_rand_un = torch.rand(3,5)
print("tensor_rand_un :", tensor_rand_un)
</code></pre>
<pre><code class="lang-auto">tensor_rand_un : tensor([[0.7576, 0.2793, 0.4031, 0.7347, 0.0293],
        [0.7999, 0.3971, 0.7544, 0.5695, 0.4388],
        [0.6387, 0.5247, 0.6826, 0.3051, 0.4635]])
</code></pre>
<p>Why I have random number every time and why I need to reset the random?   how can I get the same random all the time in colab?</p>

ptrblck: <aside class="quote no-group" data-username="hatted" data-post="1" data-topic="212447">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hatted/48/73734_2.png" class="avatar"> hatted:</div>
<blockquote>
<p>If i random the second time, it gives me different result.</p>
</blockquote>
</aside>
<p>This is expected since the seed will define the <em>sequence</em> the pseudorandom number generator will generate. Seeding the code will not force the PRNG to return the <em>same</em> values for all calls.</p>
<aside class="quote no-group" data-username="hatted" data-post="1" data-topic="212447">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hatted/48/73734_2.png" class="avatar"> hatted:</div>
<blockquote>
<p>how can I get the same random all the time in colab?</p>
</blockquote>
</aside>
<p>You could either re-seed the code after each call into the PRNG or just use a constant. Keep in mind that your values are not random at all anymore since you are sampling only once and would then reuse exactly the same constant values.</p>
Extremely slow PyTorch install

Zysora: <p>Hello, I am trying to install older versions of PyTorch and apparently, the pip install wheels do not work on anaconda, so I have to use conda install. However, the download is extremely slow when installing PyTorch. Is there a faster way to do this or somehow obtain a wheel of the previous versions? (1.0.0 and so)</p>

Zysora: <p>EDIT: conda install pytorch==1.0.0 torchvision==0.2.1 -c pytorch failed with this error. This is very frustrating.</p>
<p>pytorch-1.0.0        | 605.7 MB  | ##########################################################################<span class="hashtag">#9</span> | 100% WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename C:\Users\User\anaconda3\pkgs\pytorch-1.0.0-py3.7_cuda90_cudnn7_1\Lib\site-packages\torch\lib\libprotobuf.lib.  Please remove this file manually (you may need to reboot to free file handles)<br>
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(143): Could not remove or rename C:\Users\User\anaconda3\pkgs\pytorch-1.0.0-py3.7_cuda90_cudnn7_1\Lib\site-packages\torch\lib\libprotobuf.lib.  Please remove this file manually (you may need to reboot to free file handles)</p>
<p>InvalidArchiveError(‘Error with archive C:\Users\User\anaconda3\pkgs\pytorch-1.0.0-py3.7_cuda90_cudnn7_1.tar.bz2.  You probably need to delete and re-download or re-create this file.  Message from libarchive was:\n\nCould not unlink’)</p>

ptrblck: <p>Try to create a new virtual environment and reinstall the binaries there. The PyTorch 1.0.0 pip wheel can be found <a href="https://download.pytorch.org/whl/cu100/torch_stable.html">here</a>.<br>
In any case, I would recommend to update to the latest PyTorch version, if possible, to get the latest features and fixes.</p>

fawaz: <p>his is bcz u used pip and also conda , also sometimes it is bcz u install a different version. however, this issue is due to conda-package-handling<br>
JUST write this command<br>
conda update -n base conda-package-handling</p>

Ido_Bistry: <p>Hi friends - Just ran into same issue after reinstalling my computer<br>
and need venv reinstall again</p>
<p>I manages to excel it by changing torch to the end of the list in the requirements.txt<br>
make it install after all others in the wheel<br>
and it might fly :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c2acd07f8ab60094c83720173e5493f6488ed54.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c2acd07f8ab60094c83720173e5493f6488ed54" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c2acd07f8ab60094c83720173e5493f6488ed54.png" alt="image" data-base62-sha1="mhwbwBxW9K6oXO6fIVvC0InpTdG" width="690" height="427" data-dominant-color="212322"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">819×507 33.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Flex attention benchmarking

Lakshya_LNU: <p>I’m seeing great variance in running flex attention, some of the runs take way too much time. This is beyond the first one or two iterations where torch.compile requires time to compile.</p>
<pre><code class="lang-auto">import torch
from torch.nn.attention.flex_attention import create_block_mask, flex_attention
flex_attention = torch.compile(flex_attention, dynamic=True)
from torch.utils.benchmark import Timer

def run_timer_timeit(fn, _len=10):
    t = Timer(stmt="fn()", globals={"fn": fn})
    print(t.timeit(_len)) 

def run_block_mask_flex_attn():
    flex_attention(q, k, v, block_mask=block_mask).sum().backward()

def run_xformers():
    xops.memory_efficient_attention(q, k, v).sum().backward()

device="0"
S=20000
q = torch.randn(1, 32, S, 128, requires_grad=True, device='cuda:'+device)
k = torch.randn(1, 32, S, 128, requires_grad=True, device='cuda:'+device)
v = torch.randn(1, 32, S, 128, requires_grad=True, device='cuda:'+device)

run_timer_timeit(run_full_flex_attn, 100) # 4.66s
run_timer_timeit(run_xformers, 100) # 0.007 s

</code></pre>
<p>Result from flex attention seem extremely slow, however, while it comes from flex attention compiling, I tried running the same function multiple time, just in case the first run had the compiling run. I think most of the cases of the flex attention decently fast, but some of them might be getting stuck,<br>
when I ran it manually</p>
<pre><code class="lang-auto">start = time.time()
run_full_flex_attn()
end = time.time()
print("Full Flex Attention time: ", end-start)
</code></pre>
<p>Then, I see 0.002sec.</p>
<p>So, I’m not sure, why sometimes the flex attention run gets stuck when the compilation should have happened only once during initial runs.</p>
Efficent way of passing same input through many models

Tyzcen: <p>I want to pass the same input vector through a number of modules. These modules all have the same architecture and simply vary in their weights.<br>
Currently i am doing it like this:</p>
<pre><code class="lang-auto">    def forward(self, modules):
        embeddings = []
        for module in modules:
            embeddings.append(module(self.in_states).flatten())

        return torch.vstack(embeddings)
</code></pre>
<p>This is working fine in principal, but is very slow as i do not make use of any parallelization.<br>
I have only one machine with one GPU.<br>
Does somebody know a more efficient was of doing this?</p>

eqy: <p>Often the most efficient way to use a GPU is to leverage a single kernel at a time doing a lot of work. If the utilization of your GPU is high with this approach, then there likely isn’t a way to speed things up (trying to parallelize things by running multiple kernels in parallel would likely slow things down with the increased contention). If the utilization is low, you might see if you can increase the amount of data parallelism (e.g., batch size).</p>

Tyzcen: <p>In case anybody ever stumbles upon a similar task. I found a way, which is pretty specific to what i need.<br>
Basically as the nets i am using are all fully connected sequential Networks, i can manually execute the forwards passes via batched matrix multiplications, which can take use of cudas parallelization again.</p>
<p>here a snippet of my code:</p>
<pre><code class="lang-auto">    def forward(self, named_parameters):
        batch_size = named_parameters[self.weight_keys[0]].shape[0]
        forward_values = self.in_states.repeat(batch_size,1,1)

        for i in range(len(self.weight_keys)):
            linear_transformed = forward_values @ torch.transpose(named_parameters[self.weight_keys[i]], dim0=1, dim1=2)
            forward_values = linear_transformed + named_parameters[self.bias_keys[i]].unsqueeze(1)
            if(self.num_activations &gt; i):
                forward_values = self.activations[i](forward_values)

        return forward_values.reshape(batch_size, -1)
</code></pre>
<p>named_parameters here are the named parameters of the modules i want to pass the same input (in_states) through. They are in a special format where within each dictionary entry there is a whole batch of this specific layer parameters. (here the batch refers to number of modules).<br>
self.weigh_keys and self.bias_keys contain the keys of weights and bias parameters in the correct order.<br>
And self.activations the activations of the module.<br>
As said this is very specific and only works if all models have the same architecture and are of a fully connected sequential type.<br>
This drastically increased execution speed (when using cuda) over the implementation i showed in my question.</p>

thanrl: <p>Is there a better way to do this in 2024, preferably not having to rewrite the forward pass? My models are not FFNs; they are transformers.</p>
How should I organize my dataset to use it as training data in PyTorch?

VictorVidigal: <p>Hi everyone,</p>
<p>I would appreciate some tips on organizing my dataset or guidance on using it as input for my model.</p>
<p>The dataset contains intraday stock values, and I want to extract input and output data for model training as shown in the figure:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/e/feb7e05d31053c71f6dcdf6c4b77d0e706454513.png" data-download-href="https://discuss.pytorch.org/uploads/default/feb7e05d31053c71f6dcdf6c4b77d0e706454513" title="pytorch dúvida"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/feb7e05d31053c71f6dcdf6c4b77d0e706454513_2_689x285.png" alt="pytorch dúvida" data-base62-sha1="AllejCCCLJNtQVOYGWH2xBpK319" width="689" height="285" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/feb7e05d31053c71f6dcdf6c4b77d0e706454513_2_689x285.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/feb7e05d31053c71f6dcdf6c4b77d0e706454513_2_1033x427.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/e/feb7e05d31053c71f6dcdf6c4b77d0e706454513.png 2x" data-dominant-color="DCDCDC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch dúvida</span><span class="informations">1292×534 81.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do I need to reorganize my data to use it as a parameter for my model? If so, what should the data format look like?<br>
If reorganizing isn’t necessary, how can I load this data as PyTorch tensors to pass it to my model?</p>

ptrblck: <p>Based on the screenshot you could try to load and process the data with e.g. <code>pandas</code>. Once done you could create PyTorch tensors by reusing the numpy arrays used in <code>pandas</code> via <code>torch.from_numpy</code>.<br>
Depending on your use case, the standard inputs to a model would use a floating point format or integers if you want to use e.g. an <code>nn.Embedding</code> layer first.</p>
Installation Issue with PyTorch

enes_bajrami: <p>I am currently trying to install PyTorch and have encountered an issue. Despite my efforts to find a solution online, I have been unable to resolve it.</p>
<p>Here is the output from my command line when I attempt to install:</p>
<pre><code class="lang-auto">Python 3.13.0
pip install torch torchvision
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
ERROR: No matching distribution found for torch
</code></pre>
<p>Any help would be greatly appreciated.</p>
<p>Thank you!</p>

ptrblck: <aside class="quote" data-post="3" data-topic="212112">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/unable-to-install-pytorch-on-python-3-13/212112/3">Unable to install Pytorch on Python 3.13</a> 
  </div>
  <blockquote>
    The Python 3.13 support is tracked <a href="https://github.com/pytorch/pytorch/issues/130249">here</a> and while Windows supports source builds, the PyTorch binaries do not support it yet according to the support matrix.
  </blockquote>
</aside>
Libtorch batch jacobians and hessians

dlittlewood: <p>Are there any updates on implementing torch.autograd.functional.jacobian to the C++ API?</p>
<p>I’ve come across many threads asking about this but can’t seem to find any updates regarding if this is still being implemented or not. Using torch::autograd::grad in a loop is painfully slow.</p>
<p>Thanks for any information you can provide.</p>
Parameter loading

yichao_Liu: <p>Hi, I’m facing a weird problem. Here I use the ultralytics model to run object detection, it gives reasonable results with predicted bonding box. Later I rebuilt the model without reading that yaml file, because it’s random weights, it gives bonding box with 0 value inside. And then the weird thing happens, when I load parameters from the pretrained ultralytics model to the rebuilt model, it gives empty bounding box. And I randomly checked the maximum loaded parameters of the rebuilt model and the pretrained one, it’s same. I don’t know what happened, but I’m sure it the problem of parameter transfer problem. Here’s the loading code I used. Hope anyone can help me. Thank you so much, I really appreciate it.<br>
rebuilt_model.conv1.load_state_dict(pretrain_model.model._modules[‘model’][0].state_dict())<br>
rebuilt_model.conv2.load_state_dict(pretrain_model.model._modules[‘model’][1].state_dict())<br>
rebuilt_model.c2f1.load_state_dict(pretrain_model.model._modules[‘model’][2].state_dict())<br>
rebuilt_model.conv3.load_state_dict(pretrain_model.model._modules[‘model’][3].state_dict())<br>
rebuilt_model.c2f2.load_state_dict(pretrain_model.model._modules[‘model’][4].state_dict())<br>
rebuilt_model.conv4.load_state_dict(pretrain_model.model._modules[‘model’][5].state_dict())<br>
rebuilt_model.c2f3.load_state_dict(pretrain_model.model._modules[‘model’][6].state_dict())<br>
…<br>
Sorry, later I tried remove detect layer parameter transfer code, and now it can give bounding box with 0 values, but I want the model outputs the bounding box same as pretrained model, can someone help me? Thank you so much again.</p>

ptrblck: <aside class="quote no-group" data-username="yichao_Liu" data-post="1" data-topic="212327">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yichao_liu/48/18684_2.png" class="avatar"> yichao_Liu:</div>
<blockquote>
<p>but I want the model outputs the bounding box same as pretrained model, can someone help me?</p>
</blockquote>
</aside>
<p>In your code snippet it seems you are loading <code>state_dict</code>s for a few specific layers not all modules. Could you explain why loading the “full” <code>state_dict</code> won’t work as I would speculate a few layers might be missed and are thus still randomly initialized.</p>

yichao_Liu: <p>Hi, thank you so much for your reply. Actually I load parameters block by block, I just show part of the blocks. I’ve loaded all the 22 blocks. Just the last block detection block is where the problem happens, I feel.<br>
rebuilt_model.detect.load_state_dict(pretrain_model.model._modules[‘model’][22].state_dict())</p>

ptrblck: <p>Could you explain why you feel this block fails? Do you see any errors os warnings when loading the parameters?</p>

yichao_Liu: <p>I’m so sorry, seems there’s block I mistakely put a shortcut. But state_dict function can’t detect the mismatch. Thank you for your quick and kind answer.</p>

ptrblck: <p>I’m unsure if I understand your response correctly, but is this difference in your model causing the issue?</p>
Loading images using file name as label

RunDomRun: <p>Hi, I’m new to machine learning and looking to load a dataset from a file structure that looks something like:</p>
<p>root<br>
-star<br>
–img1.png<br>
– img2.png<br>
– img3.png<br>
…</p>
<p>-planet<br>
–img20.png<br>
–img21.png<br>
…</p>
<p>Every tutorial I find uses a preconstructed dataset and isn’t very helpful, so I would appreciate a pointer</p>

Eduardo_Lawson: <p>Hi. I think something like this would do:</p>
<pre><code class="lang-auto">import os
from torchvision import datasets, transforms

def load_image_dataset(root_dir, image_size=224):
    """
    Load image dataset from a directory structure.
    
    Args:
        root_dir (str): Path to root directory containing class subdirectories
        image_size (int): Resize images to this dimension
    
    Returns:
        torchvision.datasets.ImageFolder: Loaded dataset
    """
    # Define image transformations (data augmentation)
    data_transforms = transforms.Compose([
        transforms.Resize((image_size, image_size)),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], 
                             std=[0.229, 0.224, 0.225])
    ])
    
    # Load dataset
    train_data = datasets.ImageFolder(
        root=root_dir, 
        transform=data_transforms
    )
    
    return train_data
</code></pre>
<p>Then a example of usage, should look like this:</p>
<pre><code class="lang-auto"># You can choose the image size you want
train_data = load_image_dataset('path/to/root/directory', image_size=512) 

</code></pre>
How to limit torch.compile to CPU only?

ducha-aiki: <p>Hi,</p>
<p>I am using torch.compile for optimizing some preprocessing components, which are run on CPU, like <code>depth_to_3d_opt  = torch.compile(kornia.geometry.depth_to_3d, mode="reduce-overhead")</code></p>
<p>Then I am trying to run multi-GPU training via <code>transformers</code> + <code>accelerate</code>.<br>
However, the torch.compile fails with CUDA-related error <code>torch._dynamo.exc.InternalTorchDynamoError: Cannot re-initialize CUDA in forked subprocess.</code></p>
<pre><code class="lang-auto">[rank3]:   File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 172, in _fn
[rank3]:     cuda_rng_state = torch.cuda.get_rng_state()
[rank3]:   File "/opt/conda/lib/python3.10/site-packages/torch/cuda/random.py", line 31, in get_rng_state
[rank3]:     _lazy_init()
[rank3]:   File "/opt/conda/lib/python3.10/site-packages/torch/cuda/__init__.py", line 300, in _lazy_init
[rank3]:     raise RuntimeError(
[rank3]: torch._dynamo.exc.InternalTorchDynamoError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method


[rank3]: You can suppress this exception and fall back to eager by setting:
[rank3]:     import torch._dynamo
[rank3]:     torch._dynamo.config.suppress_errors = True
</code></pre>
<p>Is there a way to force torch.compile to run in CPU-only mode?</p>

Chillee: <p>What version of pytorch is this?</p>

ducha-aiki: <p>Latest stable, 2.4.0</p>

zbh2047: <p>I faced exactly the same issue. Can anyone help? Thank you!</p>
DDP training get slower than first few iteration

GEOLU: <p>Now I am trying the scratch training of Swin transformer in torchvision models at <a href="https://github.com/pytorch/vision/blob/main/references/classification/train.py" rel="noopener nofollow ugc">here</a></p>
<p>Even though our environment is poor (i9-10980XE, RTX3090x4, 128GB RAM, Ubuntu 22.04, pytorch 2.4 cudatoolkit=12.1), the training progressively slow down and the GPU utilization also drop. CPU cores’ are also enough.</p>
<p>It first uses the GPU utils 100% but after a few iterations, it is reduced dramatically.</p>
<p>It does not maintain the training speed. Please give me some solutions.</p>
<p>I’ve tried num_worker increament, pin_memory True,</p>
<p>Starting GPU utils<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e96971f81fe3add1a25239274255bf7d36886808.png" alt="GPU_utils_at_first" data-base62-sha1="xiRahHY3wWzPmr6QoGlOyewl8Ao" width="628" height="229"></p>

ptrblck: <p>Profile your code with a visual profiler, e.g. Nsight Systems, to narrow down where bottlenecks in your code are.</p>

GEOLU: <p>The code is the same as the <a href="https://github.com/pytorch/vision/blob/main/references/classification/train.py" rel="noopener nofollow ugc">HERE</a></p>
<p>I just run it in my environment with this command line</p>
<pre><code class="lang-auto">torchrun --nproc_per_node=4 train.py --model swin_t --epochs 300 --batch-size 128 --opt adamw --lr 0.001 --weight-decay 0.05 --norm-weight-decay 0.0  --bias-weight-decay 0.0 --transformer-embedding-decay 0.0 --lr-scheduler cosineannealinglr --lr-min 0.00001 --lr-warmup-method linear  --lr-warmup-epochs 20 --lr-warmup-decay 0.01 --amp --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 5.0 --cutmix-alpha 1.0 --random-erase 0.25 --interpolation bicubic --auto-augment ta_wide --model-ema --ra-sampler --ra-reps 4 --val-resize-size 224 --output-dir './' --print-freq 256--workers 4
</code></pre>
<p>I ran the code with workers 0, 4, 8, and 16 with 4 GPUS (nproc-per-node=4) settings</p>
<p>However, all the experiments are slowed after a few iterations.</p>
Pytorch on Android App Open Source Code

hasan10242048: <p>The task is very simple. Android app will load a pytorch model and a data file and show the output in a textbox or any suitable format.</p>
<aside class="onebox githubrepo" data-onebox-src="https://github.com/pytorch/android-demo-app">
  <header class="source">

      <a href="https://github.com/pytorch/android-demo-app" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <div class="github-row" data-github-private-repo="false">
  <img width="690" height="344" src="https://opengraph.githubassets.com/41bf3794125cd9d365cbad22d871a419/pytorch/android-demo-app" class="thumbnail">

  <h3><a href="https://github.com/pytorch/android-demo-app" target="_blank" rel="noopener nofollow ugc">GitHub - pytorch/android-demo-app: PyTorch android examples of usage in applications</a></h3>

    <p><span class="github-repo-description">PyTorch android examples of usage in applications</span></p>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I tried to follow the projects there but each time my app crashes.</p>
<p>I think this is a very simple task. Just upload models and data and show output in android app.<br>
I looked at many github repositories but most of them are 2 years ago at least.</p>
<p>Is there any <strong>recent</strong> Open Source Code available that anyone can share.</p>

Mert_Oguzhan: <p>In recent android studio version, you can use your pytorch (.pt) model into your application. But to do that you have to convert your pytorch model to onnx which enables us to convert framework models to other framework model and then convert your onnx model to TensorFlow Lite model.<br>
1- convert pytorch model to onnx<br>
2- convert onnx model to tflite model<br>
3- And add tflite model into your app<br>
android docs is here : <a href="https://developer.android.com/codelabs/digit-classifier-tflite" class="inline-onebox" rel="noopener nofollow ugc">Build a handwritten digit classifier app with TensorFlow Lite &nbsp;|&nbsp; Android Developers</a></p>

hasan10242048: <p>Do you know any resource where they have implemented the full pipeline?<br>
It can be a very simple app. But from pytorch model to android app.<br>
Thanks.</p>

Mert_Oguzhan: <p>What do you mean by saying “implemented full pipeline” ? If I am not wrong, you want to use your pytorch model in your android app right?<br>
To do that you have to convert your pytorch model to onnx and then convert from onnx to Tflite. You can see many resources about that conversation If you google it.</p>
How to lower to target backend?

wilhelm: <p>Reading the following <a href="https://pytorch.org/docs/stable/quantization.html#prototype-pytorch-2-export-quantization" rel="noopener nofollow ugc">tutorial</a>:</p>
<pre><code class="lang-auto">import torch
from torch.ao.quantization.quantize_pt2e import prepare_pt2e
from torch._export import capture_pre_autograd_graph
from torch.ao.quantization.quantizer import (
    XNNPACKQuantizer,
    get_symmetric_quantization_config,
)

class M(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = torch.nn.Linear(5, 10)

   def forward(self, x):
       return self.linear(x)

# initialize a floating point model
float_model = M().eval()

# define calibration function
def calibrate(model, data_loader):
    model.eval()
    with torch.no_grad():
        for image, target in data_loader:
            model(image)

# Step 1. program capture
# NOTE: this API will be updated to torch.export API in the future, but the captured
# result should mostly stay the same
m = capture_pre_autograd_graph(m, *example_inputs)
# we get a model with aten ops

# Step 2. quantization
# backend developer will write their own Quantizer and expose methods to allow
# users to express how they
# want the model to be quantized
quantizer = XNNPACKQuantizer().set_global(get_symmetric_quantization_config())
# or prepare_qat_pt2e for Quantization Aware Training
m = prepare_pt2e(m, quantizer)

# run calibration
# calibrate(m, sample_inference_data)
m = convert_pt2e(m)

# Step 3. lowering
# lower to target backend
</code></pre>
<p>But the last step<br>
<code>#lower to target backend</code><br>
is not described anywhere.</p>
<p>How does this step look like?</p>

jerryzh168: <p>depends on where you want to run the model, currently we have server cpu (<a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq_x86_inductor.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch 2 Export Post Training Quantization with X86 Backend through Inductor — PyTorch Tutorials 2.2.0+cu121 documentation</a>) and edge cpu (<a href="https://pytorch.org/executorch/stable/tutorials/export-to-executorch-tutorial.html#lowering-to-edge-dialect" class="inline-onebox" rel="noopener nofollow ugc">Exporting to ExecuTorch Tutorial — ExecuTorch 0.4 documentation</a>)</p>

wilhelm: <p>What about a Raspberry Pi (or in general ARM) as a target device?</p>

jerryzh168: <p>you can checkout this: <a href="https://github.com/pytorch/executorch/blob/a6328f0a3af71626e559e4bb2918a4d53826a34a/backends/arm/quantizer/arm_quantizer.py#L256" class="inline-onebox" rel="noopener nofollow ugc">executorch/backends/arm/quantizer/arm_quantizer.py at a6328f0a3af71626e559e4bb2918a4d53826a34a · pytorch/executorch · GitHub</a></p>
Torch.backends.mps.is_available = false in M1 sillicon

sms: <p>I am using M1 Mac Studio and installed ComfyUI, but an error occurs.</p>
<p>ComfyUI (<a href="https://github.com/comfyanonymous/ComfyUI" class="inline-onebox" rel="noopener nofollow ugc">GitHub - comfyanonymous/ComfyUI: The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface.</a>)<br>
first step in Apple mac silicon installation,<br>
(<a href="https://developer.apple.com/metal/pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Accelerated PyTorch training on Mac - Metal - Apple Developer</a>)<br>
I followed all the contents of this page in order.</p>
<p>In the last verification step,<br>
torch.backends.mps.is_built() =&gt; True<br>
torch.backends.mps.is_available() =&gt; False<br>
“MPS device not found.” is printed.</p>
<hr>
<ol start="0">
<li>Requirements</li>
</ol>
<ul>
<li>Mac computers with Apple silicon or AMD GPUs =&gt; <strong>M1 mac studio</strong></li>
<li>macOS 12.3 or later  =&gt; <strong>12.5.1</strong></li>
<li>Python 3.7 or later  =&gt; <strong>3.11.9</strong>, I tried it the latest 3.12, same error occur.</li>
<li>Xcode command-line tools: xcode-select --install  =&gt; <strong>the latest version installed</strong></li>
</ul>
<ol>
<li>
<p>Set up - Apple silicon<br>
conda create -n comfy python=3.11.9<br>
(I tried it the latest 3.12, same error occur.)</p>
</li>
<li>
<p>Install<br>
Miniconda3-latest-MacOSX-arm64</p>
</li>
<li>
<p>Verify</p>
</li>
</ol>
<blockquote>
<blockquote>
<blockquote>
<p>print(platform.platform())<br>
macOS-12.5.1-arm64-arm-64bit</p>
</blockquote>
</blockquote>
</blockquote>
<blockquote>
<blockquote>
<blockquote>
<p>print(torch.<strong>version</strong>)<br>
2.5.0.dev20240818</p>
</blockquote>
</blockquote>
</blockquote>
<p>(comfy) Mac-Studio:~ sms$ conda --version<br>
conda 24.7.1</p>
<blockquote>
<blockquote>
<blockquote>
<p>print(torch.<strong>version</strong>)<br>
2.5.0.dev20240818</p>
</blockquote>
</blockquote>
</blockquote>
<blockquote>
<blockquote>
<blockquote>
<p>print(torch.backends.mps.is_built())<br>
<strong>True</strong></p>
</blockquote>
</blockquote>
</blockquote>
<blockquote>
<blockquote>
<blockquote>
<p>print(torch.backends.mps.is_available())<br>
<strong>False</strong></p>
</blockquote>
</blockquote>
</blockquote>
<p>“MPS device not found.”  is printed.</p>

sms: <p>All requirements and installation were fine, but<br>
After upgrading macOS from 12.5.1 to 14.6.1<br>
“tensor([1.], device=‘mps:0’)” is printed.</p>

faelhv: <p>Is there a way to run PyTorch 2.5.1 with MPS enabled without upgrading the MacOS?</p>
<p>More precisely, I have a MacBook (macOS-12.7.6-arm64-arm-64bit) but I’m still facing the following error:</p>
<pre data-code-wrap="bash"><code class="lang-bash">RuntimeError: The MPS backend is supported on MacOS 12.3+.
Current OS version can be queried using `sw_vers`.
</code></pre>
<p>Basically, I have a OS version satisfying PyTorch requirements, still, it does not find a MPS device.</p>
Affinity By Model

James_Pinkerton: <p>Hi. I have a C++ process with multiple models, for which I use multiple threads. I want each model to run in a separate core and set affinity to reflect that. In ONNX, I know there’s a way to set affinity for the particular session. However, switching to ONNX would involve a lot of overhead for me. I was hoping to be able to select affinity in torch for each model. Is there a way to do this? So far I see thread and affinity in torch is global and that seems like a problem</p>
Libtorch: torch::autograd::grad create_graph and retain_graph malfunctioning?

Rsmeets: <p>Dear all,</p>
<p>When using the torch::autograd::grad function and setting create_graph = True and retain_graph = True, I get a resulting gradient with require_gradient() = False, which should not be the case. The excerpt from my code looks like this (it is part of a bigger function with the input being a  <code>const torch::Tensor&amp; input</code> which is batched tensor of inputs and I loop over its elements with <code>i</code>, and <code>m_function</code> is just a function from a torch::Tensor to a torch::Tensor)</p>
<pre><code class="lang-auto">torch::Tensor tmp_in = input.index({ i });
torch::Tensor tmp_out = m_function(tmp_in).reshape({ -1 });
torch::Tensor tmp_grad_outputs = torch::ones_like(tmp_out);
std::cout &lt;&lt; "tmp_in.requires_grad() is: " &lt;&lt; tmp_in.requires_grad() &lt;&lt; "\n" &lt;&lt; std::endl;
std::cout &lt;&lt; "tmp_out.requires_grad() is: " &lt;&lt; tmp_out.requires_grad() &lt;&lt; "\n" &lt;&lt; std::endl;
torch::Tensor tmp_grad = torch::autograd::grad(
		/*output=*/{ tmp_out },
		/*input=*/{ tmp_in },
		/*grad_outputs=*/{ tmp_grad_outputs },
		/*retain_graph=*/true,
		/*create_graph=*/true)[0];
std::cout &lt;&lt; "tmp_grad.requires_grad() is: " &lt;&lt; tmp_grad.requires_grad() &lt;&lt; "\n" &lt;&lt; std::endl;
</code></pre>
<p>The final line prints false, while comparing to similar pytorch code, one should expect true. With it being false, it also crashes (understandably) when trying to compute a higher order derivative. Has anybody encountered this issue before?</p>
<p>Kind regards,<br>
Robin</p>
Parallelize tensor dot-products (different sizes)

Zen: <p>I have two lists of tensors, G1 and G2, such that G1[i] and G2[i] have the same (arbitrary) shape.<br>
I want to compute some kind of dot-product between chunks of these lists. First, I compute the dot-product between the pairs of tensors given by <code>zip(G1, G2)</code>, which I put into a list (or 1D tensor). Then, I want to split this list into chunks and perform a sum over each of these chunks.</p>
<p>My current code is:</p>
<pre><code class="lang-auto">device = torch.device("cuda")
group_sizes = [2, 5, 1, 6]    # for instance (if len(G1) = 2 + 5 + 1 + 6 = 14)
H = torch.tensor([torch.dot(g1.view(-1), g2.view(-1)) for g1, g2 in zip(G1, G2)], device = device)
H_split = H.split(group_sizes)
H_final = torch.tensor([h.sum() for h in H_split], device = device)
</code></pre>
<p>I believe that some sequential operations remain sub-optimal: the existing <code>torch.dot</code> operations can be performed in parallel.</p>
<p>Is there a way to parallelize this code? I believe there is no simple way to do so, but I would like to know at which level I would have to dive to do this (CUDA backend? …).</p>
How Adam optimizer works while using Pipeline Parallelism?

Aerithy: <p>The Adam Optimizer employs the gradient after a complete batch to compute the L2 and L1 norms. However, it is unclear how Adam can be used in pipeline parallelism, such as in Gpipe or 1F1B. The pipeline parallelism utilises simple SGD to upgrade each weight partition owned by each node, which requires the gradient corresponding to their owned weight.</p>
How to avoid torch.reshape with constant shape is optimized to 1 reshape by torch.onnx.export?

nebulae: <pre data-code-wrap="python"><code class="lang-python">if attention_mask is not None:
    if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):
        raise ValueError(
            f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}"
        )
    separate_mask = [torch.reshape(attention_mask, [-1, 1, 256, 4096]) for _ in range(self.num_heads)]
    attn_weights = [aw + mask for aw, mask in zip(attn_weights, separate_mask)]
</code></pre>
<p>When I use <code>torch.onnx.export</code>, the reshape ops in all decoder layers is optimized to only 1 reshape. However, this is not the behavior I want.<br>
The target shape of reshape cannot be written as a variable shape for some reason, but has to be a constant shape.<br>
I have tried <code>do_constant_folding=False</code> for <code>torch.onnx.export</code>, but it didn’t work.</p>
<h3><a name="p-458396-version-information-1" class="anchor" href="#p-458396-version-information-1"></a>Version information</h3>
<ul>
<li>Python 3.10</li>
<li>Pytorch 2.1.2</li>
<li>onnx 1.14.1</li>
</ul>
Libtorch: resize function

Riddick_Gao: <p>How can I resize a tensor to a smaller size in  libtorch?  such as {1, 3, 704, 704} -&gt; {1, 3, 224, 224}.  I have tried the tensor.resize_( {1, 3, 224, 224}) method. But I found that it just returned a small region(224x224) of original image. please help me . thanks.</p>

tom: <p>You probably want interpolation?</p>
<p>Best regards</p>
<p>Thomas</p>

Riddick_Gao: <p>Yes, I want to do the down sampling。Does libtorch have any down sampling method? thanks</p>

tom: <p>Yes, what is it that interpolate is lacking for your purpose?</p>
<pre><code class="lang-auto">a = torch.arange(1_486_848.).view(1, 3, 704, 704)
b = torch.nn.functional.interpolate(a, size=(224, 224))
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

Riddick_Gao: <p>thanks. But I am using libtorch with C++ API. I cannot find the interpolate fuction in libtorch.</p>

Riddick_Gao: <p>I find the torch::grid_sampler() in libtorch. And it works!  Thanks for your help。 ：）</p>

ash_gamma: <p>Hi,</p>
<p>could you share the code snippet for grid_sampler?</p>
<p>Thanks</p>

yasudakn: <p>I was in trouble because I couldn’t find an API when I wanted to interpolate with C ++. I was more intuitive and easy to use upsample_bilinear2d in API library near interpolate.</p>
<pre><code class="lang-auto">at::Tensor masks_a = at::upsample_bilinear2d(masks.unsqueeze(0), {img_size, img_size}, false);
</code></pre>

ximitiejiang: <p>torch::nn::functional::interpolate() should be what you want.</p>

rbaleksandar: <p>For anyone still looking here is what I used:</p>
<pre><code class="lang-auto">auto tensor_out = torch::nn::functional::interpolate(
    tensor_in.unsqueeze(0),                       // Add batch dimension (N, C, H, W)
    torch::nn::functional::InterpolateFuncOptions()
    .size(std::vector&lt;int64_t&gt;({ 512, 512 }))
    .mode(torch::kBilinear)
    .align_corners(false)
).squeeze(0);
</code></pre>
<p>Each mode (here <code>torch::kBilinear</code>) requires a specific dimension of the input tensor. That is why I use the <code>unsqueeze(0)</code> since my tensor was <code>{3, 1024, 1024}</code> if I recall correctly (an RGB tensor basically).</p>
<p>The documentation of the C++ API is horrendous. I have spent at least 5 times more time finding (and often times not succeeding) the article than writing the actual code.</p>
How to align the resample result of librosa and torchaudio

wadewang: <p>The speed of torchauido resample attracts me after compare against librosa. But I found their results are a little different.<br>
I have read this documentation: <a href="https://pytorch.org/audio/2.4.0/tutorials/audio_resampling_tutorial.html#comparison-against-librosa" class="inline-onebox" rel="noopener nofollow ugc">Audio Resampling — Torchaudio 2.4.0 documentation</a> , but it just explained the resample method of kaiser, the default resample method of librosa is <strong>soxr_hq</strong>, so how to set the parameters of torchaudio to make its result align to that of  librosa with soxr_hq ?</p>
<p>This is my test code:</p>
<pre><code class="lang-auto">import time
import numpy as np
import soundfile as sf
import librosa
print(f"librosa version: {librosa.__version__}")
import scipy.io.wavfile as wavfile
import torchaudio
from pathlib import Path

# Define function to load wav and measure time
def measure_load_time(library_name, load_func, file_path, target_sr=16000):
    start_time = time.time()
    if library_name == "librosa":
        data, sr = load_func(file_path, sr=None)
        if sr != target_sr:
            print(f"librosa: Resampling from {sr} to {target_sr}")
            data = librosa.resample(data, orig_sr=sr, target_sr=target_sr)
            sr = target_sr
    elif library_name == "torchaudio":
        data, sr = load_func(file_path)
        if sr != target_sr:
            print(f"torchaudio: Resampling from {sr} to {target_sr}")
            data = torchaudio.functional.resample(data, orig_freq=sr, new_freq=target_sr)
            sr = target_sr
        data = data.numpy().squeeze()

    end_time = time.time()
    duration = end_time - start_time
    return data, sr, duration

# Load wav files and compare
file_path = r"test_task_123.wav"

# Measure times and compare outputs
results = {}

# Librosa
data_librosa, sr_librosa, time_librosa = measure_load_time("librosa", librosa.load, file_path)
results["librosa"] = {"data": data_librosa, "sr": sr_librosa, "time": time_librosa}

# Torchaudio
data_torchaudio, sr_torchaudio, time_torchaudio = measure_load_time("torchaudio", torchaudio.load, file_path)
results["torchaudio"] = {"data": data_torchaudio, "sr": sr_torchaudio, "time": time_torchaudio}

print("Comparison of data shapes:")
for lib, result in results.items():
    print(f"{lib}: shape = {result['data'].shape}, sample rate = {result['sr']}")
    print(f"{lib} first 10 values: {result['data'][:10]}")


print("\nComparison of loading times (in seconds):")
for lib, result in results.items():
    print(f"{lib}: {result['time']} seconds")

def compare_data(data1, data2):
    return np.allclose(data1, data2, atol=1e-6)

print(f"librosa vs torchaudio: {compare_data(results['librosa']['data'], results['torchaudio']['data'])}")
</code></pre>

wadewang: <p>Hello，I tried different parameters of <code>resampling_method</code> and <code>lowpass_filter_width</code>，find the minimal error with librosa basically converge to 2*10^(-6)</p>

wengzhenjie: <p>I have a same problem. Do you solve this problem?</p>
Request for Support in Integrating the RISC-V Vector ISA Extension

zhangfeiv0: <p>Hello everyone,</p>
<p>We are a small team from the Institute of Software, Chinese Academy of Sciences, attempt to improve PyTorch support on RISC-V.</p>
<p>I would like to draw your attention to my previously submitted <a href="https://github.com/pytorch/pytorch/pull/135570" rel="noopener nofollow ugc">PR (#135570)</a>, which aims to add support for the RISC-V Vector ISA (RVV) extension to PyTorch. This feature can significantly improve the performance of running models on RISC-V architecture, but it has not received enough response to date.</p>
<p>We have already made preliminary optimization attempts in <a href="https://github.com/pytorch/pytorch/pull/127867" rel="noopener nofollow ugc">PR (#127867)</a>, especially for the <code>DepthwiseConvKernel</code>, and initial results have shown a threefold performance improvement, proving the potential of RVV.</p>
<p>Currently, this PR has not received enough feedback and discussion. I am very eager to hear opinions and suggestions from the community. Please help to promote this feature or provide advice that can help me move forward further. If you are interested in discussing in depth or participating, you can find relevant information in the PR mentioned above.</p>
<p>Looking forward to your feedback!</p>

ptrblck: <aside class="quote no-group" data-username="zhangfeiv0" data-post="1" data-topic="212305">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhangfeiv0/48/69093_2.png" class="avatar"> zhangfeiv0:</div>
<blockquote>
<p>but it has not received enough response to date.</p>
</blockquote>
</aside>
<p>I don’t think that’s the case as the discussion is focused on enabling proper CI testing and is still ongoing. I’m sure <code>malfet</code> will follow up once he finds the time.</p>

zhangfeiv0: <p>Understood, it’s great to hear that the conversation is still active and moving forward. I appreciate the importance of thorough CI testing, and thanks for keeping me in the loop.</p>