akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Low GPU Utilization (~10%) with HF Datasets Streaming on Vertex with PyTorch Trainer

lpolisi: <p>Hi everyone,</p>
<p>I’m training a model on a <code>g2-standard-32</code> machine with a single NVIDIA L4 GPU on Google Vertex AI. My dataset consists of 4000 sharded parquet files (Snappy-compressed), each with ~4500 examples, stored in a GCS bucket. I’m using Hugging Face Datasets to stream the data and Hugging Face Trainer for training. My DL framework is PyTorch 1.13.</p>
<p>Here’s what I have tried:</p>
<ul>
<li>Different batch sizes: 512, 1024, 2048, 4096</li>
<li>Grid search on DataLoader parameters with batch size fixed to 512:
<ul>
<li><code>num_workers</code>: [2, 4, 8, 16]</li>
<li><code>prefetch_factor</code>: [2, 4, 8, 16]</li>
</ul>
</li>
</ul>
<p><strong>Observations</strong>:</p>
<ul>
<li><code>num_workers</code> improves performance but goes out of memory with more than 8 workers. At 8 workers, GPU utilization is still very low (~10%).</li>
<li><code>prefetch_factor</code> has minimal impact on GPU utilization.</li>
<li><code>batch_size</code> has minimal impact on GPU utilization, and overall GPU memory is low (biggest batch size had 50% GPU memory)</li>
</ul>
<p>Below is the collate function I use:</p>
<pre data-code-wrap="python"><code class="lang-python">def collate_fn(batch: list[dict[str, Any]]):
    labels = torch.tensor([item["label"] for item in batch], dtype=torch.long)
    col1 = torch.stack([torch.tensor(item["col1"], dtype=torch.float32).reshape(num1, num2, -1) for item in batch])
    col2 = torch.stack(
        [
            torch.tensor(item["col2"], dtype=torch.float32).reshape(num1, num2, -1)
            for item in batch
        ]
    )
    col3 = torch.stack(
        [torch.tensor(item["col3"], dtype=torch.float32).reshape(num1, num2, -1) for item in batch]
    )
    col4 = torch.stack(
        [torch.tensor(item["col4"], dtype=torch.float32).reshape(num1, num2, -1) for item in batch]
    )

    batch_processed = {
        "labels": labels,
        "col1": col1,
        "col2": col2,
        "col3": col3,
        "col4": col4,
    }
    return batch_processed
</code></pre>
<p><strong>Questions</strong>:</p>
<ol>
<li>Could this low GPU utilization be due to the dataset sharding strategy or the parquet format with Snappy compression?</li>
<li>Are there any best practices I could adopt to optimize streaming from cloud storage?</li>
<li>Is there something specific about my dataloader or collate function that could be causing the bottleneck?</li>
</ol>
<p>Any advice or recommendations would be greatly appreciated! Let me know if more information is needed.</p>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
Image_process.postprocess slow after torch.compile

bp45724: <p>Hey all! I am new to torch.compile and using torch.compile to speed up inference.</p>
<p>When apply torch.compile to a Flux pipeline transformer the vae image post processing function gets really slow (from sub 1s to 10s).  Seems to be spending most of the time on sending the tensors to the CPU ({method ‘cpu’ of ‘torch._C.TensorBase’ objects}).</p>
<p>Would like to understand why this is and if I am doing something wrong.  Happy to answer any questions.</p>
<p>For some additional background:</p>
<ul>
<li>I am working with a custom breakout of the Flux pipeline inference call in diffusers to split the pipeline across two GPUs with the transformer on GPU 0 (RTX 4090) and the rest of the components on GPU 1 (RTX 4060 ti).</li>
<li>VAE is not compiled. The VAE decode process takes very little time on GPU 1.</li>
<li>Timing of <code>postprocess</code> function without torch.compile on the transformer is 0.75 seconds.</li>
<li>Using <code>mode="reduce-overhead"</code> results in a <code>postprocess</code> time being 3.5 seconds</li>
</ul>
Compiling a method other than forward

ilykuleshov: <p>Hi everyone! Was wondering if there was any way to compile a non-forward method of an nn.Module, i.e. something similar to what <code>torch.jit.export</code> does. To the best of my knowledge, only the <code>forward</code> method gets compiled out-of-the-box, and I wasn’t able to find any docs on the subject. Would greatly appreciate any help!</p>

bdhirsh: <p>You should be able to compile other methods on your nn module by decorating them directly, e.g.:</p>
<pre><code class="lang-auto">class Mod(torch.nn.Module):
    def __init__(self):
        ....
    def forward(self):
        ....

    @torch.compile
    def my_method(self, x)
        ....

m = Mod()
x = torch.randn(4, 4)
# my_method will get just-in-time-compiled
out = m.my_method(x)
</code></pre>
<p>One thing to call out is: what are you trying to do?  Normally if you just do <code>m = torch.compile(m)</code>, this will effectively end up causing <code>m.__call__</code> to be compiled (which is a thin wrapper around the forward that also executes nn module hooks). So if you end up calling your method from inside of the your <code>Module.forward</code>, then compiling the entire module will cause its forward, as well as your custom method to get inlined into a single compiled region</p>

ilykuleshov: <p>Thank you for your answer! I am working with several modules without a forward method, so compiling them separately seems to be the way to go.</p>
Cuda not available when running multi-gpu inference

paulge: <p>Hi, I am running an inference script on a server with 7 NVIDIA GeForce GTX 1080 Ti. It’s currently running on a single GPU, which works fine. However, I’d like to parallelize things (I am using huggingface accelerate for that, this might also be an issue with their tool…). When running my script with <code>accelerate launch --config_file [my_config_file] run.py</code> torch cannot find cuda anymore. I am in the same conda env that works fine for single GPU inference. The config file sets <code>CUDA_VISIBLE_DEVICES</code> which it does in my case to ‘[1,2,3,4]’ (On GPU ‘0’ I am running the above mentioned script). Same happens when specifically setting <code>CUDA_VISIBLE_DEVICES=1,2,3,4</code> before calling <code>accelerate [...]</code>.<br>
I can’t seem to figure out why cuda is not available in this case.</p>
<ul>
<li>Cuda driver version 11.8</li>
<li>nvcc -version prints <code>nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2017 NVIDIA Corporation Built on Fri_Nov__3_21:07:56_CDT_2017 Cuda compilation tools, release 9.1, V9.1.85</code></li>
<li>torch.version.cuda is 11.8</li>
<li>pytorch version 2.0.0</li>
<li>dmesg does not have any entries regarding NVRM hinting at problems</li>
</ul>

ptrblck: <p>Could you describe your issue in more detail and what “CUDA not available” means in this context given you can use your GPU?</p>

paulge: <p>After importing torch, torch.cuda.is_available() is false and then during the initialization of an accelerate.PartialState() which under the hood uses torch.nn.parallel.DistributedDataParallel if I’m not mistaken, torch raises an AttributeError ‘torch.cpu has no attribute device_count’. Which I guess is due to the cpu being torch’s fallback device when no gpus are available and then accelerate calling device_count (checked this during debugging) on that device.</p>

ptrblck: <p>I don’t know if <code>accelerate</code> uses another PyTorch binary but you could double check it via printing <code>torch.__version__</code> and <code>torch.__path__</code> inside the standalone run (which is able to detect the GPU) and inside the script used by <code>accelerate</code>.</p>

paulge: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. Turns out it was a corrupt conda config on my end. Seems to be resolved now.</p>
find_unused_parameters=True fixes an error

thyeros: <p>I have a not-that-complex model, but it outputs this error with wrapped with DDP:</p>
<p>RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument <code>find_unused_parameters=True</code> to <code>torch.nn.parallel.DistributedDataParallel</code>;</p>
<p>Then, to find out which parm is causing an issue, I turned on find_unused_parameters=True, then the error went away while I was expecting to see a list of parameters. This is pytorch1.8. Can anyone have any clues and explain potentially why? This may not be specific enough, but like to see if anyone has faced the same issue.</p>

Yanli_Zhao: <p>find_unused_parameters=True can properly take care of unused parameters and sync them, so it fixes the error. In PT 1.9, if your application has unused parameters and you set find_unused_parameters=False, you will see the error message that includes the indices of the unused parameters</p>

thyeros: <p>good to know that, thanks!</p>

Hosein: <blockquote>
<p>find_unused_parameters=True can properly take care of unused parameters and sync them…</p>
</blockquote>
<p>What does this mean to sync a parameter which is not used?! If a parameter is not used, and then we set find_unused_parameters=True, does this mean that the parameter starts to be used and contributes to the loss?</p>
Efficiently Combine Two Tensors Based on a Boolean Mask

Shinnosuke_Uesaka: <p>Hi! I have two tensors:</p>
<ul>
<li><code>tensor_a</code> ([batch_size, seq_len_a, embedding_dim])</li>
<li><code>tensor_b</code> ([batch_size, seq_len_b, embedding_dim])</li>
</ul>
<p>The total sequence length is <code>seq_len_total = seq_len_a + seq_len_b</code>. I also have a boolean tensor <code>mask</code> ([batch_size, seq_len_total]) where <code>True</code> corresponds to positions for <code>tensor_a</code> and <code>False</code> corresponds to positions for <code>tensor_b</code>.</p>
<p>How can I efficiently combine <code>tensor_a</code> and <code>tensor_b</code> into a single tensor of shape ([batch_size, seq_len_total, embedding_dim]) using the <code>mask</code>.</p>

ptrblck: <p>I’m unsure if I understand your use case correctly, but direct indexing with the mask should work:</p>
<pre data-code-wrap="python"><code class="lang-python">batch_size, seq_len_a, seq_len_b, embedding_dim = 2, 3, 4, 5

a = torch.ones(batch_size, seq_len_a, embedding_dim)
b = torch.ones(batch_size, seq_len_b, embedding_dim) * 2

# create mask
mask = torch.cat((torch.tensor([True]*seq_len_a*batch_size), torch.tensor([False]*seq_len_b*batch_size)))
mask = mask[torch.randperm(len(mask))]
mask = mask.view(batch_size, -1)

out = torch.zeros(batch_size, seq_len_a+seq_len_b, embedding_dim)
out[mask] = a.view(-1, embedding_dim)
out[~mask] = b.view(-1, embedding_dim)
</code></pre>
Why OneCycleLR stores part of variables in optimizer groups

mombip: <p>After analyzing the source code, I noticed that <code>OneCycleLR</code> stores <code>max_lr</code> and <code>min_lr</code> (alongside <code>initial_lr</code>, which is managed by the base class) in the optimizer’s parameter groups.</p>
<pre><code class="lang-auto">    for idx, group in enumerate(self.optimizer.param_groups):
        group['initial_lr'] = max_lrs[idx] / div_factor
        group['max_lr'] = max_lrs[idx]
        group['min_lr'] = group['initial_lr'] / final_div_factor
</code></pre>
<p>This design effectively makes these variables shared within the scope of a parameter group across all <code>OneCycleLR</code> schedulers used during training (e.g., in <code>SequentialLR</code>).</p>
<p>This approach has the side effect of making it impossible to set different <code>max_lr</code> or <code>min_lr</code> values for multiple <code>OneCycleLR</code> instances applied to the same optimizer. Additionally, the documentation explicitly states that <em>“This scheduler is not chainable.”</em></p>
<p><em>My intention is not to criticize this approach but to understand its motivation</em>. Why was the decision made to store <code>OneCycleLR</code>-specific variables in the optimizer’s parameter groups, rather than keeping them local to the scheduler itself, especially given that scheduler-local storage might allow for greater isolation and flexibility?</p>
Pytorch Model Inference with CPU Parallelization

Max_Win: <p>Hi,</p>
<p>I am trying to perform inference on pytorch model saved using torch.jit.script(). Is there a C++ example where model inference is parallelized on multiple CPUs? I am using mpirun to parallelize the C++ code, and when I run C++ code with mpirun -np ${n}, it will launch n number of iterations whenever I call a Torch library function. Is there page like this one: <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Optional: Data Parallelism — PyTorch Tutorials 2.5.0+cu124 documentation</a> but for CPU parallelization instead of GPUs?</p>
<p>Max</p>
<p>Sincerely,<br>
Max</p>
Retrieving the logits for the class predictions for the Faster RCNN model in torchvision

loerbri1: <p>Hi,</p>
<p>I have followed the Object Detection Finetuning Tutorial to the train torchvision’s Faster RCNN model on my own data. I have found that the model only returns the score for the predicted category of each bounding box, but I would like to retrieve the logits behind each prediction (before softmax).</p>
<p>Is there a way of accessing those values? The only approach I could imagine is to copy all relevant source code, modify the respective return statements and then manually copy all the weights over to my “new” model. If anybody has an idea for a less ugly approach it would be highly appreciated!</p>

paula_gomez_duran: <p>Hey,</p>
<p>Have you found a solution already?</p>
<p>Thanks,</p>
<p>Paula</p>

SamShowalter: <p>Does anyone have an answer for this? I am also interested.</p>

jonathan-ibex: <p>Any updates? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

SeucheAchat9115: <p>Still no updates? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

annika_igl: <p>I just provided an answer to a similar thread: <a href="https://discuss.pytorch.org/t/how-to-see-classification-probabilities-in-the-object-detection/136887" class="inline-onebox">How to see classification probabilities in the object detection</a><br>
If you want the raw logits instead of softmax then you can just ommit the F.softmax line, I haven’t tested that though.</p>
I am trying to implement a Hierarchical Actor Critic model but my neural network doesn't return value with the correct dimension

Lordking1624: <p>So I am trying to build a kind of Actor critic model that uses hierarchical RL, and my problem is that one of my actor network returns an action with shape (1, 512) instead of (1, 6) even though it says that the dimension of the output layer is 6.</p>
<p>I am new to PyTorch and relatively new to RL so I don’t understand everything yet, if someone could help me it would be amazing. If you want more informations just ask.</p>
<p>Thank you in advance.</p>
<p>Actor network :</p>
<pre><code class="lang-auto">import os
import torch
import torch.nn as nn
import numpy as np

class ActorNetwork(nn.Module):
    def __init__(self, input_size, l1_dims=512, l2_dims=512,
                 output_size=1, name="actor", checkpoint_dir="../assets/model",
                 bound=1, offset=0):
        super(ActorNetwork, self).__init__()

        self.input_size = input_size
        self.l1_dims = l1_dims
        self.l2_dims = l2_dims
        self.output_size = output_size

        self.bound = bound
        self.offset = offset

        self.name = name
        self.checkpoint_dir = checkpoint_dir
        self.checkpoint_file = os.path.join(self.checkpoint_dir,
                                            self.name)

        self.l1 = nn.Linear(self.input_size, self.l1_dims)
        self.relu1 = nn.ReLU()
        self.l2 = nn.Linear(self.l1_dims, self.l2_dims)
        self.relu2 = nn.ReLU()
        self.l3 = nn.Linear(self.l2_dims, self.output_size)
        self.tanh = nn.Tanh()


    def call(self, state, goal):
        x = torch.cat([state, goal], dim=1)
        linear1 = self.l1(x)
        relu1 = self.relu1(linear1)
        linear2 = self.l2(relu1)
        relu2 = self.relu2(linear2)
        tanh = self.tanh(relu2)

        return (tanh.detach().numpy() * self.bound) + self.offset
</code></pre>

ptrblck: <p>You didn’t post a minimal and executable code snippet reproducing the unexpected output shape but based on the model definition you should check a few things:</p>
<ul>
<li>your model does not use <code>self.l3</code> at all while I assume this is the last linear layer with the desired output shape,</li>
<li>you are defining the forward pass in a <code>call</code> method while PyTorch expects the definition to be in the <code>forward</code> method,</li>
<li>applying <code>tanh</code> after <code>relu</code> sounds strange but is also irrelevant for your issue.</li>
</ul>

Lordking1624: <p>Thank you for you answer, I feel kinda dumb right now, it was just an error of inattention.<br>
I will also change the <code>call</code> function to <code>forward</code>.</p>
<p>Thank you for your help</p>
Why softmax training is more stable (than sigmoid)

laro: <p>I’m wondering about which activation function will be easier to train with (get better accuracy / smallest loss) - with <strong>SoftMax</strong> or <strong>sigmoid</strong> (for multiclass classification problem)</p>
<p>According to: <a href="https://www.quora.com/What-are-the-benefits-of-using-a-softmax-function-instead-of-a-sigmoid-function-in-training-deep-neural-networks" rel="noopener nofollow ugc">https://www.quora.com/What-are-the-benefits-of-using-a-softmax-function-instead-of-a-sigmoid-function-in-training-deep-neural-networks</a></p>
<p>Training model for multiclass with SoftMax - the training is more stable vs training with sigmoid</p>
<ul>
<li>Why is it true ?</li>
<li>Is it easier to train model (and get better results) with <strong>SoftMax</strong> (instead of <strong>sigmoid</strong>) ?</li>
</ul>

KFrank: <p>Hi Amit!</p>
<aside class="quote no-group" data-username="laro" data-post="1" data-topic="213069" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/laro/48/47125_2.png" class="avatar"> laro:</div>
<blockquote>
<p>I’m wondering about which activation function will be easier to train with (get better accuracy / smallest loss) - with <strong>SoftMax</strong> or <strong>sigmoid</strong> (for multiclass classification problem)</p>
</blockquote>
</aside>
<p>My reply to your other post probably answers this question.  See:</p>
<aside class="quote quote-modified" data-post="2" data-topic="213065">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/which-activation-function-for-multi-class-classification-gives-true-probability/213065/2">Which activation function for multi-class classification gives true probability?</a> 
  </div>
  <blockquote>
    Hi Amit! 

The short answer is that softmax() and sigmoid() are used for different things. 
It’s not that one is true and the other false or that one is more stable and the other 
less stable – they’re just different. 
Let me give you my perspective on this.  (I haven’t looked at the links you posted.) 
For some context, my number-one rule is that the output of a neural network means 
whatever you train it to mean. 
Let’s say that you have a problem with three classes and the final layer of your…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>
AnyModal – A Framework for Multimodal LLMs

ritabratamaiti: <p>Hi everyone,</p>
<p>I’d like to share a project I’ve been working on: <strong>AnyModal</strong>, a modular and extensible framework for integrating diverse input modalities (like images, audio, and structured data) into large language models (LLMs). It simplifies the process of combining different input types with LLMs, enabling tasks like image captioning, LaTeX OCR, and even chest X-ray interpretation.</p>
<p>GitHub: <a href="https://github.com/ritabratamaiti/AnyModal" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ritabratamaiti/AnyModal: AnyModal is a Flexible Multimodal Language Model Framework</a><br>
Reddit: <a href="https://www.reddit.com/r/AnyModal/" rel="noopener nofollow ugc">https://www.reddit.com/r/AnyModal/</a></p>
<h3><a name="p-459574-why-i-built-this-1" class="anchor" href="#p-459574-why-i-built-this-1"></a>Why I Built This</h3>
<p>Existing tools for multimodal systems often focus on specific tasks or are tightly coupled to particular models. I wanted a framework that could handle a wide variety of modalities with minimal setup, allowing researchers and developers to quickly prototype and experiment with new multimodal systems. That’s where AnyModal comes in.</p>
<h3><a name="p-459574-what-anymodal-does-2" class="anchor" href="#p-459574-what-anymodal-does-2"></a>What AnyModal Does</h3>
<p>AnyModal abstracts away much of the boilerplate in building multimodal LLMs:</p>
<ul>
<li>Seamless integration of different modalities through tokenization and embedding.</li>
<li>Flexibility to plug in pre-trained models (e.g., ViT for images) as feature encoders.</li>
<li>Simple projection layers to align modality-specific embeddings with the LLM token space.</li>
</ul>
<h3><a name="p-459574-example-use-case-3" class="anchor" href="#p-459574-example-use-case-3"></a>Example Use Case</h3>
<p>Here’s a typical workflow:<br>
You can take an image, process it with a vision transformer (like ViT), project the embeddings to match the LLM’s token space, and pass it to the LLM for tasks like caption generation or question answering. Similarly, you could handle audio inputs by encoding them into embeddings and integrating them into the LLM pipeline.</p>
<h3><a name="p-459574-current-demos-4" class="anchor" href="#p-459574-current-demos-4"></a>Current Demos</h3>
<ul>
<li>LaTeX OCR</li>
<li>Chest X-Ray Captioning (in progress)</li>
<li>Image Captioning</li>
<li>Visual Question Answering (planned)</li>
<li>Audio Captioning (planned)</li>
</ul>
<h3><a name="p-459574-whats-next-5" class="anchor" href="#p-459574-whats-next-5"></a>What’s Next</h3>
<p>AnyModal is still a work in progress, and I’m planning to expand its capabilities with more demos and better support for different modalities. I’d love feedback or contributions from anyone interested in this space.</p>
<p>Let me know what you think or if you have any questions!</p>
Correct SDPA's attn_mask for Self-attention

SotoJnthn: <p>Hello, I am trying to implement Multihead Self-Attention using torch.nn.functional.scaled_dot_product_attention, but I am not sure how to transform the src_key_padding_mask usually taken by the nn.TransformerEncoder to the desired attn_mask taken by SDPA. Is this implementation correct?</p>
<pre><code class="lang-auto">from torch import nn
import torch.nn.functional as F
from typing import Callable

class SelfAttention(nn.Module):

    def __init__(
        self,
        embed_dim: int,
        num_heads: int,
        dropout: float = 0.0,
        bias: bool = True,
        rope: Callable = None,
        is_causal: bool = False,
    ):
        super().__init__()
        assert embed_dim % num_heads == 0
        self.qkv_proj = nn.Linear(embed_dim, 3 * embed_dim, bias=bias)
        self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.dropout = dropout
        self.num_heads = num_heads
        self.embed_dim = embed_dim
        self.head_dim = embed_dim // num_heads
        self.rope = rope
        self.is_causal = is_causal

    def forward(self, x, key_padding_mask=None):
        qkv = self.qkv_proj(x)
        qkv = qkv.unflatten(-1, [self.num_heads, 3 * self.head_dim])
        q, k, v = qkv.chunk(3, dim=-1)
        if self.rope:
            q = self.rope(q)
            k = self.rope(k)

        attn_mask = None
        if key_padding_mask is not None:
            B, S = key_padding_mask.shape
            attn_mask = ~key_padding_mask\
                .view(B, 1, 1, S)\
                .expand(-1, self.num_heads, -1, -1)

        attn_output = F.scaled_dot_product_attention(
            query=q.transpose(1, 2),
            key=k.transpose(1, 2),
            value=v.transpose(1, 2),
            attn_mask=attn_mask,
            dropout_p=self.dropout if self.training else 0.0,
            is_causal=self.is_causal if self.training else False,
        )

        attn_output = attn_output.transpose(1, 2).flatten(-2)

        return self.out_proj(attn_output)
</code></pre>
Help for understanding RNN sequence inputs and output data

Ras: <p>I need to develop a system where the output data is dependent on the previous input. The system expected to have one input and one output at a time.</p>
<p>For training, I collect the sequences, say for example x1, x2, x3, …, xn, when I feed them as batches to an rnn/gru, the out put is in the size of (N, L, Hout), using batch first option. But I only need the data at the size (N, L). I’m not sure how to use a linear layer to combine them to get the output in the desired size, because when using the system (after training/testing) I need to feed only one data point and need to obtain one data point for that input.</p>

vdw: <p>Can you be more specific about task you want to learn. I can’t tell for sure if it’s a basic classification task or more akin to a sequence labeling task.</p>

Ras: <p>Thank you for the reply. I have one-dimensional sensor data as a time series, such as <code>([x_1, x_2, x_3, ..., x_n]</code>$. The input to the RNN is of dimension ((N, L, 1)), and the output is of dimension <code>(N, L, H_{out})</code>, where <code>N</code> is the batch size, <code>L</code> is the sequence length, and <code>H_{out}</code> is the hidden size. I understand that the sequence of <code>L</code> data inputs produces <code>L</code> sequences of data at the output. The length of the sequence doesn’t matter as long as the hidden data is fed back to the RNN, so I can feed one input at an instant as soon as the data arrives from the sensor and get one data point for the corresponding input after it is processed by the RNN.</p>
<p>However, my original issue was how to reduce the dimension of the output data from <code>(N, L, H_{out})</code> to <code>(N, L, 1)</code> as required. Now I realize that I can use an <code>nn.Linear(H_{out}, 1)</code> to do that job. I originally misunderstood that the matrix multiplication inside <code>nn.Linear</code> might multiply the inter-sequence data.</p>
Matrix multiplication with two common dimensions

ssamtkwon: <p>I know that Pytorch can handle batch matrix multiplication, like (B, X, Y) * (B, Y, Z) → (B, X, Z).</p>
<p>But what if the matrices had two common dimensions? I want (B1, B2, X, Y) * (B1, B2, Y, Z) → (B1, B2, X, Z), and it doesn’t seem to be handled by the default multiplication. Is there a good way?</p>

ptrblck: <p>I might misunderstand your use case but couldn’t you flatten the first two dimensions into one and unflatten them afterwards?</p>

ssamtkwon: <p>Yes, that’s actually what I’ve been doing. But code became dirty and I was just wondering if there’s a single function that took care of this, since there seems to be like a million matrix manipulating functions here… <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
How to represent the jacobian of a function where the domain field is from a cartesian product

MartensCedric: <p>Normally, if I have a function f = R^2 → R^3. It will have a 3 by 2 jacobian. But what if my function takes two arguments in R^2.</p>
<p>// pseudocode<br>
f(a, b): // a and b are  both (2,1) so this function has 4 parameters<br>
return torch.cat((a + b, [1]), 1) // output is (3,1)</p>
<p>This is the same as saying this is a function of R^4 to R^3.</p>
<p>What would be the dimension of the jacobian here? 3 by 4? 3 by 2 by 2?</p>

KFrank: <p>Hi Cedric!</p>
<aside class="quote no-group" data-username="MartensCedric" data-post="1" data-topic="213028" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/martenscedric/48/72949_2.png" class="avatar"> MartensCedric:</div>
<blockquote>
<p>Normally, if I have a function f = R^2 → R^3. It will have a 3 by 2 jacobian. But what if my function takes two arguments in R^2.</p>
</blockquote>
</aside>
<p>Pytorch’s approach (which seems fine to me) is to return a tuple of jacobians,<br>
one for each of the arguments.  Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.5.1'
&gt;&gt;&gt; 
&gt;&gt;&gt; _ = torch.manual_seed (2024)
&gt;&gt;&gt; 
&gt;&gt;&gt; def f (r2a, r2b):    # some r2, r2 --&gt; r3 function
...     m = torch.arange (6.0).reshape (3, 2)
...     return  m @ (r2a * r2b)
... 
&gt;&gt;&gt; a = torch.randn (2)
&gt;&gt;&gt; b = torch.randn (2)
&gt;&gt;&gt; 
&gt;&gt;&gt; jacfunc = torch.func.jacrev (f, argnums = (0, 1))   # tell jacrev you want both arguments of f
&gt;&gt;&gt; 
&gt;&gt;&gt; jacfunc (a, b)                                      # jacfunc returns a tuple of both jacobians
(tensor([[-0.0000,  1.3722],
        [-1.6281,  4.1165],
        [-3.2562,  6.8608]]), tensor([[-0.0000,  1.7260],
        [-0.0809,  5.1780],
        [-0.1617,  8.6300]]))
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Dynamic slicing torch.export

will-rice: <p>I want to do this:</p>
<pre><code class="lang-auto">import torch
from torch.export import export


class DynamicShapeSlicing(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.probs = torch.rand((3,))

    def forward(self, x, lengths):
        idx = self.probs.argmax()
        return x[idx, :, : lengths[idx]]


example_args = (torch.randn(3, 10, 8), torch.randint(0, 8, (3,)))
model = DynamicShapeSlicing()


ex = export(model, example_args, strict=False)
ex.module()(*example_args)
</code></pre>
<p>This is the error I get:</p>
<blockquote>
<blockquote>
<blockquote></blockquote>
</blockquote>
</blockquote>
<p>GuardOnDataDependentSymNode: Could not extract specialized integer from data-dependent expression u0 (unhinted: u0).  (Size-like symbols: none)</p>
<p>Potential framework code culprit (scroll up for full backtrace):<br>
File “/home/will/.pyenv/versions/3.10.6/envs/notebook/lib/python3.10/site-packages/torch/_export/non_strict_utils.py”, line 520, in <strong>torch_function</strong><br>
return func(*args, **kwargs)</p>
<p>For more information, run with TORCH_LOGS=“dynamic”<br>
For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL=“u0”<br>
If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1<br>
For more debugging help, see <a href="https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Dealing with GuardOnDataDependentSymNode errors - Google Docs</a></p>
<p>For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1</p>
<p>The following call raised this error:<br>
File “/tmp/ipykernel_1216314/1925435421.py”, line 16, in forward<br>
return x[idx, :, : lengths[idx]]</p>
<blockquote>
<blockquote>
<blockquote></blockquote>
</blockquote>
</blockquote>
<p>What is the proper way to do this that will allow me to export my model?</p>

will-rice: <p>This is what I ended up having to do. I’m not convinced that it is the best method. Obviously not very clean at all.</p>
<pre><code class="lang-auto">class DynamicShapeSlicing(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.probs = torch.rand((3,))

    def forward(self, x, lengths):
        idx = self.probs.argmax().item()
        torch._check(x.size(0) &gt; idx)
        torch._check(0 &lt;= idx)
        torch._check(lengths.size(0) &gt; idx)
        torch._check_is_size(idx)
        length = lengths.narrow(0, idx, 1).item()
        torch._check(length &lt; x.size(2))
        torch._check_is_size(length)
        return x.narrow(0, idx, 1).narrow(2, 0, length)
</code></pre>

bdhirsh: <p>Can you file a github issue? <a href="https://github.com/pytorch/pytorch/issues/new/choose" class="inline-onebox" rel="noopener nofollow ugc">Sign in to GitHub · GitHub</a>. Relevant folks can discuss whether there is a cleaner way to do this (or if not, whether there’s anything to do to make this code more export-friendly by default)</p>
Discrepancies Between Compiled and Non-Compiled Models with Convolutional Layers in PyTorch

WiseChoice: <p>Hello everyone,</p>
<p>I’ve been experimenting with PyTorch’s torch.compile function (with mode=‘max-autotune’) to optimize my model. I’ve observed some discrepancies in the outputs of the network when comparing the compiled version to the non-compiled version, but only when convolutional layers are involved. On networks with nothing but linear layers I observe the exact same outputs.</p>
<p>Is this expected behavior? Does torch.compile in max-autotune mode have less precision? I actually observe a substantial speedup when using the compiled version (on an RTX 3080) but I worry that it may come with reduced precision and I hesitate to embed it to my code. I use the latest stable release of torch: 2.4.1.</p>
<p>Here is a minimal example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.testing import assert_allclose

class ConvLinearModel(nn.Module):
    def __init__(self):
        super(ConvLinearModel, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, stride=1, padding=1)
        self.fc1 = nn.Linear(32 * 8 * 8, 512)
        self.fc2 = nn.Linear(512, 10)

    def forward(self, x):
        x = torch.relu(self.conv1(x))
        x = torch.relu(self.conv2(x))
        x = x.view(x.size(0), -1)  # Flatten
        x = torch.relu(self.fc1(x))
        return self.fc2(x)

def test_accuracy_over_runs(num_runs=100):
    discrepancies = 0
    model = ConvLinearModel()
    compiled_model = torch.compile(model, mode='max-autotune')

    for i in range(num_runs):
        torch.manual_seed(42 + i)
        input_data = torch.randn(10, 3, 8, 8)
        non_compiled_output = model(input_data)
        compiled_output = compiled_model(input_data)

        try:
            assert_allclose(compiled_output, non_compiled_output, rtol=1e-5, atol=1e-8)
        except AssertionError:
            discrepancies += 1

    return discrepancies

# Testing
discrepancies = test_accuracy_over_runs(100)
print(f"Number of discrepancies found: {discrepancies}")
</code></pre>

bdhirsh: <p>When I run your script on a nightly (on an H100 - I don’t have an RTX to test on), I don’t get any discrepancies.</p>
<p>Just a few ideas/questions:</p>
<p>(1) have you tried on a recent nightly?</p>
<p>(2) You specified an atol of <code>atol=1e-8</code>, which is quite small. In general, the compiler can fuse operations that can cause slight changes in numerics. What is the biggest atol/rtol numeric difference you’re seeing?</p>
<p>(3) Does this only repro with <code>mode='max-autotune</code>? Or does it also repro when you use vanilla inductor</p>
Any chance to preserve some ops while decomposing PT2E model?

siahuat0727: <p>Hi team,</p>
<p>I wonder is there any way to preserve some type of ops while doing decomposition <code>ep.run_decompositions()</code>?</p>
<p>For example,</p>
<pre><code class="lang-auto">aten.upsample_nearest2d.vec   (div_5, None, [2.0, 2.0])
</code></pre>
<p>will convert to:</p>
<pre><code class="lang-auto">opcode         name          target                   args                                                   
-------------  ------------  -----------------------  -------------------------------------------------------
call_function  arange_0      aten.arange.start_step   (0, 64)                                                
call_function  add_9         aten.add.Tensor          (arange_0, 0.0)                                        
call_function  mul_12        aten.mul.Tensor          (add_9, 0.5)                                           
call_function  _to_copy_23   aten._to_copy.default    (mul_12,)                                              
call_function  unsqueeze_0   aten.unsqueeze.default   (_to_copy_23, -1)                                      
call_function  arange_1      aten.arange.start_step   (0, 64)                                                
call_function  add_10        aten.add.Tensor          (arange_1, 0.0)                                        
call_function  mul_13        aten.mul.Tensor          (add_10, 0.5)                                          
call_function  _to_copy_24   aten._to_copy.default    (mul_13,)                                              
call_function  index_0       aten.index.Tensor        (_to_copy_22, [None, None, unsqueeze_0, _to_copy_24])  
</code></pre>
<p>But I wish to keep <code>aten.upsample_nearest2d.vec</code>. Thank you.</p>
<p>Code snippet:</p>
<pre><code class="lang-auto">model = ...
sample_inputs = ...
ep = torch.export.export(model, sample_inputs)
ep = ep.run_decompositions()
</code></pre>

bdhirsh: <p>You should be able to preserve <code>upsample_nearest2d.vec</code> now, as of a few weeks ago (<code>run_decompositions()</code> shouldn’t secretly decompose a few ops anymore). Can you try again on a nightly?</p>
Multiple compiled versions of the same model

hydrowan: <p>I want to torch compile the same model several times, optimized for different shapes, but pointing to the same large model weights. Allowing the same model to be dynamic over the shape ranges resulted in too large a slow down.</p>
<pre><code class="lang-auto">compiled_model1 = torch.compile(my_model)
_ = compiled_model1(dummy_input1) # Slow

compiled_model2 = torch.compile(my_model)
_ = compiled_model2(dummy_input2) # Slow

compiled_model3 = torch.compile(my_model)
_ = compiled_model3(dummy_input3) # Fast

compiled_model4 = torch.compile(my_model)
_ = compiled_model4(dummy_input4) # Fast


</code></pre>
<p>The ID differs between compiled model variables, but the fact that models 3, 4 and onwards suddenly compile (on first pass) quickly worries me. Does something optimize the graph generation after a few times, or are they all pointing to the same graph, and the reason <span class="hashtag-raw">#2</span> is slow, is due to it becoming dynamic with the different input size?</p>
<p>Setting dynamic=False in torch.compile() makes every JIT take the full, slow, time.</p>

hydrowan: <p>Any suggestions on this? Could it just be building triton / inductor caches?</p>

bdhirsh: <p>A couple things:</p>
<p>(1) when you call <code>m = torch.compile(m)</code>, you are not actually generating a “copy” of the model (so you don’t have to worry about your weights not being shared across compiled regions). You are effectively just annotating the <code>torch.nn.Module.__call__</code> method on your model as being marked for compilation.</p>
<p>(2) You don’t actually need to call <code>torch.compile</code> multiple times. If you just do this:</p>
<pre><code class="lang-auto">m = torch.compile(my_model)
out1 = m(inp1)
out2 = m(inp2)
out3 = m(inp3)
out4 = m(inp4)
</code></pre>
<p>When you do the above:</p>
<p>(1) the first time you invoke your compiled model, we will perform compilation to get a compiled artifact, and we will also generate a bunch of guards that tell us when this compiled artifact is safe to re-use (for example, we will guard on stuff like the shape of your input, its device/dtype, whether it requires_grad, etc)</p>
<p>(2) when you run your compiled model on another input, then we will run the series of guards that we generated, and if they all pass we’ll re-use one of the previously generated compiled artifacts. If any guards fail, we will have to recompile (causing the slowness you mentioned above).</p>
<p>If you want to see the source of recompilations, you can run with <code>TORCH_LOGS="recompiles"</code> to see why</p>
Dynamo Trace with Parameter Lifting

a1trl9: <p>Hi,</p>
<p>During the Dynamo Trace phase, is there a suitable way to lift all the parameters in the Module to input arguments? Our requirement is: for each subsequent inference, feeding the model parameters as function arguments (rather than functionalize them during the <code>aot_autograd</code> phase, allowing the backend compiler to see a functional graph).</p>
<p><em>Why do this?</em><br>
Our scenario is: the online model’s parameters updated every 10-30 minutes.</p>
<p><em>Why not use torch.export?</em><br>
Unfortunately, our model cannot always be traced into a single graph.</p>
<p><em>Why not recompile?</em><br>
Unfortunately, our users’ models require a lengthy period for tracing and compiling, which is even longer than the interval between two rounds of parameter update.</p>
<p><em>What we tried and why not work?</em><br>
We tried to create a new callable, update parameters and then call the original forward method. A demo is shown below:</p>
<pre><code class="lang-auto"># Simple module for demonstration
class M(torch.nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.kernel = torch.nn.Parameter(torch.empty(256, 4).to(device="cuda:0"))
        self.relu = torch.nn.ReLU()

    def forward(self, x: torch.Tensor, *, constant=None) -&gt; torch.Tensor:
        a = torch.matmul(x, self.kernel)
        b = self.relu(a)
        return b

def create_call_func(m):
    orig_forward = m.forward
    def call_func(x, kernel):
        m._parameters["kernel"].data = kernel
        orig_forward(x)
    return call_func
</code></pre>
<p>We’ve encountered quite a few issues, including:</p>
<ol>
<li>During the first compilation trace, the parameters passed as input arguments need to have the same ID (id(variable)) as the parameters in the model.</li>
<li>Failures of guards.</li>
<li>Handling of side effects.</li>
</ol>
<p>This makes us question whether our current approach is incorrect.</p>
<p><em>Alternatives</em><br>
Shall we just separate the logic for updating parameters from the forward function? Just use the relevant methods from torch.nn.Module to replace the parameters.</p>
<p>Looking forward to any suggestions</p>

bdhirsh: <blockquote>
<p>Our scenario is: the online model’s parameters updated every 10-30 minutes.</p>
</blockquote>
<p>Can you describe in a bit more detail what this means? are you updating the data of your model’s parameters? Or are you taking your <code>nn.Module</code> and slamming in fresh <code>nn.Parameters</code>, and you would like to avoid recompiles even though the parameters have changed). This sounds like something that should work (you can avoid the recompiles as long as your parameter shapes aren’t changing), although some code examples would be helpful</p>
AOTInductor autograd support

IgnacioPickering: <p>Reading the available documentation and a bit of the code in <code>torch/_export/</code> I couldn’t really figure out whether AOTInductor supports autograd, and if not, whether support is in the roadmap. It would be very useful to have this for models that rely on autograd for inference. Can I find more info about it somewhere? Is there a way to export a model’s <code>backward()</code> as a <code>*.so</code> library on linux?</p>

bdhirsh: <p>This is something we can support in principle, although I’m not sure of the timeline (if I remember correctly, one extra bit of work needed to get training support for AOTInductor is that it will need to handle inplace mutations to the parameters).</p>
<p>Can you file a github issue? <a href="https://github.com/pytorch/pytorch/issues/new/choose" class="inline-onebox" rel="noopener nofollow ugc">Sign in to GitHub · GitHub</a></p>
Is it possible to ignore part of the code for torch compile

divinho: <p>I have code like</p>
<pre><code class="lang-auto">if condition:
   do foo
else:
   do bla
</code></pre>
<p><code>do foo</code> is not compileable, but when using torch.compile I’m never going to enter that code path anyways. Is there a way of letting the compiler know that?</p>

ptrblck: <p>You could try to use <code>torch.compiler.disable</code> as described <a href="https://pytorch.org/docs/stable/torch.compiler_fine_grain_apis.html#torch-compiler-disable">here</a>.</p>

divinho: <p>I saw that but that seems to just allow for decorating functions? I see now it says<br>
<a href="https://pytorch.org/docs/stable/generated/torch.compiler.disable.html" rel="noopener nofollow ugc">here</a> it’s also a context manager so let me using it like that.</p>

divinho: <p>ok that doesnt work guess i have to put it in a different function</p>

bdhirsh: <p>I might be misunderstanding - but if you want to ensure that <code>foo()</code> is only executed when running with eager, and not when running with <code>torch.compile</code>, you can do:</p>
<pre><code class="lang-auto">if torch.compiler.is_compiling():
    return bla()
else:
    return foo()
</code></pre>
Where can I find the Declarations.yaml for a given Pytorch version?

darioteixeira: <p>I’m not using Pytorch directly, but its bindings to a different language (OCaml). To compile the bindings, I presently need the <code>Declarations.yaml</code> file for Pytorch 2.1.2, though in the future I’ll need the declarations file for newer versions of Pytorch.</p>
<p>Apparently <code>Declarations.yaml</code> used to ship with Pytorch, but that hasn’t been the case for a while. Unfortunately, it is far from trivial to generate the file. One needs to compile Pytorch, which is not an obvious task for someone not conversant in the Python ecosystem, and which takes <strong>forever</strong> even when everything goes well.</p>
<p>There may be good technical reasons why the <code>Declarations.yaml</code> is not shipped anymore, but could this file be at least easily available for the sake of users of non-Python languages? Anyway, maybe it already is – hence my question! Does anyone know of a place where it can be found?</p>

bdhirsh: <p>I’m not actually sure when Declarations.yaml did / did not ship with pytorch directly.</p>
<p>But you shouldn’t need to compile all of pytorch to access it - you can run the <code>torchgen</code> module if you have a local pytorch checkout:</p>
<pre><code class="lang-auto">git clone git@github.com:pytorch/pytorch.git
cd pytorch
# generates build/aten/src/ATen/Declarations.yaml,
# as well as many other code-generated files
# FWIW: all of the codegen machinery is driven from native_functions.yaml from source
python -m torchgen.gen
</code></pre>
Running Pytorch 1.13 on H100

Sia_Rezaei: <p>I installed 1.13.0+cu117. I’m getting this warning:</p>
<pre><code class="lang-auto">NVIDIA H100 80GB HBM3 with CUDA capability sm_90 is not compatible with the current PyTorch installation.
</code></pre>
<p>I have to use 1.13.0. Is there a way to get it to work with H100s?</p>

ptrblck: <p>No, since PyTorch=1.13.x supported CUDA&lt;=11.7 while Hopper support landed in CUDA 11.8+.<br>
PyTorch==2.0 was the next release which supported Hopper.</p>

Sia_Rezaei: <p>What about this: <a href="https://docs.nvidia.com/cuda/hopper-compatibility-guide/#applications-built-using-cuda-toolkit-11-7-or-earlier" class="inline-onebox" rel="noopener nofollow ugc">Hopper Architecture Compatibility</a></p>
<p>Also the compatibility matrix here seems to say that Hopper is compatible with 11.7</p>

ptrblck: <blockquote>
<p>CUDA applications built using CUDA Toolkit versions 2.1 through 11.7 are compatible with Hopper GPUs as long as they are built to include PTX versions of their kernels. This can be tested by forcing the PTX to JIT-compile at application load time with following the steps:</p>
</blockquote>
<p>You can check if PTX was shipped in these binaries and could try to JIT compile it, but I haven’t tried it using these wheels.</p>

Yangqi_Long: <p>Can I use PyTorch1.13.0+CUDA11.8?<br>
Or I should use PyTorch &gt;=2.0?</p>

ptrblck: <p>I would recommend using the latest, but any build with 11.8 should work.</p>