akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Reusing worker processes across different DataLoader instances

smolboii: <p>I’m developing a codebase for continual learning scenarios, and these scenarios often have a set of individual datasets, with a separate one for each ‘task’ being learnt. These task datasets are then iterated over in order, with (typically) multiple epochs being performed for each dataset.</p>
<p>To iterate over these datasets I have been using the PyTorch DataLoader class, and this works well within each dataset as there is no latency between epochs. However, when moving from one task dataset to the next, as the DataLoader is recreated (and hence so are its workers, I imagine), when the number of workers is large there is a delay (on the order of 8s for 128 workers).</p>
<p>Is there some way to make it so these workers persist <em>across</em> these separate DataLoader instances?  I know there is the “persistent_workers” keyword argument for DataLoader, but I believe this only applies to that DataLoader instance and is only intended to preserve workers across epochs, not across datasets.</p>
<p>I can probably refactor my code such that different tasks’ datasets are in the same dataset object, but this seems like it could be difficult…</p>
Can you do bf16 x bf16 -> FP32 matmul?

henrylhtsang: <p>Possibly a noob question here. Say I have two bf16 tensors. Is there a way to do matmul on them and return the accumulator in FP32?</p>

malfet: <p>This might be implementation dependent, but in almost all cases accumulation will be done in fp32 by default. You can try <code>out</code> variant that might work</p>
Cannot install pytorch with Given Pip command SSL error

Samuel_Chambers: <p>when running the command from the Pytorch website i get an error during install</p>
<pre><code class="lang-auto">WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)'))': /whl/cu118/torch/
Could not fetch URL https://download.pytorch.org/whl/cu118/torch/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='download.pytorch.org', port=443): Max retries exceeded with url: /whl/cu118/torch/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)'))) - skipping
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
ERROR: No matching distribution found for torch
</code></pre>
<p>I believe this is due to a security block from my company. is there any other way to install pytorch with cuda support? I am able to install pip packages without errors if im not trying to access a URL.<br>
EG: pip install numpy<br>
throws no errors</p>
<p>python version 3.10.11<br>
torch 2.6.0<br>
cuda 11.8<br>
windows 11</p>

ptrblck: <p>CC <a class="mention" href="/u/malfet">@malfet</a> in case you would know if our cert is causing issues.</p>

malfet: <aside class="quote no-group" data-username="Samuel_Chambers" data-post="1" data-topic="216272">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samuel_chambers/48/75483_2.png" class="avatar"> Samuel_Chambers:</div>
<blockquote>
<p>I believe this is due to a security block from my company.</p>
</blockquote>
</aside>
<p>In that case there isn’t much one can do there. Trying talking to them and ask adding <a href="http://download.pytorch.org" rel="noopener nofollow ugc">download.pytorch.org</a> to the list of exceptions. Or perhaps there is a corporate proxy sever one can configure to allow access to more web-sites.</p>
<aside class="quote no-group" data-username="Samuel_Chambers" data-post="1" data-topic="216272">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samuel_chambers/48/75483_2.png" class="avatar"> Samuel_Chambers:</div>
<blockquote>
<p>is there any other way to install pytorch with cuda support?</p>
</blockquote>
</aside>
<p>You can try building from source or installing anaconda package, whenever it becomes available.</p>
FlexAttention with sparse edge bias

mbaranov: <p>Motivated by this example in the flex attention blog post:</p>
<pre><code class="lang-auto">bias = torch.randn(1024, 1024)
def score_mod(score, b, h, q_idx, kv_idx):
    return score + bias[q_idx][kv_idx] # The bias tensor can change!
</code></pre>
<p>But not wanting to construct bias on the fully connected graph for memory issues.</p>
<p>I’ve tried something like:</p>
<pre><code class="lang-auto">num_neighbors = 50
edge_idx = torch.randint(low=0, high=N*A, size=(B, N*A, num_neighbors)).cuda()
bias = torch.randn(B, N*A, num_neighbors).cuda()

def score_mod(score, b, h, q_idx, kv_idx):
    true_fn = lambda : score + bias[q_idx, torch.searchsorted(edge_idx[0, q_idx], kv_idx)] 
    false_fn = lambda: score 
    boolean = torch.isin(kv_idx, edge_idx[b,q_idx])
    score = cond(boolean, true_fn, false_fn, ()) 
    return score 

block_mask = create_block_mask(window_mask, B=None, H=None, Q_LEN=N*A, KV_LEN=N*A)
flex_attention(node_h.unsqueeze(1), 
               node_h.unsqueeze(1), 
               node_h.unsqueeze(1), 
               block_mask=block_mask, score_mod=score_mod)
</code></pre>
<p>Where <code>boolean = torch.isin(kv_idx, edge_idx[b,q_idx])</code> checks for a key, query pair whether the key is one of the queries neighbors, and if yes <code>torch.searchsorted(edge_idx[0, q_idx], kv_idx)</code> will return the local index (in <code>edge_bias</code>) corresponding to the global index (kv_idx) (we can assume here edge_idx is sorted to be increasing).</p>
<p>I get the error</p>
<pre><code class="lang-auto">UncapturedHigherOrderOpError: Cond doesn't work unless it is captured completely with torch.compile. Scroll up to find out what causes the graph break.
</code></pre>
<p>Which I don’t understand, not having much experience working with torch.compile before. Other things I have tried is using sparse tensor format but ran into errors there that seemed like it was a dead end.</p>
<p>The issue seems to boil down (I could be missing a lot) where I have to do two checks (1) is the query neighbors with the key and (2) a lookup mapping global index (kv_idx) to local index (the index of the neighbor in <code>edge_bias</code> corresponding to <code>kv_idx</code>. Would precomputing the lookup instead of using <code>torch.search_sorted</code> get around the problem? Any input or thoughts would be appreciated, thank you!</p>
<p>I am working on torch==2.6.0 and CUDA 12.2</p>
Register an autograd kernel warning

Roy-Kid: <p>I got an error when I use my own c++ operator:</p>
<pre><code class="lang-auto"> UserWarning: pme::pme_reciprocal: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:62.)
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass 
</code></pre>
<p>Here is my def and impl:</p>
<pre><code class="lang-auto">TORCH_LIBRARY(pme, m) {
    m.def("pme_direct(Tensor positions, Tensor charges, Tensor neighbors, Tensor deltas, Tensor distances, Tensor exclusions, Scalar alpha, Scalar coulomb) -&gt; Tensor");
    m.def("pme_reciprocal(Tensor positions, Tensor charges, Tensor box_vectors, Scalar gridx, Scalar gridy, Scalar gridz, Scalar order, Scalar alpha, Scalar coulomb, Tensor xmoduli, Tensor ymoduli, Tensor zmoduli) -&gt; Tensor");
}
TORCH_LIBRARY_IMPL(pme, CPU, m) {
    m.impl("pme_direct", pme_direct_cpu);
    m.impl("pme_reciprocal", pme_reciprocal_cpu);
}
</code></pre>
<p>then I modify it as</p>
<pre><code class="lang-auto">TORCH_LIBRARY_IMPL(pme, CPU, m) {
    m.impl("pme_direct", torch::dispatch(c10::DispatchKey::AutogradCPU, pme_direct_cpu));
    m.impl("pme_reciprocal", torch::dispatch(c10::DispatchKey::AutogradCPU, pme_reciprocal_cpu));
}
</code></pre>
<p>It crushed immediately.<br>
How do I fix this, and where can I find a reference?</p>
<p>Thanks!!</p>

bdhirsh: <p>You need to register your autograd implementation to an autograd dispatch key. You can do that by replacing CPU in the macro with AutogradCPU:</p>
<p>TORCH_LIBRARY_IMPL(pme, AutogradCPU, m) {</p>

Roy-Kid: <p>THANKS a lot! It does fix my problem!! But I still hope the c++ doc can be more friendly for those who cant read c++ source code…</p>
JIT model is a deployment model or a quantized model?

BambooKui: <p>Hi,I have some questions about quantization and deployment:<br>
First，I’d like to ask, is the JIT model exported after FX quantization in PyTorch considered a deployment model or a quantized model? Has any relevant processing been done for the backend platform?<br>
Also, I’ve seen on GitHub that some people directly redefine the QConv and QLinear operators to replace the Conv and Linear operators in the model structure. Is it not feasible to use this method in actual deployment?<br>
Lastly,How can I export a quantized model in PyTorch to other backend platforms, such as TensorRT, OpenVINO, and ONNX Runtime? It seems that PyTorch does not support exporting quantized models to ONNX.</p>
`torch.nn.functional.conv2d` 8 times slower in torch 2.5.1 compared to 2.3.1

navidsam: <p>Hey everyone, I am noticing that in torch 2.5.1 when I run the following snippet the exact same operation takes around 8 times slower compared to torch 2.3.1 when everything else in my environment remains the same:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
import time
print(torch.__version__)
# prints the following
# 2.5.1+cu124

print("torch.backends.cudnn.deterministic = ", torch.backends.cudnn.deterministic)
print("torch.backends.cuda.matmul.allow_tf32 = ", torch.backends.cuda.matmul.allow_tf32)
print("torch.backends.cudnn.allow_tf32 = ", torch.backends.cudnn.allow_tf32)
print("torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction = ", torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction)
# prints the following
# torch.backends.cudnn.deterministic =  False
# torch.backends.cuda.matmul.allow_tf32 =  False
# torch.backends.cudnn.allow_tf32 =  True
# torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction =  True


torch.cuda.synchronize()
myconv_start = time.time()
with torch.no_grad():
    out = F.conv2d(torch.rand(75, 1, 572, 572).cuda(), torch.rand(1, 1, 51, 51).cuda(), padding=0).cpu()
torch.cuda.synchronize()
myconv_time = time.time() - myconv_start
print(f"myconv_time = {myconv_time:0.2f} seconds")
out.shape
# prints the following
# myconv_time = 4.89 seconds
# torch.Size([75, 1, 522, 522])
</code></pre>
<p>My CUDA is as follows: <code>Driver Version: 550.90.07, CUDA: 12.4</code> and I am using a L4 GPU.</p>
<p>The exact same code runs in 0.65 second on average when I run it  in torch <code>2.3.1+cu121</code> version in the same exact environment.</p>

navidsam: <p>Resolved it by upgrading to a higher cuDNN. Reference here: <a href="https://github.com/pytorch/pytorch/issues/146096" class="inline-onebox" rel="noopener nofollow ugc">`torch.nn.functional.conv2d` 8 times slower in torch 2.5.1 compared to 2.3.1 · Issue #146096 · pytorch/pytorch · GitHub</a></p>
Freezing CNN Channels

SudakshK: <p>Hi,<br>
Can I use the indexing for requires_grad function like layer.weight.data[out_idx, in_idx, :, :].requires_grad = False, to set the channels of my choice to be frozen while training and the weights for these channels must not be updated.</p>
<p>Many Thanks in advanced!</p>

anantguptadbl: <p><a class="mention" href="/u/sudakshk">@SudakshK</a>  you can definitely do that</p>
<p>Take a look at this thread for reference</p><aside class="quote quote-modified" data-post="1" data-topic="143252">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/asura/48/28647_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/correct-way-of-freezing-layers/143252">Correct way of freezing layers</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hello, 
I have a model M and I am cloning it M.clone() 
Now, I want to freeze certain layers of M.clone(). When I set requires_grad=False, it results in this error: 
RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach(). 
How to freeze the layers of M.clone() in that case? I want to ensure that when I backpropagate using the loss computed on a batch using M.…
  </blockquote>
</aside>


SudakshK: <p>Hi <a class="mention" href="/u/anantguptadbl">@anantguptadbl</a> thanks for your reply. My problem is little different. I am pruning and rebuilding a model. I want to freeze lets say only one channel of a Conv Layer i.e say the index 0 in the output channels of the Conv layer weights. I am not able to find the solution as the Torch functionality sets the whole tensor gradients to false. One solution I am working on is to freeze the gradients of the tensor to 0 in that particular index. Hence, I don’t want to have whole layer as non-trainable but only a part of that layer.</p>
<p>Thanks in advanced!</p>
FP8 training with torchao but without torchtitan

cokespace2: <p>Hi, I have a training script that trains a customize LLM with FSDP (I’m using PyTorch 2.2, so I cannot utilize FSDP2 for now).</p>
<p>Now I want to use FP8 to accelerate my training procedure, but torchtitan seems only support Llama and FSDP2, then I tried these below:</p>
<pre><code class="lang-auto">from torchao import convert_model_to_float8_training
 ...
model = build_model(config_file)
convert_model_to_float8_training(model)
model = FSDP(model, **kwargs)
dataloader = get_dataloader()
optimizer = optim.AdamW(
    model.parameters(),
    lr=train_config.lr,
    weight_decay=train_config.weight_decay,
)
train(model, dataloder, optimizer)
</code></pre>
<p>I tested FP8 Matmul with <code>torch._scaled_mm</code> and this operator runs fine, but the FP8 trainnng is hanging and I cannot figured out why…</p>
<p>Does anyone has any suggestions?</p>

supriyar: <p>Hi <a class="mention" href="/u/cokespace2">@cokespace2</a><br>
Do you have a repro script with a toy model that you can share? torchao fp8 training should work independent of torchtitan cc <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a></p>

cokespace2: <p>My bad, it failed because my server’s nvlink was off. Nothing wrong with FSDP and float8 training.</p>
Mask when batching sequence

sh0416: <p>Hi,</p>
<p>I want to get masked tensor when I batched the variable length sequences.</p>
<pre><code class="lang-auto">import torch.nn as nn

l = [torch.zeros(7, 4), torch.zeros(8, 4), torch.zeros(9, 4)]
b = nn.utils.rnn.pad_sequence(l, batch_first=True)
mask = # will be torch.BooleanTensor[3, 9] with true for valid input and false for padded input
</code></pre>
<p>Is there any simple implementation for this one?</p>
<pre><code class="lang-auto">mask = b[:, 0] != 0
</code></pre>
<p>This kind of things usually work in some case, but not all the case…</p>
<p>Thanks,</p>

yassineAlouini: <p>What doesn’t work for you? Can you provide any sample of when it fails?<br>
Otherwise, the code snippet you have provided seems fine.</p>
Support for torch.equal when exporting with Dynamo

smisks: <p>Is there a way to export models that call <code>torch.equal</code>? I don’t have access to the source code to rewrite those at the moment.</p>
<p>When I try to export using Dynamo, I get:<br>
<code>Unsupported: data dependent operator: aten.equal.default; to enable, set torch._dynamo.config.capture_scalar_outputs = True</code></p>
<p>However setting <code>torch._dynamo.config.capture_scalar_outputs = True</code> does not seem to have any effect.</p>
<p>Any insight is appreciated!</p>

richard: <p>What is the model with torch.equal() doing with it?</p>

smisks: <p>It’s using GPyTorch, which calls that function in different places. I’m mainly hitting this issue with the <a href="https://github.com/cornellius-gp/gpytorch/blob/da702691b1bdcd7a5962c46af609e1778e6e9f45/gpytorch/kernels/kernel.py#L345" rel="noopener nofollow ugc">kernel</a> classes where they check that the data sets are different during the covariance computation:</p>
<pre><code class="lang-auto">        if last_dim_is_batch:
            x1 = x1.transpose(-1, -2).unsqueeze(-1)
            x2 = x2.transpose(-1, -2).unsqueeze(-1)

        x1_eq_x2 = torch.equal(x1, x2)
        res = None

        if diag:
            # Special case the diagonal because we can return all zeros most of the time.
            if x1_eq_x2:
                return torch.zeros(*x1.shape[:-2], x1.shape[-2], dtype=x1.dtype, device=x1.device)
            else:
                res = torch.linalg.norm(x1 - x2, dim=-1)  # 2-norm by default
                return res.pow(2) if square_dist else res
        else:
            dist_func = sq_dist if square_dist else dist
            return dist_func(x1, x2, x1_eq_x2)
</code></pre>

richard: <p>See <a href="https://github.com/pytorch/pytorch/issues/145970" class="inline-onebox">DataDependentOutputException with aten.equal.default and Dynamo export · Issue #145970 · pytorch/pytorch · GitHub</a>, it’s difficult to support data-dependent control flow</p>
Best practice for integrared vs separate optimizers in actor-critic models

Ebrahim.Pichka: <p>Generally speaking, when we have two networks for Actor and Critic, namely <code>actor_net</code> and <code>critic_net</code> (with no shared modules or layers), which way (described below) is better to perform the optimization in terms of best practices and standards?</p>
<p><strong>1. Separate:</strong></p>
<pre data-code-wrap="py"><code class="lang-py">class ActorCritic:
    def __init__(*args):
        ...
        self.actor_optim = Adam(self.actor_net.parameters(), lr=self.lr_actor)
        self.critic_optim = Adam(self.critic_net.parameters(), lr=self.lr_critic)
        ...

    def learn(*args):
        for step in total_steps:
            ... # collect rollout
            for update in num _updates_per_step:
                ... # forward pass
                actor_loss = ...
                critic_loss = nn.MSELoss(..., ...)

				# backprop for actor network
				self.actor_optim.zero_grad()
				actor_loss.backward(retain_graph=True)
				self.actor_optim.step()

				# backprop for critic network
				self.critic_optim.zero_grad()
				critic_loss.backward()
				self.critic_optim.step()
                ...
</code></pre>
<p>or</p>
<p><strong>2. Integrated:</strong></p>
<pre data-code-wrap="py"><code class="lang-py">class ActorCritic:
    def __init__(*args):
        ...
        self.optim = optim.Adam([
            {'params': self.actor_net.parameters(), 'lr': self.lr_actor},
            {'params': self.critic_net.parameters(), 'lr': self.lr_critic}
        ])
        ...

    def learn(*args):
        for step in total_steps:
            ... # collect rollout
            for update in num _updates_per_step:
                ... # forward pass
                actor_loss = ...
                critic_loss = nn.MSELoss(..., ...)

                # calc total loss
                total_loss = actor_loss +  critic_loss

                self.optim.zero_grad()
                total_loss.backward()
                self.optim.step()
                ...
</code></pre>
<p>First of all, are both logically correct? And if yes, does make a difference which to use? and if so, which is preferred?</p>
<p>I haven’t experimented on it yet, but plan to do so. I wanted to see what the community’s opinion is on it.</p>

vmoens: <p>In torchrl we made some effort to use (2) instead of (1) unless the optimizers had to be called separately (eg, you need 2 different classes or one is called more often than the other)<br>
Example <a href="https://github.com/pytorch/rl/blob/c5f1565de5b066c688fe1291a9eda69ffe6b7477/sota-implementations/sac/sac.py#L112" rel="noopener nofollow ugc">here</a></p>

Ebrahim.Pichka: <p>Thanks! That’s interesting.<br>
I digged a little deeper and looks like SB3 also has done it kinda the same way e.g. <a href="https://github.com/DLR-RM/stable-baselines3/blob/b8b2d30a8399e1448d1f4c1264c343875727f053/stable_baselines3/common/policies.py#L634" rel="noopener nofollow ugc">here</a>.</p>
[torch.export] How to disable sym_size_int?

Sung_Hyuck_Hong: <p>Hello,<br>
I noticed that from torch 2.5,<code> torch.ops.aten.sym_size_int</code> is introduced to improve the coverage of dynamic shape support during torch._dynamo.export tracing.</p>
<p>For example, if the torch code is written as <code>batch_size = x.size()[0]</code>, the corresponding graph module traced through <code>torch._dynamo.export</code> with <code>tracing_mode = "symbolic"</code> resulted in the node<br>
<code>sym_size_int = torch.ops.aten.sym_size.int(l_x_, 0)</code> in torch 2.5, while in torch 2.4 the node was expressed as  <code>size = l_x.size(); getitem_1 = size[0]</code> in torch 2.4.</p>
<p>I am wondering is there a specific way to prevent the occurrence of sym_size_int in torch 2.5. The problem I am encountering is that the sym_size_int is used globally used in all the submodules. For example, given a LLM with N decoder layers, each of which is a submodule of the given LLM and contains the code<code> batch_size = hidden_state.size()[0]; reshaped_hidden_state = hidden_state.view(batch_size, -1)</code>, the captured graph in torch 2.5 contains <code>reshaped_hidden_state_n = hidden_state_n.view(sym_size_int, -1)</code> for all the N layers, n=0,1,…,N-1.</p>
<p>I am wondering if there is a way to express the graph as <code>sym_size_int_n = torch.ops.aten.sym_size.int(hidden_state_n, 0);  reshaped_hidden_state_n = hidden_state.view(sym_size_int_n, -1)</code>, i.e., enforce the calculation of tensor size in each decoder layer.</p>
<p>Thanks in advance!</p>

richard: <p>What’s the problem? That the <code>sym_size_int</code> is being reused for each of the N layers?</p>

Sung_Hyuck_Hong: <p>Yes, I have been implementing functions that transform fx.Graph in a decoder layer-wise manner. The functions work fine with torch 2.4 but not in torch 2.5 due to the issue described above. If this is something that will be enforced in torch &gt; 2.5, I am going to change the implementations but was wondering if there is anyway to revert to the original graph format.</p>
During evaluation, how to apply torch.no_grad only to part of of model?

biggerfish: <p>I have a large model where most of it are just normal structure, but there is one place that I need to manually calculated part of the gradients, something like this</p>
<pre data-code-wrap="python"><code class="lang-python">class Model(nn.Module):
    def __init__(self):
        # many other layers
        self.many_layers = ManyLayers()

    def forward(self, x, y, z):
        fc = self.many_layers(x)
        # Only this little place that I need to manually calculate gradient
        grad = torch.autograd.grad(z, y, torch.ones_like(z))
        grad_out = ManyOtherLayers(grad)
        return SomeOtherComplexLayers(grad_out, fc)
</code></pre>
<p>During evaluation, I want to use <code>torch.no_grad()</code> for performance, but got error</p>
<blockquote>
<p>RuntimeError: element 0 of variables does not require grad and does not have a grad_fn</p>
</blockquote>
<p><strong>How do I enable grad for only this specific small part of tensor while keeping the rest of it having torch.no_grad? I tried to set <code>y.require_grad = True</code>, but it tells me only leave tensor can be modified.</strong></p>

ptrblck: <p>You can use <code>with torch.no_grad()</code> as a context manager and move parts of your forward pass into it, which don’t need to create a computation graph and are thus also not part of the gradient computation.</p>

biggerfish: <p>Thanks, that is one of the solution, but I wonder if there are other ways than manually split the model. Do u know how other frameworks like tensorflow deal with such situation?</p>
DDP with SLURM hangs

Daniel_Anojan: <p>I am getting this error when I train with DDP on a single node with 4 GPUs. The error occurs at random iterations. Also when the process hangs GPU optimization is at 100%.</p>
<p>ProcessGroupNCCL’s watchdog got stuck for 600 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api, or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. workMetaList_.size() = 1</p>

ekomp: <p>I am having a similar issue, though the position during training which the hang occurs is consistent for my script. Is your problem agnostic to the hardware? It only happens for me on H100s, while using A40s has no issues.</p>
Torch operation right after TensorRT's execute_async_v3

J4Q8: <p>I am relatively new to TensorRT so I would like to clarify its intrinsic behaviour when combined with Torch. I understand that <code>execute_async_v3</code> launches the kernel on gpu and gives the control back to python interpreter immediately, hence host and device can work asynchronously. Immediately after executing the kernel I want to perform a pytorch operation e.g. <code>torch.nn.functional.interpolate</code> on the cuda tensors. Do I need to call <code>torch.cuda.synchronise()</code> to ensure that the result of the kernel execution is ready? Or does calling such a function implicitely mean that it will be executed after the first one if it is in the same stream? I ran a small experiment with and without <code>torch.cuda.synchronise()</code> and both results are correct. However, the one without <code>torch.cuda.synchronise()</code> takes only 2ms as compared to 9ms in the other case.</p>
<p>In my example I am using <a href="https://github.com/NVIDIA-AI-IOT/torch2trt/blob/4e820ae31b4e35d59685935223b05b2e11d47b03/torch2trt/trt_module.py#L133" rel="noopener nofollow ugc">torch2trt/torch2trt/trt_module.py at 4e820ae31b4e35d59685935223b05b2e11d47b03 · NVIDIA-AI-IOT/torch2trt · GitHub</a>. Thanks to this excellent repo tensorrt engine outputs the results directly to a torch tensor on cuda. An example of what I would like to do is:</p>
<pre><code class="lang-auto">        # execute
        outputs = [None] * len(self.output_names)
        for i, output_name in enumerate(self.output_names):
            dtype = torch_dtype_from_trt(self.engine.get_tensor_dtype(output_name))
            shape = tuple(self.context.get_tensor_shape(output_name))
            device = torch_device_from_trt(self.engine.get_tensor_location(output_name))
            output = torch.empty(size=shape, dtype=dtype, device=device)
            outputs[i] = output
            self.context.set_tensor_address(output_name, output.data_ptr())

        self.context.execute_async_v3(torch.cuda.current_stream().cuda_stream)

        # here
        torch.nn.functional.interpolate(outputs[0], **kwargs)
        

        if self.output_flattener is not None:
            outputs = self.output_flattener.unflatten(outputs)
        else:
            outputs = tuple(outputs)
            if len(outputs) == 1:
                outputs = outputs[0]

        return outputs
</code></pre>
<p>Thank you in advance!</p>
<h2><a name="p-464920-environment-1" class="anchor" href="#p-464920-environment-1"></a>Environment</h2>
<p><strong>TensorRT Version</strong>: 10.4.0 (docker: dustynv/l4t-pytorch:r36.4.0)<br>
<strong>Pytorch Version</strong>: 2.4.0<br>
<strong>GPU Type</strong>: Tegra<br>
<strong>System</strong>: Jetson Orin AGX</p>
Does PyTorch Already Support the NVIDIA GB200 GPU?

quanta770501: <p>Hi,</p>
<p>Does PyTorch already support the NVIDIA GB200 GPU? I couldn’t find any binary that mentions support for it.</p>
<p>If not, are there any workarounds, such as building PyTorch from source to enable compatibility?</p>
<p>Thanks!</p>

ptrblck: <p>GB200 isn’t released yet so could you describe how a workaround would be used?</p>

quanta770501: <p>Hi,</p>
<p>Thanks for pointing that out! I actually have access to a GB200 and need to conduct some workload research on it.</p>
<p>I’m exploring how to get PyTorch to work with this hardware. Specifically:</p>
<ol>
<li>Would building PyTorch from source with a specific CUDA version be a viable option, assuming the appropriate driver and CUDA are installed?</li>
<li>Since the GB200 might share similarities with the GH200, could it be possible to configure PyTorch to treat it as a GH200 for compatibility?</li>
<li>Are there general guidelines for enabling GPU support for new hardware in PyTorch?</li>
</ol>
<p>Any advice or suggestions on how to proceed would be greatly appreciated.</p>
<p>Thanks in advance!</p>

ptrblck: <p>I would recommend to get in touch with your NVIDIA contact to access software stack for hardware preview which you apparently received.<br>
Official support for GB200 will land after the release of the platform and SW stack.</p>

quanta770501: <p>Thank you so much for your thoughtful response and helpful recommendations. I truly appreciate your help!</p>

sanianagpal: <p>Any further updates on the support for GB200?</p>

ptrblck: <p>Yes, nightly PyTorch binaries are available for Linux x86 and ARM:</p>
<pre data-code-wrap="python"><code class="lang-python">pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128
</code></pre>
Loss stability when training with batch size > 1?

yassineAlouini: <p>I have a training that works well when running with a batch of 1 but that gets “shaky” when running with a bigger batch.<br>
For more context, it is a sequential audio-to-motion model (a bit similar to an NLP transformers model).<br>
What are things that could be tried to “stabilize” the training?<br>
Bigger learning rate? Some batch normalization? Bigger batch?<br>
Any ideas are welcome, thanks in advance.</p>
Wrong nvcc version when compiling plugins

cltexe: <p>I do have a 2.3.1 installation on a conda env in windows with cuda 12.6. I do also have a systemwide 12.6 and 11.x installation. I tried to run stylegan 3 on this environment but getting errors when compiling plugins. It tries to use cuda11.x and throws an error with version mismatch for vstudio 2022 latest update. I just want it to use cuda 12.6 on environment level. On the logs, it goes for systemwide cuda 11.x installation and when I change the filepath of cuda11.x system wide, throws path error not switching to other paths. To be precise:</p>
<blockquote>
<p>RuntimeError: Error building extension ‘bias_act_plugin’: [1/2] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\bin\nvcc --generate-dependencies-with-compile --dependency-output bias_act.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\Users\clt\miniforge3\envs\pytorch_2.3.1\Lib\site-packages\torch\include -IC:\Users\clt\miniforge3\envs\pytorch_2.3.1\Lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\clt\miniforge3\envs\pytorch_2.3.1\Lib\site-packages\torch\include\TH -IC:\Users\clt\miniforge3\envs\pytorch_2.3.1\Lib\site-packages\torch\include\THC “-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\include” -IC:\Users\clt\miniforge3\envs\pytorch_2.3.1\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 --use_fast_math --allow-unsupported-compiler -c C:\Users\clt\AppData\Local\torch_extensions\torch_extensions\Cache\py312_cu121\bias_act_plugin\d197188c6eb43b2685156af41c1c4adc-nvidia-geforce-rtx-3060\bias_act.cu -o bias_act.cuda.o<br>
FAILED: bias_act.cuda.o</p>
</blockquote>
Compile and vmap in custom op with quantile

Space4444: <p>Hi, I’m trying to implement vmap rule for quantile and then compile it. I don’t know how to add batching rules directly via c++, so I created custom operator around the torch.quantile function in python and registered vmap on it. Without compiling it works fine, but compiled function returns a tensor with the same elements in different order.<br>
Torch version: 2.6.0.dev20241121+cu124</p>
<pre><code class="lang-auto">import torch
from torch import Tensor

lib = torch.library.Library('mylib', 'FRAGMENT')

@torch.library.custom_op('mylib::vquantile', mutates_args=())
def vquantile(x: Tensor, q: Tensor, dim: int = -1) -&gt; Tensor:
    return torch.quantile(x, q, dim)

@torch.library.register_fake('mylib::vquantile')
def _(x, q, dim=-1):
    n = q.numel()
    x = x.index_select(dim, torch.zeros(n, dtype=int) ).squeeze(dim)
    return torch.empty_like(x)

@torch.library.register_vmap('mylib::vquantile')
def quantile_vmap(info, in_dims, x, q, dim=-1):
    x = vquantile(x.movedim(in_dims[0], -1), q, dim % (x.ndim - 1) )
    return x, x.ndim - 1

a = torch.arange(10.).reshape(2, 5)
q = torch.tensor([.2, .5, .8])

f1 = torch.vmap(lambda x: vquantile(x, q, -1))
f2 = torch.compile(f1)

r1 = f1(a)
r2 = f2(a)

print(f'a: \n{a}')
print(f'vmapped quantiles: \n{r1}')
print(f'compiled &amp; vmapped quantiles: \n{r2}')
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">a: 
tensor([[0., 1., 2., 3., 4.],
        [5., 6., 7., 8., 9.]])
vmapped quantiles: 
tensor([[0.8000, 2.0000, 3.2000],
        [5.8000, 7.0000, 8.2000]])
compiled &amp; vmapped quantiles: 
tensor([[0.8000, 3.2000, 7.0000],
        [2.0000, 5.8000, 8.2000]])
</code></pre>
<p>P.S. It seems that it is working properly only if I set out_dims=-1 when calling vmap, that is:</p>
<pre><code class="lang-auto">f1 = torch.vmap(lambda x: vquantile(x, q, -1), out_dims=-1)
</code></pre>

richard: <p>Thanks for the report, I filed an issue over at <a href="https://github.com/pytorch/pytorch/issues/146536" class="inline-onebox">vmap x compile silently incorrect · Issue #146536 · pytorch/pytorch · GitHub</a> for more investigation</p>
API domain -- pytorch.sh

rosecloud: <p>Hello – so I own the domain pytorch.sh – does anybody from the organization want to this domain and would trade me for it? I do not want it anymore. It would be great to host build scripts or an api - i have no use for it anymore.</p>

smth: <p>is this an offer for a pure transfer, if so, the pytorch foundation will probably be interested in taking it over.</p>
<p>If it’s a sale, then we’ll flag it to the pytorch foundation, but there’s unlikely an interest to pay a premium price for it.</p>

Matt_White: <p>Hi Frederick,</p>
<p>I’m the Executive Director at the PyTorch Foundation, feel free to DM me to discuss further.</p>
<p>Cheers,</p>
<p>Matt</p>

rosecloud: <p>Hey Matt – not seeing where I can direct message? Can you DM me at <span class="mention">@steamchaszer</span> on twitter? Thank you</p>

rosecloud: <p>Sent you an email and also followed you on twitter.</p>
Compilation error due to use of dropout

Joshua_Pepe: <p>Currently, I am using PyTorch version 2.6.0, but I am encountering an error where—even when applying dropout or normalization—I have to clone tensors. Even when I use</p>
<pre><code class="lang-auto">torch.compiler.cudagraph_mark_step_begin()
</code></pre>
<p>I still need to add the cloning. This is my compilation method:</p>
<pre><code class="lang-auto">torch.compile(self.model, backend="inductor", mode="max-autotune", fullgraph=False)
</code></pre>
<p>I do not know whether this is a bug or if it is intentional in this version. I am aware that the new version added more memory validations, but now even adding dropout or using RMS forces me to clone tensors.</p>

ptrblck: <p>Could you post a minimal code snippet to reproduce the issue, please?</p>
RuntimeError: "rshift_cpu" not implemented for 'Float'

Geremia: <p>How do I implement (overload?) <code>&gt;&gt;</code> for tensors?</p>

anantguptadbl: <p><a class="mention" href="/u/geremia">@Geremia</a> this is working in pytorch</p>
<pre><code class="lang-auto">a=10
print(a &lt;&lt; 2)

import torch
a = torch.tensor(10)
print(a&lt;&lt;2)
</code></pre>
<p>Result</p>
<pre><code class="lang-auto">40
tensor(40)
</code></pre>
<p>Was this your question?</p>

ptrblck: <p>You are using <code>int</code> values, where the bitshift operation is properly defined.<br>
It’s unclear to me what the expected result of a bitshift on floating point numbers is. If you really want to shift the bits, <code>view</code> the tensor in an integer format, shift it, and <code>view</code> it back:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor(1.25)
y = x.view(torch.int32)
print(y)
# tensor(1067450368, dtype=torch.int32)
print(np.binary_repr(y.numpy()))
# 111111101000000000000000000000

y = y &lt;&lt; 1
print(y)
# tensor(2134900736, dtype=torch.int32)
print(np.binary_repr(y.numpy()))
# 1111111010000000000000000000000

z = y.view(torch.float32)
print(z)
# tensor(2.5521e+38)
</code></pre>
<p>but make sure this is really what you want <a class="mention" href="/u/geremia">@Geremia</a></p>
Generating a meshgrid or itertools.product in pytorch

ncullen93: <p>Hi all,</p>
<p>I’m trying to generate a meshgrid directly in pytorch…</p>
<p>The result I’m looking for, given two arrays of [0,1], is this:</p>
<pre><code class="lang-auto">[[0,0],
 [0,1],
 [1,0],
 [1,1]]
</code></pre>
<p>You can do this with <code>np.meshgrid</code>:</p>
<pre><code class="lang-python">    grid = np.meshgrid(range(2), range(2), indexing='ij')
    grid = np.stack(grid, axis=-1)
    grid = grid.reshape(-1, 2)
</code></pre>
<p>and with <code>itertools.product</code> :</p>
<pre><code class="lang-python">grid = list(itertools.product(range(2),range(2)))
</code></pre>
<p>But does anyone reckon how you’d do that directly in pytorch?</p>
<p>Thanks!</p>
<p>EDIT: Hm, this seems to work:</p>
<pre><code class="lang-python">x = torch.Tensor([0,1])
torch.stack([x.repeat(2), x.repeat(2,1).t().contiguous().view(-1)],1)

x = torch.Tensor([0,1,2])
torch.stack([x.repeat(3), x.repeat(3,1).t().contiguous().view(-1)],1)
</code></pre>
<p>Full function for different sizes:</p>
<pre><code class="lang-python">def generate_grid(h, w):
    x = torch.range(0, h-1)
    y = torch.range(0, w-1)
    grid = torch.stack([x.repeat(w), y.repeat(h,1).t().contiguous().view(-1)],1)
    return grid

grid = generate_grid(2,3)
# 0  0
# 1  0
# 0  1
# 1  1
# 0  2
# 1  2
#[torch.FloatTensor of size 6x2]
</code></pre>

miotto: <p>Maybe:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

def generate_grid(h, w):
    x = torch.arange(h)
    y = torch.arange(w)
    return torch.cartesian_prod(x, y)


print(generate_grid(2, 3))
</code></pre>
<p>Which prints:</p>
<pre data-code-wrap="bash"><code class="lang-bash">tensor([[0, 0],
        [0, 1],
        [0, 2],
        [1, 0],
        [1, 1],
        [1, 2]])
</code></pre>
torchvision.transforms.v2.Compose equivalent in C++

L_R: <pre data-code-wrap="python"><code class="lang-python">import torchvision.transforms.v2 as v2
...
        self._input_tensor = v2.Compose(
            [v2.ToImage(), v2.ToDtype(torch.float32, scale=True)]
        )(self._input_img)[None].to(self._device)
</code></pre>
<p>I want to translate this code to C++.<br>
there seems to be no torchvision.transforms.compose functions in libtorch? I’m not sure about this.<br>
my code is like</p>
<pre data-code-wrap="C++"><code class="lang-C++">  cv::Mat input_img = ...;
  ...
  torch::Tensor input_tensor =
      torch::from_blob(input_img.data,
                       {input_img.rows, input_img.cols, input_img.channels()},
                       torch::kUInt8)
          .permute(
              {2, 0, 1})  // the image format of cv::Mat is channel-last,
                          // transfer it to channel-first format of torch tensor
          .to(torch::kFloat32)
          .div(255.0F)    // scale from [0, 255] to [0, 1]
          .unsqueeze(0);  // add a batch dimension
</code></pre>
<p>but it is much longer than using <code>Compose</code><br>
besides, I don’t know if there is a <code>scale = True</code> equivalent in C++?<br>
so I divided it by 255, but it is not strictly from 0 to 1, it is just in [0, 1]</p>

L_R: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Could I @ you to answer this?</p>
Example implementation of a variational autoencoder

Mahmoud_Abdelkhalek: <p>I am a bit unsure about the loss function in the <a href="https://github.com/pytorch/examples/blob/master/vae/main.py" rel="noopener nofollow ugc">example implementation</a> of a VAE on GitHub.</p>
<p>The <a href="https://en.wikipedia.org/wiki/Evidence_lower_bound" rel="noopener nofollow ugc">evidence lower bound</a> (ELBO) can be summarized as:</p>
<p>ELBO = log-likelihood - KL Divergence</p>
<p>And in the context of a VAE, this should be maximized. However, since PyTorch only implements gradient descent, then the negative of this should be minimized instead:</p>
<p>-ELBO = KL Divergence - log-likelihood</p>
<p>However, in the <a href="https://github.com/pytorch/examples/blob/a74badde33f924c2ce5391141b86c40483150d5a/vae/main.py#L73" rel="noopener nofollow ugc">loss function</a> in the code, the loss is defined as:</p>
<p>Binary Cross Entropy + KL Divergence</p>
<p>According to the documentation for the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html" rel="noopener nofollow ugc">BCE loss</a>, it actually implements the <strong>negative</strong> log-likelihood function of a Bernoulli distribution, which means that:</p>
<p>BCE + KLD = KLD - log-likelihood</p>
<p>Which is the same as what was derived above. Is this why the loss is defined in this way in the code?</p>
<p>Also, does the cross-entropy loss function also implement a <strong>negative</strong> log-likelihood function?</p>

InfT: <p>I have the same problem, I don’t know which form is the most correct.</p>

InnovArul: <aside class="quote no-group" data-username="Mahmoud_Abdelkhalek" data-post="1" data-topic="103427">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/8491ac/48.png" class="avatar"> Mahmoud_Abdelkhalek:</div>
<blockquote>
<p>Also, does the cross-entropy loss function also implement a <strong>negative</strong> log-likelihood function?</p>
</blockquote>
</aside>
<p>In my understanding, BCE implements negative log-likelihood for 2 classes, and CrossEntropy implements it for multiple classes.</p>

franciscocms: <p>Did you reach a conclusion about this problem? I am facing the same issue… thank you in advance!</p>

cpu1100: <p>I believe the issue is now resolved in the latest implementation. Could you please take a look (see your own link to VAE example) and confirm that its correct?</p>
How to customize a quantization algorithm and deploy it?

BambooKui: <p>Excuse me, how can I implement some quantization algorithms in PyTorch? Should I use torch.fx or torch.ao? Are there any relevant tutorials? And how can I deploy the model quantized with this quantization algorithm to hardware? I have tried some quantization tools, but there often exist problems of operator incompatibility.</p>

anantguptadbl: <p><a class="mention" href="/u/bambookui">@BambooKui</a>  there are several articles that touch upon the use-cases for torch.fx</p>
<p><a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_quant_guide.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/prototype/fx_graph_mode_quant_guide.html</a><br>
<a href="https://pytorch.org/docs/stable/quantization.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/quantization.html</a></p>
<p>For quantization using torch.fx<br>
<a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_quant_guide.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/prototype/fx_graph_mode_quant_guide.html</a></p>
<p>For quantization using torch.ao<br>
<a href="https://pytorch.org/docs/stable/quantization.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/quantization.html</a></p>
<p>If you do not have a requirement to trace your DAG flow, i would recommend going ahead with torch.ao since it is present in the stable version</p>

BambooKui: <p>Thank you very much!</p>
Looking for realiable pytorch code base finetune stable diffusion

hiru: <p>I am working on adding a custom module to stable diffusion model and training the model as a whole. But <a href="https://github.com/CompVis/stable-diffusion" rel="noopener nofollow ugc">official github for SD</a> doesnot provide a training code. Is there any simplified version of SD/<a href="https://github.com/huggingface/diffusers/tree/main" rel="noopener nofollow ugc">diffusers</a> community use?</p>