akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Flex attention and SDPA output natively equivalent?

username: lhallee
cooked: <p>Hello,</p>
<p>Really interested in flex attention but still do not fully understand it yet. Have had great luck speed running <a href="https://github.com/KellerJordan/modded-nanogpt" rel="noopener nofollow ugc">GPT2</a> or <a href="https://github.com/Synthyra/SpeedRunningESM2" rel="noopener nofollow ugc">ESM2</a> with it and want to continue using it during pretraining. As I understand it the best way to use flex attention is essentially with “batch size” 1 and flattening the entire input with particular block masks.</p>
<p>However, the ease of use with F.sdpa and typical pipelines with regular batched inputs make F.sdpa an attractive choice for inference via automatic flash attention, etc. What I’m wondering is if outputs of flex attention and F.sdpa are actually equivalent, or very close, and if not, what expected corrections may be possible to get training with flex attention and inference with sdpa?</p>
<p>Thanks,<br>
Logan</p>

username: soulitzer
cooked: <p>I’d expect you to be able to use regular batched inputs. Maybe the examples here would be helpful <a href="https://github.com/pytorch-labs/attention-gym/blob/main/examples/flex_attn.ipynb" class="inline-onebox" rel="noopener nofollow ugc">attention-gym/examples/flex_attn.ipynb at main · pytorch-labs/attention-gym · GitHub</a></p>

username: lhallee
cooked: <p>Looks like there are some correctness checks between flex and SDPA in the file, which is helpful. Thanks for sharing!</p>
Methods for checking autograd behavior

username: gideonsimpson
cooked: <p>This is a follow up my <a href="https://discuss.pytorch.org/t/checking-what-autodiff-is-differentiating/213549">earlier post</a>.  Here is an example of the sort of thing I want to consider.  Consider the scalar case, where<br>
we have:<br>
f(x) = w2 ( z1(w1 x+b1)))+b2<br>
and<br>
z1 = exp(w1 x+b1)</p>
<p>A naive implementation would be:</p>
<pre><code class="lang-auto">class ScalarNN1(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(1, 1)
        self.layer2 = nn.Linear(1, 1)

    def forward(self, x):
        x1 = self.layer1(x)
        z = torch.exp(x1)
        y = self.layer2(z * x1)
        return y
</code></pre>
<p>When I do compute gradients with respect to the parameters <code>w1</code> and <code>b1</code> (in <code>layer1</code>), the chain rule will back propagate through <code>z</code>.  Alternatively,  suppose I construct:</p>
<pre><code class="lang-auto">class ScalarNN2(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(1, 1)
        self.layer2 = nn.Linear(1, 1)

    def forward(self, x):
        x1 = self.layer1(x)
        with torch.no_grad():
            z = torch.exp(x1) # this could also be done with detach
        y = self.layer2(z * x1)
        return y
</code></pre>
<p>Because of the <code>torch.no_grad()</code>, there will be no back propagation through the <code>z</code> variable.</p>
<p>I consider these two cases because I am considering training routines where different parts of the network are trained differentially, and I want to be able to only take gradients through certain intermediates.  While I can check this example by hand, I was hoping to find a way of checking this more generally.  I. had hoped to be able to use something graphical like <code>torchview</code>, but that doesn’t seem to do it.</p>
<p>Any suggestions would be appreciated.</p>

username: soulitzer
cooked: <p>Are you looking for something like torchviz <a href="https://github.com/szagoruyko/pytorchviz" class="inline-onebox" rel="noopener nofollow ugc">GitHub - szagoruyko/pytorchviz: A small package to create visualizations of PyTorch execution graphs</a>?</p>

username: soulitzer
cooked: <p>Another thing that you can do is set the environment variable <code>TORCH_LOGS="+autograd" </code></p>
<p>e.g., <code>TORCH_LOGS="+autograd"  python test.py</code></p>
<p>This would make autograd log out the nodes the backward is computing</p>

username: gideonsimpson
cooked: <p>Yes, this seems to be extracting the information that I want.</p>
Dataloader is slow with sampling TensorDataset

username: bakhil
cooked: <p>It looks like the standard dataloader is pretty slow when we have small datasets that can fit in (CPU/GPU) memory. As an example, consider the following:</p>
<pre><code class="lang-auto">data_matrix = torch.randn(2000, 1000, 4)
# 2000 samples each with [1000x3] features and 1000-length targets 
dataset = TensorDataset(data_matrix[..., :-1], data_matrix[..., -1])
dataloader = DataLoader(dataset, batch_size=128, shuffle=True, pin_memory=True)
</code></pre>
<p>When I measure the time for sampling from the dataloader, it takes around 3-5 ms:</p>
<pre><code class="lang-auto">%%time
data_iter = iter(dataloader)
features_loader, targets_loader = next(data_iter)
print(features_loader.shape, targets_loader.shape)
</code></pre>
<p>But directly accessing the samples is much faster at 1-2 ms:</p>
<pre><code class="lang-auto">%%time
indices = torch.randint(0, 2000, (128,))
features_direct, targets_direct = data_matrix[indices][..., :-1], data_matrix[indices][..., -1]
print(features_direct.shape, targets_direct.shape)
</code></pre>
<p>Is there a way to write some custom dataloader which can sample as fast? Increasing <code>num_workers</code> in dataloader is making it way worse (more than 40 ms). I want to use the <code>DataLoader</code> class because it can be more easily integrated with other workflows.</p>

username: qq-me
cooked: <p>Hi, I had the same issue when training tiny models on a mnist1d preloaded onto GPU, and pytorch dataloader took most of the time due to how fast the training was.</p>
<p>I’ve made a custom dataloader to remedy this, and just published it as a library <a href="https://github.com/inikishev/light-dataloader" class="inline-onebox" rel="noopener nofollow ugc">GitHub - inikishev/light-dataloader: Much faster dataloader for datasets that are fully loaded into memory</a>. On my laptop and it is 9 times faster than TensorDataset + DataLoader</p>
<p>Also I found another fast library for in-memory datasets <a href="https://github.com/zhb2000/tensorloader" class="inline-onebox" rel="noopener nofollow ugc">GitHub - zhb2000/tensorloader: A faster dataloader for tensor data.</a> which does the same thing</p>
Does the license allow to copy/modify code and how to properly credit pytorch?

username: jeremyfix
cooked: <p>Hello,</p>
<p>I’m developing <a href="https://github.com/jeremyfix/torchcvnn" class="inline-onebox" rel="noopener nofollow ugc">GitHub - jeremyfix/torchcvnn: Pytorch layers for Complex Valued Neural Networks</a> which is providing complex valued building blocks for pytorch.</p>
<p>I’m currently working on porting the transformer in the complex domain based on work by Eilers et al. 2023.</p>
<p>Most of the transformer implementation is exactly the same between the complex valued domain and the real valued domain, only the <code>multi_head_attention_forward</code> differs.</p>
<p>Is it possible to copy the code that is not expected to change from pytorch ? How should I properly credit that code ?</p>
<p>For now, I add in the docstrings, sentences such as “This class is adapted from pytorch TransformerEncoderLayer”  <a href="https://github.com/jeremyfix/torchcvnn/blob/a5ead4a91f43dfec2792d7721886332ecfde7124/src/torchcvnn/nn/modules/transformer.py#L39" class="inline-onebox" rel="noopener nofollow ugc">torchcvnn/src/torchcvnn/nn/modules/transformer.py at a5ead4a91f43dfec2792d7721886332ecfde7124 · jeremyfix/torchcvnn · GitHub</a></p>
<p>Thanks for your help,</p>
<p>Best;</p>
How to categorize tensor float values into long values?

username: Omaralmaqtari
cooked: <p>I am working on a multiclass segmentation problem. My target tensors are of the shape ([1,256,256]), with the unique values ([0.0000, 0.0039, 0.0078]). However, I want to convert the float values into long values as ([0, 1, 2]). When I use <code>.long()</code>, all values turn to zeros.<br>
below I have an example of my tensors:</p>
<blockquote>
<p>([[0.0039, 0.0039, 0.0039, 0.0000, 0.0000],<br>
[0.0039, 0.0039, 0.0039, 0.0000, 0.0000],<br>
[0.0039, 0.0039, 0.0039, 0.0000, 0.0000],<br>
[0.0000, 0.0078, 0.0078, 0.0078, 0.0000],<br>
[0.0000, 0.0078, 0.0078, 0.0078, 0.0000]])</p>
</blockquote>
<p>The desired output should be like the following:</p>
<blockquote>
<p>([[1, 1, 1, 0, 0],<br>
[1, 1, 1, 0, 0],<br>
[1, 1, 1, 0, 0],<br>
[0, 2, 2, 2, 0],<br>
[0, 2, 2, 2, 0]])</p>
</blockquote>

username: soulitzer
cooked: <p>That is expected behavior for .long(), you may want to do something more complicated like the following:</p>
<pre><code class="lang-auto">import torch

def convert_float_to_labels(x):
    # Create a mapping tensor for the exact float values
    mapping = torch.tensor([0.0000, 0.0039, 0.0078])
    
    # Find the closest value for each element
    # This handles floating point precision issues
    distances = torch.abs(x.unsqueeze(-1) - mapping)
    indices = torch.argmin(distances, dim=-1)
    
    return indices

# Example usage
x = torch.tensor([
    [0.0039, 0.0039, 0.0039, 0.0000, 0.0000],
    [0.0039, 0.0039, 0.0039, 0.0000, 0.0000],
    [0.0039, 0.0039, 0.0039, 0.0000, 0.0000],
    [0.0000, 0.0078, 0.0078, 0.0078, 0.0000],
    [0.0000, 0.0078, 0.0078, 0.0078, 0.0000]
])

result = convert_float_to_labels(x)
print(result)
</code></pre>
CrossEntropy Issue

username: bruhnugget-nice
cooked: <p>Alright, so I am trying to train a ViT using CrossEntropyLoss like this:</p>
<pre><code class="lang-auto">import torch.nn.functional as F

# Train function
def train_combined(model, train_loader, criterion, optimizer, epochs=30):
  model.train()
  for epoch in range(epochs):
    running_loss=0.0
    correct=0
    total=0
    for image, label in train_loader:
      image, label= image.to(device), label.to(device)
      optimizer.zero_grad()

      fwd_output=model(image)
      loss=criterion(fwd_output, label)

      loss.backward()
      optimizer.step()

      # Calculate accuracy
      _, predicted = torch.max(fwd_output.data, 1)
      total += label.size(0)
      correct += (predicted == label).sum().item()

    train_acc = 100 * correct / total
    print(f"Epoch [{epoch+1}/{epochs}], Loss: {running_loss / len(train_loader):.4f}, Accuracy: {train_acc:.2f}%")
</code></pre>
<p>When I do this, I get this message:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-64-79153cd7a2a3&gt; in &lt;cell line: 1&gt;()
----&gt; 1 train_combined(new_vit_pathmnist, val_pathmnist_dataloader, loss_fn, optimizer_fn)

4 frames
/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   3477     if size_average is not None or reduce is not None:
   3478         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 3479     return torch._C._nn.cross_entropy_loss(
   3480         input,
   3481         target,

RuntimeError: 0D or 1D target tensor expected, multi-target not supported
</code></pre>
<p>How can I fix this?</p>

username: ptrblck
cooked: <p>Your target is supposed to contain class indices and have the same shape as the model output without the class dimension. Based on the error message it seems your target contains an additional dimension.</p>
How to save model state in pytorch fsdp

username: qibin0506
cooked: <p>I am using PyTorch’s FSDP. How can I save the completed model state every 100 batches? Is the following code correct?</p>
<pre data-code-wrap="python"><code class="lang-python">if (batch + 1) % 100 == 0:
    if isinstance(model, FSDP):
        states = model.state_dict()
        if is_main_process:
            ckpt = {'model_state_dict': states}
            torch.save(ckpt, 'ckpt.pth')
    else:
        ckpt = {'model_state_dict': model.state_dict()}
        torch.save(ckpt, 'ckpt.pth')

</code></pre>
<p>or method 2?</p>
<pre data-code-wrap="python"><code class="lang-python">            with FSDP.summon_full_params(
                    module=model,
                    rank0_only=True,
                    writeback=False,
                    offload_to_cpu=True
            ):
                states = model.state_dict()
                ckpt = {'model_state_dict': states}
                torch.save(ckpt, 'ckpt.pth')
</code></pre>

username: H-Huang
cooked: <p>Distributed Checkpoint (DCP) is the recommended utility for saving models parallelized with FSDP, TP, etc.</p>
<p><a href="https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html#how-to-use-dcp" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html#how-to-use-dcp</a></p>

username: qibin0506
cooked: <p>After reviewing the document, I still have some questions.<br>
Why do I need to use both <code>dcp.load</code> and <code>model.load_state_dict</code> if I want to load all parameters into a non-FSDP model?<br>
Will <code>dcp.load</code> take effect in this case? Does <code>model.load_state_dict</code> load all the parameters?</p>
<pre data-code-wrap="python"><code class="lang-python">def run_checkpoint_load_example():
    # create the non FSDP-wrapped toy model
    model = ToyModel()
    state_dict = {
        "model": model.state_dict(),
    }

    # since no progress group is initialized, DCP will disable any collectives.
    dcp.load(
        state_dict=state_dict,
        checkpoint_id=CHECKPOINT_DIR,
    )
    model.load_state_dict(state_dict["model"])
</code></pre>
<p><a href="https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html#how-to-use-dcp" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html#how-to-use-dcp</a></p>
Behavior of wait() on async CUDA collectives

username: x41lakazam
cooked: <p>I am confused with understanding the behavior of <code>wait()</code> on async CUDA collectives.</p>
<p>As described in <a href="https://pytorch.org/docs/stable/distributed.html#synchronous-and-asynchronous-collective-operations" rel="noopener nofollow ugc">torch distributed documentation</a>:</p>
<blockquote>
<ul>
<li><code>wait()</code> - […] In the case of CUDA collectives, will block until the operation has been successfully enqueued onto a CUDA stream and the output can be utilized on the default stream without further synchronization.</li>
</ul>
</blockquote>
<p>From that paragraph, I understand that <code>wait()</code> don’t wait for the completion of the collective, but only waits for the operation to be enqueued.</p>
<p>However from experiences I’ve done it seems that it does block the other collectives from starting. For example in this code:</p>
<pre><code class="lang-auto">r1 = dist.all_to_all_single(dst, src, group=pg1, async_op=True)
r1.wait()
r2 = dist.all_to_all_single(dst, src, group=pg2, async_op=True)
r2.wait()
</code></pre>
<p>I would expect the <code>wait()</code> to only enqueue the collective to the stream, then for the second collective to be enqueued too and start running in parallel (as they are on different PGs). However the second collective does wait for the first one to finish execution.</p>
<p>So what is the real behaviour of <code>wait()</code> at the GPU level ? Will all the collectives that are posted after if wait for all the collectives that are posted before, even if they are on a different PG ? By posted I mean calling the <code>dist.collective()</code> function like <code>dist.all_to_all_single</code>.</p>

username: H-Huang
cooked: <p>All process groups enqueue collectives under the same stream by default, so that is why the second collective always comes after the first. If they were on different streams you would see different behavior.</p>
Mps large tensor handling bug

username: Nadav_Sellam
cooked: <p>I have seem to stumble into an MPS bug.<br>
I ran this experiment on Macbook Pro M3 max, and found some very wired behavior.<br>
a = torchl.rand(55987200) &gt; 0.5<br>
a[a].all() → True<br>
a = torchl.rand(55987200, device=“mps”) &gt; 0.5<br>
a[a].all() → False</p>
<p>it seems that a[a] will have the correct shape on both devices, so it seemse the comparison part of the code is correct, but when copying to the large buffer on mps, there seems to be a bug, because a.nonzero() will have many repetitions.</p>
<p>did anyone else notice this bug ?</p>
To detach or not to detach

username: a2jinhee
cooked: <p>I know that <code>detach()</code> is used for detaching a variable from the computational graph. In that context, are the following expressions <code>x = x - torch.mean(x, dim=0).detach()</code> and <code>x = x - torch.mean(x, dim=0)</code> equivalent? I just want to subtract the mean out, don’t want to pass gradients through the average calculation.</p>
<blockquote>
<blockquote>
<p>Solution: No, in second case you’ll have additional trainable shared term. E.g. for two element vector, you’ll have xout[0] = x[0] - (x[0]+x[1])/2, so xout[0] gradient will affect x[1].</p>
</blockquote>
</blockquote>
<p>The above discussion was closed, but I want to expand on this question a little bit.</p>
<p>I understand that the two cases will differ, but how exactly does adding <code>.detach()</code> to the second term affect training? I assume it depends on the gradient of <code>xout</code>, but I’m curious about how the parameter adjusts in each scenario:</p>
<ol>
<li>When subtracting the mean without <code>.detach()</code></li>
<li>When subtracting the mean with <code>.detach()</code></li>
</ol>
<p>Generally, subtracting the mean of a tensor centers it around zero, but how does using <code>.detach()</code> influence this behavior?</p>

username: soulitzer
cooked: <p>Subtracting the mean during forward also results in the gradient being normalized this way during backward. If the detach() is done, then no normalization is done during backward on the gradient.</p>
Advanced Indexing Not Update Values as Expected?

username: chouakifares
cooked: <p>I’m encountering a confusing behavior when using advanced indexing to update values in a tensor. Here’s an example:</p>
<pre><code class="lang-auto">import torch  
A = torch.zeros(2, 2)  
Inds = torch.tensor([[0,0,1],[1,1,1]])
A[Inds.unbind()] += torch.tensor([1, 0, 1])  
</code></pre>
<p>I expected <code>A[0, 1]</code> to increment by 1, but its value remains <code>0</code> while the value of <code>A[1,1]</code> is incremented by 1. At first I thought that this was related to the in-place operation. I tried to use an explicit addition as shown bellow:</p>
<pre><code class="lang-auto">A[Inds.unbind()] = A[Inds.unbind()] + torch.tensor([1, 0, 1])  
</code></pre>
<p>After this operation, <code>A[0, 1]</code> is still <code>0</code> whereas <code>A[1,1]</code> equals 2 (which the corredct value since it was equal to 1 before).</p>
<p>My current fix is done using a loop over the values of Inds but I would like to know why does this happen and if there is a way to vecotrize this instead?</p>

username: soulitzer
cooked: <p>The behavior this usage of advanced indexing is documented here <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html" class="inline-onebox" rel="noopener nofollow ugc">Tensor Indexing API — PyTorch main documentation</a> to be equivalent to doing index_put_. And index_put_ eis undefined when indices repeat and accumulate=False. To get the behavior you want, it sounds like you want to set accumulate=True?</p>
<pre><code class="lang-auto">&gt;&gt;&gt; A.index_put_(inds.unbind(), torch.tensor([1., 0., 1.]), accumulate=True)
tensor([[0., 1.],
        [0., 1.]])
</code></pre>
Cpp torch.sparse usage?

username: Tanay_Arora
cooked: <p>Is there any way to use torch.sparse.mm() in the cpp api? I have been aimlessly wandering through the source code trying to find a way and am not getting very far.</p>
<p>I am currently implementing a 2d convolution with sparse weights and dense input, and find that batch-wise computing is too slow with python loops (hence my cpp use).</p>

username: ptrblck
cooked: <p>The <a href="https://github.com/pytorch/pytorch/blob/3df12d38cfef837b714c106cca5a80a0be413da6/torch/csrc/autograd/FunctionsManual.cpp#L1453-L1484">unit tests</a> could be a good starter showing how sparse methods can be used in the C++ backend, e.g. <code>at::sparse_sampled_addmm</code>.</p>
BatchNorm not fusing with Cone and ReLU

username: PROMIT_HALDAR
cooked: <p>I am trying to fuse Conv, batch norm and relu layers in a Resnet18 to prepare for QAT.<br>
However after I run this:</p>
<p>model.eval()</p>
<p>torch.ao.quantization.fuse_modules(model, [[“conv1”,“bn1”,“relu”]], inplace=True)</p>
<p>and see the model now:</p>
<p>ResNet(<br>
(conv1): ConvReLU2d(<br>
(0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3))<br>
(1): ReLU(inplace=True)<br>
)<br>
(bn1): Identity()<br>
(relu): Identity()</p>
<p>This is how it looks, the conv and relu layers are getting fused, but batch norm layer is not fused.</p>
<p>Please let me know the reason behind it.<br>
Thanks</p>
Torch operation time measurement using benchmark.Timer

username: sakura1
cooked: <p>When I want to measure the torch operation execution time by using torch.utils.benchmark.Timer</p>
<pre><code class="lang-auto">    globals = {
        "a": a,
        "b": b,
    }

    results.append(
        benchmark.Timer(
            stmt="torch.matmul(a, b)",
            globals = globals,
            sub_label=sub_label1,
            description="pytorch_gemm",
        ).blocked_autorange(min_run_time=1))

    compare = benchmark.Compare(results)
    compare.print()
</code></pre>
<p>result:</p>
<pre><code class="lang-auto">Times are in microseconds (us).

[-------------------  ------------------]
                          |  pytorch_gemm
1 threads: ------------------------------
      MKN=(10x4096x6144)  |      58.7    

Times are in microseconds (us).
</code></pre>
<p>The precision of the results is only displayed to one decimal place. How can I increase the displayed precision? like 58.7654?</p>

username: soulitzer
cooked: <p>You can use <code>compare.trim_significant_figures()</code> but it won’t necessarily give you more digits of precision, but that also should mean that there’s too much noise for the extra digits to be meaningful. The formatting determines the number of significant digits based on the confidence intervals from the measurements.</p>
<p>Without trim_significant_figures, as you observe, you will always have a single digit after the decimal.</p>

username: sakura1
cooked: <aside class="quote no-group" data-username="soulitzer" data-post="2" data-topic="214601">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/48.png" class="avatar"> soulitzer:</div>
<blockquote>
<p>compare.trim_significant_figures()</p>
</blockquote>
</aside>
<p>Thank you very much for your reply. After testing, I found that this function does not improve display precision but rather decreases precision. For example, the following results show:<br>
before：<br>
MKN=(128x4096x6144)  |        80.6       |     66.3     |      65.5     |  149   |  107<br>
after：<br>
MKN=(128x4096x6144)   |         81        |      66      |       66      |  149   |  107</p>
<p>So I understand that there is no universal algorithm to improve display accuracy because the results are calculated internally by themselves. Is that correct？</p>

username: soulitzer
cooked: <blockquote>
<p>So I understand that there is no universal algorithm to improve display accuracy because the results are calculated internally by themselves. Is that correct？</p>
</blockquote>
<p>Yeah the results measured are not precise enough for the extra digits shown to be useful</p>
TF32 flags when using AMP

username: CDhere
cooked: <p>Hi! I’m trying out TF32 and mixed precision training. Can they be used at the same time? Namely, is setting the flags like this ok when I enable mixed precision training, or should I actually set them back to <code>False</code>?</p>
<pre><code class="lang-auto">torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True
</code></pre>
<p>Thanks!</p>

username: ptrblck
cooked: <p>You can keep TF32 enabled and would use it for convs and matmuls outside of the <code>amp.autocast</code> region.</p>

username: CDhere
cooked: <p>Just to check my understanding, the flags would only affect things outside the <code>amp.autocast</code> context?</p>

username: ptrblck
cooked: <p>Yes, since inside the <code>autocast</code> context (assuming it’s enabled) <code>float16</code> or <code>bfloat16</code> ops will be used for TensorCore-eligible operations.</p>

username: Makor
cooked: <p>would the tf32 affect training, if wrap the all training in autocast but enabled=False</p>

username: ptrblck
cooked: <p>If you are not enabling <code>autocast</code> then TF32 operations can be used according to their setting.</p>
How to calculate the mean and the std of cifar10 data

username: Zidane
cooked: <p>Pytorch is using the following values as the mean and std for the cifar10 data: transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))</p>
<p>I need to understand the concept behind calculating it because this data is 3 channel image and I do not understand what is summed and divided over what and so on. Also if someone can share a code for calculating the mean and the std, would be so thankful.</p>

username: Farid_Alijani
cooked: <p>Originally obtained from SOF <a href="https://stackoverflow.com/a/69699979/5437090" rel="noopener nofollow ugc">CIFAR_mean_std</a> :</p>
<pre><code class="lang-auto">from  torchvision import datasets
cifar_trainset = datasets.CIFAR10(root='./data', train=True, download=True  )
data = cifar_trainset.data / 255 # data is numpy array
mean = data.mean(axis = (0,1,2)) 
std = data.std(axis = (0,1,2))
print(f"Mean : {mean}   STD: {std}") #Mean : [0.491 0.482 0.446]   STD: [0.247 0.243 0.261]
</code></pre>
Calculating the Jacobian of gradients w.r.t to true output

username: zisissour
cooked: <p>Hello,</p>
<p>I’m trying to calculate the Jacobian of the gradients of a model w.r.t. its true output.</p>
<p>Basically, I need to calculate this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/0517e389e03f091e9b23ac145313b45edb26a3c8.png" alt="image" data-base62-sha1="J3zgVdJ4bNEeJdYSfwTPb9oFqE" width="313" height="63"></p>
<p>It is mentioned in Mo et al., 2021 (<a href="https://arxiv.org/abs/2010.08762" class="inline-onebox" rel="noopener nofollow ugc">[2010.08762] Layer-wise Characterization of Latent Information Leakage in Federated Learning</a>), in chapter 2. There is no official implementation, so I don’t know how to approach this.</p>
<p>Could anyone please help out?</p>

username: qq-me
cooked: <p>This can be done with torch.autograd.grad, although I don’t know how it compares performance wise to torch.func</p>
<p>Those functions calculate jacobian of a sequence of tensors w.r.t another sequence of tensors. Two versions, normal and batched. Batched implementation corresponds to the <strong>is_grads_batched</strong> argument explained in <a href="https://pytorch.org/docs/stable/generated/torch.autograd.grad.html#torch.autograd.grad" class="inline-onebox" rel="noopener nofollow ugc">torch.autograd.grad — PyTorch 2.5 documentation</a>, basically its faster but experimental.</p>
<pre data-code-wrap="py"><code class="lang-py">from collections.abc import Sequence, Iterable
import torch

def jacobian(input: Sequence[torch.Tensor], wrt: Sequence[torch.Tensor], create_graph=False):
    flat_input = torch.cat([i.reshape(-1) for i in input])
    grad_ouputs = torch.eye(len(flat_input), device=input[0].device, dtype=input[0].dtype)
    jac = []
    for i in range(flat_input.numel()):
        jac.append(torch.autograd.grad(
            flat_input,
            wrt,
            grad_ouputs[i],
            retain_graph=True,
            create_graph=create_graph,
            allow_unused=True,
            is_grads_batched=False,
        ))
    return [torch.stack(z) for z in zip(*jac)]

def batched_jacobian(input: Sequence[torch.Tensor], wrt: Sequence[torch.Tensor], create_graph=False):
    flat_input = torch.cat([i.reshape(-1) for i in input])
    return torch.autograd.grad(
        flat_input,
        wrt,
        torch.eye(len(flat_input), device=input[0].device, dtype=input[0].dtype),
        retain_graph=True,
        create_graph=create_graph,
        allow_unused=True,
        is_grads_batched=True,
    )
</code></pre>
<p>Here is an example of calculating the Jacobian of the gradients of a model w.r.t. its output:</p>
<pre data-code-wrap="py"><code class="lang-py">import torch
model = torch.nn.Sequential(torch.nn.Linear(2,3), torch.nn.ReLU(), torch.nn.Linear(3,2))

outputs = model(torch.randn(2))

loss = (outputs ** 2).mean()

grad = jacobian([loss], list(model.parameters()), create_graph = True)

jac_wrt_output = jacobian(grad, [outputs])[0]
</code></pre>
RuntimeError: INTERNAL ASSERT FAILED

username: xiaoshendouXX
cooked: <p>I use torch=2.1.2, torchvision=0.16.2, torchaudio=2.0.2, cuda=12.1.<br>
When I used ‘deepspped’ to pretrain LLaVA with 6 RTX4090, the error was shown as image<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/1951992c26eafd7c6d6cb7cb19492e418f70568c.png" data-download-href="https://discuss.pytorch.org/uploads/default/1951992c26eafd7c6d6cb7cb19492e418f70568c" title="屏幕截图 2024-12-25 165436"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/1951992c26eafd7c6d6cb7cb19492e418f70568c_2_690x310.png" alt="屏幕截图 2024-12-25 165436" data-base62-sha1="3BYL8T6gwe7z2PZh7SGrHTZT7Fa" width="690" height="310" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/1951992c26eafd7c6d6cb7cb19492e418f70568c_2_690x310.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/1951992c26eafd7c6d6cb7cb19492e418f70568c_2_1035x465.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/1951992c26eafd7c6d6cb7cb19492e418f70568c_2_1380x620.png 2x" data-dominant-color="0F0E0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">屏幕截图 2024-12-25 165436</span><span class="informations">1514×681 210 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
What can I do to solve it?</p>

username: ptrblck
cooked: <p>Do you encounter the same error in the latest stable or nightly release?</p>
Smooth Sampling Rate Adjustment for Different Datasets

username: syt-nju
cooked: <h1><a name="p-462070-variable-weighted-sampling-for-different-datasets-1" class="anchor" href="#p-462070-variable-weighted-sampling-for-different-datasets-1"></a>Variable Weighted Sampling for Different Datasets</h1>
<p>When training a language model, we use multiple datasets concurrently. We aim to adjust the sampling ratios of different datasets as training progresses. Ideally, this should support a nested structure where weights for a category of datasets can be modified at each level, with random sampling conducted layer by layer. It is not required that sampling without replacement be enforced globally; rather, we only require that within each sub-dataset, no replacement occurs until a full traversal has been completed.</p>
<p>Is there an existing solution for this?</p>
Compile Model with TensorRT

username: shbkukuk
cooked: <p>Hello folks, i am working on Image to Image translation task. I trained <a href="https://github.com/NVlabs/I2SB" rel="noopener nofollow ugc">I2SB model</a>. I want to optimize with TensorRT so i follow instruction on given <a href="https://developer.nvidia.com/blog/accelerating-inference-up-to-6x-faster-in-pytorch-with-torch-tensorrt/" rel="noopener nofollow ugc">this page</a>. When run compile i get error.</p>
<pre><code class="lang-auto">RuntimeError: View operation returned a tensor that is the same as the input base tensor.  This is no longer allowed; you must explicitly create a new tensor (e.g., using .detach()). As a user, you could have made a mistake implementing __torch_dispatch__ or a Python operator decomposition or meta registration; if that's not the case, please report a bug to PyTorch or the backend you are using.

</code></pre>
<p>Is there anyone who struggle with same this problem ? I am wondering your solution.</p>
<p>the code compile code :</p>
<pre><code class="lang-auto">batch = next(iter(dataloader))
model = torch_tensorrt.compile(runner.net,ir='dynamo',inputs=batch, enabled_precisions= { torch_tensorrt.dtype.half})
</code></pre>
Multiple Sequential in one module

username: Dirk10001
cooked: <p>Hello all,<br>
I try to convert f-AnoGAN to C++. In the Discriminator there are two sequences:</p>
<pre><code class="lang-auto">self.model = nn.Sequential(…
self.adv_layer = nn.Sequential(…
</code></pre>
<p>Two forwarding functions call one or both of them:</p>
<pre><code class="lang-auto">features = self.forward_features(img)
validity = self.adv_layer(features)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">features = self.model(img)
</code></pre>
<p>So one forwards all, both sequences, the other one only model.</p>
<p>In C++ I tried</p>
<pre><code class="lang-auto">struct GAN_DiscriminatorImpl : nn::Module{
private:
    nn::Sequential model;
    nn::Sequential adv_layer;
    nn::Sequential all;
</code></pre>
<p>The forwarding would be the same like in python, register_module doesn’t work.</p>
<pre><code class="lang-auto">all= model + adv_layer;
register_module("Discriminator", this-&gt;all);
</code></pre>
<p>or</p>
<pre><code class="lang-auto">register_module("Discriminator", this-&gt;model, this-&gt;adv_layer);
</code></pre>
<p>None of them works, how to append the two sequences?</p>
<p>I ended up with:</p>
<pre><code class="lang-auto">torch::Tensor GAN_DiscriminatorImpl::forward_features(torch::Tensor z) {

    std::vector&lt;std::shared_ptr&lt;torch::nn::Module&gt;&gt; mods = model-&gt;modules();
    for (auto mod = std::begin(mods); mod != std::end(mods); ++mod)
    {
        std::shared_ptr&lt;torch::nn::Module&gt; m = *mod;
        torch::nn::Module* m_ = m.get();

        if (m_-&gt;name().compare("torch::nn::Conv2dImpl") == 0)
        {
            torch::nn::Conv2dImpl* c = dynamic_cast&lt;torch::nn::Conv2dImpl*&gt;(m_);
            z = c-&gt;forward(z);
        }

        if (m_-&gt;name().compare("torch::nn::ReLUImpl") == 0)
        {
            torch::nn::ReLUImpl* c = dynamic_cast&lt;torch::nn::ReLUImpl*&gt;(m_);
            z = c-&gt;forward(z);
        }

        if (m_-&gt;name().compare("torch::nn::BatchNorm2dImpl") == 0)
        {
            torch::nn::BatchNorm2dImpl* c = dynamic_cast&lt;torch::nn::BatchNorm2dImpl*&gt;(m_);
            z = c-&gt;forward(z);
        }

        if (m_-&gt;name().compare("torch::nn::Dropout2dImpl") == 0)
        {
            torch::nn::Dropout2dImpl* c = dynamic_cast&lt;torch::nn::Dropout2dImpl*&gt;(m_);
            z = c-&gt;forward(z);
        }

        if (m_-&gt;name().compare("ViewImpl") == 0)
        {
            ViewImpl* c = dynamic_cast&lt;ViewImpl*&gt;(m_);
            z = c-&gt;forward(z);
        }
    }

    return z;

}
</code></pre>
<p>So the last step, the fully connected layer is just not forwarded. That works, but it is ugly.</p>
Focal loss like classification loss

username: AviZ
cooked: <p>I have the following focal loss like implementation:</p>
<pre data-code-wrap="python"><code class="lang-python">def focal_loss(pred:torch.Tensor, target:torch.Tensor, α:float=2.0, β:float=4.0) -&gt; torch.Tensor:
  '''
  Treats the tensors as a contiguous array.
  Arguments:
    pred    (batch x c x h x w) in [0, 1]
    target  (batch x c x h x w) in [0, 1]
  '''

  pos_mask = target.eq(1).float()
  neg_mask = target.lt(1).float()

  neg_weights = torch.pow(1 - target, β)

  pos_loss = torch.log(pred) * torch.pow(1 - pred, α) * pos_mask
  neg_loss = torch.log(1 - pred) * torch.pow(pred, α) * neg_weights * neg_mask

  num_pos  = pos_mask.float().sum()
  pos_loss = pos_loss.sum()
  neg_loss = neg_loss.sum()

  total_loss = 0
  if (num_pos == 0):
    total_loss = total_loss - neg_loss
  else:
    total_loss = total_loss - (pos_loss + neg_loss) / num_pos

  return total_loss
</code></pre>
<p>Whenever I use it I get very bad results compared to using <code>binary_cross_entropy</code>.</p>
<p>The concept is from <a href="https://arxiv.org/abs/1808.01244" rel="noopener nofollow ugc">CornerNet: Detecting Objects as Paired Keypoints</a>:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4e21f0dd334536d95a10c415b2f4f0f44bb54b2.png" data-download-href="https://discuss.pytorch.org/uploads/default/c4e21f0dd334536d95a10c415b2f4f0f44bb54b2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4e21f0dd334536d95a10c415b2f4f0f44bb54b2.png" alt="image" data-base62-sha1="s5I2UacqyQoEkBAJWiR9P1Gg4CK" width="681" height="465"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">681×465 71.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I suspect it is a numerical thing.<br>
Any way to make calculation more robust?</p>
<p>Any alternative loss to such case?</p>
Distributed training with Trainer and ConstantLengthDataset classes

username: ima882112
cooked: <p>Hello! I have a custom ConstantLengthDataset class:</p>
<pre><code class="lang-auto">class ConstantLengthDataset(IterableDataset):
    def __init__(
        self, 
        tokenizer, 
        dataset, 
        infinite=False,
        seq_length=8192,
        num_of_sequences=1024,
        chars_per_token=3.6,
    ):
        self.tokenizer = tokenizer
        self.concat_token_id = tokenizer.eos_token_id
        self.dataset = dataset
        self.seq_length = seq_length
        self.infinite = infinite
        self.current_size = 0
        self.max_buffer_size = seq_length * chars_per_token * num_of_sequences

    def __iter__(self):
        iterator = iter(self.dataset)
        more_examples = True
        while more_examples:
            buffer, buffer_len = [], 0
            while True:
                if buffer_len &gt;= self.max_buffer_size:
                    break
                try:
                    buffer.append((next(iterator))["content"])
                    buffer_len += len(buffer[-1])
                except StopIteration:
                    if self.infinite:
                        iterator = iter(self.dataset)
                    else:
                        more_examples = False
                        break
            tokenized_inputs = self.tokenizer(buffer, truncation=False)["input_ids"]
            all_token_ids = []
            for tokenized_input in tokenized_inputs:
                all_token_ids.extend(tokenized_input + [self.concat_token_id])
            for i in range(0, len(all_token_ids), self.seq_length):
                input_ids = all_token_ids[i : i + self.seq_length]
                if len(input_ids) == self.seq_length:
                    self.current_size += 1
                    yield {
                        "input_ids": torch.LongTensor(input_ids),
                        "labels": torch.LongTensor(input_ids),
                    }
</code></pre>
<p>and due to <code>per_gpu_batch_size = 1</code> in trainer I want to train on 1 node using 2 GPUs with seq_len = 8192 tokens. But when I launched this training using <code>torchrun</code> command I noticed that despite of <code>yield</code> and <code>num_workers = 1</code> trainer consider more than 1 sequence of 8192 tokens per gpu and moreover different GPUs have same <code>buffer</code> elements of ConstantLengthDataset class. How can I fix it for distributed training. Thank you in advance.</p>
Temporal Fusion Transformer predict error

username: Juan_pablo_Mazo_quin
cooked: <p>Hello everyone,</p>
<p>I’m currently working with the Temporal Fusion Transformer (TFT) and facing a dimension mismatch issue. My dataset contains ~5,000 unique combinations (e.g., SKU, store, etc.) over 24 timesteps (2 years of monthly data). My goal is to forecast the next month for each combination.</p>
<p>When I calculate metrics like the Mean Absolute Error (MAE) using <code>Baseline().predict()</code> with <code>return_y=True</code>, I encounter the following error:</p>
<pre><code class="lang-auto">RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 1280 but got size 641 for tensor number 4 in the list.
</code></pre>
<p>Interestingly, if I set <code>return_y=False</code>, the code runs without issues. Below is the setup I’ve been using:</p>
<pre><code class="lang-auto"># Create validation set (predict=True) to predict the last max_prediction_length points in time for each series  
validation = TimeSeriesDataSet.from_dataset(training, df, predict=True, stop_randomization=True)  

# Create dataloaders for the model  
batch_size = 128  # Set this between 32 to 128  
train_dataloader = training.to_dataloader(train=True, batch_size=batch_size, num_workers=0)  
val_dataloader = validation.to_dataloader(train=False, batch_size=batch_size * 10, num_workers=0, drop_last=True)  

# Define baseline predictions  
baseline_predictions = Baseline().predict(val_dataloader, return_y=True)  

# Calculate MAE  
MAE()(baseline_predictions.output, baseline_predictions.y)  
</code></pre>
<p>Here are the parameters I’ve configured:</p>
<ul>
<li><code>max_prediction_length = 2</code></li>
<li><code>max_encoder_length = 12</code></li>
</ul>
<p>It appears to be a dimension mismatch issue, but I’m not sure how to address it. Has anyone encountered a similar issue or can suggest a solution?</p>
<p>Thank you in advance for your help!</p>

username: soulitzer
cooked: <p>Do you know what the expectations are from MAE() for the shape of its inputs?</p>
<p>You mention that the code runs without issues when return_y=False, would baseline_predictions.y still passed to MAE() in that case?</p>
Updated weights are not leaf tensor?

username: Gary_Tze_Hay_Lau
cooked: <p>Answer 1:</p>
<p>The <strong>initial weight</strong> (created by the user, typically via <code>torch.nn.Parameter</code>) is considered a <strong>leaf tensor</strong> if it has <code>requires_grad=True</code>. This is because it is directly created by the user and not the result of an operation.</p>
<ul>
<li><strong>Updated weights</strong> (after an operation, such as <strong>applying gradients during backpropagation) are not leaf tensors.</strong> These updated weights are the result of operations (like adding the gradients to the previous weights), and therefore they have a <code>grad_fn</code> that points to the operation used to create them. Hence, they are <strong>non-leaf tensors</strong>.</li>
</ul>
<p>So, only the <strong>initial weights</strong> (before training) are leaf tensors with <code>grad_fn=None</code>, while the <strong>updated weights</strong> are the result of a computation (e.g., weight update using gradients) and thus are not leaf nodes.</p>
<p>Answer 2:<br>
Here, <code>weights</code> is a leaf tensor, and after the update, <code>new_weights</code> is a new tensor that results from an operation on <code>weights</code>. <strong>Despite being created through an operation, new_weights is still a leaf tensor because it’s a direct result of your manual creation (the subtraction operation)</strong>, not an operation involving tensors that would produce a non-leaf tensor.</p>
<p>============<br>
Is it correct?</p>
<p>Is the updated weight considered a leaf node in pytorch or not?<br>
Could anyone help me Thanks.</p>
<p>There are two contradictory explanations after I use ChatGPT to give me an answer…</p>

username: ptrblck
cooked: <p>ChatGPT is wrong in both cases. Parameters are and stay leaf tensors as all updates are performed in-place on them.</p>
PyTorch Newbie - Help with Datasets and DatasetLoaders

username: irongeek75
cooked: <p>I am new to PyTorch and am attempting to load a custom dataset composed only of images. I’m not sure where I’m going wrong, could someone please help me to understand. Thanks in advance!</p>
<p>#!/usr/bin/env python</p>
<p>import os, glob<br>
from <a href="http://torchvision.io" rel="noopener nofollow ugc">torchvision.io</a> import read_image<br>
from torch.utils.data import Dataset, DataLoader</p>
<h1><a name="p-461942-h-1" class="anchor" href="#p-461942-h-1"></a></h1>
<h2><a name="p-461942-h-2" class="anchor" href="#p-461942-h-2"></a></h2>
<h3><a name="p-461942-example-3" class="anchor" href="#p-461942-example-3"></a>EXAMPLE</h3>
<h2><a name="p-461942-h-4" class="anchor" href="#p-461942-h-4"></a></h2>
<h1><a name="p-461942-h-5" class="anchor" href="#p-461942-h-5"></a></h1>
<p>class MyDataset(Dataset):<br>
def <strong>init</strong>(self, images_dir):<br>
self.images_dir = images_dir</p>
<pre><code>def __len__(self):
    num = len(glob.glob(os.path.join(self.images_dir, '*.png')))

    return num 

def __getitem__(self, index):
    image = read_image(self.images_dir[index])

    return image
</code></pre>
<p>def main():<br>
training_data = ‘/home/XXXX/development/Pytorch_Tutorials/Datasets_and_Dataloaders/images’<br>
dataset = MyDataset(training_data)</p>
<pre><code>loader = DataLoader(
    dataset,
    shuffle=True
)

for data in loader:
    print(data)
</code></pre>
<p>if <strong>name</strong>==“<strong>main</strong>”:<br>
main()</p>

username: ptrblck
cooked: <p>The <code>__getitem__</code> logic looks wrong. <code>images_dir</code> is a string specifying the path to the directory of all images while you are trying to index it. Create a <code>list</code> of paths to all images which can be indexed in <code>__getitem__</code>.</p>

username: irongeek75
cooked: <p>Thanks for the help!!!</p>
<p>My solution works, would you suggest a better way??</p>
<p>Solution:<br>
class MyDataset(Dataset):<br>
def <strong>init</strong>(self, images_dir):<br>
self.images_dir = Path(images_dir)</p>
<pre><code>def __len__(self):
    num = len(glob.glob(os.path.join(self.images_dir, '*.png')))

    return num 

def __getitem__(self, index):
    f = glob.glob(os.path.join(self.images_dir, '*.png'))
    img = os.path.join(self.images_dir, f[index])
    image = read_image(img)

    return image
</code></pre>
<p>def main():<br>
training_data = ‘/home/jvines/development/Pytorch_Tutorials/Datasets_and_Dataloaders/T72_Tanks’<br>
dataset = MyDataset(training_data)</p>
<pre><code>loader = DataLoader(
    dataset,
    shuffle=False
)

for data in loader:
    print(data)
</code></pre>
<p>if <strong>name</strong>==“<strong>main</strong>”:<br>
main()</p>

username: ptrblck
cooked: <p>Create the proper paths once in <code>__init__</code> to avoid the overhead of calling <code>f = glob.glob(os.path.join(self.images_dir, '*.png'))</code> in each <code>__getitem__</code> call and in the <code>__len__</code> method.<br>
Code duplication can yield to tricky errors and you are also paying overhead when each sample is loaded since you are re-creating all image paths over and over again.</p>