akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Processing unfolded tensors 2x slower?

username: thandal
cooked: <p>I’m using the unfold operation on a temporal sequence of data in order to generate a bunch of cases for a model. I send the original temporal sequence to the GPU, and I found that with careful attention to indices, I could maintain “pure views” of the data so that there was very little (no?) GPU memory overhead. Yay!</p>
<p>But then I happened to violate the pure views, and triggered a copy of the unfolded tensor (which you can also do by calling clone() on the views)… and my model ran 2x faster!  And GPU memory usage was huge, of course!</p>
<p>See the code below for a simple benchmark.</p>
<p>I don’t have a good hypothesis for why processing a view is so much slower. I thought maybe the index information was stored on the CPU, which would make it slow for the GPU to fetch… but there is the same 2x slowdown on CPU.</p>
<p>On GPU<br>
3.34 ms ± 15.3 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)<br>
6.43 ms ± 316 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)   ← view is 2x slower!</p>
<p>On CPU<br>
223 ms ± 29 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)<br>
410 ms ± 35.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  ← view is 2x slower!</p>
<p>Benchmark code:</p>
<pre><code class="lang-auto">import torch

data_length = 1_000_000
data_width = 10
sequence_length = 50
#device = 'cuda'
device = 'cpu'

print(f"Initial CUDA memory allocated: {torch.cuda.memory_allocated(0)}")
d = torch.rand((data_length, data_width), device=device)
print(f"After d:                       {torch.cuda.memory_allocated(0)}")
d_unfold = d.unfold(dimension=0, size=sequence_length, step=1).mT             # &lt;-- note the mT
print(f"After d_unfold:                {torch.cuda.memory_allocated(0)}")
# Without the matrix transpose (above), flattening d_unfold triggers a clone
d_unfold_flatten = d_unfold.flatten(start_dim=1)
print(f"After d_unfold_flatten:        {torch.cuda.memory_allocated(0)}")
d_unfold_flatten_clone = d_unfold_flatten.clone()
print(f"After d_unfold_flatten_clone:  {torch.cuda.memory_allocated(0)}")

print(f"{d.shape=}")
print(f"{d_unfold.shape=}")
print(f"{d_unfold_flatten.shape=}")
print(f"{d_unfold_flatten_clone.shape=}")

model = torch.nn.Linear(in_features=data_width * sequence_length, out_features=16, device=device)

print(f"{device=}")
%timeit y = model(d_unfold_flatten_clone)
%timeit y = model(d_unfold_flatten)
</code></pre>
Slow performance when running TransformerDecoder with low batch size in fp16

username: DanielN
cooked: <p>Hey!</p>
<p>I want to understand why this snippet of code runs slow for batch sizes of 2/4/8 when using fp16, the time it takes for bs=2 on my system is 0.55 sec for no batch version and 2.8 sec for the batched on.</p>
<pre><code class="lang-auto">from torch import nn
import torch
import time
from torch.amp import autocast
device = torch.device('cuda')

hidden_dim = 1024
USE_FP16 = True
BS = 2
TOTAL_INPUT = 250
decoder_layer = nn.TransformerDecoderLayer(d_model=hidden_dim,
                                           dim_feedforward=hidden_dim*4,nhead=8,batch_first=True,)
transformer_decoder = nn.TransformerDecoder(decoder_layer, num_layers=3,).to(device)

with autocast('cuda', enabled=USE_FP16):
    with torch.no_grad():
        memory = torch.rand((1, 1000, hidden_dim), device=device)
        tgt = torch.rand((1, 36, hidden_dim), device=device)
        out = transformer_decoder(tgt, memory)
        s = time.perf_counter()
        for _ in range(TOTAL_INPUT):
            out = transformer_decoder(tgt, memory)
        print(time.perf_counter() - s)


        memory = torch.rand((BS, 1000, hidden_dim), device=device)
        tgt = torch.rand((BS, 36, hidden_dim), device=device)


        s = time.perf_counter()
        for _ in range(TOTAL_INPUT // BS):
            out = transformer_decoder(tgt, memory)
        print(time.perf_counter() - s)
</code></pre>
<p>This is also reproduced in google colab<br>
Thanks!</p>

username: JuanFMontesinos
cooked: <p>You add overhead by imposing batch_first. Namely, the model has to internally transpose dimensions and that extra computation make sthe whole thing slower.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1ez07w-0tnAZWA36OLgHYIXDgMxkRGAfa?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/d7111a08f9e5e6c5883d312b294726e3/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1ez07w-0tnAZWA36OLgHYIXDgMxkRGAfa?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1ez07w-0tnAZWA36OLgHYIXDgMxkRGAfa?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colab</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
MultiDiscrete Observation Causes Shape Mismatch

username: rgbarishian
cooked: <p>torchrl version: 0.5.0<br>
python: 3.11.10</p>
<p>Error:</p>
<pre><code class="lang-auto">RuntimeError: All input tensors (value, reward and done states) must share a unique shape).
</code></pre>
<p>To Replicate:<br>
Follow the PPO tutorial and make the following changes:</p>
<ol>
<li>Replace the GymEnv with this</li>
</ol>
<pre><code class="lang-auto">class DummyEnv(gym.Env):
    def __init__(self, env_config={}, render_mode=None):
        self.action_space = Discrete(3)
        self.observation_space = MultiDiscrete([10,10])

    def reset(self, actions):
        return self.observation_space.sample(), {}

    def step(self, actions):
        return self.observation_space.sample(), 0, False, False, {}
</code></pre>
<p>And change the environment transform to:</p>
<pre><code class="lang-auto">env = TransformedEnv(
    base_env,
    Compose(
    DTypeCastTransform(dtype_in=torch.int64, dtype_out=torch.float32), #Necessary since we have discrete observation, observation space returns ints, but torch needs them to be floats
    StepCounter()
    )
)

</code></pre>
<p>My assumption is torch is thinking that the extra dimension is a batch size. Is there a way to flatten the MultiDiscrete space?</p>

username: vmoens
cooked: <p>Hello!<br>
This code works fine on my end (with and without ParallelEnv) using torchrl nightly. I think it should work with v0.6 too!</p>
<pre data-code-wrap="python"><code class="lang-python">import gym
import torch
from gym.spaces import Discrete, MultiDiscrete
from torchrl.envs import GymWrapper, TransformedEnv, StepCounter, Compose, DTypeCastTransform, ParallelEnv


class DummyEnv(gym.Env):
    def __init__(self, env_config={}, render_mode=None):
        self.action_space = Discrete(3)
        self.observation_space = MultiDiscrete([10, 10])

    def reset(self):
        return self.observation_space.sample(), {}

    def step(self, actions):
        return self.observation_space.sample(), 0, False, False, {}

if __name__ == "__main__":
    base_env = ParallelEnv(2, lambda:GymWrapper(DummyEnv()))

    env = TransformedEnv(
        base_env,
        Compose(
            # Necessary since we have discrete observation,
            #  observation space returns ints, but torch needs them to be floats
            DTypeCastTransform(in_keys=["observation"], out_keys=["observation"], dtype_in=torch.int64, dtype_out=torch.float32),
            StepCounter()
        )
    )

    env.check_env_specs()
</code></pre>
How should the "next" field in TorchRL's tensordict be?

username: matters
cooked: <p>Using the TorchRL framework and a collector, what should be the output of a rollout for <code>traj_ids</code>, <code>done</code> and <code>next.done</code> when an episode reaches a terminal state?</p>
<p>Considering a 6 steps rollout, I assume it should be as follows:</p>
<pre><code class="lang-auto">traj_ids:  [1, 1, 1, 2, 2, 2]
done:      [0, 0, 1, 0, 0, 0]
next.done: [0, 1, 0, 0, 0, 0]
</code></pre>
<p>Then, which <code>done</code> is used to compute the Q-value?</p>

username: vmoens
cooked: <p>The convention is this:</p>
<pre data-code-wrap="python"><code class="lang-python">TensorDict({
   "obs": torch.Tensor(...), # the observation at time t
   "action": torch.Tensor(...), # the action at time t, based on the observation at time t
   "done": torch.Tensor(...), # done state for "obs" at time t - should almost always be False (unless done at reset)
    "next": TensorDict({
        "obs": torch.Tensor(...), # the observation at time t+1, resulting from (obs, action) at time t
        "reward": torch.Tensor(...), # the reward at time t+1, resulting from (obs, action) at time t
        "done": torch.Tensor(...), # the done at time t+1, resulting from (obs, action) at time t - can be True when the transition is terminal
    }, [...]),
}, [...]) 
</code></pre>
<p>Why a “done” state at the root? Just to catch the very rare cases where the reset() returns a state that is done.</p>
<p>The decision on making reward / done belong to t+1 and action to t is based on the famous illustration from Sutton and Barto<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/411061350044bfb188cd44a8f0507ca431777ac5.png" data-download-href="https://discuss.pytorch.org/uploads/default/411061350044bfb188cd44a8f0507ca431777ac5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/1/411061350044bfb188cd44a8f0507ca431777ac5.png" alt="image" data-base62-sha1="9hA7ZHZlR6k9qovTzzD7B6keB9P" width="690" height="265" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">908×350 10.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Although I <a href="https://www.linkedin.com/posts/vincent-moens-9bb91972_technical-opinion-hear-me-out-i-think-the-activity-7219616008636895232-qBb7?utm_source=share&amp;utm_medium=member_desktop" rel="noopener nofollow ugc">personally don’t like it much</a></p>
Is this PPO training code wrong?

username: BTeddy
cooked: <p>I saw this training code in 2 different source (a open source project and my former colleague). I always consider it’s correct and use it.</p>
<pre><code class="lang-auto">ratios = torch.exp(torch.clamp(a_logp_new - torch.sum(a_logp[idx], dim=-1), max=4))
surr1 = ratios * adv[idx]
surr2 = torch.clamp(ratios, 1 - self.clip_ep, 1 + self.clip_ep) * adv[idx]
actor_loss = - torch.min(surr1, surr2).mean()
</code></pre>
<p>but now, I’m wandering it is wrong when the advantage is negetive.<br>
assume that clip_ep is 0.1, ratio is 2; when adv is positive, the loss is - 1.1 * adv; when adv is negetive, the loss is -2 * adv.<br>
positive and negetive adv with same absolute value cause unsymmetrical influence<br>
and I guess this is the reason cause my Categorical distribution become NAN like this discussion(<a href="https://discuss.pytorch.org/t/categorical-distribution-returning-breaking/165343" class="inline-onebox">Categorical distribution returning breaking</a>)</p>

username: vmoens
cooked: <p>If your advantage is negative, you want to “push away” the parameter configuration from that outcome so everything should work fine. The <code>min</code> operator ensures that you’re taking the most pessimistic estimate.</p>
Convert torchvision functions to libtorch C++ codes

username: L_R
cooked: <pre data-code-wrap="python"><code class="lang-python">from torchvision.tv_tensors import BoundingBoxes
from torchvision.tv_tensors import Mask
...
        self._pred_bboxes = BoundingBoxes(
            model_output[0]["boxes"][scores_mask]
            format="xyxy",
            canvas_size=self._input_img.size[::-1],
        )

        self._pred_masks = torch.nn.functional.interpolate(
            model_output[0]["masks"][scores_mask], size=self._color_image.size[::-1]
        )

        # mask is a two classificaiton problem
        self._pred_masks = torch.concat(
            [
                Mask(torch.where(mask &gt;= 0.5, 1, 0), dtype=torch.bool)
                for mask in self._pred_masks
            ]
        )
</code></pre>
<p>I want to transform it into C++ codes.</p>
<p>I’ve found function for <code>interpolate</code>, it is as below</p>
<pre data-code-wrap="C++"><code class="lang-C++">        torch::Tensor _pred_masks = torch::nn::functional::interpolate(
                image_tensor,
                F::InterpolateFuncOptions()
                        .size(std::vector&lt;int64_t&gt;({_color_img.rows, _color_img.cols}))
        );
</code></pre>
<p>But for torchvision, I havent found corresponding functions. I decided to store the <code>_pred_bboxes</code> as <code>torch::tensor</code> or <code>std::vector</code>.</p>
<p>For <code>torchvision::tv_tensors::Mask</code>, I don’t even know what it means. the documentation says</p>
<blockquote>
<p><a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"><code>torch.Tensor</code></a> subclass for segmentation and detection masks</p>
</blockquote>
<p>but it is very vague for what it does.</p>
<p>I have read this post <a href="https://discuss.pytorch.org/t/is-torchvision-also-available-in-c/90832/6" class="inline-onebox">Is torchvision also available in C++? - #6 by Shisho_Sama</a> but didnt find useful information about my issue.</p>
NCCL WARN NET/IB : Got async event : port error

username: wu_martin
cooked: <p>The Super MICR SYS-821GE-TNHR servers with H100，show messages like this:<br>
103 NCCL WARN NET/IB : mlx5_6:1:Got async event : port error<br>
103 NCCL WARN NET/IB : mlx5_6:1:Got async event : port error<br>
103 NCCL WARN NET/IB : mlx5_6:1:Got async event : port error<br>
103 NCCL WARN NET/IB : mlx5_6:1:Got async event : port error<br>
103 NCCL WARN NET/IB : mlx5_6:1:Got async event : GID table change<br>
103 NCCL WARN NET/IB : mlx5_6:1:Got async event : GID table change<br>
We are not sure whether it is the issue of the platform or the issue of the network or the issue of the server. which looks good. Thanks for your help!<br>
environment:<br>
ubuntu22.04<br>
NCCL:2.23.4<br>
CUDA:12.6.3</p>
GPU L1/L2 cache utilization

username: Sirish_Gambhira
cooked: <p>Hi all,</p>
<p>I was wondering if there’s any way we can visualize GPU cache occupancy while training / inferring a model in pytorch. I came across some commands such as <code>torch.cuda.memory._record_memory_history(max_entries=100000)</code>, but I am assuming this will not partition the memory into global and shared (or cache). I am interested to know if there are any ways to understand fine-grained GPU memory allocation. Thank you!</p>
Cuda out of memory even using different GPU with greater VRAM

username: EdwardJoe
cooked: <p>I was trying to train for this <a href="https://github.com/cool-xuan/BN-WVAD" rel="noopener nofollow ugc">BN-WVAD</a> model as the author trained for 2 datasets, the author had only released the codes for 1 of the dataset (XD-Violence).</p>
<p>I had successfully trained for this dataset, but failed to train for the 2nd dataset (UCF-Crime), keep getting the CUDA out of memory problem.</p>
<p>I had tried using my own RTX3050 of 6GB, Google Colab’s T4 GPU and Lightning AI Studio’s A10G which has 24GB VRAM but still getting same issue as shown in the pictures below.</p>
<p><strong>Previously had checked the shape of both dataset (N, 1024). Seems like xd-violence is greater than ucf crime</strong></p>
<p>I wonder why I can successfully trained for xd-violence when the shape is greater. Any helps will be much appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7de3d05cc8e53fe8c422f1cd3f17cca381cebe9e.png" data-download-href="https://discuss.pytorch.org/uploads/default/7de3d05cc8e53fe8c422f1cd3f17cca381cebe9e" title="error2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7de3d05cc8e53fe8c422f1cd3f17cca381cebe9e_2_487x500.png" alt="error2" data-base62-sha1="hXFLUUtGLhrjF5GEah6zHTvv354" width="487" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7de3d05cc8e53fe8c422f1cd3f17cca381cebe9e_2_487x500.png, https://discuss.pytorch.org/uploads/default/original/3X/7/d/7de3d05cc8e53fe8c422f1cd3f17cca381cebe9e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/d/7de3d05cc8e53fe8c422f1cd3f17cca381cebe9e.png 2x" data-dominant-color="1E1E1E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error2</span><span class="informations">577×592 189 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Ambiguous DTensor placements

username: Jingyue_Wu
cooked: <p>I suspect the current way of specifying shardings (via mesh and <a href="https://pytorch.org/docs/stable/distributed.tensor.html#torch.distributed.tensor.DTensor.placements" rel="noopener nofollow ugc">placements</a>) in DTensor is ambiguous. Any way to resolve that?</p>
<p>Example:</p>
<pre data-code-wrap="python"><code class="lang-python">​in_tensor = torch.arange(6)
mesh = init_device_mesh("cuda", [2, 3])  # [[GPU0, GPU1, GPU2], [GPU3, GPU4, GPU5]]
distribute_tensor(in_tensor, mesh, [Shard(0), Shard(0)])
</code></pre>
<p>doesn’t specify whether to split size=6 by 2 first or by 3 first. If first by 2, then GPU 1 will hold number 1; if first by 3, GPU 1 will hold number 2.</p>
<p>I guess this is an important distinction for context parallelism. The sequence dimension of the input activation to a transformer layer is both context parallelized and tensor parallelized. Whether to apply CP or TP first affects how data is distributed. However, maybe, in practice an arbitrary order has been good enough <img src="https://discuss.pytorch.org/images/emoji/apple/person_shrugging.png?v=12" title=":person_shrugging:" class="emoji" alt=":person_shrugging:" loading="lazy" width="20" height="20"></p>
How can I implement `aten::kthvalue.values`

username: Viranchee
cooked: <p>I want to implement this operator in MSL / MPS kernel.</p>
<p>I faced the error, operator <code>aten::kthvalue.values</code> is not implemented.</p>
<p>The code which calls it is:</p>
<pre data-code-wrap="python"><code class="lang-python">sparsities, accuracies = sensitivity_scan(
    model, dataloader['test'], scan_step=0.1, scan_start=0.4, scan_end=1.0)
</code></pre>
<p>Can someone guide me?</p>
Why is requires_grad==False after multiplication?

username: mbalazs98
cooked: <p>Hi,<br>
How is it possible that the following lines of code:</p>
<p>print(X.requires_grad, W.requires_grad)<br>
K = X * W<br>
print(K.requires_grad)</p>
<p>yield this output:</p>
<p>False True<br>
False</p>
<p>? In my understanding K.requires_grad should be True, regardless of anything else in the code.</p>
<p>Thanks in advance</p>

username: ashok_arora
cooked: <p>What version of python and pytorch are you on?</p>

username: mbalazs98
cooked: <p>Python 3.8.10<br>
2.0.1+cu117</p>

username: ashok_arora
cooked: <p>I tried the same on my system and it did come <code>True</code>.</p>
<pre><code class="lang-auto">Python 3.8.0 (default, Nov  6 2019, 21:49:08) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.0.1+cu117'
&gt;&gt;&gt; x = torch.rand(10, requires_grad=False)
&gt;&gt;&gt; w = torch.rand(10, requires_grad=True)
&gt;&gt;&gt; print(x.requires_grad, w.requires_grad)
False True
&gt;&gt;&gt; k = x * w
&gt;&gt;&gt; k.requires_grad
True
</code></pre>
<p>Could it be possible that something is changing the tensor <code>k</code>?</p>

username: mbalazs98
cooked: <p>that is what I’m wondering as well but I have no idea how, since I print K.requires_grad right after I initialized it as X*W</p>

username: ashok_arora
cooked: <p>Can you send a minimal, reproducible example of your code?</p>

username: mbalazs98
cooked: <p>it wouldn’t be too easy, since this part is called with several function calls, but the reason I am confused is I don’t think this should be possible regardless of other parts of the code</p>

username: ashok_arora
cooked: <p>Sorry, without that, it would be really hard for me to debug.</p>

username: mbalazs98
cooked: <p>I found the problem, it would be complicated to explain but I’m passing W to a wrapperfunction to implement my forward and backward pass which messes with the computational graph. I’m still slightly confused by how the 3 lines I sent is possible, but it seems clear how I can get rid of it. Thanks for the help!</p>

username: Spencer_Schneider
cooked: <p>Hey Balázs could you actually explain more how you fixed this issue? It seems like I’m running to this same exact issue.</p>
How to calculate the activation memory usage of a model

username: shadowshadow
cooked: <p>I want to write a module to track and analyze the memory usage of model activations. Here’s my current statistics code. Can it provide reasonable estimates of GPU memory consumption by activations? I think the result will be right only if torch.cuda.memory_allocate will synchorize until forward finishes.</p>
<pre data-code-wrap="python"><code class="lang-python">
memory_before_forward = torch.cuda.memory_allocate()
# do forward
model(input)
memory_after_forward = torch.cuda.memory_allocate()
print("activation memory: ", (memory_after_forward - memory_before_forward) / 1024**3)

</code></pre>

username: ptrblck
cooked: <p>Your code assumes all activations are stored on the GPU, which is a common case for a lot of models, but which will fail if e.g. CPU offloading is used. Also, libraries would allocate workspaces (e.g. cuBLAS) so you should ideally execute warmup iterations.<br>
A better way could be to actually save the number of elements and <code>dtype</code> of all forward activations during their computation using a tensor subclass.</p>
How to reduce multi-node multi-GPU communication cost in DDP

username: zbh2047
cooked: <p>Hi, I am trying to use pytorch in multi-node multi-GPU training. I train a model with just 1M parameters on 128 GPUs. However, I found that the gradient all_reduce operation takes roughly half of the time! Specifically, when adding model.no_sync() to disable gradient sync, the training speed readily doubles. I also profiled the code and confirmed that there is a cudaStreamSynchronize operation that takes half of time.</p>
<p>I’d like to ask about any method that could potentially reduce such heavy communication cost. Can anyone help me? Thank you!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="zbh2047" data-post="1" data-topic="215419">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/b5ac83/48.png" class="avatar"> zbh2047:</div>
<blockquote>
<p>all_reduce operation takes roughly half of the time</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="zbh2047" data-post="1" data-topic="215419">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/b5ac83/48.png" class="avatar"> zbh2047:</div>
<blockquote>
<p>there is a cudaStreamSynchronize operation that takes half of time.</p>
</blockquote>
</aside>
<p>Where is the sync coming from? Did you check if the communication IP is waiting for something and is being blamed wrongfully?</p>

username: zbh2047
cooked: <p>cudaStreamSynchronize is called because I use torch.amp, which checks whether gradient contains nan and thus needs a grad.item() call. I don’t know the inner imeplemtation of DDP, but in the profile I can notice that nccl_all_reduce is overlapped with cudaStreamSynchronize. So I think cudaStreamSynchronize is slow because the communication between different GPUs is slow.</p>

username: ptrblck
cooked: <p>To check the allreduce performance of your system for different sizes you could use the <code>nccl-test</code> as a reference. If the PyTorch code takes significantly longer for its communication calls, NCCL might be waiting and you would need to dig more into the profile.</p>
New AI TTS System with Emotion Control

username: Saifs_AI
cooked: <p>A cutting-edge <a href="https://saifs.ai/ai-text-to-speech" rel="noopener nofollow ugc">text-to-speech</a><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b5acf26473b4da5045b72b1251774ba7a160bf4.png" data-download-href="https://discuss.pytorch.org/uploads/default/1b5acf26473b4da5045b72b1251774ba7a160bf4" title="TTS"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b5acf26473b4da5045b72b1251774ba7a160bf4_2_690x325.png" alt="TTS" data-base62-sha1="3TZrRKT7yp5oiClPfY9stzrupNi" width="690" height="325" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b5acf26473b4da5045b72b1251774ba7a160bf4_2_690x325.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b5acf26473b4da5045b72b1251774ba7a160bf4_2_1035x487.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b5acf26473b4da5045b72b1251774ba7a160bf4_2_1380x650.png 2x" data-dominant-color="F1F2F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">TTS</span><span class="informations">1915×902 138 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
solution that converts written text into natural-sounding speech using advanced AI technology. The system supports multiple languages, voice styles, and emotional tones.</p>
<pre><code class="lang-auto">
### Unique Features
- Multi-speaker voice synthesis
- Emotion control (happy, sad, neutral, excited)
- Real-time voice cloning capabilities
- 95+ languages supported
- Custom pronunciation dictionary
- Prosody control (pitch, speed, emphasis)
- Background noise reduction
- API integration options

### Use Cases
1. Content Creation
   - Audiobook production
   - YouTube video narration
   - Podcast content generation
   - E-learning materials

2. Accessibility
   - Screen readers
   - Public announcements
   - Document reading assistance

3. Business Applications
   - Customer service automation
   - IVR systems
   - Virtual assistants
   - Corporate training materials

### Free Options
- Daily usage quota: 10,000 characters
- Access to 10 basic voices
- Standard language support
- Basic emotion controls
- Web interface access
- Community support

    ### Technical Specifications
    ```
    Technology Stack:
    - Deep Learning Framework: PyTorch
    - Voice Models: Transformer-based
    - Audio Processing: 24-bit/48kHz
    - Latency: &lt;500ms for generation
    - Format Support: WAV, MP3, OGG
    - API Protocol: REST/WebSocket</code></pre>
Torch.set_default_device(device) in CPP

username: Mohit_Kumar
cooked: <p>I am trying to replicate <a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial#:~:text=A%20character%2Dlevel%20RNN%20reads,class%20the%20word%20belongs%20to." rel="noopener nofollow ugc">this</a>.<br>
Now, this magical(to me) line of code - torch.set_default_device(device) creates all the tensors on the device - for me it is cuda.<br>
I want to know how would I implement this same line in cpp and things to keep in mind.<br>
Currently I have tried assigning some tensors(probably all), model and optimizer to CUDA. <a href="https://github.com/MohitKumar020291/DPCF/blob/main/forum.cu" rel="noopener nofollow ugc">Code link</a>.<br>
Thank you!</p>

username: EikanWang
cooked: <p><a class="mention" href="/u/mohit_kumar">@Mohit_Kumar</a> , the link raises 404.</p>

username: Mohit_Kumar
cooked: <p>My bad, the code was private - now it is public.</p>

username: Dirk10001
cooked: <p>You code initializes the device to CPU, checks for CUDA and switch to CUDA if a card is found:</p>
<pre><code class="lang-auto">torch::Device device = torch::kCPU;

//-----------------------------------------------//
void initialize_device() {
    if (torch::cuda::is_available()) {
        device = torch::kCUDA;
        std::cout &lt;&lt; "Using CUDA device\n";
    } else {
        std::cout &lt;&lt; "Using CPU device\n";
    }
}
</code></pre>
<p>If you just want CPU, change initialize_device</p>
<pre><code class="lang-auto">void initialize_device() {
}
</code></pre>
<p>Now it will be CPU.</p>

username: Mohit_Kumar
cooked: <p><a class="mention" href="/u/dirk10001">@Dirk10001</a> thanks for reading the code. We have set_default_device(my_device) in python - then my tensors over that script, probably my model and optimizer are initialized my_device. But we don’t have something similar for cpp.</p>
TORCHLIB/C++ crash Unreal Engine 4 when loading torch_cpu.dll

username: gvecchio
cooked: <p>Hi, I’m facing an issue when trying to run a network inside Unreal Engine.</p>
<p>I was able to successfully compile the Unreal project with torchlib and I’m also able to perform some basic operations inside the engine, like tensors creation and jit modules loading.<br>
The issue arises when trying to run the forward method. I’ve seen the issue reported some times in other posts online but without a definitive solution. Trying to understand where the issue is actually, I realized it is (at least it looks like that) on the loading of the torch_cpu.dll, which crashed the engine.<br>
I’m not able to understand if it’s a libtorch or an Unreal related problem.</p>
<p>Any help would be appreciated.</p>

username: Nick_Tsaizer
cooked: <p>hey, do you found any way to import libtorch onto unreal engine without using dark magic and huge amount of vodka?</p>

username: Nick_Tsaizer
cooked: <p>upd: i traced loading process to WindowsPlatformProcess.cpp (1950th line) and it looks like its crashing on LoadLibrary() call with “torch_cpu” as an argument. So i have exactly the same problem</p>

username: nypox
cooked: <p>I was able to run torchscript models at runtime using a little wrapper over libtorch stuff in UE5 EarlyAccess2. But the exact same code doesn’t work since Prevew1. The engine started to mess up with libs at the dll loading stage (just stuck at 75%). Something is definitely broken there.</p>

username: FedericoDM
cooked: <p>I have a similar problem. For me, everything works fine in UE4, but it freeze (no crash) when loading <code>torch_cpu.dll</code> in UE5.</p>
<p>Has anyone here come to a solution?</p>

username: lavoos
cooked: <p>I experience freeze on 75% on this setup - UE 5.3.2, Win 11, MSVS 2022, libtorch - downloaded .zip from official site, versions 2.0.1, and 2.1.0<br>
Unyone resolved this somehow?</p>

username: sachithpd
cooked: <p>Has anyone able to resolved this?</p>
Why is there idle time between kernels before and after gpu_memset?

username: Ning_Wang
cooked: <p>Hello,</p>
<p>While using NVIDIA Nsight Systems to analyze the performance of a Pytorch program, I noticed an issue as shown in the attached figure: there is a significant idle period between the <code>gpu_memset</code> operation and the kernel executions before and after it.<br>
Why does the <code>gpu_memset</code> operation result in such long kernel gaps? Can I avoid this?<br>
Thanks!!!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cea3aff98c501affa2fae5b9c9280fd106c9264.png" data-download-href="https://discuss.pytorch.org/uploads/default/6cea3aff98c501affa2fae5b9c9280fd106c9264" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cea3aff98c501affa2fae5b9c9280fd106c9264_2_690x310.png" alt="image" data-base62-sha1="fxvp08bII7C3PdSinJ8G3izpVHK" width="690" height="310" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cea3aff98c501affa2fae5b9c9280fd106c9264_2_690x310.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cea3aff98c501affa2fae5b9c9280fd106c9264_2_1035x465.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cea3aff98c501affa2fae5b9c9280fd106c9264.png 2x" data-dominant-color="E3E6E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1035×465 31.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And my example code is:</p>
<pre><code class="lang-auto">from torch import nn
import torch


torch.cuda.set_device(1)

conv = nn.Conv2d(384, 384, kernel_size=(3, 1), padding=(1, 0)).to(device="cuda:1").eval()

x = torch.randn(1, 384, 8, 8).to(device="cuda:1")

conv(x)

g = torch.cuda.CUDAGraph()

with torch.cuda.graph(g):
    conv(x)

g.replay()
</code></pre>
Torch.cuda.is_available() returns False with cuda12.3

username: leewww
cooked: <p>Our previous environment can use GPU well, but I don’t know why, something went wrong these days.</p>
<pre><code class="lang-auto">python -c "import torch; print(torch.cuda.is_available())"
</code></pre>
<p>and it returns:</p>
<pre><code class="lang-auto">/opt/conda/lib/python3.8/site-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() &gt; 0
False
</code></pre>
<p>Some information needs to be explained:</p>
<ol>
<li>This environment is in a container and I don’t have root privileges</li>
<li><code>nvidia-smi</code></li>
</ol>
<p><img src="https://github.com/user-attachments/assets/8c3ca5f6-3961-4641-92bd-75527c5a88a1" alt="Image" width="492" height="500"><br>
<code>nvcc -V</code></p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Nov__3_17:16:49_PDT_2023
Cuda compilation tools,release 12.3, V12.3.103
Build cuda_12.3.r12.3/compiler.33492891_0
</code></pre>
<ol start="3">
<li>pytorch version</li>
</ol>
<pre><code class="lang-auto">Version: 2.1.0
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3
Location: /opt/conda/lib/python3.8/site-packages
Requires: filelock, fsspec, jinja2, networkx, nvidia-cublas-cu12, nvidia-cuda-cupti-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-runtime-cu12, nvidia-cudnn-cu12, nvidia-cufft-cu12, nvidia-curand-cu12, nvidia-cusolver-cu12, nvidia-cusparse-cu12, nvidia-nccl-cu12, nvidia-nvtx-cu12, sympy, triton, typing-extensions
Required-by: fastai, torchaudio, torchvision
</code></pre>
<p><code>python -c "import torch; print(torch.version.cuda)"</code>got 12.1, but I think 12.3 should be backward compatible<br>
4. ~/.bashrc also Added relevant paths like</p>
<pre><code class="lang-auto">export PATH=$PATH:/usr/local/cuda/bin
export CUDA_PATH=/usr/local/cuda
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
export MATHLIBS_PATH=$CUDA_PATH/lib64
</code></pre>
<p>What should I do?</p>
Ubuntu 24.04 NCCL Seemingly Randomly Timing Out on All Reduce

username: Tanay_Arora
cooked: <p>The servers I run code on have recently migrated to Ubuntu 24.04 (and nvidia driver 565.57.01). This has caused a seemingly random timeout on all reduce calls. With INFO env variables set, the output is below (this specific one ran for 25 epochs before failing). This was under torch-nightly (2.7.0+cu126) but it persists with torch 2.5.1+cu124 (the logs for this are still being produced). What am I doing wrong (this was the same code I used on Ubuntu 22.04).</p>
<pre><code class="lang-auto">$device:64154:64154 [0] NCCL INFO AllReduce: opCount 10ecf sendbuff 0x7b069cd5b000 recvbuff 0x7b069cd5b000 count 1 datatype 7 op 0 root 0 comm 0x46044020 [nranks=4] stream 0x4625c980                                                                               
$device:64154:64154 [0] NCCL INFO 4 Bytes -&gt; Algo 1 proto 0 time 12.602401
[rank3]:[E114 22:31:00.813063304 ProcessGroupNCCL.cpp:633] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=69328, OpType=ALLREDUCE, NumelIn=2365450, NumelOut=2365450, Timeout(ms)=600000) ran for 600003 milliseconds before timing out.                                                        
[rank3]:[E114 22:31:00.813861279 ProcessGroupNCCL.cpp:2170] [PG ID 0 PG GUID 0(default_pg) Rank 3]  failure detected by watchdog at work sequence id: 69328 PG status: last enqueued work: 69331, last completed work: 69327
[rank3]:[E114 22:31:00.813915780 ProcessGroupNCCL.cpp:671] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank2]:[E114 22:31:00.850481621 ProcessGroupNCCL.cpp:633] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=69328, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 600032 milliseconds before timing out.                                                                    [rank2]:[E114 22:31:00.851808890 ProcessGroupNCCL.cpp:2170] [PG ID 0 PG GUID 0(default_pg) Rank 2]  failure detected by watchdog at work sequence id: 69328 PG status: last enqueued work: 69328, last completed work: 69327                                                                                            [rank2]:[E114 22:31:00.852806006 ProcessGroupNCCL.cpp:668] Stack trace of the failed collective:                                                            
#0 barrier from /path/to/torch/distributed/distributed_c10d.py:4551                  
#1 wrapper from /path/to/torch/distributed/c10d_logger.py:81
</code></pre>
Creating the FlexAttention BlockMask from a mask

username: divinho
cooked: <p>Is it possible to use an existing mask to create the BlockMask? Our masks are quite complex so it’s a bother to have to recreate a <code>mask_mod()</code> function when we already have a function that creates a boolean mask tensor (I know the outputs are not the same but still).</p>
<p>Looking into the code of <code>torch/nn/attention/flex_attention.py</code> I see a <code>_create_block_mask_inner()</code> which calls <code>_convert_mask_to_block_mask()</code>, which seems perfect.</p>
<p>But I see later <code>_create_sparse_block_from_block_mask()</code> is called with <code>mask_mod</code> as an argument. However it’s not clear to me whether <code>mask_mod</code> is actually used here.</p>
<p>So my Q is can one bypass having to create a <code>mask_mod()</code> function or not?</p>
<p><a class="mention" href="/u/ezyang">@ezyang</a></p>

username: divinho
cooked: <p>I think this code may show a path forward though I don’t fully understand it yet: <a href="https://github.com/KellerJordan/modded-nanogpt/blob/master/train_gpt.py#L276" class="inline-onebox" rel="noopener nofollow ugc">modded-nanogpt/train_gpt.py at master · KellerJordan/modded-nanogpt · GitHub</a></p>

username: Chillee
cooked: <p>The easiest way to do this is to make a <code>mask_mod</code> that loads from an existing mask <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>For example,</p>
<pre><code class="lang-auto">existing_mask_tensor: Tensor
def custom_mask_mod(b, h, q_idx, kv_idx):
    return existing_mask_tensor[q_idx, kv_idx]
</code></pre>
<p>This’ll allow it to take advantage of block sparsity in the existing mask, although you’ll still have to:</p>
<ol>
<li>Have the existing mask tensor in memory</li>
<li>Do loads from the existing mask tensor in cases where you have a partial mask.</li>
</ol>

username: divinho
cooked: <p>Hm I thought I tried this and got some low-level error leading to me thinking it was not viable. I’ll try again. Thank you for the reply!</p>

username: divinho
cooked: <p>Realized my lower level error was bcz of not using a sequence length % 128. But now I run into</p>
<pre><code class="lang-auto">torch._dynamo.exc.ObservedException: raised exception ExceptionVariable()
</code></pre>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji only-emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>

username: divinho
cooked: <p>that was smth silly. ran into this issue now, looks like one needs a newer PT: <a href="https://github.com/pytorch/pytorch/issues/137481" class="inline-onebox" rel="noopener nofollow ugc">[FlexAttention] Using FlexAttention with DDP complains about a "higher order optimizer" · Issue #137481 · pytorch/pytorch · GitHub</a></p>
Can ResNet50() takes as input a tensor of shape (1, 3, 95, 64) and output another tensor of same shape (1, 3, 95, 64)

username: Thierry_Roger_Bayala
cooked: <p>Hello. I am trying to figure out an issue. I would like to use Resnet architecture in my project. My task is not classification. I am intending to use the Net that takes an image as input and ouput another of same size. Can ResNet50 be used for such task. If yes could someone explained me.</p>
<p>Kind regards</p>

username: ptrblck
cooked: <p>You can use and manipulate the resnet architecture to output a 4-dom tensor. This would be possible by removing the classifier of the model. However, since resnet is downsampling in the feature extraction part of the model you would need to upsample the activations again.</p>
<p>Given than, I would rather just use a segmentation model which already does this.</p>