akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Difference between DDP vs FSDP.NO_SHARD

username: bigtree
cooked: <p>Could anyone know the difference between <a href="https://pytorch.org/docs/stable/nn.html#module-torch.nn.parallel" rel="noopener nofollow ugc">DistributedDataParallel</a></p>
<p><a href="https://pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.ShardingStrategy" rel="noopener nofollow ugc">fsdp.ShardingStrategy = NO_SHARD </a></p>
<p>They looks pretty similar. regarding memory usage and training speed are they the same ?</p>

username: XWu
cooked: <p>I think these 2 are the same. cc <a class="mention" href="/u/agu">@agu</a> <a class="mention" href="/u/weifengpy">@weifengpy</a></p>

username: agu
cooked: <p>They are the same high-level algorithm: vanilla data parallelism. However, their implementation differs.</p>
<p><code>DistributedDataParallel</code> (DDP) uses a C++ reducer to bucket gradients for all-reduce. <code>FullyShardedDataParallel</code> with <code>NO_SHARD</code> follows the module wrapping to bucket gradients for all-reduce.</p>
<p>DDP is more mature and generally the preferred solution over <code>NO_SHARD</code>.</p>

username: bigtree
cooked: <p>Since they are the same, is there a plan to replace it with the FSDP one?</p>
Is the nn.Transformer package missing nn.Generate

username: kirk86
cooked: <p>It seems like the <code>nn.Transformer</code> module might benefit from using a complementary <code>nn.Generate</code> module that allows for different decoding/sampling/generation methods to be used easily with the existing framework.</p>
<p>For instance,</p>
<pre><code class="lang-auto">class DummyModel(nn.Module, nn.Generate):
    def __init__(self, *args, **kwargs):
         # set some model ops
         self.input = nn.Linear(...)
         self.embedding = nn.Embedding(...)
         self.model = nn.Transformer(...)

    def forward(self, x):
        return self.model(self.embedding(self.input(x)))

model = DummyModel(...)

# perform training
for x,y in batch:
     y_hat = model(x)
     loss = criterion(y, y_hat)

# perform inference
generated_output = model.generate(..., method=['greedy', 'beam-search', 'top-k', 'nucleus', etc.])
</code></pre>
<p>Are there any plans to have something like that implemented in the future?</p>
Loading weights with load_state_dict takes very long

username: Adam-Mazur
cooked: <p>I’m writing my own implementation of llama 2 model in PyTorch and I want to load the weight from hugging face into my model. The problem is that the <code>state_dict</code>’s don’t match, so I wrote a method for modyfing the <code>state_dict</code> of the original llama 2 model and loading it with <code>load_state_dict</code>. My code works, but it takes around few minutes to execute, whereas loading the weights from hugging face’s hub only takes about 30 seconds on my machine. Is there a way I can make my code run faster?</p>
<pre><code class="lang-auto">def from_llama_state_dict(llama_state_dict, config, pruning_rates=None):
    if isinstance(config, LlamaConfig):
        config = LazyLlamaConfig.from_llama_config(pruning_rates, config)
    elif not isinstance(config, LazyLlamaConfig):
        raise ValueError("Config must be an instance of either LlamaConfig or LazyLlamaConfig.")
        
    new_state_dict = OrderedDict((modify_key(key), value) for key, value in llama_state_dict.items())

    model = LazyLlamaForCausalLM(config)
    model.load_state_dict(new_state_dict)

    return model
</code></pre>
<pre><code class="lang-auto">def modify_key(key):
    if "model.layers" in key:
        temp = key.split(".")
        temp.insert(3, "decoder")
        return ".".join(temp)
    else:
        return key
</code></pre>
PPO for Discrete Action Spaces (CartPole)

username: ajberlier
cooked: <p>I followed the TorchRL getting started documentation and I am running into issue with not being able to learn the cartpole environment with PPO. The tutorial learns the double pendulum environment no problem, but when I change to cartpole and modify the probabilistic actor for a discrete output it does not learn. perhaps I am not making the proper modifications?</p>
<p>Thank you for any help you can offer.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3cf249662b02db740fa2731fc582fdbe07d067db.png" data-download-href="https://discuss.pytorch.org/uploads/default/3cf249662b02db740fa2731fc582fdbe07d067db" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3cf249662b02db740fa2731fc582fdbe07d067db_2_690x308.png" alt="image" data-base62-sha1="8H9Kcf0U54C5mUjCQ7SJDjdOOsH" width="690" height="308" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3cf249662b02db740fa2731fc582fdbe07d067db_2_690x308.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3cf249662b02db740fa2731fc582fdbe07d067db_2_1035x462.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/c/3cf249662b02db740fa2731fc582fdbe07d067db.png 2x" data-dominant-color="FAFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1142×511 41.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ajberlier
cooked: <p><a href="https://www.reddit.com/r/reinforcementlearning/comments/1eer8iv/why_is_my_ppo_algorithm_not_learning/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://www.reddit.com/r/reinforcementlearning/comments/1eer8iv/why_is_my_ppo_algorithm_not_learning/</a></p>
<p>This Reddit post sorted things out for me. I am still not sure why this is the case though, so explanation would still be great. Thank you!</p>

username: vmoens
cooked: <p>We fixed these tutorials a few weeks ago, maybe worth checking the version on the main branch?<br>
<a href="https://pytorch.org/rl/main/tutorials/getting-started-5.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/rl/main/tutorials/getting-started-5.html</a></p>
How to disable fault tolerance in Torchrun

username: DanielWang
cooked: <p>Is there any way to disable the fault tolerance in Torchrun for debugging purposes?<br>
<a href="https://pytorch.org/tutorials/beginner/ddp_series_fault_tolerance.html" class="inline-onebox" rel="noopener nofollow ugc">Fault-tolerant Distributed Training with torchrun — PyTorch Tutorials 2.0.1+cu117 documentation</a> as shown in this link, the current solution will take the entire process group to a restart where I cannot find the real problem after it runs into crashes.</p>
<p>Thanks!</p>

username: wconstab
cooked: <p>I’m not sure, but can you try <code>--max_restarts=1</code> to see if that produces the crash earlier?  (IIUC you want the system to crash fast, not attempt to restart?)</p>

username: DanielWang
cooked: <p>it seems that the <code>torch.elastic</code> reset the NCCL context as well, is there any way to avoid restarting the NCCL(i.e., <code>torch.distributed</code>). Is there any way to let <code>torch.distributed</code> only retry the last failed API call (e.g., allreduce) instead of restarting the whole <code>torch.distributed</code> context?</p>

username: DanielWang
cooked: <p>Yes, I want the system just crash and not attempt to restart.</p>

username: samsja
cooked: <p>curious if anybody has found a solution for this</p>

username: zzibc
cooked: <p>Do not set --rdzv-id, --rdzv-backend, --rdzv-endpoint.</p>
How do I split an iterable dataset into training and test datasets?

username: Noman-Tanveer
cooked: <p>I have an iterable dataset object with all of my data files. How can I split it into train and validation set. I have seen a few solutions for custom datasets but iterable does not support <code>len()</code> operator. <code>torch.utils.random_sample()</code> and <code>torch.utils.SubsetRandomSample()</code> don’t work.</p>
<pre><code class="lang-auto">def __init__(self):
         bla
         bla
 
 def __iter__(self):
     bla
     .
     yield batch
</code></pre>

username: ptrblck
cooked: <p>I don’t think you could split the samples in the <code>IterableDataset</code> as it’s used for e.g. streams of data.<br>
In case your dataset is predefined, you could check if the stream source is able to split and shuffle the data.</p>

username: StefanK
cooked: <p>IMHO it is possible, but we have to load the date 2 or more times.</p>
<p>First, if our dataset returns pandas.DataFrame we can use:</p>
<pre><code class="lang-auto">import torch.utils.data as td
class SplitDataSet (td.IterableDataset):
  def __init__ (self, dataset, ranges : dict [ty.Tuple], generator = torch.Generator ()):
    self.dataset = dataset
    self.ranges = ranges
    self.generator = generator

  def __iter__ (self):
    self.iter = iter (self.dataset)
    self.rng = self.generator.clone_state ()
    return self

  def __next__ (self):
    df = next (self.iter)
    size = len (df)
    rs = torch.rand (size, generator = self.rng)
    res = {}
    for key, xy in self.ranges.items ():
      idxs = (xy [0] &lt;= rs) &amp; (rs &lt; xy [1])
      res [key] = df.loc [idxs.numpy ()]
    return res
</code></pre>
<p>NB: We must create the Dataloader by:</p>
<pre><code class="lang-auto">ranges = {'train': (0.0, 0.8), 'val': (0.8, 1.0), 'test':(0.8,1.0)}
dl = td.DataLoader (SplitDataSet (ds, ranges), batch_size = None)
</code></pre>
<p>I.e. our batches are the dataframes itself. We must transform them into Tensors.</p>
<p>Alternativly, we can split each item by itself:</p>
<pre><code class="lang-auto">class SplitItemDataSet (td.IterableDataset):
  def __init__ (self, dataset, ranges : dict [ty.Tuple], generator = torch.Generator ()):
    self.dataset = dataset
    self.ranges = ranges
    self.generator = generator

  def __iter__ (self):
    self.iter = iter (self.dataset)
    self.rng = self.generator.clone_state ()
    return self

  def __next__ (self):
    x = next (self.iter)
    num = int (torch.rand ((1,1), generator = self.rng))
    res = {}
    for key, xy in self.ranges.items ():
      use = (xy [0] &lt;= num) &amp; (num &lt; xy [1])
      res [key] = x if use else None
    return res

class FilterDataSet (td.IterableDataset):
  def __init__ (self, dataset : td.IterableDataset, pred : ty.Callable):
    self.dataset = dataset
    self.pred = pred
  def __iter__ (self):
    self.iter = iter (self.dataset)
    return self
  def __next__ (self):
    while True:
      x = next (self.iter)
      if self.pred (x): return x

dl = td.DataLoader (FilterDataSet (SplitItemDataSet (dataset, ranges),
           lambda d: d ['train'] is not None))
</code></pre>
Is possible export the model on cpu and run it gpu?

username: 111357
cooked: <p>As the title said. If no, how to export a model that can be runed on GPU.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch.export import export 

class ToyModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = torch.nn.Linear(10, 10)
        self.fc2 = torch.nn.Linear(10, 10)
        self.fc3 = torch.nn.Linear(10, 10)
    
    def forward(self, x: torch.Tensor, y: torch.Tensor) -&gt; torch.Tensor:
        
        a = torch.sin(x)
        b = torch.cos(y)
        y = self.fc1(x)
        return a + b + y
f = ToyModule()

example_args = (torch.randn(10, 10), torch.randn(10, 10))
out_ref = f(*example_args)

exported_program: torch.export.ExportedProgram = export(
    f, args=example_args
)
print(exported_program) 


out = exported_program.module()(*example_args)
assert torch.allclose(out, out_ref)

</code></pre>

username: ptrblck
cooked: <p>Your code fails with:</p>
<pre data-code-wrap="python"><code class="lang-python">ValueError: Expected `mod` to be an instance of `torch.nn.Module`, got &lt;class 'function'&gt;.
</code></pre>
<p>for me. Which PyTorch version are you using?</p>

username: 111357
cooked: <p>Oh, the <code>export</code> removed the support for function, and requires a module. <a href="https://github.com/pytorch/pytorch/pull/117528" class="inline-onebox" rel="noopener nofollow ugc">[export] require Module to be passed to export by suo · Pull Request #117528 · pytorch/pytorch · GitHub</a></p>
<p>I revised my sample code, please try again. Thx!</p>
<p>My torch version is <code>2.3.0a0+git71d0202</code>.</p>

username: ptrblck
cooked: <p>Thanks for updating the code!<br>
Based on the new code it should work fine exporting the model on the CPU and loading and running it on the GPU:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.export.save(exported_program, "test.pt")
m = torch.export.load("test.pt")
print(m)

m.module()(*[e.cuda() for e in example_args])
# RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

m.module().cuda()
out = m.module()(*[e.cuda() for e in example_args])
assert torch.allclose(out.cpu(), out_ref)
</code></pre>

username: 111357
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="200865">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>module</code></p>
</blockquote>
</aside>
<p>Cool, it works, thanks a lot!</p>
How F.adaptive_max_pool work when input size smaller than the target size?

username: Antinomy20001
cooked: <p>In the below case, the input.shape is (5, 7) but target size is (7, 7). Accoding to the standard max pooling formula, the input size seems to be smaller than the target size if padding = 0.</p>
<p>I think there is a padding stage before the max_pooling operation, but how it work ?</p>
<p>Thanks for your help.</p>
<pre><code class="lang-python">import torch
import torch.nn.functional as F

input = torch.rand((1, 3, 5, 7))
output = F.adaptive_max_pool2d(input, (7,7))
</code></pre>

username: zoraiz_ramay
cooked: <p>explaining pattern with example:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

input = torch.rand((1, 1, 5, 7))
output = F.adaptive_max_pool2d(input, (7,7))
</code></pre>
<p>produces following input and output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/091b0f850e670af15823e5c6cd7c1e2a53c02302.png" data-download-href="https://discuss.pytorch.org/uploads/default/091b0f850e670af15823e5c6cd7c1e2a53c02302" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/091b0f850e670af15823e5c6cd7c1e2a53c02302.png" alt="image" data-base62-sha1="1iyh0C9snLzE2QfhZNYUG3jVbKG" width="690" height="475" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">799×551 12.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
In this example, tensor(5,7) was upsampled to tensor(7,7). Here no. of columns are same but rows are different(5 vs 7). Now view this process as stretching spring with highlighted rows(in above figure) are kept same. As we are upsampling(stretching), we need two rows in between highlighted rows in output as opposed to one in input. It is done by maxpooling column-wise. Take first column as example:<br>
First column of input is</p>
<blockquote>
<p>0.8841<br>
0.6107<br>
0.7334<br>
0.6350<br>
0.5692</p>
</blockquote>
<p>First column of output is</p>
<blockquote>
<p>0.8841<br>
max(0.6107, 0.8841)<br>
max(0.6107, 0.7334)<br>
0.7334<br>
max(0.6350, 0.7334)<br>
max(0.6350, 0.5692)<br>
0.5692</p>
</blockquote>
<p>This pattern is repeated for all columns and you get upsized output as in figure.</p>
Understanding set_requires_grad

username: brunorcabral
cooked: <p>I have this tensor</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">x_tensor = torch::empty({size, 784});
</code></pre>
<p>And I am using <code>set_requires_grad()</code> before training</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">x_tensor.set_requires_grad(true);
</code></pre>
<p>The function <code>set_requires_grad()</code> returns a <code>Tensor</code>.<br>
It it the same <code>x_tensor</code>? Should I capture it? Should I use it? Is <code>set_requires_grad()</code> changing <code>x_tensor</code> in place?</p>

username: ptrblck
cooked: <p>You would need to assign and use the returned tensor. Alternatively, you can use the inplace operation via <code>.requires_grad_()</code> (note the trailing underscore).</p>

username: brunorcabral
cooked: <p>I’ll probably use inplace <code>requires_grad_()</code> then, thanks.<br>
But just to be clear, with <code>set_requires_grad()</code>, do I replace <code>x_tensor</code>?</p>
<pre><code class="lang-auto">x_tensor = x_tensor.set_requires_grad(true);
</code></pre>

username: ptrblck
cooked: <p>Yes, your approach with reassigning the tensor works too.</p>
Model not releasing memory even with torch.no_grad and torch.cuda.empty_cache()

username: vyrsh
cooked: <p>I have created a model using this <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">transfer learning tutorial</a> however when I use it the GPU memory finishes because whenever I do model(tensor) some memory is occupied. the larger the tensor the larger the memory occupied. and this is not released even upon torch.cuda.empty_cache() and even when I use with torch.no_grad.</p>
<p>this is the error I get<br>
<code>CUDA out of memory.  Tried to allocate 5.86 GiB.  GPU 0 has a total capacity of 15.89 GiB of which 3.68 GiB is free.  Process 3018 has 12.21 GiB memory in use.  Of the allocated memory 11.74 GiB is allocated by PyTorch, and 181.37 MiB is reserved by PyTorch but unallocated.  If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)</code></p>
<p>the process 3018 line I’m getting for the first time , however it most probably is just pytorch<br>
<a href="https://www.kaggle.com/code/vyrsh0/schizophrenia-eeg" rel="noopener nofollow ugc">here</a> is the kaggle link</p>

username: ptrblck
cooked: <p>Calling <code>torch.cuda.empty_cache()</code> will clear the cache but will not release allocated and used memory. If you want to release all memory you need to delete all objects allocating memory first. The error could this be expected assuming an intermediate tensor must to created and triggers the OOM.</p>
A few question about training densenet (and other) models

username: Anthony_Weaver
cooked: <p>I am working on rebuilding some models from Tensorflow into PyTorch.  The attached image is an example of such a model.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/91d89e2afd8643110b1c50641cae1a362b353e3f.png" alt="model" data-base62-sha1="kOdkKZfEECZ2r7sJFWT0l9v3wfB" width="139" height="452"></p>
<p>The data I’m using is pulsar time series data (256 x 256) not images.  I have a couple of questions about doing this using Pytorch’s implementation of DenseNet (and other models as I have several different models I’m trying to re-create).  I am fairly new to all of this so I apologize for any noob questions</p>
<ol>
<li>
<p>Can I use Pytorch’s DenseNet on data other than images?  Further, can I re-train/should I retrain or should I do adaptive training?</p>
</li>
<li>
<p>Does this snippet of code appear to be on the correct track for implementing the model pictured?  If I do need to train the DenseNet<br>
model would I have to add anything else to this or is it good enough to load the model without weights?<br>
In this code cnn_layer would be densenet121</p>
</li>
</ol>
<pre><code class="lang-auto">class _FreqBlock(nn.Module):
    def __init__(self, model: string, cnn_layer: string) -&gt; None:
            super().__init__()
        cnn_layer = model_params[model]["freq_cnn"]
        units = model_params[model]["units"]

        # Readjust the input size for the model to match our input
        first_layer = [nn.Conv2d(in_channels=1, out_channels=3, kernel_size=(2,2), stride=(1,1), padding="valid", dilation=(1,1), bias=True),]
        first_conv_layer.extend(list(model.features))  
        cnn_layer.features= nn.Sequential(*first_conv_layer)

        self.cnn = torch.hub.load("pytorch/vision", cnn_layer, weights=None)
        self.block = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=3, kernel_size=(2,2), stride=(1,1), padding="valid", dilation=(1,1), bias=True),
            nn.ReLU(),
            cnn_layer(), # Run the densenet121 model, will this train the model?
            nn.BatchNorm2d(num_features=freq_units, eps=0.001, momentum=0.99),
            nn.Dropout(p=0.3),
            nn.Linear(in_features=freq_units,outfeatures=units),
        )

    def forward(self, dm: Tensor) -&gt; Tensor:
        return self.block(dm)
</code></pre>
Parallelization of the for-loop in federated learning

username: cctsai
cooked: <p>I’m curious about the implementation of parallelizing the for-loop in federated learning. The for-loop I am referring to is the one that iterates through all the selected clients for locally trained models between communication rounds. Most simulations loop through the selected clients serially, but I want this for-loop to be executed in parallel.</p>
<p>Specifically, given <em>N</em> GPUs and <em>K</em> clients, there are three possible relationships between <em>N</em> and <em>K</em>. Starting with the case <em>N == K</em>, I would like each GPU to train a model on one client’s data in parallel. To achieve this, I imagine creating <em>N</em> processes, with each process executing the training task. However, I have no idea where to start, so I would appreciate some guidance on this. Additionally, I am wondering if this problem could also be solved using DDP (<code>DistributedDataParallel</code>).</p>

username: XWu
cooked: <p>my understanding of the use case is, feeding same input &lt;X_0, … &gt; to N different models &lt;M_1, …, M_n&gt;. I’m not sure if any of the existing parallelism fits into the shoe.</p>
<p>For DDP, the gradient is all-reduced to locally update all replicated models.<br>
For FSDP, the gradient is reduce-scattered to locally update the model shards.</p>
<p>Perhaps DDP with different initial model weights across processes is the closest one, but I’m not sure if this setting can lead to training convergence.</p>
Bug: REM The OpenCL loader ignores value of this variable if running with elevated privileges

username: Weiii_Tod
cooked: <p>hi, when i install pytorch, report those errors：</p>
<p>REM The OpenCL loader ignores value of this variable if running with elevated privileges</p>
<p>set “OCL_ICD_FILENAMES=D:\Softwares\Anaconda\setup\envs\pytorch_gpu\Library\lib\intelocl64.dll”</p>
<p>how can i solve it? i searched the google but seems there is little information about it.</p>

username: kh_h
cooked: <p>how did you solve it?i meet the same problem <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"></p>
Progress callback // Interrupting execution of `torch::jit::Module::forward()`

username: hugofloresgarcia
cooked: <p>Hi!</p>
<p>I’m using a torchscript model (exported from python) to perform a forward pass on a relatively large audio tensor in a C++ program. Because the forward pass may take a considerably long amount of time, I’d like for the user to be able to cancel the forward pass halfway through.</p>
<p>A (rather hacky) workaround for this would be to perform the forward pass in a separate thread, and detach it in case the user wishes to cancel:</p>
<pre><code class="lang-cpp">torch::Tensor input;
torch::Tensor output;

std::atomic&lt;bool&gt; done = {false};

torch::jit::Module model;

auto thread = std::thread(
   [&amp;model, &amp;input, &amp;output, &amp;done]()
   {
      // forward pass
      std::vector&lt;torch::jit::IValue&gt; inputs = {input};
      output = model-&gt;forward(inputs).ToTensor();

      done = true;
   }
);

// wait for the thread to finish
while (!done)
{
   if (UserWantsToCancel())
   {
      // abort if requested
      thread.detach();
      return;
   }

   wxMilliSleep(50);
}

thread.join();
</code></pre>
<p>I was wondering if the C++ torch API provided an interface for interrupting the execution of a forward pass, perhaps through some sort of callback or hook?</p>
<p>Any help would be appreciated! Thanks so much!</p>

username: spiegelball
cooked: <p>I am very interested in that too. Seems there simply is no option to terminate a forward() execution midtime.</p>

username: jonahclarsen
cooked: <p>Same here (although with the Python frontend)! Any pointers at all would be greatly appreciated.</p>
How does the whl package in pytorch.org be make?

username: hello_e
cooked: <p>I have always had a doubt. If the whl package on PyTorch’s official website was made on a CPU that supports AVX512, and my CPU does not support AVX512, what would happen if I pip install the whl package on the official website?</p>

username: ptrblck
cooked: <p>I would expect to see a <code>SIGILL, Illegal instruction</code> being raised during the <code>import</code> if your CPU does not support the needed instruction set.</p>

username: hello_e
cooked: <p>I see, I thought there was some black magic that could make the WHL package on PyTorch’s official website run on any CPU, hahaha</p>
<p>—Original—</p>
Cuda is not working

username: Ayush_Verma
cooked: <p>Recently i installed cuda toolkit from nvidia of version 12.6 but its not working when i m trying to acess with torch why is it so? i downloaded torchvision 12.4  could u please lemme know how to solve thsi issue?</p>

username: ptrblck
cooked: <p>Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension since the PyTorch binaries ship with their own CUDA runtime dependencies.</p>

username: Ayush_Verma
cooked: <p>How to built it ? Could you lemme know?</p>

username: ptrblck
cooked: <p>You don’t need to build PyTorch and can just install the binaries (pip wheels or condo binaries) as described <a href="https://pytorch.org/get-started/locally/">here</a>.</p>

username: Ayush_Verma
cooked: <p>I tried and downloaded that for 12.4 but still it’s not working.</p>

username: ptrblck
cooked: <p>Could you post the installation log as well as the output of <code>python -m torch.utils.collect_env</code>?</p>

username: Ayush_Verma
cooked: <p>Its says no module named torch.utils.collect_env?</p>
Speedup depthwise conv with AMP/Tensor cores on GPU Nvidia V100

username: Maxime_Song
cooked: <p>Hello,</p>
<p>I am right now training with AMP a vision model which is using a lot of depthwise convolution, and I am trying to speedup the training.<br>
The profile indicates that most of GPU time is used for kernel related to depthwise convolution and that it is not using Tensor Core.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f1b0285b2791873d1d23e9d1ae7f33e55f02574.png" data-download-href="https://discuss.pytorch.org/uploads/default/7f1b0285b2791873d1d23e9d1ae7f33e55f02574" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f1b0285b2791873d1d23e9d1ae7f33e55f02574_2_690x133.png" alt="image" data-base62-sha1="i8qvrfz5CaQ6CFrxkJcEkCtsLze" width="690" height="133" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f1b0285b2791873d1d23e9d1ae7f33e55f02574_2_690x133.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f1b0285b2791873d1d23e9d1ae7f33e55f02574_2_1035x199.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f1b0285b2791873d1d23e9d1ae7f33e55f02574_2_1380x266.png 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3219×625 215 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does anyone know if it is possible to speedup depthwise convolution with tensor core ? Thanks a lot!</p>
Facing issue while installing a package

username: maniron28
cooked: <p>I am trying to install spatial correlation sampler packager for one of my application. I am installing it through pip install spatial-correlation-sampler</p>
<p>When ever I try to install I am getting an error</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “C:\Program Files\Python311\Lib\site-packages\pip_vendor\pep517\in_process_in_process.py”, line 351, in<br>
main()<br>
File “C:\Program Files\Python311\Lib\site-packages\pip_vendor\pep517\in_process_in_process.py”, line 333, in main<br>
json_out[‘return_val’] = hook(**hook_input[‘kwargs’])<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “C:\Program Files\Python311\Lib\site-packages\pip_vendor\pep517\in_process_in_process.py”, line 152, in prepare_metadata_for_build_wheel<br>
return hook(metadata_directory, config_settings)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “C:\Program Files\Python311\Lib\site-packages\setuptools\build_meta.py”, line 377, in prepare_metadata_for_build_wheel<br>
self.run_setup()<br>
File “C:\Program Files\Python311\Lib\site-packages\setuptools\build_meta.py”, line 484, in run_setup<br>
self).run_setup(setup_script=setup_script)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “C:\Program Files\Python311\Lib\site-packages\setuptools\build_meta.py”, line 335, in run_setup<br>
exec(code, locals())<br>
File “”, line 69, in<br>
File “”, line 48, in launch_setup<br>
File “C:\Program Files\Python311\Lib\site-packages\torch\utils\cpp_extension.py”, line 1076, in CUDAExtension<br>
library_dirs += library_paths(cuda=True)<br>
^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “C:\Program Files\Python311\Lib\site-packages\torch\utils\cpp_extension.py”, line 1214, in library_paths<br>
paths.append(_join_cuda_home(lib_dir))<br>
^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “C:\Program Files\Python311\Lib\site-packages\torch\utils\cpp_extension.py”, line 2416, in _join_cuda_home<br>
raise OSError('CUDA_HOME environment variable is not set. ’<br>
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.</p>
</blockquote>
<p>My echo output</p>
<blockquote>
<p>echo %CUDA_HOME%<br>
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6</p>
</blockquote>
<p>Even though cuda tool kit is installed and this CUDA_HOME env is set I am getting the same error. I am trying this in windows os. If there is any solution kindly let me know</p>
Flex Attention Extremely Slow

username: whywhy
cooked: <p>I try to use flex attention in huggingface transformer, only to find it very slow. Compared to the sdpa implementation, flex attention is about 4-5 times slower, but it does save the CUDA memory.</p>
<p>Here is the example code: <a href="https://gist.github.com/why-in-Shanghaitech/8b8205f98568c6741a2e38dfcdb9d362" class="inline-onebox" rel="noopener nofollow ugc">Example benchmark for flex attention. · GitHub</a></p>

username: whywhy
cooked: <p>My bad. See <a href="https://github.com/pytorch/pytorch/issues/136261" class="inline-onebox" rel="noopener nofollow ugc">Flex Attention Extremely Slow · Issue #136261 · pytorch/pytorch · GitHub</a>.</p>
Permissions to use Pytorch logo in product

username: Ajay_Dogra
cooked: <p>Hello<br>
We (at google) would like to use the pytorch logo as part of our Cloud AI Platform services. The logo will be displayed on one or more web pages.<br>
Do we have permissions to use the logo? Let us know if you have any questions.</p>
<p>Thanks.</p>

username: smth
cooked: <p>The answer will be yes, but can you please email me at [my-first-name]<span class="mention">@pytorch.org</span> so that we can have the request and response on record.</p>

username: Jim_Tung
cooked: <p>We (at MathWorks) would like to use the pytorch logo when describing that our Simulink product can integrate pytorch models. The logo will be displayed in customer-facing presentations and possibly in one or more webpages. Do we have permission to use the logo?</p>
<p>Thanks.<br>
Jim</p>
Faster Alternative to a Loop

username: pookncheek
cooked: <p>Hi,</p>
<p>My model has a bottleneck from this loop</p>
<pre><code class="lang-auto">a = torch.zeros(B, N, D1, D2)
for t in range(N):
    Bt = b[:,:t+1,:]              #B,T,D1
    Ct = c[:,:t+1,:]              #B,T,D2
    a[:,t,:,:] = Bt.mT @ Ct       #B,D1,T @ B,T,D2 -&gt; B,1,D1,D2
</code></pre>
<p>It takes two 3d tensors - <code>b</code> &amp; <code>c</code> - of shape <code>(B,N,D)</code>, does a transpose for rows 0:t, and then outputs - <code>a</code> - a 4d tensor of shape <code>(B, N, D1, D2)</code>.  But the loop gets really slow (especially in back-propagation) as N gets large.  Is there a faster alternative?  Some way to do this all with matrix multiplication?</p>
<p>Thanks in advance for your help!!!</p>
<p>FWIW, I eventually do a reshape of <code>a</code> to be <code>(B,N,D1*D2)</code> so solutions in that shape work too!</p>

username: ptrblck
cooked: <p>Did you try to apply <code>torch.compile</code> to your method to check if it would help?</p>

username: J_Johnson
cooked: <p>You could try this:</p>
<pre><code class="lang-auto">import torch

B = 2
N = 3
D1 = 4
D2 = 5

a = torch.zeros(B, N, D1, D2)
b = torch.rand(B, N, D1)
c = torch.rand(B, N, D2)

def loop(a, b, c): #old loop definition you provided
    N = a.size(1)
    for t in range(N):
        Bt = b[:,:t+1, :]
        Ct = c[:,:t+1, :]
        a[:,t,:,:]=Bt.mT @ Ct
    return a

def parallel(b, c): #parallelized version
    N = b.size(1)
    Bt = b.unsqueeze(1).repeat(1,N,1,1).permute(0,3,2,1).triu().permute(0,3,2,1)
    Ct = c.unsqueeze(1).repeat(1,N,1,1).permute(0,3,2,1).triu().permute(0,3,2,1)
    a = Bt.mT @ Ct
    return a

output_loop = loop(a,b,c)
output_parallel = parallel(b,c)

print(torch.isclose(output_loop, output_parallel))
</code></pre>
<p>Cheers</p>

username: pookncheek
cooked: <p>I had not and this is a great reminder that torch.compile is a super useful feature.</p>
Half precision training time same as full precision

username: marcpg
cooked: <p>Hi all,</p>
<p>I am training a model (~25M params) which takes ~1day to train on the full dataset. In my attempt to reduce the training time I am testing half precision. In using half precision, it allowed me to double the batch size, yet the overall training time remains the same.</p>
<p>I was under the impression that doubling the batch size would reduce training time in ~half since many operations can be parallelized over the batch dimension. But this does not seem to be the case. Could this be a hardware (GPU architecture) or CUDA version problem? Or is my understanding of the relationship between batch size and speed just plain wrong.</p>
<p>I have tried samples to bfloat16 and weights to float32, and both samples and weights to bfloat16 with same training times as both in float32. I am using a NVIDIA Quadro GV100 with 32Gb of VRAM and CUDA 11.4.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Profile your code to narrow down the bottleneck as it seems doubling the batch size is doubling the execution time, which could point to a data loading bottleneck.</p>

username: marcpg
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thank you for your response, you were quite on the spot. After some digging I found that indeed there was a data loading bottleneck as you pointed out, just running the DataLoader on more workers alleviates the problem.</p>
<p>I however also found that using <code>bfloat16</code> is quite slower than <code>float16</code>. My guess is that my GPU architecture or CUDA version is the culprit here, but if anyone has any ideas on how to close the speed gap between <code>bfloat16</code> and <code>float16</code> I welcome them.</p>

username: marcpg
cooked: <p>I have tried running the same model on a RTX4090 with CUDA 12 and <code>bfloat16</code> is now as fast as <code>float16</code>. This indicates that indeed the Quadro GV100 did not support the <code>bfloat16</code> speed up.</p>

username: ptrblck
cooked: <p>Yes, this is correct as <code>bfloat16</code> is supported on Ampere and newer devices. Older devices could emulate it but you should not expect to see a speedup.</p>
Trying to run cuda from torch inline extension, so file is not getting created while loading the module

username: Ashutosh_Dwivedi
cooked: <p>I am new to cuda programming and trying to run a hello world program through notebooks in python like this :</p>
<pre><code class="lang-auto">%load_ext wurlitzer

import torch, os
from torch.utils.cpp_extension import load_inline


os.environ['CUDA_LAUNCH_BLOCKING'] = '1'

def load_cuda(cuda_src, cpp_src, funcs, opt=False, verbose=False):
    return load_inline(cuda_sources=[cuda_src], cpp_sources=[cpp_src], functions=funcs,
                       extra_cuda_cflags=["-O2"] if opt else [], verbose=verbose, name="inline_ext")



cuda_begin = r"""
#include 
#include 
#include 

#define CHECK_CUDA(x) TORCH_CHECK(x.device().is_cuda(), #x " must be a CUDA tensor")
#define CHECK_CONTIGUOUS(x) TORCH_CHECK(x.is_contiguous(), #x " must be contiguous")
#define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)

inline unsigned int cdiv(unsigned int a, unsigned int b) { return (a + b - 1) / b;}
"""

cuda_src = cuda_begin + r"""
__global__ void rgb_to_grayscale_kernel(unsigned char* x, unsigned char* out, int n) {
    int i = blockIdx.x*blockDim.x + threadIdx.x;
    if (i&gt;&gt;(
        input.data_ptr(), output.data_ptr(), w*h);
    C10_CUDA_KERNEL_LAUNCH_CHECK();
    return output;
}"""


cpp_src = "torch::Tensor rgb_to_grayscale(torch::Tensor input);"
module = load_cuda(cuda_src, cpp_src, ['rgb_to_grayscale'], verbose=True)
</code></pre>
<p>This results in an error saying :</p>
<pre><code class="lang-auto">ImportError: /home/zeus/.cache/torch_extensions/py311_cu121/inline_ext/inline_ext.so: cannot open shared object file: No such file or directory
</code></pre>
<p>I have already tried deleting the cache dir and re-running the code but keep getting teh same error. Not sure how to debug this, can someone help?</p>

username: ptrblck
cooked: <p>Could you check the permissions of the cache folder and make sure your Python process has the needed rights?</p>
AnoGAN example, two adversarial networks

username: Dirk10001
cooked: <p>Hi all,<br>
I tried the AnoGAN example, pytorch_cpp-master → Anomaly´_Detection → AnoGAN2d.<br>
Trying void test(… in test.cpp there is a loop</p>
<pre><code class="lang-auto">for (size_t i = 0; i &lt; search_epoch; i++) {
            output = gen-&gt;forward(z);
            anomaly_score_with_al...
</code></pre>
<p>It seems that on Tensor z, size of the latent space, before forwarding the image, the network is retrained, or more trained. Is that necessary before every forwarding?<br>
Is just running a random vector through the network and comparing the output with one image the anomaly detection?</p>
<p>I am lost in test.cpp. Nowhere the image to test is forwarded through the network and the result compared to the input?<br>
The failure images run through test show more or less what I expected.</p>
<p>Many thanks for your help.</p>
Fault tolerance with k8s

username: Shuaijian_Wang
cooked: <p>I found that fault tolerance is achieved by restarting,</p>
<ol>
<li>The number of restarts is maintained in the memory of each training node and is not the same as the number of failures</li>
<li>If a card is dropped, the Pod will be restarted continuously until the maximum number of restarts is reached<br>
I would like to ask if there are other students who have encountered the same problem, and if the community has any methods to deal with this problem</li>
</ol>
torch._C._cuda_init() returns "invalid random_device value"

username: Dom83
cooked: <p>torch._C._cuda_init() returns “invalid random_device value”. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto">&gt;&gt;&gt;torch.__version__
'2.3.0+cu118'
&gt;&gt;&gt;torch.version.cuda
'11.8'
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.cuda.device_count()
1
&gt;&gt;&gt; torch.cuda.current_device()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "D:\apps\stable-diffusion\Forge_20240811_121_21__3\system\python\lib\site
-packages\torch\cuda\__init__.py", line 778, in current_device
    _lazy_init()
  File "D:\apps\stable-diffusion\Forge_20240811_121_21__3\system\python\lib\site
-packages\torch\cuda\__init__.py", line 293, in _lazy_init
    torch._C._cuda_init()
RuntimeError: invalid random_device value
</code></pre>
<p>Can you help, please?</p>

username: Dom83
cooked: <p>solution: <a href="https://discuss.pytorch.org/t/pytorch-2-1-is-no-more-able-to-use-my-gpu/208672/13" class="inline-onebox">Pytorch &gt;= 2.1 is no more able to use my GPU :( - #13 by Dom83</a></p>