akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Gunicorn Workers Not Able to Use GPU in Parallel

sanchit1199: <p>I am trying to deploy a Pytorch image classification model wrapped in <code>Flask</code> on <code>g4dn.xlarge</code> (4 vCPU, 16GB RAM, T4 GPU with 16GB Memory) instances on AWS.</p>
<p>For selecting the optimal number of workers I performed some experiments:</p>
<pre><code class="lang-auto">Note:
* Only `model.forward` part runs on GPU - the rest of the steps run on CPU in the application.
* I have added a timing logger for every step of the application for checking 
</code></pre>
<h3>
<a name="experiment-1-1" class="anchor" href="#experiment-1-1"></a>Experiment 1:</h3>
<pre><code class="lang-auto">gunicorn main.app:app -b 0.0.0.0:8000 --workers 1
</code></pre>
<p>Concurrent Requests: 1<br>
Total Time To Process 15 Requests By A Client: 15.87s (<code>model.forward</code> takes 14.98s)</p>
<h3>
<a name="experiment-2-2" class="anchor" href="#experiment-2-2"></a>Experiment 2:</h3>
<pre><code class="lang-auto">gunicorn main.app:app -b 0.0.0.0:8000 --workers 2
</code></pre>
<p>Concurrent Requests: 2 (2 clients sending requests in parallel)<br>
Total Time To Process 15 Requests By A Client: 29.35s (<code>model.forward</code> takes 28.34s, 2x of a single request, every other step taking a similar time)</p>
<h3>
<a name="experiment-3-3" class="anchor" href="#experiment-3-3"></a>Experiment 3:</h3>
<pre><code class="lang-auto">gunicorn main.app:app -b 0.0.0.0:8000 --workers 3
</code></pre>
<p>Concurrent Requests: 3 (3 clients sending requests in parallel)<br>
Total Time To Process 15 Requests By A Client: 43.82s (<code>model.forward</code> takes 41.81s, 3x of a single request, every other step taking a similar time).</p>
<p>Using 3x workers is enabling me to process 3 requests in parallel but the overall processing time of all those requests is also becoming 3x - hence no improvement in real terms.</p>
<p>I initially thought CPU or IO processes are the bottlenecks in the app - but upon intensively logging the time taken at each step, I found the bottleneck is coming from the GPU processing (<code>model.forward</code> starts taking 2x-3x times).<br>
Upon checking the process ids of the workers for each request - I can also confirm that all the workers are getting the requests in parallel - but those are not able to perform the GPU processings in parallel at the same time.</p>
<p>Any guidance on what can be the bottleneck here will be very helpful.<br>
Also - is there a recommended worker type to be used for such kinds of processing which are GPU-dependent?</p>

Rahul_Agarwal: <p>I am also facing the same issue. Were you able to figure this out</p>
Get Image Embedding Vector and Store it in DATABASE #740

Stefan_Radonjic: <p>Hi,</p>
<p>I am trying to get image embeddings from SAM and save them in a Database, but I am a bit confused with the embedding size.</p>
<p>For example:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/17276f9d89a7125cc5fcf27d3e282308244a6b83.png" alt="image" data-base62-sha1="3iPtkqmM7UzJHFGGM8N4EltkeEb" width="647" height="216"></p>
<p>I see that size is [1, 256, 64, 64]. This is because of the batch size, correct? The embedding should be (1, 256), right?</p>
<p>How can I safely retrieve the correct embedding vector?</p>
A Conv layer in my ResNet has the wrong number of input channels, and I can't find it

kobina: <p>Hi there</p>
<p>I’m new to Pytorch, and I’m solving an image classification problem. I’ve built a ResNet (as opposed to loading it from Pytorch hub because I wanted to learn how they work). Here’s the <a href="https://github.com/maadabrandon/mushroom-identification-service/blob/testing/src/training_pipeline/models.py" rel="noopener nofollow ugc">code</a>. The relevant functions begin at line <span class="hashtag-raw">#257</span>.</p>
<p>One of the convolutional layers has the wrong number of input channels, and I’m struggling to locate the exact source of the problem.</p>
<p>The error reads:</p>
<blockquote>
<p>Given groups=1, weight of size [128, 64, 1, 1], expected input[20, 128, 45, 45] to have 64 channels, but got 128 channels instead</p>
</blockquote>
<p>I realise that many similar questions have been asked, and I’ve tried to implement relevant answers to those questions to no avail.</p>

tecdun: <p>When I run into these I normally add a <strong>print(x.shape)</strong> after line 281 and see which ConvBlock call is initiating the error.  It’s a good starting point.  Break the after the first few prints.</p>

kobina: <p>Thanks for the tip. In addition to printing <code>x.shape</code>, I decided to print <code>self.conv</code>, which is the <code>nn.Sequential</code> object that corresponds to that layer’s contents. Things broke down when the model arrived at the following:</p>
<pre><code class="lang-auto">torch.Size([20, 64, 90, 90])
Sequential(
  (0): Conv2d(128, 32, kernel_size=(1, 1), stride=(1, 1))
  (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
</code></pre>
<p>So <code>x</code> has 64 input channels, but the layer that’s supposed to be accepting it is set up for 128.</p>
<p>I somehow need to change the code to resolve this discrepancy. The trouble is that I don’t know how.</p>
mAP in Object Detection is always 0

Adrian_Contreras_Gar: <p>I’m training an RCNN model into a multiclass dataset, but when i finished configuring the pipeline and started training, everything goes right until I evaluate results. In every epoch the resulting mAP for all classes is 0.0</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/0/20a3bf4cc63900f51ba290f52f357d39866a5cf8.png" alt="image" data-base62-sha1="4EK6zhmlvPquk4pjxiQoChHBW8U" width="650" height="239"></p>
<p>Does anyone know, what could be wrong? Is the mistake related to training (I use the train_one_epoch function) or in evaluation? Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied!

jisu: <p>I’m working on cifar10 dataset and I have a problem.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/856d8f60eab47f1a48237998a7c774a541bc8760.png" alt="image" data-base62-sha1="j2mez7pKj0SVHyFZ6tTi9Aykwvu" width="542" height="421"></p>
<p>RuntimeError: mat1 and mat2 shapes cannot be multiplied (64x1000 and 2048x1000)</p>
<p>I can’t fix this error. The paper says to use pretrained resnet-50. What should I do?</p>

ptrblck: <p>All resnets implemented in <code>torchvision</code> were trained on <code>ImageNet</code> with 1000 classes and thus output logits in the shape <code>[batch_size, nb_classes=1000]</code>. If you want to replace the last linear layer, access it via <code>model.fc</code> and replace it with your new linear layer. Otherwise, if you want to really use these logits directly, set <code>in_features=1000</code> in your linear layer, but note it’s not the common use case.</p>
Saving state dict with optimizer state sharding

Arun_Mallya: <p>I am trying to use OSS to train a large model on a single machine with 4 GPUs and am running into issues when I issue the <code>optimizer.consolidate_state_dict()</code> call. The distributed data parallel job hangs and then finally dies after a long time. The final error log is below:</p>
<pre><code class="lang-auto">[E ProcessGroupNCCL.cpp:781] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=95188, OpType=BROADCAST, Timeout(ms)=1800000) ran for 1807405 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:781] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=95187, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1808126 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:781] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=95187, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1808124 milliseconds before timing out.

Traceback (most recent call last):
  File "train.py", line 143, in &lt;module&gt;
    main()
  File "base.py", line 456, in _consolidate_state_dict
    self.opt.consolidate_state_dict()
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/optim/zero_redundancy_optimizer.py", line 537, in consolidate_state_dict
    local_state_dict = _broadcast_object(
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/optim/zero_redundancy_optimizer.py", line 106, in _broadcast_object
    data_recv_tensor = torch.empty([int(length_tensor.item())], dtype=torch.uint8, device=device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate more than 1EB memory.
</code></pre>
<p>It doesn’t quite make sense that it is trying to allocate 1 exabyte of memory!<br>
This issue is with pytorch 1.13. Any clue what the issue might be?</p>

ionutmodo: <p>any solutions for this? I am facing the same error <strong>torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate more than 1EB memory.</strong> raised in my conda environment in <strong>lib/python3.9/site-packages/torch/distributed/distributed_c10d.py</strong>. This is raised when I am trying to broadcast a list of tensors to the entire group. Please help with this</p>
Data augmentation and augmented data accuracy

tecdun: <p>Hi,</p>
<p>I had quite a low data count so started my own data augmentation because it made sense in my case, and was quite easy to implement.</p>
<p>Without augmentation my CNN was extremely bad, even within the first epoch every batch had the same prediction (other than the first 3-5 batches).  Adding data completely solved this issue (augmented the data 32 fold).</p>
<p>My overall accuracy is very high (over 90%) in training and testing.</p>
<p>My predictions on unknown data (data that was not used in training or testing, not sure if this has a name) is very high on original data, yet only at about 50% on augmented data.  <strong>So my question is, is this normal (Original data accurate, augmented data not that accurate)?</strong></p>
<p>In my case I will always be using original data to predict, so it doesn’t really matter, so its just curiousity on my part.</p>
Listing the layers in a pre-built model, and adding one

SystemLord: <p>Good day.</p>
<ol>
<li>How do I view the layers in a model such as</li>
</ol>
<p>model = torch.hub.load(‘pytorch/vision:v0.10.0’, ‘resnet101’, pretrained=True)</p>
<ol start="2">
<li>How would I add or change the final layer of this model?</li>
</ol>
<p>Thanks.</p>

ptrblck: <ol>
<li>
<p>You can directly <code>print</code> the model to see all registered layers. Alternatively, you can also iterate the <code>.modules()</code>.</p>
</li>
<li>
<p>In the previous step you will see the final layer definition as <code>self.fc</code> and could replace it with your custom layer.</p>
</li>
</ol>

AidanGoldfarb: <p>To add on to what patrick said, I have done this using <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.add_module" rel="noopener nofollow ugc">add_module</a>, not sure if it is the best way, however</p>
Is it possible to let torch profiler return an organized table format rather than a raw string?

Fei_Liu: <p>E.g. sometimes the function names are very long and it’s painful to view the raw string in a console, even with adjusting the <code>max_name_column_width</code> ags in .table().</p>
<p>How can I get a more organized format, e.g. like a pandas dataframe, or even just a raw csv with full content so that I can load it with pd.read_csv or other utils? The current raw string results seem hard to further process if the view is bad.</p>

Fei_Liu: <p>I found an informal way:</p>
<pre><code class="lang-auto">
import pandas as pd

# ...
# get a prof (torch.profiler.profile)
# ...

prof_results = prof.key_averages()
df = pd.DataFrame(map(vars, profile_results))
# now df.key gives the function names.
# though df has many raw contents e.g. time usage is an int without unit, it still helps to inspect the prof results and is more convenient than a raw string output
</code></pre>
<p>Are there any better approaches?</p>

AidanGoldfarb: <p>The torch profiler is quite a disaster imo, so no, not as far as I have seen. My solution was to modify a fork of pytorch where I added functionality to return the data as a dataframe, similar to what you have done. There could be a solution buried somewhere in the source code, but for me it was faster to simply add the feature.</p>
Backprop without gradients

jloverich: <p>I have an algorithm I’d like to implement that uses backprop but does not actually pass the gradient information backwards instead passes something approximating a function inverse. The point is, I don’t want to use the gradient information but will use other information in the backprop process. Is it possible for me to redefine all the gradients to use the information I want? I should add, I know how to compute what I want from the input / output and function, I just need to pass the resulting values as “gradients” instead of what pytorch uses.</p>

ariG23498: <p>I think your best bet would be to write a custom pytorch hook for this.</p>
<p>For more details you can refer to <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html" rel="noopener nofollow ugc">this</a>.</p>
<p>The main idea would be to register the hook to your parameters and then with the custom logic, backpropagate the values you want to.</p>
<p>I would also like to see if the community has a better way to go around it.</p>
MaskRCNN: CUDA error: an illegal memory access was encountered

BatOfGotham: <p>I am working on the MaskRCNN Pytorch inbuilt model. When I trying to access the loss value that I got output from the model. I am getting following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):

File "/home/droneslab/projects/plastics/shri/DRONES_Lab/trainer-py", line 78, in _one_epoch_train batch_loss_train.append(total loss.item())

RuntimeError: CUDA error: an illegal memory access was encountered

CUDA kernel errors might

be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Compile with *TORCH_USE_CUDA_DSA* to enable device-side assertion
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/2/62ca4e8e13616fb1c5a486035031fd0da8918264.png" data-download-href="https://discuss.pytorch.org/uploads/default/62ca4e8e13616fb1c5a486035031fd0da8918264" title="image (1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/62ca4e8e13616fb1c5a486035031fd0da8918264_2_690x91.png" alt="image (1)" data-base62-sha1="e5WepTr6OulEduVOccXNcanL1FW" width="690" height="91" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/62ca4e8e13616fb1c5a486035031fd0da8918264_2_690x91.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/62ca4e8e13616fb1c5a486035031fd0da8918264_2_1035x136.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/62ca4e8e13616fb1c5a486035031fd0da8918264_2_1380x182.png 2x" data-dominant-color="2E2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image (1)</span><span class="informations">1916×253 21.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It’s not like I am getting this error every epoch. I am getting intermittent. Making me unable to log metrics by both custom logging or tensorboard.</p>

ptrblck: <p>If you are using an older PyTorch release, could you update to the latest stable or nightly release and check if you are still seeing the issue? If so, could you create a CUDA coredump and check which kernel causes the memory violation?</p>
Rife VFI error on ComfyUI on Mac

Gabrie_ZH: <p>I was running ComfyUI on a M2 Max Mac Studio.<br>
When it goes to Rife VFI node, an error prompt up.</p>
<p>“MPS: Unsupported Border padding mode”</p>
<p>Any suggestions ?</p>
<p>Thanks!</p>

toda: <p>Hi there - I’m running into something similar. And by similar I mean the exact same error, but FILM VFI is throwing the error while running Stable Video Diffusion on an M2 Max. Any luck chasing this down on your end?</p>

liocubs: <p><a class="mention" href="/u/gabrie_zh">@Gabrie_ZH</a> <a class="mention" href="/u/toda">@toda</a></p>
<p>According to <a href="https://github.com/Fannovel16/ComfyUI-Frame-Interpolation" rel="noopener nofollow ugc">ComfyUI-Frame-Interpolation</a> authors, non-CUDA support (such as Apple Silicon) is experimental.</p>
<p>Check out this doc: <a href="https://github.com/Fannovel16/ComfyUI-Frame-Interpolation?tab=readme-ov-file#support-for-non-cuda-device-experimental" rel="noopener nofollow ugc"> Support for non-CUDA device (experimental)</a> for configuration changes that might solve it for you.</p>

RamboRogers: <p>I implemented these fixes. Still not working, bummer <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>Error occurred when executing RIFE VFI:</p>
<p>MPS: Unsupported Border padding mode</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e8242393087309afb48cf4677bfab2bc21b89308.png" data-download-href="https://discuss.pytorch.org/uploads/default/e8242393087309afb48cf4677bfab2bc21b89308" title="Screenshot 2024-04-27 at 3.57.17 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8242393087309afb48cf4677bfab2bc21b89308_2_690x142.png" alt="Screenshot 2024-04-27 at 3.57.17 PM" data-base62-sha1="x7CcqM3ofglC9bgNuACxf4hSsis" width="690" height="142" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8242393087309afb48cf4677bfab2bc21b89308_2_690x142.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8242393087309afb48cf4677bfab2bc21b89308_2_1035x213.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e8242393087309afb48cf4677bfab2bc21b89308_2_1380x284.png 2x" data-dominant-color="424242"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-04-27 at 3.57.17 PM</span><span class="informations">1798×372 44.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
DDP with gradient checkpointinting: Confusing Documentation

eliphatfs: <p>This is the documentation on DDP w/ gradient checkpointing:</p>
<blockquote>
<p>DistributedDataParallel currently offers limited support for gradient checkpointing with <a href="https://pytorch.org/docs/stable/utils.html#module-torch.utils.checkpoint" rel="noopener nofollow ugc"><code>torch.utils.checkpoint()</code></a>. If the checkpoint is done with use_reentrant=False (recommended), DDP will work as expected without any limitations. If, however, the checkpoint is done with use_reentrant=True (the default), DDP will work as expected when there are no unused parameters in the model and each layer is checkpointed at most once (make sure you are not passing find_unused_parameters=True to DDP). We currently do not support the case where a layer is checkpointed multiple times, or when there unused parameters in the checkpointed model.</p>
</blockquote>
<p>“We currently do not support the case where a layer is checkpointed multiple times, or when there unused parameters in the checkpointed model.”<br>
“DDP will work as expected without any limitations.”</p>
<p>Does the first sentence scope against ‘use_reentrant=True’ or does it mean if I have to checkpoint the same layer dynamic number of times, use_reentrant=False will work?<br>
And what versions of PyTorch is this supported? Do I need nightly 2.4 version?</p>
Sentence Semitics classification with CNN

milktea_islife: <p>Hi, i have a problem using the torchtext library, its giving me an os error</p>
<pre><code class="lang-auto">OSError                                   Traceback (most recent call last)
Cell In[40], line 1
----&gt; 1 from torchtext.vocab import build_vocab_from_iterator
      2 from collections import Counter
      3 from torchtext.vocab import Vocab

File ~/anaconda3/envs/torch/lib/python3.10/site-packages/torchtext/__init__.py:18
     15     _WARN = False
     17 # the following import has to happen first in order to load the torchtext C++ library
---&gt; 18 from torchtext import _extension  # noqa: F401
     20 _TEXT_BUCKET = "https://download.pytorch.org/models/text/"
     22 _CACHE_DIR = os.path.expanduser(os.path.join(_get_torch_home(), "text"))

File ~/anaconda3/envs/torch/lib/python3.10/site-packages/torchtext/_extension.py:64
     59     # This import is for initializing the methods registered via PyBind11
     60     # This has to happen after the base library is loaded
     61     from torchtext import _torchtext  # noqa
---&gt; 64 _init_extension()

File ~/anaconda3/envs/torch/lib/python3.10/site-packages/torchtext/_extension.py:58, in _init_extension()
     55 if not _mod_utils.is_module_available("torchtext._torchtext"):
     56     raise ImportError("torchtext C++ Extension is not found.")
---&gt; 58 _load_lib("libtorchtext")
     59 # This import is for initializing the methods registered via PyBind11
...
    376     self._handle = handle
</code></pre>

ptrblck: <p>You could try to reinstall <code>torchtext</code> to check if your installation is broken for some reason. However, also note that <code>torchtext</code> is in maintenance mode and not actively developed anymore.</p>
Why Pytorch is much slower than Python dictionary?

ch0t0n: <p>Please if anybody knows the answer to my question that I asked in stackoverflow, it would be really helpful. Here is the link: <a href="https://stackoverflow.com/questions/78394039/why-pytorch-is-much-slower-than-python-dictionary" class="inline-onebox" rel="noopener nofollow ugc">Why Pytorch is much slower than Python dictionary? - Stack Overflow</a></p>
Value, reward and done states

rsarpongstreetor: <h2><a name="h-0-020000-10945-its-1" class="anchor" href="#h-0-020000-10945-its-1"></a>0%|          | 0/20000 [1:09:45&lt;?, ?it/s]</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in &lt;cell line: 7&gt;()<br>
11         # We re-compute it at each epoch as its value depends on the value<br>
12         # network which is updated in the inner loop.<br>
—&gt; 13         advantage_module(tensordict_data)<br>
14         data_view = tensordict_data.reshape(-1)<br>
15         replay_buffer.extend(data_view.cpu())</p>
<p>7 frames<br>
/usr/local/lib/python3.10/dist-packages/torchrl/objectives/value/functional.py in vec_generalized_advantage_estimate(gamma, lmbda, state_value, next_state_value, reward, done, terminated, time_dim)<br>
299         == terminated.shape<br>
300     ):<br>
 → 301         raise RuntimeError(SHAPE_ERR)<br>
302     dtype = state_value.dtype<br>
303     *batch_size, time_steps, lastdim = terminated.shape</p>
<p>RuntimeError: All input tensors (value, reward and done states) must share a unique shape.<br>
time: 2.52 s (started: 2024-04-24 04:20:38 +00:00)</p>
<p>Please how can I edit the tensordict to make the  value, reward and done states to have equal shapes?</p>

rsarpongstreetor: <p>so I found the shapes   for all<br>
(value and done states) in tensordict_data  but<br>
reward in a subdirectory of tensordict ie tensordict_data[‘next’, ‘reward’]<br>
please how can the  advantage_module read all of them?</p>
<p>It seems<br>
it is not reading all</p>

rsarpongstreetor: <p>solved , had to flatten tensors in the neural network</p>
Compile PyTorch code to LLVM IR with Torch 2.0

rogerthat: <p>I have been trying to find ways to compile <strong>native PyTorch code to LLVM IR</strong>. I have turned on environment variables like <em>TORCH_LOGS=“output_code”</em> to dump Triton kernels and LLVM IR out of PyTorch with cuda enabled code.</p>
<p>But I want to do similarly for PyTorch code with target as CPU to be compiled to LLVM IR. Executing such PyTorch programs <strong>dumps C++ kernels</strong> but doesn’t generate LLVM IR like it used to on enabling CUDA. Compiling C++ kernels using clang requires ATen header files and many other internal header files.</p>
<p>Is there any direct way to generate LLVM IR from Torch code against CPU as target? If not should I install <strong>LibTorch</strong> and then compile generated C++ kernels to LLVM IR?</p>
Parallelize a for loop (single CPU, multi-cores)

Laurent1: <p>Hi,</p>
<p>I would like to parallelize a for loop inside my model for training on a single CPU but many cores.<br>
To be more precise, at some point, I have something that looks like</p>
<pre><code class="lang-auto">def forward(self, x):
    # ...
    ys = []
    for latent in latents:
        ys.append( self.submodel(latent) )
    # do something useful with ys
    # ...
</code></pre>
<p>I tried to solve my problem with <code>Pool</code> from <code>torch.multiprocessing</code> but got the error message</p>
<p><code>grad_fn=&lt;AddmmBackward0&gt;)'. Reason: 'RuntimeError('Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).')'</code></p>
<p>I would appreciate any help.<br>
Thanks.</p>
How to concatenate feature vectors from different extractors inplace

Sabin_Grube: <p>Hi everyone,</p>
<p>I’m working on combining feature vectors from multiple extractors. To optimize memory usage and runtime, I attempted in-place operations on a single tensor. My goal was to avoid memory allocation and copying by working on a single tensor and overwriting old values with new ones. However, this resulted in a <code>RuntimeError</code>:</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward
</code></pre>
<p>Am i backwarding a second time through the graph if i overwrite the tensors content? I could make it work by detaching the tensor before overwriting it with the new values.</p>
<pre><code class="lang-auto">def forward(self, state, action):
    ...
  # get rl state feature embeddings
  robot_emb_index = 0
  robot_state_index = self._obs_dim
  # works with detaching the tensor, without detaching i get the RuntimeError specified above
  self._robot_state_emb = self._robot_state_emb.detach()
  for feature_extractor in self._robot_state_extractors:
      self._robot_state_emb[:,robot_emb_index:robot_emb_index+feature_extractor._output_size] = feature_extractor(state[:,robot_state_index:robot_state_index+feature_extractor._input_size])
      robot_state_index += feature_extractor._input_size
      robot_emb_index += feature_extractor._output_size
  
  # works with detaching the tensor, without detaching i get the RuntimeError specified above
  self._critic_input = self._critic_input.detach()
  self._critic_input[:,:self._obs_embedding_size] = encoder_features
  self._critic_input[:,self._obs_embedding_size:self._obs_embedding_size+robot_emb_index] = self._robot_state_emb
  self._critic_input[:,self._obs_embedding_size+robot_emb_index:] = action

  features1 = F.relu(self._h1(self._critic_input))
  features2 = F.relu(self._h2(features1))
  q = self._h3(features2)
  return torch.squeeze(q)
</code></pre>
<p>What is the recommended way to do this? My goal is to avoid using expensive operations  like <code>torch.hstack()</code>. It should be as efficient as possible.</p>
PyTorch Lightning reproducibility

gonzrubio: <p>I’m experiencing an issue with reproducibility in PyTorch Lightning.</p>
<p>Despite setting <code>deterministic=True</code>, <code>pl.seed_everything(42)</code> and <code>workers=True</code>, I’m getting different validation losses for the same epoch when changing the number of epochs in the trainer.</p>
<p>For example, when I set the number of epochs to 1, I get a validation loss of 5.89, but when I set it to 2, I get a validation loss of 5.13 on the first epoch. I’ve verified that the results are reproducible when running the same number of epochs multiple times (get the same sequence of validation losses), but changing the number of epochs yields different results for the same epoch.</p>
<p>What is the cause of this behavior and is there a way to ensure reproducible results across different numbers of epochs?</p>
<p>I am using PyTorch Lightning version 2.2.3</p>
<pre><code class="lang-auto">import time

import hydra
import lightning as pl
import torch
from lightning.pytorch.callbacks import LearningRateMonitor, ModelCheckpoint
from model import EarthQuakeModel
from omegaconf import DictConfig
from torchgeo.datamodules import QuakeSetDataModule


@hydra.main(config_path="configs", config_name="default", version_base=None)
def main(args: DictConfig):
    pl.seed_everything(42, workers=True)
    torch.set_float32_matmul_precision("medium")

    data_module = QuakeSetDataModule(**args.dataset)
    model = EarthQuakeModel(**args.model)

    experiment_id = time.strftime("%Y%m%d-%H%M%S")

    checkpoint_callback = ModelCheckpoint(
        monitor="val_loss",
        dirpath=f"checkpoints/{experiment_id}",
        filename="earthquake-detection-{epoch:02d}-{val_loss:.2f}",
        save_top_k=3,
        mode="min",
    )

    lr_monitor = LearningRateMonitor(logging_interval="step")

    trainer = pl.Trainer(
        **args.trainer,
        # deterministic=True,
        callbacks=[checkpoint_callback, lr_monitor],
        log_every_n_steps=50,
        precision="32-true",
    )

    trainer.fit(model, datamodule=data_module)


if __name__ == "__main__":
    main()

</code></pre>

ptrblck: <p>I don’t know where the epochs are set, but you could check if it changes the number of validation runs etc. If so, you could then check if running a validation loop calls into the PRNG and is thus changing future random numbers.</p>

Derrick_Olley: <p>I usually start looking for a solution from simple mistakes too! This is check if it changes the number of validation runs and whether running the check cycle causes a PRNG! And then there’s more to the situation))</p>
Tracing PyTorch compiled code

HodBadichi: <p>My goal is being able to adjust the backend of distributed functions according to requirements. For example, I wish to be able to change  a failed GPU to be substituted with a constant value tensor during any collective operation.</p>
<p>I experimented with running 2 GPUs with Megatron-LM and NCCL backend. I Intentionally caused a deadlock in one of them (i.e., while(True)) after 50 iterations. I then used <code>py-spy</code> to capture the call stack of the other GPU, expecting to observe collective operation such as a hanging all-reduce , and subsequently find the entry point and adjust the code as I wanted.</p>
<p>Instead, what I found was a complex stack that involves many wrappers, kernel fusion, Just-In-Time (JIT) compilation, and Triton which I’m not that familiar with but no collective operation:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa3328b1e904f38771d2b73486083bf11cdcf319.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fa3328b1e904f38771d2b73486083bf11cdcf319" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa3328b1e904f38771d2b73486083bf11cdcf319_2_567x500.jpeg" alt="image" data-base62-sha1="zHmYsCKFkdY5KXdZzqMU7FSyADD" width="567" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa3328b1e904f38771d2b73486083bf11cdcf319_2_567x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa3328b1e904f38771d2b73486083bf11cdcf319.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa3328b1e904f38771d2b73486083bf11cdcf319.jpeg 2x" data-dominant-color="11191B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">735×648 215 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But when waiting enough time during the deadlock I did encounter a timeout in a collective operation. Which came from a CPP file (<code>ProcessGroupNCCL</code>) which was not mentioned in the call stack(Couldn’t add a picture due to 1 item limitation)</p>
<p>I’m uncertain about how to proceed in my attempt to locate the entry point for the collective operation. Although I attempted to research JIT and Triton, the information is quite overwhelming. I’d really appreciate your help with this.</p>
Viterbi algorithm implementation

Michele_Proietti: <p>Hello, I’m working on a  RNN-CRF architecture for NLP task. I’ve used the CRF implementation provided by <a href="https://pytorch-crf.readthedocs.io/en/stable/" class="inline-onebox" rel="noopener nofollow ugc">pytorch-crf — pytorch-crf 0.7.2 documentation</a>, but I have a question about the implementation of Viterbi Algorthm.<br>
Looking at the online implementations of the algorithm (for example <a href="https://en.wikipedia.org/wiki/Viterbi_algorithm" class="inline-onebox" rel="noopener nofollow ugc">Viterbi algorithm - Wikipedia</a>)  seems that the score (i,j) is computed using multiplication of emission score and transition score<br>
while in the pytorch-crf they use sum to compute the next score:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/0/807e609b9db362a7a7635882c40719e909c0a9e4.png" alt="image" data-base62-sha1="ikHSsVD1MqVsXBFreq8U3O48Iba" width="585" height="30"><br>
Can someone explain to me why? In pratice, do we have to do some assumptions?<br>
Thanks a lot</p>

leejuyuu: <p>The calculation is often done in log space to avoid small floating point numbers underflowing, so the multiplication becomes addition.</p>
Pytorch profile error, output_json.cpp:468 failed to rename trace.json.tmp to trace.json

jrt-20: <p>I have error when use profile on evaluate.py(1 node 8 gpus) <a href="https://github.com/microsoft/Megatron-DeepSpeed/blob/main/tasks/eval_harness/evaluate.py" rel="noopener nofollow ugc">https://github.com/microsoft/Megatron-DeepSpeed/blob/main/tasks/eval_harness/evaluate.py</a> .</p>
<pre data-code-wrap="python"><code class="lang-python">from torch.profiler import profile, record_function, ProfilerActivity
def main():
    start = time.time()
    model = load_ds_checkpoint_and_setup_megatron(extra_args_provider=tasks_args)

    args = get_args()
    if args.deepspeed and args.adaptive_seq_len:
        # adaptive_seq_len hack #1:
        # CL automatically enables reset_activation_shape() which allows us to change input shapes
        # and it also reshapes the attenion scores in attention_mask_func
        args.curriculum_learning_legacy = 1

    task_list = ALL_TASKS if args.task_list == 'all' else args.task_list.split(',')
    task_dict = tasks.get_task_dict(task_list)

    model.module.activation_checkpoint_interval = 0
    model._compute_loss = False
    model.fwd_outputs = []

    tokenizer = get_tokenizer()
    adaptor = EvalHarnessAdaptor(model, tokenizer)
    with profile(activities=[
        ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True) as prof:
        results = evaluator.evaluate(adaptor, task_dict, False, args.num_fewshot, None)
    prof.export_chrome_trace("trace.json")
    if mpu.is_pipeline_last_stage() and mpu.get_tensor_model_parallel_rank() == 0:
        print(json.dumps(results, indent=2))
        with open(args.results_path, 'w') as outfile:
            json.dump(results, outfile, indent = 4)
    end = time.time()
    print("evaluation of {} ends in {:.2f} sec, or {:.2f} min, or {:.2f} hr".format(args.task_list, end-start, (end-start)/60.0, (end-start)/3600.0))

</code></pre>
<p>But i get a error.</p>
<pre data-code-wrap="bash"><code class="lang-bash">output_json.cpp:468 failed to rename trace.json.tmp to trace.json
</code></pre>
<p>How should I use profile tools to fix this.</p>

ptrblck: <p>Are you running the code on all ranks? If so I would assume a rank has already created the file and another one fails in overwriting it.</p>

jrt-20: <p>Thanks. the code is running on all ranks. I wonder how should I use profile on each rank</p>

ptrblck: <p>Use the rank in the file name so that you won’t overwrite the same file.</p>
How to make my faster GPU execute two batches while my other GPU does a batch?

Maitreya_Kocharekar: <p>Hello Everyone, hope you all are doing well. I had the following doubt:</p>
<p>I have a system in which I have two GPUs, say GPU0-&gt; RTX Quadro 8000 and GPU1-&gt; RTX A6000.<br>
Now I believe everyone here is aware that  RTX A6000 is faster than Quadro 8000. With some benchmarking I found out that they if benchmark tests are run on each one of them alone, RTX A6000 is 2x faster than Quadro 8000 for ResNet50 on ImageNet dataset</p>
<p>A6000, 32min, 196 batchsize<br>
Quadro8000, 65min, 196 batchsize</p>
<p>I want to run both of them together but I don’t want RTX A6000 to be under used because Quadro 8000 is slower relatively. Can I make any changes to my batch sampler?</p>
<ul>
<li>
<p>Can I make the batch sampler handle more the data for A6000, which is, that Faster GPU processing 2 batches, while the slower GPU process just 1 batch in the meantime.</p>
</li>
<li>
<p>In this case how should I accumulate the 2 batches of gradients from the Faster GPU, and 1 batch of gradients from slower GPU. How do I accumulate and calculate the updates, essentially after 3 batches of processing per step.</p>
</li>
<li>
<p>When and how do I need to call optimizer.step(), grad_zero() to achieve the above behavior.</p>
</li>
<li>
<p>I can’t just feed double the data size to the faster GPU, because it’s maxing out memory. That is, when I feed batchsize ‘X’ badata to faster gpu, because a sample size</p>
</li>
</ul>
<p>I am new to all these topics and would really appreciate any help from this community. Thank You for your time and patience.</p>
Can an int8 model derived from pytorch's QAT training be converted directly to tensorRT? 

lishanlu136: <p>Can an int8 model derived from pytorch’s QAT training be converted directly to tensorRT? Because the int8 model trained by QAT failed to convert onnx, I want to try to convert directly to tensorRT for GPU inference.</p>

jcaip: <p>Hi <a class="mention" href="/u/lishanlu136">@lishanlu136</a> , have you tried the steps outlined in this tutorial: <a href="https://pytorch.org/TensorRT/_notebooks/vgg-qat.html" class="inline-onebox" rel="noopener nofollow ugc">Deploying Quantization Aware Trained models in INT8 using Torch-TensorRT — Torch-TensorRT v1.3.0 documentation</a></p>

lishanlu136: <p>Thank you, I read the documentation carefully, is this tutorial using the library provided by TensorRT to QAT the pytorch model?</p>

HDCharles: <p>I think you can try using something along the lines of the following:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/TensorRT/blob/main/py/torch_tensorrt/fx/test/quant/test_quant_trt.py#L429-L444">
  <header class="source">

      <a href="https://github.com/pytorch/TensorRT/blob/main/py/torch_tensorrt/fx/test/quant/test_quant_trt.py#L429-L444" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/TensorRT/blob/main/py/torch_tensorrt/fx/test/quant/test_quant_trt.py#L429-L444" target="_blank" rel="noopener nofollow ugc">pytorch/TensorRT/blob/main/py/torch_tensorrt/fx/test/quant/test_quant_trt.py#L429-L444</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="429" style="counter-reset: li-counter 428 ;">
          <li>prepared = prepare(</li>
          <li>    m,</li>
          <li>    {"": self.trt_qconfig},</li>
          <li>    example_inputs,</li>
          <li>    backend_config=self.trt_backend_config_dict,</li>
          <li>)</li>
          <li>self.checkGraphModuleNodes(prepared, expected_node_occurrence=no_prepare)</li>
          <li># calibration</li>
          <li>prepared(*inputs)</li>
          <li>quantized = convert_to_reference_fx(</li>
          <li>    prepared,</li>
          <li>    backend_config=self.trt_backend_config_dict,</li>
          <li>)</li>
          <li>self.checkGraphModuleNodes(quantized, expected_node_occurrence=no_convert)</li>
          <li># lower to trt</li>
          <li>trt_mod = lower_to_trt(quantized, inputs, shape_ranges)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>its still in early prototype phase i believe but theoretically that should work if your model is traceable</p>
Partially sharded training

cholesky: <p>I have a large (100s of GB) time series dataset with high frequency samples. I would like to fit a recurrent model to predict features of this dataset in the following way:</p>
<ol>
<li>Split the data up into intervals (say, 5 minutes in length)</li>
<li>On each interval, compute in parallel F(incoming_state, parameters) = (score, outgoing_state, d_score_d_param, d_score_d_incoming_state, d_outgoing_state_d_param, d_outgoing_state_d_incoming_state)</li>
<li>Using the gradients from 2), optimize the parameters over the entire dataset at once</li>
<li>Repeat 2) and 3) until convergence.</li>
</ol>
<p>The motivation is that step 3) can fit on a single GPU, because I am kind of “compressing” the data in step 2); step 2) is very big but can be parallelized, and hopefully I can get away with computing it less frequently by passing the gradient to step 3).</p>
<p>How can I implement such a framework using PyTorch?</p>
How to write VMap function correctly

fishbotics: <p>Hi all,</p>
<p>I have this function</p>
<pre><code class="lang-auto">def rotation_matrices(angles, axis):
    """Compute rotation matrices from angle/axis representations.
    Parameters
    ----------
    angles : (n,) float
        The angles.
    axis : (3,) float
        The axis.
    Returns
    -------
    rots : (n,4,4)
        The rotation matrices
    """
    axis = axis / torch.norm(axis)
    sina = torch.sin(angles)
    cosa = torch.cos(angles)
    M = torch.eye(4, device=angles.device).repeat((len(angles), 1, 1))
    M[:, 0, 0] = cosa
    M[:, 1, 1] = cosa
    M[:, 2, 2] = cosa
    M[:, :3, :3] += (
        torch.ger(axis, axis).repeat((len(angles), 1, 1))
        * (1.0 - cosa)[:, np.newaxis, np.newaxis]
    )
    M[:, :3, :3] += (
        torch.tensor(
            [
                [0.0, -axis[2], axis[1]],
                [axis[2], 0.0, -axis[0]],
                [-axis[1], axis[0], 0.0],
            ],
            device=angles.device,
        ).repeat((len(angles), 1, 1))
        * sina[:, np.newaxis, np.newaxis]
    )
    return M
</code></pre>
<p>I’d like to be able to call this within vmap like so</p>
<pre><code class="lang-auto">angles = torch.ones(10)
axis = torch.tensor([0., 1., 0.])
rotation_matrices(angles, axis)  # Works fine

batched = torch.vmap(rotation_matrices)
batched(angles[None, ...], axis[None, ...])  # Fails
</code></pre>
<p>I get the following errors:</p>
<pre><code class="lang-auto">18 in _rotation_matrices(angles, axis)
     16 cosa = torch.cos(angles)
     17 M = torch.eye(4, device=angles.device).repeat((len(angles), 1, 1))
---&gt; 18 M[:, 0, 0] = cosa
     19 M[:, 1, 1] = cosa
     20 M[:, 2, 2] = cosa

RuntimeError: vmap: inplace arithmetic(self, *extra_args) is not possible because there exists a Tensor `other` in extra_args that has more elements than `self`. This happened due to `other` being vmapped over but `self` not being vmapped over in a vmap. Please try to use out-of-place operators instead of inplace arithmetic. If said operator is being called inside the PyTorch framework, please file a bug report instead.
</code></pre>
<p>How should I change this function to work in vmap appropriately?</p>
<p>Thanks!</p>
Torchviz graph of lstm network is very complicated

Danibahnani: <p>I am trying to visualize my lstm network using torchviz, but the graph I get is very complicated due to the nature of lstm cells. Is there a way to make torchviz simplify it?</p>
<p>I have also tried using tensorboard, but I couldn’t get it to work, I think it’s because I am packing and unpacking my sequences.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8b1255d64e4e0f9b6a95eba6fdac0e3291881e8.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b8b1255d64e4e0f9b6a95eba6fdac0e3291881e8" title="model_render"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8b1255d64e4e0f9b6a95eba6fdac0e3291881e8_2_135x499.jpeg" alt="model_render" data-base62-sha1="qlRo3yUcR6GSF2ftzcpJWcqYM7S" width="135" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8b1255d64e4e0f9b6a95eba6fdac0e3291881e8_2_135x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8b1255d64e4e0f9b6a95eba6fdac0e3291881e8_2_202x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8b1255d64e4e0f9b6a95eba6fdac0e3291881e8_2_270x998.jpeg 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">model_render</span><span class="informations">1920×7096 533 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
First Contribution help developing - Calling into non-scriptable torch.* components

duffymitch12: <p>I wanted to take on one of the issues from the repo from <a href="https://github.com/pytorch/pytorch/issues/58122" rel="noopener nofollow ugc">this issue post</a>, I am unsure what I should reference. I was looking at the third issue listed and want to know the best way to get up to speed with this area of the code. I have been looking over torch/distributions to get a better understanding. Where would this issue be fixed as well?</p>

ptrblck: <p><code>TorchScript</code> is in maintenance mode so I would not recommend spending time in trying to improve error messages and would pick another starter issue. Look for the <code>good first issue</code> label.</p>
Uninstalling pytorch is stuck in spyder IDE

NIT_India: <p>Hello all,<br>
I have installed anaconda and by launching Spyder through navigator, I install torch in it.<br>
I guess things are installed properly until, I find that the command</p>
<blockquote>
<p>torch.cuda.is_available()</p>
</blockquote>
<p>is throwing “False”.</p>
<p>Then I realized may be I did not choose the cuda version while installing, so now trying to uninstall it.<br>
But then after inputting command</p>
<blockquote>
<p>pip uninstall torch</p>
</blockquote>
<p>It gets so long and basically stuck in processing.<br>
Any solution?</p>

ptrblck: <p>Run the commands in your terminal if they are stuck in Spyder (unsure how and where you are executing these commands there).</p>
BatchMemoryManager with gans

He_Jinnan: <p>Hi! I am trying to implement BatchMemoryManager when training GAN.<br>
It seems that BatchMemoryManager need the optimizer, but I got two optimizers here. One <code>optimizerG</code> for generator and the other <code>optimizerD</code> for discriminator. Moreover, <code>optimizerD</code> is made private by Opacus while <code>optimizerG</code> is not. So how can I use BatchMemoryManager here?</p>