akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Understanding versions compatibility

Marwan_Haioun: <p>Hello! I am trying to use pytorch for the first time in a while and am facing some problems regarding versioning. I have been trying to follow installation instructions from a specific github repository relying on pytorch ( ```<br>
conda install pytorch==1.11.0 cudatoolkit=11.3 -c pytorch -y<br>
conda install pyg::pytorch-scatter=2.0.9 -y</p>
<pre><code class="lang-auto">
I'm currently working using Ubuntu on WSL and conda, and running nvidia-smi returns the following: 
![image|689x298](upload://AtcAqU1ZEXNgv3j4gYUNf3SXAUv.png)

So how should I approach this? Everything I've done so far resulted in my GPU not being recognized by torch (I've tried to install cuda toolkit 11.3 and 11.8, should I try more recent versions?).

Thanks!</code></pre>

Marwan_Haioun: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fc14914336a4a71b4aa77e7ed8f2b5110220862b.png" data-download-href="https://discuss.pytorch.org/uploads/default/fc14914336a4a71b4aa77e7ed8f2b5110220862b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fc14914336a4a71b4aa77e7ed8f2b5110220862b.png" alt="image" data-base62-sha1="zY0o3KA3PZS8eHmXe5kYSIy5KXN" width="690" height="290" data-dominant-color="1A1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1317×555 17.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
NameError: name '_C' is not defined while importing torch

rahulraj1990: <p>I’m getting below error while importing torch version 1.9.0</p>
<pre><code class="lang-auto">---&gt; 22 import torch
     23 from tqdm import tqdm
     24 from attrdict import AttrDict

/python/PROJ/torch/1.9.0-py37/exec/lib/torch/__init__.py in &lt;module&gt;()
    227 
    228 
--&gt; 229 __all__ += [name for name in dir(_C)
    230             if name[0] != '_' and
    231             not name.endswith('Base')]

NameError: name '_C' is not defined

</code></pre>
<p>Note: imported Cython 0.29.21 version as I see <a href="https://github.com/pytorch/pytorch/issues/1633" rel="noopener nofollow ugc">here</a>. But no luck.<br>
I verified that my current working directory is project home, and not torch directory.</p>
<p>Can anyone tell me whats going  wrong here?</p>

ptrblck: <p>Are you able to <code>cd</code> into an empty directory and <code>import torch</code> there? If so, I guess some scripts in your current <code>wdir</code> might be using “common” names and could thus interact with Python or PyTorch internals.</p>

rahulraj1990: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>my current working directory is:<br>
<code>/user/rahul/MyApp/examples</code></p>
<p>I created a notebook in this path and just doing <code>import torch</code> in the notebook. Then, I get the above mentioned error. The directory: <code>/user/rahul/MyApp/examples</code> has no files except the notebook that I created.  This error happens <strong>only when</strong> I change torch version from <strong>1.4.0</strong> to <strong>1.9.0</strong>.</p>

ptrblck: <p>I cannot reproduce it with a similar folder structure and the <code>1.9.0</code> binaries:</p>
<pre><code class="lang-auto"># ~/MyApp/examples$ cat tmp.py 
import torch

print(torch.__version__)

# ~/MyApp/examples$ python tmp.py 
1.9.0
</code></pre>

TTAyanlade: <p>This happened to me a while back. Restarting runtime solved it for me</p>

msxfmplus: <p>This error still happening on google colab.</p>

msxfmplus: <pre><code class="lang-auto">!pip install -q Cython torch
</code></pre>
<p>Everyone, Install the Cython first!!!</p>

ISHIKA_GARG: <p>just “disconnect and delete the runtime” and re-run. This worked for me.</p>
DDP: errno: 97 - Address family not supported by protocol

rockstar: <p>I am running distributed training; it works fine but I am getting these annoying warnings:</p>
<pre><code class="lang-auto">[W socket.cpp:426] [c10d] The server socket cannot be initialized on [::]:1234 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [host]:1234 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [::ffff:xx.xx.xx.xx]:1234 (errno: 97 - Address family not supported by protocol).
              ⋮
</code></pre>
<p><strong>IPv6</strong> is disabled on all the systems. I guess these warnings are related to <strong>IPv6</strong>. Is there any way to disable this warning?</p>

pvtien96: <p>I’ve encountered the same errors while using the high-performance computing system of TGCC/Irene where the internet is disabled.</p>
<pre data-code-wrap="bash"><code class="lang-bash">[W socket.cpp:426] [c10d] The server socket cannot be initialized on [::]:29500 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost.ccc.cea.fr]:29500 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost.ccc.cea.fr]:29500 (errno: 97 - Address family not supported by protocol).
| distributed init (rank 3): env://
| distributed init (rank 2): env://
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost.ccc.cea.fr]:29500 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost.ccc.cea.fr]:29500 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost.ccc.cea.fr]:29500 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost.ccc.cea.fr]:29500 (errno: 97 - Address family not supported by protocol).
| distributed init (rank 1): env://
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost.ccc.cea.fr]:29500 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost.ccc.cea.fr]:29500 (errno: 97 - Address family not supported by protocol).
| distributed init (rank 0): env://
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost.ccc.cea.fr]:29500 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost.ccc.cea.fr]:29500 (errno: 97 - Address family not supported by protocol).
</code></pre>

Jiancheng_Zhang: <p>So you mean, if the internet service is available, can it work?</p>
Different training speed when using 1 node with 8 GPU vs using 8 node with 1 GPU on the same 8-V100 machine

taod: <p>Hi.<br>
When training models on a 8-gpu machine with docker, I have tried the two following ways:</p>
<ol>
<li>start 1 docker container with 8 gpu and run ddp with nnodes=1 and nproc-per-node = 8</li>
<li>start 8 docker container with 1 gpu per container and run ddp with nnodes=8 and nproc-per-node = 1</li>
</ol>
<p>The first way is much faster than the second way. ( 17 steps/s vs 13 steps/s )<br>
In theory both two ways can use nccl to communicate, so what cause this training speed gap?</p>
Problem when import torchvision: Failed to load image Python extension

heycode-cpu: <p>I met below warring when import torchvision, I checked the table in <a href="https://github.com/pytorch/vision#installation" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/vision: Datasets, Transforms and Models specific to Computer Vision</a>. My torchvision is 0.18.1 and torch is 2.3.1.<br>
Could you please help to check how to fix this?</p>
<p>(yolo5) PS C:\Users\win&gt; python<br>
Python 3.9.19 (main, May  6 2024, 20:12:36) [MSC v.1916 64 bit (AMD64)] on win32<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
import torchvision<br>
D:\miniconda_data\envs\yolo5\lib\site-packages\torchvision\io\image.py:13: UserWarning: Failed to load image Python extension: '[WinError 127] The specified program could not be found.'If you don’t plan on using image functionality from <code>torchvision.io</code>, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have <code>libjpeg</code> or <code>libpng</code> installed before building <code>torchvision</code> from source?<br>
warn(<br>
torch.<strong>version</strong><br>
‘2.3.1’<br>
torchvision.<strong>version</strong><br>
‘0.18.1’</p>
</blockquote>
</blockquote>
</blockquote>
Official Bayesian Implementation!

anubhabsam: <p>Hello all! Is there an official Pytorch implementation of Bayesian NN?</p>

Aniruth_Sundararaja1: <p>You could try our Tyxe - <a href="https://paperswithcode.com/paper/tyxe-pyro-based-bayesian-neural-nets-for" class="inline-onebox" rel="noopener nofollow ugc">TyXe: Pyro-based Bayesian neural nets for Pytorch | Papers With Code</a></p>
Confused about torch.bmm

Sun-Xiaohui1: <p>hello, PyTorch team! I was confused about <code>torch.bmm</code> . In my code below:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/7/478d846878bc9a72e38395952c0a3193379cd91e.png" alt="image" data-base62-sha1="acZ6b2jclNSBLARpHYk2MQu67wi" width="560" height="208"></p>
<p><code>query_states</code>  and  <code>key_states</code>  are both <strong>4-D tensors</strong></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/badf6e9b3bd3c1ec9cd937d602e8b62f92a22492.png" alt="image" data-base62-sha1="qF9vEgKYmCgCwdjzq3SyjsUbrIm" width="612" height="129"></p>
<p>As the docs say: torch.bmm(<em>input</em> , <em>mat2</em> , *** , <em>out=None</em> ) → [Tensor]: <code>input</code> and <code>mat2</code> must be 3-D tensors each containing the same number of matrices.</p>
<p>So, why <code>torch.bmm</code> doesn’t occur en error here?  My pytorch version is ‘2.1.0+cpu’. Thanks!</p>

soulitzer: <p>Do you have a short reproducible snippet?</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.rand(1, 40, 3, 128)
&gt;&gt;&gt; b = torch.rand(1, 40, 3, 128)
&gt;&gt;&gt; torch.bmm(a, b.transpose(2, 3))
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: batch1 must be a 3D tensor
</code></pre>
<p>Also, could you try again on a later PyTorch version?</p>

Sun-Xiaohui1: <p>No, I can’t reproduce it in python terminal, which will report an error: batch1 must be a 3D tensor. Ok, I will use <code>torch.matmul</code> instead and I’m just feel confused about this.</p>
Can profiler detect memory leak across epochs with self_cpu_memory_usage?

rinkujadhav2013: <p>My code has some host memory leak which might be there due to some tensors attached in some custom autograd function while they shouldn’t be attached to the graph. Can I detect such leak by using PyTorch’s profiler? Considering the example listed in the tutorial here: <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Profiler — PyTorch Tutorials 1.12.1+cu102 documentation</a></p>
<pre><code class="lang-auto">with profile(activities=[ProfilerActivity.CPU],
        profile_memory=True, record_shapes=True) as prof:
    print(prof.key_averages())
</code></pre>
<p>would the code above print higher memory usage values across epochs if there’s a leak?</p>

ptrblck: <p>I don’t think checking the profiler output would help in this case, as it would show the memory usage of each operation, which is unrelated to storing tensors attached to a computation graph.<br>
This code snippet should illustrate it:</p>
<pre><code class="lang-python">model = models.resnet18()
x = torch.randn(1, 3, 224, 224)

outputs = []

for i in range(10):
    with profile(activities=[ProfilerActivity.CPU], profile_memory=True, record_shapes=True) as prof:
        out = model(x)
        outputs.append(out)
    
    print(prof.key_averages().table(sort_by="self_cpu_memory_usage", row_limit=10))
</code></pre>
<p>Also note that you are not seeing a memory leak (which would indicate that memory is lost and cannot be freed anymore), but an expected increase of memory usage since you are explicitly storing tensors including their attached computation graph.</p>

rinkujadhav2013: <p>Thanks!<br>
The follow up question is then, can I somehow get total tensor count or something similar so that I can check if it’s growing with each epoch or not?</p>

pujaltes: <p>Were you able to solve this?</p>

rinkujadhav2013: <p>Nope. I had to work around it somehow. I think my code might still have memory leak (or not) but it’s so low that it doesn’t really affect anything throughout training so I’m okay.</p>
Conv1d just like numpy convolve

Salman_Abbasi: <p>Hi</p>
<p>I am trying to mimic numpy.convovle jusing torch.nn.functional.conv1d. The results are not the same given my dimensions. I appreciate if someone can correct it. The script is below</p>
<p>import torch.nn.functional as F<br>
import numpy as np</p>
<p>inputs = torch.randn(2,240,60)<br>
filters = torch.randn(240,240,60)<br>
filters_flip=filters.flip(2)<br>
F.conv1d(inputs[:,:,:], filters[:,:,:]).shape<br>
conv_tens=F.conv1d(inputs, filters_flip, padding=30)[1,1,:]</p>
<p>inputs_np=inputs.numpy()<br>
filters_np=filters.numpy()<br>
conv_np=np.convolve(filters_np[1,1,:],inputs_np[1,1,:], mode=‘same’)</p>
<p>Thanks</p>

Salman_Abbasi: <p>Since I am interested in 1d convolution. Using torchaudio.functional helped. Below is small sample</p>
<p>import torchaudio.functional as F2</p>
<p>inputs2 = torch.randn(60)<br>
filters2 = torch.randn(60)<br>
filters2_flip=filters2.flip(0)<br>
conv_tens=F2.convolve(inputs2, filters2,‘same’)</p>
<p>conv_np=np.convolve(filters2,inputs2, mode=‘same’)</p>
LibTorch Inference on Android

rita19991020: <p>Hello everyone,</p>
<p>I have successfully written the code to perform inference using a model with LibTorch. I am now trying to cross-compile this code to run inference directly on an Android device using adb shell, but I am facing several challenges. Is it possible to use LibTorch as-is on an Android phone? If so, could someone guide me through the process or provide any resources to help achieve this?</p>
<p>Thank you in advance for your assistance.</p>
Create advanced neural network

jamiec: <p>I want to know what would be good more advanced model types I can use. My use case is for example house price prediction or football match score prediction. Right now I’m using <code>nn.Linear()</code> for these use cases, but I would like to use something more advanced (yes im already using non-linear activation functions such as relu) but the only thing I can find is <code>Conv1d</code> (convolutional nn), which is better fit for images.<br>
Does anyone know an example model architecture (for my use case) featuring something else than <code>nn.Linear()</code>? Because the only other one I can find are models like AlexNet which use a convolutional neural network, but I doubt that is most efficient and useful for my use case.<br>
Anything would help!</p>
GymWrapper observation spec

putskan: <p>Hey!<br>
I’m wrapping a simple MultiArmedBandit Gymnasium env I created.<br>
The original gymnasium env always returns the state 0, and its observation space is set to <code>spaces.Discrete(1)</code>.</p>
<p>However, when wrapping it with <code>GymWrapper</code>, the observations always give 1 instead of zero.</p>
<p>It seems the issue has something to do with TorchRL’s usage of <code>OneHotDiscreteTensorSpec</code>, which changes the state to 1 when calling <code>spec.encode</code> in TorchRL’s internals.</p>
<p>How can it be fixed?<br>
Much appreciated.</p>

vmoens: <p>Try <code>categorical_action_encoding=True</code> that should work</p>

putskan: <aside class="quote no-group" data-username="vmoens" data-post="2" data-topic="205506">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vmoens/48/26121_2.png" class="avatar"> vmoens:</div>
<blockquote>
<p>categorical_action_encoding</p>
</blockquote>
</aside>
<p>Completely missed that flag, thanks!</p>
How to get stream operators in custom backend

wbigat: <p>When I use a custom backend, I find that the fx graph that the custom compiler gets does not have the stream related operations.</p>
<p>Then I read the trace process and found that  the graph dropped those stream operations after aot_module_simplified.</p>
<p>When I use aot_module_simplified, how can I achieve that the fx graph obtained by custom compiler contains stream-related operations?</p>
<p>Here is my test script:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import logging
torch._logging.set_logs(dynamo=logging.DEBUG,aot=logging.DEBUG,output_code=True,graph_code=True,inductor=logging.DEBUG)

class Layer(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        stream2 = torch.cuda.Stream()
        with torch.cuda.stream(stream2):
            z = x + 1
        y = x - 1

        cur = torch.cuda.current_stream()
        cur.wait_stream(stream2)
        y = y + z
        return y

mm = Layer()
x=torch.randn([4]).cuda()

from torch._functorch.aot_autograd import aot_module_simplified
def toy_backend(gm, sample_inputs):
    return gm
def aot_toy_backend(gm, sample_inputs):
    return aot_module_simplified(gm, sample_inputs, fw_compiler=toy_backend)

mmc = torch.compile(mm, backend=aot_toy_backend)  # "inductor")
yc= mmc(x)
</code></pre>
How to set functions as black box or wrap when using torch dynamo to capture backward graph?

wmhst7: <p>I am now using</p>
<pre><code class="lang-auto">m, _ = aot_export_module(fwd, [inp], trace_joint=True, output_loss_index=0, decompositions=None)
fwd, bwd = default_partition(m, [inp], num_fwd_outputs=1)
</code></pre>
<p>to capture forward and backward graph based on aot grad and dynamo.</p>
<p>In the past, we could set a function <code>@torch.fx.wrap</code> to avoid it decomposed to lower ops. ref: <a href="https://pytorch.org/docs/stable/fx.html#torch.fx.wrap" class="inline-onebox" rel="noopener nofollow ugc">torch.fx — PyTorch 2.3 documentation</a></p>
<p>So I wonder if there are any methods to do the same thing?</p>
<p>One possible way to do it is to set a custom op as:<br>
<a href="https://pytorch.org/tutorials/advanced/python_custom_ops.html#python-custom-ops-tutorial" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/python_custom_ops.html#python-custom-ops-tutorial</a></p>
<p>But in the page, it said requires Pytorch 2.4, which is unfortunately not available to me.</p>

soulitzer: <p>You should still be able to create custom ops via the lower level torch.library.Library APIs.</p>
<pre><code class="lang-auto">lib = torch.library.Library("foo", "FRAGMENT")

lib.define("bar(Tensor x) -&gt; Tensor")

def bar_impl(a):
    return a.clone()

lib.impl("bar", bar_impl, "CPU")
lib.impl("bar", bar_impl, "CUDA")
lib.impl("bar", bar_impl, "Meta")
</code></pre>
<p>The newer custom ops APIs are intended to make the process easier and less error prone, so it is still recommended that you upgrade if possible.</p>

wmhst7: <p>I understand, thank you.</p>

wmhst7: <p>Update: <a href="https://dev-discuss.pytorch.org/t/how-to-set-wrap-function-using-torchdynamo-graph-capture/2185/4?u=wmhst7" class="inline-onebox" rel="noopener nofollow ugc">How to set wrap function using TorchDynamo graph capture? - #4 by wmhst7 - compiler - PyTorch Developer Mailing List</a></p>
Use pickle.load inside the getitem function of a dataloader is not automatically mapped to gpu

sivannavis: <p>Hi, I build a dataloader and wrapped it with a lightning datamodule that loads precomputed features saved in pkl file. So in the getitem function, I load it with pickl.load(filename), but I got the error:</p>
<pre><code class="lang-auto">*** RuntimeError: Caught RuntimeError in DataLoader worker process 2.
Original Traceback (most recent call last):
  File "/home/szding/.conda/envs/v2sa2/lib/python3.12/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/szding/.conda/envs/v2sa2/lib/python3.12/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
    return self.collate_fn(data)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/szding/.conda/envs/v2sa2/lib/python3.12/site-packages/torch/utils/data/_utils/collate.py", line 316, in default_collate
    return collate(batch, collate_fn_map=default_collate_fn_map)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/szding/.conda/envs/v2sa2/lib/python3.12/site-packages/torch/utils/data/_utils/collate.py", line 154, in collate
    clone.update({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/szding/.conda/envs/v2sa2/lib/python3.12/site-packages/torch/utils/data/_utils/collate.py", line 141, in collate
    return collate_fn_map[elem_type](batch, collate_fn_map=collate_fn_map)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/szding/.conda/envs/v2sa2/lib/python3.12/site-packages/torch/utils/data/_utils/collate.py", line 213, in collate_tensor_fn
    return torch.stack(batch, 0, out=out)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument tensors in method wrapper_CUDA_cat_out_out)
</code></pre>
<p>And I checked after loading the file with pickle, the location of the data is on CPU. I thought it will be automatically mapped to cuda, now what should I do?</p>
<p>In my code I have something like this for my getitem function:</p>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        batch = {}
        info = self.metadata[idx]
        with open(a, 'rb') as handle:
               batch['zcode'] = pickle.load(handle)
        return batch
</code></pre>
<p>If I just simply add <code>.cuda()</code> to the batch data, then another error comes: <code>RuntimeError: cannot pin 'torch.cuda.FloatTensor' only dense CPU tensors can be pinned</code></p>
Input data range after quantization

fnhdx: <p>I have a trained float32 model with input data range from 0-1. I want to do a quantization aware training. My question is for the quantized model, the input should be still 0-1 or need to scale to 0-255? should I retrain the float32 model with input 0-255 before the qat? Thanks.</p>

jerryzh168: <p>you don’t need to change the input, it should still be 0-1</p>
Unit Testing with artificial GPU limits

maxme: <p>Hello,<br>
I deploy Pytorch models across multiple devices. My training infrastructure (as per usual) has access to more GPU memory than some of the devices, where the models are deployed. I would like to set up unit tests with an artificial GPU limit, so I can figure out, if some of my models have too high memory requirement.<br>
In essence, I would like to use torch.randn with a certain batch size, the respective resolution and channels; then build my model and apply inference. When - within that unit test session - the system runs out of memory, I would like my unit test to fail.<br>
Is there a way to achieve this?</p>

ptrblck: <p><a href="https://pytorch.org/docs/stable/generated/torch.cuda.set_per_process_memory_fraction.html#torch.cuda.set_per_process_memory_fraction"><code>torch.cuda.set_per_process_memory_fraction(fraction, device=None)</code></a> might be useful for your workloads.</p>
Which index the dataloader is chosing

JuyiLin: <p>I am using torch.utils.data.DataLoader. like <code>train_features, train_labels = next(iter(train_dataloader))</code><br>
I want to determine which index the trainer is chosen?<br>
<code>for batch_idx, data in enumerate(trainloader, 0):</code> can index from 0, but if we shuffle, can we know the indices for this batch?</p>

ptrblck: <p>You could return the passed <code>index</code> from the <code>Dataset.__getitem__</code> method and use it in your <code>DataLoader</code> loop.</p>

zeh_gobbes: <p>Hello,</p>
<p>In :<br>
<a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/basics/data_tutorial.html</a></p>
<p>when using:</p>
<p>train_features, train_labels = next(iter(train_dataloader))</p>
<p>I believe that train_features takes the values ​​from train_dataloader.data and train_labels takes the values ​​from train_dataloader.targets. However, the class has several other tensors. How do I know which tensor is actually being caught, and why is this tensor being caught instead of the others?</p>

ptrblck: <p>The <code>Dataset.__getitem__</code> method is responsible for the loading and processing of the samples and will return these. The <code>DataLoader</code> will just batch these. If you want to check which tensors are processed in which way, check the <code>Dataset</code> implementation.</p>
TypeError: expected np.ndarray (got numpy.ndarray)

chinmay: <pre><code class="lang-auto">
        particle_type = self.particle_type[window["type"]: window["type"] + size].copy()
        particle_type = torch.from_numpy(np.asarray(particle_type))
       
</code></pre>
<p>In the following code snippet I am having following error</p>
<pre><code class="lang-auto">


Cell In[9], line 43, in OneStepDataset.get(self, idx)
     41 size = window["size"]
     42 particle_type = self.particle_type[window["type"]: window["type"] + size].copy()
---&gt; 43 particle_type = torch.from_numpy(np.asarray(particle_type))
     44 position_seq = self.position[window["pos"]: window["pos"] + self.window_length * size * self.dim].copy()
     45 position_seq.resize(self.window_length, size, self.dim)

TypeError: expected np.ndarray (got numpy.ndarray)
</code></pre>
<p>I tried type casting to array and without array, nothing works.<br>
Can anyone suggest any idea?</p>
<p>Link to the full notebook</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1U_glFm7bmRhfazylXYAJQjgsOANIUhBh?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/be01a7b5f02fab7b1eb2b8a2ae88eb58/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1U_glFm7bmRhfazylXYAJQjgsOANIUhBh?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1U_glFm7bmRhfazylXYAJQjgsOANIUhBh?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colab</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>Is creating an <code>np.array</code> explicitly working via <code>particle_type = torch.from_numpy(np.array(particle_type))</code>?</p>

nickums: <p>did you " import numpy as np" ?<br>
Nick</p>
How does pytorch decide the size of dataset?

lzm2256: <p>It seems that the iteration behavior of dataset depends on the idx variable in <strong>getitem</strong> function. Is it expected or a bug?</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import Dataset
class MyDataset(Dataset):
    def __init__(self):
        self.A = torch.randn(3,4)

    def __len__(self):
        return 3

    def __getitem__(self, idx):
        d = self.A[idx]  # comment out this line leads to infinite loop
        return 0


face_dataset = MyDataset()

for i, sample in enumerate(face_dataset):
    print(i)
</code></pre>
<p>As expected, the output is</p>
<pre><code class="lang-auto">0
1
2

</code></pre>
<p>However, when commenting out the line " d = self.A[idx] ", the loop never stops unless control + c is pressed.</p>
<pre><code class="lang-auto">...
1457
1458
....
</code></pre>
<p>This is not expected because the size of the dataset is already defined in the <strong>len</strong>() function. Is this a bug? How to avoid this?</p>

ptrblck: <p>Iterating a raw <code>Dataset</code> depends on a <code>StopIteration</code> signal coming from e.g. an out of bounds index. Wrap the <code>Dataset</code> into <code>DataLoader</code> or make sure a <code>StopIteration</code> will be raised.</p>

lzm2256: <p>Thanks! Wrapping dataset into Dataloader solves this problem.</p>
Attn feature agrregration

NolimitDougie: <p>I have 5 images one being a source image with 4 similar images. I’m using faiss to retrieve the source image. I’m trying to have the source image attend off the other 4 images to get a more enhanced representation. I’m thinking of using the source image as the query matrix and concatenate the other images across the channel and use that as the key, value matrix. Once I get the output how can I leverage the attn weights to better enhance my source image representation.  I need the final representation to be in B * C * H * W.  or [B, Sequence length, embed dimension] I’m using this as a hidden state for a diffusion model. Any suggestions???</p>
Really confuse on the same input but different outout

YXIANG_DAI: <p>Really confuse !!! I have two linear layer, one is I initial by myself and another is get from the model.  But they are just really different. And you can see their difference from the image below.<br>
Given the Batchsize 1 and Batchsize 2 and they give me differen results, even the dtype are the same!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/7609283c30ccf7015f9a09469a39e812e304c6b4.png" data-download-href="https://discuss.pytorch.org/uploads/default/7609283c30ccf7015f9a09469a39e812e304c6b4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/7609283c30ccf7015f9a09469a39e812e304c6b4_2_345x500.png" alt="image" data-base62-sha1="gQbXyLc3XwG4WI7BW2XIgPDIFus" width="345" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/7609283c30ccf7015f9a09469a39e812e304c6b4_2_345x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/7609283c30ccf7015f9a09469a39e812e304c6b4_2_517x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/7609283c30ccf7015f9a09469a39e812e304c6b4_2_690x1000.png 2x" data-dominant-color="232323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1150×1662 144 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Check the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> to learn more about determinism and how to make sure deterministic algorithms are selected.</p>
Custom policy with distributions for PPO

utsavdutta98: <p>The torchRL policy implementations always invoke a ProbabilisticActor when using PPO style losses. I see that at some point it calls a method called <code>.get_dist()</code>, which I assume is returning the final distribution layer. As I am not using a standard distribution from the available ones, do I need to add a ‘fake’ get_dist, log_prob method etc onto my actor network to make it compatible?</p>
<p>Is this not recommended?</p>
<p>Further, would it make sense to instead subclass torch.distributions and make my own custom distribution that adheres to their interface?</p>

vmoens: <p>Hey<br>
For PPO we need to have access to the current log probability of the action given the param configuration as well as the same value with the param configuration used at collection time.<br>
Because of this we rely on the <code>dist.log_prob</code> method which is the PyTorch distribution API (note that this is a mild requirement since PPO relies on policies that are parametric distributions so it will be more often the case than not that your policy has this <code>log_prob</code> method).<br>
<code>get_dist</code> on the other hand is TorchRL/TensorDict module API that interfaces a module with a distribution that has a <code>log_prob</code>. What this function does is rather simple and should be easy to reproduce within a custom module</p>
<pre><code class="lang-auto">class MyTdModule(TensorDictModuleBase):
    ...
    def get_dist(self, data):
        data_out = self.module(data)
        return self.dist_cls(data_out["param0"], data_out["param1"])
</code></pre>
<p>Hope that helps</p>
Feature Aggregation

NolimitDougie: <p>Hello,</p>
<p>I have 5 images I’m trying to get the best representation of all 5 images down into one single representation. I need the representation to be in B * C * H * W. I’m using this as a hidden state for a diffusion model. Any suggestions???</p>

Aniruth_Sundararaja1: <p>Try Averaging</p>
<pre><code class="lang-auto">images = [image1, image2, image3, image4, image5]
averaged_image = torch.mean(torch.stack(images), dim=0)
</code></pre>
<p>This should give u B * C * H * W</p>
How to Track GPU Memory Allocation and Release for Each Tensor During DNN Training in PyTorch?

zbtrs: <p>Hello everyone,</p>
<p>I’m currently working on GPU memory management and I would like to know how to track the memory allocated and released by each tensor throughout the DNN training process in PyTorch. From what I understand, <code>torch.profiler</code> can only provide memory usage at the kernel level, not at the tensor level. So, how can I achieve this?</p>

soulitzer: <p>Are you looking for something like this <a href="https://pytorch.org/blog/understanding-gpu-memory-1/" class="inline-onebox" rel="noopener nofollow ugc">Understanding GPU Memory 1: Visualizing All Allocations over Time | PyTorch</a></p>

zbtrs: <p>Thank you very much!!!</p>
How does DDP work with torch.cuda.make_graphed_callables

yjguo: <p><a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.3.0+cu121 documentation</a> mentions that:<br>
DDP registers an autograd hook for each parameter given by <code>model.parameters()</code> and the hook will fire when the corresponding gradient is computed in the backward pass.</p>
<p>torch.cuda.make_graphed_callables replaces the model forward and backward with cuda graph.</p>
<p>so, my question is, during backward, just a cuda graph is replay-ed, not going through the pytorch python/c++ code, how the autograd hook is triggered?  Or this is a special solution that one allreduce is specially added for this case at the end of the whole model backward?</p>
<p>thanks</p>

ptrblck: <p>The CUDA Graph will capture the corresponding NCCL operations performed in the backward pass and will replay these afterwards.</p>

yjguo: <p>thanks for the answer.</p>
<p>But I think NCCL is not involved by torch.cuda.make_graphed_callables which just captures the forward pass and backward pass without communication collective operations, and so no NCCL is captured into the cuda graph. (see the code below)</p>
<p>According to <a href="https://pytorch.org/docs/stable/notes/cuda.html#nccl-2-9-6:" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch 2.3 documentation</a><br>
Call <a href="https://pytorch.org/docs/stable/generated/torch.cuda.make_graphed_callables.html#torch.cuda.make_graphed_callables" rel="noopener nofollow ugc"><code>make_graphed_callables()</code></a> on graphable network sections <em>before</em> wrapping the network with DDP.</p>
<p>the code looks like below:</p>
<pre><code>s = torch.cuda.Stream(device=args.device)
with torch.cuda.stream(s):
    model = torch.cuda.make_graphed_callables(model, (real_inputs[0],))
    ddp_model = DDP(model)
</code></pre>
<p>btw, I know that current nccl supports cuda graph, but my purpose is to understand how this method (DDP + torch.cuda.make_graphed_callables) is implemented.</p>

ptrblck: <p>The link does not mention the section so I guess you are referring to <a href="https://pytorch.org/docs/stable/notes/cuda.html#nccl-2-9-6">this doc</a> explaining how NCCL&lt;2.9.6, which does not support full-graph capture, is handled?<br>
If you are using such an old NCCL release for some reasons, then you would be correct and you won’t be able to capture communication calls.</p>
<p>Capture the full model using newer NCCL versions instead of parts of it to also capture the communication calls in case that’s what you are asking for.</p>

yjguo: <p>thanks, our links are the same.</p>
<p>and my questions is, since the nccl is not captured, and the backward stage is replaced with cuda graph replay, how could the bwd hook be triggered to call the nccl operations?</p>
Implementing Skip Connections

jawi289p: <p>I am facing issue in implementing skip connections in my encoder-decoder model. I am getting size mismatch issue. Can anyone give a simple example for encoder-decoder architecture with skip connections?</p>

jawi289p: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> please help on priority basis<br>
thank you</p>

Brock_Brown: <p>unet isn’t defined here, please provide a reproducible example</p>

jawi289p: <p>sorry I have updated the code</p>

jawi289p: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> still waiting for your response</p>

ptrblck: <p>Please don’t tag specific users as it might discourage others to give you a valid response.<br>
It seems <a class="mention" href="/u/brock_brown">@Brock_Brown</a> was already trying to check your code, which was incomplete.</p>
How to create a "Both" option in multiple choice model training to solve IndexError: Target out of bounds error?

AlphaTT30: <p>I was trying to train a  pre-trained transformer model “bert-base-uncased”</p>
<p><code>AutoModelForMultipleChoice.from_pretrained("bert-base-uncased")</code></p>
<p>So this is a model for multiple-choice questions where the prompt is combined with each option like prompt+optionA, prompt+optionB.</p>
<p>But if there is an option called “Both”,  which has only a label say (2)  but is not paired up like optionA and optionB, then the error will raise like IndexError: Target 2 is out of bounds.<br>
Because I have 2 options but 3 labels. I can say that because If I eliminate the data containing labels 2  the training process goes fine.</p>
<p>The real case for me is:<br>
There is a prompt and 2 responses from 2 AI models response_a and response_b for the same single prompt. (i.e. 2 options )<br>
Now the model has to determine which response is better for this there are 3 labels winner_a, winner_b and winner_tie (i.e. 3 labels )</p>
<p>So how to deal with that? Should I create a third option? If so then how? If there is another solution please let me know.<br>
Here is my tokenized function :</p>
<pre data-code-wrap="python"><code class="lang-python">def tokenize_function(examples):
    # Repeat each prompt twice to go with the two possibilities of responses.
    first_sentences = [[context] * 2 for context in examples["prompt"]]
    

    # Pair the responses.
    second_sentences = [[a, b] for a, b in zip(examples['response_a'], examples['response_b'])]
    # print(second_sentences)

    # Flatten everything.
    first_sentences = sum(first_sentences, [])
    second_sentences = sum(second_sentences, [])

    # Tokenize.
    tokenized_examples = tokenizer(first_sentences, second_sentences, truncation=True)


   

    # Un-flatten.
    return {k: [v[i:i+2] for i in range(0, len(v), 2)] for k, v in tokenized_examples.items()}
</code></pre>
<p>here is the data structure :</p>
<pre data-code-wrap="python"><code class="lang-python">DatasetDict({
    train: Dataset({
        features: ['id', 'model_a', 'model_b', 'prompt', 'response_a', 'response_b', 'winner_model_a', 'winner_model_b', 'winner_tie', 'class_name', 'labels', 'encode_fail', 'options', '__index_level_0__'],
        num_rows: 45981
    })
    validation: Dataset({
        features: ['id', 'model_a', 'model_b', 'prompt', 'response_a', 'response_b', 'winner_model_a', 'winner_model_b', 'winner_tie', 'class_name', 'labels', 'encode_fail', 'options', '__index_level_0__'],
        num_rows: 11496
    })
})
 
</code></pre>
Derivation of L_out for Conv1d

whoab: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html</a></p>
<p>Can someone give me the derivation for L_out in the above documentation?</p>
<p>Specifically, how the output length of temporal data gets downsampled / affected by the kernel, dilation, etc.</p>

soulitzer: <p>There is already a formula for L_out in the documentation, but if you are asking about the intuition behind why it is the way it is:</p>
<p>For “kernel”, consider an example where kernel=3, stride=1, dilation=1, padding=0, and L_in=10.<br>
The L_out is L_in - kernel + 1 = 8 since you can tile the size 3 kernel in 8 ways.</p>
<p>To think about “dilation” and how it spreads the kernel, you can check that all else being equal, kernel=2, dilation=2 produces the same L_out as kernel=3, dilation=1.</p>

soulitzer: <p>For the rest, you can probably get at them easily by playing around with some examples.</p>

ptrblck: <p>In addition, tools such as <a class="mention" href="/u/ezyang">@ezyang</a>’s conv visualizer might also be helpful to understand the conv arguments better: <a href="https://ezyang.github.io/convolution-visualizer/" class="inline-onebox">Convolution Visualizer</a></p>
Should Transformer's causal attention mask be upper-triangular or lower-triangular?

cloudydory: <p>Hi, I am a bit confused about the way to implement a causal attention mask in Transformer, specifically, should “-inf” be placed in the upper or lower triangular part of the attention matrix.</p>
<p>In PyTorch’s recommanded way to obtain the attention mask, the “-inf” are placed on upper triangular part:</p>
<pre><code class="lang-auto">import torch.nn as nn
model = nn.Transformer()
mask = model.generate_square_subsequent_mask(sz=4)
print(mask)
</code></pre>
<p>Outputs:</p>
<pre><code class="lang-auto">tensor([[0., -inf, -inf, -inf],
        [0., 0., -inf, -inf],
        [0., 0., 0., -inf],
        [0., 0., 0., 0.]])
</code></pre>
<p>However, in PyTorch’s implementation of <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html" rel="noopener nofollow ugc">torch.nn.MultiHeadAttention</a>, the documentation on <code>attn_mask</code> is:</p>
<blockquote>
<p><strong>attn_mask</strong> – If specified, a 2D or 3D mask preventing attention to certain positions. Must be of shape (𝐿,𝑆) or (𝑁⋅num_heads,𝐿,𝑆), where 𝑁 is the batch size, 𝐿 is the target sequence length, and 𝑆 is the source sequence length.</p>
</blockquote>
<p>Here I am assuming that “source sequence length” is the length of input tokens, and “target sequence length” is the length of output tokens. Since we do not want the model to attent to future outputs, then I think the “-inf” should be placed on the lower triangular part of the attention matrix? I am getting very confused on this.</p>

Brezen: <p>The causal mask is with “-inf” in the upper triangular section. <strong>Attn_mask</strong> is needed if you want to implement a generic mask. You can use <strong>is_causal</strong> = True to add the causal mask.<br>
Anyway, I think the process is better described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html" rel="noopener nofollow ugc">scaled dot product</a> documentation.</p>
Opacus ValueError: Per sample gradient is not initialized. Not updated in backward pass?

BlueBoy: <p>Hello,</p>
<p>I have recently started working on Machine Learning and Medical Imaging and am currently attempting to integrate Opacus into the “Robbing the Fed” attack from the Breaching framework for federated learning. Unfortunately, I encountered an error that I haven’t been able to resolve.<br>
When I initiate training, the following error occurs:</p>
<p>ValueError: Per sample gradient is not initialized. Not updated in backward pass?</p>
<p>The error appears in the file <code>training_opacus.ipynb</code> in my repository:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/Bleuboy/Robbing-the-fed/blob/main/training_opacus.ipynb">
  <header class="source">

      <a href="https://github.com/Bleuboy/Robbing-the-fed/blob/main/training_opacus.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/Bleuboy/Robbing-the-fed/blob/main/training_opacus.ipynb" target="_blank" rel="noopener nofollow ugc">Bleuboy/Robbing-the-fed/blob/main/training_opacus.ipynb</a></h4>


      <pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a8ee9650",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "from collections import namedtuple\n",
    "import os\n",
    "import matplotlib.pyplot as plt\n",
    "from attacks.analytic_attack import ImprintAttacker\n",
    "from modifications.imprint import ImprintBlock\n",
    "from utils.breaching_utils import *\n",
    "\n",
    "import medmnist\n",
    "from medmnist import INFO, Evaluator\n",
</code></pre>



  This file has been truncated. <a href="https://github.com/Bleuboy/Robbing-the-fed/blob/main/training_opacus.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>As I am still a novice programmer, I would appreciate it if you could help me understand the cause of this error and how to fix it. Also, how can I remove the opacus wrapper after training to just keep the model for evaluation?</p>
<p>Kind regards.</p>