akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to let torch.compile only target GPU

woctordho: <p>For context: I’d like to help non-tech savvy users enjoy JIT acceleration in applications like ComfyUI and Unsloth. Recently I published <a href="https://pypi.org/project/triton-windows" rel="noopener nofollow ugc">triton-windows</a> wheels to PyPI, which bundles TinyCC and ptxas, so users can ‘just pip install’ it (without manually setting up compiler toolchains, which is a pain on Windows). It also helps downstream developers properly package their applications.</p>
<p>This works as long as <code>torch.compile</code> targets GPU. However, users still need a C++ compiler if they ‘accidentally’ run <code>torch.compile</code> targeting CPU. If I understand correctly, this can happen because of the graph break, although I don’t yet have a minimal reproducer for this.</p>
<p>Is there a mechanism to let <code>torch.compile</code> only target GPU, and skip compiling the CPU part?</p>
<p>(By the way… I don’t think there is a C++ compiler toolchain that is small enough to get bundled in the wheels, while complete enough for <code>torch.compile</code> targeting CPU. If you know one, please tell me)</p>
Onnx Hanging Nodes Shown In Neutron Visualizer after Python Export?

catch22: <p>I’m a bit confused by some “hanging nodes” I am seeing in the Nuetron onnx visualizer. They seem to have no input, only output. Can someone explain to me how these integrate with the rest of the tree? Here is a screen shot from the Python Onnx Export example here:</p>
<p><a href="https://pytorch.org/tutorials//beginner/onnx/export_simple_model_to_onnx_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials//beginner/onnx/export_simple_model_to_onnx_tutorial.html</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7dbd06de79aef0c8be1a067f15290e954cbce8b0.png" data-download-href="https://discuss.pytorch.org/uploads/default/7dbd06de79aef0c8be1a067f15290e954cbce8b0" title="node"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7dbd06de79aef0c8be1a067f15290e954cbce8b0_2_173x500.png" alt="node" data-base62-sha1="hWkFESRq6ws62DQdUmmBF1rcDh6" width="173" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7dbd06de79aef0c8be1a067f15290e954cbce8b0_2_173x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7dbd06de79aef0c8be1a067f15290e954cbce8b0_2_259x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7dbd06de79aef0c8be1a067f15290e954cbce8b0_2_346x1000.png 2x" data-dominant-color="F5F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">node</span><span class="informations">396×1144 33.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The page also has another screen shot which doesn’t seem related to the example code shown, but displays the same artifact(s). In this case they are transpose operations with no inputs, only outputs.</p>
<p>Thanks!</p>
Aten arange behavior when dtype is int64 and step size is greater than range

satheeshhab: <p>On CPU, i get the following behavior for arange.</p>
<blockquote>
<blockquote>
<blockquote>
<p>a = torch.arange(0, 0.5, 1, dtype=torch.int64)<br>
a<br>
tensor(<span class="chcklst-box fa fa-square-o fa-fw"></span>, dtype=torch.int64)<br>
a = torch.arange(0, 0.5, 1, dtype=torch.int32)<br>
a<br>
tensor([0], dtype=torch.int32)</p>
</blockquote>
</blockquote>
</blockquote>
<p>Why is it that size of ‘a’ is 0 when dtype is int64 where as it is 1 for int32? Logically speaking the first element is anyways 0 and size should have been 1 even for int64 type, isn’t it?</p>

KFrank: <p>Hi Satheesh!</p>
<aside class="quote no-group quote-modified" data-username="satheeshhab" data-post="1" data-topic="217735">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/satheeshhab/48/75859_2.png" class="avatar"> satheeshhab:</div>
<blockquote>
<blockquote>
<blockquote>
<blockquote>
<p>a = torch.arange(0, 0.5, 1, dtype=torch.int64)<br>
a<br>
tensor(, dtype=torch.int64)<br>
a = torch.arange(0, 0.5, 1, dtype=torch.int32)<br>
a<br>
tensor([0], dtype=torch.int32)</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</aside>
<p>I see this same behavior (on both the cpu and gpu) on pytorch 2.6.0.</p>
<aside class="quote no-group">
<blockquote>
<p>Logically speaking the first element is anyways 0 and size should have been 1 even for int64 type, isn’t it?</p>
</blockquote>
</aside>
<p>I agree with you that the <code>int32</code> behavior is correct and the <code>int64</code> behavior is wrong.<br>
This looks like a bug to me.  Perhaps you could file this as a github issue and see what<br>
the experts think.</p>
<p>Best.</p>
<p>K. Frank</p>

satheeshhab: <p>hi Frank<br>
Thank you for your review. As you suggested, I have raised a github issue - <a href="https://github.com/pytorch/pytorch/issues/149097" class="inline-onebox" rel="noopener nofollow ugc">Aten arange behavior when dtype is int64 and step size is greater than range · Issue #149097 · pytorch/pytorch · GitHub</a></p>
Not able to access GPT-Fast blogpost anymore

njkumarr: <p>I am trying to go to the GPT-Fast repo and read the blogpost, however it seems that it has been removed and replaced with something else now. Do you guys know what happened?</p>
<p><a href="https://github.com/pytorch-labs/gpt-fast" rel="noopener nofollow ugc">repo with blogpost in README.md</a><br>
<a href="https://pytorch.org/blog/accelerating-generative-ai-2/" rel="noopener nofollow ugc">attached blog post</a></p>
How to request sufficient memory in advance

chenghua_wang: <p>I would like to know if torch provides the functionality to manually request memory in advance. This is because when multiple processes share a GPU, it could lead to memory allocation conflicts and cause crashes, potentially evicting a process that has already computed a significant amount of data. I’m curious if a torch program can manually pre-allocate a sufficient block of memory for use.</p>

ptrblck: <p>Yes, you can allocate and delete a tensor at the beginning of your script manually and PyTorch will move the allocated device memory to its cache via <code>torch.empty(size, device="cuda")</code>.<br>
A better approach might be to limit the available memory per process via <a href="https://pytorch.org/docs/stable/generated/torch.cuda.set_per_process_memory_fraction.html"><code>torch.cuda.set_per_process_memory_fraction</code></a>.</p>
Pytorch OCR models for deploying to ESP32?

mavavilj: <p>I wanted to use Pytorch to target ESP32, because I don’t like TensorFlow.</p>
<p>The ESP32 docs (<a href="https://github.com/espressif/esp-dl" class="inline-onebox" rel="noopener nofollow ugc">GitHub - espressif/esp-dl: Espressif deep-learning library for AIoT applications</a>) say that Pytorch models converted to ONNX can be ran on the ESP32 as long as the esp-dl supports all the required operators.</p>
<p>However, what OCR models should I look at? I want the best accuracy and ideally a model that doesn’t use operators outside of those supported by esp-dl.</p>
Libtorch Save Network topology

Iwein: <p>Hey all,</p>
<p>I have my network topology saved in buffers, as follows:</p>
<blockquote>
<p>register_buffer(“in_channels”, torch::tensor(in_channels, torch::dtype(torch::kInt64)));<br>
register_buffer(“classes”, torch::tensor(classes, torch::dtype(torch::kInt64)));<br>
register_buffer(“depth”, torch::tensor(depth, torch::dtype(torch::kInt64)));<br>
register_buffer(“channel_base”, torch::tensor(channel_base, torch::dtype(torch::kInt64)));<br>
register_buffer(“bottleneck_depth”, torch::tensor(bottleneck_depth, torch::dtype(torch::kInt64)));</p>
</blockquote>
<p>After model initialisation and training, i save the model as follows:</p>
<blockquote>
<p>torch::save(trained_model, model_file.generic_string());</p>
</blockquote>
<p>To my supprise, the registered buffer parameters are not saved at all when i load a new model with</p>
<blockquote>
<p>torch::load(inference_model, model_file.generic_string());.</p>
</blockquote>
<p>How can i solve this problem? My network topology changes all the time in function of the type of images presented. it is crucial that the network topology is saved in the *.pt file itself.</p>
<p>Thanks in advance!<br>
I.</p>
Optimize a module with parameters per datapoint

roaiyadgar: <p>Hi,</p>
<p>I have a <code>nn.Module</code> that contains a single tensor of size <code>n x k</code> where <code>n</code> is my dataset length. Each row represents some parameters that I need to tune for each specific data point.<br>
I want to optimize over the parameters of this module (along with another module which does not scale with the data).<br>
I use Adam optimizer, and on each epoch there is a single batch in which the gradient of the i-th row is not zero.<br>
However on subsequent batches, the optimizer keeps modifying the i-th row despite it having a gradient of 0 (since Adam as well as SGD use some ‘memory’ of previous gradients).</p>
<p>What would be the best approach to prevent this from happening? What I would like to see is for the optimizer to only interpret a ‘step’ of the i-th row if the i-th data point is present in the batch. Is there an elegant solution for a problem like that?</p>
<p>Thanks!</p>

JuanFMontesinos: <p>Adam use statistics precomputed. So it has certain inertia (let’s call it like that).<br>
To avoid it, you can create n nn.Paramters instead of putting all together in a single matrix.<br>
Alternatively, you can use plain SGD with no momentum.</p>
Cura 10.2 for Ubuntu 24 and torch 1.6.0

Syp: <p>I am trying to run a script but it needs a old version for pytorch (1.6.0).</p>
<p>Cuda 10.2 seems to be the maximal compatible version. But i cannot get it working with Ubuntu 24, since 18 is the layers supported version. I cannot downgrade Ubuntu because it is not compatible with hardware.</p>
<p>Anyone else cannot install cuda 10.2 on newer machines?</p>

ptrblck: <aside class="quote no-group" data-username="Syp" data-post="1" data-topic="217746">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/syp/48/74584_2.png" class="avatar"> Syp:</div>
<blockquote>
<p>Cuda 10.2 seems to be the maximal compatible version.</p>
</blockquote>
</aside>
<p>Your locally installed CUDA toolkit won’t be used as PyTorch ships with its own CUDA runtime dependencies. You would need to install an NVIDIA driver and the PyTorch binaries afterwards.</p>
Torchvision sm_120 failed installed

Desire6666: <p>My gpu is <strong>5070ti</strong>. I have already install <strong>torch-2.7.0.dev20250305+cu128-cp39-cp39-win_amd64.whl</strong> from <a href="https://download.pytorch.org/whl/nightly/" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/</a><br>
but now I want to install torchvision, I choose to install <strong>torchvision-0.22.0.dev20250310+cu128-cp39-cp39-win_amd64.whl but failed</strong>,with error as follows：<br>
Processing c:\users\desire\downloads\torchvision-0.22.0.dev20250310+cu128-cp39-cp39-win_amd64.whl<br>
Requirement already satisfied: numpy in c:\users\desire\anaconda3\envs\pytorch_0312\lib\site-packages (from torchvision==0.22.0.dev20250310+cu128) (2.0.2)<br>
INFO: pip is looking at multiple versions of torchvision to determine which version is compatible with other requirements. This could take a while.<br>
ERROR: Could not find a version that satisfies the requirement torch==2.7.0.dev20250309+cu128 (from torchvision) (from versions: 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.5.0, 2.5.1, 2.6.0)<br>
<strong>ERROR: No matching distribution found for torch==2.7.0.dev20250309+cu128</strong></p>

ptrblck: <p>Install matching nightly versions for <code>torch</code> and <code>torchvision</code>.</p>
Avoid caching for parametrization

Franckma: <p>I would like to apply an automatic process to a non-learnable parameter (for instance L2 normalization).<br>
pytorch parametrization was a good candidate:</p>
<pre data-code-wrap="python"><code class="lang-python">parametrize.register_parametrization(
            self, "u", L2Normalize())
</code></pre>
<p>By doing so I can update my parameter u easily in a loop within a single learning step:</p>
<pre data-code-wrap="python"><code class="lang-python">for _ in range(n_iter):
  self.u = myfunction(self.u)
</code></pre>
<p>This works well when the parametrization caching is deactivated, but by activating the with parametrize.cached(), any call to self.u gives back the first value (before updates within the loop).</p>
<p>This could be a bug since the assignment self.u = …, that calls the right_inverse method and updates the self.parametrizations.u.original, it should discard the cache.</p>
<p>But the usage done in my code of parametrization is not the classic one. Does anyone have  an alternative solution that can both support multiple assignment and automatic normalization?</p>
<p>Otherwise, modification of PyTorch parametrizations could be:</p>
<ul>
<li>to discard cache of the tensor when an assignment is done (as described before)</li>
<li>to provide a flag in the register_parametrization to prevent caching for specific parameters</li>
</ul>
<p>Thanks for your help</p>
Standard way of dealing with standardization for tabular data

pentavol: <p>Hey,</p>
<p>It is well known that a model works better if the data is standardized. Here is how I deal with this in the context of regression based on tabular data:</p>
<pre><code class="lang-auto">df_train, df_test, df_val = get_data(config)
if config['scaler'] is not None:
     scaler = config['scaler'] 
else:
    scaler = StandardScaler().fit(df[config['columns']], df[target]) # I only scale some numerical columns
df_train, df_test, df_val = scaler.transform(df_train), scaler.transform(df_test), scaler.transform(df_val)
best_model, test_predictions = train(df_train, df_test, df_val, config)
target_std, target_mean = get_scaler_coeffs(scaler)
test_predictions['y_hat'] = test_predictions['y_hat'] * target_std + target_mean
store_results(test_predictions, best_model, scaler, config)
</code></pre>
<p>In short, I use StandardScaler from sklearn to scale values (target included) and pandas for operating with csv files. After training, I store the best model along with its scaler and its results. I think that this approach is not elegant at all, but I can’t figure out how should I change the code. What is the “standard” way of dealing with scalling and reverse scalling in pytorch? (preferably for regression purposes)</p>

Maria_Djeblahi: <p>Are there any updates on this issue in pytorch?</p>
Feature request: linear learning rate warmup in torch.optim.lr_scheduler

jakelevi1996: <p>As far as I can tell, there is no built-in class in <code>torch.optim.lr_scheduler</code> that can automatically do linear learning rate warm-up. I know this would be quite easy to implement, EG by subclassing <code>torch.optim.lr_scheduler.LRScheduler</code>. But considering linear learning rate warm up is used in some of the most all-time highly cited ML papers (EG “Attention is all you need”), and there are several existing forum posts + SE questions on this topic, it would make sense if this was included in <code>torch.optim.lr_scheduler</code> by default.</p>
NVLS support in pytorch

rajagond: <p>Does PyTorch support NVLS? If not, how does it manage to call NCCL’s NVLS algorithm using `torch.distributed.all_reduce?</p>

HyperHyper: <p>I think it is not supported as I see from:</p>
<ul>
<li><a href="https://github.com/microsoft/mscclpp/issues/477" class="inline-onebox" rel="noopener nofollow ugc">Nvls channel setup inside the container · Issue #477 · microsoft/mscclpp · GitHub</a></li>
<li><a href="https://github.com/pytorch/pytorch/issues/136567" class="inline-onebox" rel="noopener nofollow ugc">[RFC] Offload collectives to NVSwitch when possible · Issue #136567 · pytorch/pytorch · GitHub</a></li>
</ul>

ptrblck: <p>It’s possible to use NVLS via the <code>torch.cuda.MemPool</code> API which landed in <a href="https://github.com/pytorch/pytorch/pull/133603">this PR</a>. We are also working on enabling it in e.g. DDP related to <a href="https://github.com/pytorch/pytorch/pull/145675">this PR</a>.</p>
RAM Leak when using IterableDataset

iliasslasri: <p>Hello,</p>
<p>When using IterableDataset: For instance</p>
<pre data-code-wrap="Python"><code class="lang-Python">    def __iter__(self):
         yield input, target
</code></pre>
<p>and using the : torch.utils.data.DataLoader to wrap this dataset:</p>
<pre data-code-wrap="Python"><code class="lang-Python">dataloader = DataLoader(train_dataset)
</code></pre>
<p>The CPU memory usage increases until the process is killed.</p>
<p>Does anyone ever encountered this or have an idea of how to solve this ? I tried to use the garbage collector of python but no difference?</p>
<p>The <strong>error</strong> raised from torch/utils/data/dataloader.py line 1256</p>
<pre data-code-wrap="Python"><code class="lang-Python">  raise RuntimeError(
RuntimeError: DataLoader worker (pid(s) 144359) exited unexpectedly
</code></pre>
<p>Important remark is that when the training happens in cpu, I do not have this memory leak issue, I only get this when training on gpu, and it’s not the gpu memory that gets full, it is ram.<br>
Thank you all in advance!</p>
<p>I checked <a href="https://discuss.pytorch.org/t/memory-increasing-with-iterabledataset/203433">Memory increasing with IterableDataset</a> and <a href="https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461">DataLoader memory usage keeps increasing</a> but idea given in both.</p>
Unable to fix RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

Dave1: <p>While it seems like this issue is brought up quite frequently, I was unable to find an answer in online forums. Currently I am trying to load a model from a .pth file and fine tune it using a dataset from huggingface. However, it seems that upon loading in the model, this error keeps occuring.</p>
<p>I’ve tried putting <code>torch.multiprocessing.set_start_method('spawn')</code> in the code but then another error appears that the context has already been set. Upon some debugging it seems that the context is set when the <code>load_dataset</code> function from the <code>datasets</code> library is called. I am currently using the accelerate library for training.</p>

ptrblck: <aside class="quote no-group" data-username="Dave1" data-post="1" data-topic="208718">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dave1/48/61905_2.png" class="avatar"> Dave1:</div>
<blockquote>
<p>Upon some debugging it seems that the context is set when the <code>load_dataset</code> function from the <code>datasets</code> library is called. I am currently using the accelerate library for training.</p>
</blockquote>
</aside>
<p>Check if the dataset is moving the samples to the GPU internally and if so, keep them on the CPU. Alternatively, reduce the <code>num_workers=0</code> and use the main thread to move the data to the GPU if you really want to move it to the GPU inside the dataset.</p>

iliasslasri: <p>Hey, What do you mean by :</p>
<blockquote>
<p>Check if the dataset is moving the samples to the GPU internally</p>
</blockquote>
<p>Did you mean if for instance if the the <code>IterableDataset</code> does <code>yield batch.to(device)</code> and we should change it to <code>yield batch</code>. Or something else ?</p>

ptrblck: <p>Yes, check if somewhere inside the data loading pipeline the data is moved to the device as this could fail using the wrong start method and multiprocessing.</p>

iliasslasri: <p>It worked, thanks, I wonder why <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
PyTorch + CUDA on python:3.11-bullseye

pbonito: <p>HI all,<br>
I want to run dataflow with PyTorch, Cuda and Debian bullseye.<br>
Dataflow can run with docker image on Container-Optimized OS.</p>
<p>This docker container works fine and torch is able to use GPU:</p>
<pre><code class="lang-auto">FROM pytorch/pytorch:2.6.0-cuda12.6-cudnn9-runtime

WORKDIR /pipeline

COPY requirements.txt .
COPY *.py ./


RUN pip install --no-cache-dir --upgrade pip \
    &amp;&amp; pip install --no-cache-dir -r requirements.txt \
    &amp;&amp; pip check


COPY --from=apache/beam_python3.11_sdk:2.61.0 /opt/apache/beam /opt/apache/beam
ENTRYPOINT [ "/opt/apache/beam/boot" ]
</code></pre>
<p>When I try to install torch+cuda following the instruction here <a href="https://pytorch.org/get-started/locally/#mac-verification" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a>, torch is installed but orch.cuda.is_available() return false. In both cases NVIDIA driver are installed separately.</p>
<pre><code class="lang-auto">FROM python:3.11-bullseye

WORKDIR /pipeline

COPY requirements.txt .
COPY *.py ./


RUN pip install --no-cache-dir --upgrade pip \
    &amp;&amp; pip install --no-cache-dir -r requirements.txt \
    &amp;&amp; pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126 \
    &amp;&amp; pip check

 

COPY --from=apache/beam_python3.11_sdk:2.61.0 /opt/apache/beam /opt/apache/beam
ENTRYPOINT [ "/opt/apache/beam/boot" ]
</code></pre>
<p>GPU looks ok:<br>
ldd --version</p>
<p>ldd (Debian GLIBC 2.31-13+deb11u11) 2.31</p>
Is there any efficient way to build a dataset from zip archive file?

Eta_C: <p>Is there any efficient way to build a dataset from zip archive file?<br>
I have some huge data with <code>.zip</code> format. (size &gt; 1TB)<br>
Extracting these make me painful.</p>
<p>Perhaps a zip dataset/dataloader can help me to load the corresponding data as needed during training…</p>
In-API way of catching `torch._C._LinAlgError`

ego-thales: <p>Hello,</p>
<p>Is there an in-API way of catching <code>torch._C._LinAlgError</code> – risen upon <a href="https://pytorch.org/docs/stable/generated/torch.linalg.inv_ex.html" rel="noopener nofollow ugc"><code>torch.linalg.inv_ex</code></a> failure on singular matrices when <code>check_errors=True</code>?</p>
<p>Thanks!</p>
<p>Edit: I should precise that I also get a <code>mypy</code> error when trying to use it as is.</p>
<pre><code class="lang-auto">error: Module has no attribute "_LinAlgError"  [attr-defined]
</code></pre>

ego-thales: <p>I found <code>torch.linalg.LinAlgError</code>!</p>
<pre data-code-wrap="pycon"><code class="lang-pycon">&gt;&gt;&gt; torch.linalg.LinAlgError is torch._C._LinAlgError
True
</code></pre>
Help with a 3d unet

Zotnam: <p>hi,<br>
i am trying to apply a 3d unet to a set of 3d images that I have for segmentation but it’s the first time i do it and i am facing some issues.</p>
<p>My images have only 1 channel but are very big  (ct scans) so    i reduced by a factor 3 so that  they have now  a shape that is around 800 in height and 700 in width. the issue is that the depth,i.e. the number of slices, vary a lot from one volume to the other,from  a minimum of 16 to a maximum of 109 and padding everything to the maximum shape is very inefficient. what should i do?</p>
<p>I have a v100 gpu with 32 gb of ram, can i process whole images or should i use patches?  thank you for any help</p>
CPU and GPU inference differences

ck92: <p>I’m running the following script that creates a pytorch model, runs inference on CPU and GPU. I get differences as high as 2e-4. Is this expected with float32?</p>
<pre><code class="lang-auto">import torch
from monai.networks.nets import VarAutoEncoder

model.eval()
input = torch.ones((1,6,64,64))
recon, mu, logvar, _ = model(input)

device = torch.device('cuda:0')
model = model.to(device)
input = input.to(device)
recon_gpu, mu_gpu, logvar_gpu, _ = model(input)


diff = mu_gpu.detach().cpu().numpy() - mu.detach().numpy()
print(f'diff: {diff}')
</code></pre>
<pre><code class="lang-auto">diff: [[-7.45356083e-05 -3.43009830e-04  1.75867230e-04 -1.16109848e-04
  -4.17232513e-05  2.66432762e-05 -1.32430345e-04 -1.16214156e-04
  -5.60581684e-05  2.44081020e-04 -9.73343849e-05  2.08556652e-04
   1.40473247e-04 -1.85698271e-04 -2.05576420e-04  4.55528498e-05
   1.46627426e-04  2.26497650e-06 -1.25408173e-04 -1.09910965e-04]]
</code></pre>

KFrank: <p>Hi Chait!</p>
<aside class="quote no-group" data-username="ck92" data-post="1" data-topic="217633">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/91b2a8/48.png" class="avatar"> ck92:</div>
<blockquote>
<pre><code class="lang-auto">diff: [[-7.45356083e-05 -3.43009830e-04  1.75867230e-04 -1.16109848e-04
  -4.17232513e-05  2.66432762e-05 -1.32430345e-04 -1.16214156e-04
  -5.60581684e-05  2.44081020e-04 -9.73343849e-05  2.08556652e-04
   1.40473247e-04 -1.85698271e-04 -2.05576420e-04  4.55528498e-05
   1.46627426e-04  2.26497650e-06 -1.25408173e-04 -1.09910965e-04]]
</code></pre>
</blockquote>
</aside>
<p>Differences at this level don’t surprise me.  (As an aside, without knowing the typical<br>
scale of <code>mu</code>, we don’t actually know how large <code>diff</code> is <em>relative</em> to the size of <code>mu</code>, so<br>
I am assuming that <code>mu</code> is roughly of order one.)</p>
<p>Your cpu and gpu computations could easily be performing various arithmetic operations<br>
in mathematically equivalent, but numerically different orders, thereby leading to differing<br>
round-off error.  A single float32 operation would typically have a round-off error on the<br>
order of 1e-7.  But such round-off errors accumulate with further computation.  It wouldn’t<br>
surprise me for the output of a reasonably deep model (lots of layers) to end up with cpu<br>
and gpu accumulated round-off errors that differ on the order of 1e-4.</p>
<p>You might try converting your model to <code>double()</code> (<code>torch.float64</code>) and see if <code>diff</code><br>
shrinks by several (about nine) orders of magnitude.</p>
<p>Best.</p>
<p>K. Frank</p>
Pooling with a custom kernel?

shroomite: <p>In my current application I would like to perform max pooling with a diamond kernel. This is possible by performing 3x1 and 1x3 max pools in the dimensions of both axes, and then selecting the element-wise max of both, but this is wasteful as it needs to process the image twice.</p>
<p>I would ideally like to pool only once, with a diamond kernel. I don’t think the max pooling operation supports kernel input, but I would ideally like to be able to do something like this:</p>
<pre><code class="lang-auto">pool = nn.functional.max_pool2d(x, kernel=[[0,1,0],[1,1,1],[0,1,0]], stride=1, padding=1)
</code></pre>
<p>I tried to mimic this behaviour by convolving with the same kernel ([[0,1,0],[1,1,1],[0,1,0]]), but I end up with a sum of the kernel and I’m not sure how I could implement the convolution to use max instead of sum to reduce the elements. Is it even possible? I need the solution to work with 3D images too, and it seems unfold only supports 2D images.</p>
<p>I would be thankful for any help with this issue!</p>

KFrank: <p>Hi Karol!</p>
<aside class="quote no-group" data-username="shroomite" data-post="1" data-topic="217660">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shroomite/48/76075_2.png" class="avatar"> shroomite:</div>
<blockquote>
<p>I would ideally like to pool only once, with a diamond kernel.</p>
</blockquote>
</aside>
<p>I don’t think that pytorch supports what you want with a single operation.</p>
<p>As an alternative to max-pooling along the two axes and then computing the<br>
element-wise max of the two results, you could construct a <code>Conv2d</code> with five<br>
<code>out_channels</code>, where each channel of its kernel is just a 3x3 slice with a single<br>
<code>1.0</code> corresponding to each of the five <code>1.0</code>s that appear in your “diamond kernel.”<br>
Then perform a max-pool along the channel dimension of you convolved image.</p>
<p>I don’t have any intuition about about which approach would be faster, but I would<br>
expect a single “diamond-kernel” max-pool with a well-tuned custom implementation<br>
(at the c++ or cuda level) to be faster than either of the two multi-step approaches.<br>
But writing such a custom implementation would be a project in its own right.</p>
<p>Having said that, as long you’re not looping (in python) over the pixels in your images,<br>
pytorch should be quite fast in applying the multi-step max-pools and/or convolutions.</p>
<p>You might try timing both, and I would expect that one (or both) will prove fast enough<br>
for your use case.</p>
<p>Best.</p>
<p>K. Frank</p>
Pytorch Model struggles to learn for large number of datapoints

simonduell: <p>I have a problem with my model, as it is struggling to learn when i increase the amount of training data.</p>
<p>My model relates to a physical system where the input is intended to be followed. It can also be described by a state space system.</p>
<p>My NN input basically consists of the state space input that needs to befollowed <strong>u</strong>(6 Values) and the current state of the system <strong>x</strong> (18 Values). The output of the system is an internal intermediate state <strong>z</strong> which is used to update the current system state <strong>x</strong> where <strong>z</strong> is a time derivative of <strong>x</strong> and is then passed through a second pre-trained NN to be further processed. The second NNs output <strong>w</strong> is then subject to another Matrix multiplication, where the result corresponds to the state space model output <strong>y</strong> which is supposed to follwow the input. Therefore, the loss of the network that outputs <strong>z</strong> is teh difference between <strong>u</strong> and <strong>y</strong>.</p>
<p>My state space input <strong>u</strong> consists of 20000 Datapoints, each with 6 Features as a timeseries. When i train my model to overfit onto one single datapoint this works without a problem, for 10 datapoints it still works, however, as soon as is use a larger subset of e.g. 100 Datapoints, the model fails to train for 3 of the 6 Features. For those 3 features the computations subsequent to the NN are more complicated than for the other 3 features.</p>
<p>However, the output transformation from <strong>z</strong> to <strong>y</strong> should’nt be a problem, as even if the implementation is not physically acurate, the model should still be able to learn according to the falsely implemented formulas. Nonetheless I checked these formulas and model multiple times and they should be fine.</p>
<p>As my model struggles, when I use more amount of training data, my suspision would be, that the model complexity is not sufficient to capture the dependencies of the problem. However, even doubling the amount of layers and the amount of neurons  per layer did not help in any way.</p>
<p>Unfortunately my code includes sensitive data, therefore i am not able to share my code. I hope that my explanation is enough and someone has an idea on how I can get my model to learn for larger amounts of training data.</p>
Cannot use Pytorch on A100-PCIE-40GB x 8

George_Polya: <p>I’m trying to run PyTorch on a server with eight A100-PCIE-40GB GPUs,<br>
but torch.cuda.is_available() returns False.<br>
The server doesn’t have an NVSwitch, so I don’t believe the issue is related to nvidia-fabricmanager.</p>
<p>Running <code>lspci | grep -i nvswitch</code> produces no output.<br>
<code>nvcc --version</code> shows CUDA 12.4, and <code>nvidia-smi</code> appears normal.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/795937cc6ed7ddfa911f1f34276df3bca234f2d9.png" data-download-href="https://discuss.pytorch.org/uploads/default/795937cc6ed7ddfa911f1f34276df3bca234f2d9" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/9/795937cc6ed7ddfa911f1f34276df3bca234f2d9.png" alt="image" data-base62-sha1="hjuVccrKD6JB048BFkHPhFNCHzP" width="411" height="500" data-dominant-color="272727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">749×909 26.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee05504ee936dd8eeb30ac6daf91c989a4595ac7.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee05504ee936dd8eeb30ac6daf91c989a4595ac7" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee05504ee936dd8eeb30ac6daf91c989a4595ac7.png" alt="image" data-base62-sha1="xXD1oAquU5G5u2bN0LJwrQM2AVV" width="690" height="88" data-dominant-color="272525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">950×122 6.11 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Disable MIG or make sure it’s properly configured and the Python process uses the desired MIG slice.</p>
Purpose of `torch.neg`, `torch.square`, and `torch.sqrt`

pylearn: <p>What is the purpose of <code>torch.neg</code>, <code>torch.square</code>, and <code>torch.sqrt</code>?<br>
Based on the doc, they are just regular operations. That is, the following lines are exactly the same.</p>
<ol>
<li><code>[torch.neg(x), torch.square(x), torch.sqrt(x)]</code></li>
<li><code>[x.neg(), x.square(), x.sqrt()]</code></li>
<li><code>[-x, x**2, x**-0.5]</code></li>
</ol>
<p>Which one should I use?</p>

ptrblck: <p>You can use whichever function name better suits you and some methods have aliases as different users have different preferences.</p>

lzm2256: <p>Just a quick note. torch.sqrt is **0.5, not **-0.5 as indecated in the 3rd point in the question.</p>
Optical Flow through backward warping?

ppoyazero: <p>I have read the documentations for torch.nn.functional.grid_sample and ended up having questions regarding the “correct” way to backward warp an img.</p>
<p>Suppose I have two images, prev and cur, which are used to calculate the 2D optical flow field(denoted as opt_flow).</p>
<p>The opt_flow denotes the movements of each pixel moving from prev to cur.</p>
<p>Suppose that pixel (3, 5) has optical flow (2, 2), and pixel (5, 7) has optical flow (3, 4).</p>
<p>Then, in order to grid_sample, the grid requires (-2, -2) in (5, 7), and (-3, -4) in (8, 11), but I cannot think of a clear way to create such a grid from the obtained opt_flow, since simply converting to - will place the displacements in wrong positions.</p>
<p>One way will be to apply cur-&gt;prev optical flow(inv_opt_flow) to prev instead, which if ideal will have (-2, -2) in (5, 7) and (-3, -4) in (8, 11).<br>
However this seems a bit unintuitive as we are applying the inverse optical flow to the previous image.</p>
<p>Is there a way to create the sampling grid for grid_sample from the forward optical flow?</p>
Torch.gather() without repeating the tensor

John_Snow: <p>I have a tensor A with shape: <code>torch.Size([10, 20, 30])</code></p>
<p>I also have a tensor B with shape: <code>torch.Size([1, 2, 3, 5])</code></p>
<p>I want to gather values from tensor A using indices in tensor B. The gathering should occur along the last dimension, such that the 5 values in tensor B correspond to the indices of the last dimension of tensor A. So <code>B.max()&lt;30</code> .</p>
<p>The resulting tensor should have the shape <code>torch.Size([1, 2, 3, 5, 10, 20])</code>.</p>
<p>Currently I can achieve this result by repeating both tensors such that A becomes <code>torch.Size([10, 20, 1, 2, 3, 30])</code> and B becomes <code>torch.Size([10, 20, 1, 2, 3, 5])</code> and then performing a torch.gather() operation on them. The dimensions in reality are much larger and this takes too much memory. I was wondering if there is a way to achieve the same without repeating the tensors.</p>

ptrblck: <p>Could you post your current code so we could use it as a reference for a potentially better implementation?</p>

John_Snow: <p>Sure, here it is:</p>
<pre><code class="lang-auto">A = torch.rand(10, 20, 30)
B = torch.rand(1, 2, 3, 5) 
B = (B*30).type(torch.long)

A = A[:, :, None, None, None, :].repeat(1, 1, *B.shape[:3], 1) # 10, 20, 1, 2, 3, 30
B = B[None, None, :, :, :, :].repeat(*A.shape[:2], 1, 1, 1, 1) # 10, 20, 1, 2, 3, 5

result = torch.gather(A, -1, B) # 10, 20, 1, 2, 3, 5
</code></pre>

ada-b19: <p>I’m also searching for a good solution here. However one workaround is to use expand instead of repeat because it doesn’t create a copy of the tensor.</p>
Pytorch suddenly slow down at a batch

Liu_Zesheng: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d450d0db5c5e6b0428680d8e9db744660d3bc681.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d450d0db5c5e6b0428680d8e9db744660d3bc681" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d450d0db5c5e6b0428680d8e9db744660d3bc681.jpeg" alt="image" data-base62-sha1="uiemZHhZbdLTHLVQAVX6yRHHg09" width="690" height="175" data-dominant-color="D0D3D6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">706×180 49 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am training my network. It works well and fast if I use part of the train and test dataset. However, if I use the full dataset, it starts with the normal fast speed but suddenly slow down at a batch in epoch 1…</p>
<p>Is there anything that got full at that time? Also notice at the beginning of epoch 2, it will be faster again then slow down at somepoint again…</p>

ptrblck: <p>You could profile your code to narrow down where the bottleneck appears. E.g. maybe your system is overheating and is thus reducing the clocks.</p>
Hyperparameter Sweeping by Only Measuring Convergence

benjamin-perry-duke: <p>One very simple question I have found in my research that I haven’t been able to fully answer revolves around hyperparameter sweeping.</p>
<p>Of course, there is the method of searching the entire hyperparameter space; however, this becomes computationally infeasible when considering the scale of that space.</p>
<p>The natural next answer people propose is using some package such as Optuna, which uses statistical estimators to provide some sort of “supervision” to the process, yet often I find that I still have to train my models until convergence (early stopping based on a validation loss) then use the resultant test metric as the optimization criterion for Optuna.</p>
<p>But the problem still requries full training of each iteration of hyperparameters. I am wondering, in cases where models are massive, if anyone knows of a quicker way to essentially “assess” the performance of a certain combination of hyperparameters.</p>
<p>For example, could I only train for 2-5 epochs on a given combination then assess the rate of convergence and use that in my optimization rather than full training and assessment on a validation loss?</p>
<p>I’m curious if anyone has any ideas!</p>
Forward Mode AD with multiple tangents for each primal

Mathieu_Simon: <p>Hello,</p>
<p>I am currently exploring ways to compute Jacobian-vector products (JVP) in PyTorch. I recently came across the following resources:</p>
<ul>
<li><a href="https://pytorch.org/tutorials/intermediate/forward_ad_usage.html" rel="noopener nofollow ugc">PyTorch Tutorial on Forward-Mode AD</a></li>
<li><a href="https://discuss.pytorch.org/t/forward-mode-ad-vs-double-grad-trick-for-jacobian-vector-product/159037">Forum Discussion: Forward-Mode AD vs. Double-Grad Trick for JVP</a></li>
</ul>
<p>However, I am still relatively new to auto-differentiation and find it challenging to fully understand how forward-mode AD operates and integrates with the forward pass.</p>
<h3><a name="p-462560-questions-1" class="anchor" href="#p-462560-questions-1"></a>Questions:</h3>
<ol>
<li>
<p>Compared to computing VJP (vector-Jacobian product) with <code>grad</code> in the backward pass, how does forward-mode AD differ in terms of:</p>
<ul>
<li>Speed of computation?</li>
<li>Memory usage?</li>
</ul>
</li>
<li>
<p>My main goal is to compute JVPs efficiently for <strong>multiple tangents</strong> for each primal i.e, each primal is associated with multiple tangents. Is there a way to compute JVPs for multiple tangents in parallel without having to recompute <code>f(primal)</code> for each tangent vector?</p>
</li>
</ol>
<p>Any insights or examples would be greatly appreciated!</p>
<p>Thank you!</p>

soulitzer: <ol>
<li>
<p>Speed of computation should be similar. Memory usage for forward AD is lower since you no longer need to save a bunch of activations for backward.</p>
</li>
<li>
<p>You likely want to do vectorized jvps. This is possible to do using torch.func by combining <a href="https://pytorch.org/docs/stable/generated/torch.func.vmap.html#torch.func.vmap" class="inline-onebox" rel="noopener nofollow ugc">torch.func.vmap — PyTorch 2.5 documentation</a> with <a href="https://pytorch.org/docs/stable/generated/torch.func.jvp.html#torch.func.jvp" class="inline-onebox" rel="noopener nofollow ugc">torch.func.jvp — PyTorch 2.5 documentation</a>.</p>
</li>
</ol>

torchjunkie: <p>So if I have a function f(x,y) of 2 variables, if I want the partials f_x and f_y I will effectively need to make 2 calls with tangents (1,0) and (0,1)? Is it possible to create higher dimensional duals so that x-&gt; (x, epsilon1,epsilon2) and same for y such that if I set z=f(x,y) the partials are found in z[1] and z[2]? one could get all sensitivities in 1 shot?</p>

soulitzer: <p>Yes, you can create a Batched dual with one-hot values and use functorch’s vmap + jvp to compute that in one forward pass.</p>
FSDP reduce operation

nasosg10: <p>Hello everyone!<br>
I am studying distributed training with PyTorch, and more precisely, Distributed Data Parallel (DDP) and Fully Sharded Data Parallel (FSDP). I have two questions regarding how some things function under the hood.</p>
<ol>
<li>
<p>From the PyTorch documentation of DDP, it is clear that the gradients are reduced with the “average” operation. Does the same thing hold for FSDP too? Basically, I am asking which is the reduction operation passed in the reduce scatter collective communication. Given that nobody mentions it, i guess it should be the same.</p>
</li>
<li>
<p>Another question, relevant to both settings, is whether the model.no_sync() is necessary when accumulating gradients to simulate a larger batch size. As far as I have understood (and did some math), it is key for efficiency and training speed,but does not change the final gradients, so an implementation without it is also correct.</p>
</li>
</ol>
<p>I would appreciate much any help with these, thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Forgot to mention, I use PyTorch 2.1.0</p>
Loading donut transformers model getting error

ankita1: <p>self.model = self.model.to(self.device)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1145, in to<br>
return self._apply(convert)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 797, in _apply<br>
module._apply(fn)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 797, in _apply<br>
module._apply(fn)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 797, in _apply<br>
module._apply(fn)<br>
[Previous line repeated 1 more time]<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 820, in _apply<br>
param_applied = fn(param)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1143, in convert<br>
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)<br>
RuntimeError: CUDA error: device-side assert triggered<br>
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.</p>
<p>my code is<br>
class Donut:<br>
def init(self):<br>
self.processor = DonutProcessor.from_pretrained(“naver-clova-ix/donut-base-finetuned-docvqa”)<br>
self.model = VisionEncoderDecoderModel.from_pretrained(“naver-clova-ix/donut-base-finetuned-docvqa”)<br>
self.device = “cuda” if torch.cuda.is_available() else “cpu”<br>
self.model.to(self.device)</p>
<p>torch version :- torch==2.0.1+cu118<br>
torchaudio==2.0.2+cu118<br>
torchvision==0.15.2+cu118<br>
Nvidia Driver :- Driver Version: 550.144.03 CUDA Version: 12.4</p>

ptrblck: <p>Often device asserts are triggered by failing indexing operations so check if e.g. an embedding layer received an invalid input or any other indexing fails.</p>

ankita1: <p>Thanks for reply. It has resolve , my max_position_embeddings mismatching.</p>
I am unable to export my model properly and use it through onnxruntime

yinjun1131: <p>I have PyTorch code that requires input of bfloat16, but some calculations need to be converted to float32. After the calculation is complete, it will be converted back to bfloat16 and calculated with the value of another bfloat16. This code runs normally, but if I try to convert to onnx and run it through onnxruntime, it will prompt ‘Could not find an implementation for MatMul (13) node with name’. What should I do</p>
<p>and this is my code</p>
<pre><code class="lang-auto">class Qwen2RMSNorm(nn.Module):
    def __init__(self, hidden_size, eps=1e-6):
        """
        Qwen2RMSNorm is equivalent to T5LayerNorm
        """
        super().__init__()
        self.weight = nn.Parameter(torch.ones(hidden_size))
        self.variance_epsilon = eps

        self .test1 = torch.full([5,5],1,dtype=torch.bfloat16)
        self .test2 = torch.full([5,5],1,dtype=torch.float32)

    def forward(self, hidden_states,kk = 0):
       #this input type was torch.bfloat16
        input_dtype = hidden_states.dtype
        # if not (input_dtype == torch.float32 or input_dtype == torch.float16 or input_dtype == torch.bfloat16 or input_dtype == torch.float64):
        hidden_states = hidden_states.to(torch.float32)
        variance = hidden_states.pow(2).mean(-1, keepdim=True)
        
       
        rsqrt = torch.rsqrt(variance + self.variance_epsilon)
        hidden_states = hidden_states * rsqrt
       

        return self.weight * hidden_states.to(input_dtype)
</code></pre>

yinjun1131: <p>In this code, if I try to modify the last line of return self. height * hidden_dates. to (input_detype) to return (self. height * hidden_dates). to (input_detype), the error will disappear. Why is this</p>

ptrblck: <p>Based on the error message and the code workaround it seems a lower dtype <code>matmul</code> operation is not supported for the device you are using in ONNX.</p>

yinjun1131: <p>Yes, it seems that in PyTorch, using operators such as mat and add on bfloat16 and float32 would be considered illegal, but I have already performed type conversion and still have this error. Is there any good solution?</p>