akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Single core of TPU gives prediction results different than the CPU results

Mohamed_Amara: <h2><a name="p-464023-questions-and-help-1" class="anchor" href="#p-464023-questions-and-help-1"></a><img src="https://discuss.pytorch.org/images/emoji/apple/question.png?v=12" title=":question:" class="emoji" alt=":question:" loading="lazy" width="20" height="20"> Questions and Help</h2>
<p>I encountered a discrepancy when training a model using PyTorch XLA on a TPU, where the results differed significantly from those obtained using CPU or GPU. After further investigation using a toy example, I noticed that the predictions made with PyTorch XLA were not consistent with those made on the CPU. Interestingly, when using PyTorch Lightning for training on TPU, the results were identical to the CPU output PyTorch Lightning depends on pytoch XLA when training with tpu. This led me to suspect that there may be some device-specific differences or initialization issues when using PyTorch XLA directly.</p>
<h3><a name="p-464023-code-2" class="anchor" href="#p-464023-code-2"></a>Code</h3>
<pre><code class="lang-auto">def generate_random_data(batch_size=1, num_channels=3, height=224, width=224):
    return torch.randn(batch_size, num_channels, height, width, dtype=torch.float32)

def load_model():
    return models.efficientnet_b0(weights='DEFAULT').eval()

random_data = generate_random_data()
# CPU inference
model = load_model()
cpu_result = inference_on_device(model, torch.device('cpu'), random_data)

# TPU inference
tpu_device = xm.xla_device() #single core of tpu
tpu_result = inference_on_device(model, tpu_device, random_data)

# Compare CPU and TPU results
print("Difference between CPU and XLA TPU results:", np.abs(cpu_result - tpu_result).max())
</code></pre>
<h3><a name="p-464023-output-3" class="anchor" href="#p-464023-output-3"></a>Output</h3>
<p><code>Difference between CPU and XLA TPU results: 0.025713682</code></p>
<p>Best Regard.</p>
Single core of TPU gives inference results different than the CPU results

Mohamed_Amara: <p>I encountered an issue when using PyTorch XLA to train a model on TPU. My main code gives a different results than training with CPU or GPU so I decided to check using a toy example and found that prediction using pytorch XLA gives results different than prediction using CPU.<br>
I also tried to check using pytorch lightning but it gives the same result like CPU so how to setup pytorch xla to give identical results like lightning?<br>
<a href="https://www.kaggle.com/code/saadsallam/tpu-cpu" rel="noopener nofollow ugc">Notebook</a></p>

frank58rich: <p>Hello,</p>
<p>I encountered a discrepancy when training a model using PyTorch XLA on a TPU, where the results differed significantly from those obtained using CPU or GPU. After further investigation using a toy example, I noticed that the predictions made with PyTorch XLA were not consistent with those made on the CPU. Interestingly, when using PyTorch Lightning for training on TPU, the results were identical to the CPU output. This led me to suspect that there may be some device-specific differences or initialization issues when using PyTorch XLA directly.</p>
<p>Best Regard,<br>
Frank<br>
<a href="https://www.read-theory.com" rel="noopener nofollow ugc">ReadTheory</a></p>
Change channels order (Python's permute equivalent)?

BaptisteAmato: <p>Hi,</p>
<p>I need to classify in C++ a BMP image within 20ms.<br>
Once the file opened, I have a pointer to an array, where the RGB pixels are stored one after the other.<br>
So I need to fill a Tensor as fast as possible from this array. Here is the code I use:</p>
<pre><code class="lang-auto">torch::Tensor image = torch::zeros({window_height, window_width, 3});
auto p = image.accessor&lt;float, 3&gt;();
for (int yy = origin_y; yy &lt; origin_y + window_height; yy++) {
	// unsigned char *addr_line = addr_origin + yy * width * step;
	unsigned char *addr_line = addr_origin + (height - 1 - yy) * width * step;

	for (int xx = origin_x; xx &lt; origin_x + window_width; xx++) {
		// 1 - Image coded in BGR, but Tensors are RGB
		// 2 - BMP images are coded from bottom to top. Pixel (i, j) once flipped is then (height - 1 - i, j)
		auto tensor_pixel = p[yy - origin_y][xx - origin_x];
		auto array_pixel = addr_line + xx * step;
		tensor_pixel[0] = *(array_pixel + 2) / 255.;
		tensor_pixel[1] = *(array_pixel + 1) / 255.;
		tensor_pixel[2] = *(array_pixel) / 255.;
	}
}
image = torch::unsqueeze(image, 0);
</code></pre>
<p>I put the channel dimension in 3rd position to speed up the filling, but how can I put it back in first position before unsqueezing (for the batch dimension)? I’m looking for a function similar to “permute” in Python</p>
<p>Best</p>

ptrblck: <p><code>permute</code> is also available in the C++ backend via:</p>
<pre><code class="lang-auto">image.permute({2, 0, 1}); 
</code></pre>

BaptisteAmato: <p>Awesome, thanks a lot!<br>
Would you have any tip to have a better understanding of the documentation? I’m having a hard time using the C++ backend of Libtorch, there aren’t that many examples of use (at least, not easily findable…).</p>
<p>Best</p>

ptrblck: <p>I’m checking the <a href="https://pytorch.org/cppdocs/search.html?q=permute&amp;check_keywords=yes&amp;area=default#">C++ docs</a> first for a matching function (for <code>permute</code> I knew it was there) and if I cannot find it, I’m usually checking the C++ code (the libtorch tests are a good place to see examples) for the method.</p>

BaptisteAmato: <p>Alright, will try to remember that before asking obvious questions. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Sure, but also feel free to post here in case you cannot find anything and think the documentation doesn’t mention the method. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Qirui_Sun: <p>Hi, May I ask a question?<br>
I found that tensor1.permute({0,2,1,3}); in C++ has the different value from tensor1.permute(0,2,1,3)  in python3. Are there any insights on that</p>

ptrblck: <p>No, this shouldn’t be the case if you index the result tensor in the correct way.<br>
Note that <code>permute</code> does not return a contiguous tensor. So check if you are trying to index the raw data without respecting the strides in C++.</p>

Qirui_Sun: <p>great！ Thanks a lot.</p>
Permute function has the different values in CPP and python3

Qirui_Sun: <p>Hi there,<br>
I am using <code>tensor.permute({0,2,1,3});</code> in C++ however, the output value is different from  <code>tensor.permute(0,2,1,3)</code> in python3. Are there any ideas on what happends</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/change-channels-order-pythons-permute-equivalent/105661/7">here</a>.</p>
`addThreadLocalCallback` concurrent modification

sternj: <p>Is there any reasonable case in which:</p>
<ol>
<li>Start callbacks (as added in a given thread in <code>addThreadLocalCallback</code>) are called in a different order than they’re added when no sampling is used?</li>
<li><code>addThreadLocalCallback</code> is called while callbacks are being iterated over and executed?</li>
</ol>
<p>Additionally, is there any case in which not all thread-local callbacks are called?</p>
Error loading for Error loading "torch\lib\shm.dll after running the generated exe file

surf_reta: <p>I have a pytorch program running successfully. Then I used the Pyinstaller to wrap us this python program into an exe file. This process was also finished successfully and generate d the related exe file. However, when I run this exe file, I got the following error message.</p>
<p>The program was developed under miniconda.</p>
<p>Python = 3.8.5<br>
Pytorch = 1.11.0  -cpu version<br>
Pyinstaller = 5.0.1</p>
<pre><code class="lang-auto">Traceback (most recent call last):
&gt;   File "cli.py", line 8, in &lt;module&gt;
&gt;   File "&lt;frozen importlib._bootstrap&gt;", line 991, in _find_and_load
&gt;   File "&lt;frozen importlib._bootstrap&gt;", line 975, in _find_and_load_unlocked
&gt;   File "&lt;frozen importlib._bootstrap&gt;", line 671, in _load_unlocked
&gt;   File "PyInstaller\loader\pyimod03_importers.py", line 495, in exec_module
&gt;   File "scripts\integration_work\mdflag.py", line 20, in &lt;module&gt;
&gt;   File "&lt;frozen importlib._bootstrap&gt;", line 991, in _find_and_load
&gt;   File "&lt;frozen importlib._bootstrap&gt;", line 975, in _find_and_load_unlocked
&gt;   File "&lt;frozen importlib._bootstrap&gt;", line 671, in _load_unlocked
&gt;   File "PyInstaller\loader\pyimod03_importers.py", line 495, in exec_module
&gt;   File "torch\__init__.py", line 137, in &lt;module&gt;
&gt; OSError: [WinError 126] The specified module could not be found. Error loading "C:\Users\u\AppData\Local\Temp\_MEI180242\torch\lib\shm.dll" or one of its dependencies.
&gt; [4656] Failed to execute script 'cli' due to unhandled exception!
</code></pre>

bairizhuozhuo: <p>Hello, have you solved this problem?</p>

LiuGaoyong: <p>pip install intel-openmp</p>

Enzo_Salvadori: <p>any news? I have the same problem here.</p>

superleon: <p>suggest to reinstall pytorch. install pytorch==2.2.2, torchvision==0.17.2, and torchaudio=2.2.2.</p>

lwl1120211123: <p>have you solve this problem</p>
How to checkpoint/pickle FlexAttention with sliding window mask

cathal: <p>Hello, I am using flex attention in the following way. I have ‘sliding_window_mask’ defined inside forward. ‘forward’ fails to checkpoint because of the nested function definition. However I have to define it this way because window_size does not exist outside forward.</p>
<p>def forward(self, window_size, …):<br>
def sliding_window_mask(b, h, q_idx, kv_idx):<br>
return abs(q_idx - kv_idx) &lt;= window_size<br>
self.block_mask = create_block_mask(sliding_window_mask, B=None, H=None, Q_LEN=seq_len, KV_LEN=seq_len, _compile=True)</p>
<p>I have tried:<br>
1: creating sliding_window_mask outside forward, adding a window_size parameter. This fails bc create_block_mask requires sliding_window_mask to have 4 parameters<br>
2: I have tried deleting sliding_window_mask from the checkpoint by overloading <strong>getstate</strong> but this didn’t work.</p>
<p>any advice would be greatly appreciated, thanks!</p>
Does order of transforms applied for data augmentation matter in Torchvision transforms?

Amit_Sur: <p>I have the following Custom dataset class for image segmentation task.</p>
<pre><code class="lang-auto">class LoadDataset(Dataset):

    def __init__(self, img_dir, mask_dir, apply_transforms = None):
        self.img_dir = img_dir
        self.mask_dir = mask_dir
        self.transforms = apply_transforms
        self.img_paths, self.mask_paths = self.__get_all_paths()
        self.__pil_to_tensor = transforms.PILToTensor()
        self.__float_tensor = transforms.ToDtype(torch.float32, scale = True)
        self.__grayscale = transforms.Grayscale()

    def __get_all_paths(self):
        img_paths = [os.path.join(self.img_dir, img_name.name) for img_name in os.scandir(self.img_dir) if os.path.isfile(img_name)]
        mask_paths = [os.path.join(self.mask_dir, mask_name.name) for mask_name in os.scandir(self.mask_dir) if os.path.isfile(mask_name)]
        img_paths = sorted(img_paths)
        mask_paths = sorted(mask_paths)
        return img_paths, mask_paths

    
    def __len__(self):
        return len(self.img_paths)

    
    def __getitem__(self, index):
        img_path, mask_path = self.img_paths[index], self.mask_paths[index]
        img_PIL = Image.open(img_path)
        mask_PIL = Image.open(mask_path)
        img_tensor = self.__pil_to_tensor(img_PIL)
        img_tensor = self.__float_tensor(img_tensor)
        mask_tensor = self.__pil_to_tensor(mask_PIL)
        mask_tensor = self.__float_tensor(mask_tensor)
        mask_tensor = self.__grayscale(mask_tensor)
        if self.transforms:
            img_tensor, mask_tensor = self.transforms(img_tensor, mask_tensor)
        return img_tensor, mask_tensor

</code></pre>
<p>When I am applying the following <code>transforms.RandomHorizontalFlip()</code> either the image or the mask is being flipped. But if the change the order of transformations in <code>__getitem__</code> to the following then it works fine.</p>
<pre><code class="lang-auto">    def __getitem__(self, index):
        img_path, mask_path = self.img_paths[index], self.mask_paths[index]
        img_PIL = Image.open(img_path)
        mask_PIL = Image.open(mask_path)
        if self.transforms:
            img_PIL, mask_PIL = self.transforms(img_PIL, mask_PIL)
        img_tensor = self.__pil_to_tensor(img_PIL)
        mask_tensor = self.__pil_to_tensor(mask_PIL)
        img_tensor = self.__float_tensor(img_tensor)
        mask_tensor = self.__float_tensor(mask_tensor)
        mask_tensor = self.__grayscale(mask_tensor)
        return img_tensor, mask_tensor
</code></pre>
<p>Does the order transformation matter somehow? I am using <code>torchvision.transforms.v2</code> for all the transformations.</p>

Bjorn_Lindqvist: <p>Yes, the order matters.</p>
Nvrtc error with CUDA

Laya1: <p>I am using PyTorch 2.5.1 (CUDA12.4)  with Python 3.10.15 on Windows 11. I am performing operations on complex numbers as follows:</p>
<pre><code class="lang-auto">a = torch.randn(2, 10, 5, 6)  # dtype -&gt; torch.float32
c = (1j * a).exp()  # expected dtype -&gt; torch.complex64
</code></pre>
<p>I am getting the following error:</p>
<pre><code class="lang-auto">nvrtc: error: failed to open nvrtc-builtins64_124.dll.
Make sure that nvrtc-builtins64_124.dll is installed correctly.
</code></pre>
<p>Executing only <code>1j * a</code> does not raise an error. The error occurs in the exponentiation. The complete error message is very long to paste here, but the first few lines are:</p>
<pre><code class="lang-auto">RuntimeError: 
  #ifdef __HIPCC__
  #define ERROR_UNSUPPORTED_CAST ;
  // corresponds to aten/src/ATen/native/cuda/thread_constants.h
  #define CUDA_OR_ROCM_NUM_THREADS 256
  // corresponds to aten/src/ATen/cuda/detail/OffsetCalculator.cuh
  #define MAX_DIMS 16
  #ifndef __forceinline__
  #define __forceinline__ inline __attribute__((always_inline))
  #endif
  #else
</code></pre>
<p>It mentions something about unsupported cast error, but I am not sure why this occurs and how to fix it. I have tried changing the Python versions (3.11, 3.12) and PyTorch versions (2.5.1cu121, 2.5.1cu118, 2.4.0cu124) but nothing seems to work. I have the CudaToolkit and cuDNN installed from NVIDIA’s website. Thanks in advance for your help.</p>

Laya1: <p>As an update, this happens for <code>abs()</code> and <code>log()</code> operations, but not for <code>sum()</code>. It is not clear why some of the operations are not executed on CUDA. I need to resolve this to run the rest of the code on GPU. I would appreciate any insights on this.</p>
<p>Thanks in advance for your help.</p>

ptrblck: <p>The error is pointing to a missing <code>nvrtc</code> library in your build, which is used for runtime compilation of some kernels. I don’t have a Windows system to reproduce the issue and cannot reproduce it on Linux.<br>
Ad a workaround you could try to install nvrtc manually.<br>
CC <a class="mention" href="/u/malfet">@malfet</a> in case you have seen this issue before.</p>

Mole_Turner: <p>I’ve encountered a similar problem and whilst I don’t know the reason for the issue, I think I’ve found a possible cause.</p>
<p>In my case I’m using cuda environments, and when the environment is created in cuda’s default environment path <code>C:\Users\user_name\AppData\Local\anaconda3\envs\my_env</code> I don’t appear to have the issue. However, when I specify a different location for the env folder to be created i.e <code>C:\Users\user_name\.conda\envs\my_env</code> then the issue presents itself.</p>
<p>I’ve tested this with several blank environments where I only install python and the Pytorch libraries and the effect appears to be consistent.</p>
<p>In all cases I can also locate the <code>nvrtc</code> dlls within the relevant <code>bin</code> directory.</p>
<p>Any ideas?</p>
VGG16 for CIFAR10

Bjorn_Lindqvist: <p>Hello fellow deep learners,</p>
<p>To learn more about image classification I have implemented VGG16 for CIFAR10 in PyTorch. My code works and the training converges. But the model is capable of ~95% accuracy whereas mine only reaches ~89%. So I wonder if anyone could take a loot at my training code to see what could be improved?</p>
<aside class="onebox githubgist" data-onebox-src="https://gist.github.com/bjourne/855e22acc36ca6a7ffb958c6f2bb3eb0">
  <header class="source">

      <a href="https://gist.github.com/bjourne/855e22acc36ca6a7ffb958c6f2bb3eb0" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/bjourne/855e22acc36ca6a7ffb958c6f2bb3eb0" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/bjourne/855e22acc36ca6a7ffb958c6f2bb3eb0</a></h4>

  <h5>vgg16_cifar.py</h5>
  <pre><code class="Python">from itertools import islice
from pathlib import Path
from torch.nn import *
from torch.nn.functional import cross_entropy, mse_loss, one_hot
from torch.nn.init import constant_, kaiming_normal_, normal_
from torch.optim import SGD
from torch.optim.lr_scheduler import CosineAnnealingLR
from torch.utils.data import DataLoader
from torchinfo import summary
from torchtoolbox.transform import Cutout</code></pre>
   This file has been truncated. <a href="https://gist.github.com/bjourne/855e22acc36ca6a7ffb958c6f2bb3eb0" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>TIA!</p>

wilhelm: <p>A learning rate of 0.1 is IMHO too much.<br>
You should start with something like 1-e4.<br>
Secondly you need a scheduler for your training.</p>
<p>Ok, I saw right now, that you are using already a <code>CosineAnnealingLR</code>.<br>
I would something more conventional like a <code>LinerLR</code> or <code>StepLR</code>.</p>

Bjorn_Lindqvist: <p>After tweaking a bit I got 93.6%. Better, but still not very impressive.</p>

aavraham: <p>Hi Bjorn, I would like to take part in the efforts of improving the accuracy. I would appreciate if you could provide me with access to your github repo and maybe collaborate together.</p>

Bjorn_Lindqvist: <p>Hello Aviv. My low performance was due to insufficient data augmentation. With better augmentation I’m getting way better results. All my code is here: <a href="https://github.com/bjourne/ml-stuff" class="inline-onebox" rel="noopener nofollow ugc">GitHub - bjourne/ml-stuff</a></p>
Communication behavior of FSDP

HariSeldon11988: <p>Hello together,</p>
<p>I’m currently doing some profiling with a FSDP wrapped model. I use both, the <code>full_shard</code> and <code>hybrid_shard</code> policy.</p>
<p>My model has 12 units so I have 12 <code>FSDP.foward</code> calls or blocks, similar for the backward pass. These forwards blocks consists always of the same three functions:</p>
<ul>
<li><code>pre_forward</code> (launches <code>all_gather</code>)</li>
<li>compute</li>
<li><code>post_forward</code></li>
</ul>
<p>I would expect to have <code>12 all_gather</code> calls triggered from the 12 <code>pre_forward</code> functions part of the 12 forward blocks, but at the beginning I have one additional single <code>pre_forward</code> which launches an additional <code>all_gather</code>. So overall I have 13 <code>all_gather</code> and 12 units.</p>
<p>In the backward pass I find as expected 12 <code>all_gather</code> calls but 13 <code>reduce_scatter</code>. One additionally triggered from an separate <code>post_backward</code> after the last backward block. If I use the <code>hybrid_shard</code> policy I have as expected some additional <code>all_reduces</code>, but again I have 13 of them.</p>
<p>My question is:</p>
<p>Does someone know what the 13th <code>all_gather</code> in forward and the 13th <code>reduce_scatter</code> and <code>all_reduce</code> in the backward is doing?<br>
Why do I have these additional communication calls and why not at <code>all_gather</code></p>
<p>Would appreciate any kind of help!</p>
RuntimeError: result type Float can't be cast to the desired output type Long---

fuad47: <p>This is my code, I get this error. Tried to converrt df[labels] to torch.float but the issue still exists.<br>
Getting —&gt; trainer.train() error.</p>
<pre><code class="lang-auto"># Convert to Hugging Face dataset
dataset = Dataset.from_pandas(df)

# Load tokenizer and model
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=len(mlb.classes_), problem_type="multi_label_classification")

# Tokenize data
def preprocess_function(examples):
    return tokenizer(examples['job_title'], truncation=True, padding=True)

tokenized_dataset = dataset.map(preprocess_function, batched=True)

# Ensure labels are of type torch.float (this is required for multi-label classification)
def cast_to_float(example):
    example['labels'] = torch.tensor(example['labels'], dtype=torch.float)  # Convert labels to torch.float
    return example

tokenized_dataset = tokenized_dataset.map(cast_to_float)

# Training arguments
training_args = TrainingArguments(
    output_dir="./results",
    evaluation_strategy="epoch",
    save_strategy="epoch",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    logging_dir="./logs",
)

# Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    eval_dataset=tokenized_dataset,  # Ideally, you should split this into train/test datasets.
)

# Train model
trainer.train()
</code></pre>

ptrblck: <p>It’s unclear which operation fails in your model, but I would assume you are trying to perform an inplace operation with an integral and floating type which is disallowed as explained in the <a href="https://pytorch.org/docs/stable/tensor_attributes.html#torch-dtype">docs</a>:</p>
<blockquote>
<p>An integral output tensor cannot accept a floating point tensor.</p>
</blockquote>
<p>Example:</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.tensor(1)
print(a.type())
# torch.LongTensor

b = torch.tensor(2.)
print(b.type())
# torch.FloatTensor

a * b
# tensor(2.)
a *= b
# RuntimeError: result type Float can't be cast to the desired output type Long
</code></pre>

fuad47: <p>This is tokenized_dataset which I am going to train:<br>
Dataset({<br>
features: [‘job_title’, ‘labels’, ‘input_ids’, ‘token_type_ids’, ‘attention_mask’],<br>
num_rows: 3<br>
})</p>
<p>job_title and token_type_ids are list type objects other columns are torch.LongTensor type. Should I exclude them from training dataset?</p>
Multi-node, Multi-gpu training

DHARMENDRA_PRAJAPAT: <p>I have given the link to some code that does not have any specific dataloader defined, is there any way it can be modified for the multi-node training? Kindly help</p><aside class="onebox githubblob" data-onebox-src="https://github.com/TonyNemo/UBAR-MultiWOZ/blob/master/train.py">
  <header class="source">

      <a href="https://github.com/TonyNemo/UBAR-MultiWOZ/blob/master/train.py" target="_blank" rel="noopener nofollow ugc">github.com/TonyNemo/UBAR-MultiWOZ</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/TonyNemo/UBAR-MultiWOZ/blob/master/train.py" target="_blank" rel="noopener nofollow ugc">train.py</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/TonyNemo/UBAR-MultiWOZ/blob/master/train.py" rel="noopener nofollow ugc"><code>master</code></a>
</div>


      <pre><code class="lang-py">from transformers.optimization import AdamW, get_linear_schedule_with_warmup
from transformers import GPT2Tokenizer, GPT2LMHeadModel, GPT2Model
from eval import MultiWozEvaluator
from damd_net import DAMD, cuda_, get_one_hot_input
from reader import MultiWozReader
import utils
from torch.optim import Adam
import torch
import torch.nn as nn

import os
import random
import argparse
import time
import logging
import json
import tqdm
import numpy as np
from torch.utils.tensorboard import SummaryWriter
from tqdm import tqdm
</code></pre>



  This file has been truncated. <a href="https://github.com/TonyNemo/UBAR-MultiWOZ/blob/master/train.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Is there a way to use list of indices to simultaneously access the modules of `nn.ModuleList` in python?

Adex: <p>Is there a way to use list of indices to simultaneously access the modules of<br>
<code>nn.ModuleList</code> in python?</p>
<p>I am working with pytorch <code>ModuleList</code> as described below,</p>
<pre><code class="lang-auto">decision_modules = nn.ModuleList([nn.Linear(768, 768) for i in range(10)])
</code></pre>
<p>Our input data is of the shape <code>x=torch.rand(32,768)</code>. Here <code>32</code> is the batch size and <code>768</code> is the feature dimension.</p>
<p>Now for each input data point in a minibatch of <code>32</code> datapoints, we want to select <code>4</code> decision modules from the list of <code>decision_modules</code>. The <code>4</code> decision engines from <code>decision_engine</code> are selected using an index list as explained below.</p>
<p>I have a index matrix of dimensions <code>ind</code>. The <code>ind</code> matrix is of dimension <code>torch.randint(0,10,(4,4))</code>.</p>
<p>I want to us a solution without use of loops as loops slows down the xecution significantly.</p>
<p>But the following code throws and error.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

linears = nn.ModuleList([nn.Linear(768, 768) for i in range(10)])
ind=torch.randint(0,10,(4,4))
input=torch.rand(32,768)

out=linears[ind](input)
</code></pre>
<p>The following error was observed</p>
<blockquote>
<p>File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\container.py:334, in ModuleList.<strong>getitem</strong>(self, idx)<br>
332     return self.<strong>class</strong>(list(self._modules.values())[idx])<br>
333 else:<br>
 → 334     return self._modules[self._get_abs_string_index(idx)]</p>
<p>File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\modules\container.py:314, in ModuleList._get_abs_string_index(self, idx)<br>
312 def _get_abs_string_index(self, idx):<br>
313     “”“Get the absolute index for the list of modules.”“”<br>
 → 314     idx = operator.index(idx)<br>
315     if not (-len(self) &lt;= idx &lt; len(self)):<br>
316         raise IndexError(f"index {idx} is out of range")</p>
<p>TypeError: only integer tensors of a single element can be converted to an index</p>
</blockquote>
<p>Any help will be highly useful.</p>
Restarting a distributedDataParallel

pytorcher: <p>I know this is a weird use case, but I need to reset my DDP in the middle of a training run.  ie. I call</p>
<pre><code class="lang-auto">model = myModel()
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu], find_unused_parameters=True)
...
model = model.module
...
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu], find_unused_parameters=True)
</code></pre>
<p>When I do this I get the error:</p>
<pre><code class="lang-auto">RuntimeError: Tensors must be CUDA and dense
</code></pre>
<p>I have read that this is when tensors are sparse, but I don’t believe any of them are. I changed the second call to:</p>
<pre><code class="lang-auto">        foundOne = False
        for name, param in model.named_parameters():
            if param.is_sparse:
                print(name)
                foundOne = True
        print('Found a sparse parameter: %d' % foundOne)
        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu], find_unused_parameters=True)
</code></pre>
<p>and it does not find any sparse tensors.  This also appears to not work even with only a single GPU, though the code runs properly with regular DataParallel or no DataParallel and just sending to(cuda).</p>

pytorcher: <p>Turns out error that said two possible things was the other thing. <img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=12" title=":man_facepalming:" class="emoji" alt=":man_facepalming:" loading="lazy" width="20" height="20"></p>
<p>Problem solved by adding</p>
<pre><code class="lang-auto">model = model.to('cuda')
</code></pre>
<p>Before the second call to DDP.</p>
Interactions between ThreadLocalCallback and forward pre-hook

sternj: <p>Is there any case in which, in a single thread, pre-hooks that add thread-local callbacks and the <code>StartCallback</code> function of those callbacks will occur out of order of each other? For the sake of simplicity, let’s say that <code>addThreadLocalCallback</code> is called only with <code>RecordScope::Function</code></p>
<p>This is to say, could it ever be the case that you have a call chain like:</p>
<ol>
<li>forward pre_hook for Module instance 1</li>
<li>forward pre_hook for Module instance 2</li>
<li>StartCallback from LocalCallbackManager is induced by an operation associated with Module 1, but not Module 2</li>
<li>forward post_hook for Module 1 is called</li>
<li>forward post_hook for Module 2 is called</li>
</ol>
<p>as opposed to:</p>
<ol>
<li>forward pre_hook for Module instance 1</li>
<li>forward pre_hook for Module instance 2</li>
<li>StartCallback from LocalCallbackManager is induced by an operation associated with Module 2 (and also Module 1 if module 1 calls module 2)</li>
<li>forward post_hook for Module 2 is called</li>
<li>forward post_hook for Module 1 is called</li>
</ol>
<p>Perhaps more succinctly-- if a forward pre_op hook is called and calls <code>addThreadLocalCallback</code>, will the next operation for that specific thread only ever be one of the following:</p>
<ul>
<li>the StartCallback is called</li>
<li>the forward post_op hook is called</li>
<li>A different forward pre-op hook is called</li>
</ul>
<p>I’m interested in any possible corner case here, including with asynchronous execution and with tasks pausing/resuming in a way that’s not observable from the APIs that I talked about.</p>
Multiclass semantic segmentation giving zero in all output channels except for the first channel

Amit_Sur: <p>I am training a UNet model for multiclass semantic segmentation. My training loop is as follows</p>
<pre><code class="lang-auto">for X, y in dataloader:
        X, y = X.to(device), y.to(device).squeeze().long()
        optimizer.zero_grad()
        pred = model(X) # [N, C, H, W]
        loss = loss_fn(pred, y)
</code></pre>
<p>I am using <code>nn.CrossEntropyLoss</code> as loss function which requires the input to have shape <code>[N, C, H, W]</code> and target to have shape <code>[N, H, W]</code>. After training the model, I am having 0.37 training loss and 0.32 validation loss. But the prediction that I am getting is like shown below, only the first channel in the prediction has some valid logit while all other channels have 0</p>
<pre><code class="lang-auto">array([[[4.554523 , 4.376083 , 4.3859653, ..., 4.385881 , 4.3868537,
         4.5185075],
        [4.372363 , 4.3757625, 4.387993 , ..., 4.387873 , 4.3772345,
         4.3609285],
        [4.3757234, 4.3763394, 4.378887 , ..., 4.3940897, 4.3748627,
         4.374463 ],
        ...,
        [4.377365 , 4.3794756, 4.3913045, ..., 4.3968244, 4.386844 ,
         4.36672  ],
        [4.379352 , 4.3720374, 4.3786154, ..., 4.3863754, 4.376202 ,
         4.368247 ],
        [4.4423575, 4.385245 , 4.3882155, ..., 4.3879733, 4.387912 ,
         4.3996263]],

       [[0.       , 0.       , 0.       , ..., 0.       , 0.       ,
         0.       ],
        [0.       , 0.       , 0.       , ..., 0.       , 0.       ,
         0.       ],
        [0.       , 0.       , 0.       , ..., 0.       , 0.       ,
         0.       ],
        ...,
        [0.       , 0.       , 0.       , ..., 0.       , 0.       ,
         0.       ],
        [0.       , 0.       , 0.       , ..., 0.       , 0.       ,
         0.       ],
        [0.       , 0.       , 0.       , ..., 0.       , 0.       ,
         0.       ]]], dtype=float32)
</code></pre>
<p>What should I do?</p>

ptrblck: <p>Is your model returning logits or are you applying e.g. <code>nn.ReLU</code> on the output?</p>

Amit_Sur: <p>I am applying ReLU after the very last convolution layer that has <code>out_channels</code> = <code>no of classes</code>.</p>

ptrblck: <p>Remove it so your model is returning raw logits.</p>

Amit_Sur: <p>Thank you, but I have another doubt, should I keep the Batch Normalization layer or should I remove that as well?</p>

ptrblck: <p>Keep all layers and call <code>model.eval()</code> for the validation loop and <code>model.train()</code> before starting the training. This makes sure some layers change their behavior according to training/validation runs. E.g. batchnorm layers will use the running stats during validation runs to normalize the activation inputs.</p>
Problem with pytorch and cuda installation

Jothi_mani: <p>I tried to install pytorch with cuda enabled using the following command in conda prompt: conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia.<br>
but I got the following error: (‘Connection broken: IncompleteRead(217545068 bytes read, 176616422 more expected)’, IncompleteRead(217545068 bytes read, 176616422 more expected)).</p>
<p>I am not sure what this means to me. Could anyone please explain why this is happening?</p>

ptrblck: <p>It seems your internet connection might have issues so you might want to retry the download.</p>

robertopfneto: <p>I tried this and not working. Is the same error when I run the install command</p>

ptrblck: <p>Works for me:</p>
<pre data-code-wrap="python"><code class="lang-python">conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
...
The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
...
   pytorch-2.5.1              |py3.10_cuda11.8_cudnn9.1.0_0        1.54 GB  pytorch
    pytorch-cuda-11.8          |       h7e8668a_6           7 KB  pytorch
    pytorch-mutex-1.0          |             cuda           3 KB  pytorch
    pyyaml-6.0.2               |  py310h5eee18b_0         193 KB
    requests-2.32.3            |  py310h06a4308_1         101 KB
    sympy-1.13.3               |  py310h06a4308_0        11.3 MB
    torchaudio-2.5.1           |      py310_cu118         6.2 MB  pytorch
    torchtriton-3.1.0          |            py310       233.5 MB  pytorch
    torchvision-0.20.1         |      py310_cu118         8.3 MB  pytorch
...
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">python -c "import torch; print(torch.__version__); print(torch.version.cuda); print(torch.randn(1).cuda())"                                                                                                                              
2.5.1                                                                                                                                                                                                                                                                             
11.8                                                                                                                                                                                                                                                                              
tensor([-1.2686], device='cuda:0')    
</code></pre>
Trying torch.jit.script with a super().forward

milongo: <p>Hi, I have this code:</p>
<pre data-code-wrap="python"><code class="lang-python">class Decoder(nn.ConvTranspose1d):
    """A decoder layer that consists of ConvTranspose1d.

    Arguments
    ---------
    *args : tuple
    **kwargs : dict
        Arguments passed through to nn.ConvTranspose1d

    Example
    -------
    &gt;&gt;&gt; x = torch.randn(2, 100, 1000)
    &gt;&gt;&gt; decoder = Decoder(kernel_size=4, in_channels=100, out_channels=1)
    &gt;&gt;&gt; h = decoder(x)
    &gt;&gt;&gt; h.shape
    torch.Size([2, 1003])
    """

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
    
    def forward(self, x):
        """Return the decoded output.

        Arguments
        ---------
        x : torch.Tensor
            Input tensor with dimensionality [B, N, L].
                where, B = Batchsize,
                       N = number of filters
                       L = time points

        Returns
        -------
        out : torch.Tensor
            The decoded outputs.
        """

        if x.dim() not in [2, 3]:
            print("heeehee")
        
        x = super().forward(x if x.dim() == 3 else torch.unsqueeze(x, 1))

        if torch.squeeze(x).dim() == 1:
            x = torch.squeeze(x, dim=1)
        else:
            x = torch.squeeze(x)
        return x
    
decoder = Decoder(100, 100, 4)

torch.jit.script(decoder)
</code></pre>
<p>And I’m getting this error:</p>
<pre><code class="lang-auto">RuntimeError: 
'Tensor' object has no attribute or method 'forward'.:
  File "/var/folders/7p/pb1c1byn4j7b1ryyh67rh9gc0000gp/T/ipykernel_34769/3603230582.py", line 42
            print("heeehee")
        
        x = super().forward(x if x.dim() == 3 else torch.unsqueeze(x, 1))
            ~~~~~~~~~~~~~ &lt;--- HERE
</code></pre>
<p>Why does TorchScript say <code>super()</code> is a Tensor?</p>
<p>Thanks!</p>
RuntimeError: PyTorch 1.8.1+cu102 and NVIDIA A100-SXM4-40GB

GGlearning: <p>Hi Folks,</p>
<p>The current version of PyTorch in my Conda environment is 1.8.1+cu102. My code throws following warning and error</p>
<p><strong>UserWarning:<br>
NVIDIA A100-SXM4-40GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.</strong></p>
<p><strong>RuntimeError: CUDA error: no kernel image is available for execution on the device</strong>.</p>
<p>It will be really great to have a solution without disturbing the current PyTorch version.</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Your Ampere GPU (A100) needs CUDA 11.x and is not compatible with CUDA 10.2, which you are using right now. Install the current binaries with any CUDA 11 or 12 runtime and it will work.</p>

GGlearning: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, what is the correct approach to install the current binaries with any CUDA 11 or 12 ? Do I need to create another conda environment and install all the dependencies ? Or there is any way  to fix the current environment ?</p>

ptrblck: <p>No, you don’t need conda and can just copy paste the install command from the <a href="https://pytorch.org/get-started/locally/">install matrix</a> into a clean and empty environment.</p>

Bernard_Arnault: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I have NVIDIA A100 80GB PCIe and I get no available CUDA while running PyTorch.<br>
Which CUDA version is compatible for me?</p>

ptrblck: <p>All current PyTorch binaries will work since they ship with CUDA&gt;=11.</p>

Ambuj-Choudha: <p>Hey hi thanks for the helpful reply, I couldn’t find a comprehensive table/matrix for this. It would be helpful to know, where can one learn more about this.</p>
<p>I could only find this information beacuse you pointed it out. Thanks a lot really!!<br>
More power to you.</p>
<p>Regards,<br>
Ambuj C</p>