akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Can't run torch conv2d with AMD Instinct MI250X

lbj_obj: <p>I am on AMD 250X GPU after installed rocm5.6 with pytorch version 2.2 following commend</p>
<pre><code class="lang-auto"> pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 --index-url https://download.pytorch.org/whl/rocm5.6
</code></pre>
<p>I am not able to run the test code:</p>
<pre><code class="lang-auto">import torch

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
x = torch.rand(1, 3, 64, 64, device=device)  # Random tensor
conv = torch.nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1).to(device)
y = conv(x)
print(y.shape)
</code></pre>
<p>error:</p>
<pre><code class="lang-auto">MIOpen(HIP): Warning [FindWinogradSolutions] /MIOpen/src/sqlite_db.cpp:220: Internal error while accessing SQLite database: disk I/O error
MIOpen(HIP): Warning [FindDataGemmSolutions] /MIOpen/src/sqlite_db.cpp:220: Internal error while accessing SQLite database: disk I/O error
MIOpen(HIP): Warning [FindDataDirectSolutions] /MIOpen/src/sqlite_db.cpp:220: Internal error while accessing SQLite database: disk I/O error
MIOpen(HIP): Warning [FindDataImplicitGemmSolutions] /MIOpen/src/sqlite_db.cpp:220: Internal error while accessing SQLite database: disk I/O error
MIOpen(HIP): Warning [FindFftSolutions] /MIOpen/src/sqlite_db.cpp:220: Internal error while accessing SQLite database: disk I/O error
MIOpen Error: /MIOpen/src/ocl/convolutionocl.cpp:517: Forward Convolution cannot be executed due to incorrect params
Traceback (most recent call last):
....
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: miopenStatusUnknownError

</code></pre>
<p>However I don’t have issue with AMD 210X GPU.</p>

fluidnumerics_joe: <p>Hey <a class="mention" href="/u/lbj_obj">@lbj_obj</a> ,<br>
This issue doesn’t appear to be related to the GPU’s that you’re using.<br>
Specifically, the error</p>
<pre><code class="lang-auto">Internal error while accessing SQLite database: disk I/O error
</code></pre>
<p>suggests there’s an issue with the filesystem where MIOpen is attempting to create an sqlite database.</p>
<p>Can you confirm that you have sufficient space on the disk or file system that hosts your <code>/home</code> directory ?</p>
<pre><code class="lang-auto">df -h
</code></pre>
<p>MIOpen, by default, will use <code>~/.cache/miopen</code> for storing pre-compiled kernels and other performance related information. Can you confirm this path exists and you have permissions to read/write there.</p>
<pre><code class="lang-auto">ls -l ~/.cache/
ls -l ~/.cache/miopen/
</code></pre>
<p>Alternatively, can you confirm that the environment variable <code>MIOPEN_CACHE_DIR</code> is not set</p>
<pre><code class="lang-auto">echo $MIOPEN_CACHE_DIR
</code></pre>
<p>If it is set to some path, make sure that you have read/write access to that location ?</p>
ROCm: hipBLASLt error with gfx1103

lumie: <p>Hi,</p>
<p>I get the following error when using the Pytorch ROCm version:</p>
<blockquote>
<p>UserWarning: Attempting to use hipBLASLt on an unsupported architecture! Overriding blas backend to hipblas</p>
</blockquote>
<p>Here is a minimal example to recreate the error:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; t = torch.rand((5,5), device=0)
&gt;&gt;&gt; l = torch.nn.Linear(5,5).to(0)
&gt;&gt;&gt; l(t)
</code></pre>
<p>I set HSA_OVERRIDE_GFX_VERSION=11.0.0 before running python.</p>
<p>I installed ROCm 6.2.2 from this repository: <a href="https://gitlab.archlinux.org/archlinux/packaging/packages/rocm-hip-sdk" class="inline-onebox" rel="noopener nofollow ugc">Arch Linux / Packaging / Packages / rocm-hip-sdk · GitLab</a> and build and installed hipBLASLt 0.8.0 from this repository: <a href="https://github.com/ROCm/hipBLASLt" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ROCm/hipBLASLt: hipBLASLt is a library that provides general matrix-matrix operations with a flexible API and extends functionalities beyond a traditional BLAS library</a><br>
According to the latter hipBLASLt should be compatible with gfx110x cards. I have an AMD Radeon 780M (gfx1103).</p>
<p>Pytorch was installed with</p>
<pre><code class="lang-auto">pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2
</code></pre>
<p>Does anyone have an idea on how to solve this or a suggestion where to ask for help with this problem?</p>
<p>Thanks<br>
Lukas</p>

fngarrett: <p>This post has been communicated as an issue on the <a href="https://github.com/ROCm/ROCm/issues/4440" rel="noopener nofollow ugc">ROCm GitHub repository</a>.</p>
<p>Could you verify if upgrading to the latest stable PyTorch release resolves this issue?</p>
Portable parallelization over batch dimension

BeachHut: <p>I can parallelize computation over the batch dimension in my C++ extension using at::parallel_for, however this seems to require compiling with OpenMP enabled (“-fopenmp” on GCC). I am reluctant to do this as the wheel will be used by many different users, some of whom might be using a version of PyTorch compiled with a different OpenMP library, or TBB, which I worry might cause conflicts. (I had an issue in the past where I did something like this, but then PyTorch changed the OpenMP library used on Windows and my Windows users started getting a warning message about two different OpenMP libraries being loaded.)</p>
<p>Another option is to move the multithreading up into the call from Python, using multiple threads to call the C++ code for different ranges of the batch dimension. I don’t think PyTorch provides any facility for this, but I could probably use Python’s own ThreadPoolExecutor.</p>
<p>I would have imagined that this would be a common issue, so I am surprised that it is not discussed in the tutorials and that I haven’t found much about it elsewhere. Is ThreadPoolExecutor a good approach, or is there some other option, preferably a PyTorch facility, to enable portable multithreaded execution over the batch dimension in C++ extensions?</p>
How does the torch.profiler measure memory?

MirkoDeVita98: <p>Hello,</p>
<p>I have been working on profiling LLMs from HuggingFace and I have always assumed that I could trust the torch.profiler and the self_device_memory_usage metric.<br>
In order to double check the result, I started using psutils and for a small model it turns out that psutils tells me I have used 100 MBs.<br>
torch.profiler, instead, tells me 21 MBs when I sum the self_device_memory_usage of each event.<br>
What am I missing? Shouldn’t the two results more or less be the same?</p>
<p>Thank you in advance for the help!</p>
Alternative for CUDAExtension for intel GPU(xpu)

yash3056: <aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/1eba9b3aa3c43f86f4a2c807ac8e12c4a7767340/torch/utils/cpp_extension.py#L1018">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/1eba9b3aa3c43f86f4a2c807ac8e12c4a7767340/torch/utils/cpp_extension.py#L1018" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/1eba9b3aa3c43f86f4a2c807ac8e12c4a7767340/torch/utils/cpp_extension.py#L1018" target="_blank" rel="noopener nofollow ugc">torch/utils/cpp_extension.py</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/pytorch/pytorch/blob/1eba9b3aa3c43f86f4a2c807ac8e12c4a7767340/torch/utils/cpp_extension.py#L1018" rel="noopener nofollow ugc"><code>1eba9b3aa</code></a>
</div>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="1008" style="counter-reset: li-counter 1007 ;">
          <li>        libraries.append('torch_python')</li>
          <li>    if IS_WINDOWS and platform.machine().lower() != "arm64":</li>
          <li>        libraries.append("sleef")</li>
          <li></li>
          <li>    kwargs['libraries'] = libraries</li>
          <li></li>
          <li>    kwargs['language'] = 'c++'</li>
          <li>    return setuptools.Extension(name, sources, *args, **kwargs)</li>
          <li></li>
          <li></li>
          <li class="selected">def CUDAExtension(name, sources, *args, **kwargs):</li>
          <li>    """</li>
          <li>    Create a :class:`setuptools.Extension` for CUDA/C++.</li>
          <li></li>
          <li>    Convenience method that creates a :class:`setuptools.Extension` with the</li>
          <li>    bare minimum (but often sufficient) arguments to build a CUDA/C++</li>
          <li>    extension. This includes the CUDA include path, library path and runtime</li>
          <li>    library.</li>
          <li></li>
          <li>    All arguments are forwarded to the :class:`setuptools.Extension`</li>
          <li>    constructor. Full list arguments can be found at</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I am trying to utilize FlashMLA for intel GPUs, I need some help with the dependencies.</p>
Dataloader slowing down and getting stuck

thomasB: <p>Hi all,<br>
I’m using the pytorch dataloader for 3d medical imaging. The only thing that really happens in my dataset is that cropped patches from 3d volumes (saved as .npy files) are read from the local SSD. However I experienced recently that the dataloader gets stuck or slows down significantly after hours of training. I get messages like</p>
<p>kernel:[438302.215353 watchdog: BUG: soft lockup - GPU#1 stuck for 22s! [python:1397221]</p>
<p>Usually I have four python processes running for the four GPUs in our servers. If one of the processes is stopped with Crtl C the others are running again.<br>
I reduced the number of workers from 8 to 5 and experienced that in the middle of the training the time per epoch went up by a factor of 20. The Crtl C stoppped all processes at the dataloader trying to get data and showed something like</p>
<p>data = self._data_queue.get(timeout=timeout) dataloader.py, line 990 in _try_get_data<br>
self.not_empty.wait(remaining) queue.py line 180, in get<br>
gotit = waiter.acquire(True, timeout) threading.py, line 316 in wait</p>
<p>I’m using a server with four Quadro P6000 GPUs, one AMD EPYC 7352 24-Core Processor CPU and a custom version of Ubuntu 20.02 that our IT created.</p>
<p>Does anyone know what could be going on and how to fix this?</p>
<p>Many thanks!</p>

itsnamgyu: <p>I’m also having this issue. Did you find a solution?</p>
Should I start learning to use Pytorch?

fjesuslsanchez: <p>Hello,</p>
<p>I have a project in mind, and to be honest, I am not yet familiar with PyTorch. I am trying to determine if it would be a good solution for this project and whether it’s worth starting to learn it.</p>
<p>I have a set of thousands of “.step” files representing mechanical parts. My goal is to identify and count several 3D features, such as the number of holes in each part. I am wondering if it’s possible to create a model using PyTorch to “train” it, enabling it to identify the number of holes in new parts.</p>
<p>Is this feasible with PyTorch?</p>
<p>Thank you in advance for your insights.</p>

Naming-isDifficult: <p>For your case, I would not recommend you to learn PyTorch unless you are familiar with machine learning.</p>
<blockquote>
<p>Is this feasible with PyTorch?</p>
</blockquote>
<p>I would say most likely yes. However, learning PyTorch itself would not give you the ability to create a proper model. PyTorch only helps you to implement your model, but you would possibly need to learn how machine learning works to design a proper model for your goal.</p>
<p>You might want to do some searching to find an open source model or something else instead… Maybe this link: <a href="https://stackoverflow.com/questions/38896066/counting-holes-in-a-mesh" rel="noopener nofollow ugc">python - Counting Holes in A Mesh - Stack Overflow</a>.</p>
Missing Torch Version

Michael_Sh: <p>Hi all,</p>
<p>First of all, I want to thank you for the amazing library!</p>
<p>I’m currently trying to find a binary version that meets the following specifications:</p>
<pre><code>torch==2.5.1
CUDA version: 12.4
Architecture: ARM 7.5 for Graviton2
</code></pre>
<p>Could you please guide me on how to obtain or build this?</p>
<p>Thank you in advance!</p>

Michael_Sh: <p><a class="mention" href="/u/malfet">@malfet</a> I’ve opened an issue here. Could you please guide me to the appropriate place to ask it?"</p>
<p>Feel free to adjust further if needed!</p>
<p>Source: <a href="https://github.com/pytorch/pytorch/issues/147953" rel="noopener nofollow ugc">Request for Binary Version of torch==2.5.1 with CUDA 12.4 for ARM 7.5 (Graviton2) · Issue #147953 · pytorch/pytorch · GitHub</a></p>
What is the difference between flex_attention.create_mask and flex_attention.create_block_mask

shadowshadow: <p>I want to use flex_attention and need to use custom mask. I find there are three create_mask function in flex_attention.</p>
<ul>
<li>create_mask</li>
<li>create_block_mask</li>
<li>create_nested_block_mask</li>
</ul>
<p>I wonder what’s the difference between <code>create_mask</code> and <code>create_block_mask</code>, especially what does block mean?</p>
How can I implement a chunked dataset in Torch?

s5j3: <p>I want to efficiently load a dataset that has many samples and is larger then my system’s memory.</p>
<p>TensorFlow has TFRecords which saves many samples into a single TFRecord. TensorFlow’s data pipeline reads in the files, shuffles the data in each TFRecord and then passes it to the network. Since many samples are combined into a single file, the data pipeline spends less time loading individual files.</p>
<p>Is there an equivalent in Torch?</p>
<p>I have seen some people implement TFRecord readers for Torch. However, I would think there are other ways to accomplish the same capabilities in Torch. I think I could implement something    similar using an iterable dataset but I feel that such capabilities should already exist.</p>
Shared memory between multiple nodes pytorch

External_happy: <p>Hi, I try to create a memory bank to store image features along with their labels. I am utilizing 2 nodes, each equipped with 4 GPUs. I aim for this memory bank to be shared among all processes and be used for updates during training. I have this code snippet.</p>
<pre><code class="lang-auto">@torch.no_grad()
def eval_build_bank(model, data_loader, len_dataset, device, world_size):
    features = torch.zeros(len_dataset, model.model.visual.classifier.shape[0]).to(device)
    labels   = torch.zeros(len_dataset, dtype=torch.long).to(device)
    for _, batch in enumerate(data_loader):
        inputs, _, idx = batch
        image = inputs[0].to(device)
        with torch.no_grad():
            logit, feats = model(image)
        features[idx] = feats.detach()
        proba = torch.softmax(logit, dim=-1)
        pseudo_targets = torch.argmax(proba, dim=-1)
        labels[idx] = pseudo_targets.detach()
    dist.all_reduce(features, op=dist.ReduceOp.SUM)
    dist.all_reduce(labels, op=dist.ReduceOp.SUM)
    bank = {
        "features" : features,
        "labels" : labels
    } 
    return bank
</code></pre>
<p>Is this correct?</p>

AKT_TARAFDER: <p><a class="mention" href="/u/external_happy">@External_happy</a> Hello,<br>
I kind of need the same thing. Lets assume I need a shared memory of [row, col]<br>
how to do it?<br>
I need 1 pair for each node(each node with 2 gpu)<br>
another for 2 nodes.</p>
nn.MultiheadAttention output

saimandahal: <p>multihead_attn = nn.MultiheadAttention(embed_dim, num_heads)<br>
attn_output, attn_output_weights = multihead_attn(query, key, value)</p>
<p>'attn_output_weights` only contains the softmax-normalized attention scores. Is this true?<br>
Or it is the final output after multiplication with the value?</p>
Training accuracy significantly decreases and doesn't go back up when loading from a checkpoint

EaswarGn: <p>I am training an encoder only transformer model that plays chess like a human, with two prediction heads that find logits over the board encoder. This is how I am saving the checkpoint:</p>
<pre><code class="lang-auto">state = {
        "step": step,
        "model_state_dict": model.state_dict(),
        "optimizer_state_dict": optimizer.state_dict(),
    }
    checkpoint_folder = f'{config_name}/{checkpoint_folder}'
    pathlib.Path(checkpoint_folder).mkdir(parents=True, exist_ok=True)
    filename = rating + "_step_" + step + ".pt"
    torch.save(state, os.path.join(checkpoint_folder, filename))
</code></pre>
<p>and I load the checkpoint like this:</p>
<pre><code class="lang-auto">checkpoint = torch.load(
            CONFIG.CHECKPOINT_PATH,
            weights_only=True,
        )
        
        step = checkpoint['step']
        step = int(step)
        start_epoch = step//CONFIG.STEPS_PER_EPOCH + 1
        
        state_dict = checkpoint['model_state_dict']
        new_state_dict = {}
        for key, value in state_dict.items():
            new_key = key.replace('_orig_mod.', '')  # remove the '_orig_mod' prefix
            new_key = new_key.replace('module.', '')
            new_state_dict[new_key] = value
        model.load_state_dict(new_state_dict, strict=CONFIG.USE_STRICT)
        optimizer.load_state_dict(checkpoint["optimizer_state_dict"])
</code></pre>
<p>I tried validating the model after loading the checkpoint, and the accuracy is fine. But when I resume training for the model with model.train(), the accuracy drops a good 5-7% and it’s on the same dataset, the accuracy doesn’t go back up to what it was before pausing training either. After letting this checkpointed model train for some 500 steps, the validation accuracy decreases from where it started, which means that there is definitely something wrong with the model weights. The learning rate is also saved and loaded. I’ve looked at a few other issues related to this, and most people fixed it by finding a difference in their dataloader for training and validation, but nothing like this exists from my case.  Could someone please help me with this, I’ve been stuck on this for a few days now and I’m pretty desperate for answers.</p>

ptrblck: <p>Could you explain why you are manipulating the <code>state_dict</code> before loading it?<br>
Also, do you load it with the <code>strict</code> option or not? While <code>strict=False</code> has valid use cases you should not use it in case real errors were raised due to a mismatching key etc.</p>

EaswarGn: <p>In the code that I sent above, strict=True, and I am manipulating the state dict because I was saving a model that was run with<br>
torch.compile, and before I was testing it out with ddp, so I was stripping the prefix that pytorch was appending to each key. That shouldn’t make a difference to the state dict right?</p>
Loading and saving a checkpoint changes it?

amagibaba: <p>I’m doing some research on local minima and I’m trying to train a simple auto-encoder with only 1 weight matrix (W) and 1 bias (b) on a toy dataset, and found some very very peculiar behavior. At some epoch, the encoder is almost perfectly at a local optimum - I save this into a checkpoint. When I try to reload my checkpoint and continue training it, like so:</p>
<pre><code class="lang-auto">model = ToyModel(input_dim=7, hidden_dim=2, bias=True)
chkpt = torch.load(pth_file)
model.load_state_dict(chkpt['model_state_dict'])
</code></pre>
<p>The model continues training just fine, in that I see the weights updating. However, when I add in a simple load and save, like so:</p>
<pre><code class="lang-auto">model = ToyModel(input_dim=7, hidden_dim=2, bias=True)

# Load 1
chkpt = torch.load(pth_file)
model.load_state_dict(chkpt['model_state_dict'])

# Save as is
torch.save({
    'epoch': -1,
    'model_state_dict': model.state_dict(),
    'optimizer_state_dict': SOME_SGD_OPTIMIZER,
    'scheduler_state_dict': SOME_CONSTANT_LR_SCHEDULER,
    'train_loss': 0,
    'test_loss': 0,
}, save_loc)

# Load 2
chkpt = torch.load(save_loc)
model.load_state_dict(chkpt['model_state_dict'])
</code></pre>
<p>The model does not continue training just fine, in that the weights do not update anymore. This implies that the weights in the original <code>chkpt</code> are different from the weights in <code>save_loc</code>. What’s going on here?</p>
<ul>
<li>I don’t think it has to do with whether gradients are required (i.e. not a problem with calling <code>.detach()</code>, or turning <code>requires_grad</code> on or off, etc.), because when I try to modify <code>model.W</code> between the <code># Load 1</code> and <code>Save as is</code> steps, the training continuation proceeds fine, so it’s not a case of the code turning off gradient computation.</li>
<li>I think it has to do with the loss of some precision when calling <code>model.state_dict()</code> or <code>torch.save()</code> (or maybe even <code>model.load_state_dict()</code>), but I can’t seem to find any leads on what this is.</li>
</ul>
<p>Help much appreciated!!</p>

amagibaba: <p>Woops, I found the problem was due to my newly created scheduler being incompatible with the optimizer, and somewhere there was a <code>lr</code> key in the scheduler that was 0.0 and/or there was aggressive weight decay and I didn’t provide code to illustrate that.</p>
<p>Not a pytorch problem, case closed</p>
How to convert .mp3 file to .PTH file

erkjghzebfv: <p>Someone can help me please I need to convert an .mp3 file to a .PTH file but don’t know how to do it</p>

ptrblck: <p>Could you describe your use case in more details, please?<br>
Do you want to store the decoded waveforms from the <code>mp3</code> file as a tensor?<br>
If so, you could use <code>torchaudio</code> to read the data, transform it to a tensor, and store it via <code>torch.save</code>.</p>

Victor_Gitu: <p>Kits dot ai requires you to upload the file in .pht format, so I think that’s why he wants the file in .pht. Additionally, Mobile phone recorded audio that you are trying to use for training is in .dat that can be converted to .mp3. The issue now is converting from MP3 to .pht.</p>

Shahirul_Aiman: <p>Hey sorry for late reply. u can use this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><a href="https://colab.research.google.com/drive/1hmKPNeeReO4NHzOktJFMI2plYKy5F7ZI?usp=sharing#scrollTo=CuEvrNVj9ruN" rel="noopener nofollow ugc">BahaaAI - RVC Model Training &amp; Inference- 19 January 2025 - Colab</a></p>
Using OpenAI image models to extract encondings

chrathans: <p>I am using code that can be found in this <a href="https://github.com/NielsRogge/Transformers-Tutorials/blob/master/ImageGPT/Linear_probing_with_ImageGPT.ipynb" rel="noopener nofollow ugc">link</a> that loads an image model and extracts encodings to perform some linear probing of my model. It makes use of <code>cifar10</code> and <code>"openai/imagegpt-small"</code>. The code I am using is the following:</p>
<pre><code class="lang-auto">    train_ds, test_ds = load_dataset('cifar10', split=['train[:10]', 'test[:10]'])
    # split up training into training + validation
    splits = train_ds.train_test_split(test_size=0.1)
    train_ds = splits['train']
    val_ds = splits['test']

    dataset = load_dataset('cifar10')
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    feature_extractor = ImageGPTFeatureExtractor.from_pretrained("openai/imagegpt-small")
    model = ImageGPTModel.from_pretrained("openai/imagegpt-small")
    model.to(device)

    url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
    image = Image.open(requests.get(url, stream=True).raw)

    encoding = feature_extractor(image, return_tensors="pt")
    pixel_values = encoding.pixel_values.to(device)
</code></pre>
<p>I am facing a problem in the last line of my code, where the attribute pixel_values is not recognized and I am getting an error: <code>raise AttributeError</code>.</p>
<p>I am firstly, trying to understand what this <code>pixel_values</code> does in general and how to solve this issue.</p>

foti: <p>Can you check if you are getting the encodings from calling feature_extractor. It could be the case that the method returns None and the pixel_values attribute is not found.</p>
Torch._check failing with torch compile

Ch0ronomato: <p>This is an example</p>
<pre data-code-wrap="py"><code class="lang-py">def f(x):
    torch._check(x &gt; 0)
    return torch.log(x)

torch.compile(f)(torch.rand(1))
</code></pre>
<p>Returns the following error:</p>
<pre data-code-wrap="txt"><code class="lang-txt">TorchRuntimeError: Failed running call_function &lt;function _check at 0x7f7cf0322fc0&gt;(*(FakeTensor(..., size=(1,), dtype=torch.bool),), **{}):
cond must be a bool, but got &lt;class 'torch._subclasses.fake_tensor.FakeTensor'&gt;

from user code:
   File "/tmp/ipykernel_1454634/3269954531.py", line 2, in f
    torch._check(x &gt; 0)

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
</code></pre>

BeachHut: <p><code>torch._check(bool(x &gt; 0))</code> and <code>torch._check_tensor_all(x &gt; 0)</code> seem to work.</p>
Imbalance in training data

Parthiv_Narayan: <p>Hellow everyone,<br>
I am a very new user trying to implement a deep learning UNet model for forest disturbance dectection using remote sensing data. As a new commer I am just trying to implement the generic model. I have a 4 band tif image. I have created a binary mask manually. Then I have created 128*128pixel patches. But the problem is that there is very little representation of the disturbed areas.<br>
So my model is unable to identify the disturbed areas.<br>
Please help me to rectify my issue.</p>

manan: <p>you can limit the the amount of all black mask created. for eg make it only 40-50% of the mask with the disturbed area.<br>
Also for the white part, try dividing mask into patches with some overlap, it will increase the dataset. There’s augmentation too, which can add variation in your disturbed area patches.</p>

J_Johnson: <p>What you’re doing is a semantic segmentation task. And it’s important to understand that this is effectively a classification task.</p>
<p>With that out of the way, we just need to calculate weights to pass into the cross_entropy_loss function. This post addresses how to do that with code:</p>
<aside class="quote quote-modified" data-post="2" data-topic="194579">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-do-i-get-the-accuracy-of-an-unbalanced-dataset-or-segmentation-task/194579/2">How do I get the accuracy of an unbalanced dataset or segmentation task?</a> <a class="badge-category__wrapper " href="/c/faq/30"><span data-category-id="30" style="--category-badge-color: #0088CC; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="The FAQ category contains commonly-asked questions and their answers. Please refer to this section before you post your query."><span class="badge-category__name">FAQ</span></span></a>
  </div>
  <blockquote>
    Here is a function for calculating the weight vector used by CrossEntropyLoss based on a set of targets with integers. 
def scaled_weights(targets):
    '''
    Gets class weights
    :param targets: pass in the matrix of size (batch, classes) containing correct targets or
        labels as one_hot
    :return: returns a vector for weight rescaling for use in cross entropy loss function
    '''

    N, C = targets.size(0), targets.size(1)

    # get vector of total for each class
    totals = ta…
  </blockquote>
</aside>
Which cuda version to choose when installing pytorch

davidhkw: <p>Hi, I am new to using pytorch and I have a question about installation.</p>
<p>In the guide, I have to choose the Cuda version I want to install (Compute Platform) where I can choose CUDA 11.8, CUDA12.4 or CUDA 12.6. As I check my cuda version, I have CUDA 12.7. Which one should I choose when installing the pytorch library?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d0bf3abd863008bdaa1021a7f7ea0c5e97304aa.png" data-download-href="https://discuss.pytorch.org/uploads/default/8d0bf3abd863008bdaa1021a7f7ea0c5e97304aa" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d0bf3abd863008bdaa1021a7f7ea0c5e97304aa.png" alt="image" data-base62-sha1="k7KVF3e270eALiwzMz7BuPcujF0" width="690" height="277" data-dominant-color="1B1B1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1734×698 18.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks so much.</p>

ptrblck: <p>PyTorch binaries ship with their own CUDA runtime dependencies and you would only need to install an NVIDIA driver. Since your driver is new enough you can install any PyTorch binary and I would recommend sticking with the latest, i.e. CUDA 12.6.3 unless you have a Blackwell GPU which requires CUDA 12.8.</p>

pzlong: <p>Hi ptrblck, I was excited to see the new CUDA 12.8 support in the nightly builds — I’ve waited too long for this because my Blackwell GPU can’t work without cu128.</p>
<p>When I installed it with:<br>
pip3 install --pre torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/nightly/cu128" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu128</a></p>
<p>it returned:</p>
<p>ERROR: Could not find a version that satisfies the requirement torchaudio (from versions: none)</p>
<p>Any idea why this happens? Thanks for your help!</p>

ptrblck: <p>Remove <code>torchaudio</code> from your install command for now as the build or upload seems to fail.</p>

pzlong: <p>Okay, I’ll try again. Thank you.</p>

ou17770103456: <p>I also encountered similar problems and look forward to pythorch being released as soon as possible</p>
Finding issue for high memory bandwidth between CPU and GPU

MLangner: <p>I have implemented some models for generating text with LSTM layers etc., mostly according<br>
to the typical encoder-decoder scheme and autoregressive training. Now, training one of  the more complex models produced the issue of relatively constant GPU VRAM usage of about 30 percent, but the bandwidth of data being copied from CPU to GPU or backwards is at about 40GB/s.</p>
<p>In the training loop, I copy the batches to the GPU and do the computations (forward step etc.).<br>
All layers and tensors are initialized directly on the GPU.<br>
The result is a horribly slow training, and I am unsure where this overhead of copying comes from, because I do not use a single explicit copying from GPU to CPU in the code, nor do I get errors that tell me that not all tensors are on the same device.</p>
<p>Does anyone have any idea how to identify the issue and / or what the reason may be?</p>
<p>Thank you in advance</p>

ptrblck: <p>I don’t understand your issue entirely. It seems you are describing a slow host2device copy, which I assume you are able to reproduce with a standalone CUDA code. If so, you should check what your system specs expect.<br>
Later you describe you are copying the data to the GPU only and train the model. Assuming the data is not huge even a slow host2device copy should not slow down the entire training unless your model is tiny.</p>

MLangner: <p>The issue was, that a function call copied once per forward step all tensors etc. from the GPU to the CPU and back again, which led to a bandwidth usage of 40 GB/s between GPU and CPU. THis slowed down the training process. Effectively, I overlooked that I used “any” instead of “torch.any”, the former being executed on the CPU, the latter on GPU. Now its fixed and training is lightening fast. Is there a method to alert users with e.g. a warning when between pytorch operations, one is using a pythonic function that copies back to CPU? I just did not find the error due to the fact that I wasn’t aware of the copying process that happened under the hood.</p>

ptrblck: <p>I don’t know if the copy itself is slow but since the op is synchronizing you would indeed expect a performance hit. You could use <a href="https://pytorch.org/docs/stable/generated/torch.cuda.set_sync_debug_mode.html#torch-cuda-set-sync-debug-mode"><code>torch.cuda.set_sync_debug_mode</code></a> to raise a warning or error on synchronizations.</p>
AWS NVIDIA Driver Issue with PyTorch 2.2.2 and 2.4.1

vitaminc: <p>I have a repo that was using torch 2.2.2 (CUDA 12.1) hitting a Sagemaker inference endpoint with instance type ml.g4dn.xlarge. NVIDIA SMI tells me this is a Tesla T4 running CUDA 12.4. Cloudwatch is giving me:</p>
<pre><code class="lang-auto">`venv/lib/python3.10/site-packages/torch/cuda/__init__.py:141: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 11040). Please update your GPU driver by downloading and installing a new version from the URL: ` `http://www.nvidia.com/Download/index.aspx` ` Alternatively, go to: ` `https://pytorch.org` ` to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)`
`  return torch._C._cuda_getDeviceCount() &gt; 0`
</code></pre>
<p>I updated the repo’s torch version to 2.4.1+cu124 but am getting the same Cloudwatch message.</p>
<p>Is there something I’m missing?</p>

ptrblck: <p>Your NVIDIA driver is too old so either install a PyTorch binary with CUDA 11 runtime dependencies or update your driver.</p>
Having trouble getting the mps version of torch to install

voxvox142: <p>I’m on Sonoma 14.7.1, using conda. A few weeks ago, I followed this guide: <a href="https://developer.apple.com/metal/pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Accelerated PyTorch training on Mac - Metal - Apple Developer</a></p>
<p>I used the conda installation method: <code>conda install pytorch torchvision torchaudio -c pytorch-nightly</code>, at least as far as i remember, and according to conda’s list, installed: <code>pytorch=2.3.1=gpu_mps_py310h7b7e308_100</code></p>
<p>Then, yesterday, I broke my installation when installing a requirements.txt file of a codebase i was working with. Stupid, i know, but then I tried to recreate the conda env from scratch. After doing the above, my installed pytorch shows:<br>
<code>pytorch=2.6.0.dev20241112=py3.10_0</code></p>
<p>I have ended up in multi day troubleshooting holes because of conda and pip issues with torch, so i’m hoping someone may be able to clearly explain why this even happened? Specifically, how did i get the mps-specific torch installed before, and why isn’t it installing now?</p>

roach: <p>Hello, i installed it from this fork and it “kinda” works. <a href="https://github.com/LalithShiyam/pytorch-mps" class="inline-onebox" rel="noopener nofollow ugc">GitHub - LalithShiyam/pytorch-mps: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a>, the version is 2.3.0 (torch base version). There are a LOT of things that are not supported in torchmps right now, like transpose convolutions etc etc. This package seems to have various implementations, however there are things that it will crash hard.</p>
<p>tl;dr pip3 install git+https://github.com/LalithShiyam/pytorch-mps.git</p>
TorchRL cpu-only installation

Peter78: <p>Dear all,<br>
I was wondering if installing TorchRL for CPU-only usage is possible.</p>
<p>Regards,</p>

ptrblck: <p>I don’t think <code>torchrl</code> ships with custom CUDA kernels (please correct me if I’m wrong) and depends on the PyTorch installation for its execution. So yes, install the CPU-only PyTorch binary, <code>torchrl</code> afterwards, and it should work.</p>

vmoens: <p>As <a class="mention" href="/u/ptrblck">@ptrblck</a> mentioned, nothing in torchrl requires you from having a GPU. You can confidently use it on cpu only.</p>

Peter78: <p>Hi everyone,</p>
<p>Many thanks for all the replies.</p>
<p>I was wondering if anyone could provide me with a simple “Hello World” example using TorchRL to help me get started.</p>
<p>Best regards,<br>
P.</p>

vmoens: <p>Check out the getting started!<br>
<a href="https://pytorch.org/rl/stable/tutorials/getting-started-0.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/rl/stable/tutorials/getting-started-0.html</a></p>
How can I use an iterable dataset and dataloader for ML training?

saff: <p>Hi everyone,<br>
I have a huge dataset (we’re talking about trillions and trillions of samples here). The samples are read from multiple files and are then streamed into the dataloaders. For my implementation, I’ve used an iterable dataset then called the dataloader.</p>
<p>From my understanding, if you use an iterable dataset then that means you need to take care of the shuffling? And we need to shuffle the data in each epoch to do so, to avoid overfitting.</p>
<p>I’m confused about how should I initalize the dataset and dataloader before training. Do I make a call to the dataset and dataloader in each epoch? How does the sequence go? Also do I need an dataset shuffler to shuffle the data in that case?</p>

ptrblck: <p>Could you describe why you are using an iterable dataset instead of a map-style one?</p>

saff: <p>From my research (could be wrong) if the data doesn’t fit in memory, then you use iterable-style dataset</p>

ptrblck: <p>Not necessarily, since the standard approach is to lazily load the data anyway. I.e. in a map-style dataset you would store e.g. the paths to all images (without loading the actual image or data sample) and would then load it via the <code>index</code> in the <code>__getitem__</code>.</p>

saff: <p>So does that mean the data is not loaded into memory at once? Also, if we use the map-style dataset we dont have to create a custom shuffler?</p>

ptrblck: <p>Yes, if you use the lazy loading approach described before no data will be preloaded and only the needed sample for the current batch will be loaded into memory (times the batch size, number of workers, and prefetch factor but these are details).<br>
Yes, you won’t need to apply any custom shuffling since the <code>DataLoader</code> will shuffle the indices if <code>shuffle=True</code>.</p>

saff: <p>Amazing, thanks for the help! Does that mean that I initialize the dataset and the data loader once before looping through the number of epochs?</p>

ptrblck: <p>Yes, check the ImageNet example in the PyTorch repository for an example.</p>
Modifying an instance segmentation model to incorporate images and measurement data

mycul: <p>Hi,</p>
<p>I’m wondering if anyone has any experience with modifying the RPN section of a Mask RCNN model.</p>
<p>I’m building a model meant to identify certain items that don’t fit inside of a certain grid that is viewable within an image and have noticed it doesn’t take that info into account as much as i’d hope.</p>
<p>I have the measurements of the grid and the pixel locations (in other words, I could paint it on), I’m wondering if it might be more beneficial to just push this information into the bbox regressor within the RPN considering it is such vital information to the classification of the items.</p>
<p>My current issue is that the instance segmentation models are great at recognizing the pattern of the classes in the image, but accidentally classify things in areas of the image where it would make the result a FP.</p>
<p>Considering how vital those grid measurements are to the classification of the items, it seems to me that it would be beneficial, but I’m curious to if anyone has tried something similar to this or has any recommendations.</p>
<p>NOTE: I cannot share the data at the moment and there are not many annotated images (around 300, they’re tedious to annotate, but I understand that this is definitely an issue). The model I would be modifying would be the v2 version of the resnet50-Maskrcnn from pytorch.</p>
Where to Post for Tensor Subclass Support in SKA?

BouarfaMahi: <p>I am developing <strong>Structured Knowledge Accumulation (SKA)</strong>, a new framework for entropy-driven learning without backpropagation. It requires a <strong>custom tensor subclass</strong> in PyTorch to handle <strong>knowledge accumulation, decision probabilities, and entropy-based updates.</strong></p>
<p>Where is the best place to post for <strong>support on implementing a PyTorch tensor subclass</strong> for this type of learning?<br>
Thanks in advance for any guidance!</p>
How do you think about fp4, fp3?

eastsky: <p>I think it would be better to use if pytorch supports fp4(e3m0, e2m1,…), fp3(e1m1, e2m0,…) some types like that.<br>
forward and backward, several caculation supports while we using… like fp32 or bf16. It will be far better to use when users need PTQ or QAT or making small models.<br>
So I wonder if pytorch has plan to make fp4 and fp3 data type officially in pytorch.<br>
I think 4 bit float and 3 bit float is needed and fp8 does not seem to support ‘any kinds of calculation in any cpu or gpu device’. ('un’limited support)<br>
-<code>torch.float8_e4m3fn</code> and <code>torch.float8_e5m2</code> implement the spec for 8-bit floating point types from <a href="https://arxiv.org/abs/2209.05433" class="inline-onebox" rel="noopener nofollow ugc">[2209.05433] FP8 Formats for Deep Learning</a>. The op support is very limited.<br>
I think many users want a new update about that…<br>
Of course there will be someone who needs smaller…<br>
And I think they (smaller float types) would be soon used widely…<br>
As I’ve been watching someone who made some small data type code and telling to some company like nvidia…</p>
<p>I want pytorch provides unlimited support on ‘officially optimized’ reasonable small float point data types.</p>
<p>Not so excellent in english.</p>
<p>Thanks and best regards…</p>

ptrblck: <p>I think you misunderstood the question.</p>
Optimizing Tensor Multiplication in PyTorch Without Excessive Memory Usage

Alireza_Alireza: <p>I have two tensors, A and B, with the following dimensions:</p>
<ul>
<li>A.shape: [1024, 1, 1024, 1, 2048]</li>
<li>B.shape: [1024, 1024, 1, 2048, 1]</li>
</ul>
<p>I want to compute the matrix multiplication of these two tensors using <code>A @ B</code> in PyTorch. The output is correct, but there is an issue with memory consumption. When this computation is performed in PyTorch, it tries to first broadcast the two tensors to the same dimensions. For example, it creates an intermediate tensor from A with dimensions [1024, 1024, 1024, 1, 2048], repeating the second dimension 1024 times. It does the same for B, resulting in dimensions [1024, 1024, 1024, 2048, 1]. These intermediate tensors are very large.</p>
<p>Is there a more efficient way to perform the multiplication without such broadcasting? Are there any other libraries that can handle this multiplication better than PyTorch?</p>

Soumya_Kundu: <p>Would einsum work here?</p>

Alireza_Alireza: <p>I tried torch.einsum. But output was not correct. also speed was not fast.</p>
Optimizer_state_dict with multiple optimizers in FSDP

andreasst: <p>Hi,</p>
<p>I am using PyTorch 2.5.1 with FSDP and 2 optimizers that each handle a separate set of parameters.</p>
<p>Training runs fine, but I am running into errors when trying to get the full_state_dict during checkpointing. I am calling</p>
<pre><code class="lang-auto">get_state_dict(model, [optimizer1, optimizer2], options=StateDictOptions(full_state_dict=True))
</code></pre>
<p>and getting <code>RuntimeError: some.weight is not in the optimizer state</code> about optimizer1 about some weight that is handled by optimizer2.</p>
<p>If I change the call to pass <code>[optimizer2, optimizer1]</code> I get errors about optimizer2 not knowing about a weight that is handled by optimizer1.</p>
<p>How do I get around this?</p>
<p>Thanks.</p>
How to manually replicate module.to(dtype=dtype)

jackzhxng: <p>How can I manually replicate dtype-converting an <code>nn.module</code>?</p>
<p>Would the following be enough, or are there other items that need to be converted as well?</p>
<pre><code class="lang-auto">dtype = torch.float32
for buf in model.buffers():
        buf.data = buf.data.to(dtype=dtype)
for param in model.parameters():
        param.data = param.data.to(dtype=dtype)
</code></pre>
_ivalue_ INTERNAL ASSERT FAILED at "../torch/csrc/jit/api/object.h":37, please report a bug to PyTorch

Brendan_Lydon: <p>Hi,</p>
<p>I am trying to load a trained python, pytorch “.pt” file into a C++ application. I have had no problem in the past doing this with a similar code block:</p>
<blockquote>
<pre><code>try{
    m_model.eval();
    torch::Tensor output = m_model.forward({xTensor}).toTensor();
    torch::Tensor rndOutput = output.round().to(torch::kInt);
    predCls = rndOutput.item&lt;int&gt;();

} catch(std::exception &amp;e) {
    std::string errMsg = e.what();
    std::cout &lt;&lt; errMsg &lt;&lt; std::endl;
}
</code></pre>
</blockquote>
<p>I get this error at the m_model.eval() line…</p>
<p>The entire e.what() msg is:</p>
<blockquote>
<p><em>ivalue</em> INTERNAL ASSERT FAILED at “…/torch/csrc/jit/api/object.h”:37, please report a bug to PyTorch.<br>
Exception raised from _ivalue at …/torch/csrc/jit/api/object.h:37 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;) + 0x6b (0x7ffff7f7005b in /home/brendan/libtorch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span> c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0xc9 (0x7ffff7f6b0e9 in /home/brendan/libtorch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#2:</span>  + 0x45c23fb (0x7fffe4b9e3fb in /home/brendan/libtorch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#3:</span> torch::jit::Module::train(bool) + 0x6da (0x7fffe4b98d3a in /home/brendan/libtorch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#4:</span>  + 0xac559 (0x555555600559 in /home/brendan/Repos/prodml/build/main)<br>
frame <span class="hashtag-raw">#5:</span>  + 0xa9dd0 (0x5555555fddd0 in /home/brendan/Repos/prodml/build/main)<br>
frame <span class="hashtag-raw">#6:</span>  + 0xddd71 (0x555555631d71 in /home/brendan/Repos/prodml/build/main)<br>
frame <span class="hashtag-raw">#7:</span>  + 0xda105 (0x55555562e105 in /home/brendan/Repos/prodml/build/main)<br>
frame <span class="hashtag-raw">#8:</span>  + 0xdc36f (0x55555563036f in /home/brendan/Repos/prodml/build/main)<br>
frame <span class="hashtag-raw">#9:</span>  + 0x9dd73 (0x5555555f1d73 in /home/brendan/Repos/prodml/build/main)<br>
frame <span class="hashtag-raw">#10:</span>  + 0x29d90 (0x7fffdfe29d90 in /lib/x86_64-linux-gnu/libc.so.6)<br>
frame <span class="hashtag-raw">#11:</span> __libc_start_main + 0x80 (0x7fffdfe29e40 in /lib/x86_64-linux-gnu/libc.so.6)<br>
frame <span class="hashtag-raw">#12:</span>  + 0x13aa5 (0x555555567aa5 in /home/brendan/Repos/prodml/build/main)</p>
</blockquote>
<p>Any help would be appreciated, thank you.</p>

tianlanqqcom: <p>I got same error. The C++ code throws c10:Error when calling eval(). Have you solved the problem?</p>