akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
When using worker_init_fn, dataset length breaks

maxfreu: <p>Hi!<br>
Setting: My dataset wraps a large pandas dataframe. The dataset is replicated for each dataloader worker, filling my ram, limiting the workers i can use. So I thought to split it via worker_init_fn.</p>
<p>Problem: When I initialize the dataset the way I do, I get index errors.</p>
<p>Sample code showing how I do things:</p>
<pre><code class="lang-python">#%%
import torch
import numpy as np
from time import time, sleep
from torch.utils.data import DataLoader, Dataset
# %%

class Data(Dataset):
    def __init__(self) -&gt; None:
        super().__init__()
        self.indices = np.arange(1000)

    def __getitem__(self, index):
        return self.indices[index]
    
    def __len__(self):
        return len(self.indices)
    
    @staticmethod
    def worker_init_fn(worker_id):
        worker_info = torch.utils.data.get_worker_info()
        print(f"setting up worker {worker_id}")
        dataset = worker_info.dataset
        all_ids = dataset.indices
        subset = np.array_split(all_ids, worker_info.num_workers)[worker_id]
        print(f"worker {worker_id} subset size: {len(subset)} / {len(all_ids)}\n")
        dataset.indices = subset
    
loader = DataLoader(Data(), num_workers=4, worker_init_fn=Data.worker_init_fn)
# %%
t0 = time()
s = 0
for i in loader:
    print(i)
    s += i
    sleep(0.01)
print(time()-t0)

&gt; IndexError: index 250 is out of bounds for axis 0 with size 250
</code></pre>
<p>What am I doing wrong?</p>

tom: <p>I don’t think you’re supposed to change the length of the dataset at that point. For indexed datasets, the indices that each worker gets will be global and drawn from the sampler (which by default does not know about the split).<br>
I think the conceptually easiest thing is to use for splitting data is to use an <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset" rel="noopener nofollow ugc">IterableDatasset</a> which will have each worker return data until it is exhausted, meaning that you sidestep the length. The link above has an example along the lines of your use case.</p>
<p>Best regards</p>
<p>Thomas</p>

maxfreu: <p>Thanks, that avoids the problem.</p>
Different results of Griffin-Lim using torchaudio

ZhangMingxin1997: <p>I’m trying to transform the spectrogram back to the audio. First I used <code>librosa.griffinlim</code> and it worked well, but it was time-consuming. Therefore I am trying to use torchaudio on GPU to boost the transformation. However I obtained different reconstruction results compared to the librosa.</p>
<p>This is my code:</p>
<pre><code class="lang-auto"># Preprocess
data, fs = librosa.load('waveform.wav', sr=44100)
b, a = signal.butter(3, [20 / fs, 1000 / fs], 'bandpass')
data = signal.filtfilt(b, a, data)
plt.plot(data)

# STFT
DMatrix = librosa.stft(data, n_fft=2048, hop_length=int(2048 * 0.1), window='hann')
dbMatrix = librosa.amplitude_to_db(np.abs(DMatrix), ref=np.max)
</code></pre>
<p>And I obtained results similar to the original waveform using librosa:</p>
<pre><code class="lang-auto">spec = librosa.db_to_amplitude(dbMatrix)
re_wav = librosa.griffinlim(spec, n_iter=100, n_fft=2048, hop_length=int(2048 * 0.1), window='hann')
plt.plot(re_wav)
</code></pre>
<p>But when I changed to torchaudio, the result is different.</p>
<pre><code class="lang-auto">griffinlim = torchaudio.transforms.GriffinLim(n_fft=2048, n_iter=100, hop_length=int(2048 * 0.1)).to('cuda')
spec = librosa.db_to_amplitude(dbMatrix)
re_wav = griffinlim(torch.tensor(spec).to('cuda'))
plt.plot(re_wav.cpu().detach().numpy())
</code></pre>
<p>What am I missing?</p>

ZhangMingxin1997: <p><a href="https://stackoverflow.com/questions/77479348/different-results-of-griffin-lim-from-librosa-and-torchaudio" rel="noopener nofollow ugc">https://stackoverflow.com/questions/77479348/different-results-of-griffin-lim-from-librosa-and-torchaudio</a></p>
<p>Setting power=1 works.</p>
Reagrding compatibilty of cudaV 9.1.85 when running code on multiple gpu's

Akash_Negi: <p>I am getting nccl error while running my code on multiple gpu’s.so i want to install pytorch version compatible with cudaWhen i was running on singlen gpu everything was working fine but on multiple gpu’s it is throwing error so can anyone tell me how to rsolve it. I have pytorch 1.10.0</p>

ptrblck: <p>CUDA 9 is quite old by now and not supported anymore in current PyTorch builds. However, note that the PyTorch binaries ship with their own CUDA dependencies and will not use your locally installed CUDA toolkit unless you build PyTorch from source or custom CUDA extensions.<br>
I thus guess you mean the NVIDIA driver?<br>
Could you update PyTorch to the latest stable or nightly release and check if you would still see these errors?</p>
Dataloader problem in pyinstaller executable file

DonghunP: <p>Has anyone ever built a PyTorch inference file with pyinstaller?</p>
<p>When iterating torch.utils.data.dataloader using enumerate, there was no problem when executing the Python file directly,</p>
<p>but if I build it as an executable file and run it, dataloader part is not executed and the executable file is restarted.</p>
<p>Hard to debug:(</p>
<p>Does anyone know a solution or has similar experience?</p>
<p>Thank ahead:)</p>

Nazar_Pankov: <p>Hi<br>
I have the same problem<br>
Have u fixed this issue?</p>

DonghunP: <p><a class="mention" href="/u/nazar_pankov">@Nazar_Pankov</a> you can refer to this page<br>
<a href="https://www.reddit.com/r/learnpython/comments/o0ln04/why_is_parallel_processing_not_working_with/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://www.reddit.com/r/learnpython/comments/o0ln04/why_is_parallel_processing_not_working_with/</a></p>
<p>try ‘num_worker=1’</p>
Changing dtype drastically affects training time

skodeiri: <p>I’ve implemented a Wide Resnet and I’m trying to fine-tune it on Cifar10. The model’s last layer is a linear layer, and before that it’s:<br>
<code>out = torch.mean(out, dim=(2,3), dtype=torch.float64)</code><br>
When the dtype was float32, each epoch was taking around 15 seconds at most, but now it’s taking almost 4 minutes which is a drastic increase. Any idea why?</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the slowdown, please?</p>
Pytorch and Torchvision compiling with different versions of CUDA

jrsaffy: <p>Hi!</p>
<p>I am installing ultralytics(YoloV8) on a new-ish computer with CUDA 12.1 installed.  I was trying:</p>
<pre><code class="lang-auto">pip3 install torch torchvision torchaudio 
</code></pre>
<p>but when I run</p>
<pre><code class="lang-auto">from ultralytics import YOLO
</code></pre>
<p>I get:</p>
<pre><code class="lang-auto">RuntimeError: Detected that PyTorch and torchvision were compiled with different CUDA major versions. PyTorch has CUDA Version=12.1 and torchvision has CUDA Version=11.7. Please reinstall the torchvision that matches your PyTorch install.
</code></pre>
<p>I then tried:</p>
<pre><code class="lang-auto">pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
</code></pre>
<p>But got the same error.  How can I resolve this?</p>

ptrblck: <p>Uninstall <code>torch</code> and <code>torchvision</code> until no proper installation can be found (you might have multiple installations in your current setup), then install both packages once, and post the install log here.</p>
PyTorch on ROCm in Docker in QEMU fails

Imred_Gemu: <p>I know the obvious answer to this is probably that there are just too many layers of jank, and it’s even worse than the title. I’m running PyTorch from the <a href="https://hub.docker.com/r/rocm/pytorch" rel="noopener nofollow ugc">rocm/pytorch</a> Docker container. I’m starting the container by running:</p>
<pre><code class="lang-auto">docker run -it --network=host --device=/dev/kfd --device=/dev/dri --group-add=video --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --shm-size 8G -v /home/username/dockerx:/dockerx -w /dockerx rocm/pytorch
</code></pre>
<p>Specifically I’m trying to run ExLlamaV2, but the problem occurs even running simple tools like <a href="https://pypi.org/project/test-pytorch-gpu/" rel="noopener nofollow ugc">test-pytorch-gpu</a>. Various function calls fail with a runtime error and always the frustratingly unhelpful error:</p>
<pre><code class="lang-auto">HIP error: the operation cannot be performed in the present state
</code></pre>
<p>I’ve tried this running on multiple distros, including a fresh Ubuntu 23.04 with just ROCm kernel modules and Docker installed and loading correctly. These systems are running on Libvirt/QEMU on KVM, with the GPU accessible through PCIe passthrough. My GPU is an RX 6800 XT, which according to ROCm’s documentation isn’t officially supported apparently, but ROCm seems to load normally in spite of this, though possibly with missing functionality? Probably not helping things, I’m working on a laptop with no internal GPU, the 6800 is connected via a Thunderbolt 4 eGPU case. I just need this working for testing purposes, so I’m open to suggestions even if they would sacrifice performance. At the very least I’m hoping for some help debugging this to find exactly what the problem is. If it is just the case that it’s not possible to run on my setup at all, I would like to confirm that.</p>
Tensor copied over to multiple GPUs on its own

ubx: <p>I am having a strange issue with pytorch on a machine that has 2 GPUs. When I run the following code with VRAM of both GPUs empty:</p>
<pre><code class="lang-auto">Python 3.11.6 | packaged by conda-forge | (main, Oct  3 2023, 10:40:35) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; dev = torch.device('cuda:0')
&gt;&gt;&gt; a = torch.rand(1000,1000,1000, device=dev)
</code></pre>
<p>The ‘a’ tensor gets copied over to both GPUs even though I explicitly piped it to GPU <span class="hashtag-raw">#0</span>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/3/032b39857d47d462a24442f6281797facf49a385.png" data-download-href="https://discuss.pytorch.org/uploads/default/032b39857d47d462a24442f6281797facf49a385" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/3/032b39857d47d462a24442f6281797facf49a385.png" alt="image" data-base62-sha1="s22xb3PfzK36a3YPcwfTwt5kyh" width="689" height="87" data-dominant-color="2C2D28"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1374×175 7.22 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, the same thing doesn’t happen on another system with similar hardware with the same code.</p>
<p>Specs of the faulty setup:</p>
<pre><code class="lang-auto">GPU: Quadro P6000 (2x)

$ conda list torch*
# packages in environment at ....:
#
# Name                    Version                   Build  Channel
pytorch                   2.1.0           py3.11_cuda11.8_cudnn8.7.0_0    pytorch
pytorch-cuda              11.8                 h7e8668a_5    pytorch
pytorch-mutex             1.0                        cuda    pytorch
torchaudio                2.1.0               py311_cu118    pytorch
torchtriton               2.1.0                     py311    pytorch
torchvision               0.16.0              py311_cu118    pytorch
</code></pre>
Why this triton kernel crashes?

Thibault712: <p>I can’t find what is the error in this code, it suppose to merge p tensors of size (n, k) such that the merged tensor of size (n, k) has alternating element of these p tensors without doublon per row. (If alternatively you can suggest an other code (in triton or full python) that does the same and that is computationally and memory efficient it would be fine). Thank you for help.</p>
<pre><code class="lang-auto">def _merge_edge_index_kernel(
    edge_index_stacked_ptr,
    num_edges,
    num_edge_types,
    edge_index_merged_ptr,
    BLOCK_SIZE: tl.constexpr,
):
    i = tl.program_id(axis=0)

    offsets = tl.arange(0, BLOCK_SIZE)
    mask = offsets &lt; num_edges

    edge_index_merged = tl.full((BLOCK_SIZE,), value=-1, dtype=tl.int32)
    stride = num_edges * num_edge_types

    count = 0
    num_duplicates = 0
    while count &lt; num_edges:
        j = tl.load(edge_index_stacked_ptr + i * stride + count + num_duplicates)
        not_already_in = tl.sum(edge_index_merged == j, axis=0) == 0
        if not_already_in:
            edge_index_merged = tl.where(offsets == count, j, edge_index_merged)
            count += 1
        else:
            num_duplicates += 1

    tl.store(edge_index_merged_ptr + i * num_edges + offsets, edge_index_merged, mask=mask)


def merge_edge_index(edge_index_stacked: torch.Tensor) -&gt; torch.Tensor:
    """
    Inputs
    ------
        * edge_index_stacked: (sum_i seq_lens[i], num_edges, num_edge_types)

    Output
    ------
        * edge_index_merged: (sum_i seq_lens[i], num_edges)
    """

    assert edge_index_stacked.is_cuda, "edge_index_stacked is not on cuda"
    assert edge_index_stacked.is_contiguous(), "edge_index_stacked is not contiguous"

    total_size, num_edges, num_edge_types = edge_index_stacked.shape

    edge_index_merged = torch.empty_like(edge_index_stacked[..., 0])
    BLOCK_SIZE = triton.next_power_of_2(num_edges)

    grid = (total_size,)
    _merge_edge_index_kernel[grid](
        edge_index_stacked,
        num_edges,
        num_edge_types,
        edge_index_merged,
        BLOCK_SIZE=BLOCK_SIZE,  # type: ignore
    )

    return edge_index_merged
</code></pre>
How to load extremely large tensors into memory

J_B_28: <p>I have an extremely large tensor in my disk whose dimension is [3000000, 128, 768], it cannot be loaded entirely into memory at once because the code crashed. Is there a way to load only specific rows based on the provided indices using the current DataLoader? For example, only load [1, 55555, 2673], three rows from that large tensor in the disk to memory? Thanks!</p>

ptrblck: <p>You could use <code>mmap</code> in <code>torch.load</code> as described in the <a href="https://pytorch.org/docs/stable/generated/torch.load.html#torch.load">docs</a>:</p>
<blockquote>
<p><strong>mmap</strong> (<a href="https://docs.python.org/3/library/typing.html#typing.Optional"><em>Optional</em></a><em>[</em> <a href="https://docs.python.org/3/library/functions.html#bool"><em>bool</em></a><em>]</em>) – Indicates whether the file should be mmaped rather than loading all the storages into memory. Typically, tensor storages in the file will first be moved from disk to CPU memory, after which they are moved to the location that they were tagged with when saving, or specified by <code>map_location</code>. This second step is a no-op if the final location is CPU. When the <code>mmap</code> flag is set, instead of copying the tensor storages from disk to CPU memory in the first step, <code>f</code> is mmaped.</p>
</blockquote>

J_B_28: <p>Hello ptrblck,</p>
<p>Thanks for your reply. Actually I tried to directly use mmap as follows:</p>
<pre><code class="lang-auto">import torch
import mmap
import numpy as np

tensor_size = 1000
tensor = torch.randn(1000, 128, 768)

print("tensor", tensor[torch.tensor([166, 896, 66, 888, 1])])

torch.save(tensor, '/test.pt')

# Path to the file containing the tensor
file_path = '/test.pt'

# Specify the indices of rows to select
indices = [166, 896, 66, 888, 1]

# Define the tensor shape
num_rows = len(indices)
row_size = 128*768  # Assuming each row has a shape of [128, 768]

# Load specific rows using mmap
selected_rows = []
with open(file_path, 'r+b') as file:
    # Memory map the file
    mmapped_file = mmap.mmap(file.fileno(), length=0, access=mmap.ACCESS_READ)

    # Calculate the byte size of each row
    row_byte_size = row_size * np.dtype(np.float32).itemsize

    # Iterate through indices and read specific rows
    for idx in indices:
        mmapped_file.seek(idx * row_byte_size)
        row_data = np.frombuffer(mmapped_file.read(row_byte_size), dtype=np.float32)
        row_tensor = torch.tensor(row_data, dtype=torch.float32).view(128, 768)
        selected_rows.append(row_tensor)

# Concatenate selected rows into a new tensor
concatenated_tensor = torch.stack(selected_rows, dim=0)

# Confirm the shape of the new tensor
print(concatenated_tensor)  # This should result in [5, 128, 768]
</code></pre>
<p>However, the two printed results are not same, could I have some further advices for this issue? Thanks again.</p>

ptrblck: <p>I don’t think you can directly use <code>mmap.mmap</code> on a tensor as the saved file is a archive containing some metadata as well as the actual tensor data, while you are expecting to read the binary data directly from offset 0.</p>
How to call C++ Kernels independently

Shubhi_Shukla: <p>I have installed PyTorch from source. Now, I want to use the CPU Maxpool implementation which is in “aten/src/ATen/native/cpu/MaxPoolKernel.cpp” directly. So basically I want to write a code which calls this maxpool function directly and try giving various kinds of inputs to it, but I’m having problems with it. What is the correct way to do this? I also would like to try this for other operations as well later.</p>
<p>What I tried was that I first created a .h file for MaxPool which somehow is there for MaxUnpool but not MaxPool. Then I wrote a code which included the .h file and then called the function from my code. The problem is there are multiple other includes inside these files as well for which the path is not getting set correctly hence I get the error of file not found.</p>

ptrblck: <p>I’m unsure what your exact use case is but you might want to use the C++ frontend via e.g.:</p>
<pre><code class="lang-cpp">auto model = torch::nn::MaxPool2d(kernel_size=[5, 5], stride=[5, 5], padding=[0, 0], dilation=[1, 1], ceil_mode=false);
auto x =  torch::ones({1, 1, 25, 25}, torch::requires_grad());
auto y = model(x);
</code></pre>
<p>which would then call into these layers.</p>
How to do Instance segmentation with overlapping labels?

henry_de_lange: <p>I’m working on a project to segment rooftop objects, I have a labeled dataset for YOLOv8 from roboflow.<br>
I’m creating a custom model using PyTorch, but I’m stuck as to how PyTorch gets the labels for instance segmentation.<br>
Because it’s about objects on a roof, the labels I have are overlapping, here is an example:<br>
"0 0.8151095629770003 0 0.7230230457121672 0.06468577367652012 0.629497676615071 0.1376042821845973 0.9193334366944352 0.4491040183592357 1 0.3833300151572437 1 0.16354047283849452 0.8438865996222608 0</p>
<p>0 0.5468747017258877 0.04966784730650573 0.41719574128953224 0.12416961826626435 0.35124472712475713 0.17141464375294052 0.7402872243045079 0.5745475758597562 0.8066917261524205 0.5246918030069853 0.9188415514955619 0.44950608104353224</p>
<p>0 0.2841732368719464 0.208170580740801 0.16228395620321057 0.29134432383450326 0.23119906089224523 0.2986127892939919 0.571328448551029 0.6468934258944896 0.5431665666792898 0.6944899882906383 0.5597133627503145 0.706839090537974 0.6625912687571204 0.6121626399750674 0.7316561567057452 0.5674706508894717 0.3505037044936923 0.17323176011781283"<br>
the format is in class, x y coordinates, what is the format PyTorch need to do instance segmentation on this?<br>
Online I found that it’s usually transformed into a mask with 0 as background, class 1 as 1, class 2 as 2, etc. how does that work when an overlapping class has a large overlap with a different class?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f40d697f96b3a5d4b169ccdda06fa97862aca3f8.png" alt="lAUzHNe" data-base62-sha1="yOZeLd1hyD2Mn3bNw4nV2Q4ofrO" width="392" height="443"><br>
In this case, the orange class overlaps for the most part with the pink class (the orange class is on top of the pink class)</p>

Harish_S: <p>may be dataset issue,at the time of createing bounding boxes issue occured,you can resolve this issue by createing dataset again</p>

henry_de_lange: <p>i don’t really have an issue with the dataset, it’s more the representation of the labels that i don’t really understand, specifically when labels overlap.</p>
Which PyTorch version supports GeForce GTX 480?

my-name-s: <p>Which PyTorch version supports GeForce GTX 480?</p>

ptrblck: <p>None of the current binaries as they support compute capabilities 3.7 - 9.0, while your GPU uses the Fermi architecture with compute capability 2.0.<br>
CUDA 8.0 was the last version supporting it, so you might be able to find a really old PyTorch release with this CUDA version.</p>
AMP on cpu: No Gradscaler necessary / available?

Hackster: <p>I want to implement automatic mixed precision into my training framework. According to<br>
<a href="https://pytorch.org/docs/stable/amp.html#torch.autocast" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/amp.html#torch.autocast</a> autocasting is available for cuda and cpu.</p>
<p>I wondered that for their training example on cpu</p>
<pre><code class="lang-auto"># Creates model and optimizer in default precision
model = Net()
optimizer = optim.SGD(model.parameters(), ...)

for epoch in epochs:
    for input, target in data:
        optimizer.zero_grad()

        # Runs the forward pass with autocasting.
        with torch.autocast(device_type="cpu", dtype=torch.bfloat16):
            output = model(input)
            loss = loss_fn(output, target)

        loss.backward()
        optimizer.step()
</code></pre>
<p>they do not use a Gradscaler, which, however, turned out to be crucial for my training on gpu. It seems that GradScaler is only available for cuda (<code>torch.cuda.amp.GradScaler</code>) and it throws errors when trying to use it with tensors on cpu.</p>
<p>Thus my question(s):</p>
<ol>
<li>Is there any reason why one would, in contrary to cuda, <strong>not</strong> need a GradScaler on cpu?</li>
<li>If not, is there any implementation for a GradScaler when running amp on cpu?</li>
</ol>

ptrblck: <p>AMP on the CPU supports <code>bfloat16</code>, which does not need gradient scaling.</p>
Frozen Layers as a Function of Training Memory

ofir1080: <p>Hello there,<br>
I train an Encoder-Deceder model implemented in Torch, where each stage I freeze one component.<br>
My model capacity is ~800M params, where the encoder and the decoder are 50M, 750M params respectively.<br>
I noticed that when I freeze the Decoder and train only the Encoder (meaning ~50M trainable params), it takes MORE memory than when I freeze the Encoder and train only the decoder (~750M trainable params).<br>
My intuition is that in the first case the computation graph must be full, as opposed to the first case where only the deeper part of is necessary (because I don’t need to backprop the whole graph up the encoder).<br>
I am wondering if this makes sense, or I have a bug…</p>
<p>Thanks!</p>
Pytorch model and exported ONNX differs in inference

Ars_ML: <p>I exported Pytorch model to ONNX, and created TensorRT engine from this ONNX. But Pytorch model and TensorRT engine produce different results given the same input data.<br>
(This is exactly the same data I passed to the torch.onnx.export(…) function)</p>
<p>Can you describe shortly, how Pytorch exporter creates ONNX graph?</p>
<blockquote>
<p>it first does the equivalent of torch.jit.trace(), which executes the model once with the given <code>args</code> and records all operations that happen during that execution</p>
</blockquote>
<p>How does exporter record operations? In my case model have some operations, implemented in C++, how exporter can record these operations?</p>
Pthread stack size restrictions

Chr: <p>Hi,</p>
<p>Does anybody know why <code>pthread_create()</code> would return <code>EINVAL</code>  for certain thread stack sizes when linking PyTorch, but not if PyTorch is not linked ?</p>
<p>Lets take this small example:<br>
<strong>main.cpp</strong></p>
<pre><code class="lang-auto">#include "pthread.h"

static void *thread1(void *pstVoid)
{
  std::cout &lt;&lt; "Hello from pthread" &lt;&lt; std::endl;
}

int main(int argc, char **argv)
{
  pthread_t tid1;
  pthread_attr_t pa; 
  iErr = pthread_attr_init(&amp;pa);
  iErr = pthread_attr_setstacksize (&amp;pa, 16*1024);


   iErr = pthread_create(&amp;tid1, &amp;pa, thread1, NULL);
   printf("iErr = %d  \n", iErr);

   pthread_join(tid1, NULL);
}
</code></pre>
<p><strong>CMakeLists.txt</strong></p>
<pre><code class="lang-auto">find_package(Torch REQUIRED)

add_executable(test main.cpp)
target_link_libraries(test PRIVATE torch pthread)
</code></pre>
<p>In this example <code>pthread_create()</code> returns the error <code>EINVAL</code>. However, if I remove <em>torch</em> from <code>target_link_libraries</code> then it works fine. Or alternatively  if I change the stack to 1024x1024 then it also works.</p>
<p>We’re using version 1.7.1 of PyTorch.</p>
<p>And before anyone asks, I’m using a internal library where I work which uses pthread directly and not <code>std::threads</code> <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"> Which is where I encountered this issue first.</p>
How to convert a model to ONNX before conversion?

seamoonlight-YBY: <p>For some reason, I need to store a intermediate state model  into ONNX，The state of this model is between ‘prepare’ and ‘convert’，My current approach is as follows</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
#from torch.ao.quantization import QConfigMapping
from torch.quantization.quantize_fx import prepare_fx, convert_fx, prepare_qat_fx

class Net(torch.nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.bn1 = nn.BatchNorm2d(6)
        self.relu1 = nn.ReLU()
        self.conv2 = nn.Conv2d(6, 1, 1)
        self.bn2 = nn.BatchNorm2d(1)
        self.relu2 = nn.ReLU()
    def forward(self, x):
        x = self.relu1(self.bn1(self.conv1(x)))
        x = self.relu2(self.bn2(self.conv2(x)))
        return x

example_inputs = torch.randn((1, 3, 48, 48))
model = Net()
model.train()
qconfig_dict = {"":torch.quantization.get_default_qat_qconfig('qnnpack'),}
prepared = prepare_qat_fx(model,qconfig_dict)
quantized_model = convert_fx(prepared)
torch.onnx.export(prepared, example_inputs, "quantized_model.onnx", input_names=['input1'], output_names=['result'],opset_version=14, export_params=True,  operator_export_type=torch.onnx.OperatorExportTypes.ONNX_FALLTHROUGH, keep_initializers_as_inputs=True )

</code></pre>
<p>but i got</p>
<p>File “/usr/local/anaconda3/envs/yolo/lib/python3.8/site-packages/torch/onnx/utils.py”, line 1111, in _model_to_graph<br>
graph, params, torch_out, module = _create_jit_graph(model, args)<br>
File “/usr/local/anaconda3/envs/yolo/lib/python3.8/site-packages/torch/onnx/utils.py”, line 987, in _create_jit_graph<br>
graph, torch_out = _trace_and_get_graph_from_model(model, args)<br>
File “/usr/local/anaconda3/envs/yolo/lib/python3.8/site-packages/torch/onnx/utils.py”, line 883, in _trace_and_get_graph_from_model<br>
orig_state_dict_keys = torch.jit._unique_state_dict(model).keys()<br>
File “/usr/local/anaconda3/envs/yolo/lib/python3.8/site-packages/torch/jit/_trace.py”, line 71, in _unique_state_dict<br>
filtered_dict[k] = v.detach()<br>
AttributeError: ‘torch.qscheme’ object has no attribute ‘detach’</p>
<p>My PyTorch version is 1.13.1+cu117</p>
<p>Could you please advise on how to resolve this issue? <img src="https://discuss.pytorch.org/images/emoji/apple/melting_face.png?v=12" title=":melting_face:" class="emoji" alt=":melting_face:" loading="lazy" width="20" height="20"></p>
How to build up pytorch framework efficiently?

deeperlearner: <p>Hi, I’m new here.</p>
<p>I think there are a lot of pytorch veterans here. And I want to share my bootstrap of building PyTorch files sturcture, which auto-generates and organizes dataloaders, models, trainers in pytorch.<br>
<a href="https://github.com/deeperlearner/pytorch-template" rel="noopener nofollow ugc">PyTorch Template</a></p>
<p>I know there are some package like <a href="https://github.com/Lightning-AI/lightning" rel="noopener nofollow ugc">pytorch-lightning</a> etc. But I just want to share my project and also want to know some people sharing their SOP of building up PyTorch projects. Thanks!</p>
Unclear variable lengths in Emformer

Valeriya: <p>Hello!</p>
<p>Thank you for your implementation of Emformer.<br>
However, it is not clear for me what should be the variable ‘lengths’ equal to in the implementation?</p>
<p>Should it be lengths = utterance.length + right_context.length or just lengths = utterance.length?<br>
And if the latter, where should the right context be located if I feed the batch of waveforms: after the paddings or before?</p>
[Question] Trying to train deep learning on Python Pyside2-based GUI

Hye-Min_Moon: <p>I’ve made a deep learning training tool based on python pyside2.<br>
I’m using torch version :  ‘0.12.0+cu113’<br>
well, current cuda I’m using is 11.1, but the training works on python script.</p>
<p>I’m trying to export it as .exe file using pyinstaller,<br>
but when it comes to training phase,<br>
it doesn’t work.</p>
<p>Do I need any extra .dll or anything that relates to using pytorch gpu(cuda) when converting the GUI to .exe using pyinstaller?</p>
Is it possible to directly write activation to CPU memory?

zizhao.mo: <p>Hi,</p>
<p>I am wondering how I can implement the logic that the result evaluated in GPU can be directly written to CPU memory, without temporarily storing it in GPU memory and then transferring it. I think it may be feasible due to the existence of the DMA technique but I am not sure about that. The motivation is that in my project the activation is large, and I want to save activation in CPU memory to save GPU memory.</p>
<p>I know this question may sound weird, but I really appreciate any help you can provide.</p>
<p>BR</p>

eqy: <p>Currently I think the closest thing to this is CPU activation checkpointing which is implemented by projects such as deepspeed: <a href="https://deepspeed.readthedocs.io/en/latest/activation-checkpointing.html" class="inline-onebox" rel="noopener nofollow ugc">Activation Checkpointing — DeepSpeed 0.12.3 documentation</a><br>
In practice you may find that on many systems Device-to-Host bandwidth is actually quite limited compared to GPU memory bandwidth, which may greatly slow your application. This limitation may change in the future though with higher bandwidth unified memory systems becoming more popular.</p>

zizhao.mo: <p>Hi,</p>
<p>Thanks for your reply. I understand that the device-to-host bandwidth is slow. However, in my case, the availability (i.e., the usability of training) is more important than training throughput. When the activation is large (typically common in LLM training), the memory consumption spur caused by activation generation can lead to OOM in a memory-constrained landscape. As such, I intend to avoid temporary storage for activation on GPU, which makes the activation checkpointing mechanism in deepspeed unsuitable. What I expect is that the GPU can directly write the result tensors to CPU memory, without involving the GPU memory. Is it possible?</p>

ptrblck: <aside class="quote no-group" data-username="zizhao.mo" data-post="1" data-topic="191599">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/0ea827/48.png" class="avatar"> zizhao.mo:</div>
<blockquote>
<p>without temporarily storing it in GPU memory and then transferring it.</p>
</blockquote>
</aside>
<p>I don’t think this is possible, but you could use the aforementioned utils. or offload the activation to the host after it was stored temporarily on the GPU as described <a href="https://discuss.pytorch.org/t/modifying-forward-backward-pass/169687/2">here</a>.</p>

zizhao.mo: <p>Thank you,</p>
<p>My doubt is gone.</p>
Looking for a speaker to talk about Pyro

bparbhu: <p>Hi All,</p>
<p>I’m looking for a speaker to give a talk about Pyro/NumPyro for the Bayesian Data Analysis Meetup starting Next February.</p>
<p>Here’s a link to our Meetup.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.meetup.com/bda-group/">
  <header class="source">
      <img src="https://secure.meetupstatic.com/next/images/general/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://www.meetup.com/bda-group/" target="_blank" rel="noopener nofollow ugc">Meetup</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:600/450;"><img src="https://secure.meetupstatic.com/photos/event/3/4/e/b/600_450913547.jpeg" class="thumbnail" width="600" height="450"></div>

<h3><a href="https://www.meetup.com/bda-group/" target="_blank" rel="noopener nofollow ugc">Bayesian Data Analysis | Meetup</a></h3>

  <p>This is a meetup for people interested in Bayesian Statistics, Stan, and related technologies. We'll have talks, Q&amp;A sessions, and workshops. Stan (http://mc-stan.org) is a probabilistic programming language implementing full Bayesian statistical...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I would like to do an in-person meetup in NYC in February so if you’re in the tri-state area and would like to give this talk let me know and I can work with you around expenses.</p>
<p>Also, if you’re interested and far away that’s cool too we can make it more a webinar if that’s the case.</p>
<p>Love the work that’s being done here and would welcome any speaker to give an intro or a series of talks about Pyro!</p>
<p>Also, if interested please reach out to me on here or at my email: <a href="mailto:brian.parbhu@gmail.com">brian.parbhu@gmail.com</a></p>
<p>-Brian Parbhu|</p>
Optimizing sliding window function with Tensor operations

jkswin: <p>Hi all <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I have a function that takes a 2D tensor <code>output_ids</code> and a 1D tensor <code>ignore_span</code>. Using a sliding window approach it creates a mask equal to <code>penalty</code> wherever the <code>ignore_span</code> appears in <code>output_ids</code></p>
<p>The function looks as below (the commented line will become apparent later):</p>
<pre><code class="lang-auto">import torch

batch_size, sequence_length, vocab_size = (4,8,12) #(32, 1024, 2048) #(4,12,8)
logits = torch.randint(0,2, size=(batch_size, sequence_length, vocab_size))
ignore_span = torch.tensor([0, 1, 2])
output_ids = torch.argmax(logits, dim=-1) # (batch_size, sequence_length)


def penalty_mask_loop(output_ids, ignore_span, penalty=5):
    """
    Slow, looped sliding window approach.
    """
    mask = torch.zeros_like(output_ids)
    for i in range(output_ids.size(0)):
                for j in range(output_ids.size(1) - len(ignore_span) + 1):
                    if torch.all(output_ids[i, j:j+len(ignore_span)] == ignore_span):
                        mask[i, j:j+len(ignore_span)] = penalty
                        #mask[i, j] = penalty
    return mask
</code></pre>
<p>I have developed a version that uses tensor operations to avoid the loops and it is much much faster, however the resulting mask is equivalent to the commented line above i.e. it only masks the first index of the span.</p>
<p>See below:</p>
<pre><code class="lang-auto">@torch.no_grad()
def penalty_mask_tensor_ops(output_ids, ignore_span, penalty = 5):
    """
    Faster tensor-op variant.
    """
    mask = torch.ones_like(output_ids)
    # make all len(ignore_span) sized views of output_ids to simulate a sliding window
    slices = output_ids.unfold(1, len(ignore_span), 1)  # shape (batch_size, sequence_length-len(ignore_span)+1, len(ignore_span))                                                     
    matches = torch.all(slices == ignore_span, dim=2)  # shape (batch_size, sequence_length-len(ignore_span)+1) i.e. along the dimension of the length of the span
    mask[:, :matches.size(1)] *= (penalty+1) ** matches 
    return mask - 1

loopmask = penalty_mask_loop(output_ids, ignore_span)
tensmask = penalty_mask_tensor_ops(output_ids, ignore_span)

</code></pre>
<p>Any ideas how I can get this to efficiently mask each occurrence of the full span rather than just the first index of each discovered span match?</p>
<p>Thank you in advance!</p>
Torch.cuda.is_available() is TRUE But No CUDA GPUs are available

i_amRifat: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/8/98b89f3deded988b03c3b7a7ff11c1a98a3b9c7f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/98b89f3deded988b03c3b7a7ff11c1a98a3b9c7f" title="error_cuda"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/98b89f3deded988b03c3b7a7ff11c1a98a3b9c7f_2_553x499.jpeg" alt="error_cuda" data-base62-sha1="lN27PT7EijZCrM3koryXFnVbJIb" width="553" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/98b89f3deded988b03c3b7a7ff11c1a98a3b9c7f_2_553x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/98b89f3deded988b03c3b7a7ff11c1a98a3b9c7f_2_829x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/8/98b89f3deded988b03c3b7a7ff11c1a98a3b9c7f.jpeg 2x" data-dominant-color="1A0E0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error_cuda</span><span class="informations">958×866 145 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hi,<br>
I am having an issue while running my script inference.py Please see the screenshot.</p>
<p>I am using <code>torch==2.1.0</code> and my Nvidia configurations are <code>nvidia-cublas-cu12==12.1.3.1 nvidia-cuda-cupti-cu12==12.1.105 nvidia-cuda-nvrtc-cu12==12.1.105 nvidia-cuda-runtime-cu12==12.1.105 nvidia-cudnn-cu12==8.9.2.26 nvidia-cufft-cu12==11.0.2.54 nvidia-curand-cu12==10.3.2.106 nvidia-cusolver-cu12==11.4.5.107 nvidia-cusparse-cu12==12.1.0.106 nvidia-nccl-cu12==2.18.1 nvidia-nvjitlink-cu12==12.3.52 nvidia-nvtx-cu12==12.1.105</code></p>
<p>I do not have any idea why I am getting this error although <code> torch.cuda.is_available()</code> is returning True and my <code>torch.version.cuda</code> is 12.1.</p>
<p>I need expert help. Thank you.</p>

eqy: <p>Could you verify that the the <code>torch</code> used by the <code>fairseq</code> library is the same as the one in your interpreter (e.g., via <code>__path__</code>) as described here: <a href="https://discuss.pytorch.org/t/cuda-is-available-returns-true-in-consel-but-false-in-program/182892/2" class="inline-onebox">Cuda.is_available returns True in consel but False in program - #2 by ptrblck</a>?</p>

i_amRifat: <p><code>fairseq</code> is using <code>torch==1.3.1</code> version, But while I am trying to install this version I am getting the error <code>ERROR: No matching distribution found for torch==1.3.1</code> and when using wheel getting <code>ERROR: torch-1.3.1-cp37-cp37m-manylinux1_x86_64.whl is not a supported wheel on this platform.</code> error.</p>
<p>What does it mean?</p>

ptrblck: <p>Your Python version is <code>3.10</code> based on the screenshot while you are trying to install a binary built for <code>3.7</code>.</p>

i_amRifat: <p>is there any version available for <code>3.10</code> or should I change my python version to <code>3.7</code>?</p>
CUDA not available after installing PyTorch v1.8.0 via conda

santiagorg2401: <p>Greetings,<br>
I am building a fine-grained image classifier with an already-implemented architecture which requires PyTorch v1.8.0. After following the installation tutorial in <a href="https://pytorch.org/get-started/previous-versions/#v180" class="inline-onebox" rel="noopener nofollow ugc">Previous PyTorch Versions | PyTorch</a> and running:</p>
<pre><code class="lang-auto">conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge
</code></pre>
<p>Everything installs correctly but PyTorch is not compiled for GPUs and <code>torch.cuda.is_available()</code> returns <code>False</code>  even though I have a CUDA-supported GPU. My specifications are as follows:</p>
<ul>
<li>Pop!_OS 22.04 LTS 64-bit</li>
<li>NVIDIA Corporation GA106M [GeForce RTX 3060 Mobile / Max-Q] / RENOIR (renoir, LLVM 15.0.7, DRM 3.54, 6.5.6-76060506-generic)</li>
<li>Driver Version: 535.113.01   CUDA Version: 12.2</li>
<li>conda 23.10.0</li>
<li>Python 3.7.0</li>
</ul>
<p>I have already tried deleting and creating the environment, cleaning Conda packages, and deleting all environments. It is worth mentioning that I do not possess a system-wide CUDA installation.</p>

ptrblck: <p>Try to update to the latest stable or nightly PyTorch release as <code>1.8.0</code> is quite old by now.</p>

santiagorg2401: <p>Hello,<br>
While I totally agree that using newer versions is preferable, the architecture mentioned explicitly requires PyTorch v1.8.0, thus the reasoning behind this post.</p>

ptrblck: <p>Do you know what exactly is required from <code>1.8.0</code> as updating the code might not be too hard?<br>
I.e. did you try to run the code using the latest release and did you see errors you could post here?</p>