akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Unable to exit the deep learning program and free GPU memory

username: wangyurun
cooked: <p>Hello. My deep learning program runs on a ubuntu server with multiple graphics cards. Instead of specifying the graphics card by CUDA_VISIBLE_DEVICES, I specify a specific number when creating tensors in the program. This leads to the fact that although the program mainly uses the specified graphics card, a small part of the memory of the remaining graphics cards is also occupied by this program.Today, when the program was running, a graphics card was broken for some reason. This graphics card is not my tensor storage graphics card. As a result, the program could not be closed at the end, and the graphics memory of the graphics card I used could not be released. I try to use kill command to end the program, but it is useless. Is there a solution other than reboot?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="wangyurun" data-post="1" data-topic="208162">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/dec6dc/48.png" class="avatar"> wangyurun:</div>
<blockquote>
<p>Is there a solution other than reboot?</p>
</blockquote>
</aside>
<p>I don’t fully understand why a reboot would be problematic based on:</p>
<aside class="quote no-group" data-username="wangyurun" data-post="1" data-topic="208162">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/dec6dc/48.png" class="avatar"> wangyurun:</div>
<blockquote>
<p>Today, when the program was running, a graphics card was broken for some reason.</p>
</blockquote>
</aside>
<p>which sounds as if you want to shut down the workstation in any case and replace or at least remove the broken device?</p>

username: wangyurun
cooked: <p>Because other graphics cards can still be used, and someone is still using the server to run their program.</p>

username: wangyurun
cooked: <p>Because someone else is using other graphics cards. Reboot is the final solution.</p>
CUDA 12.2.1 and PyTorch

username: Halil_Huseyin_Calisk
cooked: <p>Hello, I have CUDA 12.2.1 installed on my computer. Will this CUDA version be compatible with the <code>torch</code>, <code>torchvision</code>, and <code>torchaudio</code> libraries that are compatible with CUDA 12.1 as listed on PyTorch’s official website? Thank you very much.</p>

username: ptrblck
cooked: <p>Yes, since your locally CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension.</p>
Pytorch / Torchscript Pro needed!

username: spiegelball
cooked: <p>I am looking for someone who is able to convert existing Pytorch model to JIT / Torchscript. Permanent role or freelance option possible.</p>

username: Lauren1
cooked: <p>I would be glad to assist you.<br>
To discuss further in detail kindly reach me at <a href="mailto:-lauren@cisinlabs.com">-lauren@cisinlabs.com</a> or Skype me:live:lauren_8606<br>
Looking forward to your response.</p>
<p>Thanks,<br>
Lauren W.</p>

username: Andrew2
cooked: <p>Hi,<br>
If you are still on the lookout for some help I would be glad to help you out, as what you are wanting done is well within my area of expertise.<br>
You can reach me on <a href="mailto:andrewjohnson.56782@gmail.com">andrewjohnson.56782@gmail.com</a><br>
Cheers and Have a great day ahead,<br>
Andrew</p>

username: spiegelball
cooked: <p>Our company is still looking!</p>

username: Akhil
cooked: <p>Hi <a class="mention" href="/u/spiegelball">@spiegelball</a><br>
I am a Robotics graduate student from Northeastern University.<br>
I have experience in model quantization and pruning which also had an element of tracing with torch.jit.<br>
If you are still on the lookout for a candidate, please feel free to contact me at sasikumar.ak@northeastern.edu.</p>
Torch profiler makes the result different?

username: Letmehome
cooked: <p>I am using torch profiler to measure my model’s Flops.</p>
<pre><code class="lang-auto">model = model.cuda()
input_tensor = torch.randn((1,3, img_size, img_size)).cuda()
with torch.profiler.profile(
    activities=[
        torch.profiler.ProfilerActivity.CPU,
        torch.profiler.ProfilerActivity.CUDA],
    profile_memory=True,
    record_shapes=True,
    with_stack=True,
    with_flops=True,
) as prof:
    model(input_tensor)

print(prof.key_averages().table(sort_by="cpu_time_total", row_limit=10))

flops = sum([k.flops for k in prof.key_averages()])
print(f"Total G FLOPs: {flops / 1e9} G")

input_tensor = input_tensor.cpu().detach()
del input_tensor
</code></pre>
<p>When I load test data to test before this code block, the result is always the same,</p>
<p>But after this code, the result is always different.</p>
<p>Is there any reason?</p>
Training dataloader problem

username: asdemirel
cooked: <p>We are using the YoloV7 object detection model and want to train it with a custom dataset. Our GPU hardware consists of <strong>Nvidia V100</strong>. Our dataset includes <em>60k training images, 2k validation images, and 4k test images</em>. The image resolution is 640x640. We are using a <em><strong>batch size of 16 and 8 workers.</strong></em></p>
<p>We are encountering a problem during training, which I believe is related to the dataloader. During training, <strong>GPU utilization sometimes drops to 0%</strong> and at other times increases to 50%-70%, resulting in very long iteration times. One epoch takes almost 50 minutes. I tried increasing the number of workers (16, 24), but it did not fix the issue.</p>
<p>Summary, dataloader cannot transfer data synchronously to GPU. So GPU consumption is decreasing.</p>
<p>Consumption %0 during the training. Anormaly<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a01bbdbc56e56bb5ae9403a88fb55ce2d7ffa57.jpeg" alt="anormal" data-base62-sha1="3I47ZBibeKru0MMEOxumNnEYAIv" width="442" height="52"></p>

username: Soumya_Kundu
cooked: <p>Think this is better to ask on their repo as they have custom loaders generally for YOLO.</p>
<p>Have you tried doing/seen if they have added the pin_memory=True and prefetching parameters in the data loader?</p>

username: Tanmay_Pani
cooked: <p>It seems like most of the overload might be coming from too many host to gpu copies, especially from unpinned memory (if pin_memory=True is not already set). If you subclass Dataset, you could do something like implementing the<code> __getitems__()</code> instead of the usual <code>__getitem__()</code>, directly returning a batch created on the GPU (I would recommend increasing batch_size to as many images you can fit in your GPU and setting dataloader workers to one for this). I generally deal with tabular data, so I sometimes am able to get significant speedups by loading my entire dataset on the GPU (if possible) from the begining and then using a combination of <code>torch.utils.data.RandomSampler</code> and <code>torch.utils.data.BatchSampler</code> to obtain training batches, but then you dont get multiple dataloader workers obviously.</p>
Getting unusual strides when using Pytorch's autograd

username: Vedant_Roy
cooked: <p>I’m getting strides like <code>(0, 0)</code> when writing custom Pytorch autograd functions. Here’s a minimal, reproducible example:</p>
<pre><code class="lang-auto">import torch

class Identity(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        return input

    @staticmethod
    def backward(ctx, grad_output):
        print(grad_output.shape, grad_output.stride())
        return grad_output

# Create a callable object for the Identity function
identity = Identity.apply


# Example usage
x = torch.randn(3, 3, requires_grad=True)
y = identity(x)
z = y.sum()
z.backward()

print(x.grad)  # Should be a 3x3 tensor of ones
</code></pre>
<p>This is causing problems / leading to incorrect results when using more complicated autograd functions where I try to make <code>grad_output</code> contiguous.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Vedant_Roy" data-post="1" data-topic="208093">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vedant_roy/48/51087_2.png" class="avatar"> Vedant_Roy:</div>
<blockquote>
<p>This is causing problems / leading to incorrect results when using more complicated autograd functions where I try to make <code>grad_output</code> contiguous.</p>
</blockquote>
</aside>
<p>Could you describe this issue in more detail?<br>
The code snippet you’ve posted is not problematic, since the input will be broadcasted to the needed shape and indeed <code>x.grad</code> shows a tensor of the shape <code>[3, 3]</code>.</p>
Cpp extension example for mixed precision, torch.compile

username: crazyboy9103
cooked: <p>Is there any example for cpp extension to support for mixed precision training and to support torch.compile? im not sure how to make my cuda kernel to support it.</p>

username: Tahamustapha_Nehdi
cooked: <p>i Think its better to use ATen DISPATCH method so your cuda kernel or c++ function adapt on any floating point type specially 16 float point number and this is a call example</p>
<pre><code class="lang-auto">AT_DISPATCH_FLOATING_TYPES(feats.type(), "trilinear_fw_cu", 
    ([&amp;] {
        trilinear_fw_kernel&lt;scalar_t&gt;&lt;&lt;&lt;blocks, threads&gt;&gt;&gt;(
            feats.packed_accessor&lt;scalar_t, 3, torch::RestrictPtrTraits, size_t&gt;(),
            points.packed_accessor&lt;scalar_t, 2, torch::RestrictPtrTraits, size_t&gt;(),
            feat_interp.packed_accessor&lt;scalar_t, 2, torch::RestrictPtrTraits, size_t&gt;()
        );
    }));
</code></pre>
Detectron2 Non-Maximum Suppression

username: mycul
cooked: <p>Has anyone done this before? I can’t seem to find any code online that streamlines this functionality, even the nms functions within detectron2 don’t exactly make it all that simple. You still have to calculate the IoU, but can only do it for two objects at a time?</p>
<p>Was wondering if maybe I have read this incorrectly or if someone could push me in the right direction.</p>
In-place operation causes a memory leak. Why?

username: Amos_Haviv_Hason
cooked: <p>I have the following piece of code:</p>
<pre><code class="lang-auto">eigvecs = torch.randn(n, b, dtype=eigval_approximations.dtype, device=eigval_approximations.device)
eigvecs /= torch.linalg.norm(eigvecs, dim=0)

for _ in range(iterations):
    eigvecs = torch.linalg.solve(mats - eigval_approximations * identity, eigvecs.T).T
    eigvecs /= torch.linalg.norm(eigvecs, dim=0)
</code></pre>
<p>It is part of loss calculation in PyTorch, and it causes GPU OOM after a few epochs.</p>
<p><code>eigvecs</code> is re-initialized at every training/validation step, and it is of the same size at each step. It exists in a very short-lived scope.</p>
<p>Changing this piece of code such that it avoids in-place division (<code>/=</code>) eliminates the memory leak.</p>
<p>Why was there a memory leak due to an in-place operation in the first place? I don’t see any reason for this.</p>
<p>Is it expected in Python, or might it be an implementation detail of or a bug in PyTorch?</p>

username: Amos_Haviv_Hason
cooked: <p>Does anyone have an idea?</p>

username: AlphaBetaGamma96
cooked: <aside class="quote no-group" data-username="Amos_Haviv_Hason" data-post="1" data-topic="207527">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amos_haviv_hason/48/61652_2.png" class="avatar"> Amos_Haviv_Hason:</div>
<blockquote>
<p><code>torch.linalg.solve</code></p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/amos_haviv_hason">@Amos_Haviv_Hason</a>,</p>
<p>I’m not quite sure, so it’d be best to get a dev’s opinion on this, but I do know the <code>torch.linalg.solve</code> syncs with the CPU when it’s run on the GPU (so perhaps there’s a memory leak from that?).</p>
<p>In the documentation of <a href="https://pytorch.org/docs/stable/generated/torch.linalg.solve.html" rel="noopener nofollow ugc">torch.linalg.solve</a>, it states:</p>
<pre><code class="lang-auto">When inputs are on a CUDA device, this function synchronizes that device with the CPU. For a version of this function that does not synchronize, see torch.linalg.solve_ex()
</code></pre>
<p>So, you could try replacing <code>torch.linalg.solve_ex</code> method (docs here: <a href="https://pytorch.org/docs/stable/generated/torch.linalg.solve_ex.html#torch.linalg.solve_ex" class="inline-onebox" rel="noopener nofollow ugc">torch.linalg.solve_ex — PyTorch 2.4 documentation</a>) and see if you get the same memory leak? That would be one way to test this hypothesis (although not conclusive)</p>
<p>Also, if you’re trying to use in-place operations to speed-up pytorch (it’ll make minimal difference in this use case). If you’re purely computing the eigenvalues (and don’t want any gradients), run your code within a <code>torch.no_grad()</code> context manager, which will speed up your code. Docs here: <a href="https://pytorch.org/docs/stable/generated/torch.no_grad.html#torch.no_grad" class="inline-onebox" rel="noopener nofollow ugc">no_grad — PyTorch 2.4 documentation</a></p>
Nvcc missing when installing cudatoolkit?

username: Binbose
cooked: <p>I have installed cuda along pytorch with<br>
<code>conda install pytorch torchvision cudatoolkit=10.0 -c pytorch</code></p>
<p>However, it seems like nvcc was not installed along with it. If I want to use for example <code>nvcc -V</code>, I get the error that nvcc was not found, and that I should install it with <code>sudo apt install nvidia-cuda-toolkit</code>. Can I do this (I dont want to just try and then find out that it is not working/messes up the whole cuda setup).<br>
And is this a bug or expected behaviour?</p>
<p>I am using Ubuntu 18.04 and have cuda 10.2</p>

username: Deepali
cooked: <p>I think <code>nvcc</code> is available with <code>cudatoolkit-dev</code> package.<br>
Please check with <code>conda install -c conda-forge cudatoolkit-dev</code></p>

username: Binbose
cooked: <p>I just tried it, unfortunately it didnt work though. I will just uninstall cudatoolkit and install cuda in a proper way.</p>
Pytorch to Onnx

username: saeed_Ahmad
cooked: <p>Hi there community!<br>
I have a general question if someone can refer any resources on the topic, I’ll be thankful.<br>
question is while writing pytorch models what should be kept in mind to make sure that the model is easily convertible to onnx. And if we encounter some model which is not traceable, how do we debug that, because stack traces are usually so complicated and not accurate.</p>
Flex Attention return score values

username: jwillette
cooked: <p>I saw the newly released Flex Attention <a href="https://pytorch.org/blog/flexattention/" class="inline-onebox" rel="noopener nofollow ugc">FlexAttention: The Flexibility of PyTorch with the Performance of FlashAttention | PyTorch</a> and I have a question.</p>
<p>I am wondering if it is possible to write to some globally scoped tensor the way that the alibi bias example in the link above reads from a globally scoped tensor.</p>
<p>Say I wanted to retrieve all scores from the model to plot the attention matrix, or maybe sum the columns of the attention matrix to modify a KV cache eviction policy, or some other use case.</p>
<p>Is it possible to accomplish this by writing to a globally initialized tensor, I tried the following, but it didn’t work. Is there a way to accomplish this with flex attention?</p>
<pre><code class="lang-auto">import torch
from torch.nn.attention.flex_attention import flex_attention

query = torch.randn(1, 8, 256, 128)
key = torch.randn(1, 8, 128, 128)
value = torch.randn(1, 8, 128, 128)

scores_out = torch.zeros(1, 8, 256, 128)

def noop(score, b, h, q_idx, kv_idx):
    scores_out[b, h, q_idx, kv_idx] = score
    return score

out = flex_attention(query, key, value, score_mod=noop)

print(out.size())
print(scores_out)
</code></pre>

username: jwillette
cooked: <p>started a discussion at: <a href="https://github.com/pytorch-labs/attention-gym/issues/19" class="inline-onebox" rel="noopener nofollow ugc">Writing to a globally scoped tensor from score_mod function · Issue #19 · pytorch-labs/attention-gym · GitHub</a> because it looks like a more appropriate place to discuss this topic.</p>
Training on multiple datasets without mixing them

username: afshin
cooked: <p>I want to train on, say, 2 datasets ds1, ds2. However, I want the samples in each batch come from only one dataset. Hence, concatenating datasets and using a DataLoader with default batch sampling and shuffle won’t work. What is a good and efficient method to do that?<br>
I can think of two possible solutions:<br>
1- defining a custom batch sampler, where I can sample the dataset index, and for each dataset, I use its designated batch sampler to generate the training batch. This way the DataLoader can be used with other libraries such as pytorch lightning or huggingface transformers.<br>
2- each dataset has its own DataLoader, and write a custom training loop where at each iteration I choose the dataloader and then get a batch from it.</p>
<p>Any suggestions?</p>

username: ptrblck
cooked: <p>Both approaches sound fine. Depending on your overall use case might might want to prefer the first approach if you want to make sure higher-level APIs can consume a single <code>DataLoader</code>.</p>
Loading from checkpoint giving Lookahead error

username: Ken_Pierce
cooked: <p>When I try to load the trainer from a checkpoint I get ‘Lookahead’ object has no attribute ‘_optimizer_load_state_dict_pre_hooks’</p>
<p>What do I need to do to alter the checkpoint callback to add the right pieces. I’m using lightning and I thought the checkpoint system would save everyting it needed to get going again. Here’s the cell that does the training and loading.</p>
<p>log_name = f’{local_config[“weights_name”]}_bands{bindices}’<br>
log_path = r’E:/data/lsa_pytorch_logger’<br>
logger = CSVLogger(log_path, name=log_name)</p>
<p>EPOCHS = 10<br>
model.train()<br>
torch.set_float32_matmul_precision(‘medium’)</p>
<p>checkpoint_callback = ModelCheckpoint(save_top_k=2, monitor=“val_F1”,mode=‘max’, dirpath = logger.log_dir,  filename=‘{epoch:02d}-{val_F1:.2f}’)</p>
<p>trainer = pl.Trainer(accelerator=“gpu”,<br>
devices=1,<br>
max_epochs=EPOCHS,<br>
logger=logger,<br>
profiler=“simple”,<br>
callbacks=[checkpoint_callback])</p>
<h1><a name="trainerfitmodel-train_loader-valid_loader-1" class="anchor" href="#trainerfitmodel-train_loader-valid_loader-1"></a>trainer.fit(model, train_loader, valid_loader)</h1>
<p>##<span class="hashtag-raw">#TO</span> LOAD from previous CHECKPOINT<br>
trainer.fit(model,<br>
train_loader,<br>
valid_loader,<br>
ckpt_path=mpath)</p>
output of custom attention mechanism implementation does not match torch.nn.MultiheadAttention

username: carlitosbala931
cooked: <p>I was trying to create my own attention function for a project I’m working on. However, when I compared the output and weights from my code with those from <code>torch.nn.MultiheadAttention</code>, I noticed that the <code>softmax(QK^T/d_k^0.5)</code> is calculated incorrectly. Here is my code:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn.functional as F
from torch.nn import MultiheadAttention

def attention(Q, K, V):
    d_k = Q.size(-1)
    scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_k**0.5)
    attn_output_weights = F.softmax(scores, dim=-1)
    attn_output = torch.matmul(attn_output_weights, V)
    return attn_output, attn_output_weights

embed_dim = 8
num_heads = 1
batch_size = 2
seq_len = 5

Q = torch.randn(batch_size, seq_len, embed_dim)
K = torch.randn(batch_size, seq_len, embed_dim)
V = torch.randn(batch_size, seq_len, embed_dim)

multihead_attn = MultiheadAttention(embed_dim=embed_dim, num_heads=num_heads, batch_first=True)
attn_output_pytorch, attn_output_weights_pytorch = multihead_attn(Q, K, V)

attn_output_custom, attn_output_weights_custom = attention(Q, K, V)

assert torch.allclose(attn_output_custom, attn_output_pytorch, rtol=1e-6, atol=1e-8), "Attention output does not match."
assert torch.allclose(attn_output_weights_custom, attn_output_weights_pytorch, rtol=1e-6, atol=1e-8), "Attention weights do not match."
</code></pre>
<p>I tried changing the hyperparameters, printing each matrix, not normalizing by the d_k^0.5 factor, matching with <code>torch.nn.functional.scaled_dot_product_attention</code>, and checking the shape of each tensor, but I still didn’t get good results. I am primarily concerned with matching <code>attn_output_weights_custom</code> and <code>attn_output_weights_pytorch</code>.</p>
<p>Can someone spot what I might be doing wrong?</p>
Create a view using subsampling

username: Craig_Hicks
cooked: <p>For example with a tensor</p>
<pre><code class="lang-auto">[ [0,1,2,3],[4,5,6,7],[8,9,10,11],[12,13,14,15]]
</code></pre>
<p>create a view exposing</p>
<pre><code class="lang-auto">[ [0,2],[4,6],[8,10],[12,14]]
</code></pre>
<p>Obviously this is not a reshaping with AFAIK is all you can do with view.  However, it doesn’t seem like it would be impossible so I though I would ask.</p>

username: ptrblck
cooked: <p>Direct indexing should work:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor([[0,1,2,3],[4,5,6,7],[8,9,10,11],[12,13,14,15]])
y = x[:, ::2]
print(y)
# tensor([[ 0,  2],
#         [ 4,  6],
#         [ 8, 10],
#         [12, 14]])

y[0, 0].fill_(100)
print(x)
# tensor([[100,   1,   2,   3],
#         [  4,   5,   6,   7],
#         [  8,   9,  10,  11],
#         [ 12,  13,  14,  15]])
</code></pre>

username: Craig_Hicks
cooked: <p>Yes, that can be used to create the vector of indices, and <code>gather</code> can then be used to subsample.</p>
Best practice for multiple losses on different parts of batch

username: heth27
cooked: <p>I have a pose estimation model which I want to train. Some instances have 3d ground truth data, and some have 2d ground truth data. The prediction model is the same for both types, only the loss differs. They are loaded by different dataloaders.</p>
<p>I want both to be in the same batch, what is the (software engineering) best practices for achieving this.</p>
<ol>
<li>Pass them both separately though the model, so two model forward and backward calls, then run the optimizer on the accumulated gradient.</li>
<li>Combine both batches (track how many samples in each), feed them through the model, and then take apart the result before putting it through the loss function.</li>
<li>Calculate both losses, on both batches, with dummy ground truth and multiply with 1 or 0 to only include the valid items in the overall loss.</li>
</ol>
<p>What approach is likely to be the easiest to understand, are there any recipes on how best to approach this problem?</p>
<p>cross post with: <a href="https://stackoverflow.com/questions/78871584/best-practice-for-multiple-losses-on-different-parts-of-batch" class="inline-onebox" rel="noopener nofollow ugc">machine learning - Best practice for multiple losses on different parts of batch - Stack Overflow</a></p>
IndexError: pop from empty list

username: Zunaira_Nadeem
cooked: <p>/usr/local/lib/python3.10/dist-packages/opacus/grad_sample/grad_sample_module.py in rearrange_grad_samples(self, module, backprops, loss_reduction, batch_first)<br>
396                 batch_dim=batch_dim,<br>
397             )<br>
 → 398         activations = module.activations.pop()<br>
399<br>
400         n = module.max_batch_len</p>
<p>IndexError: pop from empty list<br>
Hi I tried all possible code for differential privacy in time series but end up with this error. Please help. I am using this code # Privacy engine hyper-parameters<br>
max_per_sample_grad_norm = 1.5<br>
delta = 8e-5<br>
epsilon = 12.0</p>
<pre><code># criterion = CustomLoss().to(device)
</code></pre>
<p>optimizer = torch.optim.Adam(net.parameters(), lr=0.05)<br>
from opacus import PrivacyEngine<br>
privacy_engine = PrivacyEngine()</p>
<p>model, optimizer, train_loader = privacy_engine.make_private_with_epsilon(<br>
module =net,<br>
optimizer=optimizer,<br>
data_loader=trainloader,<br>
max_grad_norm=max_per_sample_grad_norm,<br>
target_delta=delta,<br>
target_epsilon=epsilon,<br>
epochs=epochs,<br>
)</p>
Best Python Version for PyTorch Compatibility

username: Marcus_Wey
cooked: <p>Hi everyone,</p>
<p>I’m currently working with PyTorch and wanted to know which Python version is recommended for the best compatibility and performance. I’m using Python 3.11.9 and 3.12.5, but they don’t seem to be compatible with PyTorch. I’m considering downgrading if it would provide better stability or support with PyTorch. Although the official website mentions support for Python 3.8 to 3.11, I wonder why my 3.11.9 doesn’t work with PyTorch and often shows an <code>OSError</code>. Any insights or suggestions would be greatly appreciated!</p>
<p>Thank you in advance!</p>
Stride has 2 1s in it

username: Vedant_Roy
cooked: <p>I have a tensor called <code>grads</code> with stride:</p>
<pre><code class="lang-auto">(1866240, 155520, 64, 1)
</code></pre>
<p>and shape:</p>
<pre><code class="lang-auto">shape: torch.Size([1, 12, 2430, 64])
</code></pre>
<p>But, when I do a backwards pass (using a custom autograd function), like this:</p>
<pre><code class="lang-auto">result.backwards(grads)
</code></pre>
<p>In other words:</p>
<pre><code class="lang-auto">        @staticmethod
        def backward(ctx, grad_output):
</code></pre>
<p>The stride of <code>grad_output</code> becomes <code>(1, 155520, 64, 1)</code>, even though the stride of <code>grads</code> is <code>(1866240, 155520, 64, 1)</code>.</p>
<p>The stride of this tensor becomes: <code>(1, 155520, 64, 1)</code>, even though the shape hasn’t changed. Why does this happen?</p>
Torchtext not supported

username: SatyaDasika
cooked: <p>If torchtext is no longer supported. is there an alternative for the same with pytorch 2.0</p>

username: AbdulsalamBande
cooked: <p>I think it depends on what exactly you use torchtext for, can you be more specific?</p>

username: SatyaDasika
cooked: <p>Vocab Builder, tokenizer etc., which were a part of torchtext. Also, on the whole for natural language processing other than nltk is there anything else like torch text?</p>

username: AbdulsalamBande
cooked: <p>I think your problem is solved here:</p><aside class="quote" data-post="3" data-topic="175067">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/torch-2-0-0-is-not-compatible-with-anything-i-want-to-install/175067/3">Torch 2.0.0 is not compatible with anything I want to install</a> 
  </div>
  <blockquote>
    Are you still running into the issue? 
I just tried to install torchtext in the latest version and I get 0.15.1 as seen here: 
&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torchtext
&gt;&gt;&gt; torch.__version__
'2.0.0+cu118'
&gt;&gt;&gt; torchtext.__version__
'0.15.1+cpu'
  </blockquote>
</aside>
Where is the torchvision.ops.nms() method located?

username: Bodo
cooked: <p>I am looking for a method in torch, but for the love of god I can not not find the function definition!<br>
The reason I need to find it is that I need to get rid of the torch dependency and I want to try to convert it into numpy.</p>
<p>I am speaking about <code>torchvision.ops.nms()</code><br>
This method is located in <code>torchvision/ops/boxes.py</code> and returns<code> torch.ops.torchvision.nms()</code>.</p>
<p>This method is generated code which can be found in<code> torch/_ops.py</code> where they initialize <code>ops</code> with <code>ops: _Ops = _Ops()</code>.</p>
<p>Thats the point where I am lost, the class is located in the same file, but I cant figure out which library it calls to get the nms() method.</p>
<p>Please help me <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>The CPU kernel can be found <a href="https://github.com/pytorch/vision/blob/8f73afacb81df1fcc6e2d308c5b734b126db3426/torchvision/csrc/ops/cpu/nms_kernel.cpp">here</a> and the CUDA kernel can be found <a href="https://github.com/pytorch/vision/blob/8f73afacb81df1fcc6e2d308c5b734b126db3426/torchvision/csrc/ops/cuda/nms_kernel.cu">here</a>.</p>
Unexpected error from cudaGetDeviceCount() out of memory (Triggered internally at …/c10/cuda/CUDAFunctions.cpp:108.)

username: hgong
cooked: <p>I am running miniconda in Ubuntu 22.04 under WSL. I first installed wsl cuda toolkit from Nvidia and then installed miniconda. The original cuda version was 11.5 which I then upgraded to 12.4 with conda install nvidia::cuda-toolkit. Then I installed pytorch with cuda support 12.1 using pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu121" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu121</a>. Now when I print torch version cuda, it is returning 12.1. However when I try to do  print(torch.cuda.is_available()), it returns the following:<br>
/root/miniconda3/lib/python3.12/site-packages/torch/cuda/<strong>init</strong>.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory (Triggered internally at …/c10/cuda/CUDAFunctions.cpp:108.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
False</p>
<p>If I try  print(“Number of GPUs:”, torch.cuda.device_count()) it will return 4.</p>
Confusion Regarding Quantization on GPUs with PyTorch

username: Woosung
cooked: <p>Hello everyone,</p>
<p>First, I want to mention that I am a beginner in the field of quantization, so my question might seem basic.</p>
<p>I would like to run quantized DNN models on a GPU. However, as far as I understand from the PyTorch documentation, most quantization techniques are only supported on CPUs, and GPU support for these features seems to be quite limited. (I also tried and got the result as quantied_linear is not supported by CUDA backends…)</p>
<p>What confuses me is that in repositories like <a href="https://github.com/artidoro/qlora" rel="noopener nofollow ugc">QLoRA</a>, it appears that quantization is being performed using torch.</p>
<p>Is the quantization functionality being provided by Hugging Face or the <code>bitsandbytes</code> library, or is it actually utilizing torch for this process?</p>
<p>Additionally, if I provide FP16 data as input to a standard FP32 kernel, will the computation proceed correctly?</p>
<p>Thank you in advance for your help!</p>

username: supriyar
cooked: <p>Hey,</p>
<p>Quantization is supported on GPUs using PyTorch. A lot of our work in this space has been in this repo <a href="https://github.com/pytorch/ao" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/ao: The missing pytorch dtype and layout library for training and inference</a>. Please check it out and let us know if you run into any issues, thanks!</p>
Register for PyTorch Conference 2024!

username: kyliewd
cooked: <p><span class="discourse-local-date" data-date="2024-09-18" data-email-preview="2024-09-18T07:00:00Z UTC" data-range="from" data-timezone="America/Los_Angeles">2024-09-18T07:00:00Z</span>→<span class="discourse-local-date" data-date="2024-09-19" data-email-preview="2024-09-19T07:00:00Z UTC" data-range="to" data-timezone="America/Los_Angeles">2024-09-19T07:00:00Z</span></p>
<p>Join us at <strong>PyTorch Conference</strong> in San Francisco on September 18th-19th! <img src="https://discuss.pytorch.org/images/emoji/apple/purple_heart.png?v=12" title=":purple_heart:" class="emoji" alt=":purple_heart:" loading="lazy" width="20" height="20"></p>
<p>You’ll have opportunities to learn about the latest advancements for PyTorch, and collaborate with top-tier researchers, developers, and academic communities.</p>
<p>The schedule is now live and keynote speakers are being announced! View them <a href="https://events.linuxfoundation.org/pytorch-conference/program/schedule/?utm_source=www&amp;utm_medium=homepage&amp;utm_campaign=Pytorch-Conference-2024&amp;utm_content=hero" rel="noopener nofollow ugc">here</a>.</p>
<p><a href="https://events.linuxfoundation.org/pytorch-conference/?utm_source=www&amp;utm_medium=homepage&amp;utm_campaign=Pytorch-Conference-2024&amp;utm_content=hero" rel="noopener nofollow ugc">Register here to secure your spot</a> - We hope to see you there!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/1/0174abd1f14910086ade6dce09a69665c4b4ca68.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0174abd1f14910086ade6dce09a69665c4b4ca68" title="PyTorch Conference 2024"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/0174abd1f14910086ade6dce09a69665c4b4ca68_2_690x361.jpeg" alt="PyTorch Conference 2024" data-base62-sha1="cSrxU6yPV9jfNGTPRHPpUj346c" width="690" height="361" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/0174abd1f14910086ade6dce09a69665c4b4ca68_2_690x361.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/0174abd1f14910086ade6dce09a69665c4b4ca68_2_1035x541.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/1/0174abd1f14910086ade6dce09a69665c4b4ca68.jpeg 2x" data-dominant-color="2C151F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">PyTorch Conference 2024</span><span class="informations">1200×628 56.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
action_code: pinned_globally.enabled
File from Tutorial nn.Module causing error

username: EarlyBird
cooked: <p>Hello all,</p>
<p>When I run the tutorial code&gt;<br>
<a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/nn_tutorial.html</a><br>
the cell for “pickel.load” causes an error:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c14a7dc7e9437479cc38d0652ab759543b1e315e.png" alt="Bildschirmfoto 2024-08-11 um 16.44.44" data-base62-sha1="rzVKq9oj5bFdOc5UFShEMBNjfOK" width="624" height="238"><br>
After my mac loaded the file and uncompressed it automatically, I compressed it back in Terminal with gzip. Still causes the error.</p>
<p>How can I fix it?<br>
Regards<br>
Sven</p>

username: ptrblck
cooked: <p>The issue might be related to <a href="https://discuss.pytorch.org/t/unpicklingerror-invalid-load-key-x0a/143235">this one</a> pointing to a file corruption error.</p>

username: EarlyBird
cooked: <p>Hi ptrblck</p>
<p>No, I downloaded the model with 2 different browser (Safari, Firefox) and got the same error.<br>
changed the code to os.open the uncompressed file and got the same error.<br>
Downloaded the file with the python code from the tutorial and downloaded it manually, both create the same error.<br>
Can the mnist.pkl.gz file be not compatible with current pytorch versions?<br>
Any idea?</p>
<p>Regards<br>
Sven</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="EarlyBird" data-post="3" data-topic="207901">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/3ab097/48.png" class="avatar"> EarlyBird:</div>
<blockquote>
<p>Can the mnist.pkl.gz file be not compatible with current pytorch versions?</p>
</blockquote>
</aside>
<p>No, I doubt it as it’s working fine for me using <code>torch==2.4.0</code> and a recent nightly binary:</p>
<pre data-code-wrap="python"><code class="lang-python">from pathlib import Path
import requests

DATA_PATH = Path("data_lala")
PATH = DATA_PATH / "mnist"

PATH.mkdir(parents=True, exist_ok=True)

URL = "https://github.com/pytorch/tutorials/raw/main/_static/"
FILENAME = "mnist.pkl.gz"

if not (PATH / FILENAME).exists():
        content = requests.get(URL + FILENAME).content
        (PATH / FILENAME).open("wb").write(content)
        
import pickle
import gzip

with gzip.open((PATH / FILENAME).as_posix(), "rb") as f:
        ((x_train, y_train), (x_valid, y_valid), _) = pickle.load(f, encoding="latin-1")

print(x_train.shape)
# (50000, 784)
</code></pre>

username: EarlyBird
cooked: <p>Thank you ptrblk,</p>
<p>On MAC with torch 2.3.0 it fails, but on Linux with torch 2.5.0 it works.<br>
Looks like a version issue. Anyway thanks for your great help</p>
<p>Regards<br>
Sven</p>
Apply transform to neural network output

username: hannanmustajab
cooked: <p>Hey, I am training a PINN model, which is basically a FCN. I want to apply hard boundary conditions by transforming the neural network output. I am currently doing it in this way, but am not getting the expected results. What would be the best way to do it ?</p>
<pre><code class="lang-auto">class HardPINN(Model):

    def initial_condition(self, x,y):
        return torch.exp(-0.5 * (((x - 2) / 0.1) ** 2 + ((y - 2) / 0.1) ** 2))

    def forward(self, x):
        # Extract inputs
        t_in = x[:, 0:1]
        x_in = x[:, 1:2]
        y_in = x[:, 2:3]
        
        # Take the input, pass it into the first input layer and pass it to the hidden layers from there.           
        x = self.input_layer(x)
        
        # Get from first layer and pass it on to the hidden layers. 
        x = self.hidden_layers(x)
        
        # get the output from last hidden layer and get the output. 
        x = self.output_layer(x)
        

        # Define sigmaX (example value, replace with actual value)
        sigmaX = 0.1  # Adjust this value as needed

        # Apply initial condition ansatz
        term1 = torch.sigmoid( 5 * (2 - (t_in/sigmaX)) ) * self.initial_condition(x_in, y_in)
        term2 = torch.tanh( t_in / sigmaX ) ** 2 * x
        u = term1 + term2

        return u
</code></pre>
<p>Basic idea here is to remove the first term from the neural network output at t&gt;&gt;2*sigmaX.</p>
<p>I am not getting the expected results, can you please tell me if I am doing something wrong here in terms of the output transform?</p>

username: Tahamustapha_Nehdi
cooked: <p>why you are not using the newtork output can you explain more your prob</p>

username: hannanmustajab
cooked: <p>I am training a physics informed neural network for solving wave equation. For it to be well posed, it has to exactly satisfy the initial condition, at t=0. With soft constraint, it never exactly matches that condition. So I want to apply it using hard boundaries or constraints. Idea here is to transform the neural network output by a function F(x,y,t) which matches / satisfies boundary conditions.</p>
How To Apply Early Stopping in The Case of Person Reid

username: Tahamustapha_Nehdi
cooked: <p>Im training  on Reid Dataset and i want to perform early stopping the problem is i have 3 sets<br>
train with id on each sample , gallery the set of available ids , query the set of sample that i need to match on gallery the scores is calculated are mAP , cmc form rank 1 , 5 , 20 …</p>
<p>when i split the train and val i dont have gallry and query on splited train im wondring how to evaluate and save best perfroming checkpoint acc is not a relevant eval in this case</p>
Does numba works in pytorch?

username: Jackson_brown
cooked: <p>Hi, I found numba libarary helps run code faster than usual python code. I am wondering, to make the training time faster, does Numba works for pytorch? If not, do you have any alternative to make it faster in terms of speed. Thanks</p>

username: Daniel_Gonzalez1
cooked: <p>Hi, I understand Numba does some kinds of precompilations for python code making in it faster. In that sense it should work fine with pytorch, but maybe is not very helpful as pytorch already wraps a lot of critical operations in C++.</p>
<p>For inference or evaluation during training, make sure you use <code>with torch.no_grad()</code>. Or avoid any other unnecessary operations during the model evaluation.</p>
<p>Daniel Gonzalez<br>
Embedded SW Engineer at RidgeRun</p>

username: AlphaBetaGamma96
cooked: <p>You could have a look into <code>torch.compile</code>?</p>
How to access a Tensor of a class derived from torch::nn::Module

username: DavidHr
cooked: <p>I have a class derived from torch::nn::Module. In the constructor I define a Tensor and I register it as buffer: this-&gt;register_buffer(“pe”, pe);</p>
<p>I need to get it for modifiying… I have used: auto pe_buffer= this-&gt;named_buffers()[“pe”];</p>
<p>but I can’t modify it this way.</p>
<p>Any help would be appreciated.</p>
'torch/csrc/jit/mobile/import.h' file not found

username: alisonqiu
cooked: <p>I tried to copy and paste the inference module from <a href="https://github.com/pytorch/ios-demo-app/tree/master/SpeechRecognition" class="inline-onebox" rel="noopener nofollow ugc">ios-demo-app/SpeechRecognition at master · pytorch/ios-demo-app · GitHub</a> into my own xcode project</p>
<p>I always get the error in LibTorch-Lite.h saying ‘torch/csrc/jit/mobile/import.h’ file not found.</p>
<p>I tried pod ‘LibTorch_Lite’, ‘~&gt;1.9.0’ and pod ‘LibTorch_Lite’, ‘~&gt;1.10.0’ and tried both <span class="hashtag">#import</span> &lt;Libtorch-Lite.h&gt; and <span class="hashtag">#import</span> &lt;Libtorch-Lite/Libtorch-Lite.h&gt; but nothing worked. I didn’t find a solution to this issue through Google. Any help would be appreciated!</p>

username: alisonqiu
cooked: <p>I found at this issue appears whenever I put InferenceModule.h and InferenceModule.m into my pods folder and it might have to do with header search path. Any idea why it’s happening?</p>

username: Shivam_Tripathi
cooked: <p>Did u found any solution?</p>
Is it safe to cache tensors inside of an autograd function?

username: Vedant_Roy
cooked: <p>I roughly have the following:</p>
<pre><code class="lang-auto">def make_autograd_function():
     cache = {}
     class TestFunc(torch.autograd.Function):
         @staticmethod
         def forward(ctx, q):
             if 'foo' not in cache:
                  cache['foo'] = torch.zeros((5, 5), device=q.device, dtype=q.dtype)
</code></pre>
<p>I’m caching a tensor inside of the <code>cache</code> variable. Is this safe?</p>
<p>(I know there’s <code>ctx</code>, but I want to avoid re-initializing the tensor every time.)</p>
How to map LOCAL_RANK to its RANK equivalent?

username: Claudio_dos_Santos_F
cooked: <p>I’m writing a data loader that is very IO and memory intensive, and as such, I want only one worker per node handling the data loading and <code>scatter</code>ing the results to the other local workers.</p>
<p>To to this, I need to know the RANKs of all other local workers in my local worker group. So my questions are:</p>
<p>Given the <code>RANK</code> of a worker whose <code>LOCAL_RANK</code> is <code>0</code>, can I assume that the <code>RANK</code>’s of the other local workers will <strong>always</strong> be <code>RANK+1</code>, <code>RANK+2</code>, …, <code>RANK+(LOCAL_WORLD_SIZE-1)</code>? Where is this documented?</p>
<p>If not, then how can I map the <code>LOCAL_RANK</code>’s <code>1</code>, <code>2</code>, …, <code>(LOCAL_WORLD_SIZE-1)</code> to <code>RANK</code> values?</p>

username: irisz
cooked: <p>This API might be helpful for your use case.<br>
<code>get_global_rank</code>, which translate a group rank into a global rank.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/torch/distributed/distributed_c10d.py#L934">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/distributed_c10d.py#L934" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/distributed_c10d.py#L934" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/torch/distributed/distributed_c10d.py#L934</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="924" style="counter-reset: li-counter 923 ;">
          <li>        raise ValueError(</li>
          <li>            f"Group {group} is not registered, please create group with torch.distributed.new_group API"</li>
          <li>        )</li>
          <li>    group_ranks = _world.pg_group_ranks[group]</li>
          <li>    if global_rank not in group_ranks:</li>
          <li>        raise ValueError(f"Global rank {global_rank} is not part of group {group}")</li>
          <li></li>
          <li>    return group_ranks[global_rank]</li>
          <li></li>
          <li></li>
          <li class="selected">def get_global_rank(group: ProcessGroup, group_rank: int) -&gt; int:</li>
          <li>    """</li>
          <li>    Translate a group rank into a global rank.</li>
          <li></li>
          <li>    ``group_rank`` must be part of `group` otherwise this raises RuntimeError.</li>
          <li></li>
          <li>    Args:</li>
          <li>        group (ProcessGroup): ProcessGroup to find the global rank from.</li>
          <li>        group_rank (int): Group rank to query.</li>
          <li></li>
          <li>    Returns:</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Calculating mean and std for images?

username: Danibahnani
cooked: <p>Hi!<br>
I am new to torchvision and I am trying to normalize my images. I am using transforms.v2.Normalize(). It takes mean and std as parameters. If I have understood it correctly it needs one value for each channel.</p>
<p>I am wondering if you should calculate the mean and std of the pixel distributions over the whole training dataset or for each picture?</p>

username: Tahamustapha_Nehdi
cooked: <p>Hi<br>
a good practice to use Imagnet Mean and std</p>
<p>Using the mean and std of Imagenet is a common practice. They are calculated based on millions of images. If you want to train from scratch on your own dataset, you can calculate the new mean and std. Otherwise, using the Imagenet pretrianed model with its own mean and std is recommended.</p>
Eval mode or training mode for WGAN-GP

username: DC1
cooked: <p>I use Batchnorm2d in my WGAN-GP model. I have heard that it is preferred for GAN models to stay on training mode because eval mode is highly unstable, which is particularly the case when you have batchnorm in generator. Is this true?</p>
<p>I already did a bit of testing and it does seem like the model on training more generated more realistic and diverse samples while the model on eval mode outputted very low quality images that almost looked like mode collapse. I originally thought eval mode would give improved results, but it seems like my WGAN-GP model does way worse on eval mode. Is this normal, and should I just use my model on training mode for evaluation?</p>
GroupNorm vs BatchNorm

username: DC1
cooked: <p>Is it better to use BatchNorm or GroupNorm in the generator of my conditional WGAN-GP model? Although BatchNorm gave me good results, my small batch size of 10 meant the eval mode gives bad images. I decided to use GroupNorm, and it allowed me to use eval mode. Yet GroupNorm caused my generator outputs to look very similar even with very different inputs. Is this a normal problem with GroupNorm and how do I fix this?</p>
MPI Run with Pytorch

username: Prannesh
cooked: <p>Hi Everyone,<br>
I amt trying to run MPI with Pytorch from Source for distributed runs. I am able to build, compile and instal. But post installation, i am unable to import torch.</p>
<p>I am using OpenMPi and Pytorch latest version.</p>
<p>Let me know if i have to export any variables or if there is anything other information needed from side to proceed further.</p>

username: fduwjj
cooked: <p>Can you let us know what errors you run into? Looks like there is another post asking about this.</p>

username: fduwjj
cooked: <p>what is your use case for MPI here?</p>