akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.multiprocessing on CUDA turns tensors to zeros

Boris_Zhao: <p>I am trying to send a CUDA tensor from main process to another process, and below is the minimum example:</p>
<pre><code class="lang-auto">import torch

def process_1(pipe: torch.multiprocessing.Queue, event: torch.multiprocessing.Event):
    a = torch.rand([100, 100], device='cuda', dtype=torch.float)
    print(f'tensor sent: {a}')
    print(torch.sum(a))
    pipe.put(a)
    event.wait()
    event.clear()
    print(f'tensor after del: {a}')

if __name__ == '__main__':
    pipe = torch.multiprocessing.Queue()
    event = torch.multiprocessing.Event()
    p = torch.multiprocessing.Process(target=process_1, args=(pipe, event, ))
    p.start()
    recv = pipe.get()
    print(f'tensor received: {recv}')
    print(torch.sum(recv))
    del recv
    event.set()

</code></pre>
<p>I tried to run this code on Windows, but I found that CUDA tensors that went through the shared memory (Pipe, Queue, etc.) will be turned to all-zero tensors.</p>
<pre><code class="lang-auto">tensor sent: tensor([[0.0738, 0.4750, 0.2477,  ..., 0.6408, 0.8999, 0.3425],
        [0.9697, 0.0946, 0.6554,  ..., 0.6812, 0.9557, 0.5535],
        [0.0681, 0.4022, 0.7647,  ..., 0.1023, 0.1328, 0.1847],
        ...,
        [0.8344, 0.9620, 0.5390,  ..., 0.2282, 0.6173, 0.3060],
        [0.1959, 0.5154, 0.5861,  ..., 0.3451, 0.1385, 0.2135],
        [0.3778, 0.0317, 0.0770,  ..., 0.6761, 0.7165, 0.1330]],
       device='cuda:0')
tensor(4990.6367, device='cuda:0')
tensor received: tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')
tensor(0., device='cuda:0')
tensor after del: tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')
</code></pre>
<p>I suspect this is because of issues related to Windows.</p>
<p>Any remedies now? Or do I have to switch to the Linux version of PyTorch?</p>

tobiaswuerth: <p>same issue here. (<a href="https://discuss.pytorch.org/t/pytorch-multiprocessing-with-cuda-sets-tensors-to-0/179117/13">commented</a>)</p>
Install PyTorch 1.7

adb57: <p>Hello I am trying to install PyTorch 1.7 to use for a Github repo I found quite interesting, however when I run the pip commands found on <a href="https://pytorch.org/get-started/previous-versions/" class="inline-onebox" rel="noopener nofollow ugc">Previous PyTorch Versions | PyTorch</a>, it appears the version of PyTorch could not be found. Is there a way to manually download it, or is the repo simply no longer feasible to use?</p>
<p>Command I ran:</p>
<blockquote>
<p>pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>
</blockquote>
<p>Output I got</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d6121f6c5d76c51efced35dfb3949c6f5d304a3.png" data-download-href="https://discuss.pytorch.org/uploads/default/9d6121f6c5d76c51efced35dfb3949c6f5d304a3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d6121f6c5d76c51efced35dfb3949c6f5d304a3.png" alt="image" data-base62-sha1="msf4fTbjt5ClOM5NXEnAD7Kws1R" width="690" height="97" data-dominant-color="1E0E0F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1496×211 11.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I don’t know which Python version you are using but you could try to use an older one which was available when PyTorch 1.7 was released.</p>

adb57: <p>I have Python 3.11.4, I guess I would need to downgrade and the above code should work?</p>

ptrblck: <p>A quick search shows PyTorch==1.7 was released in Oct. 2020 while Python==3.11 was released in Oct. 2022, so yes you would need to downgrade your Python version.</p>

adb57: <p>Good to know! Thank you very much for the help!</p>
<p>3.8 was the trick!</p>
RuntimeError: Quantized cudnn conv2d is currenty limited to groups = 1; received groups =16 , during QAT

ANKIT_KUMAR1: <p>what dose it mean , getting erroe during QAT to convert model</p>
<p>model_qat = torch.quantization.convert(quantized_model, inplace=False)</p>

HDCharles: <p>the groups argument for this function (<a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html" class="inline-onebox" rel="noopener nofollow ugc">Conv2d — PyTorch 2.0 documentation</a>) isn’t supported in the quantized conv2d op.</p>

Ankit_kumar_2: <p>Hey Ankit, (<a class="mention" href="/u/ankit_kumar1">@ANKIT_KUMAR1</a> )</p>
<p>Did you find any solution to fix this issue ?</p>

HassanBinHaroon: <p><strong>Solution for me was:</strong></p>
<p>This error may also arise if the computational graph is defined using the dynamic data types (like, dict or list).</p>
Is there a KL divergence operation supported by coreML conversion?

JimW: <p>I have a <code>nn.KLDivLoss()</code> in my model to calculate the distance between two tensors.<br>
However,  when I was trying to convert PyTorch to coreML, I came across the following error:</p>
<pre><code class="lang-auto">  File "/mnt/miniconda/envs/PY3/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 134, in convert_single_node
    raise RuntimeError(
RuntimeError: PyTorch convert function for op 'kl_div' not implemented.
</code></pre>
<p>What is the alternative way of implementing this KL divergence calculation for a successful conversion?  Thanks.</p>

esteban78edbrf: <p>CoreML doesn’t directly support nn.KLDivLoss. Implement KL divergence manually using basic operations, create a custom PyTorch layer, or consider <a href="https://www.adp-run.com/" rel="noopener nofollow ugc">Run ADP</a> an alternative loss function supported by CoreML. Refer to CoreML documentation and explore online resources for specific implementations.</p>
How to write a inplace operator in extension?

hzhangxyz: <p>I am trying to write a extension for a operator, which is a in-place operator, let’s take <code>sort_</code> as example, how to define it in <code>TORCH_LIBRARY</code> ?</p>
<p>The <a href="https://pytorch.org/tutorials/advanced/cpp_custom_ops.html#creating-mutable-operators" rel="noopener nofollow ugc">document</a> is confusing, which takes <code>m.def("myadd_out(Tensor a, Tensor b, Tensor(a!) out) -&gt; ()");</code> as an example, but I do not understand the meaning of <code>a</code> in <code>(a!)</code>, does it refer to the first argument? or why this argument needs two name(<code>a</code> and <code>out</code>)?</p>
<p>In the <code>sort_</code> case, it seems <code>m.def("sort_(Tensor(a!) x) -&gt; ()");</code> works even we do not have an argument named <code>a</code>, but what is the meaning of <code>(a!)</code> ?</p>
Specific implementation of the function scaled_dot_product_attention？

Du_ni_kai: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html#torch-nn-functional-scaled-dot-product-attention" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html#torch-nn-functional-scaled-dot-product-attention</a><br>
The implementation code introduced in this document does not actually work the same as torch.nn.functional.scaled_dot_product_attention<br>
I came to this conclusion through experimentation. My experimental method was to replace the code implementation described in the document with the place where this built-in function was called.</p>
Is it possible to run a model created in CUDA 10.2 in a CUDA 11.1+ GPU?

MahatKC: <p>So I would like to train this model (<a href="https://github.com/tianqi-wang1996/DeepAccident" class="inline-onebox" rel="noopener nofollow ugc">GitHub - tianqi-wang1996/DeepAccident: Code for the benchmark - DeepAccident: A Motion and Accident Prediction Benchmark for V2X Autonomous Driving.</a>) that uses CUDA 10.2, but the GPUs I have available are either an RTX 3090, an A40, or even newer than these. The lowest CUDA version they are compatible with is CUDA 11.1. Is it possible for me to run the code for this model at all, or would I need to rewrite its code or reimplement the model?</p>

ptrblck: <p>Yes, the CUDA version is irrelevant and you would need to install a PyTorch binary with CUDA runtime dependencies for your GPU.</p>
How to disable ___check_obj_id guards on dict?

Xynon: <p>Hi,</p>
<p>This is a fairly simple question but I have a global imported dict that I know will never change, but is quite large. This causes the majority of dynamo guards to check obj ids in the dict since dynamo doesn’t seem to know that it is immutable. Is there any way to mark a dict as immutable to dynamo or manually disable the guards on it?</p>
Support for quantization in int16

m-chenie: <p>Hi there,</p>
<p>I see in the wiki about Quantized Tensors <a href="https://github.com/pytorch/pytorch/wiki/Introducing-Quantized-Tensor" rel="noopener nofollow ugc">Introducing Quantized Tensor · pytorch/pytorch Wiki · GitHub</a>, it mentions that there are plans to support int16 quantization. Although, that page was last edited in 2020.</p>
<p>Are there any updates on support for int16 quantization?</p>
<p>Thanks,<br>
Miranda</p>

HDCharles: <p>no if you wanted to do that you’d normally just use bf16 or fp16 quantization. There are no plans to support it in pytorch at present.</p>

m-chenie: <p>Is there a way to do custom quantization so I can quantize to int16?</p>

jerryzh168: <p>yeah please check out our new flow: <a href="https://pytorch.org/docs/main/quantization.html#prototype-pytorch-2-export-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch main documentation</a></p>
<p>which backend do you want to deploy to?</p>

m-chenie: <p>a custom hardware acceleration that does fixed point arithmetic</p>

jerryzh168: <p>I see the pt2e flow should be helpful here, you can custom what you want to do with: <a href="https://pytorch.org/tutorials/prototype/pt2e_quantizer.html" class="inline-onebox" rel="noopener nofollow ugc">How to Write a Quantizer for PyTorch 2 Export Quantization — PyTorch Tutorials 2.5.0+cu124 documentation</a></p>
Not seeing any training time speedups when using torch.compile

sdangi: <p>Hello, I’m training a DLRM recommender system model (similar to <a href="https://ai.meta.com/blog/dlrm-an-advanced-open-source-deep-learning-recommendation-model/" class="inline-onebox" rel="noopener nofollow ugc">DLRM: An advanced, open source deep learning recommendation model</a>) with DCNv2 in PyTorch.  I’m trying to see if applying torch.compile on my model prior to training can help to provide some training time speedup.  I’ve removed as many graph breaks as I can (there are 5 remaining, which are not easy to remove), but I’m still seeing essentially zero difference in training time when I train without torch.compile vs. with torch.compile.  In fact, the forward() and optimizer.step() times take slightly longer (~6-8% more) when using torch.compile.  Most of my training step time is in the backward pass, and that doesn’t seem to be affected by torch.compile either.  Is this expected?  Even with a few graph breaks, I would have expected that torch.compile would be able to give some training time speed up.  Also – should I expect torch.compile to have a different effect on training time if I train with 1 vs. multiple (e.g., 8) GPUs?  Any insights here would be much appreciated.  Thanks!</p>

bdhirsh: <p>Have you taken a look at the performance section of <a href="https://docs.google.com/document/d/1y5CRfMLdwEoF1nTk9q8qEu1mgMUuUtvhklPKJ2emLU8/edit?tab=t.0#heading=h.cfjs9aarcek" class="inline-onebox" rel="noopener nofollow ugc">torch.compile, the missing manual - Google Docs</a>?</p>
<p>A few things I would look into are:</p>
<p>(1) having ~5 graph breaks might be fine and still give you good perf, or it might not, depending on the context. For example, if you have a graph break in a deeply nested region of code, Dynamo will generate a separate graph at each function boundary. One way to tell is by using <code>tlparse</code> (lots more detail in this section: <a href="https://docs.google.com/document/d/1y5CRfMLdwEoF1nTk9q8qEu1mgMUuUtvhklPKJ2emLU8/edit?tab=t.0#bookmark=id.muskl365sauz" class="inline-onebox" rel="noopener nofollow ugc">torch.compile, the missing manual - Google Docs</a>)</p>
<p>(3) Try running the pytorch profiler, and compare profiles between eager and torch.compile: <a href="https://pytorch.org/docs/stable/torch.compiler_profiling_torch_compile.html" class="inline-onebox" rel="noopener nofollow ugc">Profiling to understand torch.compile performance — PyTorch 2.5 documentation</a></p>
<p>If you are able to generate any of the above outputs (tlparse output, or output of the profiler), sharing them would also be helpful to diagnosing the slowness!</p>
Generating random tensor based on seed tensor

Space4444: <p>Suppose, I have an integer tensor and it contains some unique and some duplicative numbers. Is it possible to create a tensor of the same length filled with random numbers that will be the same at those indices, where elements from initial tensor are the same? Like if elements from first tensor were seeds for producing random numbers in second tensor?</p>

Soumya_Kundu: <p>Can you give an input and expected output?</p>
<p>I do not understand what this bit means:</p>
<blockquote>
<p>where elements from initial tensor are the same?</p>
</blockquote>

Space4444: <p>Ok, for example:<br>
input: <code>[5, 2, 2, 4, 2, 7, 5]</code>;<br>
output: <code>[322, 67, 67, 3938, 67, 190, 322]</code></p>

Soumya_Kundu: <pre><code class="lang-auto">import torch

# Define the input tensor
input_tensor = torch.tensor([5, 2, 2, 4, 2, 7, 5])

# Create an empty tensor to store the output
output_tensor = torch.empty_like(input_tensor, dtype=torch.int32)

# Use a dictionary to store the mapping of input value to generated random number
seed_to_random = {}

# Generate random numbers using the input values as seeds
for i, value in enumerate(input_tensor):
    if value.item() not in seed_to_random:
        torch.manual_seed(value.item())  # Use the value as a seed
        seed_to_random[value.item()] = torch.randint(0, 10000, (1,)).item()
    output_tensor[i] = seed_to_random[value.item()]

print(input_tensor)
print(output_tensor)
</code></pre>
<p>I think this works?</p>

Space4444: <p>Thanks, but my input tensor will be about half of a million long and for loop will take a very long time. Actually, I’ve already found a vectorized solution for my task</p>
What does `foreach` do in AdamW?

Kore_ana: <p>Hello! AdamW has a <code>foreach</code> parameter, which states :</p>
<pre><code class="lang-auto">foreach (bool, optional) – whether foreach implementation of optimizer is used (default: None)
</code></pre>
<p>I tried search for what the “foreach implementation” is, but couldn’t find it. Could anyone explain what this is?</p>
<p>Thank you in advance!</p>

crcrpar: <p>there are some functions with the prefix of <code>_foreeach_</code> such as <code>torch._foreach_exp</code> and <code>torch._foreach_add</code> that take one or more lists of tensors. They apply some counterpart native function such as <code>torch.exp</code> and <code>torch.add</code> to each element of input tensor(s). If a certain condition is met such as tensors are hosted on the same device and of the same dtype then we can expect much less CUDA kernel calls than just iterating over the input lists and call torch functions.</p>

Kore_ana: <p>I see! Thank you for your classification.</p>

Max_Unhold: <p>Hey, thank you for your answer. Could you give a source, where <code>torch._foreach_add</code> is explained in greater detail?</p>
<ul>
<li>Is it an inplace operation for one of the two arguments?</li>
<li>Is there any way to replace this operation by Pytorch functions, looping directly over both lists?</li>
</ul>

janeyx99: <p><a class="mention" href="/u/max_unhold">@Max_Unhold</a> late response, not sure if still relevant, but will answer for the record:</p>
<p>The foreach ops are private in PyTorch today, so the best way I use to check what’s supported is by looking through native_functions.yaml: <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/native_functions.yaml#L10411" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/native_functions.yaml at main · pytorch/pytorch · GitHub</a></p>
<blockquote>
<p>Is it an inplace operation for one of the two arguments?</p>
</blockquote>
<p>It can be! <code>torch._foreach_add</code> is out of place, but <code>torch._foreach_add_</code> (note the underscore) is inplace on the leftmost argument.</p>
<blockquote>
<p>Is there any way to replace this operation by Pytorch functions, looping directly over both lists?</p>
</blockquote>
<p>If I understand the question correctly, yes! foreach_add is a performance optimization over add. Semantically, foreach_blah should boil down to doing blah in a for loop. To give an example, <code>cs = torch._foreach_add(as, bs)</code> should compute similarly to <code>cs = [a + b for (a, b) in zip(as, bs)]</code>. (Don’t quote me on the python haha I didn’t actually run the code.)</p>
How can I parallelize this module

Fox11: <p>I use the following code in my program. Can I avoid a loop within a batch? I have tried transposing some channels of qkv, but the output always failed to match correctly.</p>
<pre><code class="lang-auto">q = torch.rand(32, 256, 768)
k = torch.rand(64, 256, 768)
v = torch.rand(64, 256, 768)

# batch_first=False
cross_attn_0_to_1 = nn.MultiheadAttention(768, 8, dropout=0.0, batch_first=False)
res = []
for i in range(32):
    tmp = cross_attn_0_to_1(q[i].unsqueeze(0), k[i * 2:(i + 1) * 2], v[i * 2:(i + 1) * 2])[0].squeeze(0)
    res.append(tmp)
res = torch.stack(res)
</code></pre>

Space4444: <p>May be like this</p>
<pre><code class="lang-auto">q = torch.rand(32, 256, 768)
k = torch.rand(64, 256, 768)
v = torch.rand(64, 256, 768)

# batch_first=False
cross_attn_0_to_1 = nn.MultiheadAttention(768, 8, dropout=0.0, batch_first=False)

q = q.view(32, 1, 256, 768)
k = k.view(32, 2, 256, 768)
v = v.view(32, 2, 256, 768)
res = torch.vmap(cross_attn_0_to_1)(q, k, v)[0].squeeze(1)
</code></pre>

Fox11: <p>Wow, I just tried this, it worked! Thanks a lot for your help!</p>
Error: ‘PyArray_Descr’ has no member named ‘elsize’

tuler: <p>I’m trying to compile a specific version of PyTorch (2.1.2) from source in Linux, but I’m facing the following issue:</p>
<pre><code class="lang-auto">[1861/1881] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/utils/tensor_numpy.cpp.o
  FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/utils/tensor_numpy.cpp.o
  /src/pytorch/torch/csrc/utils/tensor_numpy.cpp: In function ‘at::Tensor torch::utils::tensor_from_cuda_array_interface(PyObject*)’:
  /src/pytorch/torch/csrc/utils/tensor_numpy.cpp:404:34: error: ‘PyArray_Descr’ has no member named ‘elsize’
    404 |     dtype_size_in_bytes = descr-&gt;elsize;
        |                                  ^~~~~~
</code></pre>
<p>Maybe the cause is an incompatible version of NumPy that is being pulled in.</p>
<p>ps: my environment is actually a bit different. It’s a slightly patched PyTorch being built on riscv64 using qemu. But maybe the error is not related.</p>

ptrblck: <p>Downgrade <code>numpy</code> to <code>&lt;2.0</code> and try to rebuild PyTorch.</p>

tuler: <p>That worked. Thanks for the tip.</p>
Understanding logits in GPT2

firolommones3: <p>Hello. I have the following task: given a string such as <code>This is a nice string</code> I have to tokenize it and  compute the log probs of each token. I assume this means that for e.g. <code>nice</code> I need to compute <code>log p(nice|this is a)</code>.</p>
<p>This is the code I have written to achieve this:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel

model_name = "gpt2"
tokenizer = GPT2Tokenizer.from_pretrained(model_name)
model = GPT2LMHeadModel.from_pretrained(model_name)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model.to(device)
model.eval()

text = "This is a nice string."
tokens = tokenizer.encode(text, return_tensors="pt")
tokenized_sequence = tokenizer.convert_ids_to_tokens(tokens[0])
print(f"Tokenized sequence: {tokenized_sequence} \n")

outputs = model(tokens)
logits = outputs.logits  # shape: [1, sequence_length, vocab_size]

log_probs = torch.log_softmax(logits, dim=-1)

for idx, token in enumerate(tokenized_sequence):
    token_id = tokenizer.convert_tokens_to_ids(token)  
    log_prob = log_probs[0, idx, token_id].item()      
    print(f"token: {token},  log prob: {log_prob}")
</code></pre>
<p>There is something that I’m not sure about. in the line <code> log_prob = log_probs[0, idx, token_id].item()</code>, should I use <code> log_prob = log_probs[0, idx-1, token_id].item()</code> instead? In other words, do the logits at position $j$ in a sequence give the prediction for the token at position $j$ given the previous tokens as context, or is a shift necessary?</p>
^C when training YOLO on google colab

Haku_Yamada: <p>Hello this is my first ever project with PyTorch and idk what’s going on.</p>
<p>I followed two tutorials (one of them being from <a href="https://www.youtube.com/watch?v=LNwODJXcvt4" rel="noopener nofollow ugc">ultralytics</a>) but both had the same problem.</p>
<p>When running</p>
<pre><code class="lang-auto">!yolo task=detect mode=train model=yolov5l6u.pt data={dataset.location}/data.yaml epochs=20 imgsz=640
</code></pre>
<p>it always end up like this and note the ^C. I am not ending the run, it does on its own.</p>
<pre><code class="lang-auto">Logging results to runs/detect/train3
Starting training for 20 epochs...

      Epoch    GPU_mem   box_loss   cls_loss   dfl_loss  Instances       Size
  0% 0/27 [00:00&lt;?, ?it/s]^C
</code></pre>
<p>I also changed the path of train/test/valid of data.yaml so that shouldn’t be an issue.</p>
<p>Any thoughts?</p>

Ngoc_Ninh_Nguyen: <p>Because you are using CPU,<br>
**Edit  =&gt; Notebook settings =&gt; Hardware accelerator =&gt; GPU on Colab to sovle the problem</p>

kangwuing: <p>Nice. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"></p>
TorchLens: New package for feature extraction and visualization of arbitrary models

johnmarktaylor: <p>I recently released a new package, <a href="https://github.com/johnmarktaylor91/torchlens" rel="noopener nofollow ugc">TorchLens</a>, for extracting hidden layer activations and visualizing arbitrary PyTorch models. Here’s the bullet point pitch for anyone interested:</p>
<p>– It’s designed to work for any arbitrary PyTorch model whatsoever, not just a predefined library (but if I’ve missed any edge cases, let me know)<br>
– It can extract activations from and visualize not just PyTorch module outputs, but the results of any PyTorch function call involved in the forward pass.<br>
– It works not just for static computational graphs, but also for dynamic computational graphs (unlike the torchvision feature_extraction module, which only works for static graphs).<br>
– In addition to layer activations, it gives you a bunch of metadata about both the overall model and about each individual layer. The goal is to give you every last bit of information you could possibly want to know about the model.<br>
– It can identify and visualize recurrent feedback loops in the model–this is done both by finding modules that apply the same parameters multiple times, and by finding repeated adjacent loops of the same operations.<br>
– The visualization can show hierarchically nested modules, and allows you to specify how many levels deep you want to show.<br>
– If there’s “if-then” logic in the forward pass, this is visualized as well.<br>
– In addition to the forward pass activations, it can also extract the gradients from a backward pass.</p>
<p>I am trying to make this as useful a tool as possible for folks, so please do let me know if you ever have any wishlist items or grievances :] Hope this is useful for folks.</p>

Ravenwater: <p>just came across this repo and wanted to know who has been using it.</p>

johnmarktaylor: <p>Hope it helps you out :] I welcome any feedback you’ve got</p>
GPU Compatibility Error: NVIDIA GeForce GT 710 with CUDA Capability 3.5 not supported by PyTorch

Ryck: <p>Hello everyone, I’m new here and new to programming. I downloaded a voice cloning program and was using it successfully on Colab. However, when I tried to run it on my PC, I encountered the following error:</p>
<p>runtime\python.exe extract_feature_print.py cuda:0 1 0 0 C:\Users\Rick\Desktop\RVC/logs/test11<br>
[‘extract_feature_print.py’, ‘cuda:0’, ‘1’, ‘0’, ‘0’, ‘C:\Users\Rick\Desktop\RVC/logs/test11’]<br>
C:\Users\Rick\Desktop\RVC/logs/test11<br>
load model(s) from hubert_base.pt<br>
C:\Users\Rick\Desktop\RVC\runtime\lib\site-packages\torch\cuda_<em>init</em>_.py:122: UserWarning:<br>
Found GPU0 NVIDIA GeForce GT 710 which is of cuda capability 3.5.<br>
PyTorch no longer supports this GPU because it is too old.<br>
The minimum cuda capability supported by this library is 3.7.<br>
As I mentioned earlier, I don’t have much programming knowledge. I tried changing the PyTorch version to a lower one, and my current PyTorch version is <strong>version</strong> = ‘1.11.0+cu113’. I also installed CUDA 11.4, which is the highest version supported by my GPU, NVIDIA GeForce GT 710. I understand that the error message is indicating that my GPU is old and doesn’t meet the required compute capability of at least 3.7. I thought that by installing an older version of PyTorch, the issue would be resolved, but it didn’t work. The program is executed through Gradio, and I’m using Python 3.9.</p>
<p>I would greatly appreciate any guidance or suggestions to overcome this issue. Thank you!"</p>
<p>Feel free to make any further modifications or additions to the question as needed.</p>
<p>PD:  I’m using a Windows 10 operating system.</p>

ptrblck: <p>The currently built and released PyTorch binaries support NVIDIA GPUs with a computer capability of 3.7 to 9.0 (if you are using the binaries with the CUDA 11.8 or 12.1 runtime). Your GPU is too old and you might be able to build PyTorch from source for it.</p>

Ryck: <p>The CUDA version I have installed is 11.4, and the PyTorch version is ‘<strong>version</strong> = ‘1.11.0+cu113’’. However, I’m still getting the same error. I’m very new to this and have limited computer knowledge. Am I understanding correctly that you’re suggesting I compile from the source code? Is this process complicated? Is there any other way to resolve this, or is my GPU simply too outdated for this?</p>

ptrblck: <p>Your GPU is too old to <code>pip</code> or <code>conda install</code> any recent PyTorch binary and you could thus try to build PyTorch from source explicitly for your GPU. It depends on your experience in building open-source projects. While I might claim it’s easy, I should also mention that I am constantly building PyTorch from source for testing and development.</p>

the_series_dev: <p><a class="mention" href="/u/ryck">@Ryck</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> did you solve this I am getting the same error when I try to run comfyUI locally, I am having nvidia GT 710</p>

ptrblck: <p>You would need to install PyTorch from source, too, since the binaries do not support your device anymore.</p>
Can I use libtorch-1.12.0 ABI version in CentOs7?

complexhhs: <p>As I mentioned in title, I’m struggling to use libtorch-1.12.0 version.</p>
<p>I want to use libtorch-1.12.0 ABI version in the specific software, which is operated in CentOs7.<br>
The problem is the CentOs7 only supports upto GLIBC-2.15, but libtorch-1.12.0 ABI version requires GLIBC version upto 2.27 version.</p>
<p>Can anyone help?</p>
TorchRL Probabilistic Actor Returning Continuous Actions Despite Given Bounded Discrete Action Spec

SashaBoguraev: <p>Hi! I am currently using TorchRL to build an instance of a reference game, and as such I need my action spaces to be discrete. As such, I have defined my speaker action space as</p>
<pre><code class="lang-auto">self.action_spec = Composite(
                    {"action": Bounded(
                        shape = self.action_space,
                        low = 0,
                        high = self.vocab_size,
                        domain = "discrete",
                        dtype = torch.int64
                    )}, shape=torch.Size([self.n_envs, 1]),
                )
</code></pre>
<p>and my listener action pace as</p>
<pre><code class="lang-auto">self.action_spec = Composite(
                    {"action": Bounded(
                        shape = self.action_space,
                        low = 0,
                        high = self.n_images-1,
                        domain = "discrete",
                        dtype = torch.int64
                    )}, shape=torch.Size([self.n_envs, 1])
                )
</code></pre>
<p>I am then using the <a href="https://pytorch.org/rl/stable/tutorials/multiagent_competitive_ddpg.html" rel="noopener nofollow ugc">MADDPG tutorial</a> to build a training loop, but noticing that the actions for my listener and speaker end up being continuous in nature (I.E. on the continuum between 0-vocab_size and 0-self.n_images-1 respectively). This seems to violate the specs I have designed so I am confused as to what is going on and would appreciate any pointers.</p>

vmoens: <p>Hello!<br>
Do you know / can you show where your actions are cast to a tensor?<br>
The specs are not enforced because that would be costly - it would mean that at every iteration the dtype, device, shape etc of every tensor would be checked against the spec which is time consuming.<br>
We use specs to</p>
<ul>
<li>preallocate buffers in distributed / multiproc settings</li>
<li>check (not project) if tensors are in the space defined by the spec in the env</li>
<li>project the value of the action to the required space in the Actor (eg, clamp the action) given some heuristic in <code>TensorSpec.project</code>.</li>
</ul>
<p>What I think is happening here is that your action is created as a float tensor and unless you run <code>check_env_specs(env)</code> or use your env in multiproc settings, you will not get an error</p>
Initialization of a layer with different function than its forward function

alireza378: <p>I have a layer in my network, suppose it is a matrix that inputs x and outputs y=Ax.<br>
Now, the matrix A is just a placeholder, it is defined based on a function f(x) where x is the input (i.e A=f(x)). So we can say y=Ax=f(x)*x. Trainable parameters are parameters inside f, not A.<br>
Implementing such network is quite easy, the problem is that I want to initialize the placeholder, A, in another fashion, meaning that A=g(x) in initialization, where g is different than f.</p>
<p>I’m not sure how to properly implement this. Using a flag for initialization feels hacky and doesn’t seem like true initialization—it just switches between two computations. Additionally, I’m concerned that this kind of network might not be fully end-to-end differentiable.</p>
<p>Does this make sense, and is it possible to implement this in a principled way in PyTorch? Let me know if anything is unclear or if you have questions about my explanation.</p>

ptrblck: <aside class="quote no-group" data-username="alireza378" data-post="1" data-topic="213778">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alireza378/48/61353_2.png" class="avatar"> alireza378:</div>
<blockquote>
<p>I have a layer in my network, suppose it is a matrix that inputs x and outputs y=Ax.<br>
Now, the matrix A is defined based on a function f(x) where x is the input (i.e A=f(x)). So we can say y=Ax=f(x)*x.</p>
</blockquote>
</aside>
<p>This statement is a bit unclear since you are claiming the layer contains the matrix <code>A</code>, which sounds as if <code>A</code> is a trainable <code>nn.Parameter</code>, while later you are explaining <code>A</code> is computed from the input via <code>A = f(x)</code>. In this case, the layer would be a pure functional layer and would not contain any trainable parameters (which is fine). If so, you could indeed add some logic to the <code>forward</code> method to switch between the different <code>A</code> computations.</p>

alireza378: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="213778">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>This statement is a bit unclear since you are claiming the layer contains the matrix <code>A</code>, which sounds as if <code>A</code> is a trainable <code>nn.Parameter</code>,</p>
</blockquote>
</aside>
<p>I agree, sorry for the confusion. I edited the question for better clarity.<br>
The matrix itself is not trainable, the parameters inside the function f are trainable parameters, matrix A is just a placeholder for function f, during training, A is filled with f(x), but in initializaiton, A is based on g(x).</p>
Flex_attention returning logits

pikantrop: <p>I am running some experiments where I need to use attention logits (attention tensor pre-softmax). One example would be CoPE (<a href="https://arxiv.org/abs/2405.18719" class="inline-onebox" rel="noopener nofollow ugc">[2405.18719] Contextual Position Encoding: Learning to Count What's Important</a>)</p>
<p>Any chance that we can get an option to skip softmax?</p>
Residual Connection breaks or not while calling separately

hasan11085: <p>My question is that does the connection of residual break here? I have a concern that do I need to give residual connection again to make it work like resnet18?</p>
<pre><code class="lang-auto">class preprocessor(nn.Module):
    def __init__(self,num_channels=3):
        super(preprocessor,self).__init__()
        
        self.conv1 = nn.Conv2d(3,192,1)
        self.resnet = models.resnet18(pretrained=True)
        self.resnet.aux_logits=False
        self.conv2 = nn.Conv2d(64,1,1)
        self.dropout= torch.nn.Dropout2d(p=0.5, inplace=False)
        self.seb = SEBlock(64)
        self.sp = SpatialAttention(7)

    def forward(self,x):

        identity = x[:,2:3,:,:]
        b,c,h,w = x.shape

        x = self.resnet.conv1(x)
        x = self.resnet.bn1(x)
        x = self.resnet.maxpool(x)

        #layer1
        x = self.resnet.layer1[0].conv1(x)
        x = self.resnet.layer1[0].bn1(x)
        x = self.resnet.layer1[0].conv2(x)
        x = self.resnet.layer1[0].bn2(x)
        x = self.resnet.layer1[1].conv1(x)
        x = self.resnet.layer1[1].bn1(x)
        x = self.resnet.layer1[1].conv2(x)
        x = self.resnet.layer1[1].bn2(x)
        x = self.dropout(x)
        spa = self.sp(x)
        x = x+spa
        x = self.seb(x)
        x = F.interpolate(self.conv2(x),(h,w))
        x = identity+x
        return x
</code></pre>

ptrblck: <p>I don’t fully understand the question as your code already shows <code>x = identity + x</code>, which could be seen as a residual connection. <code>resnet18</code> does not have an <code>aux_logits</code> attribute so you are creating it and are initializing it to <code>False</code>.</p>
PyTorch > TorchScript tensor.view() alternative?

WLDRMND: <p>0</p>
<p>I have the following code:</p>
<pre><code class="lang-auto">@torch.jit.script
def my_function(t: torch.Tensor) -&gt; torch.Tensor:
    if t.dtype != torch.bfloat16:
        raise ValueError("Input tensor must be of dtype torch.bfloat16")
    int_type = torch.int16
    t_bits = t.view(int_type) &lt;- error
</code></pre>
<p>That throws:</p>
<pre><code class="lang-auto">RuntimeError: shape '[2]' is invalid for input of size 1
</code></pre>
<p>Because, given the documentation:</p>
<pre><code class="lang-auto">.. warning::
This overload is not supported by TorchScript, and using it in a 
Torchscript program will cause undefined behavior.
</code></pre>
<p>So the question is, what’s the alternative for use?</p>
How to deploy multiple instances PyTorch model API for inference on a single GPU?

SyedAunZaidi: <p>I am a beginner in MLOps and I have a Python script that uses a PyTorch model (Whisper Tiny) for speech-to-text (STT). According to the model card, this model has about 39 million parameters and is very small in size compared to my GPU memory (24 GB).</p>
<p>I want to deploy multiple instances of this model on the same GPU and process requests in parallel, so that I can make use of the GPU memory and improve the throughput. However, when I try to do that, the requests are processed sequentially and the GPU utilization is low.</p>
<p>I am using FastAPI and Docker to build and run my app. I have created a Dockerfile that uses <code>pytorch/pytorch:latest</code> as the base image and runs the app with <code>gunicorn</code>. I have deployed two containers from this image, one on port <code>8000</code> and another on port <code>8001</code>. When I send two concurrent requests to these containers, the first request takes 5 seconds and the second request takes 10 seconds, implying that it waits for the first request to complete.</p>
<p>Following is how I am running these containers:</p>
<pre><code class="lang-auto">docker run -d -p 8000:8000 eng_api
docker run -d -p 8001:8000 eng_api
</code></pre>
<p>Following is my <code>Dockerfile</code> file:</p>
<pre><code class="lang-auto">FROM pytorch/pytorch:latest
RUN pip install fastapi uvicorn transformers ...
COPY main.py /main.py
WORKDIR /
CMD ["uvicorn", "main:app", "--host=0.0.0.0", "--port=8000"]
</code></pre>
<p>Following is my <code>main.py file</code>:</p>
<pre><code class="lang-auto">import ...
@app.post("/asr")
async def asr(audio: UploadFile = File(...)):

    audio_data = await audio.read()

    dset = Dataset.from_dict({"audio": [audio_data]})

    dset = dset.cast_column("audio", Audio(sampling_rate=16000))

    audio_array = dset[0]["audio"]["array"]
    sampling_rate = dset[0]["audio"]["sampling_rate"]

    input_features = processor(audio_array, sampling_rate=sampling_rate, return_tensors="pt").input_features

    output = model.generate(input_features)
    transcription = processor.batch_decode(output, skip_special_tokens=True)[0]

    return {"transcription": transcription}
</code></pre>
<p>How can I resolve this issue? How can I ensure that the containers run in parallel and use the GPU resources efficiently? Is there a way to specify the GPU memory allocation for each container? Or do I need to use a different framework or tool to manage the deployment?</p>

marksaroufim: <p>Typically you’re better off serving one model per GPU unless your GPU supports <a href="https://www.nvidia.com/en-us/technologies/multi-instance-gpu/" class="inline-onebox">Multi-Instance GPU (MIG) | NVIDIA</a></p>
<p>The way to increase throughput should be instead to increase the batch size</p>

Karthick_Durai: <p>Can you expand on how to increase the throughput by increasing the batch size. What do you mean by that. Sorry I am new to pytorch, so might be a silly question.</p>
Does computation matter whether it is in the loss function or in the model?

Terence_Chow: <p>I realize this is a simple question, but I didn’t think it mattered and the LLMs told me to move some computation I had in the model to the loss fn and I can’t seem to determine if it hallucinated that answer and am hoping for some clarification.</p>
<p>For concreteness, I was doing ordinal threshold prediction with 6 classes. My last layer of my model was a linear layer, followed by softplus, followed by torch.cumsum and divided by the sum of the rows to get values to between 0 and 1. Essentially I was trying to output 5 threshold probabilities that were monotonically increasing.  The LLMs suggested that I was constraining the optimization space and suggested moving the softplus and cumsum and division to the loss function.</p>
<p>From my understanding this shouldn’t matter because theoretically it’s differentiable regardless of if it’s in the models last layer or if it’s in a loss function. Is this accurate?</p>
<p>Is there any reason to what the LLM is saying? Maybe it matters from a weight initialization pov?</p>

T0BIAS: <p>I’m not to familiar with ordinal threshold regression, but from what i understand the functions you apply at the end (softplus, cumsum and division) behave like a custom activation. Moving it to the loss function would, as far as i know not make a difference since the gradients would be unaffected. However for the sake of functionality i would recommend you keep the current setup since if you dont your model would have to be evaluated with an extra module just to adjust the output to the preferred range [0,1]. Also if you want to compile it to .pt or .pth for any other usecases its best to have the correct last layer activation in the model.<br>
Hope this made sense</p>
CUDA out of memory while using Llama3.1-8B for inference

tomwagstaff-opml: <p>Cross-post from <a href="https://stackoverflow.com/questions/79251900/cuda-out-of-memory-while-using-llama3-1-8b-for-inference" rel="noopener nofollow ugc">Stack Overflow</a></p>
<p>I have written a simple Python script that uses the HuggingFace <code>transformers</code> library along with <code>torch</code> to run <code>Llama3.1-8B-instruct</code> purely for inference, after feeding in some long-ish bits of text (about 10k-20k tokens). It runs fine on my laptop, which has a GPU with 12GB RAM, but can also access up to 28GB total (I guess from the main system RAM?)</p>
<p>However, when I run the code on a “Standard NC4as T4 v3” Windows Virtual Machine, with a single Tesla T4 GPU with 16GB RAM, it very quickly throws this error: <code>CUDA out of memory. Tried to allocate XXX GiB. GPU</code></p>
<p>According to my calculations, this code should run fine given the available RAM. Nevertheless, I’ve tried to make the script more memory efficient:</p>
<ul>
<li>Changing the attention mechanism - by setting the <code>attn_implementation</code> when instantiating the model - first to “sdpa”. Then tried to adopt flash attention, but found it impossible to install the package <code>flashattn</code> on Windows</li>
<li>Using <code>xformers</code> to run <code>enable_xformers_memory_efficient_attention</code> - think this is also about the attention mechanism, but couldn’t install / run this</li>
<li>Using <code>torch.inference_mode()</code></li>
<li>Setting flags like <code>low_cpu_mem_usage = True</code> during model instantiation.</li>
<li>Explicitly setting <code>torch_dtype</code> to <code>torch.float16</code> or <code>torch.bfloat16</code> during model instantiation.</li>
<li>Using <code>BitsAndBytesConfig</code> to trigger 8-bit and 4-bit quantization.</li>
</ul>
<p>The last step was the only one to have any effect - the model manages to generate one or two responses, but still fails to complete the loop, and still fails on the very first inference. If I’ve understood the documentation properly, 4-bit quantization should drastically reduce memory requirements and make it almost trivial to do a forward pass with this model size with this GPU.</p>
<p>So I’m suspicious about trying to further optimise the code - the next step apparently being to manually map all the model layers to the GPU and CPU respectively by defining a <code>device_map</code>.</p>
<p>So my questions are:</p>
<ol>
<li>Is this likely to be truly a memory issue, or is it a red herring?</li>
<li>If it’s memory, what could I do beyond what I’ve tried?</li>
<li>If it’s not, are there obvious things to check? (I’m suspecting to do with system configuration or package installations…)</li>
</ol>
Ubuntu 20.04 end of life

eleven: <p>So Ubuntu 20.04 is approaching End of Life (EOL) in April 2025.<br>
Will PyTorch/Cuda still be supported on 20.04?<br>
If we already have a working stack is upgrading Ubuntu recommended?<br>
I am concerned that upgrading the OS might make my current projects no longer work.  I’ve seen past Ubuntu upgrades brick servers because of incompatible NVidia/Cuda drivers.</p>
<p>Is upgrading Ubuntu recommended, or should we leave our server as is?</p>
Train model with multiple independent time series

zuliani99: <p>I have a multiple datasets that describe the status of independent engines and I would like to create a model that identifies anomalies over the given time-series like a <a href="https://ieeexplore.ieee.org/document/9053558" rel="noopener nofollow ugc">LSTM-VAE</a> or <a href="https://dl.acm.org/doi/10.1145/3394486.3403392" rel="noopener nofollow ugc">USAD</a>, the point is that I do not know how to make the model to generalize over multiple independent datasets, ie how to create a well formatted Dataloader.</p>
<p>One of the possible idea that had come in my mind is set the batch to be equal of the number of engines and then iterate over the batch splitted by the sequence length, since it is unfeasible to stack all the datasets into a single one since the model window possibly get different engines timestamp.</p>
<p>Thanks for your help!</p>
<p>Best regards<br>
Riccardo</p>