akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Setting Up ExecuTorch doc for macOS out of date

crossingpatterns: <p>I ran into a <em>“Too many open files (os error 24)”</em> issue trying to build ExecuTorch on macOS Sonoma 14.5.</p>
<p>I resolved it by simply building from <a href="https://github.com/pytorch/executorch/commit/d288076192f548a25457b72cd8021ee4d1952db6" rel="noopener nofollow ugc">main</a>.</p>
<p>The Setting Up ExecuTorch doc instructs you to clone a specific branch: <em>git clone --branch v0.2.1 <a href="https://github.com/pytorch/executorch.git" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/executorch: On-device AI across mobile, embedded and edge for PyTorch</a></em></p>
<p>This should probably be updated. Apologies if there’s a different process for flagging this that I’m unaware of <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

lucylq: <p>Hey Bill,<br>
Glad you were able to resolve the issue by building from main! We recently released ExecuTorch v0.3.0 and the <a href="https://pytorch.org/executorch/stable/getting-started-setup.html" rel="noopener nofollow ugc">Setting Up ExecuTorch doc</a> has been updated to point to that.</p>
Can I use @torch.no_grad() upon classes?

jaehyung.ca: <p>Is it okay to use the decorator <span class="mention">@torch.no_grad</span>() upon classes?<br>
Or should I put the decorator on every method of a class?</p>
<p>Thank you in advance.</p>

albanD: <p>Hi,</p>
<p>I don’t think you can actually decorate a class definition in python.<br>
You will have to do it for each function one by one I’m afraid.</p>

jaehyung.ca: <p>No worries. Thank you again.</p>

mohsenhariri: <p>Decorators are higher-order functions, it’s like a function over another function. You can use multiple decorators over one function but you can’t use it in a class.</p>
<p>Decorators are higher-order functions, they are like functions applied to other functions. You can use multiple decorators on a single function but you can’t use them on classes. (no meaning to define a higher-order function over a class)</p>
Distributing Individual Convolutions

maw2025: <p>I’m a beginner working on distributing a convolution neural network. Does the abstraction of PyTorch take care of assigning each CuDA block the correct convolution and therefore by using the abstracted PyTorch distribution it would take care of this?</p>

H-Huang: <p>Take a look at the getting started guide for PyTorch distributed: <a href="https://pytorch.org/tutorials/beginner/dist_overview.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Distributed Overview — PyTorch Tutorials 2.4.0+cu124 documentation</a></p>
<p>PyTorch distributed APIs typically work at the module level, so the convolution would not be split up. But you can use something like <a href="https://pytorch.org/docs/stable/notes/ddp.html" rel="noopener nofollow ugc">DistributedDataParallel</a>, which would replicate the module and allow you to perform the same convolution on different devices feeding in different data.</p>

maw2025: <p>So with the DDP I would have to worry about distributing the individual convolutions as it’s abstracted away already and does it for me?</p>

H-Huang: <p>Yes, just initialize the process group and wrap your model with <code>DDP(model, ...)</code>. Tutorials should help you out there</p>

maw2025: <p>If I would like to learn more about how DDP works where should I go? I don’t only want to learn how to use DDP but how it works under the hood.</p>

H-Huang: <p>This has the high level architecture and code pointers <a href="https://pytorch.org/docs/master/notes/ddp.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch master documentation</a></p>
<p>code is also available on github: <a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/distributed/c10d/reducer.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/csrc/distributed/c10d/reducer.cpp at main · pytorch/pytorch · GitHub</a></p>
Hadamard product with expanded query and key

User6: <p>Hi all,</p>
<p>In transformer architecture. I have confusion about the process of expanding the query and key to form token-wise pairs to compute the attention by copying and concatenating their elements.</p>
<p>Suppose I have a query and key with shape [1, 4, 16] where 1 is the batch size, 4 is the number of tokens and 16 is the dimension. I supposed that the result is going to have the shape [1, 4, 4, 16] but I cannot calculate the hadamard product between them if I transpose the key [1,4,16,4]<br>
hadamard product does not work with the following shapes [1, 4, 4, 16] and [1, 4, 16, 4].</p>
<p>I searched for solutions online but could not find sufficient explanations. Thank you in advance for your help!</p>
Upload CT scan files with Dinov2

Abdelkareem_Elkhateb: <p>I want to upload Ct scans images into Dinov2 but i get mismatch error<br>
the input to the model is in the following size</p>
<pre data-code-wrap="py"><code class="lang-py">torch.Size([16,1,10,18,18])
</code></pre>
<h1><a name="batch-channel-depth-height-width-1" class="anchor" href="#batch-channel-depth-height-width-1"></a>Batch, Channel, Depth, Height, Width</h1>
<p>but Dinov2 needs the channel numbers to be 3 and no width.<br>
This is my thoughts, any help or guide to fix this problem ! thanks in advance</p>
Why nn.LSTM still use float16 in hidden_state, even if set to bfloat16 or float32 already?

stgzr: <p>Here is the code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


rnn = nn.LSTM(10, 20, 2).cuda()
input = torch.randn(5, 3, 10).cuda()
h0 = torch.randn(2, 3, 20).cuda()
c0 = torch.randn(2, 3, 20).cuda()

with torch.amp.autocast("cuda", dtype=torch.bfloat16):
    output, (hn, cn) = rnn(input, (h0, c0))
    print(hn)
</code></pre>
<p>Then I find the dtype of hn is <code>float16</code>, I don’t know why.<br>
My torch version is <code>2.4.0</code></p>
Seeking Guidance on Integrating Custom Backends in Eager Mode with PyTorch

siahuat0727: <p>Hi team,<br>
I am exploring how to integrate a custom backend with PyTorch. After reading about <a href="https://pytorch.org/docs/main/torch.compiler_custom_backends.html" rel="noopener nofollow ugc">custom backends</a> and <a href="https://docs.google.com/document/d/1_W62p8WJOQQUzPsJYa7s701JXt0qf2OfLub2sbkHOaU/edit#heading=h.ptttacy8y1u9" rel="noopener nofollow ugc">The Custom Operators Manual</a>, it looks like modern custom backend support often relies on computational graphs. However, in scenarios where there are more dynamic operations and a computational graph cannot be extracted, it seems that integrating a custom backend in eager mode is necessary.<br>
Is my understanding correct? If so, could you kindly share the latest recommended way to integrate a custom backend in eager mode with PyTorch? Thank you.</p>
Can't Export the model to ONNX using either Dynamo or JIT-based approach

Mohammad_Mahdi: <p>I’ve already asked a question about why I couldn’t export a simple Vision Transformer model to ONNX (and then to TRT) and I had a workaround answer for it <a href="https://stackoverflow.com/questions/78787534/converting-a-pytorch-onnx-model-to-tensorrt-engine-jetson-orin-nano" rel="noopener nofollow ugc">here</a>.</p>
<p>Can anybody explain why PyTorch couldn’t export the model to ONNX unless I had to use JIT alongside disabling <code>checkpoint</code> mode. Why didn’t the Dynamo-based approach work?</p>
Libtorch tensor.permute() - malloc(): corrupted top size

5had3z: <p>I’m currently developing an op that operates over a contiguous final dimension. Both the cpu and gpu variants use the same code to pre-shape input and post-shape the output tensor. Currently it seems only the cpu variant has an issue with reshaping the output: <code>malloc(): corrupted top size</code>. I’ve tried a few combinations of view/reshape assigning a new tensor or mutating inplace, but I keep getting this problem. Again, the gpu variant is fine (even with same code), just the cpu is a problem, all the requested shapes look okay with the good ol print debugging (requested permute is just 1,0 on a 2dim tensor).</p>
<pre data-code-wrap="c++"><code class="lang-c++">void restoreOutputShape(torch::Tensor&amp; output, c10::IntArrayRef inShape, int64_t dim)
{
    output = output.reshape(getPermutedShape(inShape, dim));
    if (dim != (output.ndimension() - 1))
    {
        output = output.permute(getReversePermutation(output.ndimension(), dim));
    }
    output = output.contiguous();
}
</code></pre>
<p><a href="https://github.com/5had3z/torch-discounted-cumsum-nd" rel="noopener nofollow ugc">Open sourcing now</a> even though still WIP so it can be inspected.</p>

5had3z: <p>There was a bad stride calculation in the cpu kernel, rather than the post-process reshaping. Just classic UB things, error gets thrown not in the place where the actual memory bug is <img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=12" title=":upside_down_face:" class="emoji" alt=":upside_down_face:" loading="lazy" width="20" height="20">.</p>
How do I decide on the final model when using cross validation?

cylanokim: <p>I’m trying to build a multi classification model which is selecting an one out of 20 categories. But the problem is that I have very few data. So I am going to use the Cross Validation method and I made 5 models by dividing whole data as below picture.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb7aa5712d17c62e75c0ddc8231e8805ecf2fc0a.png" data-download-href="https://discuss.pytorch.org/uploads/default/bb7aa5712d17c62e75c0ddc8231e8805ecf2fc0a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb7aa5712d17c62e75c0ddc8231e8805ecf2fc0a_2_345x123.png" alt="image" data-base62-sha1="qKw3qQhqwS2NTGfBTjTgE18nHrI" width="345" height="123" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb7aa5712d17c62e75c0ddc8231e8805ecf2fc0a_2_345x123.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb7aa5712d17c62e75c0ddc8231e8805ecf2fc0a_2_517x184.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb7aa5712d17c62e75c0ddc8231e8805ecf2fc0a_2_690x246.png 2x" data-dominant-color="DAA670"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">784×280 20.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you know, in this case I can get 5 models with the .pth extension and I don’t know what results I will return when new data is entered into my neural network. The methods I personally thought of are as follows.</p>
<ol>
<li>When a new data enter into the five models, select the category with the highest frequency among the results. For example, when the results are calculated as follows, the answer is “dog”</li>
</ol>
<p>model1 : “dog”<br>
model2 : “dog”<br>
model3 : “cat”<br>
model4 : “sparrow”<br>
model5 : “dog”</p>
<ol start="2">
<li>
<p>Among the five models, the model with the highest accuracy is selected and used for the service.</p>
</li>
<li>
<p>Instead of newly defining the model for each fold, use only one model and training the model</p>
</li>
</ol>
<p>There are many difficult to develop a practical model. I’d like to hear advice from those who have developed models and services using neural network. Thanks</p>

Soumya_Kundu: <p>You can look into ensembling as a first step: <a href="https://pytorch.org/tutorials/intermediate/ensembling.html" class="inline-onebox" rel="noopener nofollow ugc">Model ensembling — PyTorch Tutorials 2.4.0+cu124 documentation</a></p>
<p>Nice intro theory to it: <a href="https://machinelearningmastery.com/tour-of-ensemble-learning-algorithms/" rel="noopener nofollow ugc">https://machinelearningmastery.com/tour-of-ensemble-learning-algorithms/</a></p>
<p>But, if you have a very scarce dataset, make sure your splits have a representation of each class in each fold.</p>
Advanced Slicing

steinraf1: <p>I would like to slice a torch tensor in a pattern where the slice has a width that is not equal to one (what would basically be slicing off columns of a matrix) for a 1D tensor.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.tensor([0.0, 0.0, 0.0, 0.0, 1.0, 1.0] * 3)
&gt;&gt;&gt; b = a.view(3, -1)[:, 4:]
&gt;&gt;&gt; b
tensor([[1., 1.],
        [1., 1.],
        [1., 1.]])
&gt;&gt;&gt; b.view(-1, 6)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
</code></pre>
<p>Given the rules of torch.Tensor.view this of course makes sense since this cant be expressed as a uniform slice, but I was wondering if there are any other ways to perform such an operation while not doing a copy of the original storage.</p>

Soumya_Kundu: <p>Maybe looking into this would be useful: <a href="https://github.com/pytorch/pytorch/issues/60466" class="inline-onebox" rel="noopener nofollow ugc">PyTorch unfold could be faster · Issue #60466 · pytorch/pytorch · GitHub</a></p>
<pre><code class="lang-auto">import torch

a = torch.tensor([0.0, 0.0, 0.0, 0.0, 1.0, 1.0] * 3)
b = a.view(3, -1)[:, 4:]

# Using as_strided
result = torch.as_strided(b, size=(1, 6), stride=(2, 1))

print(result)
</code></pre>
<p>Is this what you were looking for? I maybe mistaken.</p>

steinraf1: <p>torch.as_strided actually performs the stride operation on the original memory layout, so the value that is output from this program is actually</p>
<pre><code class="lang-auto">tensor([[1., 1., 0., 0., 0., 0.]])
</code></pre>
<p>and not the vector of ones one might expect.</p>

Soumya_Kundu: <p>Ah, my bad.</p>
<p>Making it contiguous solves that problem (but then you’re making a copy). My understanding was a bit wrong I guess then. Unfortunately do not have a solution then. Even <code>.reshape</code> is only for contiguous.</p>
How do I apply Batch Normalization on a sequential data?

rectified_sigmoid: <p>I am trying to train an NLP model that takes in the entire sequence at once instead of passing each time step individually as this approach is faster afaik. So, my data is of shape <code>(seq_size, batch_size, length)</code>. This format is working fine for all other layers but I am facing problem with <code>BatchNorm1d</code>. So, normally when I am using a non-sequential data of shape <code>(batch_size, length)</code>, I do the following:</p>
<pre><code class="lang-auto">self.linear = nn.Linear(length1, length2)
self.bn = nn.BatchNormalization1d(length2)
</code></pre>
<p>and then in the <code>forward</code> function:</p>
<pre><code class="lang-auto">x = self.linear(x)
x = self.bn(x)
</code></pre>
<p>and this works fine. But when I use the same method for sequential, it takes the second dimension, i.e. the batch_size and then gives a <code>RuntimeError</code>. eg, my batch size is 32, and the length of output from the previous <code>Linear</code> layer is 64, so I have defined my batch norm as <code>nn.BatchNorm1d(64)</code>. but this gives me error <code>RuntimeError: running_mean should contain 32 elements not 64</code> on forward pass. How do I fix this? what is the correct method for this kind of problem?</p>

ptrblck: <p>Try to permute your input so that it has the shape <code>[batch_size, seq_len, features]</code> and rerun the code again.<br>
<code>nn.Linear</code> and <code>nn.BatchNorm*d</code> expect the batch dimension to be in <code>dim0</code>, which might yield the error you are seeing.</p>

rectified_sigmoid: <p>will it give the same result as using each timestep individually and then combining? like will permute change and calculations that would have taken place if I passed the sequence one by one?</p>

ptrblck: <p>The linear layer will be applied to the additional input dimensions (the <code>*</code> in <code>[batch_size, *, in_features]</code>) as if you would pass the input sequentially in <code>dim*</code>.</p>
<p>The batchnorm layer will work differently, since you are currently using the batch size as the feature dimension, which sounds wrong.</p>

Seth_Briney: <aside class="quote no-group" data-username="rectified_sigmoid" data-post="1" data-topic="102106">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rectified_sigmoid/48/30094_2.png" class="avatar"> rectified_sigmoid:</div>
<blockquote>
<p>(seq_size, batch_size, length)</p>
</blockquote>
</aside>
<p>I think the simplest solution is to treat the sequence and the batch dimensions equally. So you could do:</p>
<p>x = self.bn(x.reshape(seq_size*batch_size, length)).reshape(x.shape)</p>
<p>from the perspective of self.bn the batch size is seq_size*batch_size using this method.</p>

jiyuechenxing: <p><a class="mention" href="/u/rectified_sigmoid">@rectified_sigmoid</a> Excuse me, has the problem been solved yet?</p>
Torch Error: magic_number = pickle_module.load(f, **pickle_load_args) _pickle.UnpicklingError: invalid load key, 'v'

Adex: <p>When I am trying to load a <code>.pt</code> file i am seeing the following error,</p>
<pre><code>    str1='Dataset/ALL_feats_cgqa.pt'
    m = torch.load(str1)
</code></pre>
<p>the error is as follows,</p>
<pre><code class="lang-auto">  File "/home/Storage1/pythonCodeArea/train.py", line 21, in load_embeddings
    m = torch.load(str1)
  File "/home/.local/lib/python3.10/site-packages/torch/serialization.py", line 1040, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/.local/lib/python3.10/site-packages/torch/serialization.py", line 1262, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.
</code></pre>
<p>I have no idea about this error. Any help will be highly appreciated.</p>
<p>I have gone through these references without any solution, <a href="https://github.com/danielgatis/rembg/issues/262" rel="noopener nofollow ugc">[1]</a>, <a href="https://github.com/WongKinYiu/yolor/issues/269" rel="noopener nofollow ugc">[2]</a>, <a href="https://stackoverflow.com/questions/63063723/how-to-fix-pickle-module-loadf-pickle-load-args-pickle-unpicklingerror-i" rel="noopener nofollow ugc">[3]</a>, <a href="https://discuss.pytorch.org/t/trying-to-load-a-model-in-pytorch-1-6-results-in-pickle-unpicklingerror-invalid-load-key-xad/95615">[4]</a>.</p>
Is there a PyTorch equivalent of TensorFlow.js?

mrdbourke: <p>Hello,</p>
<p>I’m wondering if there’s an equivalent of TensorFlow.js in PyTorch?</p>
<p>TensorFlow.js works right in the browser and I’d like to do something similar with a PyTorch model (if possible).</p>
<p>Has anyone done this?</p>
<p>Or is using an API for deployment too easy these days?</p>
<p>Running directly in the browser has a fair few benefits.</p>
<p>Thank you</p>

marksaroufim: <p>There doesn’t seem overwhelming community demand for it but if you think I’m wrong please let me know!</p>
<p>An alternative may be <a href="https://github.com/microsoft/onnxjs" rel="noopener nofollow ugc">https://github.com/microsoft/onnxjs</a> but the repo is read only so idk if this the best option long term</p>

garymm: <p>You can do <a href="https://pytorch.org/docs/stable/onnx.html" rel="noopener nofollow ugc">PyTorch → ONNX</a> → <a href="https://onnxruntime.ai/docs/get-started/with-javascript.html" rel="noopener nofollow ugc">ONNX Runtime Web</a>.</p>
<p>ONNX Runtime Web is under active development.</p>

Emma_N: <p>FYI, ONNX.js has been replaced by ONNX Runtime Web which offers enhanced user experience and improved performance.</p>

lex.lutsenko: <p>One way would be to go Pytorch → Keras directly using <a href="https://github.com/AlexanderLutsenko/nobuco" rel="noopener nofollow ugc">Nobuco</a>, then convert to TFLite/TFJS as normal.</p>

eduardoleao052: <p>A library similar to what you’re referring to is <a href="https://github.com/eduardoleao052/js-pytorch" rel="noopener nofollow ugc">JS-PyTorch</a>.</p>
Hardware memory pool as a torch.device

Pan_Yue: <p>We have a hardware remote memory pool device and would like to add functions related to tensor allocation and migration between remote and local memory. The device (CXL memory pool) shows up as a NUMA node, and does not have any compute function. This could be useful such as retrieving long context KV cache that have not been actively used for a while. I am currently thinking about extending the <code>privateuseone</code> device and only implement memory allocation and migration related ops, such as <code>Tensor.to</code>. For example, I’m looking to achieve the following:</p>
<pre><code class="lang-auto">tensor = torch.rand((500, 500), device='cpu')
tensor.to(device='cuda') # move from CPU to GPU
# some computation
result_tensor.to(device='mempool')  # move from GPU to mem pool
result_tensor.to(device='cuda')  # move from mem pool to GPU
</code></pre>
<p>Two questions</p>
<ul>
<li>Is it doable?</li>
<li>Is this is approach a good idea or there are more explicit and easier solutions?</li>
</ul>
<p>Would greatly appreciate if the community can provide advice or related materials!</p>

ptrblck: <p>If I understand your use case correctly, you are interested in using custom memory allocations, so <a href="https://github.com/pytorch/pytorch/pull/131152">this PR</a> adding the <code>torch.cuda.MemPool()</code> API might be interesting for you.</p>
How to Control Staged Backward Execution in PyTorch for Llama2 Model?

zbtrs: <p>Hello everyone,</p>
<p>I am training a Llama2 model using PyTorch, and I noticed that directly using <code>loss.backward()</code> takes too long to complete within the specified time slot. Therefore, I want to split the Llama2 model into multiple submodules based on decoder layers, with each submodule containing several decoder layers.</p>
<p>However, I am unsure how to control the staged execution of backward propagation. I found that once the forward pass is completed, the computation graph is automatically built, and executing <code>loss.backward()</code> will directly execute the entire backward process. Is there a good way to control the staged execution of backward propagation?</p>
<p>Thank you!</p>
Can I ignore the `not writable` warning if I'm sure I won't write to it?

Fei_Liu: <p>I converted a readonly numpy array to torch by <code>torch.from_numpy</code>, which gives the following warning:</p>
<blockquote>
<p>UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program.</p>
</blockquote>
<p>I’m sure that I’ll just use the resulting tensor as “read-only” and I just like to use pytorch APIs to do stuff than numpy (torch API is a lot more coherent and consistent imo). In this case, can I safely discard the warning? Based on the message I think the answer is yes. But one has to be very cautious when undefined behaviors are involved. Additional context: this <code>from_numpy</code> conversion needs to be done for every IO steps and it’s beneficial to avoid unnecessary copy (I have no control over what external library returns. They gave me a numpy array instead of a torch tensor).</p>

JuanFMontesinos: <p>Hi,<br>
typically, when you convert a numpy array to pytorch, the memory is shared.<br>
Namely, if you modify the array in pytorch “object”, you should observe the same change in the numpy “object”.<br>
The warning simply states that there will be un undefined behavior, and therefore I think nobody can ensure it is ok doing nothing. I believe it depends on what operations you carry out, there are in-place ops that you cannot probably do or they raise hidden bugs. If you reaaally know what you are doing it might be safe. Note that pytorch will modify the memory and therefore you have no clear way of detecting the change (as no further issues will be raised). However, you can simply make a copy of the array as<br>
<code>tensor=torch.from_numpy(array.copy())</code>, if you are using pytorch, making a copy is probably not a bottleneck in your algorithm.</p>
<p>This could happen if you use numpy memory map, for example. Also I believe numpy arrays have a “non-writtable” mode controlled by a flag.</p>

Fei_Liu: <p>If I want to create a fresh new tensor I’ll just use the more convenient api <code>torch.tensor(np_array)</code>. I just want to understand the scope of this warning. The warning itself says it’s undefined behavior to “write to it”. The databuffer itself is large so extra copy does affect performance (otherwise I won’t care). My read of the warning is that (in plain words): torch does not have ways to prevent you writing to a protected memory buffer (other than np, which has read-only flag protections), so it warns me that “I can’t stop you, so be cautious”. This is where I want to get clarification, and make sure my understanding is correct.</p>
How to load csv file into a PyTorch datasets

Small_Red_69: <p>Sorry that I am still a tiro in Pytorch, and so may raise a naive question: now I managed to collect a great deal of application data in a csv file, but got no idea on how to load the .csv file into a PyTorch “datasets”.</p>
<p>Alternatively, can I bypass the PyTorch datasets but instead use the PyTorch DataLoader() class to load those CSV data directly?</p>
<p>Thanks a lot for any help!</p>

ptrblck: <p>You could write a custom <code>Dataset</code> and load the data there as described in <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a>.</p>

srishti-git1110: <p>Feel free to checkout <a href="https://github.com/srishti-git1110/Lets-go-deep-with-PyTorch/blob/main/Dataset%20and%20DataPipes%20blog/Dataset_and_DataPipe_colab.ipynb" rel="noopener nofollow ugc">my notebook</a>.<br>
You’ll learn how to load any type of data stored in csv files using <code>torch.utils.data.Dataset</code> and <code>torch.utils.data.DataLoader</code>.<br>
It also includes the new TorchData (DataPipes) functionality in case you are interested.</p>

L_Z: <p>I have a question regarding to this, so I need to upload the rows from .csv files and I am using pandas.read_csv() and then .iloc() to get the exact part I want. And then I used np.array(.iloc(csv_file)). Then, after uploading all the files, I just convert the whole thing as np.array().<br>
file_path = glob.glob(file_path)<br>
data_frames = np.array([np.array(pd.read_csv(file)<strong>.iloc[3:, 1:]</strong>) for file in file_path])<br>
And then I group my data with labels with a class where I convert torch.tensor(sequence, dtype=torch.float32) and for labels converted as torch.tensor(label, type=torch.long). And then I call the DataLoader() to separate into batches. But if I only upload <strong>one row</strong> from each .csv file, then when I iterate through the train_loader, I am getting an error message saying "can’t convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint64, uint32, uint16, uint8, and bool.<br>
file_path = glob.glob(file_path)<br>
data_frames = np.array([np.array(pd.read_csv(file)<strong>.iloc[3, 1:]</strong>) for file in file_path])</p>
Torch Problem fbgemm.dll" or one of its dependencie

Hamza_Fayaz: <p>\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\lib\fbgemm.dll" or one of its dependencies</p>

ptrblck: <p>The issue is related to <a href="https://discuss.pytorch.org/t/failed-to-import-pytorch-fbgemm-dll-or-one-of-its-dependencies-is-missing/201969/">this topic</a> and <a href="https://github.com/pytorch/pytorch/issues/131662">this issue</a>.</p>
Is it smart to revert to the best model weights when the learning rate changes?

eTuDpy: <p>I am currently looking deeper in the usage of learning rate schedulers. Hereby, a critical thought came by: if my learning rate goes down by using StepLR or ReduceLRonPlateau, the decreased learning rate leads to a more detailed search for the local minimum. Which is what I want. However, from the gradient perspective, this continued more detailed search does not happen at my current best minimum (based on validation loss), but rather at the current spot within the n-dimensional optimization space. Hence, the optimizer starts a more detailed search at a less optimal spot it could. So, I was thinking of writing a<code>OptimizationCheckpointManager</code>. The main idea behind this class is to save the model weights whenever the validation loss decreases and to collect the learning rate of the optimizer. If the learning rate changes, the <code>OptimizationCheckpointManager</code> overwrites the model weights of the current model with the best weights seen so far. This way, the training can continue from the point of the lowest validation loss with the new learning rate.</p>
<p>Here’s my first version of the class:</p>
<pre><code>class OptimizationCheckpointManager(object):
    def __init__(self, model, optimizer):
        self.best_model_state = model.state_dict()
        self.best_optimizer_state = optimizer.state_dict()
        self.best_loss = float('inf')
        self.current_lr = self._get_lr(optimizer)

    @staticmethod
    def _get_lr(optimizer):
        for param_group in optimizer.param_groups:
            return param_group['lr']

    def update(self, val_loss, model, optimizer):
        updated_lr = self._get_lr(optimizer)

        if val_loss &lt; self.best_loss:
            self.best_loss = val_loss
            self.best_model_state = model.state_dict()
            self.best_optimizer_state = optimizer.state_dict()

        if updated_lr &lt; self.current_lr:
            self.current_lr = updated_lr

            ## update learning rate in best optimizer state
            for param_group in optimizer.param_groups:
                param_group['lr'] = updated_lr

            return True
        else:
            return False

    def return_best_states(self):
        return self.best_model_state, self.best_optimizer_state
</code></pre>
<p>The order within the training loops goes something like:<br>
training → validation → lr_scheduler → OptimizationCheckpointManager</p>
<p>Well, currently I do not see any big difference in my results. So, my questions are:<br>
1.) Is this ideal actually any good?<br>
2.) And if yes, might I have a logical error within my class.</p>
Info on how to make custom backend with primtorch

aadulla: <p>If we would like to work on a custom backend using the canonicalized operators in PT2.0’s PrimTorch, are there any examples or tutorials on how to go about doing this?</p>

khushi-411: <p>Hi, there!<br>
To use the primTorch backend, you can do the following, and the rest of the story remains the same.</p>
<pre><code class="lang-python">In [1]: import torch

In [2]: import torch._prims as prims

In [3]: prims.add(torch.tensor([1, 2, 3]), torch.tensor([6, 5, 4]))
Out[3]: tensor([7, 7, 7])
</code></pre>
<p>PS: If you are looking at the working of primTorch, there’s a great post by <a class="mention" href="/u/mruberry">@mruberry</a> on <a href="https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-0/577" rel="noopener nofollow ugc">Tracing with primitives</a>. Although, it’s more of a developer discussion rather than a tutorial or example.</p>
<p>Out of curiosity, why do you want to use primTorch operators?<br>
Thanks!</p>

aadulla: <p>Thanks for the reply!</p>
<p>I was more so asking about the process for plugging in a custom backend at the ATen IR level. According to a previous PT2.0 announcement, it seems like if I were to write a backend that minimally implements the ~200 PrimTorch operators, that would be sufficient. However I cannot seem to find any additional directions on how to go about doing this.</p>

aadulla: <p>Hi, just wanted to bump this post again and see if there is any info available on custom backend integration through primtorch.</p>

awf: <p>It would be interesting to get a pointer here, agreed.  I guess the general question is “how do we best interpose some graph transforms before our custom backend”?</p>

Marcus_Nilis: <p>+1. I’m experimenting with out-of-tree backend using PivateUse1 as a disptach key. I’ve started with eager mode and aten kernels.<br>
At first it looked as it would be sufficient to impelment only those that declared as “dispatch=True” &amp; “default=False” based on <a href="https://pytorch.org/tutorials/advanced/extend_dispatcher.html" class="inline-onebox" rel="noopener nofollow ugc">Extending dispatcher for a new backend in C++ — PyTorch Tutorials 1.13.1+cu117 documentation</a> . There are ~1k of them. However this ended up in RuntimeError. More kernels need be implemented to run a neural network, e.g.:<br>
aten::convolution_overrideable … “dispatch”: “True”, “default”: “True”<br>
aten::cross_entropy_loss … “dispatch”: “False”, “default”: “True”<br>
And now it feels like I’ll need to implement all the 3k of kernels to make it work.</p>
<p>Noticed PrimTorch project that targets to reduce the number of must have kernels. Is there any way to use PrimTorch in Eager mode without modification of original python code of a neural network?</p>

jelly_brain: <p>I am also thinking about trying to write a new PrimTorch backend, particularly using Kompute.<br>
Where can I find the API which is needed to be implemented, so I would comprehend the complexity of the task ahead.</p>

Hacker1337: <p>I am also very interested!</p>
How to a layer-wise learning rate scheduler?

Wongboo: <p>How to apply layer-wise learning rate scheduler? e.g., different warmup epochs for different layer in cosine decay scheduler with linear warmup.<br>
I try a lot but no clue.</p>

ptrblck: <p>The schedulers will apply their learning rate update to all parameter groups defined in the optimizer, if I’m not mistaken, so I would assume creating separate optimizers with their corresponding schedulers might be a valid approach. <a class="mention" href="/u/alband">@albanD</a> might correct me if I’m missing a more flexible approach for schedulers.</p>

Wongboo: <p>Yes, it works, I actually have thought about these ideas before, but it seems way more complicated than layer-wise learning rate in optimizer.</p>

Wongboo: <p>It seems over complicated since I’ll create multiple optimzers and multiple schedulers, any simplification?</p>
Sin = sin[position_ids].unsqueeze(1)

TaeYeon_Lee: <p>In pytorch,</p>
<p>sin = sin[position_ids].unsqueeze(1)</p>
<p>position_ids.shape is<br>
torch.Size([1, 1024])</p>
<p>sin.shape is<br>
torch.Size([1024, 64])</p>
<p>In C++ Torch lib,<br>
How to convert it as same ?</p>

ptrblck: <pre><code class="lang-auto">auto out = sin.index({position_ids}).unsqueeze(1)
</code></pre>
<p>should work based on the <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html#getter">docs</a>.</p>

TaeYeon_Lee: <p>Found another solution</p>
<p>torch::Tensor position_ids_temp = position_ids.reshape(1024);<br>
sin = sin.index_select(0, position_ids_temp).unsqueeze(0).unsqueeze(0);</p>
Implementing LBFGS in PyTorch

Ashima_Kalathingal: <pre><code class="lang-auto"># The main program

# Initialization
init_i    = gen_i(I1)
init_Tamb = T_amb1[0]
func = NeuralODE(init_i,init_Tamb).double()
func.load_state_dict(torch.load('func_params_3500.pth'))
# Defining the loss function
lossfun = nn.MSELoss()

optimizer = optim.LBFGS(func.parameters(),lr=1e-3)


loss_total = []
epch       = []
for itr in range(201):  # 201 epochs
    
    # Random values of s from 1 to 15
    s = np.random.choice(np.arange(1,16),15,replace=False)
    start_time = time.time()
    
    def closure():
        optimizer.zero_grad()
        loss_batch = 0
        for i in range(len(s)):
            loss,SOC_pred,T_pred,batch_t,batch_T_act = calculations(s[i])
            loss_batch += loss
            # Visualizing the results
            if itr %100 == 0:
             plt.figure()
             plt.plot(batch_t.detach().squeeze(),T_pred.detach().squeeze(),'r',label = 'Prediction') # detach is used fot T_pred because it is calculated from the odeint_adjoint. And its good practice to use detach() when plotting
             plt.plot(batch_t.detach().squeeze(),batch_T_act.detach().squeeze(),'b',label = 'Actual') 
             plt.xlabel('Time (s)')
             plt.ylabel('Temperature(K)')
             plt.title(f"Temperature prediction for the epoch {itr} for the data set {s[i]}")
             plt.legend()
             plt.savefig(f"Temperature prediction for the data set {s[i]} for epoch {itr}")
             plt.show()
        
             plt.figure()
             plt.plot(batch_t.detach().squeeze(),SOC_pred.detach().squeeze(),'r',label = 'Prediction')
             plt.title(f" SOC prediction for the epoch {itr} for the data set {s[i]}")
             plt.xlabel('Time (s)')
             plt.ylabel('SOC')
             plt.legend()
             plt.show()
        
            
        loss_batch.backward()
        return loss_batch
        
    loss_total.append(loss_batch)
    epch.append(itr)
    optimizer.step(closure)
    end_time = time.time()

    if itr %100 == 0:
        torch.save(func.state_dict(),f'func_paramsLBGS_{itr}.pth')
    print('Epoch',itr,'Finished','Time Taken is',end_time-start_time)
plt.figure()
plt.plot(epch,loss_total)
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.title('Loss vs epoch')
print('\n training complete')
</code></pre>
<p>I am trying to implement LBFGS in Pytorch. This is my code. It is working well. But I am not sure whether this is the right way to implement it.<br>
When I plot my loss_total all the values it is zero. So I think there is a mistake somewhere. Can someone help me with it</p>
Please help, error loading fbgemm.dll

asor-1: <p>I have tried to uninstall Pytorch and reinstall it. I have uninstalled cuda and reinstalled it and nothing has worked for me. I have provided the error. What am I doing wrong.</p>
<p>(myenv) PS C:\Users\adida\Desktop\OpenMind_bot&gt; python flask_app.py<br>
Traceback (most recent call last):<br>
File “C:\Users\adida\Desktop\OpenMind_bot\flask_app.py”, line 1, in <br>
import torch<br>
File “C:\Users\adida\Desktop\OpenMind_bot\myenv\Lib\site-packages\torch_<em>init</em>_.py”, line 148, in <br>
raise err<br>
OSError: [WinError 126] The specified module could not be found. Error loading “C:\Users\adida\Desktop\OpenMind_bot\myenv\Lib\site-packages\torch\lib\fbgemm.dll” or one of its dependencies.<br>
(myenv) PS C:\Users\adida\Desktop\OpenMind_bot&gt;</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/failed-to-import-pytorch-fbgemm-dll-or-one-of-its-dependencies-is-missing/201969/27">This topic</a> might be relevant.</p>
Out of memory error on GH200

Jared4: <p>Hi, I am running inference on a HF llama 70B model with pytorch backend. The model initially uses the GPU memory and then quickly runs out memory. This GH200 has unified memory. Is there anything I can do to use this unified memory so when the model inference runs of GPU memory it starts using the host memory? Does pytorch support memory spill over to host memory?</p>
<p>thanks</p>

ptrblck: <p><a class="mention" href="/u/syed-ahmed">@syed-ahmed</a> has an open PR to allow fur custom memory allocations via <code>torch.cuda.MemPool</code> <a href="https://github.com/pytorch/pytorch/pull/131152">here</a>, but it’s not merged yet.</p>

Jared4: <p>Thanks. From what I understand is with this pytorch memory pooling, I could pool the GPU and host memory into a single memory pool and this would allow spillover from GPU to host memory?</p>
RuntimeError: The size of tensor a (3) must match the size of tensor b (512) at non-singleton dimension 1

suri_g: <p>Hi All,<br>
I’m working on the kaggle comp Chatbot Arena Human Preference Predictions , I created the tokinization for that , But I got this error. please help me , how to resolve this</p>
<blockquote>
<blockquote>
<p>tokenizer = AutoTokenizer.from_pretrained(‘microsoft/deberta-v3-base’)<br>
model = AutoModelForSequenceClassification.from_pretrained(‘microsoft/deberta-v3-base’, num_labels=3)<br>
max_length = 512<br>
def tokenize_function(examples):<br>
return tokenizer(examples[‘prompt’], examples[‘response_a’], examples[‘response_b’], padding=“max_length”, truncation=True,max_length=max_length)</p>
<p>train_dataset = train_dataset.map(tokenize_function, batched=True)<br>
test_dataset = test_dataset.map(tokenize_function, batched=True)</p>
</blockquote>
<p>I got this error:<br>
training_args = TrainingArguments(</p>
<p>output_dir=“kaggle/working/bert_mode”,<br>
report_to=‘none’,<br>
learning_rate=2e-5,<br>
per_device_train_batch_size=3,<br>
per_device_eval_batch_size=3,<br>
num_train_epochs=2,<br>
weight_decay=0.01,<br>
evaluation_strategy=“epoch”,<br>
save_strategy=“epoch”,<br>
load_best_model_at_end=True,<br>
)</p>
<p>trainer = Trainer(</p>
<p>model=model,<br>
args=training_args,<br>
train_dataset=train_dataset,<br>
eval_dataset=test_dataset,<br>
tokenizer=tokenizer,<br>
data_collator=data_collator,<br>
#compute_metrics=compute_metrics,<br>
)</p>
<p>trainer.train()</p>
<p>File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1527, in Module._call_impl(self, *args, **kwargs)<br>
1522 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1523 # this function, and just call forward.<br>
1524 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks<br>
1525         or _global_backward_pre_hooks or _global_backward_hooks<br>
1526         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1527     return forward_call(*args, **kwargs)<br>
1529 try:<br>
1530     result = None</p>
<p>File /opt/conda/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:1335, in DebertaV2ForSequenceClassification.forward(self, input_ids, attention_mask, token_type_ids, position_ids, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)<br>
1333     else:<br>
1334         log_softmax = nn.LogSoftmax(-1)<br>
 → 1335         loss = -((log_softmax(logits) * labels).sum(-1)).mean()<br>
1336 elif self.config.problem_type == “regression”:<br>
1337     loss_fct = MSELoss()</p>
<p>RuntimeError: The size of tensor a (3) must match the size of tensor b (512) at non-singleton dimension 1</p>
</blockquote>

ptrblck: <p>Could you post the shapes of <code>logits</code> and <code>labels</code> before calling:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = -((log_softmax(logits) * labels).sum(-1)).mean()
</code></pre>
Why pytorch use different lambda options for lerp?

cyk2018: <p>here, this <a href="https://github.com/pytorch/pytorch/commit/b5ad90932ab990d3cb013d13f789d8f04c3cc8fa#diff-18a0c3e7cd07089b34760028c387d39de1aa393172f1d5f123692106b3a3d144" rel="noopener nofollow ugc">link</a> update the Lerp.cu. And it use <span class="chcklst-box fa fa-square-o fa-fw"></span> for tensor lerp but [=] for scalar lerp, I am confused why this choice, can Anyone give me some helps? Thanks.<br>
I don’t know if this is suitable to commit an issue so I asked here. If you think this should be asked in github repo, I will update.</p>

ptrblck: <p>CC <a class="mention" href="/u/khushi-411">@khushi-411</a> as the PR author - do you know why different capture clauses were used?</p>
Detach() and requires_grad_() again?

Do_Thanh_Dat_LE: <p>Hi,<br>
I am training a variant of VAE. I would like to store a tensor <code>z_v</code> for each variety and a tensor  <code>z_e</code> for each environment in a dictionary after training one mini-batch. Then I want to use them in training the next mini-batch and update them. If I do not use <code>detach()</code>, I will run into the the error that the variables needed for gradient computation has been modified by an inplace operation. But if I use <code>detach()</code>, I need to use <code>requires_grad_()</code> again when I use the tensors <code>z_v</code> and <code>z_e</code> in the dictionary to compute the reconstruction term. Is it ok to do like that (<code>detach()</code> to store and <code>requires_grad_()</code> again) ? Thank you.</p>

ptrblck: <aside class="quote no-group" data-username="Do_Thanh_Dat_LE" data-post="1" data-topic="206956">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/do_thanh_dat_le/48/66316_2.png" class="avatar"> Do_Thanh_Dat_LE:</div>
<blockquote>
<p>Is it ok to do like that (<code>detach()</code> to store and <code>requires_grad_()</code> again) ?</p>
</blockquote>
</aside>
<p>Yes, it’s fine and also the recommendation based on this warning:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(1, requires_grad=True)
y = torch.tensor(x)
# UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
</code></pre>
RuntimeError: CUDA error: device-side assert triggered CUDA 

yusako: <p>I am trying to run my project on Google Colab’s L4 GPU. However, I get the error:</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
</code></pre>
<p>Its weird because everytime I run the project, this error seems to pop up in different places.</p>
<p>I also checked <a href="https://discuss.pytorch.org/t/runtimeerror-cuda-error-device-side-assert-triggered-cuda-kernel-errors-might-be-asynchronously-reported-at-some-other-api-call-so-the-stacktrace-below-might-be-incorrect/139735" class="inline-onebox">RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect</a>. I also use <code>nn.CrossEntropyLoss()</code>, but the error occurs before that line.</p>
<p>I’m not sure what is the problem. Could someone help me?</p>

egor.chil.1999: <p>Hi! Do you use pin_memory or/and non_blocking data transfers?</p>

yusako: <p>Yeah, I used pin_memory.</p>
<p>I actually figured it out, turns out there was a place in my code where there was index out of bounds error. I fixed it and the error no longer appeared.</p>

ptrblck: <aside class="quote no-group" data-username="yusako" data-post="3" data-topic="206937">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/eb9ed0/48.png" class="avatar"> yusako:</div>
<blockquote>
<p>turns out there was a place in my code where there was index out of bounds error. I fixed it and the error no longer appeared.</p>
</blockquote>
</aside>
<p>Great! Thanks for the follow-up as device assert issues are usually caused by invalid indexing.</p>