akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dealing with in-place operation in custo function

Vendrick17: <p>Hello, I am trying to implement the function below and compute its gradient using <code>backward()</code>, but I have an in-place operation that I cannot solve, below I show an option I tried and didn’t work.</p>
<pre><code class="lang-auto">def modified_gram_schmidt(A):
    m, n = A.shape

    Q = torch.zeros((m, n), requires_grad=True).clone()
    R = torch.zeros((n, n), requires_grad=True).clone()

    for j in range(n):
        v = A[:, j].clone()
        for i in range(j):
            R[i, j] = torch.dot(Q[:, i], v)
            v_prev = v.clone()
            v = v_prev - torch.dot(Q[:, i], v_prev) * Q[:, i]
        R[j, j] = torch.norm(v)
        v = v / torch.norm(v)
        Q[:, j] = v.clone()

    return Q, R
</code></pre>
<p>The operation which is causing the issue is <code>v = v_prev - torch.dot(Q[:, i], v_prev) * Q[:, i]</code>, when I try to compute the gradient by <code>Q.sum().backward()</code> I have the error: <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</code>. I would appreciate any help.</p>

albanD: <p>Hi,</p>
<p>A simple solution is just to remove the inplace change into Q and only create it at the end:</p>
<pre><code class="lang-auto">def modified_gram_schmidt(A):
    m, n = A.shape

    Q = []
    R = torch.zeros((n, n), requires_grad=True).clone()

    for j in range(n):
        v = A[:, j].clone()
        for i in range(j):
            R[i, j] = torch.dot(Q[i], v)
            v_prev = v.clone()
            v = v_prev - torch.dot(Q[i], v_prev) * Q[i]
        R[j, j] = torch.norm(v)
        v = v / torch.norm(v)
        Q.append(v)

    return torch.stack(Q), R
</code></pre>

Vendrick17: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="177356">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<pre><code class="lang-auto">def modified_gram_schmidt(A):
    m, n = A.shape

    Q = []
    R = torch.zeros((n, n), requires_grad=True).clone()

    for j in range(n):
        v = A[:, j].clone()
        for i in range(j):
            R[i, j] = torch.dot(Q[i], v)
            v_prev = v.clone()
            v = v_prev - torch.dot(Q[i], v_prev) * Q[i]
        R[j, j] = torch.norm(v)
        v = v / torch.norm(v)
        Q.append(v)

    return torch.stack(Q), R
</code></pre>
</blockquote>
</aside>
<p>Thank you so much for the quick reply, that solved the problem!</p>
Loading Sequential Data of Varying Lengths into GPU

akshathsa: <p>Hi, so I have time series data collected on different individuals of varying lengths. I’m trying to train an LSTM on that data using PyTorch. I followed some threads on here to successfully implement a first attempt. I extended the Dataset class to which takes in my data as a Pandas series containing numpy arrays and correctly returns (features, labels) tuples of arrays which correspond to the data for each individual. Then I use a DataLoader with a custom collate function which uses pack_sequence to pad the arrays on a per-batch basis and pack them. I want to pad the data to the length of the longest sequence in a mini-batch instead so I avoid unnecessarily padding all of the data to the length of some outlier series. From my understanding, this is still all on the CPU. Then in my training loop when I get each mini-batch, I send them to the GPU. Using CUDA in this fashion has cut my training time by a factor of 5, but I’m wondering if all the data transfers from the CPU to GPU are still slowing it down.</p>
<p>Since my dataset is relatively small, I wanted to try putting it all on the GPU and see if that improves running time. From what I’ve seen online, a good way to achieve this is to send it all to the GPU in the <strong>init</strong> method of the Dataset class I’m using. However, from my understanding, data has to be in PyTorch Tensors if I want to send them to the GPU, and since the data isn’t padded at this point yet and I can’t stack Tensors of different lengths, I’m running into a problem.</p>
<p>Is there a solution to this or perhaps a better way to achieve what I’m trying to do? I’m relatively new to PyTorch, so I may very well be unaware of better options. Thank you in advance for any and all help!</p>

ptrblck: <p>I would recommend profiling the code first to check where the bottleneck is before trying to optimize one part of the code, which might not slow down the training at all.<br>
To do so you could use the native PyTorch profiler or e.g. Nsight Systems, which would give you an overview of the current processing.<br>
Preloading the data and moving it directly to the GPU could work with the drawback of a potentially slower startup time and more GPU memory usage.</p>

akshathsa: <p>Alright, thanks for the suggestion! I’ll try using a profiler and will follow up.</p>
In_features , out_features in nn.Linear

Oualid: <p>I have a question regarding <strong>nn.Linnar</strong>.</p>
<p>In the documentation it says:</p>
<p>Parameters:<br>
<strong>in_features</strong> – size of each input sample<br>
<strong>out_features</strong> – size of each output sample</p>
<p>As an example we have the following model:</p>
<pre><code class="lang-auto">model_cnn = nn.Sequential(nn.Conv2d(1, 32, 3, padding=1), nn.ReLU(),
                          nn.Conv2d(32, 32, 3, padding=1, stride=2), nn.ReLU(),
                          nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(),
                          nn.Conv2d(64, 64, 3, padding=1, stride=2), nn.ReLU(),
                          Flatten(),
                          nn.Linear(7*7*64, 100), nn.ReLU(),
                          nn.Linear(100, 10)).to(device)

</code></pre>
<p>Where do the values (7*7*64, 100) in the first nn.Linear and (100, 10) in the second nn.Linear come from?<br>
I can’t figure out how these values are calculated.</p>

spacecraft1013: <p>It is the shape of the features after coming out of the final conv2d layer, it is a tensor of shape (64, 7, 7). The <code>Flatten()</code> layer will just flatten these features into a 1 dimensional tensor, which will be of shape (64*7*7,), which is what get’s inputted into the <code>Linear</code> layer</p>

Tejan_Mehndiratta: <p>Hi,</p>
<p>The output of your last Conv2d  would be like (N, 64, 7, 7), where N stands for batch_size, 64 for number of channels and 7x7, the height and width of the image.<br>
So, now Flatten() will convert this into shape (N, 64 x 7 x 7). Now, when it will go to the first Linear, the output will be (N, 100) and after second Linear (N, 10).</p>
<p>Please do ask if you are still confused…</p>

Oualid: <p>Thank you for the explanation.<br>
But how do we know that the size of the image is 7x7 in the final conv2d?</p>

Tejan_Mehndiratta: <p>Oh! See, I use a trick. Before using the linear or the flatten layer, you run the model on a dummy sample by passing say torch.randn(32, 3, 60, 60), where 32 is the batch_size, 3 is the input num_channels and 60x60 is the dimension of the images. The output you get will have a shape of (N, out_channels, height, width). So, this is how you can get the output of the last conv2d layer, otherwise you have to calculate the shape manually.</p>

Oualid: <p>OK, thank you for the explanation.</p>

Kailasha_Sahoo: <p>Hi, I have a question here?</p>
<p>How out features is calculated?<br>
According to liner equation: y = Ax + b</p>
<p>How dimensions of Y getting altered with given out-feature parameter?</p>
PyTorch; forward without for-loop with multiple neural networks

JinraeKim: <p>Hi,</p>
<p>I’d like to replace a for-loop for multiple NNs with something like matrix operation GPU usage.</p>
<p>(Note: the following code is conceptual; would not be runnable)<br>
For example, I have a bunch of NNs, which are contained in a <code>torch.nn.ModuleList</code>.<br>
That is,</p>
<pre><code class="lang-auto">my_list = torch.nn.ModuleList()
for _ in range(N):
    my_list.append(construct_my_NN())
</code></pre>
<p>Now, in the method <code>forward</code>, I use the output of all NNs generated before.</p>
<pre><code class="lang-auto">def forward(self, xs):
    d = xs.shape[0]
    ys = torch.empty(d, 1)
    for i in range(N):
        ys[i] = my_list[i](xs)
    return my_custom_func(ys)
</code></pre>
<p>
Note that the <code>torch.sum</code> can be a different custom function.
<br>
I think this makes forward method very slow.<br>
Can I replace the for-loop with a single operation (something like matrix operation)?</p>
<p>Or, any ideas to detour this issue?</p>
<p>EDIT: if one would replace the multiple NNs with a single NN with high-dimensional outputs.<br>
For example, the above forward method is replaced with</p>
<pre><code class="lang-auto">def forward(self, xs):
    d = xs.shape[0]
    ys = huge_NN(xs)
    return my_custom_func(ys)
</code></pre>
<p>I think the two cases may be different in terms of numerical sensitivity or training performance.<br>
Note that I actually implemented the two cases and found that, of course, they showed different learning progress.<br>
Can you guys give me an intuition for this?</p>

Clion: <p>Hey did you ever find a solution for this? I’m trying to figure out the exact same thing. Pretty straightforward in tensorflow, but cannot for the life of me figure out how to do it in torch</p>

bdhirsh: <p>Have you taken a look at model ensembling using ‘vmap’? <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>
<p><a href="https://pytorch.org/functorch/stable/notebooks/ensembling.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/functorch/stable/notebooks/ensembling.html</a></p>

Clion: <p>Holy moly, ty for this!</p>
Ensembled LSTM fails on inference

Clion: <p>Alright, here’s the issue I’m facing. I created a model ensemble using functorch’s combine_state_for_ensemble. The purpose of this is so I can run the same model with multiple different values for its parameters in parallel without having to loop over a forward pass for each version of this model one at a time. The model takes three different inputs and produces one output - most of the model is just a bunch of linear layers, but part of it is an LSTM; one of the three inputs is a sequence, the other two are just simple vectors. This is the error I’m getting:</p>
<pre><code class="lang-auto">  File "/home/USERNAME/miniconda3/envs/ENVNAME/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 774, in forward
    result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
RuntimeError: Batching rule not implemented for aten::lstm.input. We could not generate a fallback.
</code></pre>
<p>The following is how I create/call the ensemble:</p>
<pre><code class="lang-auto">class MultiModel:

	def __init__( self, mFile="", iterSpan=-1 ):

		GPU = pt.device( 'cuda:0' )
		self.IterSpan = iterSpan; self.device = GPU

		ValNets = []

		if iterSpan &lt; 1:
			ValNet_0 = ValueNet( modelIter=iterSpan )
			ValNet_0.to( GPU ); ValNets.append( ValNet_0 )

		for t in range( 1,iterSpan+1 ):
			ValNet_t = ValueNet( modelIter=t, load_from_file=mFile )
			ValNet_t.to( GPU ); ValNets.append( ValNet_t )

		multimodel, multiparameters, multibuffers =\
			combine_state_for_ensemble( ValNets )

		self.MM = multimodel
		self.MP = multiparameters
		self.MB = multibuffers

	def __call__( self, hInputs, oInputs, aInputs ):

		#in_dims is what it is because the intention is to feed the 
		#same batch of data into all models in the ensemble
		return vmap( self.MM, in_dims=(0,0,None,None,None) )(
			self.MP, self.MB, hInputs,oInputs,aInputs )
</code></pre>
<p>If it’s relevant, I have no need to use this for training - the individual ValNets are trained in isolation and only need to be ensembled for inference. Can post the code defining exactly what a ValNet is too but I’ll wait to be asked for that one.</p>
How to speed up convolution with kernels that contain many zeros?

iminfine: <p>Suppose there are two set of kernels with a same shape and almost half elements in the kernels are zeros. For the same input x, the kernels are used to calculate convolution respectively. See  example code blow:</p>
<pre><code class="lang-auto">initial
conv=nn.conv2d(c,n,k)  # shape(n,c,k,k)

forward 
kernel1=F.relu(conv.weight) 
kernel2=F.relu(-conv.weight) 
out1=F.conv2d(x,kernel1)
out2=F.conv2d(x,kernel2)
</code></pre>
<p>As 0 multiple anything is 0, near half of the multiplications in the convolutions are unnecessary. My question is:<br>
Is there possible to speed up the two convolutions ? Can the two convolutions calculate in parallel ?</p>
Semi-supervised learning problem

Katono88: <p>Im currently working on semi-supervised learning problems, and when Im confused of the idea that semi-supervised learning can use unlabeled samples to make predictions,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c74c94b599e1c0ce5b555873120c429d04ea321.png" alt="image" data-base62-sha1="hKZq1Sb8QbhvSpfo2w2udRXAFCp" width="560" height="179"><br>
For example this picture, it aims to predict if the middle point is positive or negative, and it observes the other unlabeled samples(those gray ones) and label it with <strong>positive with high confidence</strong>.</p>
<p>How does it work? From my point of view, I don’t know why the middle point can be positive.</p>
Spawning DDP processes with multiprocessing randomly reads files from disk

jwillette: <p>I do not know if this behavior comes from the multiprocessing module, or PT itself, but I find that when I start running a model with DDP, it randomly reads the files from disk sometimes in the middle of the processes which prevents me from being able to change anything on the disk when the process is running.</p>
<p>This can cause an annoyance when I want to run a model and then implement something else in the codebase while that is running, because if there is ever an intermediate save which has some kind of syntax error, then the whole running program will crash.</p>
<p>If it were a non-DDP module, I could start the code, and then once the code is loaded into memory, I could go on implementing whatever I need to which would not affect the running process.</p>
<p>Does this behavior come from PT or is it part of <code>mp.spawn</code>? Is there any way to avoid accidentally crashing the running program without being forced to work on a different machine?</p>

wanchaol: <p>you can refer to a similar issue here <a href="https://discuss.pytorch.org/t/ddp-is-affected-by-code-modification/148046" class="inline-onebox">DDP is affected by code modification</a></p>

jwillette: <p>hmm strange… That posts links to a SO post which says that Linux does not re-read the code from file. I am using Linux (Ubuntu 18.04) and it is re-reading the code from file.</p>
<p>I guess there may be nothing I can do about it…</p>

ptrblck: <p>From my linked post:</p>
<blockquote>
<p>seems to depend on the OS or rather the method to fork or spawn the processes</p>
</blockquote>
<p>The OS selects the default start method, but assume you are using the <code>spawn</code> start method Linux would also re-read the file.</p>
Is nn.MultiheadAttention attn_mask working differently in pytorch 2.0?

Thibault712: <pre><code class="lang-auto">    return torch._native_multi_head_attention(
RuntimeError: Mask shape should match input. mask: [2048, 172, 172] input: [128, 16, 172, 172]
</code></pre>
<p>they are talking to me about a problem of dimension but 2048 = batchsize * num_heads = 128 *16 (172=L, 172=S, self attention in my case)</p>
<p>According to the <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html" class="inline-onebox" rel="noopener nofollow ugc">MultiheadAttention — PyTorch 2.0 documentation</a></p>
<p>It’s supposed to be of the right shape</p>

soulitzer: <p>Yes the number of elements is correct, but you need to match the shape exactly. You can do that by calling mask.flatten(0, 1).</p>

Thibault712: <p>No in my error I do have the good dimension for the mask, it’s the “input” dimension, which is derived by the module MultiheadAttention, I can’t modify it there.</p>
<p>Furthermore if I can make a suggestion I feel this is less intuitive for reading to do (batch_size * num_heads,…) also since 95% of the time you apply same mask on all heads it would make more sense the other way cause you could just broadcast over the dimension 1 that corresponds to heads:<br>
like with a mask (batch_size, 1, source, target)</p>

soulitzer: <p>Hmm do you have a short self-contained snippet for reproduction?</p>

Alexie: <p>Hi, I’ve just encountered the exact same issue, have you found a way to resolve?</p>
Confused why GPU total capacity is bigger than PyTorch reserved memory(23.69 GiB vs 20.79 GiB)

ztysdu: <p>Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.47 GiB (GPU 3; 23.69 GiB total capacity; 19.38 GiB already allocated; 1.44 GiB free; 20.79 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
<p>Dear all,<br>
I am confused why GPU 3 total capacity is 23.69 GiB while PyTorch only reserved  20.79 GiB.<br>
I am sure that no other process is running on the GPU<br>
I tried to set os.environ[“PYTORCH_CUDA_ALLOC_CONF”] = “max_split_size_mb:1024”.<br>
However, it increased nearly 30% of the  computation time. It can be seen that the required<br>
memory is only little more than the free memoey</p>

ptrblck: <p>Besides the memory allocations PyTorch owns your GPU will also use memory for the driver, for each loaded CUDA kernel etc. Depending on the GPU, the CUDA version, if CUDA’s lazy module loading is enabled (it’s enabled for all PyTorch binaries shipping with CUDA &gt;= 11.7), the loaded libraries, the actually used memory would differ.</p>
How to assign a tensor to another tensor at different rows and columns?

Charlie0257: <p>For example, I have a tensor</p>
<pre><code class="lang-auto">a = tensor([[ 0,  1,  2,  3,  4],
        [ 5,  6,  7,  8,  9],
        [10, 11, 12, 13, 14],
        [15, 16, 17, 18, 19]])
</code></pre>
<p>indices matrixes are <code>b = torch.LongTensor([0, 2, 3])</code> and <code>c = torch.LongTensor([2, 4])</code></p>
<p>I can get the d = a[b,:][:,c], but how can I assign the <strong>-d</strong> to <strong>a[b,:][:,c]</strong>?</p>
<p>Furthermore, how can I assign the tensor <strong>having the same dimensions as d</strong> to <strong>a[b,:][:,c]</strong>?</p>

ptrblck: <p>I assume <code>a[b,:][:,c]</code> represents the working reference and you would like to assign a new value to these indexed values?<br>
If so, this would work:</p>
<pre><code class="lang-python">a = torch.tensor([[ 0,  1,  2,  3,  4],
                  [ 5,  6,  7,  8,  9],
                  [10, 11, 12, 13, 14],
                  [15, 16, 17, 18, 19]])

b = torch.tensor([0, 2, 3])
c = torch.tensor([2, 4])

a[b.unsqueeze(1), c] = 1000
print(a)
# tensor([[   0,    1, 1000,    3, 1000],
#         [   5,    6,    7,    8,    9],
#         [  10,   11, 1000,   13, 1000],
#         [  15,   16, 1000,   18, 1000]])
</code></pre>

Charlie0257: <p>Thanks for your reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I found it works even if a[b.unsqueeze(1), c] = d (where d is a 3x2 tensor)!<br>
e.g.,</p>
<pre><code class="lang-auto">d = tensor([[ -2,  -4],
        [-12, -14],
        [-17, -19]])
</code></pre>
<p>Thank you very much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
RTX 3060 vs RTX 3090 Benchmarks - Tested Torch 1.13, Torch 2, cudNN 8.8.0.1, xFormers, OPT-SDP-Attention, DreamBooth, IT/s, NansException all NaNs Solution, Watt Usage, Dual Cards Performance

FurkanGozukara: <p>Tested so many stuff. I hope you enjoy and leave a comment</p>
<p>28.) Automatic1111 Web UI - PC - Free<br>
<a href="https://youtu.be/lgP1LNnaUaQ" rel="noopener nofollow ugc"><strong>RTX 3090 vs RTX 3060 Ultimate Showdown for Stable Diffusion, ML, AI &amp; Video Rendering Performance</strong></a><br>
<a href="https://youtu.be/lgP1LNnaUaQ" rel="noopener nofollow ugc"><img src="https://user-images.githubusercontent.com/19240467/231295610-478e7a21-6763-4148-941f-dce6441c5ab0.png" alt="image" width="640" height="360"></a></p>
<div class="youtube-onebox lazy-video-container" data-video-id="lgP1LNnaUaQ" data-video-title="RTX 3090 vs RTX 3060 Ultimate Showdown for Stable Diffusion, ML, AI &amp; Video Rendering Performance" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=lgP1LNnaUaQ" target="_blank" rel="noopener nofollow ugc">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/lgP1LNnaUaQ/maxresdefault.jpg" title="RTX 3090 vs RTX 3060 Ultimate Showdown for Stable Diffusion, ML, AI &amp; Video Rendering Performance" width="690" height="388">
  </a>
</div>
Register full backward hook vs. register module full backward hook

mike3: <p>I am trying to understand the differences between the functions in the title. What are the appropriate use cases for both? I especially would like to see an example where register_module_full_backward_hook is used properly. Thank you!</p>

soulitzer: <p>(there’s actually a bug if you use nightly/build from source that needs to be fixed!)<br>
edit: Actually I wasn’t aware that the bug was very recently reverted. So only certain nightlies would be broken.</p>
<p>Otherwise, this is how it should be used:</p>
<pre><code class="lang-auto">import torch

a = torch.rand(10, requires_grad=True)
lin = torch.nn.Linear(10, 10)

def hook(mod, grad_inputs, grad_outputs):
    # grad_outputs is what your module's backward receives as the gradient
    # grad_inputs is the gradient wrt the module's inputs (in this case the
    # input is `a``)
    assert mod is lin
    return (grad_inputs[0] * 100,)

torch.nn.modules.module.register_module_full_backward_hook(hook)

out = lin(a)
out.sum().backward()
print(a.grad)
</code></pre>

mike3: <p>Thank you! Follow up question: how does register_module_full_backward_hook(hook) know which module to be called on? Or will it be called whenever .backward() is called (with the module as an argument), no matter on which module?</p>

soulitzer: <p>Yeah it would be called whenever .backward() is called and it’d be applied to all modules. To figure out which module its being called for you’d look at the first argument yes.</p>
Final Model Training Problem - Overfitting

Zelreedy: <p>Hello. I am working on a CNN project for multiclass classification. I implemented hyperparameter optimization to find the most suitable model, during which I got a best accuracy of 97.38%. I then took this model and applied Early Stopping to it, but the results were much worse. To start, the accuracy dropped greatly. Second, the model seems to be overfitting the data, as shown in the learning curve below. Test loss stops dropping after the shown point, and the training loss goes down to almost zero. I have tried implementing some techniques like Dropout, weight decay, and using different batch sizes, but the issue still persists: the train loss goes down rapidly and converges quickly while the test loss saturates at a specific point and does not yield good prediction results. What are some other steps which I can implement to solve this issue. Can I fix the model during Early Stopping straight away, or do I have to rerun the hyperparameter optimization from the beginning? Also, what is the reason that the same model performs exceptionally well in K-Fold, but not in Early Stopping?</p>
<p><a href="https://i.stack.imgur.com/8iOme.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/8iOme.png" alt="enter image description here" width="690" height="345"></a></p>
How to export Pytorch model to ONNX with variable-length tensor loop?

payne_zhang: <p>I simplify my complex Pytoch model like belows.</p>
<pre><code class="lang-python">import torch
from torch import nn
import onnx
import onnxruntime
import numpy as np


class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.template = torch.randn((1000, 1000))

    def forward(self, points):
        template = self.template
        points = points.reshape(-1, 2)
        heatmaps = [template[point[0]:point[0] + 10, point[1]:point[1] + 20] for point in points]
        return heatmaps


model = Model()
points = torch.randint(100, 200, (1, 8, 2))


torch.onnx.export(model, args=points, f='toy.onnx',
                  export_params=True,
                  opset_version=13,
                  do_constant_folding=True,
                  verbose=False,
                  input_names=['input1'],
                  output_names=['output1'],
                  dynamic_axes={'input1': {0: 'batch_size'},
                                'output1': {0: 'batch_size'},
                                }
                  )

session = onnxruntime.InferenceSession("./toy.onnx")
inputs = np.random.randint(100, 200, (2, 8, 2))
ort_inputs = {'input1': inputs}
ort_outs = session.run(None, ort_inputs)
</code></pre>
<p>I can export Pytoch model to ONNX successfully, but when I change input batch size I got errors.</p>
<pre><code class="lang-auto">onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Split node. Name:'Split_3' Status Message: Cannot split using values in 'split' attribute. Axis=0 Input shape={16,2} NumOutputs=8 Num entries in 'split' (must equal number of outputs) was 8 Sum of sizes in 'split' (must equal size of selected axis) was 8
</code></pre>
<p>I know this error caused by loop through a dynamic sized tensor, but I don’t know how to solve this problem. <strong>Note, I cannot move this operation out of model due to some restrictions.</strong></p>
<p>Any suggestions will help me, thanks.</p>
Backendcompent are different in torchgen/model.py and c10/core/DispatchKey.h

small-chen-wen: <p>why backendcompent definition are different in torchgen/model.py and c10/core/DispatchKey.h<br>
the number of backendcompent are different<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4ce0f3a3b1884dbba4c1e34b201899042c86c551.png" alt="image" data-base62-sha1="aY6fabdlGrzxDHRBZwzoctlsu09" width="581" height="426"><br>
the number in c10/core/DispatchKey.h is 15 but in torchgen/model.py is 14</p>
Error on shape in tensor multiplication

ado_sar: <p>I am experimenting with Pytorch and I want to build a 3D CNN. My input is a tensor of size <code>25x25x25</code> and the output is a binary <code>[0, 1]</code>. I have defined the following architecture:</p>
<pre><code class="lang-auto"> # Architecture of CNN
 class Net(nn.Module):
     def __init__(self):
         super().__init__()
         self.conv1 = nn.Conv3d(in_channels=1, out_channels=3, kernel_size=5)
         self.pool = nn.MaxPool3d(kernel_size=3) # Stride equals `kernel_size`
         self.fc1 = nn.Linear(7*7*7*3, 20)
         self.fc2 = nn.Linear(20, 5)
         self.fc3 = nn.Linear(5, 2)
 
     def forward(self, x):
         x = self.pool(F.relu(self.conv1(x)))
         x = torch.flatten(x, 1)
         x = F.relu(self.fc1(x))
         x = F.relu(self.fc2(x))
         x = self.fc3(x)
         return x


</code></pre>
<p>I am getting the following error: <code>RuntimeError: mat1 and mat2 shapes cannot be multiplied (3x343 and 1029x20)</code>. Note that <code>3*343 == 1029</code> and I am flatteting except the batch size dimension.</p>

ptrblck: <p>Your model works fine for me:</p>
<pre><code class="lang-python">model = Net()

x = torch.randn(2, 1, 25, 25, 25)
out = model(x)
print(out.shape)
# torch.Size([2, 2])

x = torch.randn(64, 1, 25, 25, 25)
out = model(x)
print(out.shape)
# torch.Size([64, 2])
</code></pre>

ado_sar: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the answer and your time. I tried your snippet and works fine. The error happened during training where I forgot to reshape my training inputs to include 1 channel.</p>
Why it's hard to deal with the loop and condition in PyTorch graph capture?

111357: <p>Why it’s hard to deal with the loop and condition in PyTorch graph capture? but cpp can deal with loops and conditions well?</p>

ptrblck: <p>CUDA Graphs captures the workload and is able to significantly reduce the launch overheads by replaying the captured work.<br>
This also requires to read and write to the same (virtual) memory addresses and to avoid CPU workload.<br>
Conditions in Python code are executed on the host and are thus not captured.<br>
<a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-graphs">These docs</a> explain the limitations in more detail.</p>
Skiping training iteration in DDP setting results in stalled training

danish_nazir: <p>Hi All,</p>
<p>I want to skip training iteration for a single GPU when training loss is greater than 10,000 (for example). The naive way of doing it is as follows.</p>
<pre><code class="lang-auto">  for x in train_dataloader:
            
            optimizer.zero_grad()

            out = model(x)

            out_criterion = criterion(out, x)
           
            out_criterion.backward()

            if out_criterion &gt; 10000: #skipping optimizer update
                continue

            optimizer.step()
</code></pre>
<p>However, this results in “stalled training” due to synchronizaton issue between the GPU’s. Is there any other efficient way of achieving this especially in the DDP setting?</p>

ptrblck: <p>Maybe you could <code>allreduce</code> the losses, check its value, and skip the <code>step()</code> on all ranks instead of checking it on each rank separately which would diverge the runs.</p>

danish_nazir: <p>Thank you for your reply, Do you mean like this?</p>
<pre><code class="lang-auto">        dist.barrier()
        dist.all_reduce(out_criterion, op=torch.distributed.ReduceOp.SUM)
</code></pre>
<p>The drawback of this is that if we have N GPU’s for training and only single GPU has e.g. out_criterion = 10000, while the others has normal losses, then the gradients of all those other GPU’s will also be wasted…, since we are not taking the optimization step.</p>

ptrblck: <p>Yes, but this would also be expected since all parameters in each rank are equal in each iteration. If you skip the update on one rank your training would diverge since parameters would differ.</p>

danish_nazir: <p>Got it thanks for your response.</p>
Where is. THPVariable_conv2d backward source code

jinyuan.feng: <p>Hi, I just find the conv2d forward code but do not know where to set its backward function.</p>

soulitzer: <p>aten/src/ATen/native/Convolution.cpp is a good starting point, but depending on the inputs/device etc, it will dispatch to different backends</p>

jinyuan.feng: <aside class="quote no-group" data-username="soulitzer" data-post="2" data-topic="177111">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/48.png" class="avatar"> soulitzer:</div>
<blockquote>
<p>ill dispatch to different backen</p>
</blockquote>
</aside>
<p>I have find it. at</p>
<ol>
<li>pytorch/torch/csrc/autograd/generated/python_torch_functionsEverything.cpp</li>
<li>pytorch/torch/csrc/autograd/generated/VariableTypeEverything.cpp</li>
<li>pytorch/torch/csrc/autograd/generated/Functions.cpp</li>
<li>Finally, aten/src/ATen/native/Convolution.cpp</li>
</ol>
ViT for segmentation

dirkcremers: <p>Hi,</p>
<p>Currently, I am trying to apply a ViT transformer as a backbone for my image segmentation.<br>
Using a pre-trained ViT, I obtain the following summary for my images:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3bc16e030887e19907e2d9f3fbd77cba2e233551.png" data-download-href="https://discuss.pytorch.org/uploads/default/3bc16e030887e19907e2d9f3fbd77cba2e233551" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3bc16e030887e19907e2d9f3fbd77cba2e233551.png" alt="image" data-base62-sha1="8wCAHQ3krUxHduG6n47vxenmSnD" width="690" height="254" data-dominant-color="414141"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1009×372 11.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Of course, as one can see, I transformed the output to be of size [1,35] and hence being a classification procedure for each image. Therefore, I would like to transform the last output of the encoder (ViT, either of size [1,35] or the standard [1,1000] size) into the following shape of [1,num_classes,384,384]. This is compatible with an image segmentation task.</p>
<p>Reading several papers online (e.g. <a href="https://arxiv.org/pdf/2105.05633v3.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2105.05633v3.pdf</a>), I find some interesting observations, but they do not clearly indicate how to address such problems in terms of code. Therefore, I was wondering if somebody could point me in the right direction to scale the image back to its original format to perform image segmentation.</p>
<p>Thanks!<br>
Dirk</p>

ptrblck: <p>Often an encoder-decoder model architecture is used where the decoder uses the (low-dimensional) output of the encoder to create the segmentation mask e.g. via transposed convolutions etc.<br>
I would guess the linked paper might apply a similar technique.</p>
DDP, convert_sync_batchnorm and eval()

spnova12: <p>When using DistributedDataParallel (DDP) to train a model with batch normalization, you may encounter the following error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128]] is at version 4; expected version 3 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

</code></pre>
<p>Batch normalization seems to handle batches in an inplace manner, which can cause conflicts when using DDP. To avoid these conflicts, it is recommended to use SYNCBATCHNORM instead of batch normalization when using DDP. You can use <code>nn.SyncBatchNorm.convert_sync_batchnorm</code> to convert the batch normalization layers to SYNCBATCHNORM layers.</p>
<p>However, even after converting the batch normalization layers, you may encounter the same error if the model is in <code>eval()</code> mode. In this case, you need to set <code>broadcast_buffers=False</code> in your model to avoid synchronization errors.</p>
<p><strong>Is it possible to train a model with batch normalization using DDP while the model is in <code>eval()</code> mode with <code>broadcast_buffers=True</code>?</strong></p>
<p>(I referred to <a href="https://stackoverflow.com/questions/66165366/trying-to-use-distributed-data-parallel-on-gans-but-getting-runtime-error-about" rel="noopener nofollow ugc">this post</a> for guidance.)</p>

wanchaol: <p>if your model is in <code>eval()</code> model, then you are not suppose to train the model, you might need to put your model under <code>trainI()</code> model first and a lot of nn.Module relies on <code>eval</code> or <code>train</code> to determine whether their parameters need to record gradient or not (i.e. batch norm layer as you observed).</p>
<p>can you describe why you want to train the model under <code>eval</code> mode?</p>

spnova12: <p>I am using batch normalization in my model, but I don’t want to train it from the beginning.</p>
<p>Instead, I would like to activate the batch normalization training after some initial training. Training batch normalization using the train() mode means that the network learns to satisfy the distribution of the entire batch, which can be challenging during the initial training phase.</p>
<p>Therefore, I have been using eval() mode to suppress batch normalization’s ability to see the entire batch during the initial training phase. I want to continue using eval() mode during training so that I can activate batch normalization training later in the process when the training has become more stable.</p>
Accumulating grads with DDP with one forward pass and multiple backwards

bary: <p>Hi,<br>
In my training I need to perform a <code>forward</code> pass, followed by multiple <code>backward</code> passes with <code>retain graph = True</code>, and finally using <code>backward</code> final time before <code>optimizer.step</code>.</p>
<p>I do this because I compute a heavy loss that causes cuda OOM before reaching the final <code>backward</code> if I accumulate the ‘partial’ losses before using <code>backward </code> on everything together.</p>
<p>I now want to train on multiple GPUs,but from what I see <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.no_sync" rel="noopener nofollow ugc">here</a> it seems that I can’t use <code>no_sync</code> for my use case, as the forward pass also need to be inside the context.<br>
Another point is that the number of backwards between the final one won’t be the same across the different inputs.</p>
<p>Is there a way for me to disable syncing until the final backward pass?</p>

wanchaol: <p><a class="mention" href="/u/bary">@bary</a> <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html?highlight=distributeddata#torch.nn.parallel.DistributedDataParallel" class="inline-onebox" rel="noopener nofollow ugc">DistributedDataParallel — PyTorch 2.0 documentation</a></p>
<p>Maybe you could try use static_graph=True option as that supports re-entrant backward, but I am not sure if that would help disable the gradients sync to the final backward pass.</p>
<p>cc <a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a></p>

bary: <p>What I ended up doing as a workaround is to have a dummy forward pass before the final backward pass - this made the gradients different the final backward, and equal after it.</p>
Get environment variables dynamically

rmekdma: <p>When using torchrun with elasticity, nodes can join or leave the group.</p>
<p>I want to current state of environments and I found <code>torch.distributed.get_world_size()</code>, <code>torch.distributed.get_rank()</code>.<br>
I am not sure, but these two functions seems to return values of current state.<br>
However I can’t find the way to get the number of current nodes.</p>
<p>My questions are</p>
<ol>
<li>Do <code>get_world_size()</code>, <code>get_rank()</code> return value of current state?</li>
<li>Is there a function that return the number of nodes?</li>
</ol>
<p>Thanks!</p>

rmekdma: <p>According to <a href="https://pytorch.org/docs/stable/elastic/run.html#membership-changes" rel="noopener nofollow ugc">document</a>, whenever a node join or leave, all workers restart with new <code>RANK</code> and <code>WORLD_SIZE</code> values.<br>
Also I found <code>GROUP_RANK</code> and <code>GROUP_WORLD_SIZE</code> environment variables.</p>
<p>In conclusion, I will use environment variables below</p>
<ul>
<li>local: <code>LOCAL_RANK</code>, <code>LOCAL_WORLD_SIZE</code>
</li>
<li>global: <code>RANK</code>, <code>WORLD_SIZE</code>
</li>
<li>node: <code>GROUP_RANK</code>, <code>GROUP_WORLD_SIZE</code>
</li>
</ul>
Which version of CUDA 12.1 should be used

ItachiWithSasuke: <p>My CUDA version is 12.1, which version of pytorcn should I choose, the latest 2.0, or should I choose to downgrade the driver and lower the CUDA version?</p>

ptrblck: <p>The PyTorch binaries ship with their own CUDA runtime and math libs. Your local CUDA toolkit will be used if you build PyTorch from source or custom extensions. You can thus install the current release with the bundled CUDA 11.7 or 11.8 libs.</p>

ItachiWithSasuke: <p>Thanks, I’m a beginner and don’t know how to install the current release with the bundled CUDA 11.7 or 11.8 libs, can you tell me what command to use?</p>

ptrblck: <p>Select the desired CUDA version from the install matrix <a href="https://pytorch.org/get-started/locally/">here</a>, copy/paste the command into your terminal, and execute it.<br>
I would also recommend to either use a new and empty virtual environment or to make sure no other PyTorch packages are installed in the current environment.</p>
Index 599 is out of bounds for dimension 0 with size 10

Ranim_Saidi: <p>Hello,</p>
<p>I am getting this error<br>
out = torch.gather(v,0,t).float()<br>
RuntimeError: index 599 is out of bounds for dimension 0 with size 10</p>
<p>Any suggestion on what to do?</p>

ptrblck: <p>The <code>gather</code> operation fails since <code>t</code> contains an invalid index (<code>599</code>) related to the size of <code>dim0</code> of <code>v</code> (<code>10</code>). Here is a small example code showing the issue:</p>
<pre><code class="lang-python">v = torch.randn(10)
print(v)
# tensor([ 1.1481,  1.1344,  0.7485,  0.7008, -1.4947,  0.0202,  1.0304, -0.6202,
#         -0.6328,  1.1113])

# works since 0 is a valid index
t = torch.tensor([0])
out = torch.gather(v,0,t).float()
print(out)
# tensor([1.1481])

# fails since 599 is invalid
t = torch.tensor([599])
out = torch.gather(v,0,t).float()
# RuntimeError: index 599 is out of bounds for dimension 0 with size 10
</code></pre>

Ranim_Saidi: <p>thank you for your response! much appreciated</p>
Enforcing high accuracy for function fitting

hnakao: <p>I’m trying to fit a neural network to a known function f with input and output dimensions both being 6 (which I would like to scale to 100~ in the future). I am using mean squared error as a loss function with ADAM optimizer and my target is 1e-6 accuracy. As f is known and the sample space is given, there is basically an infinite number of data available to feed into NN so I give 128 uniform samples in each step. In the first 1000 steps, I see a steep decline in the loss function but after that, the loss curve draws an L-like shape at 5e-3 (the decrease is not visible in the log plot) for about 1e6 steps until I stop. I tried varying the learning rate from 1e-3 to 1e-5, and the neural network size from 24 width * 3 hidden layers to 48 width * 9 hidden layers. However, I observe that there is a barrier to the loss function at the same 5e-3 level.  I’m not sure what else I can try. While universal approximation holds ‘in theory’, is this a common phenomenon in deep learning?</p>
Where is `apply` defined in torch.autograd.Function?

Sam-gege: <pre><code class="lang-python">import torch
from torch.autograd import Function

class ExpTest(Function):
    @staticmethod
    def forward(ctx, i):
        print(ctx)
        result = i.exp()
        return result


x=torch.tensor([1,2,3.], requires_grad=True)
output = ExpTest.apply(x)
print(output.grad_fn)

# &lt;torch.autograd.function.ExpTestBackward object at 0x7f0f376d84f0&gt;
# &lt;torch.autograd.function.ExpTestBackward object at 0x7f0f376d84f0&gt;
</code></pre>
<p>As shown above, looks like the <code>apply</code> method first instantiated an <code>ExpTestBackward</code> object, then pass it to forward. but where is this <code>apply</code> defined? There is no such method in torch.autograd.function.py file</p>

soulitzer: <p>It’s defined in cpp, see python_function.cpp. The Function class in python inherits from the one defined in cpp.</p>
[Multiple Process on same GPU device]

heitorrapela: <p>Hello,</p>
<p>I have access to one server that I can use one GPU per time with 40 GB memory. As each of my experiments only take 7gb±, sometimes I run multiple experiments with different hparams. But I have been noticing that when I do that, the experiments run too slowly. Even the CPU and GPU have a lot of free resources.</p>
<p>My guess is some strange resource that is not fully isolated when I run multiple processes on the same GPU. Is there a way to fully isolate the process in the same GPU to take advantage of the free memory that I have?</p>
<p>ps: I know that maybe the SLURM can do something like that, emulating the GPU with scheduling for different users, but as I am not the root, I would like to know if there is maybe a simple solution or another possible solution.</p>
<p>Best Regards,</p>

marksaroufim: <p>GPUs for the most part don’t have great resource isolation, for newer GPUs you might want to take a look at MIG support</p>

heitorrapela: <p>Oh, this MIG support seems to be what I was looking for. Thanks! If someone else also knows another tool, it would be great.</p>
TorchFunctionMode for getting current function : equivalent for pytorch 1.10

Aditya_Desai1: <p>I want to make some changes in save_hooks based on which function is running.  In one of the discussions on this forum <a href="https://discuss.pytorch.org/t/how-to-have-different-saved-tensor-hooks-for-different-functions/176301">How to have different saved_tensor_hooks for different functions - autograd - PyTorch Forums</a>, a  solution where we have to use TorchFunctionMode to get current function is suggesgted. However, this support was  not available in pytorch 1.10 . Is there a equivalent way for older pytorch versions to identify which function is running  when save hooks are called ?</p>

soulitzer: <p>Writing a torch function subclass would do something similar <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch — PyTorch 2.0 documentation</a>, the trickiness though is that you’ll have to make sure that all the tensors you pass in are of that subclass.</p>
<p>Just out of curiosity, why aren’t you able to upgrade to a later version?</p>
Weight initialization across releases

Javier_Grau: <p>I’m not sure if this issue has been posted before, but here it goes.</p>
<p>I’ve been training networks on two different machines with same running parameters, but getting very different results. The only different is that one machine I use version 1.9.0 and in the other 1.13.0.</p>
<p>I’m wonder what could be so different between these two releases that is producing such effect. Is the default weight initialization the same on these two versions?</p>
<p>Thanks</p>

marksaroufim: <p>Are you observing these changes with the same seed across releases, for example can you repro your results if you do <code>torch.manual_seed(0)</code></p>

Javier_Grau: <p>Thanks for your answer.</p>
<p>I can’t do that unfortunately. The hardware in both machines belong to two different generations of gpus, hence very different cuda versions. That’s why the different pytorch versions.<br>
But I am manually setting the same seeds with torch.manual_seed.</p>