akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Need help for tracing the internal CPU matmul Code

brushington1: <p>Hello, I am attempting to trace the sequence of parallel multiplication and addition operations in the matrix multiplication function, torch.matmul. From the C++ code in the PyTorch GitHub repository, I’ve tracked the actual execution to a call to <code>at::cpu::mm_out(out, mat1, mat2)</code>. This function appears to be dynamically generated by the ATen module during the compilation of PyTorch.</p>
<p>Upon compiling the code in PyTorch, I discovered that the dynamically generated <code>mm_out</code> further invokes the method <code>structured_mm_out_cpu::impl(mat1, mat2, out)</code>. This is declared in <code>torch\include\ATen\ops\mm_native.h</code>, which might also be a dynamically generated file. However, I haven’t been able to locate any implementation of this method in the compiled code. I’ve searched possible inheritance scenarios, taking polymorphism into consideration.</p>
<p>Could you assist me in tracking this down or directing me to where the actual code for the parallel multiplication and addition of matrix multiplication elements in the tensor is located?</p>
How to do normalization over specific dimension?

batx: <p>I know LayerNorm and InstanceNorm1d can do normalization over the last dimension. If I want to do it over the first dimension, I have to transpose the input tensor or calculate the mean and standard-deviation by my self, which would consume much time and memory.</p>
<p>if I do it by my self or transpose the input:</p>
<pre><code class="lang-auto">class Net1(nn.Module):
    def __init__(self):
        super(Net1, self).__init__()
        
    def forward(self, x):
        return (x - x.mean(dim=0, keepdim=True))/ (x.var(dim=0, unbiased=False, keepdim=True) + 1e-5).sqrt()


class Net2(nn.Module):
    def __init__(self):
        super(Net2, self).__init__()
        
    def forward(self, x):
        return nn.functional.layer_norm(x.transpose(0, 2), (1000,)).transpose(0, 2)
    

class Net3(nn.Module):
    def __init__(self):
        super(Net3, self).__init__()
        
    def forward(self, x):
        return nn.functional.instance_norm(x.transpose(0, 2)).transpose(0, 2)

net1 = Net1()
net2 = Net2()
net3 = Net3()

x = torch.randn(1000, 100000, 10, requires_grad=True)

%timeit net1(x)   #4.7 s ± 32.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit net2(x)   #2.71 s ± 46.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit net3(x)   #3.33 s ± 82.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit net1(x).mean().backward()   #22.1 s ± 52.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit net2(x).mean().backward()   #11.3 s ± 169 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit net3(x).mean().backward()   #10.3 s ± 134 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
</code></pre>
<p>But if I do normalization over the last dimension without transpose, I can save half of time.</p>
<pre><code class="lang-auto">y = torch.randn(10, 100000, 1000, requires_grad=True)

class Net5(nn.Module):
    def __init__(self):
        super(Net5, self).__init__()
        
    def forward(self, x):
        return nn.functional.layer_norm(x, (1000,))
    

class Net6(nn.Module):
    def __init__(self):
        super(Net6, self).__init__()
        
    def forward(self, x):
        return nn.functional.instance_norm(x)

net5 = Net5()
net6 = Net6()

%timeit net5(y)   #1.24 s ± 22.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit net6(y)   #1.73 s ± 16.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit net5(y).mean().backward()   #4.07 s ± 28.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit net6(y).mean().backward()   #4.69 s ± 32.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
</code></pre>
<p>Is there any method to do it over specific dimension effectively?</p>

kirk86: <p>What they should be doing is adding another argument to <code>nn.LayerNorm(axsi/dim=-1)(x)</code> so that users can specify which axis they want, especially useful in situations where you don’t know a priori the shapes of the inputs.</p>
Minimize loss function

Kallel_Mariem: <p>i work on a complex model, the loss function used is SSIM_loss+ rmi_loss+texture_loss i want to minimize it but it is always in the range of 1.7 and 1 what should i do ?  i use optimizer = optim.SGD(model.parameters(), lr=args.lr, weight_decay=0.01)<br>
scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.001, max_lr=0.1, step_size_up=100, step_size_down=100, mode=‘triangular’, cycle_momentum=True, base_momentum=0.8, max_momentum=0.9, last_epoch=-1, verbose=False)</p>
NSight Compute CUPTI Error

azhao: <p>Hi! When running NSight Compute on a remote cluster with the command</p>
<p>run_script.sh profile_test.py</p>
<p>I get the error:</p>
<p>==PROF== Connected to process 1700339 (&lt;path_to_conda_env&gt;/bin/python3.10)<br>
WARNING:2023-11-29 19:41:07 1700339:1700339 init.cpp:155] function cbapi-&gt;getCuptiStatus() failed with error CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED (39)<br>
WARNING:2023-11-29 19:41:07 1700339:1700339 init.cpp:156] CUPTI initialization failed - CUDA profiler activities will be missing<br>
INFO:2023-11-29 19:41:07 1700339:1700339 init.cpp:158] If you see CUPTI_ERROR_INSUFFICIENT_PRIVILEGES, refer to <a href="https://developer.nvidia.com/nvidia-development-tools-solutions-err-nvgpuctrperm-cupti" class="inline-onebox" rel="noopener nofollow ugc">NVIDIA Development Tools Solutions - | NVIDIA Developer</a><br>
==PROF== Disconnected from process 1700339<br>
==WARNING== No kernels were profiled.</p>
<p>run_script.sh loads some modules, then calls NCU (I removed --target-processes all in case that was causing the issue)</p>
<p>#!/bin/bash</p>
<p>module load python/3.10.12<br>
module load gcc/9.5.0<br>
module load cuda/12.0.1<br>
module load cudnn/8.9.2.26_cuda12</p>
<p>HOME_DIR=&lt;path_to_my_home_dir&gt;</p>
<p>mamba activate $HOME_DIR/env</p>
<p>module load cmake</p>
<p>ncu --set full --export $HOME_DIR/ncu_data/profile_results $HOME_DIR/mamba_env/bin/python3 $1</p>
<p>and all profile_test.py does is import torch, with some nvtx markers:</p>
<p>import torch<br>
from torch.cuda import nvtx</p>
<p>nvtx.range_push(‘start’)<br>
print(‘Hello World’)<br>
nvtx.range_pop()</p>
<p>Do you know why this might be? According to NSight Compute documentation,</p>
<ul>
<li>Added error code <em>CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED</em> to indicate the presense of another CUPTI subscriber. <em>API cuptiSubscribe()</em> returns the new error code than <em>CUPTI_ERROR_MAX_LIMIT_REACHED</em>,</li>
</ul>
<p>though I’m not sure what might be causing the error on my end. I’m running the commands on an isolated node with V100s.</p>
<p>If I run the python program that simply calls “import os” or “import numpy as np,” I get a no kernels profiled warning—not the CUPTI error. Therefore, I’m wondering if this has something to do with torch also using CUPTI. Notably, I built torch from source; in the build logs, I found “Using Kineto with CUPTI support.”</p>
<p>At the end of the day, I would like to be able to profile PyTorch models using NSight Compute.</p>
<p>Thanks!</p>

ptrblck: <p>The issue was most likely introduced in <a href="https://github.com/pytorch/pytorch/pull/112623">this PR</a> and should be reverted. I’ll ping the code owners again as we are hitting the same issue internally and already bisected it to this PR.</p>
<p>CC <a class="mention" href="/u/malfet">@malfet</a> as we just discussed it.</p>

aaronenyeshi: <p>A revert landed in <a href="https://github.com/pytorch/pytorch/pull/116201" rel="noopener nofollow ugc">PR116201</a>. I didn’t see any comments in the regressing PR, so I thought a workaround was being developed at NVIDIA. Sorry about this issue.</p>
Losses start very high and drop to near zero extremely quickly

FedeHeinrichsCorrado: <p>Hi all,</p>
<p>I’m training a network to solve a system of partial differential equations, and the loss is extremely large at iteration 1000, by iteration 2000 it’s already down to almost zero and it stays down, although it decreases incrementally. Clearly there is something wrong, but I can’t narrow down what it is. I imagine that something like this is probably symptomatic of some kind of pathology that many of you would have seen/know about, that it’s a tell-tale sign of something specific that I’m messing up. I’d very grateful for any suggestions on what you’d look at.</p>
<p>Many thanks,<br>
Federico</p>
What is the cpu() in pytorch

J_Na: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/cd6ed8b3e481be6bebb088ed05f879b43eecc097.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd6ed8b3e481be6bebb088ed05f879b43eecc097" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/cd6ed8b3e481be6bebb088ed05f879b43eecc097.png" alt="image" data-base62-sha1="tjlmSi1j75IeJKc6rsR7JEvbxIj" width="690" height="272" data-dominant-color="F9F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">778×307 22.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<strong>correct += pred_eq(target.data).cpu().sum()</strong></p>
<p>What`s the meaning of cpu() ?<br>
Can anyone understand that code ?</p>

Diego: <p>This is used to move the tensor to cpu(). Some operations on tensors cannot be performed on cuda tensors so you need to move them to cpu first.</p>

jpeg729: <p><code>tensor.cuda()</code> is used to move a tensor to GPU memory.<br>
<code>tensor.cpu()</code> moves it back to memory accessible to the CPU.</p>

vinaykumar2491: <p>But after doing <code>tensor.cpu()</code> when I check the device of <code>tensor</code> using <code>tensor.device</code> it gives the original <code>cuda:0</code> where it was before moving to cpu. How can I be sure that the tensor is moved to CPU?</p>

ptrblck: <p>You have to assign the tensor after moving:</p>
<pre><code class="lang-python">tensor = tensor.cpu()
# or using the new method
tensor = tensor.to('cpu)
</code></pre>

vinaykumar2491: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. I see that <code>.cpu()</code> or <code>.cuda()</code> works differently for <code>model</code> and <code>tensor</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=6" title=":+1:" class="emoji" alt=":+1:"></p>

peterkim95: <p>Is there an official list of operations that are unable to performed on cuda tensors?</p>
<p>I’m wondering if there is some guideline to understand when you can and cannot perform certain operations?</p>
<p>Thanks!</p>

mary-ashk: <p>I have a big model and my RAM is 6.<br>
Is it helpful to send some tensors to CPU, and others remain on cuda to solve the cuda out of memory error?</p>

ptrblck: <p>I don’t know what “my RAM is 6” means, but CPU offloading can generally reduce the GPU memory usage.</p>
Theory discussion

Angry_potato: <p>Hi,<br>
I was thinking if we could add a section in discussion where people can discuss theoritical aspects of ML/DL.What say <a class="mention-group" href="/groups/admins">@admins</a></p>

ptrblck: <p>Sounds like a valid request and I would like others to chime in, if they would be interested in this category instead of using “Uncategorized”.</p>

mathematics: <p>Yeah i was asking this kind of things , nobody replied.It would be great if we have discussion of theorems and things category.</p>

chetan06: <p>Yes Its a great I idea as this will motivate many to post their ideas on some topic and also many to learn from them.</p>

vdw: <p>Yes, a category where to place fundamental ML questions would be good. Many questions are already often not really PyTorch specific and I feel in other categories they get sometimes ignored.</p>
Saving and loading model with libtorch C++

Amadou: <p>Hello,</p>
<p>How can I properly save and load my model with libtorch C++ ?</p>
<p>Indeed, my class module is defined like <code>class Net : public torch::nn::Module {}</code>.<br>
After training my model, I saved it using <code>torch::save(model, "my_model.pt")</code>. And for Inference, I used <code>torch::load(model, "my_model.pt")</code>, but forward in <code>model.forward(inputs)</code> is not recognized.</p>
<p>I also tried to load with torch jit : this loading method doesn’t work.</p>
<p>Thanks in advance!</p>

beazt: <p>What is the exact error when you say the forward method is not recognized? If I understood correctly, you need to load a traced model via torch::jit::load(), the saved model should also be a traced model.</p>

Amadou: <p><a class="mention" href="/u/beazt">@beazt</a> thanks for contributing!</p>
<p>Not recognized like forward is not a member of model.</p>
<p>I solved the problem by saving and loading the parameters instead of <code>model</code>.<br>
Concretely, I saved <code>model.parameters()</code> as <code>std::vector&lt;torch::Tensor&gt;</code> after training using <code>torch::save</code>. And for Inference, I loaded the saved weights and didn’t use grad.</p>
<p>Thanks!</p>

Kris1: <p>I have the same problem with you. When I load the model and use it for inference, it throws out the error “Method forward is not defined.”. Could you share how you save the parameters and load it? Thanks!</p>
Function 'AddmmBackward0' returned nan values in its 1th output

AAArAAA: <p>I have simple NN network which is consist of convolution layer followed by Linear layer to train RL agent using REINFORCE algorithm. I use</p>
<p>torch.optim.AdamW, for optimizer<br>
nn.init.kaiming_normal_(m.weight, nonlinearity=‘relu’) for initialization</p>
<p>Although I set the torch.manual_seed(seed), numpy seed and random seed, I sometimes get the following error:</p>
<p>Function ‘AddmmBackward0’ returned nan values in its 1th output.</p>
<p>I have couple of question:<br>
(1) nan value in its 1th output means there is problem in the weights, or the ipnput?<br>
(2) Even when I have seed, Why do I sometimes get the error and sometimes my program successfully ends with reproducible output?<br>
(3) How can I fix it?</p>
<p>I appreciate your help</p>

wilhelm: <p>In my case it was an exploding loss calculation.<br>
Post your code entirely, otherwise it is almost impossible to help you</p>
Torchtext: Setting field with preprocessing

usakira: <p>Hi, I’m working with my original data and struggling with creating a dataset on torchtext.</p>
<p>The data consist of sentences.<br>
i.e.) [[“He”, “plays”, “piano”],[“He”, “plays”, “guitar”, “well”],…]</p>
<p>I would like to choose one of them for train.<br>
So I set the Field for them as followings.</p>
<pre><code class="lang-auto">n_samples_field = Field(use_vocab=True,
                        eos_token=SpecialToken.EOS.value,
                        pad_token=SpecialToken.Padding.value,
                        unk_token=SpecialToken.Unknown.value,
                        preprocessing=lambda sen: random.choice(sen) \
                        if sen != [] else [""],
                        include_lengths=True)
</code></pre>
<p>But it failed and there is the error code.<br>
I thought I need to convert each sen into int after preprocessing, but how?</p>
<pre><code class="lang-auto">  File "/home/ubuntu/test/train.py", line 132, in run
    for batch in X:
  File "/home/ubuntu/anaconda3/envs/test_env/lib/python3.7/site-packages/torchtext/data/iterator.py", line 156, in __iter__
    yield Batch(minibatch, self.dataset, self.device)
  File "/home/ubuntu/anaconda3/envs/test_env/lib/python3.7/site-packages/torchtext/data/batch.py", line 34, in __init__
    setattr(self, name, field.process(batch, device=device))
  File "/home/ubuntu/anaconda3/envs/test_env/lib/python3.7/site-packages/torchtext/data/field.py", line 237, in process
    tensor = self.numericalize(padded, device=device)
  File "/home/ubuntu/anaconda3/envs/test_env/lib/python3.7/site-packages/torchtext/data/field.py", line 336, in numericalize
    arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
  File "/home/ubuntu/anaconda3/envs/test_env/lib/python3.7/site-packages/torchtext/data/field.py", line 336, in &lt;listcomp&gt;
    arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
  File "/home/ubuntu/anaconda3/envs/test_env/lib/python3.7/site-packages/torchtext/data/field.py", line 336, in &lt;listcomp&gt;
    arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
AttributeError: 'Field' object has no attribute 'vocab'
</code></pre>

usakira: <p>I got processing is like that.<br>
So I need to pick 1 example before adopting field.</p>
<p><code>preprocessing – The Pipeline that will be applied to examples using this field after tokenizing but before numericalizing. Many Datasets replace this attribute with a custom preprocessor. Default: None.</code></p>

Bridget_Murphy: <p>Good luck on your project.</p>
RuntimeError: output with shape [64, 12, 1, 1] doesn't match the broadcast shape [64, 12, 1, 64]

ahmedrizwan239: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/eece9c9360442b611d8c4f8d2f6b365f48832f62.png" data-download-href="https://discuss.pytorch.org/uploads/default/eece9c9360442b611d8c4f8d2f6b365f48832f62" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eece9c9360442b611d8c4f8d2f6b365f48832f62_2_690x194.png" alt="image" data-base62-sha1="y4AiGjcI2Gw0eTcRJM2v4W0mvOW" width="690" height="194" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eece9c9360442b611d8c4f8d2f6b365f48832f62_2_690x194.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eece9c9360442b611d8c4f8d2f6b365f48832f62_2_1035x291.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eece9c9360442b611d8c4f8d2f6b365f48832f62_2_1380x388.png 2x" data-dominant-color="3A3B3C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1845×520 34 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m trying to make a project on English to SQL Query, I’m using T5-base model and while evaluating the model I ran into this error.</p>
<p>Here are my basic configurations:</p>
<h1><a name="define-constants-1" class="anchor" href="#define-constants-1"></a>define constants</h1>
<p>MODEL_NAME = ‘t5-base’<br>
MAX_LENGTH = 64<br>
BATCH_SIZE = 64<br>
NUM_EPOCHS = 1</p>
<p>I’m attaching the notebook link:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/14F9M-U2Sc13VACHSxVLnoj6zXAj9AXax?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/6a08971d26f794d4e970c8cafd32434c/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/14F9M-U2Sc13VACHSxVLnoj6zXAj9AXax?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/14F9M-U2Sc13VACHSxVLnoj6zXAj9AXax?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Please review the code and let me know where I’m doing wrong.<br>
TIA</p>
Does the foreach operators have the same shape?

Larry_Li: <p>foreach operators receive TensorList as input. Does the tensors stored in TensorList must have same shape for foreach kernels?</p>

ptrblck: <p>No as seen in this small example:</p>
<pre data-code-wrap="python"><code class="lang-python">device = "cuda"
l = [torch.randn(20, 20, device=device), torch.randn(10, 10, device=device)]
out = torch._foreach_abs(l)

out = torch._foreach_add(l, 100.)
</code></pre>
<p>Note that API is in beta and could thus change.</p>

Larry_Li: <p>Thank you, ptrblck. I have another question, when we add new backend, we found that pytorch frontend will do broadcast. For example: x1_shape[2, 3, 4, 5], x2_shape[4, 5]. torch._foreach_add([x1], [x2]), x2 will be broadcasted to [2, 3, 4, 5] automatically. Backend don’t need to do it again. Is this correct? And do you know the code location of these broadcasting?</p>

ptrblck: <p>Could you post an executable code example?</p>

Larry_Li: <p>Sorry for late reply because of the forums migration. I have found the answers, the frontend will expand x2 to [1, 1, 4, 5], but don’t change data stored in memory.<br>
Thanks for your support!</p>
Dataloader with num_workers >0 not working on jupiter

AlexRanda: <p>Hi All, I read several post and I already know that there are issues with using jupyter notebook with multi threading. Since those posts were some years ago, I am wondering if this issue was fixed. Below is my machine and a snippet of my code (nested for loop). It runs indefinitely without giving any errors if I put inside Dataloader num_workers &gt; 0 (default is zero)  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>Is CUDA supported by this system? True<br>
CUDA version: 12.1<br>
Using device: cuda<br>
ID of current CUDA device: 0</p>
</blockquote>
<p>Name of current CUDA device: NVIDIA GeForce RTX 3060<br>
NVIDIA GeForce RTX 3060</p>
<p>It is working on a single thread and is extremely slow and I need to speed the execution up.<br>
Below a snippet of my code: X_PM10_train_pt, Y_PM10_train_pt are cuda tensors, but nothing change if they move on cpu.</p>
<pre><code class="lang-auto">train_loader = DataLoader(TensorDataset(X_PM10_train_pt, Y_PM10_train_pt), batch_size=params['batch_size'],shuffle=False) 
.....
for epoch in range(num_epochs):
    model_mlp2_pm10.train()
    for inputs, labels in train_loader:
        optimizer.zero_grad() #zero the gradients after updating, before starting a new optimization
        outputs = model_mlp2_pm10(inputs) #predict = forward pass with our model
        loss = criterion(outputs, labels) #loss
        loss.backward() #calculate gradients
        optimizer.step()#update weights
# Valutazione del modello sul set di validazione

</code></pre>
<p>Please give me any hint how to overcome this issue</p>

J_Johnson: <p>Hello, is your system Windows or Linux? If Windows, I’m afraid the PyTorch dataloader can only use num_workers=0.</p>

AlexRanda: <p>windows 11 professional</p>

ptrblck: <aside class="quote no-group" data-username="J_Johnson" data-post="2" data-topic="194815">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar"> J_Johnson:</div>
<blockquote>
<p>If Windows, I’m afraid the PyTorch dataloader can only use num_workers=0.</p>
</blockquote>
</aside>
<p>That’s not strictly true and should still work, if e.g. the needed <code>if-clause</code> protection is used as explained <a href="https://pytorch.org/docs/stable/notes/windows.html#usage-multiprocessing">here</a>.</p>
How to Implement Asynchronous Request Handling in TorchServe for High-Latency Inference Jobs?

aroop: <p>I’m currently developing a Rails application that interacts with a TorchServe instance for machine learning inference. The TorchServe server is hosted on-premises and equipped with 4 GPUs. We’re working with stable diffusion models, and each inference request is expected to take around 30 seconds due to the complexity of the models.</p>
<p>Given the high latency per job, I’m exploring the best way to implement asynchronous request handling in TorchServe. The primary goal is to manage a large volume of incoming prediction requests efficiently without having each client blocked waiting for a response.</p>
<p>Here’s the current setup and challenges:</p>
<ul>
<li>Rails Application: This acts as the client sending prediction requests to TorchServe.</li>
<li>TorchServe Server: Running on an on-prem server with 4 GPUs.</li>
<li>Model Complexity: Due to stable diffusion processing, each request takes about 30 seconds.</li>
</ul>
<p>I’m looking for insights or guidance on the following:</p>
<ol>
<li>Native Asynchronous Support: Does TorchServe natively support asynchronous request handling? If so, how can it be configured?</li>
<li>Queue Management: If TorchServe does not support this natively, what are the best practices for implementing a queue system on the server side to handle requests asynchronously?</li>
<li>Client-Side Implementation: Tips for managing asynchronous communication in the Rails application. Should I implement a polling mechanism, or are there better approaches?</li>
<li>Resource Management: How to effectively utilize the 4 GPUs in an asynchronous setup to ensure optimal processing and reduced wait times for clients.</li>
</ol>
<p>Any advice, experiences, or pointers to relevant documentation would be greatly appreciated. I’m aiming to make this process as efficient and scalable as possible, considering the high latency of each inference job.</p>
<p>Thank you in advance for your help!</p>
`compiled_autograd`

Jerome_Ku: <p>What is the purpose of the <code>compile_autograd</code> <a href="https://github.com/pytorch/pytorch/blob/bfbb8d82201450f8336285ee075b391467921952/benchmarks/dynamo/common.py#L3154" rel="noopener nofollow ugc">argument</a> in the <code>dynamo</code> benchmarking` script?</p>
<p>I thought that forward / backwards graphs were automatically compiled for a <code>nn.Module</code> when using <code>torch.compile</code> (with default <code>inductor</code> backend)?</p>
Sharing weights

papoo13: <p>Assume I have a linear layer in PyTorch. For the weight matrix of this layer, can I define only one learnable parameter and then expand this parameter to fit the required matrix size of the layer? so far, I think this is impossible, so I appreciate any tips. I want to reduce the number of learning parameters for this layer to one. It doesn’t matter if in the end, I have the same values for all the values in this matrix.</p>
Dataloader execution freezes

liampour: <p>Hello guys,<br>
when I run the following script, it is executed till to the print(“just before for loop”) command and after it “freezes” (nothing prints in terminal)</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

batch_size = 4

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,
                                         shuffle=False, num_workers=2)


print("just before for loop")

for data in trainloader:    
    print(data)

print("Script run succesfully")

</code></pre>
<p>I waited 10 mins but nothing happened</p>
<p>Then I tried to run the script without the  for loop and  it executed perfectly. After it was completed I run the for loop from terminal and it also executed.<br>
It seems to be executed perfectly when I run them separately but it “freezes” when they are in the same script.<br>
I run pytorch to:</p>
<ul>
<li>Windows</li>
<li>Conda 23.7.4</li>
<li>Python 3.11.5</li>
<li>Cuda 11.8</li>
<li>Spyder 5.4.3</li>
</ul>
<p>Any suggestions??</p>
Mac M1 mps error mps.scatter_nd when iterating across architectures

mercicle: <p>I’m using nanoGPT but this post is more about mps than nanoGPT.</p>
<p>Hi all,</p>
<p>I’m on Apple M1 and used the nanoGPT train.py to make a train_gpt(…) function, so I can sweep across architectures (varying number of layers, heads and embeddings).</p>
<p>When only using the cpu, I get no errors. But when using mps, I get this error:</p>
<pre><code class="lang-auto">loc("edb"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/97f6331a-ba75-11ed-a4bc-863efbbaf80d/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":49:0)): error: 'mps.scatter_nd' op invalid input tensor shape: updates tensor shape and data tensor shape must match along inner dimensions

/AppleInternal/Library/BuildRoots/97f6331a-ba75-11ed-a4bc-863efbbaf80d/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphExecutable.mm:1710: failed assertion `Error: MLIR pass manager failed'

</code></pre>
<p>If I use mps option and keep the architecture fixed (e.g. <code>{'n_layer': 4, 'n_head' : 4, 'n_embd' : 128}</code>), then mps works without error. However, when I loop across varying architectures I get the error above ^^.</p>
<p>The issue is not with compatible <code>n_layer</code>, <code>n_head</code> and <code>n_embd</code> dimensions or memory issues. I’ve tested that the architectures in which the error occurs work fine if ran alone (rather than in a loop after a different architecture). The general rules I use for generating architectures are: 1. n_head is a multiple of n_layer, and 2. n_embd is divisible by n_head.</p>
<p>So it smells like the GPU cache is not getting cleared after each train_gpt(…) iteration and there is some stale dimensions being expected in operations.</p>
<p>Is there a way to clear the GPU cache in mps framework? I’ve tried throwing a <code>mps.empty_cache()</code> at the end of each iteration (by iteration, I mean a round of calling train_gpt(…) for a specific architecture, not an iteration in each training loop). But that doesn’t work.</p>
<p>Thanks in advance!</p>
Efficient way to remove trailing NaN values

alaria: <p>Is there an efficient way to remove trailing nan values from a tensor?</p>
<p>Given Input:         [NaN, 1, 2, NaN, 4, NaN, NaN]<br>
Desired Output:   [NaN, 1, 2, NaN, 4]</p>
<p>In my specific case, the amount of values to be removed is likely to be &lt;50 while the entire tensor has more than a thousand elements. So it might just be most efficient to iterate over it to find the right cutoff point, starting from the last element, and then do “output = input[:cutoff]”. I’m just curious if there is a more efficient PyTorch way.</p>
How to control or tell torch.compile for training or inference?

ayl: <p>The provided code is extracted from <a href="https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/fusions/fused_bias_swiglu.py" rel="noopener nofollow ugc">here</a>. I have two questions:</p>
<ol>
<li>How can one control <code>torch.compile</code> to operate in training mode, where it generates the backward pass, or in inference mode, where it doesn’t generate the backward pass?</li>
<li>What is the behavior when the <code>torch.compile</code> functions (<code>swiglu</code> and <code>swiglu_back</code> as shown below) are used in the <code>forward</code> and <code>backward</code> static methods within <code>torch.autograd.Function</code>? Do they operate in the inference mode (which doesn’t generate the backward pass) as default?</li>
</ol>
<pre><code class="lang-auto">@torch.compile
def swiglu(y):
    y_1, y_2 = torch.chunk(y, 2, -1)
    return F.silu(y_1) * y_2

@torch.compile
def swiglu_back(g, y):
    y_1, y_2 = torch.chunk(y, 2, -1)
    return torch.cat(
        (g * torch.sigmoid(y_1) * (1 + y_1 * (1 - torch.sigmoid(y_1))) * y_2, g * F.silu(y_1)), -1
    )

class SwiGLUFunction(torch.autograd.Function):
    @staticmethod
    # bias is an optional argument
    def forward(ctx, input):
        ctx.save_for_backward(input)
        return swiglu(input)

    @staticmethod
    def backward(ctx, grad_output):
        input = ctx.saved_tensors
        tmp = swiglu_back(grad_output, input[0])
        return tmp
</code></pre>
Cannot import name 'LSTM' from 'torchrl.modules.tensordict_module.rnn'

Clion: <p>So after about 8 months of wishing I could use VMap to stack LSTMs, I’ve finally found this: <a href="https://pytorch.org/rl/reference/generated/torchrl.modules.LSTM.html" class="inline-onebox" rel="noopener nofollow ugc">LSTM — torchrl main documentation</a></p>
<p>On this page, I see: “This class is implemented without relying on CuDNN, which makes it compatible with torch.vmap()”, which is great for me. However, when following the instructions on that page, I run into an error at the very first step, the import:</p>
<pre><code class="lang-auto">from torchrl.modules.tensordict_module.rnn import LSTM
</code></pre>
<p>That line produces this error:</p>
<pre><code class="lang-auto">ImportError: cannot import name 'LSTM' from 'torchrl.modules.tensordict_module.rnn' (/home/username/miniconda3/envs/envname/lib/python3.8/site-packages/torchrl/modules/tensordict_module/rnn.py)
</code></pre>
<p>Versioning info for context:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
&gt;&gt;&gt; '2.1.1'
&gt;&gt;&gt; import torchrl
&gt;&gt;&gt; torchrl.__version__
&gt;&gt;&gt; '0.2.1'
</code></pre>
<p>And python itself is 3.8.18.</p>
<p>TorchRL was installed via:</p>
<pre><code class="lang-auto">pip install torchrl
</code></pre>
<p>as per instructions on this page: <a href="https://pytorch.org/rl/tutorials/torchrl_demo.html#installing-the-library" class="inline-onebox" rel="noopener nofollow ugc">Introduction to TorchRL — torchrl main documentation</a></p>
<p>Any takers on this? Frustrating error because I’ve been waiting for a way to stack LSTMs for almost a year now, and I finally find this torchrl LSTM that can supposedly be VMapped, seemingly follow usage instructions exactly, and can’t get past the import step.</p>
<p>Mega thanks to anyone that can help with this (:</p>

Clion: <p>Ok nevermind, problem solved. For posterity: <code>tensordict_module.rnn.LSTM</code> only currently ( as of 27.01.2024 ) exists in the nightly build of <code>torchrl</code>, which requires one to have the nightly build of pytorch itself installed. Uninstalling <code>torch 2.1.1</code>, then installing the nightly build of pytorch, then installing the nightly build of torchrl ( pytorch nightly installed via conda instructions from <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">this page</a> and torchrl nightly installed via <code>pip install torchrl-nightly</code> ) worked to solve this problem for me.</p>

vmoens: <p>Yes this feature will be in the v0.3 release next week. It should work with torch.compile as well as vmap.</p>
Efficiently computing log probabilities of a batch of samples points on K different Multivariate Gaussian distributions

elkop: <p>Suppose I have N 2D Multivariate Gaussian distribution. I can define this using <code>torch.distributions.multivariate_normal.MultivariateNormal</code> as</p>
<pre><code class="lang-auto">inp_dim=2
mu = nn.Parameter(torch.rand(n_gaussians, inp_dim))
cov = nn.Parameter(torch.eye(inp_dim).unsqueeze(0).repeat(n_gaussians, 1, 1)

mvn = MultivariateNormal(mu, scale_tril=torch.tril(cov))

</code></pre>
<p>Suppose I have B sample points and I want to evaluate log probabilities of these B sample points at all of these N 2D Multivariate Gaussian distributions i.e. I will have N*B log probabilities. Simply calling the following will not work as it assumes that n_gaussians is the batch shape but in my case, N is different from B.</p>
<pre><code class="lang-auto">log_prob = mvn.log_prob(x)
</code></pre>
<p>How to efficiently evaluate these log probabilities of B sample points at N multivariate Gaussians?</p>

KFrank: <p>Hi Elkop!</p>
<aside class="quote no-group" data-username="elkop" data-post="1" data-topic="196134" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/c57346/48.png" class="avatar"> elkop:</div>
<blockquote>
<p>I will have N*B log probabilities. Simply calling the following will not work as it assumes that n_gaussians is the batch shape but in my case, N is different from B.</p>
<pre><code class="lang-auto">log_prob = mvn.log_prob(x)
</code></pre>
<p>How to efficiently evaluate these log probabilities of B sample points at N multivariate Gaussians?</p>
</blockquote>
</aside>
<p>Add a singleton dimension to <code>x</code> that will broadcast over the <code>n_gaussians</code><br>
dimension of <code>mvn</code>.  That way the <code>n_gaussians</code> and “batch” dimensions will<br>
be kept separate and you will get your full set of N*B log probabilities.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.1.2
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2024)
&gt;&gt;&gt;
&gt;&gt;&gt; inp_dim = 2
&gt;&gt;&gt; n_gaussians = 5
&gt;&gt;&gt; n_batch = 3
&gt;&gt;&gt;
&gt;&gt;&gt; mu = torch.nn.Parameter (torch.rand (n_gaussians, inp_dim))
&gt;&gt;&gt; cov = torch.nn.Parameter (torch.eye (inp_dim).unsqueeze (0).repeat (n_gaussians, 1, 1))
&gt;&gt;&gt;
&gt;&gt;&gt; mvn = torch.distributions.multivariate_normal.MultivariateNormal (mu, scale_tril = torch.tril (cov))
&gt;&gt;&gt;
&gt;&gt;&gt; x = torch.randn (n_batch, 1, inp_dim)   # singleton dimension will broadcast over mvn's n_batch
&gt;&gt;&gt; mvn.log_prob (x)
tensor([[-1.9006, -2.4927, -2.0306, -1.8988, -1.9011],
        [-2.9162, -3.7090, -2.4418, -2.4743, -2.8308],
        [-1.9674, -2.0202, -1.8672, -1.8966, -1.9255]], grad_fn=&lt;SubBackward0&gt;)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Accessing the intermediate layer of timm model in pytorch

Adex: <p>I am currently working on a timm model where I want to access some specific layers of the model.</p>
<pre><code class="lang-auto">import torch
import timm

model=timm.create_model('timm/vit_base_patch16_224', pretrained=True)

for i in range(6):
    strn='blocks.'+str([i])+'.mlp'
    container=model.strn
</code></pre>
<p>This code is showing error. Although coddes like the following works</p>
<pre><code class="lang-auto">model.blocks[7].mlp
</code></pre>
<p>It also looks like the model is not subscriptable i.e. approaches like <code>model[8]</code><br>
will not work. Is there any way to access the modules of a model using automated approach?</p>

ptrblck: <p>You could use <code>getattr</code> and pass the attribute name to it as I assume <code>model.strn</code> fails as it’s not a valid expression.</p>

Adex: <p>Thanks a lot for your reply. The following works,</p>
<pre><code class="lang-auto">container=getattr(getattr(self.model,'blocks')[i],'mlp')
</code></pre>
<p>But I cannot repace the modules with custom modules. Can you suggest any solution to the following problem?</p>
<pre><code class="lang-auto">getattr(getattr(self.model,'blocks')[i],'mlp')=nn.Linear(10,20)
</code></pre>
<p>I am getting following error,</p>
<pre><code class="lang-auto">SyntaxError: cannot assign to function call
</code></pre>

ptrblck: <p>Try <code>setattr</code> after accessing the layer via <code>getattr</code>.</p>
Dose DDP code compatible with a single card?

hitbuyi: <p>I have only one RTX3060, but I want to write my training code as form of DDP, hoping that DDP can compatible with one card condiction, is it possible?</p>

ptrblck: <p><code>DDP</code> is compatible with single GPUs in a sense as it will fallback to a plain model and won’t use any distributed calls.</p>
Input data for Multi Class Segmentation

vsat10: <p>I am building a multi class image segmentation model. In each image there could be 11 classes.</p>
<p>In mask image, pixels are marked with class no’s of category they belong to. So,<br>
Image shape - [3, 224, 224]<br>
Mask shape - ?</p>
<p>How should masks be shaped?</p>

ptrblck: <p>For a multi-class segmentation you could use <code>nn.CrossEntropyLoss</code> with a model output in the shape <code>[batch_size, nb_classes, height, width]</code> and a target containing class indices in the range <code>[0, nb_classes-1]</code> in the shape <code>[batch_size, height, width]</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">criterion = nn.CrossEntropyLoss()
batch_size, nb_classes, height, width = 2, 10, 24, 24
output = torch.randn(batch_size, nb_classes, height, width, requires_grad=True)
target = torch.randint(0, nb_classes, (batch_size, height, width))

loss = criterion(output, target)
loss.backward()
</code></pre>

vsat10: <p>Thanks for your immediate response <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>So, my target has this shape,<br>
Target = [nb_classes, height, width]</p>
<p>Batch size gets added later by the Dataloader.</p>
<p>Is this ok?</p>

ptrblck: <p>No, check my code which shows that the target does not have a class dimension. You can copy/paste my code as it’s also executable.</p>

vsat10: <p>Sorry for not explaining completely.</p>
<p>My target image is already a single channel image and containing pixel values based on which class they belong to.<br>
Each pixel can belong to only 1 class.<br>
So, target is,</p>
<ul>
<li>grayscale</li>
<li>shape : [224, 224]</li>
</ul>
<p>There are 10 possible classes including background</p>

ptrblck: <p>In this case the target would have the unbatched shape <code>[height, width]</code> and the batched shape <code>[batch_size, height, width]</code>, which is correct.</p>

vsat10: <p>The model is training now. Thanks</p>
<p>I am also trying to create a VIT with the same dataset. This model is not returning correct shape of output - [2, 10, 224, 224] - [B, C, H, W]</p>
No module named 'utils.torch_utils'

5_5: <p>Hello.I have pythorch 2.0.1+cu117.I tried to import select_device from utils.torch_utils by the command:<br>
from utils.torch_utils import select_device .<br>
I got the error: ModuleNotFoundError: No module named ‘utils.torch_utils’.</p>
<p>I tried to :</p>
<p>1.pip install python_utils<br>
2.pip install utils<br>
3.pip install torch-utils<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1ca1d5f7e82554ac166578e512ba4be0de4ca26.png" data-download-href="https://discuss.pytorch.org/uploads/default/c1ca1d5f7e82554ac166578e512ba4be0de4ca26" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1ca1d5f7e82554ac166578e512ba4be0de4ca26.png" alt="image" data-base62-sha1="rElbckmhH8kQ209JaO8JJDudCtM" width="690" height="229" data-dominant-color="171817"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1883×626 31.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
4.pip install --upgrade torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu117" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu117</a><br>
5.pip install --upgrade torch<br>
6.pip install --upgrade torch torchvision torchaudio<br>
7.pip install --upgrade torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu118" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118</a></p>
<p>Now i have pytorch 2.1.2+cu118, but, unfortunatelly i still have the same problem.How can i fix that ?</p>

ptrblck: <p>Could you explain where the <code>utils</code> namespace is defined, please?</p>

5_5: <p>Hello.Sorry for the late reply.I tried to find where the <code>utils</code> namespace is defined and found out that the problem was associated with yolov5 and not with the pytorch.After that i could fix the problem.<br>
Thanks.</p>
Reproducibility issues with ReflectPad2D

Harry_Lee: <p>Hello everyone,I ran into this issue when I was reproducing the experiment:reflection_pad2d_backward_cuda does not have a deterministic implementation, but you set ‘torch.use_deterministic_algorithms(True)’. You can turn off determinism just for this operation, or you can use the ‘warn_only=True’ option, if that’s acceptable for your application. You can also file an issue at <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a> to help us prioritize adding deterministic support for this operation.<br>
This seems to show that ReflectPad2D does not implement a deterministic algorithm when backpropagating, so if the network contains ReflectPad2D, it will not be able to fully reproduce the previous results(Even if I’ve fixed the random number seed)?<br>
Or how can I fully reproduce the results of a network experiment that includes ReflectPad2D？</p>
Torchaudio.functional.rnnt_loss crashes for logits with >2**31 elements

gpuplz1: <p>I filed an issue <a href="https://github.com/pytorch/audio/issues/3736" class="inline-onebox" rel="noopener nofollow ugc">`torchaudio.functional.rnnt_loss` crashes for `logits` with &gt;2**31 elements · Issue #3736 · pytorch/audio · GitHub</a> but saw a warning that the torchaudio repository is no longer actively monitored.  Is this forum a better place?  Any advice?</p>
Why is the input of hook function a tuple?

MahdiNazemi: <p>I have implemented a forward hook according to what is explained <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>
<p>When I apply the hook to an <code>nn.Linear</code> module, I see that the <code>input</code> in the hook function is a tuple of length one, i.e. (tensor, ), while the <code>output</code> is a tensor. Is there a reason why input is a tuple but the output is not?</p>

AntoineC: <p>I am asking myself the same question and was wondering if you had found an answer.</p>
<p>By way of example, I also started from the link you paste, but for a minimal working example I think <a href="https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html#forward-and-backward-function-hooks" rel="nofollow noopener">the official documentation</a> is very relevant.</p>

rhossein: <p>I also find it misleading:</p>
<p>Form the doc:</p>
<p>The <code>grad_input</code> and <code>grad_output</code> are tuples that contain the gradients with respect to the inputs and outputs respectively.</p>
Modify data of a tensor in-place

Nicu_Buliga: <p>Hello, I try to modify data of a PyTorch tensor in-place, specifically assign data of a new tensor to the existing tensor.  The problem is that I don’t want to reassign the new tensor to old tensor (old = new), because it will make old point to a new memory address. I cannot use copy_, because tensors are not broadcastable, and more than that the new tensors will always be larger than the old one (it is the result of concatenation).<br>
I tried to use data attribute and do something like old.data = new.data, but this causes some problems with autograd and the modifications of tensors will not affect the computation graph (<a href="https://github.com/pytorch/pytorch/releases/tag/v0.4.0" class="inline-onebox" rel="noopener nofollow ugc">Release Trade-off memory for compute, Windows support, 24 distributions with cdf, variance etc., dtypes, zero-dimensional Tensors, Tensor-Variable merge, , faster distributed, perf and bug fixes, CuDNN 7.1 · pytorch/pytorch · GitHub</a>).<br>
Do you have any idea how can I do this or at least if that’s possible?</p>

ptrblck: <aside class="quote no-group" data-username="Nicu_Buliga" data-post="1" data-topic="196110">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nicu_buliga/48/67264_2.png" class="avatar"> Nicu_Buliga:</div>
<blockquote>
<p>I cannot use copy_, because tensors are not broadcastable, and more than that the new tensors will always be larger than the old one (it is the result of concatenation).</p>
</blockquote>
</aside>
<p>I don’t understand how inplace operations should be possible if the sizes differ.</p>