akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to turn off requires_grad for individual params?

username: Sam_Lerman
cooked: <p>I know it’s trivial for a parameter vector, but when I iterate through the params in, for example:</p>
<p><code>torch.cat([param.flatten() for param in self.policy.model.parameters()])</code></p>
<p>setting requires_grad to false returns an error. How do I turn off the gradient for the individual param scalar weights?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Could you share a code sample of what you’re trying to do exactly and what is the exact error please?</p>

username: Sam_Lerman
cooked: <p>I have a <code>model</code> and I want to disable some of the weights, so I flatten the weights of the model in order to iterate through them and turn off <code>requires_grad</code> for a subset:</p>
<pre><code class="lang-auto">params = torch.cat([param.flatten() for param in model.parameters()])
for i, param in enumerate(params):
     if should_be_disabled[i]:
           param.requires_grad_(False)
</code></pre>
<p>This returns error that <code>requires_grad</code> can only be changed for leaves.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Your first line actually concatenates all the parameters in a single big Tensor in a differentiable manner.<br>
So the new Tensor params requires_grad property is independent from the one in your parameters.<br>
You can disable gradients for the Tensors in <code>model.parameters()</code> though:</p>
<pre><code class="lang-python">for j, p in enumerate(model.parameters()):
    if should_be_disabled[j]:
        p.requires_grad_(False)
</code></pre>

username: Sam_Lerman
cooked: <p>The problem is that <code>p</code> is a vector in your example and I would like to disable individual scalar weights within that vector. (A subset of them)</p>

username: albanD
cooked: <p>I am afraid that this is not possible. Tensors are “elementary” autograd objects. And so either the whole Tensor requires gradients or not.</p>
<p>Note that you can just zero-out the gradients after they are computed if you just want to not have gradients for some entries in there. (you can even do that with a hook to make sure it happens every time a gradient is computed for that Tensor).</p>

username: Sam_Lerman
cooked: <p>That would still be good. Thanks! Could you provide an example of this please?</p>
<p>My only concern is that I do three separate backwards passes for three separate loss terms, and I’m worried it’ll get convoluted because each one requires different gradients to be zeroed out.</p>

username: albanD
cooked: <p>Sure</p>
<pre><code class="lang-python">def get_hook(param_idx):
   def hook(grad):
        grad = grad.clone() # NEVER change the given grad inplace
        # Assumes 1D but can be generalized
        for i in grad.size(0):
            if should_be_disabled[param_idx][i]:
                grad[i] = 0
        return grad
    return hook

for j, p in enumerate(model.parameters()):
    p.register_hook(get_hook(j))
    
</code></pre>
How to understand memory usage with PyTorch

username: Tomash
cooked: <p>Hi guys,<br>
I am trying to train model on a modified COCO database. During loading data to <code>dataloaders</code> (images and targets - code in the end of post) <code>htop</code> command shows me that I am running like 100 process and every one of it uses 60 GB of <code>VIRT</code> and about 40 GB of <code>RES</code>, but summary the <code>mem bar</code> shows only 50 GB/ 504 GB. How to understand it? How to be sure that I will not use to much memory?</p>
<p>Can you look on my code if I am doing it alright?</p>
<blockquote>
<pre><code>class LoadDataset(Dataset):
def __init__(self): 
    self.images = []
    self.targets = []
    
    img_path, ann_path = (
        "path_to_images",
        "path_to_annotations_json",
    )
    coco_ds = torchvision.datasets.CocoDetection(img_path, ann_path)
    for i in range(0, len(coco_ds)):
        img, ann = coco_ds[i]
        for a in ann:
            images, targets = collate(
                [img.copy(), img.copy()], [[a], [a]], coco_ds.coco
            )
            for t in targets:
                self.targets.append(t)
            for image in images:
                self.images.append(image)
                        
def __len__(self):
    return len(self.images)

def __getitem__(self, idx):
    img = self.images[idx]
    target = self.targets[idx]
    
    return (
        img,
        target,
    )
</code></pre>
<p>and later in code: …</p>
<p>train_loader = DataLoader(LoadDataset, batch_size=24, shuffle=True)</p>
</blockquote>
create_graph=True in backward() causes memory leakage

username: Huimin_ZENG
cooked: <p>Hi! I was trying to do backward on the first derivative (Jacobian). I observed that the usage of memory continues to grow if <code>y.backward(retain_graph=True,create__graph=True) </code>.</p>
<p>I have read this post <a href="https://discuss.pytorch.org/t/how-to-free-the-graph-after-create-graph-true/58476/4">https://discuss.pytorch.org/t/how-to-free-the-graph-after-create-graph-true/58476/4</a>, where it is said the graph will be deteted if the referece is deleted.</p>
<p>But I also found this post: <a href="https://github.com/pytorch/pytorch/issues/4661" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/4661</a>, stating that the leakage issue is still open.</p>
<p>I am confused. Could you please help me out? And I can’t use <code> torch.autograd.grad</code>, since my outputs <code>y</code> are vectors, not scalar outputs.</p>
<p>Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>The conclusion from the issue you linked is that this is expected behavior mostly (or something we should forbid people from doing).<br>
<code>torch.autograd.grad</code> works for vectors as well. What is the issue you encounter when trying to use it?</p>

username: Huimin_ZENG
cooked: <p>Ah! I see! So I should stick to <code>torch.autograd.grad</code>, right?</p>
<p>I just figured out how to use <code>torch.autograd.grad</code> for vectors couple minutes ago.</p>
<p>Thanks!</p>

username: albanD
cooked: <p>Yes you should stick to <code>torch.autograd.grad</code> and all will be good.</p>

username: ybj14
cooked: <p>Hi, I meet the same problem, but I want to backward on the first derivative w.r.t. network parameters. Since both <code>torch.autograd.grad</code> and <code>torch.autograd.functional.jacobian</code> only takes vector inputs while network parameters are tuple of tensors, is there a feasible way to do this?  Thanks in advance for any possible help!</p>
<p>This post: <a href="https://discuss.pytorch.org/t/get-gradient-and-jacobian-wrt-the-parameters/98240" class="inline-onebox">Get gradient and Jacobian wrt the parameters</a> helps get jacobian but I’m trying to backward further on Jacobian.  It would be really nice if PyTorch supports gradient w.r.t <a href="https://jax.readthedocs.io/en/latest/pytrees.html" rel="noopener nofollow ugc">PyTree</a> objects like jax.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Both these functions take either a single Tensor or a tuple of Tensors as input. So it should work just fine.</p>
__getattr__ of subclass of nn.Module misbehaves

username: ybj14
cooked: <p>I define a  <code>NE_Conv2d</code>  class and override its  <code>__getattr__</code> . When I try to get an existing attribute from one instance of this class,  <code>__getattr__</code>  is called and enter infinite recursion.</p>
<p>I know one need to deal with  <code>__getattr__</code>  carefully to avoid infinite recursion, but apparently the  <code>conv</code>  attribute already exists after  <code>__init__</code>  is called. So when I try to get  <code>conv</code>  attribute,  <code>__getattr__</code>  should not be called.</p>
<pre><code class="lang-auto">from torch import nn
class NE_Conv2d(nn.Module):
    '''Nonexpansive conv2d'''

    def __init__(self, *k, **kw):
        super(NE_Conv2d, self).__init__()
        self.conv = nn.Conv2d(*k, **kw)
        print('foo')

    def __getattr__(self, attr):
        return getattr(self.conv, attr)

a = NE_Conv2d(3, 32, 5)
print(a)
print(a.conv)
</code></pre>
<p>Above code should print info about  <code>a</code>  and  <code>a.conv</code> , but enter infinite recursion when trying to get  <code>a.conv</code> .</p>

username: ybj14
cooked: <p>OK silly me. After overloads <code>__getattr__</code>, <code>self.conv</code> causes infinite recursion.</p>
Problem with static quantization

username: pizuzadan
cooked: <p>Hi!<br>
I have some model:</p>
<pre><code class="lang-auto">class SomelLayer(nn.Module):
    # some code here
    def forward(self, x):
        dim = 3
        mean_x = x.mean(dim)
        mean_x2 = (x * x).mean(dim)
        std_x = torch.nn.functional.relu(mean_x2 - mean_x*mean_x).sqrt()
    #some code
</code></pre>
<p>I want to use static quantization.</p>
<pre><code class="lang-auto">class SomelLayer(nn.Module):
    def __init__(self, mode):
        super(SomelLayer, self).__init__()
        self.quant = QuantStub()
        self.dequant = DeQuantStub()

        self.f_mul = torch.nn.quantized.FloatFunctional()
        self.f_add = torch.nn.quantized.FloatFunctional()
        self.f_add_relu = torch.nn.quantized.FloatFunctional()
        self.f_mul_scalar = torch.nn.quantized.FloatFunctional()

    def forward(self, x):
        dim = 3
        x = self.quant(x)
        mean_x = x.mean(dim)
        mean_x2 = self.f_mul.mul(x, x).mean(dim)
        std_x = self.f_add_relu.add_relu(mean_x2, self.f_mul_scalar.mul_scalar(self.f_mul.mul(mean_x, mean_x), -1.0))
        std_x = std_x.sqrt()
        # some code
        x = self.dequant(x)
        return x
</code></pre>
<p>But I’m getting error during inference:</p>
<pre><code class="lang-auto">Traceback of TorchScript, original code (most recent call last):
  File ".../model.py", line 235, in forward
        std_x = self.f_add_relu.add_relu(mean_x2,
                                         self.f_mul_scalar.mul_scalar(self.f_mul.mul(mean_x, mean_x), -1.0))
        std_x = std_x.sqrt()
                ~~~~~~~~~~ &lt;--- HERE
RuntimeError: Could not run 'aten::empty.memory_format' with arguments from the 'QuantizedCPU' backend. 'aten::empty.memory_format' is only available for these backends: [CPU, CUDA, MkldnnCPU, SparseCPU, SparseCUDA, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].
</code></pre>
<p>Do I have to implement the sqrt operator myself to solve this problem? As here: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/README.md" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/README.md</a><br>
I have similar problems with other operators.</p>

username: pizuzadan
cooked: <p>Solution for this problem:</p>
<pre><code class="lang-auto">self.quant0 = QuantStub()
self.dequant0 = DeQuantStub()

std_x = self.dequant0(std_x)
std_x = std_x.sqrt()
std_x = self.quant0(std_x)
</code></pre>

username: Vasiliy_Kuznetsov
cooked: <p>Hi <a class="mention" href="/u/pizuzadan">@pizuzadan</a>, yes, a quantized kernel for sqrt is not implemented at the moment.  Doing it in fp32 is the workaround.</p>
Conv1d to detect anomalies in ecg signal

username: Alicja_K
cooked: <p>Hi, so I want to detect anomal peaks in ECG signal using 1d convolution by sliding normal looking ECG peak across the signal.  Here is function that I’m using.</p>
<pre><code class="lang-auto">def convolve(signal, filt, stride):
    
    signalt = torch.from_numpy(signal).float().view(1,1,-1)
    filt = torch.from_numpy(filt).view(1,1,-1)
    
    output = F.conv1d(input=signalt, weight=filt, stride=stride, padding=0)
    
    return output.numpy().ravel()
</code></pre>
<p>The output I get is an ecg signal that  has reduced anomaly peaks - I want to know which one those are so I was trying to get the difference between the original and the output, however the length does not match.</p>
<pre><code class="lang-auto">ef explain_anomalies(y, fil, stride, sigma=1.0):

    avg = convolve(y, fil, stride).tolist()
    residual = y - avg
    # Calculate the variation in the distribution of the residual
    std = np.std(residual)
    return {'standard_deviation': round(std, 3), 
            'anomalies_dict': collections.OrderedDict([(index, y_i) for 
                                                       index, y_i, avg_i in zip(count(), y, avg) 
              if (y_i &gt; avg_i + (sigma*std)) | (y_i &lt; avg_i - (sigma*std))])}
</code></pre>
<p>after calling explain_anomalies(ecg[‘signal’], sig_filt, stride=1)<br>
where sig_filt is normal QRS peak that I want to do convolution with I get</p>
<pre><code class="lang-auto">&lt;ipython-input-115-71fe810b0625&gt; in explain_anomalies(y, fil, stride, sigma)
     25 
     26     avg = convolve(y, fil, stride).tolist()
---&gt; 27     residual = y - avg
     28     # Calculate the variation in the distribution of the residual
     29     std = np.std(residual)

ValueError: operands could not be broadcast together with shapes (71006,) (70977,)
</code></pre>
<p>Why does convolution reduce the numer of samples even though I have stride set to 1?</p>

username: klory
cooked: <p>not sure what you data and filter looks like, but  <code>padding</code> may also change your output size</p>

username: Alicja_K
cooked: <p>thank you for your answer <a class="mention" href="/u/klory">@klory</a></p>
<p>here is my signal with length of 71006</p>
<pre><code class="lang-auto">array([ -152.,  -330.,  -306., ..., -2522., -2724., -2886.], dtype=float32)
</code></pre>
<p>and filter length is 30</p>
<pre><code class="lang-auto">array([ -320.,  -322.,  -340.,  -410.,  -678., -1100.,  -712.,  1580.,
        5190.,  8574., 11104., 12664., 12998., 12104., 10110.,  7030.,
        3232.,  -574., -3596., -4724., -3832., -2558., -2184., -2416.,
       -2560., -2474., -2374., -2394., -2434., -2394.], dtype=float32)
</code></pre>
<p>So padding param pads signal on both sides - it changes the data to compensate for filtering, am I getting this right?</p>

username: klory
cooked: <p>yes, you can change padding according to the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html" rel="noopener nofollow ugc">doc</a></p>
How to draw a plot like this

username: huanyu_zang
cooked: <p>Hi friends,<br>
I have read a paper named " ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks". Is there anyone have idea how this picture drawn? Like each curve with different colors in different painting palette? Sorry I don’t know how to describe this but looks fancy, and wanna get some ideas how to plot similar to this style. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7de06225b54445908a80620c7d6e1458b3e999c7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7de06225b54445908a80620c7d6e1458b3e999c7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7de06225b54445908a80620c7d6e1458b3e999c7_2_369x500.jpeg" alt="image" data-base62-sha1="hXyqe09seea0kp1qHcJTaNUXPef" width="369" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7de06225b54445908a80620c7d6e1458b3e999c7_2_369x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/7/d/7de06225b54445908a80620c7d6e1458b3e999c7.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/d/7de06225b54445908a80620c7d6e1458b3e999c7.jpeg 2x" data-dominant-color="677185"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">465×630 67.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: klory
cooked: <p>try <a href="https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.fill_between.html" rel="noopener nofollow ugc">fill_between</a> in Matplotlib</p>
About convert torch.LongTensor(l) to C++

username: Chen0729
cooked: <p>Hi, All</p>
<p>I am wondering how to convert label= torch.LongTensor(l) as torch::LongTensor does not exist.</p>
<p>Any suggestion is welcome.</p>
<p>Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can use the equivalent of <code>torch.tensor(l, dtype=torch.long)</code>.<br>
Not sure what is the exact c++ way to write that though <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: Chen0729
cooked: <p>Thanks a lot.</p>
<p>Actually, I use y=l[0].to(tensor:kFloat32)<br>
in dype in C++ in libtorch it seems that there is no “Long” but I am not sure if they are good approximation in this case.</p>

username: albanD
cooked: <p>I think the name is <code>torch::kInt64</code> of the torch namespace (or <code>at::kLong</code> but the torch version should be used I think).</p>
Bert for summarization (BertSum) problems

username: bk_geek
cooked: <p>I am trying to implement the model as defined here --&gt; <a href="https://github.com/nlpyang/BertSum" rel="noopener nofollow ugc">https://github.com/nlpyang/BertSum</a></p>
<p>Each input to bertsum can be considered a document with multiple sentences separated by [SEP][CLS]</p>
<p>In order to ensure that I have the basic model working, I am trying to overfit the model with a toy dataset where I randomly assign sentences to uppercase or lowercase and I am trying to get the model to predict all uppercase = 1 and all lowercase = 0.</p>
<p>The model does not converge when the input has multiple sentences separated by CLS. It appears to me that the CLS tokens are not training. However, when I have a single sentence and therefore a single CLS token it works (basic classification task).</p>
<p>Any ideas what may be wrong? My notebook with all debugging output is here --&gt; <a href="https://colab.research.google.com/drive/1NvrwbNAdYMguvu4KR2ReZUBHmj93PNkP?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1NvrwbNAdYMguvu4KR2ReZUBHmj93PNkP?usp=sharing</a></p>

username: Abhilash_Srivastava
cooked: <p>Can you try using the Adam optimizer instead of SDG optimizer?</p>

username: bk_geek
cooked: <p>Its worse with Adam… The predictions become overconfident and loss stops changing after a while</p>
Training Graphic

username: Giuseppe
cooked: <p>Hy guys i don’t understand why i have this kind of graphic, why i have these up and down in the training curve? My LR = 0.00000001 and WD = 0.00001, as optimizer i use Adam.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b299cc3b510b8d88e036cb438e24bb62d190e2de.png" data-download-href="https://discuss.pytorch.org/uploads/default/b299cc3b510b8d88e036cb438e24bb62d190e2de" title="Schermata del 2020-11-11 11-41-57"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b299cc3b510b8d88e036cb438e24bb62d190e2de_2_690x197.png" alt="Schermata del 2020-11-11 11-41-57" data-base62-sha1="ptYuZRrFcXrtwNg4MjRngFmdCF0" width="690" height="197" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b299cc3b510b8d88e036cb438e24bb62d190e2de_2_690x197.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b299cc3b510b8d88e036cb438e24bb62d190e2de_2_1035x295.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b299cc3b510b8d88e036cb438e24bb62d190e2de_2_1380x394.png 2x" data-dominant-color="F5F8FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Schermata del 2020-11-11 11-41-57</span><span class="informations">1737×498 54.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Is the plot showing the training loss for each iteration and are the peaks at the end of each epoch?<br>
If so, are you shuffling the data?</p>

username: Giuseppe
cooked: <p>In the dataloader i don’t suffle the data!</p>

username: ptrblck
cooked: <p>This could explain the peaks, as some samples might be harder to classify than others.<br>
If you shuffle the dataset, this cyclic behavior should disappear.</p>

username: Giuseppe
cooked: <p>Oh yes, i didn’t think about this. i’ve forgotten this thing. Thanks a lot.</p>
About converting for (datum, label) in self.test to C++ in libtorch

username: Chen0729
cooked: <p>Hi, All</p>
<p>I am trying to convert the following python code to C++ in libtorch</p>
<p>The python code is:<br>
d, l = [], []<br>
for (datum, label) in self.test:<br>
d.append(datum)<br>
l.append(label)</p>
<p>When I convert it in C++ code, do I need to obtain the size of the dataset for test please ?<br>
Any suggestion is welcome.</p>
<p>Thanks,</p>

username: Chen0729
cooked: <p>Typicall, when I write</p>
<p>std::vector<a>torch::Tensor</a> d;<br>
std::vector<a>torch::Tensor</a> l;<br>
for (auto&amp; datum: test.size())<br>
{<br>
…<br>
}</p>
<p>It gives me an error at “auto&amp; datum: test.size()” saying “Invalid range expression, no viable begin function available”</p>
<p>Thanks</p>

username: Chen0729
cooked: <p>And I have changed it to:<br>
for (int i=0; i&lt;test.size().value();i++)<br>
{<br>
torch::Tensor datum=test.get_batch([i]).data;</p>
<pre><code>    }
</code></pre>
<p>It is saying “expected lambda body of experession”</p>

username: ptrblck
cooked: <p>Assuming you are creating the <code>data_loader</code> via:</p>
<pre><code class="lang-auto">auto data_loader = torch::data::make_data_loader(
      std::move(dataset), ...
</code></pre>
<p>then this should work:</p>
<pre><code class="lang-auto">for (auto&amp; batch : *data_loader) {
    ...
</code></pre>

username: Chen0729
cooked: <p>Thanks a lot for your response. I appreciate that.</p>
How should I build this LSTM model in Pytorch?

username: temps1101
cooked: <p>On Pytorch, if you want to build a model like this, ⇓<a href="https://i.stack.imgur.com/AFr0j.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/AFr0j.png" alt="normal bi-LSTM" width="690" height="390"></a><br>
the code will be:</p>
<pre><code class="lang-python">import torch.nn as nn

BLSTM = nn.LSTM(3, 3, bidirectional=True) # input and hidden sizes are example. it doesn't have to be 3.
</code></pre>
<hr>
<p>But I want to build a model like this. ⇓<a href="https://i.stack.imgur.com/Fast7.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/Fast7.png" alt="I don't know" width="690" height="383"></a><br>
This model has two-layerd forward &amp; backward LSTM.</p>
<p><em><strong>My question is how to build this model in Pytorch.</strong></em></p>
<hr>
<p>The code I can think of is this:</p>
<pre><code class="lang-python">import torch.nn

BLSTM = nn.LSTM(3, 3, 2, bidirectional=True) # input and hidden sizes are example. it doesn't have to be 3.
</code></pre>
<p>I thought it’s wrong because this ⇓ would be the model if I used this code.<br>
<a href="https://i.stack.imgur.com/cvWgC.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/cvWgC.png" alt="cat" width="529" height="500"></a><br>
I feel that this model is a little different from the model I want to build.</p>
[Two same network update one] Apply two same network grad to one network

username: Isaac_Sim
cooked: <p>I have two same network A, B, which A is one step backprop behind of B.<br>
So B is one step ahead.</p>
<ol>
<li>I am making samples from A and B, grad tensor  “result” came from each networks’ forward function: result-A, result-B</li>
<li>Copying B’s weights to A,</li>
<li>and want to update both gradients to B therefore B is one step ahead again.</li>
</ol>
<p>To do <span class="hashtag">#3</span> I concatenated result-A and result-B but I realized that it wouldn’t work like that.<br>
How can I update both result-A and result-B to B?</p>
<p>====<br>
Just someone might wonder why I am trying to do the above:<br>
I am coding to make distributed learning for reinforcement learning, where multiple ‘player’ agents only produce the outputs from inputs, and apply to ‘trainer’ agent.</p>
<p>They all have same networks but different weights, all players might have weights of trainer at less than or equal to trainer agent.</p>
<p>If I use replay buffer like off-policy module it is easy - requiring just coding since it only replaces the reward at the action representing dimension, find loss, do backward.</p>
<p>But if it is on-policy using Categorical log_prob then it would be hard since the action candidates and actions are produced from the previous weights, not current one. I don’t think the trainer would update correctly when I simply change the values …</p>
Batchnorm for N*C*L

username: dhyey
cooked: <p>my tensor shape is 8 X 5 X 512 (N X C X L) and i want to apply batchnorm on L how to do it??</p>

username: Preetham_R_Patlolla
cooked: <p>You should try using torch.nn.BatchNorm1d(). Please find the relevant info on this pytorch page.</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html" target="_blank" rel="noopener nofollow ugc">BatchNorm1d — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: dhyey
cooked: <p>batchnorm1d normalize over C in N X C X L</p>

username: ptrblck
cooked: <p>You could <code>permute</code> the input tensor and swap <code>C</code> with <code>L</code> before applying the batchnorm and swap it back afterwards.</p>
Find adjacency matrix given a list of class labels

username: jpainam
cooked: <p>Hi,<br>
Supposing that I have <code>L = [1 1 1 0 0 2 2]</code>, where each number indicates a different class. Using MATLAB, I can write:</p>
<pre><code class="lang-auto">A = zeros(numel(L));
A = bsxfun(@eq,L,L.');
A(~L,~L) = 0;
</code></pre>
<p>which gives</p>
<pre><code class="lang-auto">A =
     1     1     1     0     0     0     0
     1     1     1     0     0     0     0
     1     1     1     0     0     0     0
     0     0     0     0     0     0     0
     0     0     0     0     0     0     0
     0     0     0     0     0     1     1
     0     0     0     0     0     1     1
</code></pre>
<p>Please, how can I do this in pytorch and with no loops?<br>
Thanks</p>
<p>This question is similar to this <a href="https://stackoverflow.com/questions/28276215/find-adjacency-matrix-given-a-list-of-class-labels" rel="noopener nofollow ugc">stackoverflow question</a></p>

username: jpainam
cooked: <aside class="quote no-group" data-username="jpainam" data-post="1" data-topic="102376">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpainam/48/5870_2.png" class="avatar"> jpainam:</div>
<blockquote>
<pre><code class="lang-auto">A = zeros(numel(L));
A = bsxfun(@eq,L,L.');
A(~L,~L) = 0;
</code></pre>
</blockquote>
</aside>
<p>Even this way doesn’t work as expected</p>
<pre data-code-wrap="python"><code class="lang-python">L = torch.tensor([1, 1, 1, 0, 0, 2, 2])
A = (L == L.view(-1, 1)).long()
tensor([[1, 1, 1, 0, 0, 0, 0],
        [1, 1, 1, 0, 0, 0, 0],
        [1, 1, 1, 0, 0, 0, 0],
        [0, 0, 0, 1, 1, 0, 0],
        [0, 0, 0, 1, 1, 0, 0],
        [0, 0, 0, 0, 0, 1, 1],
        [0, 0, 0, 0, 0, 1, 1]])
T = ~ (L != 0)
A[T, T] = 0
tensor([[1, 1, 1, 0, 0, 0, 0],
        [1, 1, 1, 0, 0, 0, 0],
        [1, 1, 1, 0, 0, 0, 0],
        [0, 0, 0, 0, 1, 0, 0],
        [0, 0, 0, 1, 0, 0, 0],
        [0, 0, 0, 0, 0, 1, 1],
        [0, 0, 0, 0, 0, 1, 1]])
&gt;&gt;&gt; 
</code></pre>
<p>which is a little bit different.<br>
Any help will be useful</p>
Same probability for every class

username: utkuumetin
cooked: <p>I trained Transformer model for mask word prediction but my model output always same for every class</p>
<pre><code class="lang-auto">tensor([[[-2.3026, -2.3026, -2.3026,  ..., -2.3026, -2.3026, -2.3026],
         [-2.3026, -2.3026, -2.3026,  ..., -2.3026, -2.3026, -2.3026],
         [-2.3026, -2.3026, -2.3026,  ..., -2.3026, -2.3026, -2.3026],
         ...,
         [-2.3026, -2.3026, -2.3026,  ..., -2.3026, -2.3026, -2.3026],
         [-2.3026, -2.3026, -2.3026,  ..., -2.3026, -2.3026, -2.3026],
         [-2.3026, -2.3026, -2.3026,  ..., -2.3026, -2.3026, -2.3026]]],
       device='cuda:0', grad_fn=&lt;LogSoftmaxBackward&gt;)
</code></pre>
<p>After torch.exp()</p>
<pre><code class="lang-auto">tensor([[[0.1000, 0.1000, 0.1000,  ..., 0.1000, 0.1000, 0.1000],
         [0.1000, 0.1000, 0.1000,  ..., 0.1000, 0.1000, 0.1000],
         [0.1000, 0.1000, 0.1000,  ..., 0.1000, 0.1000, 0.1000],
         ...,
         [0.1000, 0.1000, 0.1000,  ..., 0.1000, 0.1000, 0.1000],
         [0.1000, 0.1000, 0.1000,  ..., 0.1000, 0.1000, 0.1000],
         [0.1000, 0.1000, 0.1000,  ..., 0.1000, 0.1000, 0.1000]]],
       device='cuda:0', grad_fn=&lt;ExpBackward&gt;)
</code></pre>
<p>What can be wrong here? Thanks in advance</p>
What is the best input for lstm?

username: suits_cloud
cooked: <p>I have 1-hour resolution temperature data with the following sample input-1 format. I use the LSTM to find anomalies with this format successfully. But I am not sure if LSTM’s learn any temporal dependencies with this type of format. Do I need to use the format as shown in input-2 where I create subsequences for each day?</p>
<p>input1</p>
<pre><code class="lang-auto">2020-01-01 01:00:00,100
2020-01-01 02:00:00,200
2020-01-01 03:00:00,900
2020-01-01 04:00:00,300
2020-01-02 01:00:00,100
2020-01-02 02:00:00,200
2020-01-02 03:00:00 100
2020-01-02 04:00:00 300
</code></pre>
<p>input2</p>
<pre><code class="lang-auto">,1,2,3,4
2020-01-01,100,200,900,300
2020-01-02,100,200,100,300
</code></pre>
Replacing Trainable Parameters of a Model

username: kko
cooked: <p>Hi,</p>
<p>I have a very specific use case of PyTorch and I am not sure what would be the suggested way to implement it. So, I will first describe what I want to do. Then, how I attempted to do it.</p>
<p>Goal: Let say I have a model called net which has trainable parameters from those I am interested in say only in convolutional layers:</p>
<pre><code class="lang-auto">for module in net.modules():
   if is_conv_layer(module):
      module.weight # This is the weight I am interested in.
</code></pre>
<p>Now, what I want to do is to replace module.weight’s with another set of  Parameter’s let us call weight_new where weight_new=weight*weight2 Here, the important thing is that I want to update weight2 values and only use the initial values of module.weight.</p>
<p>Attempt:</p>
<pre><code class="lang-auto">for module in net.modules():
  if is_conv_layer(module):
    data = module.weight.data
    module.weight = weight2
    module.weight.data *= data
</code></pre>
<p>So, basically, I replace trainable parameters with new Parameters weight2 and only after that, I updated .data term so that it is the multiplication of weight and weight2. This is because I cannot modify Parameter before assigning it to module.weight.</p>
<p>When I minimize the losses for these, I realized that weight2 is not really updating itself. Am I doing it wrong, what would be the suggested way of doing this?</p>
<p>If I am not mistaken, I read that overwriting weight.data is not a suggested way to do things in PyTorch and it may be the reason behind the weird behaviors. Is it really the case?</p>
Creating tensors from input vectors in libtorch

username: John_J_Watson
cooked: <p>I have some vectors which I create by reading a CSV (using <code>rapidcsv.h</code>) and I was wondering how I can get it to an input tensor so that I can use this as my model input.</p>
<p>I have:</p>
<blockquote>
<pre><code>      std::vector&lt;float&gt; some_a = doc.GetColumn&lt;float&gt;(0);
      std::vector&lt;float&gt; some_b = doc.GetColumn&lt;float&gt;(1);
      std::vector&lt;float&gt; some_c = doc.GetColumn&lt;float&gt;(2);
      std::cout &lt;&lt; "Read " &lt;&lt; some_a.size() &lt;&lt; " values." &lt;&lt; std::endl; //out: 512
      std::cout &lt;&lt; "Read " &lt;&lt; some_b.size() &lt;&lt; " values." &lt;&lt; std::endl; //out: 512
      std::cout &lt;&lt; "Read " &lt;&lt; some_c.size() &lt;&lt; " values." &lt;&lt; std::endl; //out:512

      std::vector&lt;torch::jit::IValue&gt; inputs;
      inputs.push_back(torch::zeros({1, 3, 512}));
</code></pre>
</blockquote>
<p>I would like to assign <code>some_a</code>, <code>some_b</code> and <code>some_c</code> to the 3 tensors in <code>inputs</code> [0,1,:], [0,2,:], [0,3,:].</p>
<p>how does one go about achieving this?</p>
Q-values increase, also with negative TD

username: HappyLemon
cooked: <p>I set up a simple DQL task, but it wasn’t learning, so I decreased the number of steps and samples to 1 to track down the error.</p>
<p>The TD values are calculated correctly, but no matter if they are positive or negative, after the update, the corresponding Q-value increased. I use :</p>
<p>optimiser = torch.optim.Adam(model.parameters(), lr=1e-3, eps=1e-3, amsgrad=True)<br>
gamma = 0.9<br>
for episode in range(episodes):<br>
TD = []<br>
…<br>
for step in range(run_length):<br>
…<br>
TDstep = reward + gamma*torch.max(Qnext.detach(), dim =1).values - Q[range(batch_size),action]<br>
TD.append(TDstep)</p>
<pre><code>TD = torch.stack(TD).sum()
TD.backward()

optimiser.step()
optimiser.zero_grad()
</code></pre>
<p>With example output:</p>
<p>action taken: 2<br>
Q-values: tensor([-0.0832, -0.5799,  0.6861, -0.1725, -0.0538])<br>
reward: tensor(-0.0050)<br>
max Q(s+1): tensor([0.6873])<br>
TD: tensor([-0.0726], grad_fn=)<br>
action taken: 4<br>
Q-values: tensor([-0.0823, -0.5819,  0.6942, -0.1716, -0.0544])<br>
reward: tensor(-0.1900)<br>
max Q(s+1): tensor([0.6797])<br>
TD: tensor([0.4761], grad_fn=)<br>
action taken: 2<br>
Q-values: tensor([-0.0809, -0.5839,  0.7002, -0.1697, -0.0506])<br>
reward: tensor(-0.0050)<br>
max Q(s+1): tensor([0.6873])<br>
TD: tensor([-0.0867], grad_fn=)<br>
action taken: 1<br>
Q-values: tensor([-0.0796, -0.5859,  0.7069, -0.1680, -0.0477])</p>
<p>As you can see, though the TD for action 2 in the first step is negative, the Q-value for action 2 increased after the update. The TD for action 4 is positive and the Q-value for action 4 also increases after the corresponding update.</p>
<p>I checked the updated weights and only the right weights are affected. So I’m clueless.</p>

username: Chris_Freiburg
cooked: <p>I have a similar problem to bad nobody answered<br>
Did you find your error?</p>
OOM, process is killed

username: Erik_Bertolino
cooked: <p>What can I do to mitigate this? Is it a version issue? I have the newest version of pytorch and python.</p>
<p>Is it an issue with<br>
-the code (refrain from saving all variables, making the mini-batch smaller?)<br>
-the operating system (I use ubuntu 20.04)</p>
<p>The specific error I get when running dmesg in the terminal is “Out of memory: Killed process”</p>

username: albanD
cooked: <p>Hi,</p>
<p>It is very hard to say with no information about what you were doing.<br>
Is it actually your python process that was eating up all the memory or something else?<br>
If it was the python process, what were you doing when it happened?</p>
Accessing constructors for function-specific outputs in torch.return_types

username: jemisjoky
cooked: <p>I’m working on some wrappers for Pytorch functions with the goal of <a href="https://pytorch.org/docs/1.7.0/notes/extending.html#extending-torch" rel="noopener nofollow ugc">extending the Tensor class</a>, and am running into an issue with functions like <code>torch.mode</code> (among many others) that return function-specific types like <code>torch.return_types.mode</code>.</p>
<p>I’d like to replace the <code>values</code> attribute of the output object with a different tensor before returning it, but the output is immutable and I can’t find the constructors for the classes in <code>torch.return_types</code> anywhere.</p>
<p>Any idea where these constructors are located (assuming they’re implemented somewhere in Python)? I’d be alright with returning a plain tuple instead, but would love if there is a means of handling this that’s more faithful to the original Pytorch behavior. Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can re-use the original result to create a new instance of the same class:</p>
<pre><code class="lang-python">out = torch.mode(inp)
final_out = out.__class__(new_values)
</code></pre>

username: jemisjoky
cooked: <p>Perfect, thank you <a class="mention" href="/u/alband">@albanD</a>!</p>
Simple feed forward network with targets between 0 and 1 doesn't learn well

username: agt
cooked: <p>I’m trying to fit this very simple data with targets in the range (0, 1), but even when two values are far apart, like .2 and .45, the loss is very small. So I end up with a very small loss, but the predictions are very wrong. How can improve my predictions?</p>
<p>Here’s my data and the model predictions after training for 1000 epochs with Adam, MSE, and a learning rate of .001. Trying BCELoss didn’t improve the results.</p>
<blockquote>
<p>Inputs = [[14.0110, 2.0000], [18.9990, 3.0000], [12.0110, 0.0000], [16.9990, 1.0000]]<br>
Targets = [0.6774895, 0.12747164, 0.02246823, 0.00056751847]<br>
Predictions = [0.37114963, 0.31669545, 0.1016788, 0.09256815]</p>
</blockquote>
<p>The last sample’s prediction is 163 times bigger than its target, but the loss is tiny (0.0084).</p>
<p>Here’s my model:</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self, num_features):
        super().__init__()

        self.linear_hidden_1 = torch.nn.Linear(num_features, num_features * 2)
        self.linear_out = torch.nn.Linear(num_features * 2, 1)

    def forward(self, inputs):
        hidden = torch.nn.ReLU()(self.linear_hidden_1(inputs))

        out = torch.nn.Sigmoid()(self.linear_out(hidden)).flatten()

        return out
</code></pre>
<p>I would really appreciate any suggestions how to get it to predict properly.</p>

username: ptrblck
cooked: <p>If your target values have a small values range and are thus also yielding a small loss, you could try to e.g. increase the learning rate. Alternatively, you could also normalize the targets to a “proper” range during training and “unnormalize” the model predictions during evaluation to get the real prediction values again.</p>

username: agt
cooked: <p>If the target range is so small, does it really matter that the loss is small too?</p>
<p>Since updates to the weights have to be small since the values are packed together in such a small range, I feel like the small loss might not be a problem (e.g. if the range was (0,10), the weights &amp; loss might just be scaled up by 10 &amp; 10^2 without changing learning performance much).</p>
<p>I tried to train with targets scaled up to (0,100) but it didn’t noticeably improve performance.</p>
How to trace jit in evaluation mode using multi-gpu learned model?

username: LvJC
cooked: <p>I used multi-gpu to train my model, but when I wanted to use jit trace to get <code>.pt</code> model, an error occured.</p>
<p>Here’s my training codes.</p>
<pre><code class="lang-python">train_loader, test_loader = get_dataloader(batch_size, root)
    gpus = list(range(torch.cuda.device_count()))
    se_resnet = nn.DataParallel(se_resnet50(num_classes=102),
                                device_ids=gpus)
    optimizer = optim.SGD(params=se_resnet.parameters(), lr=0.6 / 1024 * batch_size, momentum=0.9, weight_decay=1e-4)
    scheduler = optim.lr_scheduler.StepLR(optimizer, 30, gamma=0.1)
    trainer = Trainer(se_resnet, optimizer, F.cross_entropy, save_dir="./102flower_model_weights")
</code></pre>
<p>Here’s my jit tracing code.</p>
<pre><code class="lang-python">gpus = list(range(torch.cuda.device_count()))
model = nn.DataParallel(se_resnet50(num_classes=102),device_ids=gpus).cuda()
checkpoint = torch.load('102flower_model_weights/model_epoch_70.pth')
model.load_state_dict(checkpoint["weight"])#, strict=False)
model.eval()
# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)
# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)
# save
traced_script_module.save("102flower_model_weights/model_epoch_70.pt")
</code></pre>
<p>Error info:</p>
<pre><code class="lang-auto">    traced_script_module.save("102flower_model_weights/model_epoch_70.pt")
RuntimeError:
could not export python function call Scatter. Remove calls to python functions before export.:
</code></pre>

username: smth
cooked: <p>this is a known limitation of the tracing / jit at the moment – that it doesn’t support nn.DataParallel right now.</p>
<p>What you can do is to trace your model inside of DataParallel.</p>
<p>For example:</p>
<pre><code class="lang-auto">traced_script_module = torch.jit.trace(model.module, example)
</code></pre>

username: rtkaratekid
cooked: <p><a class="mention" href="/u/smth">@smth</a> is this resolved? Or are we still tracing inside of DataParallel? Sorry to dredge this post up after so long</p>

username: Michael_Suo
cooked: <p>We do not support passing a DataParallel module to <code>torch.jit.trace()</code>. As <a class="mention" href="/u/smth">@smth</a> mentioned, one alternative is to trace the inner module, then call <code>DataParallel()</code> on it.</p>

username: Arvind_Vepa
cooked: <p>So I traced a module and then wanted to apply <code>DataParallel</code>. While it doesn’t throw an error, inference is much slower than the regular torch version (5x-10x).</p>

username: Erun
cooked: <p>I have encountered the same problem ? Have you solved it ? I need help urgently, thank you.</p>

username: zimingzheng
cooked: <p>Is there any plan to resolve this issue please?</p>

username: Michael_Suo
cooked: <p>There’s a lot of interest in this, so I’d like to consolidate the discussion here: <a href="https://github.com/pytorch/pytorch/issues/47650" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/47650</a>.</p>
Get_default_qconfig appears in docs but not in API Reference

username: ResidentMario
cooked: <p>The <a href="https://pytorch.org/blog/introduction-to-quantization-on-pytorch/" rel="noopener nofollow ugc">Quantization Release Blog Post</a> and <a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">various</a> <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="noopener nofollow ugc">demos</a> in the docs make heavy use of the <code>torch.quantization.get_default_qconfig</code> method for configuring quantization algorithm behavior. Oddly, the <a href="https://pytorch.org/docs/stable/torch.quantization.html?highlight=quantization#module-torch.quantization" rel="noopener nofollow ugc">torch.quantization API Reference</a> makes no reference to this function.</p>
<p>This is unfortunate, because it means there isn’t anywhere to go to get a list of valid parameters to this method. If others agree, and this isn’t intentional, I’d be happy to fine a ticket for this in the GH tracker.</p>

username: Vasiliy_Kuznetsov
cooked: <p>Hi <a class="mention" href="/u/residentmario">@ResidentMario</a>, that’s a good point.  You can check out the definition of the function here: <a href="https://github.com/pytorch/pytorch/blob/20ac7362009dd8e0aca6e72fc9357773136a83b8/torch/quantization/qconfig.py#L81" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/20ac7362009dd8e0aca6e72fc9357773136a83b8/torch/quantization/qconfig.py#L81</a>.  Filed <a href="https://github.com/pytorch/pytorch/issues/48106" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/48106</a> to track adding it to the docs.</p>
Multiple NCCL calls reduce training speed?

username: Yi_Zhang
cooked: <p>Hi,</p>
<p>I tried to run 2 training jobs on a GPU cluster with 8 GPUs, both jobs need to use NCCL AllReduce. And I notice the training speed is slower if running the 2 jobs at same time than running them separately. Is this because they are competing for the bandwidth between GPU communications (the AllReduce call)? Thanks.</p>

username: rvarm1
cooked: <p>Hi <a class="mention" href="/u/yi_zhang">@Yi_Zhang</a> Yes, I would think that this is likely the case and running multiple training jobs will result in more competition for the GPU’s bandwidth and thus overall a slower performance compared to running either job individually.</p>
<p>Are you noticing any extreme slowness/hangs that may be more indicative of a bug?</p>

username: Yi_Zhang
cooked: <p>Hi <a class="mention" href="/u/rvarm1">@rvarm1</a>, thanks for reply. I’m not sure if it is a bug, since the program doesn’t hang, but it affects the speed greatly. And if I reduce the sync frequency, it can help to speed up, so I feel the communication between GPUs may be a bottleneck.</p>
Pickle local object with DistributedDataParallel

username: Jimmy2027
cooked: <p>Hi, I am trying to make my code work with <code>torch.nn.parallel.DistributedDataParallel</code>. However in my code I need to load some files which gives me the error: <code>AttributeError: Can't pickle local object</code>.<br>
I guess the problem is the processes trying to access the data concurrently? Is there any way to make this work?</p>

username: rvarm1
cooked: <p>I’m not clear on how this is an issue with <code>DistributedDataParallel</code> which does not in itself do any pickling. Could you share a minimal script that reproduces the issue you’re seeing so we can debug further? Thanks!</p>

username: Jimmy2027
cooked: <p>Hi <a class="mention" href="/u/rvarm1">@rvarm1</a> thanks for your answer, again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> An example would be using tensorboard, as I mentioned in another post: <a href="https://discuss.pytorch.org/t/using-tensorboard-with-distributeddataparallel/102555" class="inline-onebox">Using tensorboard with DistributedDataParallel</a></p>
Copy a row from one tensor to another tensor row

username: Robin_Lobel
cooked: <p>What would be the easiest way with pytorch to say copy the 2nd row from tensor A(32,1,16,16) to the 5th row of tensor B(32,1,16,16)?</p>
<p>By “row” I mean an index from the first dimension. So my copied block would be of size (1,1,16,16).<br>
By “copy” I mean overwriting the 5th row of tensor B with data from the 2nd row of tensor A.</p>
<p>Bonus question, is there a swap operation doing that? (otherwise 3 copies with an intermediate buffer will do the trick)</p>
<p>For a bit of context, the bigger picture is that I have several tensors like A (my training set sliced in batch of 32 samples (1,16,16)) and I’d like to shuffle them at each epoch, which requires swapping rows around across the different training tensors.</p>

username: Robin_Lobel
cooked: <p>Found how to do it:</p>
<pre><code class="lang-auto">void Training::swapTensorRow(torch::Tensor tensorA, int indexA, torch::Tensor tensorB, int indexB)
{
    torch::Tensor cache=tensorA.index({indexA}).clone();
    tensorA.index_put_({indexA},tensorB.index({indexB}));
    tensorB.index_put_({indexB},cache);
}
</code></pre>
<p>More detail about indexing PyTorch vs LibTorch: <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/notes/tensor_indexing.html</a></p>