akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to plot roc curve for custom NN classifier

username: maito
cooked: <p>Hi!</p>
<p>I have a custom classifier (binary class), and would like to plot ROC curve. I understand that I have to feed the X test values and the probabilities that either value would belong to class 1 or 0. This is where I’m stuck: all the codes that I have seen use libraries such as Scikit Learn, which have their own methods for this. How do I calculate these probabilities using my own classifier?</p>
<p>The code to my classifier is:</p>
<pre><code class="lang-auto">
class NNClassifier(torch.nn.Module):
    def __init__(self):
        super(NNClassifier, self).__init__()
        self.linear = torch.nn.Linear(672, 1)
        self.layer_1 = nn.Linear(672, 300) 
        self.layer_2 = nn.Linear(300, 300)
        self.layer_out = nn.Linear(300, 1) 
        
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(p=0.1)
        self.batchnorm1 = nn.BatchNorm1d(300)
        self.batchnorm2 = nn.BatchNorm1d(300)
        
        for m in self.modules():
            if m.__class__ == nn.Conv2d or m.__class__ == nn.Linear:
                init.xavier_normal_(m.weight.data)
                m.bias.data.fill_(0)
                
    def forward(self, inputs):
        x = self.relu(self.layer_1(inputs))
        x = self.batchnorm1(x)
        x = self.relu(self.layer_2(x))
        x = self.batchnorm2(x)
        x = self.dropout(x)
        x = self.layer_out(x)
        
        return x
</code></pre>
<p>Any help is appreciated!</p>

username: tom
cooked: <p>Hi,</p>
<p>so you should be getting outputs of shape <code>batch_size x 1</code> from your model. If you loop over your test set and collect them and the labels into two lists as discussed in <a href="https://discuss.pytorch.org/t/issues-calculating-roc-score-for-binary-classification/94675">this thread</a> and concatenate (torch.cat) to get prediction scores and labels for the entire dataset.</p>
<p>You can then either feed the result to SKLearn or do this manually as we did <a href="https://github.com/deep-learning-with-pytorch/dlwpt-code/blob/master/p2ch14_malben_baseline.ipynb" rel="nofollow noopener">in Chapter 14 or our book</a> by sorting the predictions and then getting tp / fp per threshold with broadcasting (cell 4) and then doing the numerical integration using the trapezoidal rule (cell 5).</p>
<p>Best regards</p>
<p>Thomas</p>
Data parallelization in PyTorch

username: Shahrzad
cooked: <p>I’m running a simple code performing 1D convolution on an input array, and a given filter specification, as follows:</p>
<pre><code class="lang-auto">import argparse
import sys
import torch.nn as nn
import torch
import time

parser = argparse.ArgumentParser(description='Parameters')
parser.add_argument('integers', type=int, nargs=4,
                    help='batch, length, out_channels, kernel_size')

args = parser.parse_args()

batch = args.integers[0]
length = args.integers[1]
out_channels = args.integers[2]
kernel_size = args.integers[3]

array = torch.rand(batch, 3, length)
conv1 = nn.Conv1d(in_channels=3,out_channels=out_channels, kernel_size=kernel_size, bias=False)

t_start = time.time()
out = conv1(array)
t_end = time.time()
t = t_end-t_start

</code></pre>
<p>I run my code on a single machine with 40 cores. When I measure the time I see that for some filter lengths I don’t see any performance degradation when the number of batches increases, which is strange. Since I’m not even using <code>nn.parallel</code> or <code>distributed</code> module, so my question is does these results make sense? How is PyTorch designed to handle data parallelization in this case? Is there a threshold for batch size from which it starts parallelization? If it does parallelization does it simply divide the batches among the cores and assigns each chunk to a core?<br>
The following image shows the results of running the above-mentioned code on one node while keeping input length, filter length, and out channels fixed (50, 10 ,5 in this image) and increasing the batch size.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b3fbf920d6e9546199ef8deb26ddb9b2dd7b97f.png" alt="medusa_1_50-10-5" data-base62-sha1="8s8KyucZFzMvVz7ZWZ78hlb7YyX" width="482" height="278"></p>

username: tom
cooked: <p>You typically have two forms of parallelism on the CPU, intra-op (so one operation is executed on several CPUs) and inter-op (so several operations are launched in parallel).<br>
You might see that the first works better for larger batches.</p>
Shared variable in GPU

username: Yi_Zhang
cooked: <p>Hi,</p>
<p>I want to have two parallel processes in one GPU, one for training (calculating) and the other for communicating parameter updates with other GPUs. And both processes can modify a shared variable, (like a buffer to store the most updated parameters). Anyone knows is it possible to do this?<br>
I checked this documentation: <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/multiprocessing.html</a>, and it mentions “multiprocessing.Queue”, not sure is it suitable in my case? Or any good examples?</p>

username: mrshenli
cooked: <p>This should be doable, see the example code in this post:</p>
<aside class="quote quote-modified" data-post="9" data-topic="90636">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/exception-process-0-terminated-with-exit-code-1-error-when-using-torch-multiprocessing-spawn-to-parallelize-over-multiple-gpus/90636/9">`Exception: process 0 terminated with exit code 1` error when using `torch.multiprocessing.spawn` to parallelize over multiple GPUs</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    You can use <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.share_memory_" rel="noopener nofollow ugc">share_momery_()</a> and torch.multiprocessing.SimpleQueue to implement IPC. E.g.: 
import numpy as np
import torch
import torch.multiprocessing as mp


def func(rank, x, p2c, c2p):
    x_power = x.to(rank) ** rank
    c2p.put(x_power)
    # citing multiprocessing doc: Unlike CPU tensors, the 
    # sending process is required to keep the original tensor 
    # as long as the receiving process retains a copy of 
    # the tensor. The refcounting is implemented under the 
    # hood but re…
  </blockquote>
</aside>
Is there a way to optimize/vectorize this pytorch operation?

username: tiwa
cooked: <p>I have two tensors,  <code>A</code>  and  <code>B</code> , and the values at corresponding indices  <code>A[i][j]</code>  and  <code>B[i][j]</code>  parametrize a distribution ( <code>Kpdf</code>  below). I’d like to, for every index, generate  <code>n_samples</code>  samples (see  <code>samples</code> , which are generated from  <code>noise</code>  using  <code>T_</code> ) and calculate a statistic ( <code>kl_</code> ). Then I want to sum up all of these statistics and return the result.</p>
<p>Is there a good way to do this? Here is the code I have so far:</p>
<pre><code class="lang-auto">eps = 1e-12

T_ = lambda x, a, b: torch.pow(1 - torch.pow(1-x,1/b), 1/a)
Kpdf = lambda x, a, b: a * b * torch.pow(x,a-1) * torch.pow((1-torch.pow(x,a)), b-1)

kl = torch.tensor([0.0], dtype=torch.float, requires_grad=True).to(A.device)

for i in range(A.shape[0]):
    for j in range(A.shape[1]):
        a = A[i][j]
        b = B[i][j]
        noise = torch.FloatTensor(n_samples).uniform_(0, 1).to(A.device)
        samples = T_(noise, a, b)
        kl_ = torch.log(Kpdf(samples, a, b)/(eps + prior(samples)))
        kl = kl + kl_.sum()       

return kl
</code></pre>

username: googlebot
cooked: <p>Yes, generally, you can use (i,j,1) shapes for distribution parameters, (i,j,n_samples) for noise &amp; samples. With that, you have expanded everything and can reduce results as needed.</p>
Weights & Biases

username: chaslie
cooked: <p>morning,</p>
<p>I am confused. When i store the weights and bias from a convolutional layer using:</p>
<pre><code class="lang-auto">kernels_int_in =conv2d.weight.detach().cpu()
conv2d.bias.detach().cpu()
</code></pre>
<p>the output shape for weight is [out_channel,in_channel,kernal_height,Kernal_width] and the size of the bias is a 1D vector of size [out_channel].</p>
<p>when the weights are plotted using:</p>
<pre><code class="lang-auto">            kernels_t = kernels_int_in - kernels_int_in.min()
            kernels_t = kernels_t / kernels_t.max()
            ind=0
            fig, axarr = plt.subplots(4,32)
            plt.subplots_adjust(wspace=.02, hspace=.02)
            while ind !=128:
                for idx in range(4):
                    for idy in range(32):
                        axarr[idx,idy].imshow(kernels_t[ind,0,:,:].squeeze())
                        # axarr[idx,idy].imshow(act[ind])
                        ind=ind+1
</code></pre>
<p>the question is, why are the in_channels stored and what are they telling me? Also over 30 epochs in my case the values of the out_channels do not change significantly, is this suggesting that the kernels are not appropriate since the overall error stays quite high?</p>

username: ayalaa2
cooked: <aside class="quote no-group" data-username="chaslie" data-post="1" data-topic="94718">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/48.png" class="avatar"> chaslie:</div>
<blockquote>
<p>why are the in_channels stored and what are they telling me?</p>
</blockquote>
</aside>
<p>Can you expand on this? I’m not sure what you mean by the in_channels being stored. Are you asking about that dimension in general? I see that in your plot making that you’re always indexing into in_channels with 0 (so you might not be seeing all the data in your plots).</p>
<aside class="quote no-group" data-username="chaslie" data-post="1" data-topic="94718">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/48.png" class="avatar"> chaslie:</div>
<blockquote>
<p>values of the out_channels do not change significantly</p>
</blockquote>
</aside>
<p>Convolution kernel weights are typically pretty small in the first place. For example, in resnet18, some of their kernels weights, on average, are around 0.002 or even 0.0002 (depending on where you sample). I think it would be more appropriate to take a look at their gradients overtime. That will be a better indicator of seeing how those layers are doing.</p>

username: chaslie
cooked: <p>what i am trying to plot is the actual kernals that are used in conv2D</p>

username: ayalaa2
cooked: <p>You need to iterate over your in channels to get all the information about your kernels. Conv2D kernels are 4 dimensional shapes that act upon every single input channel.</p>
I can train resnet model without changing the last layer of the model

username: Sassan_Owssia
cooked: <p>I’m using a resnet model to train my data which has 4 classes. However, the resnet model in torchvision trained on the Imagenet dataset has 1000 output features. I have not changed the last layer of the resnet to match the classes of my data. However, the model does not raise any errors and starts the training process without issue. I am a beginner and was wondering how this works.</p>

username: RaLo4
cooked: <p>You are basically still training your model to classify images into 1000 classes but your trainingsset just does not include images for 996 of those classes.<br>
There are no errors to raise since this is “technically not wrong”. It’s just like you are using a really shitty dataset.</p>
Transformer Classification (Overfitting ?)

username: Unity05
cooked: <p>I have built a simple transformer classifier consisting of multiple <code>TransformerEncoderLayers</code>.<br>
I have tried a variety of different values for hyperparameters (<code>model dimension</code>, <code>number of heads</code>, <code>hidden size</code>, <code>number of layers</code>, <code>initial learning rate</code> and <code>batch size</code>). I have also tried different <code>dropout probabilities</code> (0.2, 0.5 and 0.75). My <code>vocabulary size</code> is 100,000.<br>
All different combinations have lead to a following loss pattern:<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/741f28973d0380e8d74d7512ec53813224f34ba0.png" alt="loss_graph_0" data-base62-sha1="gzg8TvTH1sf1mkM9xJFtbOOz5iE" width="640" height="480"></p>
<pre><code class="lang-auto">blue graph: training loss
orange graph: validation loss
</code></pre>
<p>Of course, they differ by a tiny amount, but the overall result has always been the same.<br>
I have also tried another Dataset to make sure the data is not the problem, but the results were even worse.<br>
The <code>training loss</code> just stays at about the same value from the 15th unit on until the next epoch starts from unit 22 on, where the <code>training loss</code> suddenly decreases by about 0.02, but then stays the same again for the entire epoch.<br>
The <code>validation loss</code> does not change significantly from unit 15 on.<br>
It looks very much like overfitting, but it seems quite strange too me, that the model starts overfitting that early with such a bad loss.</p>
<p>I appreciate any hints or help, if somebody has already experienced something like this.</p>
<p>Regards,<br>
Unity05</p>
Open Source contribution

username: SHIVAM_PANDEY
cooked: <p>I have been using Pytorch in my my college years, but I have never done any open source contribution, I would be glad to know, on how to start open source contribution in pytorch</p>

username: suraj.pt
cooked: <p>Welcome Shivam! Take a look at <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md</a> to get started.</p>
GrayScale to RGB

username: Himani_Gulati
cooked: <p>Can I know how to convert Grayscale images to RGB so I can use a pretrained model on them.</p>

username: Lin_Jia
cooked: <p>would the answer here be helpful to you?<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/40119743/convert-a-grayscale-image-to-a-3-channel-image" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1957969/dylan" target="_blank" rel="nofollow noopener">
    <img alt="Dylan" src="https://www.gravatar.com/avatar/d4eab0de4090438ab23b2ddecd3b0368?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/40119743/convert-a-grayscale-image-to-a-3-channel-image" target="_blank" rel="nofollow noopener">convert a grayscale image to a 3-channel image</a>
</h4>

<div class="tags">
  <strong>python, numpy</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1957969/dylan" target="_blank" rel="nofollow noopener">
    Dylan
  </a>
  on <a href="https://stackoverflow.com/questions/40119743/convert-a-grayscale-image-to-a-3-channel-image" target="_blank" rel="nofollow noopener">11:34PM - 18 Oct 16 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>They just make RGB channel all equal to the grayscale value. I suspect it should be good enough from deep learning perspective, though the actual image can be visually distorted a bit. BTW, if you use a pretrained model, I high recommend you fine tune it a bit. Cause I think it is likely the pretrained model is trained on real RGB image.</p>

username: RaLo4
cooked: <p>torchvisions transforms has a function called <code>torchvision.transforms.Grayscale(num_output_channels=1)</code>. See <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Grayscale" rel="nofollow noopener">here</a>.</p>
<p>Using <code>num_output_channels=1</code> this can be used to convert an 3 channel RGB image into a 1 channel grayscale image.</p>
<p>But if used with <code>num_output_channels=3</code> this creates a 3 channel image with R=G=B.</p>
<p>You can use this function on your 1 channel image to make a 3 channel image where R=G=B, if this is what you want.</p>
Alfred workflow to quickly jump to PyTorch API docs

username: lsgrep
cooked: <p>Hi, I’ve created an <a href="https://www.alfredapp.com/" rel="noopener nofollow ugc">Alfred</a> workflow to quickly jump to the official PyTorch API docs<br>
<a href="https://github.com/lsgrep/mldocs" rel="noopener nofollow ugc">https://github.com/lsgrep/mldocs</a><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8088e1af8c3345385e77d5b24e889898b01bc97.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b8088e1af8c3345385e77d5b24e889898b01bc97" title="CleanShot 2020-09-01 at 17.55.30@2x"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8088e1af8c3345385e77d5b24e889898b01bc97_2_538x500.jpeg" alt="CleanShot 2020-09-01 at 17.55.30@2x" data-base62-sha1="qg2boroOk82xKuNOtr8x3P8bWzJ" width="538" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8088e1af8c3345385e77d5b24e889898b01bc97_2_538x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8088e1af8c3345385e77d5b24e889898b01bc97_2_807x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8088e1af8c3345385e77d5b24e889898b01bc97_2_1076x1000.jpeg 2x" data-dominant-color="424548"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">CleanShot 2020-09-01 at 17.55.30@2x</span><span class="informations">1364×1266 335 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Results of training on 3D data

username: shangguan91
cooked: <p>i did some reimplementation of benchmark(best model) Rsconv on my own dataset. Normally, other algorithm perform well, at start the train and test accuracy both surges and be steady at certain epoch.</p>
<p>However, when i train this model. after i adjust learning rate the performance of model is till around 60.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/aeb2d2e7792c3582c5bf880dcab5f73f52c780a6.png" data-download-href="https://discuss.pytorch.org/uploads/default/aeb2d2e7792c3582c5bf880dcab5f73f52c780a6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/e/aeb2d2e7792c3582c5bf880dcab5f73f52c780a6.png" alt="image" data-base62-sha1="oVscUeEh16cTAVE3ZkwS3uD9Gse" width="690" height="305" data-dominant-color="EBF2EC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1227×543 18.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I also reimplement it on benchmark dataset modelnet10, it works well.<br>
I guess the reason is that the Rsconv designed for large dataset?</p>
Converting from pytorch.tensor() to numpy array is too slow!

username: soaxeus
cooked: <p>I’m converting <code>pytorch.tensor()</code> object to <code>numpy array</code> like the below code.</p>
<p><code>tensor_data.cpu().detach().numpy()</code></p>
<p>But it takes approximately 0.33 seconds. Is it normal?</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is most likely the <code>tensor_data.cpu()</code> that is slow? Not the <code>.detach().numpy()</code> right?</p>

username: soaxeus
cooked: <p>Yes <code>tensor_data.cpu()</code> is slowing the operation. Do you know how I can convert numpy array without converting <code>cpu()</code> ?</p>

username: ptrblck
cooked: <p>If your data is on the <code>GPU</code>, you would have to transfer the data to the RAM first via <code>.cpu()</code> and call <code>numpy()</code> on it.<br>
Note that (as <a class="mention" href="/u/alband">@albanD</a> mentioned) the <code>numpy()</code> call should be really cheap, as the underlying data will be shared and no copy will be involved.</p>
<p>Since CUDA operations are asynchronous, the <code>.cpu()</code> call will create a synchronization point, so that all currently executed and queued ops on the GPU have to finish, before the tensor will be pushed to the host.<br>
The data transfer might be indeed not taking the majority of the time, but other CUDA operations in the background (e.g. your model’s forward/backward pass).</p>

username: Hao_Hao_Tan
cooked: <p>So does this means that if <code>.cpu()</code> is waiting super long for synchronization, it basically says that my GPU is not strong enough to compute all the results I need to transfer to CPU? Am I understanding it correctly?</p>

username: ptrblck
cooked: <p>If you are not synchronizing the code, the next sync point will just accumulate the runtime.</p>
<aside class="quote no-group" data-username="Hao_Hao_Tan" data-post="5" data-topic="66095">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hao_hao_tan/48/28338_2.png" class="avatar"> Hao_Hao_Tan:</div>
<blockquote>
<p>it basically says that my GPU is not strong enough to compute all the results I need to transfer to CPU?</p>
</blockquote>
</aside>
<p>No, the result will be calculated and pushed to the CPU once it’s finished. Since the CUDA operation is executed asynchronously, the Python script executes the next line of code right after launching the CUDA kernel. Since the calculation on the GPU will take “some” time, the next line of code would wait, if it’s a sync point.</p>
Size mismatch in MADDPG

username: dani_kal
cooked: <p>Hello everyone!<br>
My question seems common but I have not found the solution yet…<br>
I have implemented MADDPG algorithm and I receive this error message</p>
<blockquote>
<p>RuntimeError: size mismatch, m1: [20 x 20], m2: [40 x 64] at …\aten\src\TH/generic/THTensorMath.cpp:41</p>
</blockquote>
<p>This is the code</p>
<blockquote>
<p>class Actor(nn.Module):</p>
<pre><code>def __init__(self, state_size, action_size, seed, fc1_units=400, fc2_units=300, use_bn=False):
    
    super(Actor, self).__init__()
    self.seed = torch.manual_seed(seed)
    self.use_bn = use_bn
    
    self.fc1 = nn.Linear(state_size, fc1_units)
    self.fc2 = nn.Linear(fc1_units, fc2_units)            
    self.fc3 = nn.Linear(fc2_units, action_size)
    if self.use_bn:
        self.bn1 = nn.BatchNorm1d(state_size)
        self.bn2 = nn.BatchNorm1d(fc1_units)
        self.bn3 = nn.BatchNorm1d(fc2_units)

    self.reset_parameters()
</code></pre>
</blockquote>
<blockquote>
<p>def forward(self, state):</p>
<pre><code>    print('state shape',state.shape)
    
    if self.use_bn:
        x = self.fc1(self.bn1(state))
    else:
        x = self.fc1(state)

    x = F.relu(x)
    if self.use_bn:
        x = self.bn2(x)
    x = self.fc2(x)
    x = F.relu(x)
    if self.use_bn:
        x = self.bn3(x)
    return F.relu(self.fc3(x))
</code></pre>
</blockquote>
<blockquote>
<p>class Critic(nn.Module):</p>
<pre><code>def __init__(self, state_size, action_size, seed, fc1_units=400, fc2_units=300, use_bn=False):
   
    super(Critic, self).__init__()
    self.seed = torch.manual_seed(seed)
    self.use_bn = use_bn
    self.fc1 = nn.Linear(state_size, fc1_units)
    self.fc2 = nn.Linear(fc1_units + action_size, fc2_units)
    self.fc3 = nn.Linear(fc2_units, 1)
    if self.use_bn:
        self.bn1 = nn.BatchNorm1d(fc1_units)
        self.bn2 = nn.BatchNorm1d(fc2_units)
</code></pre>
<p>def forward(self, state, action):<br>
x  = self.fc1(state)<br>
xs = F.relu(x)<br>
if self.use_bn:<br>
x = self.bn1(x)<br>
x = torch.cat((xs, action), dim=1)<br>
x = self.fc2(x)<br>
x = F.relu(x)<br>
if self.use_bn:<br>
x = self.bn2(x)<br>
return self.fc3(x)</p>
</blockquote>
<p>The error is when I call this</p>
<blockquote>
<p>Q = self.critic(states, actions)</p>
</blockquote>
<p>states = 10<br>
actions = 10</p>
<p>Can someone help me!!!</p>

username: ptrblck
cooked: <p>Are you calling the <code>Critic</code> model with two integers in this line of code?</p>
<pre><code class="lang-python">states = 10
actions = 10
Q = self.critic(states, actions)
</code></pre>
<p>If so, make sure you are passing tensors to the model in the shape <code>[batch_size, in_features]</code>.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: dani_kal
cooked: <p>ok thank you I will try it!!!</p>
Expecting a face similarity loss function

username: weisunding
cooked: <p>Hi gurus</p>
<p>I am finding a face similarity loss function which accepts two tensors as input, any idea?</p>
<p>My goal is to find a loss function which can do backward, as SSIM is very good for replacement of L2, but it detects the global structure, I want to compare the face only.</p>
<p>The expected usage</p>
<pre><code class="lang-auto">loss = 1 - face_similarity(tensor1, tensor2)
loss.backward()
</code></pre>
<p>I know there is a facenet-pytorch project, MTCNN + ResNet(vggface2) combo which can compare the faces, but it breaks the gradients.</p>
<p>Any clue will be appreciated!</p>
Masking/Threshold Tensor

username: ganLover
cooked: <p>I have a tensor(A) of the Shape-&gt;  [1,3,13,13]<br>
Another tensor(B) of the shape-&gt; [3]<br>
And a Threshold Value-&gt; 0.5</p>
<p>I want to iterate over the -&gt; X dimension of -&gt;[1,X, 13, 13]<br>
Eg. B= [0.1, 0.3, 0.6]</p>
<p>For each index of B[index], If its value &gt; threshold make that [1,index, 13, 13] -&gt; ZERO.</p>
<p>I tried with-&gt; A(dim1, dim2&gt; threshold, dim3, dim4) = 0<br>
But it gives error-&gt; [Index out of Bounds]</p>
<p>Any other way…</p>

username: ptrblck
cooked: <p>This code should work if I understand your issue correctly:</p>
<pre><code class="lang-python">A = torch.randn(1, 3, 13, 13)
B = torch.randn(3)
thresh = 0.0
A[:, B &gt; thresh] = 0
</code></pre>

username: saba
cooked: <p>Sorry, i need to do thresholding in the case that be usable for backward.() <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I need to do the inverse of the case that defined in nn.Threshold<br>
would you please help me with that. I write some code the thresholding is correct but not usable for backward.()</p>

username: ptrblck
cooked: <p>The posted operation would be similar to a relu (which zeroes out the values above a threshold), wouldn’t it? What issues are you seeing in the backward?</p>

username: saba
cooked: <p>It works now. But how I can understand the generated gradient is correct. For example my result is not the thing that I expected but I am nt sure is teh problem of gradient or totally I should change the loss function. I used this code</p>
<pre><code class="lang-auto">      
        zzz=torch.zeros(bbb.shape)
        zzz1=torch.zeros(bbb1.shape)
        
        for ii in range(bbb.shape[0]):
             for ii1 in range(bbb.shape[1]):
                  for ii2 in range(bbb.shape[2]):
                            if bbb[ii,ii1,ii2]&gt;=0.98:
                                zzz[ii,ii1,ii2]=1
                            elif  bbb[ii,ii1,ii2]&lt;0.98:
                                zzz[ii,ii1,ii2]=0
                
        for ii in range(bbb1.shape[0]):
            for ii1 in range(bbb1.shape[1]):
                for ii2 in range(bbb1.shape[2]):
                        if bbb1[ii,ii1,ii2]&gt;=0.98:
                           zzz1[ii,ii1,ii2]=1
                        elif  bbb1[ii,ii1,ii2]&lt;0.98:
                            zzz1[ii,ii1,ii2]=0

        L1=criterion2(zzz,zzz1)
        loss2=errG1+1*L1
        loss2.backward()

</code></pre>
<p>It create gradient but I don’t know is it true or not <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
How to load and save multiple schedulers?

username: RJaikanth
cooked: <p>Hello,</p>
<p>I am currently working on a project for computer vision to implement papers. I want to generalize the training script so that we can use <strong>multiple schedulers</strong> if we want.</p>
<p><strong>I had 2 questions -</strong></p>
<ul>
<li>Is it absolutely necessary to save schedulers?</li>
<li>How to save/load multiple schedulers?</li>
</ul>
<p>The workflow goes something like this -</p>
<h4>train.yml</h4>
<pre><code class="lang-auto">schedulers:
  # Scheduler1
  - name: "reduce_lr"
    mode: "min"
    factor: 0.1
    patience: 5
    verbose: True
  # Scheduler2
  # ..
  # Scheduler n 
</code></pre>
<h4>Loading scheduler(s) initially</h4>
<pre><code class="lang-auto">def get_scheduler(optimizer, scheduler_params):
  scheduler_params["name"] in SCHEDULER_DICT:
  scheduler = scheduler_params["name"]
  del scheduler_params["name"]
  logging.info("Using '{}' scheduler for this run.".format(scheduler.upper()))

  scheduler = SCHEDULER_DICT[scheduler](optimizer=optimizer, **scheduler_params)
  return scheduler

sch_dict = {}  # scheduler(s)
    for scheduler_params in config.schedulers:
        sch_name = scheduler_params["name"]
        sch_dict[sch_name] = get_scheduler(optimizer=optimizer, scheduler_params=scheduler_params)
</code></pre>
<h4>Calling schedulers</h4>
<pre><code class="lang-auto">for name, scheduler in sch_dict.items():
                if name == "reduce_lr":
                    scheduler.step(val_metrics["loss"])
                else:
                    scheduler.step()
</code></pre>
Share weights within the model in dynamic way

username: zeeel_patel
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba9589127930dfc58629bb1483f29f359095c99b.png" data-download-href="https://discuss.pytorch.org/uploads/default/ba9589127930dfc58629bb1483f29f359095c99b" title="ADAIN"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba9589127930dfc58629bb1483f29f359095c99b_2_690x413.png" alt="ADAIN" data-base62-sha1="qCBbDwG4IXcQ6zXFI1WjuzvyGXF" width="690" height="413" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba9589127930dfc58629bb1483f29f359095c99b_2_690x413.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba9589127930dfc58629bb1483f29f359095c99b_2_1035x619.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba9589127930dfc58629bb1483f29f359095c99b.png 2x" data-dominant-color="EFEFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ADAIN</span><span class="informations">1069×640 81.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I’m trying to deploy the following model in PyTorch.</p>
<p>Consider the right part of the network. I have data related to each station from 1 to K where K is dynamic. layers on the right should share weights for all stations.</p>
<p>Can someone tell me how can I share weights for this kind of application?</p>
PyTorch DataLoader error- AttributeError: 'DataLoader' object has no attribute 'generator'

username: almog
cooked: <p>Just updated to PyTorch 1.6 and got this error while looping over a dataloader object:<br>
AttributeError: ‘DataLoader’ object has no attribute ‘generator’</p>
<p>The loop works on PyTorch 1.5</p>
<p>Any ideas?</p>

username: ptrblck
cooked: <p>The <code>generator</code> attribute was added ~3months ago in <a href="https://github.com/pytorch/pytorch/pull/39737">this PR</a>. Could you post the complete stack trace with a minimal code snippet, which would reproduce this issue?</p>
Concat dataloaders for a unified interleaving dataloader?

username: dvirginz
cooked: <p>I Try to train on two dataloaders, one attached to a dataset where each <code>__get_item__</code> call fetches a predefined batch of varying length (thus the batch_size I transfer to the <code>dataloader</code> object is 1), and one where I sample randomly from a set of sequences, thus <code>__get_item__</code> call fetches one sample each time.</p>
<p>I’m looking for something like</p>
<pre><code class="lang-auto">            loader = DataLoader(
                batched_dataset,
                batch_size=1,
            )
            loader_tdm = DataLoader(
                random_samples_dataset,
                batch_size=8,
            )
            from data.concat_dataloaders import AlternateIterator
            loader = AlternateDataloader(loader,loader_tdm)
</code></pre>
<p>Is this possible?</p>

username: ptrblck
cooked: <p>You could create the iterators directly via:</p>
<pre><code class="lang-python">iter0 = iter(loader)
iter1 = iter(loader_tdm)
</code></pre>
<p>and either use some utility functions from <code>itertools</code> or manually alternate between these iterators.<br>
Note that you would have to manually take care or recreating the iterators if they run out of samples.</p>
How can l clear the old cache in GPU, when training different groups of data continuously?

username: tcexeexe
cooked: <p>I need to train the image classifiers in different groups. When one group is trained, the script will automatically load and train the next group by “for” statements.<br>
But an error occurred saying there was not enough memory in GPU when it start to train the next group. How can I clear the GPU memory used by the last group training before the script start train the next group?<br>
l have try to use torch.cuda.empty_cache() after each group training finished but it doesn’t work.</p>
<pre><code class="lang-auto">    time.sleep(5)
    del model
    del loss
    gc.collect()
    torch.cuda.empty_cache()#清空GPU缓存
    # print(torch.cuda.memory_stats(0))
    time.sleep(10)
    torch.cuda.empty_cache()#清空GPU缓存
    # print(torch.cuda.memory_stats(0))
</code></pre>
<p>Pytorch’s version is 1.6.0<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b94ad2e444c53dd5cb1ad62fe8334543856d612.png" data-download-href="https://discuss.pytorch.org/uploads/default/8b94ad2e444c53dd5cb1ad62fe8334543856d612" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b94ad2e444c53dd5cb1ad62fe8334543856d612_2_690x390.png" alt="image" data-base62-sha1="jUMUbO0ch7piBwgn6DJh9yXlSjo" width="690" height="390" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b94ad2e444c53dd5cb1ad62fe8334543856d612_2_690x390.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b94ad2e444c53dd5cb1ad62fe8334543856d612_2_1035x585.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b94ad2e444c53dd5cb1ad62fe8334543856d612_2_1380x780.png 2x" data-dominant-color="221E1E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1884×1065 1.04 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>If you delete all references to the model and other tensors, the memory can be freed or reused. <a href="https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762/4">Here</a> is a small example.<br>
Make sure you are not storing the model output, loss etc. without detaching it, as this would keep the computation graph with all intermediate tensors alive.</p>
Unexpected NaN from TransformerEncoder forward function

username: rwang11
cooked: <p>Hi,</p>
<p>I’m getting unexpected NaN at certain cells of the output tensor, could somebody help me understand why? Especially given that I have the padding_mask specified to avoid the nan cells, why does that still impact my output? Also is input padding cells’ value supposed to be NaN, or what’s the best practice?</p>
<p>Example Input:</p>
<blockquote>
<p>in_embed = torch.tensor(<br>
[[[ 0.0000,  1.1115],<br>
[ 0.0015,  1.1115],<br>
[ 0.0015,  1.1115]],<br>
[[ 0.9552,  0.0000],<br>
[float(‘nan’),float(‘nan’)],<br>
[float(‘nan’),float(‘nan’)]],<br>
[[ 1.0119, -0.4620],<br>
[float(‘nan’),float(‘nan’)],<br>
[float(‘nan’),float(‘nan’)]],<br>
[[ 0.0463, -1.2204],<br>
[float(‘nan’),float(‘nan’)],<br>
[float(‘nan’),float(‘nan’)]]]<br>
)<br>
mask = torch.tensor(<br>
[[0., -float(‘inf’), -float(‘inf’), -float(‘inf’)],<br>
[0., 0., -float(‘inf’), -float(‘inf’)],<br>
[0., 0., 0., -float(‘inf’)],<br>
[0., 0., 0., 0.]]<br>
)<br>
seqmasks = torch.tensor(<br>
[[False, False, False, False],<br>
[False,  True,  True,  True],<br>
[False,  True,  True,  True]]<br>
)</p>
</blockquote>
<p>Calling TransformerEncoder:</p>
<blockquote>
<p>tel = torch.nn.TransformerEncoderLayer(d_model=2, nhead=2, dim_feedforward=8)<br>
bert = torch.nn.TransformerEncoder(tel, num_layers=6)<br>
out_embed = bert(src=in_embed, mask=mask, src_key_padding_mask=seqmasks)</p>
</blockquote>
<p>Real Output:</p>
<blockquote>
<p>tensor([[[-1.0000,  1.0000],<br>
[    nan,     nan],<br>
[    nan,     nan]],<br>
[[ 1.0000, -1.0000],<br>
[    nan,     nan],<br>
[    nan,     nan]],<br>
[[ 1.0000, -1.0000],<br>
[    nan,     nan],<br>
[    nan,     nan]],<br>
[[ 1.0000, -1.0000],<br>
[    nan,     nan],<br>
[    nan,     nan]]], grad_fn=)</p>
</blockquote>
<p>Expected Output (something like this):</p>
<blockquote>
<p>tensor([[[-1.0000,  1.0000],<br>
[-1.0000,  1.0000],<br>
[-1.0000,  1.0000]],<br>
[[ 1.0000, -1.0000],<br>
[    nan,     nan],<br>
[    nan,     nan]],<br>
[[ 1.0000, -1.0000],<br>
[    nan,     nan],<br>
[    nan,     nan]],<br>
[[ 1.0000, -1.0000],<br>
[    nan,     nan],<br>
[    nan,     nan]]], grad_fn=)</p>
</blockquote>
Curriculum learning with CNNs

username: bjr96571
cooked: <p>I’m currently working on a project and using on the methodology described in this paper: <a href="https://arxiv.org/pdf/1808.02212.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1808.02212.pdf</a></p>
<p>Essentially, the target labels are hierarchical (binary --&gt; 7 categories --&gt; 25 categories), and as I understand it, the authors first train a model on the binary, and then fine tune that model on the next level of labels (and then the third).</p>
<p>I’m struggling a bit on understanding the correct implementation. I’ve fine-tuned a resnet model on the binary labels and got a fairly good accuracy, and I took that same model, changed the final classifier layer, and began training on the second level of labels – but the accuracy is terrible.</p>
<p>Has anyone implemented a model in a similar fashion or know if this approach is correct?</p>
Backward step in the generator regularization is much slower in Libtorch than in pytorch

username: Carol_Ye_Liu
cooked: <p>I have this path length penalty loss calculated in the function g_path_regularize. And the generator regularization step looks like this:</p>
<pre><code>        std::vector&lt;torch::Tensor&gt; res = generator-&gt;forward(noise, true);
        fake_img = res[0];
        latents = res[1];
   
        weighted_path_loss = args.parameter * path_loss;
        generator-&gt;zero_grad();

        at::cuda::CUDAStream stream = at::cuda::getCurrentCUDAStream();
        AT_CUDA_CHECK(cudaStreamSynchronize(stream));
        std::clock_t start;
        double duration;
        start = std::clock();

        weighted_path_loss.backward();
        stream = at::cuda::getCurrentCUDAStream();
        AT_CUDA_CHECK(cudaStreamSynchronize(stream));

        duration = ( std::clock() - start ) / (double) CLOCKS_PER_SEC;
        std::cout&lt;&lt;"Time spent backward: "&lt;&lt; duration &lt;&lt;'\n';
</code></pre>
<p>From the output window I saw “Time spent backward: 294.73”, which means it took around 5 minutes to finish one regularization step. But in python using pytorch, the regularization step finished in less than 10 seconds. I wonder what could be the bottleneck. Is there any way to debug the backward step? Any input would be greatly appreciated!</p>
<pre><code>std::vector&lt;torch::Tensor&gt; g_path_regularize(const torch::Tensor&amp; fake_img, const torch::Tensor&amp; 
latents,  const torch::Tensor&amp; mean_path_length, Scalar decay=0.01){
std::vector&lt;torch::Tensor&gt; res;
auto noise = torch::randn_like(fake_img) / sqrt(
        fake_img.size(2) * fake_img.size(3)
);

auto grad = torch::autograd::grad(
        {(fake_img * noise).sum()}, {latents}, {}, true, true)[0];

auto path_lengths = torch::sqrt(torch::mean(torch::sum(torch::mul(grad, grad), 2),1));

auto path_mean = mean_path_length + decay * (path_lengths.mean() - mean_path_length);//tensor

auto path_penalty = torch::mean(torch::mul(path_lengths - path_mean, path_lengths - path_mean) );

res={path_penalty, path_mean.detach(), path_lengths};
return res;
}</code></pre>
How to sort tensor by given order?

username: fan_percy
cooked: <p>Hi,</p>
<p>I’m looking for a function to sort the value of a 2D tensor by a given order.</p>
<p>For example, A is a 2D tensor ([1, 0, 2, 2, 1], [0, 2, 1, 2, 0]), and I want to sort A following an order tensor B as ([3, 2, 4, 1, 0], [2, 1, 4, 3, 3]). So the sorted A should be ([2, 2, 1, 0, 1], [1, 2, 0, 2, 2]). There can be repeating elements in the same row of B and each row of A is sorted by the order in corresponding row of B.</p>
<p>Is there any function I can directly apply to get the sorted A?</p>
<p>Any suggestion is appreciated!</p>

username: beaupreda
cooked: <p>Hello,</p>
<p>As far as I know, there is no such function to get a 2D tensor sorted from another 2D tensor. However, with a simple <code>for</code> loop and indexing, it is possible to achieve the desired result:</p>
<pre><code class="lang-auto">    a = torch.tensor([[1, 0, 2, 2, 1], [0, 2, 1, 2, 0]], dtype=torch.int32)
    # index tensor (b) must be of type long, byte or bool
    b = torch.tensor([[3, 2, 4, 1, 0], [2, 1, 4, 3, 3]], dtype=torch.long)
    for i in range(b.shape[0]):
        # change the order of the ith tensor of a with the indexes of b[i]
        c = a[i][b[i]]
        print(c)
    # tensor([2, 2, 1, 0, 1], dtype=torch.int32)
    # tensor([1, 2, 0, 2, 2], dtype=torch.int32)
</code></pre>
<p>Don’t know if this helps!</p>

username: agadetsky
cooked: <p>Tensorized version can be implemented as follows:</p>
<pre><code class="lang-auto">def smart_sort(x, permutation):
    d1, d2 = x.size()
    ret = x[
        torch.arange(d1).unsqueeze(1).repeat((1, d2)).flatten(),
        permutation.flatten()
    ].view(d1, d2)
    return ret
</code></pre>
<p>And it sorts as you want:</p>
<pre><code class="lang-auto">a = torch.tensor([[1, 0, 2, 2, 1], [0, 2, 1, 2, 0]])
b = torch.tensor([[3, 2, 4, 1, 0], [2, 1, 4, 3, 3]])
smart_sort(a, b)

output: tensor([[2, 2, 1, 0, 1],
               [1, 2, 0, 2, 2]])
</code></pre>

username: fan_percy
cooked: <p>Thanks for the help!</p>

username: fan_percy
cooked: <p>Thank you for your advice!</p>

username: Vinaypatil
cooked: <p>you can try this,also</p>
<pre><code class="lang-auto">def sort_tensor(X, dim=-1):
  d1, d2 = X.size()
  ten, ind=torch.sort(X, dim=dim)
  if dim == -1:
    return X[torch.arange(d1).unsqueeze(1).repeat((1, d2)), ind]
  elif dim == 0:
    return X[ind, torch.arange(d2)]
  else:
    raise IndexError("wrong dim parameter")
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">t = torch.tensor([[23, 45, 66, 100], 
                [90, 56, 8,132],
                [34, 90, 88, 200],
                [250, 15, 90, 5]])

sort_tensor(t, dim=-1)
output:
tensor([[ 23,  45,  66, 100],
        [  8,  56,  90, 132],
        [ 34,  88,  90, 200],
        [  5,  15,  90, 250]])

sort_tensor(t, dim=-0)
output:
tensor([[ 23,  15,   8,   5],
        [ 34,  45,  66, 100],
        [ 90,  56,  88, 132],
        [250,  90,  90, 200]])
                  
</code></pre>
Weighing cross entropy loss

username: BaruchG
cooked: <p>I am trying to assign different weights to tensors in my batch when computing cross entropy loss.  The issue I am having is that these weights are not based on labels so I can’t seem to give them to <code>nn.CrossEntropyLoss()</code> as outlined at <a href="https://pytorch.org/docs/stable/nn.functional.html#cross-entropy" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.functional.html#cross-entropy</a>.  For an example of what I am trying to do inside each batch:</p>
<pre><code class="lang-auto">criterion = CrossEntropyLoss_modified()
fake_outputs = torch.rand((2,2)) #single batch
labels = torch.randint(0,2,(2,))
weights = torch.tensor([.7,.3]) #these values will be changed for each batch and each item in the batch will have a weight assigned to it
criterion(fake_outputs, labels, weights) # should output a weighted loss
</code></pre>
<p>The length of the weights will be the length of the batch, not the length.<br>
Is this doable?  Is there a better way to go about this?</p>

username: ayalaa2
cooked: <p>I would set the reduction to none: <code>CrossEntropyLoss(reduction='none')</code></p>
<p>This returns you a <code>num_batch</code> size tensor. So you can then do something like this:<br>
<code>(weights * criterion(output, labels)).mean()</code></p>

username: BaruchG
cooked: <p>Perfect, just what I was looking for.  Thanks!</p>
Reshaping 1D vector in a 4D tensor without allocating huge amounts of GPU memory

username: Anej_Sterle
cooked: <p>Hello guys. I have following parameters from a class I have written:</p>
<pre><code class="lang-auto">self.m = 4
self.Pnum = 2*self.m -1 
self.Nl = 1000
</code></pre>
<p>In the object’s method i’m doing some linear algebra. I’m doing this on the GPU. I need to make a 4D matrix out of my vector z, which is of length Pnum.</p>
<pre><code class="lang-auto">z = torch.linspace(0, 2*np.pi, steps = self.Pnum)
</code></pre>
<p>I transform it into a 4D matrix with a specific shape with:</p>
<pre><code class="lang-auto">z_tile = z.repeat( self.Nl * self.m, self.m).reshape( (self.Nl, self.m, self.m, self.Pnum))
</code></pre>
<p>This operation seems to be taking a lot of memory, if I have bigger values of self.Nl ( which is an input ). Is there a way to rewrite this z_tile tensor with views and expand functions  in a way which saves some GPU memory?</p>

username: JuanFMontesinos
cooked: <p>Hi,<br>
rapeat makes new copies of the tensor each time. This is expected (depending on the behaviour you expect for your gradients)</p>
<p>You can alternatively use expand<br>
<a href="https://pytorch.org/docs/stable/tensors.html?highlight=expand_as#torch.Tensor.expand" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/tensors.html?highlight=expand_as#torch.Tensor.expand</a><br>
Please read the docs in order to be aware of the differences.</p>
<p>From docs:</p>
<blockquote>
<p>More than one element of an expanded tensor may refer to a single memory location. As a result, in-place operations (especially ones that are vectorized) may result in incorrect behavior. If you need to write to the tensors, please clone them first.</p>
</blockquote>