akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Gpu memory cost

username: as754770178
cooked: <pre><code class="lang-auto">      out = self.conv1(x)
      out = self.norm1(out)
      out = self.relu(out)

      out = self.conv2(out)
      out = self.norm2(out)
      out = self.relu(out)

      out = self.conv3(out)
      out = self.norm3(out)
</code></pre>
<ol>
<li>
<code>training</code> of  conv and norm is false</li>
<li>track_running_stats of norm is True</li>
</ol>
<p>When I use pycharm to debug the code, from <code>conv1</code> to <code>conv3</code>, the gpu memory not increase. But I exec <code>norm3</code>, the gpu memory increase.<br>
If <code>training</code> of  conv and norm is True, I test that I exec every sentence, the gpu memory increase.</p>
<p>When I exec every sentence， that create new out, why the gpu memory not increase?<br>
But why I exec norm3 the gpu memory increase?</p>

username: as754770178
cooked: <p>I see the requires_grad of weight and bias in norm is False.</p>
<p>The code in <code>mmdet/models/backbones/resnet.py</code> of  <a href="https://github.com/open-mmlab/mmdetection" rel="nofollow noopener">mmdetection</a>.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Keep in mind that the GPU api is asynchronous. So you might want to add a <code>torch.cuda.syncrhonize()</code> after the line to make sure it finished executing.</p>
<p>Also since, you reuse the <code>out</code> variable, the old Tensor that out was pointing to is not reachable anymore and can be deleted when you’re not training (when training, it needs to be kept around to be able to compute the backward).</p>
How to watch pytorch code

username: as754770178
cooked: <p>I use pycharm write my code.</p>
<p>When I want to watch the pytorch source code, only comment in python file, such as</p>
<pre><code class="lang-auto">    def asin(self, input, out=None): # real signature unknown; restored from __doc__
        """
        asin(input, out=None) -&gt; Tensor
        
        Returns a new tensor with the arcsine  of the elements of :attr:`input`.
        
        .. math::
            \text{out}_{i} = \sin^{-1}(\text{input}_{i})
        
        Args:
            input (Tensor): the input tensor
            out (Tensor, optional): the output tensor
        
        Example::
        
            &gt;&gt;&gt; a = torch.randn(4)
            &gt;&gt;&gt; a
            tensor([-0.5962,  1.4985, -0.4396,  1.4525])
            &gt;&gt;&gt; torch.asin(a)
            tensor([-0.6387,     nan, -0.4552,     nan])
        """
        pass

    def asin_(self, *args, **kwargs): # real signature unknown
        pass

    def as_strided(self, *args, **kwargs): # real signature unknown
        pass

    def as_strided_(self, *args, **kwargs): # real signature unknown
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Most of these functions are implemented in cpp I’m afraid. So you won’t be able to find the source code in python. We just use these to easily store the documentation and signature in a way that doc generator and mypy understands.</p>

username: as754770178
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="83170">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Most of these functions are implemented in cpp I’m afraid. So you won’t be able to find the source code in python. We just use these to easily store the documentation and signature in a way that doc generator and mypy understands.</p>
</blockquote>
</aside>
<p>Thanks, I want to know how to generate these documentation. I want to have some insight into pytorch</p>

username: albanD
cooked: <p>All the docs are generated from here: <a href="https://github.com/pytorch/pytorch/tree/master/docs">https://github.com/pytorch/pytorch/tree/master/docs</a></p>
Siamese network data loader

username: dendi
cooked: <p>I am trying to implement a Siamese network dedicated to detecting similarities between sentences.<br>
The input to the dataset is three lists (anchor, positive, and negative).<br>
My custom dataset is implemented in the following way:</p>
<pre><code class="lang-python">max_seq_length = 16

class DescriptionDataset(data.Dataset):
    def __init__(self,positive_list, negative_list, anchor_list):
        self.negative_list = negative_list
        self.positive_list = positive_list
        self.anchor_list = anchor_list
        
    def __getitem__(self,index):
        return (self.to_tokens(self.anchor_list[index]),
                self.to_tokens(self.positive_list[index]),
                self.to_tokens(self.negative_list[index])
               )
    
    
    def _to_tokens(description):
        tokenized_description = tokenizer.tokenize(description)
        
        if len(tokenized_description) &gt; max_seq_length:
            tokenized_description = tokenized_description[:max_seq_length]
            
        ids_description  = tokenizer.convert_tokens_to_ids(tokenized_description)
        padding = [0] * (max_seq_length - len(ids_description))
        ids_description += padding
        ids_description = torch.tensor(ids_description)
    
    def __len__(self):
        return len(self.anchor_list)
</code></pre>
<p>My network is implemented in the following way:</p>
<pre><code class="lang-auto">   def __init__(self, encoder, encoder_dim, n_hidden, use_gpu=True):
        super(SiameseNetwork, self).__init__()
        self.use_gpu=use_gpu
        self.encoder = encoder
        self.n_hidden = n_hidden
        self.fc = nn.Linear(encoder_dim, hidden_dim,bias=False)
   
   def forward_embedding(self, x):
        x = self.encoder(x)
        self.hidden = cache
        output = self.fc(output)
        return output
    
   def forward(self, x, y, z):
        embedded_x = self.forward_embedding(x)
        embedded_y = self.forward_embedding(y)
        embedded_z = self.forward_embedding(z)
        dist_a = F.pairwise_distance(embedded_x, embedded_y, 2)
        dist_b = F.pairwise_distance(embedded_x, embedded_z, 2)
        return dist_a, dist_b, embedded_x, embedded_y, embedded_z
</code></pre>
<p>As an encoder, I am going to use the pre-trained BERT model.<br>
The problem is that I have not idea how to implement custom DataLoader, in particular how to overload the default <em>collate_fn</em>  function.<br>
Can you help me?</p>
CPU utilization too high

username: happyprotean
cooked: <p>Hi, I set num_workers=8 in dataloader, but  I use top to check the CPU usage and found that only one CPU worker’s utilization is high(500%), the others is quite low(20%).<br>
How can I balance the CPU usage?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/10e89d5e8884be730bfc6d14217067f3fb65e683.png" data-download-href="https://discuss.pytorch.org/uploads/default/10e89d5e8884be730bfc6d14217067f3fb65e683" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/0/10e89d5e8884be730bfc6d14217067f3fb65e683.png" alt="image" data-base62-sha1="2pA0SRXh6ylcpe4W07hmOPlLBNp" width="690" height="199" data-dominant-color="DFDDDD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">723×209 11.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Transfer learning error

username: chalapathinagavarma
cooked: <pre><code class="lang-auto">
from torchvision import models
model = models.resnet18(pretrained = True)
mean = [0.485,0.456,0.406]
std = [0.229,0.224,0.225]
for parms in model.parameters():
    parms.requires_grad = False
model.fc = torch.nn.Linear(512,8)
optimzer = torch.optim.Adam([parms for parmas in model.parameters() if (parms.requires_grad == True)],lr = 0.05 )
</code></pre>
<h2>
<strong>This code is giving me error like this</strong>  I don’t understand where is mistake.</h2>
<pre><code class="lang-auto">ValueError                                Traceback (most recent call last)
&lt;ipython-input-18-c701a2bba695&gt; in &lt;module&gt;
      6     parms.requires_grad = False
      7 model.fc = torch.nn.Linear(512,8)
----&gt; 8 optimzer = torch.optim.Adam([parms for parmas in model.parameters() if (parms.requires_grad == True)],lr = 0.05 )

C:\Miniconda\envs\py37_pytorch\lib\site-packages\torch\optim\adam.py in __init__(self, params, lr, betas, eps, weight_decay, amsgrad)
     42         defaults = dict(lr=lr, betas=betas, eps=eps,
     43                         weight_decay=weight_decay, amsgrad=amsgrad)
---&gt; 44         super(Adam, self).__init__(params, defaults)
     45 
     46     def __setstate__(self, state):

C:\Miniconda\envs\py37_pytorch\lib\site-packages\torch\optim\optimizer.py in __init__(self, params, defaults)
     44         param_groups = list(params)
     45         if len(param_groups) == 0:
---&gt; 46             raise ValueError("optimizer got an empty parameter list")
     47         if not isinstance(param_groups[0], dict):
     48             param_groups = [{'params': param_groups}]

ValueError: optimizer got an empty parameter list
</code></pre>

username: Nikronic
cooked: <p>Hi,</p>
<p>This might be funny but you have typo in your code.</p>
<aside class="quote no-group" data-username="chalapathinagavarma" data-post="1" data-topic="83248">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chalapathinagavarma/48/24660_2.png" class="avatar"> chalapathinagavarma:</div>
<blockquote>
<p><code>parms for parmas in model.parameters() if (parms.requires_grad == True)</code></p>
</blockquote>
</aside>
<p><code>parms</code> is from loop of original model that you turned its grad off. <code>parmas</code> is the variable you defined in loop within <code>optim.Adam</code> method to get params with required grads.</p>
<p>Bests</p>

username: chalapathinagavarma
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=9" title=":man_facepalming:" class="emoji" alt=":man_facepalming:"><img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=9" title=":man_facepalming:" class="emoji" alt=":man_facepalming:"><img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=9" title=":man_facepalming:" class="emoji" alt=":man_facepalming:">.<br>
Thank you very much</p>
Computing multiple sums over columns efficiently in GPU

username: mineiro
cooked: <p>Let’s say I have a 2d tensor, named x. I have also a python list (a) which is of the form a = [a_1, a_2, …, a_n]. a_i is a list that contains column indexes of the tensor x. I want to compute the sum of the columns of x for each a_i as efficiently as possible in the GPU. For this reason, I do:</p>
<p>new_tensor = torch.stack([torch.sum(x[:, i], 1) for i in a], 1).</p>
<p>However, even though tensor x is loaded to GPU already, the process takes way too long and the GPU utilization during execution drops to 0%.</p>
<p>Any idea how to make it run faster on GPU?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Would it be possible to create an index tensor of <code>a</code> and index <code>x</code> only once instead of in a loop?<br>
Could you print the shapes and values for some dummy inputs of <code>x</code> and <code>a</code>?</p>

username: mineiro
cooked: <p>x could be x = random(5,5) (5 x 5 random tensor). Also a = [[0,1], [1,2,3], [4]] (In general entries of a have different lengths).</p>
<p>I don’t follow on the “only once” part you mention above.</p>
Restore dataloader within an epoch

username: Paul_Koch
cooked: <p>Dear Community,</p>
<p>I would like to restore my Dataloader similar to the model/scheduler/optimizer with the “state_dict”. Given a special circumstance I may want to pause my training inside a epoch I would like to restore my training right where it stopped. However, the Dataloader has no state_dict unlike other pytorch classes. I use “islice” from the “itertools” library to continue my training from a given step in the epoch. Unfortunately I can not find any way to set the iter list inside the sampler of the Dataloader. This is not an issue if my data is iterated sequentially. Thou for shuffled iteration this is not the case. Inside the “RandomSampler” function “_<em>iter</em>_(self)” the index list is set like this: iter(torch.randperm(n).tolist()) [1]. If this list would become part of the sampler I would be able to get and set it. My question is whether there is any way which I could not find to restore the iter index list of the sampler? Else you might want to consider and discuss my proposal to make the “torch.randperm(n).tolist()” list a part of the sampler self.</p>
<p>Best,<br>
Paul</p>
<p>[1] <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/sampler.html#Sampler" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/utils/data/sampler.html#Sampler</a></p>

username: Paul_Koch
cooked: <p>If I set a seed with (torch.manual_seed(n)) before I iter my dataloader in every epoch I have ensured that my randomness is the same if i restore the training at a given epoch step. After every epoch i generate a new seed and then I just have to save that seed when I stop my training within an epoch.</p>
TypeError: 'DataLoader' object does not support indexing

username: Surbhi_Khushu
cooked: <p>I am implementing cycle Gans  .<br>
My dataloader is written as : class MyDataset(Dataset) :<br>
def <strong>init</strong>(self , patch_CT, patch_PET):<br>
‘characterizes a dataset for pytorch’<br>
self.patch_CT = patch_CT<br>
self.patch_PET= patch_PET<br>
self.transforms = transforms.ToTensor</p>
<pre><code> def __len__(self):
    'denotes the total number of samples'
    return  len(self.patch_CT)
    print(len(self.patch_PET))

 def __getitem__(self,index):
    'Generates one sample of data'
    #select sample
    x = self.patch_CT[index]

    y = self.patch_PET[index]

    # Unsqueeze channel dimension
    #x = x.unsqueeze(0)
    #y = y.unsqueeze(0)
    return x,y
</code></pre>
<p>train_A_dataset, test_A_dataset = torch.utils.data.random_split(<br>
patch_CT ,[int(0.7<em>len(patch_CT)), int(len(patch_CT) - int(0.7</em>len(patch_CT)))])</p>
<p>train_B_dataset, test_B_dataset = torch.utils.data.random_split(<br>
patch_PET ,[int(0.7<em>len(patch_PET)), int(len(patch_PET) - int(0.7</em>len(patch_PET)))])</p>
<p>and for dataloader:<br>
a_loader = torch.utils.data.DataLoader(train_A_dataset, batch_size=5, shuffle=False<br>
b_loader = torch.utils.data.DataLoader(train_B_dataset, batch_size=5, shuffle=False)<br>
for i,batch in enumerate((a_loader, b_loader)):</p>
<pre><code>            # step
            step = epoch * min(len(a_loader), len(b_loader)) + i + 1

            # Generator Computations
            ##################################################

            set_grad([self.Da, self.Db], False)
            self.g_optimizer.zero_grad()

            a_real = Variable(batch[0])
            b_real = Variable(batch[0])
            a_real, b_real = utils.cuda([a_real, b_real])
</code></pre>
<p>The error I get is<br>
a_real = Variable(batch[0])<br>
TypeError: ‘DataLoader’ object does not support indexing</p>
<p>How can I solve this?<br>
Thank you in advance.</p>

username: Pranavan_Theivendira
cooked: <aside class="quote no-group" data-username="Surbhi_Khushu" data-post="1" data-topic="83149">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/8c91f0/48.png" class="avatar"> Surbhi_Khushu:</div>
<blockquote>
<p>DataLoader</p>
</blockquote>
</aside>
<p>Hi,</p>
<p>PyTorch Dataloaders are accessed as follows.</p>
<pre><code class="lang-auto">for index, data enumerate(a_loader)
</code></pre>
<p>They do not support indexing.</p>
<p>Thanks<br>
Regards<br>
Pranavan</p>

username: Surbhi_Khushu
cooked: <p>Thank you, it worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Implmentations of the Laplacian regularization loss

username: yl-precious
cooked: <h3>Referrence implementaion:  tfl.lattice_layer.LaplacianRegularizer</h3>
<p>I am trying to reimplemente the <strong>laplacian regularization loss</strong> using pytorch. But found the code might be wrong.</p>
<pre><code class="lang-auto">    def calc_laplacian_regularizer_loss(self, weights, lattice_sizes, l1=0.0, l2=0.0):
        # weight:  B C H W
        if not l1 and not l2:
            return 0.0
        B, _, H, W = weights.shape

        weights = weights.view(B, lattice_sizes[0], lattice_sizes[1], H, W)

        diff1 = weights[:, 1:, :, :, :] - weights[:, :-1, :, :, :]
        diff2 = weights[:, :, 1:, :, :] - weights[:, :, :-1, :, :]
        if l1:
            resutl1 = torch.abs(diff1).sum()
            resutl1 += torch.abs(diff2).sum()
        if l2:
            resutl2 = torch.pow(diff1, 2).sum()
            resutl2 += torch.pow(diff2, 2).sum()

        if l1 and not l2:
            return resutl1
        elif not l1 and l2:
            return resutl2
        else:
            return resutl1 + resutl2
</code></pre>
<ol>
<li>if  therer are one tensor( <strong>[4, 12, 256, 256]</strong>), How to comput the laplacian regularization loss over their <strong>six-connected neighbors ?</strong>
</li>
<li>Waiting for your reply!!!</li>
<li>Waiting for your reply!!!</li>
<li>Waiting for your reply!!!</li>
</ol>
<h1><a href="https://github.com/tensorflow/lattice/blob/v2.0.4/tensorflow_lattice/python/lattice_layer.py#L930-L1003" rel="nofollow noopener"><strong>Tensorflow implementation is here</strong></a></h1>
How can i change the computation graph of a saved model?

username: Nick_Young
cooked: <p>only given a saved model file, after i load it, can i modify the computation graph of the model? insert new operators between layers (down to op level)? or it highly relies on how the model code is written?</p>

username: Kushaj
cooked: <p>Saved model file only contains the weights of a neural network and code for only the base class of the model. All the other methods it calls are not included in the save. Also, there are some things that are not pickled like lambdas (I think) which if the model includes will not be in the save. So you need model code, if you want to do anything meaningful.</p>

username: Nick_Young
cooked: <p>Thank you for the reply! <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>
Super().__init__(x, y) vs super(Model, self).__init__()

username: sahiluppal2k
cooked: <p>I’m new to pytorch and i have few questions.</p>
<p>while defining a model, we have to provide “super().<strong>init</strong>()” inside constructor. then what is difference between</p>
<ol>
<li>super(Model, self).<strong>init</strong>()</li>
<li>super().<strong>init</strong>(x, y)</li>
</ol>

username: ptrblck
cooked: <p>If I’m not mistaken, you had to use the first approach in Python2.x, while both approaches should work in Python3.x (not sure about the exact minor versions).</p>
<p>For backward-compatibility reasons, you will probably see a lot of the first approach (and I’m also used to it), which wouldn’t be necessary anymore, as Python2 is dead by now. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Autoencoder: how to add sparsity

username: MngFrc
cooked: <p>Hi everyone!</p>
<p>I am deploying a sparse autoencoder.</p>
<p>Briefly, an autoencoder is a feedforward NN that is formed by a series of layers of decreasing dimension (the encoder), followed by a series of layers of increasing dimension (the decoder). The loss is computed by the MSE between NN’s input and decoder’s output. At the end of the encoder, instead, there is the encoded input.</p>
<p>I want to limit the number of activated neurons at the encoder’s ouput layer, in order to “simplify” the encoded data. But I am not sure that the steps I have followed are the right ones.<br>
I thought to count the number of times in which each encoder’s output neuron was activated over the entire training dataset, compare such values with the desidered ones (an hyperparameter) and so the sparsity loss.<br>
Then start fitting the model batch by batch, epoch by epoch, adding each time the sparsity loss to the training one, before the backpropagation step.<br>
The sparsity loss is updated after each epoch.</p>
<p>Following this procedure, I have not obtained much satisfactory results. It seems that the sparsity term has not effects on the training. So before start looking for code bugs, I would like to have a confirm or a denial about the followed high-level procedure.</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>How did you calculate the sparsity loss?<br>
Based on your description it seems as if you’ve implemented a “counting” approach, which might not be differentiable.<br>
To verify it, you could check, if the sparsity loss has a valid <code>.grad_fn</code>.<br>
If that’s not the case, your approach is currently just adding an offset to the loss, which won’t have an effect on the training.</p>
<p>In case your sparsity loss doesn’t have a <code>.grad_fn</code>, you could maybe try to add an L1 penalty to the parameters of the last layer of your encoder, which <em>might</em> make them sparse.</p>
Moving tensors between devices

username: Andreas_Georgiou
cooked: <p>Let’s say we have a simple model like this:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        self.conv1 = nn.Conv2d(3,32, 3)
        self.param1 = nn.Parameter(torch.rand(3,3))

    def set_devices(self):
        self.conv1 = self.conv1.to('cuda:0')
        self.param1 = self.param1.to('cuda:1')
        return self
</code></pre>
<p>Then after the creation of the model I call <code>model.set_devices()</code>. Is PyTorch going to accumulate the gradients in the new moved parameters <code>.grad</code>? If so, why? .to will create a new tensor that is no longer a leaf tensor from what I understand, is that correct? Is the behaviour different for nn.Parameter and nn.Module? What about <code>model = model.set_devices()</code>? Does that yield different results?</p>
<p>Thanks in advance for your help</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="Andreas_Georgiou" data-post="1" data-topic="83261">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andreas_georgiou/48/11871_2.png" class="avatar"> Andreas_Georgiou:</div>
<blockquote>
<p>If so, why? .to will create a new tensor that is no longer a leaf tensor from what I understand, is that correct?</p>
</blockquote>
</aside>
<p>Yes</p>
<aside class="quote no-group" data-username="Andreas_Georgiou" data-post="1" data-topic="83261">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andreas_georgiou/48/11871_2.png" class="avatar"> Andreas_Georgiou:</div>
<blockquote>
<p>Is the behaviour different for nn.Parameter and nn.Module?</p>
</blockquote>
</aside>
<p>Yes, nn.Module does some peculiar things to manage nn.Parameters and their movement. You may check source code for details.</p>
<aside class="quote no-group" data-username="Andreas_Georgiou" data-post="1" data-topic="83261">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andreas_georgiou/48/11871_2.png" class="avatar"> Andreas_Georgiou:</div>
<blockquote>
<p>What about <code>model = model.set_devices()</code> ? Does that yield different results?</p>
</blockquote>
</aside>
<p>Nope, as model is never cloned in your code.</p>
<p>PS in general, <code>self.param.data = self.param.data.to(device)</code> does the trick</p>

username: Andreas_Georgiou
cooked: <p>Hmm, but then why model.to() creates a copy? Where in the code does this happen? Looking at the nn.Module the <code>_apply()</code> method, should just move the internal parameters and modules in-place.</p>

username: Andreas_Georgiou
cooked: <p>I see, it does do it in place. I guess it returns <code>self</code> just for convenience so that methods can be chained. But calling model.to() without <code>model=model.to()</code> still works</p>
PyTorch fails on me when using local Jupyter to run epochs

username: Onur_Ozbek
cooked: <p>Hi.</p>
<p>I am trying to run a classifier on my local notebook to use my own GPU instead of relying on Colab.  When I try to run a single epoch to find the learning rate, I get this many repeats of this message:</p>
<pre><code class="lang-auto">/home/onur/.local/lib/python3.6/site-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details. 
</code></pre>
<p>I get that its just a warning but my epoch does not run at all.</p>

username: SANTOSH_S
cooked: <p>Are you able to train when you set  <code>recompute_scale_factor=True</code>?</p>

username: Onur_Ozbek
cooked: <p>Where do I set that and what is its significance?</p>

username: Onur_Ozbek
cooked: <aside class="quote no-group" data-username="SANTOSH_S" data-post="2" data-topic="83161">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/santosh_s/48/24798_2.png" class="avatar"> SANTOSH_S:</div>
<blockquote>
<p><code>recompute_scale_factor=True</code> ?</p>
</blockquote>
</aside>
<p>It doesn’t solve the problem</p>
Save tensor append mode

username: Xiaoyu_Song
cooked: <p>Hi, How to save several tensor appending using <code>torch.save()</code>?<br>
for example:</p>
<pre><code class="lang-auto">for i in range(20):
       ......
    loss = criterion(scores, labels) 
    torch.save(loss,'loss.pt')
</code></pre>
<p>How to save these all 20 losses?</p>

username: DanyalTajik
cooked: <pre><code class="lang-auto">for i in range(20):
       ......
    loss = criterion(scores, labels) 
    torch.save(loss,'loss'+i+'.pt')
</code></pre>
<p>this is what you are talking about? each file will be saved with a different name. i understood your question like this. or you want to append all the losses into a single tensor and then save it?<br>
with <code>torch.save(Tensor)</code> you can save tensors</p>

username: Xiaoyu_Song
cooked: <p>Hi, Thank you for your reply. I would like to save them in a list or tensor whatever for plotting afterwards, what are your suggestion?</p>

username: DanyalTajik
cooked: <p>okay, you can do something like:</p>
<pre><code class="lang-auto">Losses=[]
for i in range(20):
       ......
    loss = criterion(scores, labels) 
    Losses.append(loss.item())
StackedLossTensor=torch.stack(Losses)
torch.save(StackedLossTensor,'loss.pt')
</code></pre>
<p>i think this will work for you.</p>

username: Rachel_Tan
cooked: <p>is there a way where i can append them only upon saving. i dont have enough memory on my cpu so i was hoping i could save them iteratively through the data</p>

username: ptrblck
cooked: <p><code>loss.item()</code> should be a <code>float32</code> scalar value. How large is the expected <code>StackedLossTensor</code>, that you are running out of memory?</p>
<p>The posted code snippet assumed that the loss is stored for each batch during training.<br>
Let me know, if I misunderstood your question. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Difference between F.embedding and indexing

username: omer_sahban
cooked: <pre><code class="lang-auto">w = torch.rand(10000, 256)
x = torch.tensor([0, 1, 2])

y1 = F.embedding(x, w)
y2 = w[x]

print((y1-y2).abs().sum().item())
# Out: 0
</code></pre>
<p>it seems like F.embedding and numpy style indexing can be used interchangeably, but F.embedding is about 20% faster. is there any other differences between these two methods? for example their behavior during backward pass</p>

username: omer_sahban
cooked: <p>apparently the weight parameter of F.embedding has to be a 2d tensor, otherwise it fails to backprop.</p>
<pre><code class="lang-auto">w = torch.rand(10000, 10, 256, requires_grad=True)
o = torch.rand(256, 10000, requires_grad=True)

x = torch.tensor([0, 1, 2])
y = F.embedding(x, w)
logits = y.mean(dim=1) @ o
loss = F.cross_entropy(logits, x)
loss.backward()
# RuntimeError: shape '[3, 256]' is invalid for input of size 7680
</code></pre>

username: omer_sahban
cooked: <p>I’ve also discovered that multiplying embedding matrix with onehot vectors is much faster than F.embedding, but creating onehot vectors with F.onehot is exteremly slow. I created a “whiteboard” vector filled with zeros, then scattered the indices onto it to create onehot vectors, this is much faster than creating a new tensor at each training step.</p>
<pre><code class="lang-auto">batch_size=128
seq_len =100
vocab_size=10000
w = torch.rand(vocab_size, 256, requires_grad=True)
whiteboard = torch.zeros([batch_size, seq_len, vocab_size])

for i in range(100):
  x = torch.randint(vocab_size, size=[batch_size, seq_len])
  onehot = whiteboard.scatter(2, x[:,:,None], 1)
  # onehot = F.onehot(x, vocab_size) # very slow
  
  y = onehot @ w
  # y = F.embedding(x, w)

  # ...
</code></pre>
Need help in model selection

username: jetjodh
cooked: <p>I am training mulitple pretrained models from torchvision on my own dataset. The performance of two models have caught my attention. The first model has a normally decreasing training loss and a increasing accuracy but its validation loss decreases and then keeps on decreasing but giving a bit better validation accuracy than the second model, which has a decreasing training loss and a increasing accuracy(albeit much lower than first model’s) but has a decreasing validation loss and improving validation accuracy(but still lower than first models).<br>
How do I go over choosing which model to work with further? Do I keep training for more epochs?</p>

username: ptrblck
cooked: <p>Usually you would select the model yielding the “best” performance.<br>
However, it seems you are interested in further fine-tuning or optimizing the model, so it’s a bit hard to tell, which model you should select.<br>
Note, that the second model might learn a bit slower due to non-optimal hyperparameters, so you might want to experiment with changing these.</p>
<p>You could train the model as long as the training and validation losses decrease by a significant amount.</p>
Python doubt in mean

username: julioeu99
cooked: <pre><code class="lang-auto">a=[]
for ne, naa in zip(ne, naa): 
   de+=distance.euclidean(naa,ne)
a.append(de/size_batch)
print(*a, sep = ", ")
</code></pre>
<p>I wanted to put all data inside the array, and print whats inside it.<br>
I think the array is overwriting, and its stuck on the first position. How can i store all values in this array?</p>
<p>I tried this to:</p>
<pre><code class="lang-auto">v=0
a.insert(v,de/size_batch)
++v
</code></pre>
<p>And i get the same result.<br>
The output is just a value and not all the values.<br>
Can someone help me?</p>

username: ptrblck
cooked: <p>The approach should work.<br>
Could you check the size of <code>ne</code> and <code>naa</code>?<br>
<code>zip</code> will just use the smaller of both lengths. If one of these tensors only contains a single sample, you would only use a single iteration.</p>

username: ptrblck
cooked: <p>Ah sorry, I missed the errors.</p>
<p>You are currently overwriting <code>ne</code> and <code>naa</code> in the loop, so use other variable names for the single elements and also move the <code>a.append</code> call inside the loop:</p>
<pre><code class="lang-python">for ne_, naa_ in zip(ne, naa): 
   de+=distance.euclidean(naa_,ne_)
   a.append(de/size_batch)
</code></pre>

username: julioeu99
cooked: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Embedding on CUDA error

username: attardi
cooked: <p>I wrote a parser using Embedding that works fine on CPU.<br>
I have added .to(torch.float64), .to(device(‘cuda’) in the appropriate places, to get to run on GPU, in particular after the creation of an Embedding.<br>
However, when extracting the Embedding for a sequence of indices, with pdb3, I get:</p>
<p>.    exec(cmd, globals, locals)<br>
File “”, line 1, in <br>
File “/project/tools/lhrparser/examples/train.py”, line 18, in <br>
import sys<br>
File “/project/tools/lhrparser/examples/train.py”, line 88, in main<br>
trainer.train(training_set=training_sentences, validation_set=test_sentences, model_path=model_path)<br>
File “/project/tools/lhrparser/src/traininghelper.py”, line 254, in train<br>
loss = self.train_sentence(sentence=sentence)<br>
File “/project/tools/lhrparser/src/traininghelper.py”, line 203, in train_sentence<br>
concatenated_embedding = self.parser.encode_embeddings(sentence=sentence)<br>
File “/project/tools/lhrparser/src/lhrparser.py”, line 41, in encode_embeddings<br>
pos_embedding = self.model.pos_embeddings(pos_sequence)<br>
File “/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/project/tools/lhrparser/src/embeddings.py”, line 45, in forward<br>
embeds = self.embeddings(sequence)<br>
File “/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python3.5/dist-packages/torch/nn/modules/sparse.py”, line 108, in forward<br>
self.norm_type, self.scale_grad_by_freq, self.sparse)<br>
File “/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py”, line 1076, in embedding<br>
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)<br>
RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.LongTensor for argument <span class="hashtag-raw">#3</span> ‘index’<br>
Uncaught exception. Entering post mortem debugging<br>
Running ‘cont’ or ‘step’ will restart the program</p>
<blockquote>
<p>/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py(1076)embedding()<br>
 → return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)<br>
(Pdb) weight.dtype<br>
torch.float64<br>
(Pdb) input<br>
tensor([  3,  12,  13,  11,  11,   1,  14,   0,  15,   3,  12,  14,<br>
0,   9,  16,   2,  12,   6,  15,  12,  12,  11,   0,  14,<br>
11,  16,  12,  13])<br>
(Pdb) weight.shape<br>
torch.Size([17, 50])<br>
(Pdb) padding_idx<br>
-1<br>
(Pdb) scale_grad_by_freq<br>
False<br>
(Pdb) sparse<br>
False</p>
</blockquote>
<p>Any help appreciated.</p>

username: zywu
cooked: <p>Hey, have you soloved this problem?<br>
Can you shade some light on it?</p>

username: attardi
cooked: <p>It was due to the fact that the argument was not moved to cuda.<br>
Thanks for your help…</p>
Making a custom class input compatible with DataParallel

username: Ali2500
cooked: <p>From what I understand, <code>torch.nn.DataParallel</code> works by splitting the input(s) to a module along the batch dimension and feeding the split inputs to copies of the model in each GPU. This imposes a somewhat annoying restriction that the input to my model cannot be, for example, a custom class, because then <code>DataParallel</code> just creates a shallow copy that can’t be distributed among multiple GPUs.</p>
<p>My question is: can I design a custom class in such a way that when I provide it as input to a torch module with <code>DataParallel</code> wrapped around it, the input is properly split. It would be ideal if I could just write a class function like <code>def split(self, num_gpus)</code> that implements this splitting, and then <code>DataParallel</code> internally internally just call it.</p>

username: justanhduc
cooked: <p>Hello. Have you figured out the solution?</p>

username: justanhduc
cooked: <p>For anyone coming here, I solved this by writing a custom scatter (for custom input) for <code>torch.nn.parallel.DataParallel</code>.</p>
Initialize weights as a constant array?

username: byersin
cooked: <p>I’m interested in using the Glow stand-alone bundles for microcontroller applications but I ran into the following memory issue. When declaring the constant weight array</p>
<pre><code class="lang-auto">uint8_t constantWeight[CONSTANT_MEM_SIZE] = {
#include "MODEL_NAME.weights.txt"
};
</code></pre>
<p>the array is not defined as constant. You cannot simply add a <code>const</code> keyword because the model entry function expects a non-constant format. This is a problem because non-constant variables are stored in the (too small) RAM segment instead of the flash. Is there a simple way to work around this problem?</p>

username: srohit0
cooked: <p>Check out  <a href="http://bit.ly/deep-C" rel="nofollow noopener">bit.ly/deep-C </a> for an easy approach to put pytorch &amp; onnx models into microcontroller applications</p>

username: jfix
cooked: <p>I’m not sure I understand the issue. Why do you need to mark it as <code>const</code>? Your microcontroller application requires the weights to be <code>const</code>? Can you not just create a constant reference to use in APIs other than the model entry function?</p>
Cyclic Learning Rate Max LR

username: Kumar_Rajamani
cooked: <p>Can the Max_LR value for CycleLR be less than the lr that is used for optim.Adam ?<br>
I used the values below (lr 0.002 and max_lr 0.0005 (I did that by mistake) and that seemed to be giving better results than Max_LR of 0.005 )<br>
Max_LR of 0.005 did not seem to give better result</p>
<p>optimizer = optim.Adam(list(net.parameters()),lr=0.002)</p>
<p>scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.0005, steps_per_epoch=len(dataloaders[‘train’]), epochs=total_epocs_var)</p>

username: ptrblck
cooked: <p>Yes, you could set the <code>max_lr</code> to a smaller value than the original learning rate, which would create the new limit as seen here:</p>
<pre><code class="lang-python">net = nn.Linear(1, 1)
optimizer = torch.optim.Adam(list(net.parameters()),lr=0.002)
scheduler = torch.optim.lr_scheduler.OneCycleLR(
    optimizer, max_lr=0.0005, steps_per_epoch=2, epochs=10)

for epoch in range(20):
    optimizer.step()
    scheduler.step()
    print(scheduler.get_last_lr())

&gt; [6.58359213500126e-05]
[0.00018583592135001264]
[0.00033416407864998737]
[0.00045416407864998736]
[0.0005]
[0.0004937320031175437]
[0.0004752423160067368]
[0.00044545808878552497]
[0.0004058728269748815]
[0.0003584715008956504]
[0.0003056310109681446]
[0.00025000099999999997]
[0.00019437098903185538]
[0.0001415304991043496]
[9.412917302511849e-05]
[5.454391121447504e-05]
[2.4759683993263143e-05]
[6.269996882456277e-06]
[2e-09]
[6.269996882456277e-06]
</code></pre>

username: Kumar_Rajamani
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your kind reply<br>
I will experiment with the code block that you have shared and see the difference if lr was also small like 0.0005 vesus a large lr of 0.002<br>
I guess that lr might start with larger value and then settle into the max value set by max_lr</p>
<p>What could be the advantages of such a setting ? (or any draw backs too of such setting)</p>

username: ptrblck
cooked: <p>One drawback could be a slightly confusing code, as the optimizer defines another learning rate than the scheduler is allowing. Besides that the scheduler will drive the learning rate, so I don’t think there are other issues.</p>
Data Agumentation. How to agument a single image into 30 images at time

username: chalapathinagavarma
cooked: <p>Hello everyone.<br>
I have dought that how to implement a data augmentation for the image that should be augmented to 30 images with custom data set. can anyone please provide me any reference or code</p>

username: SANTOSH_S
cooked: <p>Hi <a class="mention" href="/u/chalapathinagavarma">@chalapathinagavarma</a> can you please elaborate your question?<br>
Do you have any confusion with how to create Dataloaders?</p>

username: henry_Kang
cooked: <p>Hello. If you are the first time,<br>
I recommend to use the albumentation.<br>
Which is very easy to use for data augmentation in the computer vision.<br>
For example</p>
<pre><code class="lang-auto">import cv2
import numpy as np
from albumentations import (CropNonEmptyMaskIfExists,HorizontalFlip,VerticalFlip, RandomBrightness,RandomContrast,MotionBlur,RandomCrop,RandomRotate90)

def rr(img,mask):
    aug=RandomRotate90(p=1)
    augmented = aug(image=img, mask=mask)
    img=augmented['image']
    mask=augmented['mask']
    return(img,mask)

#This is is data loader section. train_imgr is img, target_imgr is ground truth mask
train_img2,target_img2=rr(train_imgr,target_imgr) #random brightness
self.train_data.append(train_img2)   # this is train data list
self.train_labels.append(target_img2) # this is ground truth list
</code></pre>
Combinations from 2 tensors

username: IGero228
cooked: <p>Have two tensors like:</p>
<pre><code class="lang-auto">tensor([ 5, 27]
tensor([ 1, 20,3].
</code></pre>
<p>I need  to get this tensor:<br>
<code>tensor([[ 5, 1], [5,20], [5,3], [ 27, 1], [27,20], [27,3]]</code></p>
<p>How i can do it without loop!?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>This will do the trick and as it is the builtin function, it is the most efficient way!:</p>
<pre><code class="lang-auto">x = torch.tensor([5, 27])
y = torch.tensor([1, 20, 3])

torch.cartesian_prod(x, y)
</code></pre>
<p>For more info, visit <a href="https://pytorch.org/docs/master/generated/torch.cartesian_prod.html#torch-cartesian-prod" rel="nofollow noopener">official docs</a>.</p>
<p>Bests</p>
How to add dropout to a squeezenet?

username: alx
cooked: <p>Here’s what I normally do:</p>
<pre><code class="lang-auto">    model = models.inception_v3(pretrained=True)
    num_ftrs = model.fc.in_features
    model.fc = nn.Sequential(
      nn.Dropout(0.5),
      nn.Linear(num_ftrs, 5))
</code></pre>
<p>However this does not work at all with the squeezenet, it does not have an in_features</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>Yes, because squeezenet does not use Linear in its definition so there is no need to define the structure like ResNet or Inception. But you can get the structure of model using <code>print(model)</code> and extract layers that you want.<br>
Based on your example code, I think you looking for dropping final classifier and train your own linear classifier for it.</p>
<p>Original classfier of squeezenet:</p>
<pre><code class="lang-auto">(classifier): Sequential(
    (0): Dropout(p=0.5, inplace=False)
    (1): Conv2d(512, 1000, kernel_size=(1, 1), stride=(1, 1))
    (2): ReLU(inplace=True)
    (3): AdaptiveAvgPool2d(output_size=(1, 1))
  )
</code></pre>
<p>To change it the way you want, this snippet may help, although you can modify it in anyway you like:</p>
<pre><code class="lang-auto">x = torch.randn(1, 3, 512, 512)

model = torchvision.models.squeezenet.squeezenet1_0(pretrained=True)
model.classifier = nn.Sequential(
    nn.AdaptiveAvgPool2d(output_size=(1, 1)),
    nn.Dropout(0.5),
    nn.Flatten(),
    nn.Linear(in_features=512, out_features=5),
    )
model(x)
</code></pre>
<p>Bests</p>

username: alx
cooked: <p>Works great thank you!</p>
Libtorch sequential missing an #include?

username: J_S
cooked: <p>OS: Ubuntu 18.04<br>
libtorch: 1.4</p>
<p>I’ve set up libtorch as described on the site and been using it for a few weeks without problems. Now trying to use Sequential and as soon as I add the following line I can’t build:</p>
<p><span class="hashtag">#include</span> “torch/nn/modules/container/sequential.h”</p>
<p>error:<br>
libcpp/libtorch/include/torch/csrc/api/include/torch/nn/modules/container/sequential.h:118:49: error: ‘NamedAnyModule’ was not declared in this scope<br>
explicit SequentialImpl(std::initializer_list named_modules) {</p>
<p>The error message is pretty clear and it’s obvious what to do… but I’m new to c++ and libtorch so I seriously doubts that I found a problem in libtorch itself and I keep thinking I must be doing something wrong. Yet, I haven’t run in any other problem with libtorch and if I go in sequential.h and do the following modifications then I can build:</p>
<ol>
<li>add <span class="hashtag">#include</span> “torch/nn/modules/container/named_any.h”</li>
<li>add torch::nn:: in front of every NamedAnyModule</li>
</ol>
<p>I looked at sequential.h in libtorch 1.5 and it looks the same.</p>
<p>Does anyone else have the same problem?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The safe bet for you here would be to <code>#include "torch/torch.h"</code> to avoid any further issue</p>
<p>But 1) sounds like the right fix. Can you send a PR adding this include to sequential.h?</p>

username: J_S
cooked: <p>Thanks and yes, I will send a PR.</p>
Expected integer literal for index

username: WaterKnight
cooked: <p>I opened this issue becasue torch.jit.script is not working with hrnet</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/39165" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/39165" target="_blank" rel="nofollow noopener">[JIT] Expected integer literal for index:</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-05-28" data-time="16:21:54" data-timezone="UTC">04:21PM - 28 May 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/WaterKnight1998" target="_blank" rel="nofollow noopener">
          <img alt="WaterKnight1998" src="https://avatars1.githubusercontent.com/u/41203448?v=4" class="onebox-avatar-inline" width="20" height="20">
          WaterKnight1998
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
I have tried to convert this model to TorchScript using torch.jit.script. However, I am getting this issue:
RuntimeError: 
Expected integer literal...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">jit</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>