akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Multiply two tensors with different dimensions

Lame: <p>Hi all,</p>
<p>How to multiply two tensors with A ~&gt; torch.Size([4]) and B ~&gt; torch.Size([3, 1])</p>
<p>Any help will be appreciated.</p>

ptrblck: <p>How should the multiplication take place, i.e. which value of <code>A</code> should be multiplied with which value(s) of <code>B</code>? An elementwise multiplication isn’t possible, as both tensors have a different number of elements so I guess you would like to apply some kind of matrix multiplication?</p>
PyTorch1.7 seems to consume more memory?

feiyuhuahuo: <p>I was training a model on two 11G memory GPUs, and the memory consumption is about 10.3 G per GPU observed by <code>nvidia-smi</code>. After I upgrade PyTorch to 1.7, this project could not be run any more and encountered <code>CUDA out of memory</code>. So does PyTorch1.7 need more memory? Besides, my CUDA version is 10.2.</p>

ptrblck: <p>Are you seeing this for a variety of models or a specific model in particular?</p>

feiyuhuahuo: <p>That’s strange. I tested on another two GPUs now and everything is OK. Since the two GPUs which I meet the problem is training now with PyTorch 1.4. After the training is over, I’ll test whether the problem occurs by accident.</p>

Julien_Maille: <p>I noticed too that with 1.7 I can run twice less threads than I used to to with 1.6<br>
I’m curious about any explanation for that. Could the default precision on Cuda 11 be float64 or something like that?</p>

ptrblck: <p>What kind of threads do you mean?</p>
<aside class="quote no-group" data-username="Julien_Maille" data-post="4" data-topic="101674">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/julien_maille/48/18680_2.png" class="avatar"> Julien_Maille:</div>
<blockquote>
<p>Could the default precision on Cuda 11 be float64 or something like that?</p>
</blockquote>
</aside>
<p>No, that’s not the case.</p>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation for a loss function

g33sean: <p>Hey, so I have this custom loss where I have a generator, called Gen, some latent codes that are z (which is also learned), and the input data x.  I wrote this loss custom loss function, below and I am getting an in-place operation error. I used suggestions for the traceback, and it is coming from the loss.backward(). I am aware you are typically not supposed to use in place operations on tensors that require gradients. I believe it coming from the fact that z requires a grad, but there is an in-place operation on it with the dummy variable. Any help is appreciated</p>
<pre><code class="lang-python">class CustomLoss(nn.Module):
    def __init__(self):
        super(CustomLoss, self).__init__()
    def forward(self,Gen,z,x,device):
        loss_list=[]
        z_tun=torch.zeros(1,z.size(1)).to(device)
        for i in range(z.size(0)):
            z_tun
            for j in range(z.size(1)):
                z=z.data.clone()
                z_dummy=z[i,:]
                z_tun[:,j]=z_dummy[j]
                loss=torch.norm(Gen(z_tun)-x,p=2)
                loss_list.append(loss)
        total_loss=sum(loss_list)
        total_loss/=z.numel()
        return total_loss
</code></pre>

ptrblck: <p>Could you remove the <code>.data</code> usage and check, if you are still getting the error?<br>
Using <code>.data</code> is not recommended and might yield unwanted side effects.</p>

g33sean: <p>Hey thanks for the response, no the error still exist. I am having output 0 of Viewbackward. I am not sure how to fix that. I believe I need to clone z correctly.</p>

ptrblck: <p>Check, if the inplace division of <code>total_loss</code> might cause this issue by using the out-of-place operation.<br>
If not, use <code>.clone()</code> to narrow down the particular line of code.</p>
When calculate loss in model forward with multi-gpu training then get a tuple loss

Alva-2020: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab7ead71cb8d610a61c7d0f4575c228c30e05f5b.png" data-download-href="https://discuss.pytorch.org/uploads/default/ab7ead71cb8d610a61c7d0f4575c228c30e05f5b" title="捕获"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab7ead71cb8d610a61c7d0f4575c228c30e05f5b.png" alt="捕获" data-base62-sha1="ot73gS8FZyFj1rAOQ1ICpq3Y9N9" width="690" height="183" data-dominant-color="151414"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">捕获</span><span class="informations">1655×441 27.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Hi everyone, when I use F.nn_loss() in model forward as above. Then I two GPUs to train the model in form of model = torch.nn.DataParallel(model).cuda(). I get a tuple  speed_loss. Before I called loss.backward, I use torch.sum(speed_loss) to get a scalar. Is this right? The tuple is from two gpus calculation. I add them together then go backward directly.</p>

ptrblck: <p>The output of <code>nn.DataParallel</code> should be a single tensor on the default device and thus the target as well.<br>
I’m not sure, if you are using a custom data parallel approach, but if you are calculating the loss based on the model output and target, you should get a tensor (also on the default device) and not a <code>tuple</code>.</p>

Alva-2020: <p>Thanks very much, I fixed it.</p>

gangqiang_hu: <p>I got the same problem,  getting multiple loss. How do you solve that? Thanks a lot!</p>

gangqiang_hu: <p>OK…I fixed that. If you compute and return the loss inside the model, then you will get a loss list. Just average the loss list.</p>
Plot magnitude of gradient in SDM with training step

Vish2020: <p>I’m using CNN2D for text sentiment analysis with the following settings:</p>
<blockquote>
<p>optimizer = optim.Adam(model_pyt.parameters())</p>
<p>criterion = nn.BCEWithLogitsLoss()</p>
</blockquote>
<p>I’m currently training various neural networks and I’m trying to ascertain how the gradient is behaving at each training step and thus would like to plot the gradient magnitude vs time step to observe if the gradient is vanishing or exploding.  Is there a way to do this?  Also is there a way to plot the learning rate per training step so we can observe how that’s changing, as it should get smaller as we reach a stationary point on the convex loss function?</p>

ptrblck: <p>You can get all gradients by iterating the parameters after the <code>backward</code> operation:</p>
<pre><code class="lang-python">for name, param in model.named_parameters():
    print(name, param.grad)
</code></pre>
<p>Using this approach you can store the gradient magnitude and plot it for each iteration using e.g. <code>matplotlib</code>.</p>
<p>To get the learning rate and other internal states you can use <code>optimizer.state_dict()</code> or <code>optimizer.param_groups[id]</code> and check the values you would like to track.</p>

Vish2020: <p>Yes this worked thank you, one comment though is that from param.grad we need to take the clone param.grad.clone() in order to capture the gradient at each time step t that we can plot.</p>
<p>Thx!</p>
Converting a torch::Tensor to a std:vector

iiSeymour: <p>I’m writing a shim to use a C++ function with PyTorch and would like to know how to convert <code>torch::Tensor</code> to  a <code>std:vector&lt;int32_t&gt;</code>?</p>
<pre><code class="lang-auto">float f_cpp(std::vector&lt;int32_t&gt;&amp; result);

float f(torch::Tensor results_t) {

  //  std:vector&lt;int32_t&gt; &lt;-- torch::Tensor?
  std::vector&lt;int32_t&gt; result = result_t.data_ptr&lt;std::vector&lt;int32_t&gt;&gt;; 

  return f_cpp(*result);
}

</code></pre>

tom: <p>The problem here is that a std::vector can’t use “foreign” memory.<br>
Some avenues that might work:</p>
<ul>
<li>Use a pointer (<code>int32_t* </code>) as an array or a <code>ArrayRef&lt;int32_t&gt;</code> (available in c10). In this case you need to keep the tensor allocated while you are using them. Also note that you need to be a bit careful with strides if your tensor can be non-contiguous.</li>
<li>Allocate the memory in the vector and then use from_blob to get a tensor. In this case you need to keep the vector around while using the tensor.</li>
<li>Copy the data.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

iiSeymour: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="101921">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>from_blob</p>
</blockquote>
</aside>
<p>Hey Thomas,</p>
<p>Thanks for replying, I’m not a C++ developer so I need a little bit more clarification.</p>
<p>I’m only dealing with 1d tensors so if I go with <code>int32_t*</code> am I on the right track with -</p>
<pre><code class="lang-auto">auto r_ptr = result_t.data_ptr&lt;int32_t&gt;();
std::vector&lt;int32_t&gt; result{r_ptr, r_ptr + result_t.?};
</code></pre>
<p>If so, how do I get the size of the <code>torch::Tensor result_t</code>?</p>
<p>Regards,</p>

tom: <p>I’d recommend result_t.size(0)</p>
Learning rate Decay combined with differential Learning Rate

Karan_Chhabra: <p>Hi,<br>
I am using differential learning rates for different layers. But at the same time I am also using LR decay (I might change to OneCycle policy). But I wanted to know for which layers is the LR decay applied, below is the sample code:</p>
<p><code>optimizer = optim.SGD([ {'params': model.base.parameters()}, {'params': model.classifier.parameters(), 'lr': 1e-3} ], lr=1e-2, momentum=0.9)</code></p>
<p><code>MultiStepLR(optimizer, milestones=[5,10], gamma=0.1)</code></p>
<p>Is the rate decay only applied to base layers or to both the classifier and base layers?</p>

tom: <p>They loop over the parameter groups, i.e. the classifier lr will always be a tenth of the base in your example.</p>
<p>Best regards</p>
<p>Thomas</p>

Karan_Chhabra: <p>Is the a way to make the classifier learning rate be constant while the base learning only decay’s?</p>

tom: <p>Yes, stick the parameters in two optimizers. That should not make much of a difference w.r.t. performance, you just need to zero_grad and step twice.</p>
Pytorch: How to create a random int tensor where a certain percent are of a certain value? For example, 25% are 1s, and rest 0s

Santosh-Gupta: <p>In pytorch I can create a random zero and one tensor with around %50 distribution of each</p>
<pre><code class="lang-auto">import torch 
torch.randint(low=0, high=2, size=(2, 5))
</code></pre>
<p>I am wondering how I can make a tensor where only 25% of the values are 1s, and the rest are zeros?</p>

tom: <p>I typically use either<br>
<code>(torch.rand((2, 5)) &lt; 0.25).float()</code><br>
or<br>
<code>torch.full((2, 5), 0.25).bernoulli_()</code></p>
<p>So technically, the second saves one step (rand + compare + float vs. full + bernoulli), but I haven’t really seen this as the bottleneck in anything I do, so it entirely depends on my mood - whether the glass is half full or half rand.</p>
<p>Best regards</p>
<p>Thomas</p>
Shared Dict errors in pytorch dataloader

Muhammad_Inam: <p>I have implemented a custom dataset using pytorch. I am stating the sample code below:</p>
<p>class MyDataset(Dataset):<br>
def <strong>init</strong>(self, shared_dict, length):<br>
self.shared_dict = shared_dict<br>
self.length = length</p>
<pre><code>def __getitem__(self, index):
#Check share dict for the index as the key, if key exists return the value.(function then returns from here)

ELSE
#Heavy pre-processing of images happen here that consumes a lot of time
#Add the pre-processed data to the shared dict with the key set as the index.
#return the pre-processed data.
</code></pre>
<p>Since I wanted to save the time taken to pre-process the data in every epoch, I used the following approach:</p><aside class="quote quote-modified" data-post="4" data-topic="32658">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloader-re-initialize-dataset-after-each-iteration/32658/4">Dataloader re-initialize dataset after each iteration?</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    That’s an interesting use case. You are right, variable sized input won’t work with my first approach. 
However, luckily Python provides some implementation for a shared dict. 
Here is a small example using it: 
from multiprocessing import Manager

import torch
from torch.utils.data import Dataset, DataLoader


class MyDataset(Dataset):
    def __init__(self, shared_dict, length):
        self.shared_dict = shared_dict
        self.length = length
        
    def __getitem__(self, index):
     …
  </blockquote>
</aside>

<p>During the training process multiple errors rise including:<br>
Broken Pipe error occurring on the for loop of data loader enumerator (already tried including this for loop in python’s main function scope)<br>
Workers quitting unexpectedly , Memory Buffer Errors, etc…<br>
Tracing the errors always lead towards multiprocessing.manager libraries.</p>
<p>Training configuration:<br>
number of workers = 40<br>
batch size = 150</p>
<p>Hardware Specs:<br>
GPU: Nvidia A100<br>
CPU: EPYC 128 cores processor<br>
RAM: 1 TB<br>
HDD/SDD: Its not a limitation either</p>
<p>Software Specs:<br>
Pytorch: latest version,<br>
CUDA 11,<br>
OS: Ubuntu</p>
<p>Dataset:<br>
Consists of images and json files making up a size of 17GB.</p>
<p>NOTE: Code works perfectly without shared dict.</p>
How to use max_indices given by torch.max(...)?

aydo: <p>Hello,</p>
<p>I have a (soft) adjacency tensor <code>adj</code> of size <code>B x N x M</code> with batch dimension <code>B</code>. When I perform <code>adj.max(dim=2)</code> I get a tuple of max values and max value indices, indicating  for each row in <code>N</code> dimension where it’s maximal in regards to <code>M</code> dimension. Now I would like to use the returned max value indices to <em>select</em> entries from another tensor <code>features</code> of size <code>B x N x M x 10</code>, so that I get a tensor of size <code>B x N x 10</code> which has kept only the entries of it’s <code>dim=2</code> where <code>adj</code> was maximal with respect to <code>dim=2</code>.</p>
<p>I have been able to do this without the batch dimension:</p>
<pre><code class="lang-auto">import torch

adj = torch.tensor([[1,0,0],[0,1,0],[0,1,0],[0,0,1]]) # 4 x 3
features = torch.stack([torch.arange(12).reshape((4,3))]*10).permute((1,2,0)) # 4 x 3 x 10

m = adj.max(dim=1)[1] #
result = features[range(4),m]
print(result)
#result is:
#tensor([[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0],
#            [ 4,  4,  4,  4,  4,  4,  4,  4,  4,  4],
#            [ 7,  7,  7,  7,  7,  7,  7,  7,  7,  7],
#            [11, 11, 11, 11, 11, 11, 11, 11, 11, 11]])

</code></pre>
<p>Now using <code>range(4)</code> there just doesn’t seem like the correct way to handle this to me. Also I have no idea how to do this with an additional batch dimension. Any ideas?</p>

albanD: <p>Hi,</p>
<p>You can use these indices with the <code>other_tensor.gather(dim, indices)</code> function.<br>
Note that you either need to use <code>keepdim=True</code> when you call max or unsqueeze the indices before giving them to gather.</p>

aydo: <p>Thanks, it appears <code>tensor.gather(...)</code> is the right tool. Unfortunately, I still couldn’t get it to work. When I add an extra dimension at the end of <code>adj</code> and use <code>keepdim=True</code>, <code>features.gather(1,m)</code> only returns a tensor of size <code>4 x 1 x 1</code> while I need the whole <code>4 x 10 x 1</code> tensor. So gather somewhat ignores the last dimension of size 10 of features, it only picks the first element of that dimension.</p>
<p><strong>Update</strong>: So I got it to work by using <code>m.repeat(1,1,10)</code> and using that as index tensor for the <code>gather</code> call. This still doesn’t seem like the most elegant solution to me. Now I gotta figure out how to do this with batches.</p>
<p><strong>Update 2</strong>: So batching it is no problem, it’s literally the same with an additional dimension. I’m still interested whether there is a better way than calling repeat. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
database of training and test of a program

Sylvain_Ard: <p>Hello,</p>
<p>I want to test the code <a href="https://github.com/gathierry/FashionAI-KeyPointsDetectionOfApparel" rel="noopener nofollow ugc">https://github.com/gathierry/FashionAI-KeyPointsDetectionOfApparel</a> but the training and test database is no longer available. Have some of you downloaded it before so you can send it to me please ?</p>
<p>Thank you very much</p>
<p>Best regards</p>
How to make two tensor dimensions the same

fyy: <p>if I have two tensor A and B,A= [1,128,4,6,6],B= [1,96,4,6,6],How to make the dimension of A the same as B，I don’t know if we can use the convolution operation</p>

Henry_Chibueze: <p>Does A and B u specified in the question represent the shapes of the actual A and B or are they the actual A and B?</p>

fyy: <p>A and B are the actual sizes in the network, that is, the one mentioned above.</p>

Henry_Chibueze: <p>That’s alota dimensions u got there😑</p>
<p>Can u tell me the context of ur data, like is it a 3d CAD model or sth else?<br>
Also what does each dimension represent?</p>
Visualizing weight updates where loss itself has gradients

Tushar_Gautam: <p>Hi, I’ve written a NN where the loss function consists of gradients of the output with respect to the inputs. The code works fine, but I want to inspect how the gradients flow when I’m updating the NN weights with respect to the loss function. I can’t figure out how the computation graph is laid out here. I hope someone can help me understand how exactly are the gradients computed.</p>
<p>Here’s my code</p>
<pre><code class="lang-auto">dtype = torch.float
device = torch.device("cuda:0") # Uncomment this to run on GPU
x = torch.tensor([[1., 1.]], device=device, dtype=dtype, requires_grad=True)

w1 = torch.tensor([[ 0.6561,  0.6202, -1.5620],
        [ 1.1547, -1.3227,  0.4719]], device=device, dtype=dtype, requires_grad=True)
w2 = torch.tensor([[-0.4917],
        [-0.3886],
        [-1.4218]], device=device, dtype=dtype, requires_grad=True)

learning_rate = 0.01

z1 = x.mm(w1)
a1 = torch.sin(z1)
z2 = a1.mm(w2)
p = torch.sin(z2)
grads, = torch.autograd.grad(p, x, 
                grad_outputs=p.data.new(p.shape).fill_(1),
                create_graph=True, only_inputs=True)
dpdx, dpdt = grads[:,0], grads[:,1]
pde = dpdt + dpdx
loss = pde.pow(2).mean()

loss.backward()
with torch.no_grad():
    w1 -= learning_rate * w1.grad
    w2 -= learning_rate * w2.grad
    # Manually zero the gradients after updating weights
    w1.grad.zero_()
    w2.grad.zero_()       
</code></pre>
<p>I just can’t understand how I get <code>w1.grad</code> and <code>w2.grad</code> (i.e. what are the connections in the computation graph) when I run <code>loss.backward()</code>.</p>
Custom Loss function multiple NN outs

karuu6: <p>Hi,<br>
I’m trying to train a NN to rank objects. The NN takes as input one object at a time and outputs it’s score. For the loss, I’m trying to implement a function of several outputted scores. However, I’m getting stuck on the loss function because I cant do a backward pass as the resulting loss doesn’t have a gradient because it’s composed of several outputs of the NN. How should I go about implementing this?</p>
Multiple Inputs, Concatenate Layers

Tornac: <p>I need a generalizable solution to the following problem:</p>
<p>A neural network has multiple inputs, for example some sort of image (A) which I want to use some convolution layers on etc, and some numerical values (B).<br>
(A) and (B) should at some point feed into the same layer(s) © and eventually produce a result. Graphically it might look like this</p>
<pre><code class="lang-auto">A1 -&gt; A2 -&gt; A3 \
                           C1 -&gt; C2 -&gt; C3
B1 -&gt; B2 -&gt; B3 /
</code></pre>
<p>In keras this would be solved by creating a model out of the (A) layers, creating a second model out of the (B) layers and calling keras.layers.concatenate on them.</p>
<p>Is something similar possible by stacking torch’s Sequential models and if so, how? The examples I’ve seen use classes derived from nn.Module which doesn’t fit my needs unfortunately.</p>

ptrblck: <aside class="quote no-group" data-username="Tornac" data-post="1" data-topic="101792">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/2bfe46/48.png" class="avatar"> Tornac:</div>
<blockquote>
<p>The examples I’ve seen use classes derived from nn.Module which doesn’t fit my needs unfortunately.</p>
</blockquote>
</aside>
<p>I’m unsure why this wouldn’t fit your need. Could you explain your concerns a bit?</p>
<p>The general approach for your use case would be:</p>
<ul>
<li>define <code>modelA</code>, <code>modelB</code>, and <code>modelC</code></li>
<li>register all models in a <code>parentModel</code></li>
<li>pass two inputs to <code>parentModel.forward</code>, call each branch using the corresponding data, concatenate the activations, and feed it to <code>modelC</code></li>
</ul>
<p>Here is an example:</p>
<pre data-code-wrap="python"><code class="lang-python">class ParentModel(nn.Module):
    def __init__(self, modelA, modelB, modelC):
        super(ParentModel, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        self.modelC = modelC
        
    def forward(self, x1, x2):
        x1 = self.modelA(x1)
        x2 = self.modelB(x2)
        x = torch.cat((x1, x2), dim=1)
        x = self.modelC(x)
        return x
        
modelA = nn.Linear(10, 10)
modelB = nn.Linear(10, 10)
modelC = nn.Linear(20, 10)
parent = ParentModel(modelA, modelB, modelC)
x1 = torch.randn(1, 10)
x2 = torch.randn(1, 10)
out = parent(x1, x2)
print(out.shape)
&gt; torch.Size([1, 10])
</code></pre>
<p>Instead of the <code>nn.Linear</code> modules for <code>modelX</code> you can also use <code>nn.Sequential</code> if you want.</p>
Time is running backwards!

Albert_Suso: <p>Executing some pytorch code I get this message by stdout: “Time is running Backwards!”. What does it mean?</p>
<p>It doesn’t seems to affect the program execution at all.</p>

ptrblck: <p>That’s an interesting message you are seeing. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Could you post more information which part of the code is creating this output?</p>
Where should I place .zero_grad()?

unnir: <p>Hi all,</p>
<p>I just want to be sure, where should I use the .zero_grad() function?<br>
In the official MNIST example, the .zero_grad() function is used in the beginning of the training loop.</p>
<pre><code class="lang-auto">def train(args, model, device, train_loader, optimizer, epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % args.log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))
            if args.dry_run:
                break
</code></pre>
<p>from here: <a href="https://github.com/pytorch/examples/blob/a74badde33f924c2ce5391141b86c40483150d5a/mnist/main.py#L37" rel="noopener nofollow ugc">https://github.com/pytorch/examples/blob/a74badde33f924c2ce5391141b86c40483150d5a/mnist/main.py#L37</a></p>
<p>Also, in the official tutorials, the zero_grad() function is used right before the backward() function.</p>
<pre><code class="lang-auto">for t in range(500):
    # Forward pass: compute predicted y by passing x to the model. Module objects
    # override the __call__ operator so you can call them like functions. When
    # doing so you pass a Tensor of input data to the Module and it produces
    # a Tensor of output data.
    y_pred = model(x)

    # Compute and print loss. We pass Tensors containing the predicted and true
    # values of y, and the loss function returns a Tensor containing the
    # loss.
    loss = loss_fn(y_pred, y)
    if t % 100 == 99:
        print(t, loss.item())

    # Zero the gradients before running the backward pass.
    model.zero_grad()

    # Backward pass: compute gradient of the loss with respect to all the learnable
    # parameters of the model. Internally, the parameters of each Module are stored
    # in Tensors with requires_grad=True, so this call will compute gradients for
    # all learnable parameters in the model.
    loss.backward()
</code></pre>
<p>In my understanding, the first way makes sense, because we want to have gradient only for the current batch…</p>

ptrblck: <p>Both approaches are valid for the standard use case, i.e. if you do not want to accumulate gradients for multiple iterations.<br>
You can thus call <code>optimizer.zero_grad()</code> everywhere in the loop but not between the <code>loss.backward()</code> and <code>optimizer.step()</code> operation.</p>
Teacher force training

learner47: <p>I am trying to do a seq2seq prediction. For this, I have an LSTM layer followed by a fully connected layer. I employ Teacher training during the training phase and would like to skip this (I maybe wrong here) during testing phase. I have not found a direct way of doing this so I have taken the approach shown below.</p>
<pre><code class="lang-auto">def forward(self, inputs, future=0, teacher_force_ratio=0.2, target=None):
	outputs = []
	for idx in range(future):
		rnn_out, _ = self.rnn(inputs)
		output = self.fc1(rnn_out)
		if self.teacher_training:
			new_input = output if np.random.random() &gt;= teacher_force_ratio else target[idx]
		else:
			new_input = output
		inputs = new_input
</code></pre>
<p>I use a <code>bool</code> variable <code>teacher_training</code> to check if Teacher training is needed or not. Is this correct? If it is, is there a better way of doing it? Thanks.</p>

ptrblck: <p>Your approach is fine, if you want to set the <code>teacher_training</code> argument independently.<br>
Alternatively, you could also use the internal <code>self.training</code> flag, which will be changed by calling <code>model.train()</code> or <code>model.eval()</code>.</p>
How to trace just neuron activations

dugr: <p>Hello, I am wondering if there is some written method already to extract whether neuron is just activated or not.<br>
Something like this:</p>
<pre><code class="lang-auto">if neuron_value &gt;=0:
    then neuron_value =1
else:
   neuron_value = 0
</code></pre>
<p>but for all neurons in the network.  The final output of one layer (linear or convolutional) in this case would be e.g.</p>
<blockquote>
<p>[0 1 0 1 1 1 1 1 1]</p>
</blockquote>
<p>or</p>
<blockquote>
<p>[[0 1 0 0 1 0 ], [0 1 0 1 0 0], [0 1 0 1 0 0], [0 1 1 1 0 0]]</p>
</blockquote>
<p>…</p>
<p>I know now how to get activations values (from this post: <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/55" class="inline-onebox">How can l load my best model as a feature extractor/evaluator? - #55 by dugr</a>), but now want just to track whether the neuron is on or off in a network.<br>
It can be calculated manually also (wit <code>if</code> and <code>else</code> in every layer), but in case of convolutional and dense neural networks and requires lots of programming and considering tensors shapes and dimensions (especially in convolutional layers), so I was wondering if there is some other solution that calculates it for all layers at once for specific input.</p>
<p>Thank you.</p>
nn.Linear paramater meaning

jack_giorgio: <p>Hello,<br>
Just started PyTorch, and I cannot understand the meaning of nn.Linear(input_features, output_features, bias = True)<br>
From my point of view, linear regression is supposed to be determined by its weight and bias, what does input_features and output_features mean?</p>

Henry_Chibueze: <p>Ok lemme explain this:<br>
U see the input feature simply means how many features of a single data point is going to be fed into the network, the output feature means the size of the output  and the bias is simply a matrix of values that will be added to the product of ur input and weight.</p>
<p>Eg:<br>
Say u have a training data with shape (N, F) where N is the number of data points an F is the number of features per data point.<br>
Let’s select a random data point from the training data, it’s shape will be (1, F).<br>
Let’s assume that F = 5, therefore there are 5 features per data point</p>
<p>In this case if ur network is a single dense / linear layers network, the input feature is 5.<br>
The output feature in this case will be the number of points per target label (continuous or categorical) simply because it’s a single layer network.<br>
So now let’s assume that the points per target in ur training data is 1, therefore ur output feature will be 1.<br>
Your linear layer will look sth like this:<br>
nn.Linear(5, 1).<br>
Now that these numbers have been specified, the shape of your weights will be (5, 1) and the size of ur bias will be (1, 1) because the the output neuron / feature  is 1</p>
<p>So let’s say ur weights is denoted by ‘w’ and bias by ‘b’ and input by ‘x’ and output by ‘y_pred’ and target per data point by ‘y_actual’:</p>
<p>y_pred = xw + b ≠ wx + b (coz of matrix multiplication rules)</p>
<p>Now the shape of y_pred is always equal to the shape of y_actual so that the loss can be computed fine.<br>
Also from what I initially said, there is 1 point per target so the target data is a shape of (N, 1) therefore, the target for a single data point (y_actual) should be a of shape (1, 1)</p>
<p>Now if u remember from matrix multiplication, let’s check if there shapes are compatible:</p>
<p>y_pred =  (1, 5) x (5, 1) + (1, 1)</p>
<p>The shape of y_pred is (1, 1) for a single data point which is correct coz it’s also the same as that of y_actual.</p>
<p>Now these weights and biases are initalized randomly and behind the scenes so u can’t really see them except u print them out.<br>
U can also initialize the weights however u please, but just be careful not to make them too large and with same values in each cell (can cause exploding gradients).<br>
Sometime u would want to set the bias to false simply coz u don’t need them Eg: When u are using batch Normalization after a layer.</p>
<p>I Know I talked too much but hopefully after this u’ll understand this in full detail.</p>
<p>Cheers🙃</p>

Henry_Chibueze: <p>So in summary of it all the input and output feature parameters are used to instanciate the shape of the weight and bias.</p>
Confused about my self-attention implementation

razent: <p>Hi everyone,</p>
<p>I tried implementing self-attention mechanism for sequence labeling task (like NER). I fed BiLSTM outputs with (batch_size, seq_length, embed_dim) shape and mask with (batch_size, seq_length) shape into the self-attention. I want my model can focus on the input sentence itself. But the performance is worst than the model without attention.</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">class SelfAttention(nn.Module):
  """
    Self-Attention mechanism.
  """
  def __init__(self, emb_dim, kqv_dim, num_heads=1):
    super(SelfAttention, self).__init__()
    self.emb_dim = emb_dim
    self.kqv_dim = kqv_dim
    self.num_heads = num_heads

    self.w_k = nn.Linear(emb_dim, kqv_dim*num_heads, bias=False)
    self.w_q = nn.Linear(emb_dim, kqv_dim*num_heads, bias=False)
    self.w_v = nn.Linear(emb_dim, kqv_dim*num_heads, bias=False)
    self.w_out = nn.Linear(kqv_dim * num_heads, emb_dim)
  
  def get_mask(self):
    pass

  def forward(self, inputs, attention_mask=None):
    b, t, _ = inputs.shape
    e = self.kqv_dim
    h = self.num_heads

    if attention_mask is not None:
      attention_mask = attention_mask.unsqueeze(-1)
      inputs = inputs * attention_mask
    
    keys = self.w_k(inputs).view(b, t, h, e)
    values = self.w_v(inputs).view(b, t, h, e)
    queries = self.w_q(inputs).view(b, t, h ,e)

    keys = keys.transpose(1, 2).contiguous().view(b*h, t, e)
    queries = queries.transpose(1, 2).contiguous().view(b*h, t, e)
    values = values.transpose(1, 2).contiguous().view(b*h, t, e)

    attn_scores = torch.bmm(queries, keys.transpose(1, 2))
    attn_scores = attn_scores / np.sqrt(e)

    if attention_mask is not None:
      attn_scores = attn_scores * attention_mask
    attn_scores = F.softmax(attn_scores, dim=2)
    weighted_values = torch.bmm(attn_scores, values).view(b, h, t, e)
    weighted_values = weighted_values.transpose(1, 2).contiguous().view(b, t, h*e)
    if attention_mask is not None:
      weighted_values = weighted_values * attention_mask
    outputs = self.w_out(weighted_values)
    
    return outputs
</code></pre>
<p>What’s wrong with my code?<br>
Thank you!</p>

Abhilash_Srivastava: <p>Have you thought of using the Transformers instead?</p>

razent: <p>Hi,</p>
<p>I already thought of using the Transformers; yet, I want to try this way first.<br>
My implementation based on Self-Attention in Transformers. Maybe it does not coincide with my goal.</p>
Fuse_conv_bn_eval does not work as expected

coincheung: <p>Hi,</p>
<p>My test code is like this:</p>
<pre><code class="lang-auto">import torch


conv = torch.nn.Conv2d(3, 32, 3, 1, 1)
bn = torch.nn.BatchNorm2d(32)
conv.eval()
bn.eval()


fused = torch.nn.utils.fuse_conv_bn_eval(conv, bn)

inten = torch.randn(2, 3, 224, 224)

out1 = bn(conv(inten))
out2 = fused(inten)
print((out1 - out2).abs().sum())
</code></pre>
<p>The output is <code>tensor(0.1754, grad_fn=&lt;SumBackward0&gt;)</code>. Why is there so large difference between the separate computation and fused computation, and how could I make the fussd version work like separate version?</p>

ptrblck: <p>If you check the <code>(out1 - out2).abs().max()</code> value you’ll see that it’s in the range <code>~1e-6</code>, which is most likely created due to the limited floating point precision.<br>
Summing these small absolute errors might result in the larger difference.</p>

coincheung: <p>Thanks for telling me that!!</p>
<p>I tried to replace one <code>conv-bn</code> pair with the fused conv in a trained model, and test the model on some test set. The acc drops almost 1%. It seems that the point precision error is not neglectful. How could I fuse <code>conv-bn</code> pair without losing accuracy please ?</p>

ptrblck: <p>If you think these changes are causing a drop in model performance, you could fine-tune the model with the fused layers for some iterations and check if this would recover the performance.</p>

coincheung: <p>Thanks !!</p>
<p>By the way, is there any suggestions about finetune like this ? I plan to fine tune for 5 epochs, with the lr value and its annealing strategy not changed (I used cosine lr to train my model before fuse).  Shall I use a smaller lr or have the lr annealing curve changed ?</p>

ptrblck: <p>Unfortunately I don’t know what the best approach would be.<br>
If possible, I would try to add the fused layers from the beginning, but if your model is already trained and you would like to fuse these layers now, you would have to try out different approaches. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED

rina: <p>I have the following model:</p>
<pre><code class="lang-auto">model = torch.nn.Sequential(
        torch.nn.Conv3d(1, 128, kernel_size=3, padding=1),
        torch.nn.ReLU(),
        torch.nn.Conv3d(128, 128, kernel_size=3, padding=1),
        torch.nn.ReLU(),
        torch.nn.Conv3d(128, 128, kernel_size=3, padding=1),
        torch.nn.ReLU(),
        torch.nn.Conv3d(128, 128, kernel_size=3, padding=1),
        torch.nn.ReLU(),
        torch.nn.Conv3d(128, 128, kernel_size=3, padding=1),
        torch.nn.ReLU(),
        torch.nn.Conv3d(128, 1, kernel_size=3, padding=1),
        torch.nn.ReLU()
    ).to(device)
</code></pre>
<p>For which, I use batches of the size N x 160 x 256 x 256<br>
I was running into the <code>RuntimeError: CUDA out of memory</code> error so I reduced the batch size to 2, which resulted in this error instead:<br>
<code>RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.</code><br>
A quick search in google suggested that my batch size was still too large so I reduced it 1, which surprisingly gave me the <code>RuntimeError: CUDA out of memory</code> error again.</p>
<p>How can I resolve this issue?</p>
<p>Thank you.</p>

ptrblck: <p>The first two <code>nn.Conv3d</code> layers with an an input of <code>[1, 1, 160, 256, 256]</code> will already take ~20GB during the forward pass (with intermediate activation needed for the backward pass). The complete model will thus have a much higher memory usage and I assume your GPU doesn’t have enough memory.</p>
How do i count the list of nested list?

Ripley: <p>Hi, I would like to ask how can i read a list of list and count the numbers of True inside?</p>
<p>For example,</p>
<p>tensor([[ True, False,  True, False,  True],<br>
[ True,  True,  True,  True,  True],<br>
[False, False, False,  True,  True]])</p>
<p>How do I iterate and count the numbers of True inside the whole list?</p>

ptrblck: <p>You could use <code>x.sum()</code> to get the number of <code>True</code> values inside the tensor.</p>

Ripley: <p>Thank you! it seem to work when i tried it on a random list. Gonna try it on my actual function after its done training! thank you!</p>

Ripley: <p>sry, what if I want to calculate the entire amount in the entire list? this x.sum() surprisingly sums the total number of True. Awesome!</p>
<p><span class="hashtag">#Edit</span></p>
<p>Hi, I tried using len(), but does not work like how sum() works… Is there other way possible?</p>

spurscoder: <p>Maybe, you could use <code>x.numel()</code> to get the number of values inside the tensor.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.numel.html" rel="noopener nofollow ugc">torch.numel() link</a></p>
How to calculate the memory requirement for running inferences

bing: <p>Hi Everyone,<br>
I have trained a deep learning model for image classification.<br>
The trained model size is 250 Mb. How to calculate, how much computation memory, it’ll require to run inferences.<br>
I am trying to run it on my 16GB RAM Macbook Pro, but it is failing after the first iteration.</p>

anujd9: <p>Hi. I think you can use the PyTorch Profiler here: <a href="https://pytorch.org/tutorials/recipes/recipes/profiler.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/recipes/profiler.html</a>   for getting the amount of memory required per layer, execution time per layer in the model as well as show the amount of memory (used by the model’s tensors) that was allocated (or released) during the execution of the model’s operators.<br>
Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Pack_padded_sequence and pad_packed_sequence

Vandana_Rajan: <p>Hi,</p>
<p>I want to use the Keras ‘masking layer’ equivalent in PyTorch. I came up with the ‘pack_padded_sequence’ and ‘pad_packed_sequence’ examples and I have 3 doubts.</p>
<p>My input data is of the shape (batch_size, seq_length, feat_dim) = (10, 63, 100). For each batch, I am executing the following code in my model’s ‘forward’ method.</p>
<blockquote>
<p>seq_lens = np.count_nonzero(x.cpu().numpy(), axis=1)[:, 0]<br>
pack = pack_padded_sequence(x.float(), -np.sort(-seq_lens), batch_first=True)<br>
gru_out,h = self.Gru(pack)<br>
unpacked, unpacked_len = pad_packed_sequence(gru_out, batch_first=True)</p>
</blockquote>
<p>‘pack.data’ has dimension [256,100]. So my 1st doubt is how does the GRU understand the ‘time-steps’ from this data input?</p>
<p>For every batch, the actual (without zero padding) maximum sequence length varies. Is that permitted or will it create any issues?</p>
<p>From the ‘unpacked.data’ how do I recreate the new data [10, 63, new_feat_dim]? This is needed because my labels are also zero padded with size [10, 63].</p>
<p>Thanks,</p>

googlebot: <p>that’s basically cudnn’s internal format, you don’t need to know it under normal circumstances</p>
<aside class="quote no-group" data-username="Vandana_Rajan" data-post="1" data-topic="101775">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vandana_rajan/48/19834_2.png" class="avatar"> Vandana_Rajan:</div>
<blockquote>
<p>For every batch, the actual (without zero padding) maximum sequence length varies. Is that permitted or will it create any issues?</p>
</blockquote>
</aside>
<p>no major issues, besides some extra memory heap fragmentation</p>
<aside class="quote no-group" data-username="Vandana_Rajan" data-post="1" data-topic="101775">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vandana_rajan/48/19834_2.png" class="avatar"> Vandana_Rajan:</div>
<blockquote>
<p>From the ‘unpacked.data’ how do I recreate the new data [10, 63, new_feat_dim]? This is needed because my labels are also zero padded with size [10, 63].</p>
</blockquote>
</aside>
<p>“unpacked” is a tensor that is already padded. don’t use the .data attribute there.</p>
Improve Mask R-CNN segmentation

FiReTiTi: <p>Hi,<br>
I’ve been using Mask R-CNN on my project and I’m very happy with the detection results. However, I’m not fully satisfied by the segmentation as I don’t think it is precise enough for my needs.</p>
<p>I tried to modify the <code>MaskRCNNPredictor</code> without success. So:</p>
<ul>
<li>What can I do to improve the segmentation part?</li>
<li>Can I change the loss?</li>
<li>Can I modify other parts of the model?</li>
</ul>
<p>Thank you for your help</p>
Left shift tensor

learner47: <p>I have a tensor <code>a</code> of shape <code>(1, N, 1)</code>. I need to left shift the tensor along dimension <code>1</code> and add a new value as replacement. I have found a way to make this work and following is the code.</p>
<pre><code class="lang-auto">a = torch.from_numpy(np.array([1, 2, 3]))
a = a.unsqueeze(0).unsqeeze(2)  # (1, 3, 1), my data resembles this shape, therefore the two unsqueeze
# want to left shift a along dim 1 and insert a new value at the end
# I achieve the required shifts using the following code
b = a.squeeze
c = b.roll(shifts=-1)
c[-1] = 4
c = c.unsqueeze(0).unsqueeze(2)
# c = [[[2], [3], [4]]]
</code></pre>
<p>My question is, is there a simpler way to do this? Thanks.</p>

learner47: <p>I found out that unsqueezing and squeezing are not required and can be combined.</p>
<pre><code class="lang-auto">a = torch.from_numpy(np.array([1, 2, 3])[None, :, None])  # (1, 3, 1)
b = torch.roll(a, shifts=-1, dims=1)
b[:, -1, :] = 4
</code></pre>
Introducing augmentations at specific time during training?

kirk86: <p>What’s the pytorch way of dataloader to introduce specific augmentations at specific time/epoch during training? Can we pass any parameter we want to <code>__getittem__()</code> of the dataloader (e.g. epoch)?</p>

ptrblck: <p>Yes, you could pass more arguments to <code>__getitem__</code>, but would need to feed them from the sampler.<br>
You could change some internal attributes of the <code>Dataset</code> before starting each new epoch.<br>
During the epoch the <code>DataLoader</code> will create copies of the <code>Dataset</code> for each worker, so changing internal attributes won’t be reflected in all copies.</p>

kirk86: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, if I understood you correctly something like the following MWE is what you’re describing?</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
     def __init__(self, *args, epoch=0):
         self.epoch = epoch

     def __getittem__(self, index):
         x, y = self.data[index]
         if self.epoch == epoch:
            x = transforms(x)
         return x, y

# then during training
for epoch in range(epochs):
     dataset.epoch = epoch
     for x, y in loader:
          # train as usual
</code></pre>

ptrblck: <p>Yes, with a minor change: since you’ve wrapped the <code>Dataset</code> in a <code>DataLoader</code>, you should be using <code>loader.dataset.epoch = epoch</code> in the outer loop.</p>
Prepare_qat on the top level vs each layer

thyeros: <p>Hello, everyone.</p>
<p>I have a question on how to use prepare_qat properly.</p>
<p>I’m interested in quantizing a subset of models and like to apply prepare_qat selectively. However, a simple test code below reveals that such a selective case will not lead to the same quant-prep as the case where it is done at the top-level.</p>
<blockquote>
<p>class LeNet(nn.Module):</p>
<pre><code>def __init__(self):
    super().__init__()
    self.l1 = nn.Linear(28 * 28, 10)    
    self.relu1 = nn.ReLU(inplace=True)

def forward(self, x):             
    return self.relu1(self.l1(x.view(x.size(0), -1)))  
</code></pre>
<p>model_a = LeNet()<br>
model_b = LeNet()<br>
qconfig = torch.quantization.get_default_qat_qconfig()</p>
</blockquote>
<p>Now, when comparing two ways of apply prepare_qat</p>
<ul>
<li>top-level prepare_qat</li>
</ul>
<blockquote>
<p>model_a.qconfig = qconfig<br>
torch.quantization.prepare_qat(model_a, inplace=True)<br>
print(model_a)</p>
</blockquote>
<ul>
<li>layer-wise prepare_qat</li>
</ul>
<blockquote>
<p>model_b.l1.qconfig = qconfig<br>
torch.quantization.prepare_qat(model_b.l1, inplace=True)<br>
model_b.relu1.qconfig = qconfig<br>
torch.quantization.prepare_qat(model_b.relu1, inplace=True)<br>
print(model_b)</p>
</blockquote>
<p>You can see below that print(model_a) and print(model_b) will yield different results, notably, weight_fake_quant is missing in model_b.l1. Can someone explain why these two have different behaviors and the right way to do selective quantization?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec988995308761e32b4bc98d01f628d55ec9a9b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec988995308761e32b4bc98d01f628d55ec9a9b7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec988995308761e32b4bc98d01f628d55ec9a9b7_2_614x500.png" alt="image" data-base62-sha1="xL1uyY9nrHCJslaYV0wkXpdoP9d" width="614" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec988995308761e32b4bc98d01f628d55ec9a9b7_2_614x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec988995308761e32b4bc98d01f628d55ec9a9b7_2_921x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec988995308761e32b4bc98d01f628d55ec9a9b7_2_1228x1000.png 2x" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2486×2022 527 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>