akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: The expanded size of the tensor (1) must match the existing size (10) at non-singleton dimension 1. Target sizes: [1, 1]. Tensor sizes: [10]

insung3511: <p>I got a trouble with this…</p>
<pre><code class="lang-python">p_h = F.sigmoid(
            F.linear(v.view(1, self.batch), 
                     self.W.view(1, self.batch).expand(1, self.batch), 
                     self.h_bias)
        )
</code></pre>
<p>help me plz</p>

eqy: <p>Could you post a runnable code snippet that reproduces this error? I can’t seem to reproduce this issue e.g.,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; W = torch.randn(10)
&gt;&gt;&gt; W.view(1, 10)
tensor([[ 0.0540,  0.6975,  0.9764, -2.3373,  1.4731, -0.4768,  0.8725,  1.1909,
         -0.3579, -1.1160]])
&gt;&gt;&gt; W.view(1, 10).expand(1, 10)
tensor([[ 0.0540,  0.6975,  0.9764, -2.3373,  1.4731, -0.4768,  0.8725,  1.1909,
         -0.3579, -1.1160]])
</code></pre>

insung3511: <p>Hopefully I solved it myself. solutions was similar with your code.<br>
My solutions is <a href="https://stackoverflow.com/questions/72081872/runtimeerror-output-with-shape-1-doesnt-match-the-broadcast-shape-10" rel="noopener nofollow ugc">here</a>.</p>
<p>I really thank you that u tried to help me.</p>
RuntimeError: output with shape [] doesn't match the broadcast shape [10]

insung3511: <p>I got problem with RBM code… Here is the code of problem part.</p>
<pre><code class="lang-python">def v_to_h(self, v):
        h_bias = (self.h_bias.clone()).expand(10)
        v = v.clone().expand(10)
        w = self.W.clone().squeeze()

        print(h_bias.size(), v.size(), w.size())
        print(h_bias.dim(), v.dim(), w.dim())

        p_h = F.sigmoid(
            F.linear(v, w, bias=h_bias)
        )

        sample_h = self.sample_from_p(p_h)
        return p_h, sample_h
</code></pre>
<p>and got errors like this.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/Users/bahk_insung/Documents/Github/ecg-dbn/model.py", line 67, in &lt;module&gt;
    v, v1 = rbm(sample_data)
  File "/Users/bahk_insung/miniforge3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/bahk_insung/Documents/Github/ecg-dbn/RBM.py", line 55, in forward
    pre_h1, h1 = self.v_to_h(v)
  File "/Users/bahk_insung/Documents/Github/ecg-dbn/RBM.py", line 37, in v_to_h
    F.linear(v, w, bias=h_bias)
  File "/Users/bahk_insung/miniforge3/lib/python3.9/site-packages/torch/nn/functional.py", line 1850, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: output with shape [] doesn't match the broadcast shape [10]
</code></pre>
<p>weight, v (is data from visible layer) and bias is same size and same dimension. 10 tensor size and 1-D. Seriously Im figuring out for 3days. I still can’t find what’s wrong with me.</p>

ptrblck: <p>This post seems to be related to <a href="https://discuss.pytorch.org/t/runtimeerror-the-expanded-size-of-the-tensor-1-must-match-the-existing-size-10-at-non-singleton-dimension-1-target-sizes-1-1-tensor-sizes-10/150512">your previous topic</a>. Could you respond to the other topic as <a class="mention" href="/u/eqy">@eqy</a> already followed up there, please?</p>

insung3511: <p>oh okay. I forgot that I asked there. I’m apologize that.</p>
Do I need to uninstall cuda 11.6?

Ged: <p>Hello,</p>
<p>I have installed cuda 11.6, and realize now that 11.3 is required. Would you recommend to uninstall cuda 11.6 and re-install cuda 11.3?</p>
<p>Thanks in advance!</p>

ptrblck: <aside class="quote no-group" data-username="Ged" data-post="1" data-topic="150568">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/85e7bf/48.png" class="avatar"> Ged:</div>
<blockquote>
<p>I have installed cuda 11.6, and realize now that 11.3 is required.</p>
</blockquote>
</aside>
<p>I guess you are referring to the binaries (pip wheels and conda binaries), which both ship with their own CUDA runtime.<br>
If so, then no you do not need to uninstall your local CUDA toolkit, as the binaries will use their CUDA runtime.<br>
Your local CUDA toolkit will be used if you are building PyTorch from source or a custom CUDA extension. To run the binaries you would only need to install an NVIDIA driver.</p>

Ged: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a> for your quick reply.</p>
<p>Not sure actually if these are the binaries you mentioned</p>
<ol>
<li>I have first installed cuda 11.6 from <a href="https://developer.nvidia.com/cuda-downloads" class="inline-onebox" rel="noopener nofollow ugc">CUDA Toolkit 11.6 Update 2 Downloads | NVIDIA Developer</a>
</li>
<li>I have then realized 11.3 is required whilst downloading Pytorch for windows with pip, python and cuda 11.3. So using this command: pip3 install torch torchvision torchaudio --extra-index-url <a href="https://download.pytorch.org/whl/cu113" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu113</a>
</li>
</ol>

ptrblck: <p>Yes, I was referring to the pip wheels mentioned in your second step as the “binaries”.<br>
The pip wheels do not require a matching local CUDA toolkit (installed in your first step), as they will use their own CUDA runtime (CUDA 11.3 in your selection), so you can keep your local CUDA toolkit (11.6U2).</p>

Ged: <p>Thanks again <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
How to translate my own sentence using attention mechanism?

Sri93: <p>For every language translation repos, I can see the existing sentences which are translated. If i give my own sentence ( eventhough the words of my own sentence are existed in the dataset) the translation I am getting is the existing sentence from the dataset.</p>
<p>Please help me to resolve this issue</p>
<p>Reference link for the code <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">NLP From Scratch: Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials 1.11.0+cu102 documentation</a></p>
<p>Reference sentence which I tried to translate on <a href="https://i.stack.imgur.com/GuoSE.png" rel="noopener nofollow ugc">enter image description here</a></p>

djlindsay: <p>Hi, I’m sort of in the same boat as you, getting transformer inference stages to work right. Are you including the  token at the beginning of your input?</p>

Sri93: <p>Yes, I included the Tokens</p>
How to create a sparse tensor give following value?

yiwc: <p>Dear Team,</p>
<p>value=torch.tensor(xx) # shape=[64,10]<br>
ids=torch.tensor(xx) # shape=[10], eg [94,13,20,6,27,45,15,7,53,2]</p>
<p>–how-to-create–&gt;</p>
<p>sparse-tensor= x # shape [64,100], with ids’ position values in each row equals to value.</p>
<p>one more question,<br>
does the operation torch.sparse.mm support GPU?</p>
<p>THanks Team!</p>

Matias_Vasquez: <p>If your sizes are relatively small, then you can use something like this to create a sparse matrix using the default strided tensor layout.</p>
<pre><code class="lang-python">value=torch.rand(64, 10) # shape=[64,10]
ids=torch.randint(0, 99, (10,)) # shape=[10], eg [94,13,20,6,27,45,15,7,53,2]

sparse_tensor = torch.zeros(64, 100)
sparse_tensor[:, ids] = value
</code></pre>
<p>It will not make that much of a difference in memory if you define it like this or as a <code>sparse_tensor</code>. For matrix multiplication you can then use <code>@</code> or <code>torch.matmul</code> or <code>torch.mm</code>.</p>
<p>However, if you do have very large sparse matrices, then you can either create a <code>torch.sparse_coo_tensor</code> or a <code>torch.sparse_csr_tensor</code>.</p>
<p>According to the <a href="https://pytorch.org/docs/stable/sparse.html#sparse-coo-tensors" rel="noopener nofollow ugc">documentation</a>, <code>torch.sparse_csr_tensor</code> does not support CUDA, so I will show you how to do a <code>torch.sparse_coo_tensor</code> for your case.</p>
<pre><code class="lang-python">value=torch.rand(64, 10) # shape=[64,10]
ids=torch.randint(0, 99, (10,)) # shape=[10], eg [94,13,20,6,27,45,15,7,53,2]

# First you need to redefine your indices to be coordinates.
#   There are many ways to do it as shown on the documentation
#   For this example I did it like this
#   [[0, 94], [0, 13], [0, 20], ..., [63, 7], [63, 53], [63, 2]] 
idx = [[i, int(j)] for i in range(64) for j in ids]
sparse_tensor = torch.sparse_coo_tensor(list(zip(*idx)), value.view(-1), (64, 100))
</code></pre>
<p>also, if you want to see which operations support gradient, you can look <a href="https://pytorch.org/docs/stable/sparse.html#supported-linear-algebra-operations" rel="noopener nofollow ugc">here</a></p>

yiwc: <p>THanks Matias, that is a very detailed answer.</p>
The weight of the convolution kernel become NaN after training several batches

ojipadeson: <p>My network’s weights suddenly change to NaN during the training process. That is, it trains normally for a period of time and suddenly go to NaN in a random batch(not the same batch). In debug mode, I found all inputs are normal. And the item() of the last loss is normal. What may cause the network weights to be NaN?</p>

BrianPulfer: <p>Weights going to NaN is typically due to overflow. The most common cause I know for this issue is a too high learning rate and no gradient clipping. This makes the parameters of your network to diverge towards +/- infinity.</p>
<p>Have you tried lowering the learning rate? You could also log the norm of the gradients (<a href="https://discuss.pytorch.org/t/check-the-norm-of-gradients/27961" class="inline-onebox">Check the norm of gradients</a>).</p>

ojipadeson: <p>I applied <code>nn.utils.clip_grad_norm_(max_norm=2)</code> in my learning process, but there is still the case of NaN. My init learning rate is 0.0004 and I use an <code>ExponentialLR(gamma=0.9)</code> learning schedule. I’m wondering if the <code>reduction</code> of loss to <code>mean</code> or <code>sum</code> or the weight of every sub-loss (<code>loss = w1*loss1 + w2*loss2 + .. + wn*lossn</code>) would have a significant effect to the gradients. And is there a way to ensure that the gradient in the network does not appear Nan?</p>

BrianPulfer: <p>There is a similar thread at <a href="https://discuss.pytorch.org/t/gradient-value-is-nan/91663/3" class="inline-onebox">Gradient value is nan - #3 by saumya0303</a></p>
<p>As <a class="mention" href="/u/ptrblck">@ptrblck</a> suggested, you could use <code>torch.autograd.set_detect_anomaly(True)</code> to see when the gradient go to NaN and debug from there.</p>
<p>Hope this helps.</p>
Params stored in GPU during training

base-y: <p>During training on GPUs, what are the exact parameters which take up the GPU memory? From my understanding, I think these are the ones:</p>
<ol>
<li>The model weights (and bias terms) which need to be loaded into memory.</li>
<li>The (batch size x input_seq_length) for the inputs.</li>
<li>The activations when using forward pass.</li>
<li>The gradients when doing backward pass.</li>
<li>The optimizer states assuming adam (first moment for every weight param, second moment for every weight param).</li>
</ol>
<p>Did I miss something else? Please shed some light if something is wrong or I missed out something. Would like to understand what is consuming GPU memory.</p>

ptrblck: <p>Your summary looks correct. Besides that some method could also create intermediate tensors on the specified device in the backend, which would (temporarily) use more memory. Some libraries also use workspaces (e.g. cuDNN, cublas), others need to initialize some buffers (NCCL), so a small overhead could also come from these.</p>
RuntimeError: shape '[-1, 128]' is invalid for input of size 97250

praggya: <blockquote>
<p>Blockquote<br>
class Linear_overtime(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden_size,bias=True):<br>
super(Linear_overtime, self).<strong>init</strong>()<br>
self.fc = nn.Linear(input_size, hidden_size, bias=bias)<br>
self.input_size = input_size<br>
self.hidden_size = hidden_size</p>
</blockquote>
<pre><code>def forward(self, x):
    y = x.contiguous().view(-1, self.input_size)
    y = self.fc(y)
    y = y.view(x.size()[0], x.size()[1], self.hidden_size)
    return y
</code></pre>
<p>class UnICORNN(nn.Module):<br>
def <strong>init</strong>(self, ninp, nhid, nout, dt, alpha, n_layers, drop=0.1):<br>
super(UnICORNN, self).<strong>init</strong>()<br>
self.nhid=nhid<br>
self.drop = drop<br>
self.nlayers = n_layers<br>
self.DIs=nn.ModuleList()<br>
denseinput=Linear_overtime(ninp, nhid)<br>
self.DIs.append(denseinput)<br>
for x in range(self.nlayers - 1):<br>
denseinput = Linear_overtime(nhid, nhid)<br>
self.DIs.append(denseinput)<br>
self.classifier = nn.Linear(nhid, nout)<br>
self.init_weights()<br>
self.RNNs = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for x in range(self.nlayers):<br>
rnn = UnICORNN_recurrence(nhid,dt,alpha)<br>
self.RNNs.append(rnn)<br>
self.RNNs = torch.nn.ModuleList(self.RNNs)</p>
<pre><code>def init_weights(self):
    for name, param in self.named_parameters():
        if ('fc' in name) and 'weight' in name:
            nn.init.kaiming_uniform_(param, a=8, mode='fan_in')
        if 'classifier' in name and 'weight' in name:
            nn.init.kaiming_normal_(param.data)
        if 'bias' in name:
            param.data.fill_(0.0)

def forward(self, input):
    rnnoutputs={}
    rnnoutputs['outlayer-1']=input
    for x in range(len(self.RNNs)):
        rnnoutputs['dilayer%d'%x]=self.DIs[x](rnnoutputs['outlayer%d'%(x-1)])
        rnnoutputs['outlayer%d'%x]= self.RNNs[x](rnnoutputs['dilayer%d'%x])
        rnnoutputs['outlayer%d' % x] = dropout_overtime(rnnoutputs['outlayer%d' % x], self.drop,self.training)
    temp=rnnoutputs['outlayer%d'%(len(self.RNNs)-1)][-1]
    output = self.classifier(temp)
    return output
</code></pre>
<p>Blockquote<br>
here input_size I pass is 128,when I change input_size to 97250,RuntimeError: shape ‘[-1, 97250]’ is invalid for input of size 96450 this error message shows up<br>
Please help!!!Urgently</p>

ptrblck: <p>As the error message shows, the <code>view</code> operation is invalid for the currently used input shape.<br>
Based on the reported shapes, it seems the input to the model has a shape of <code>[96450]</code> while you are trying to create a view of <code>[-1, 97250]</code> which requires more elements than the tensor contains.</p>
hello I want some help to solve the problem of the program below

chao2022: <p>83 / 5,000</p>
<h2><a name="translation-results-1" class="anchor" href="#translation-results-1"></a>Translation results</h2>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c874ee19c2ba28df19f10aac1a0484ea4d3f2d95.png" alt="Capture d’écran 2022-05-01 à 17.56.29" data-base62-sha1="sBk0Y5yswkt13p52enAga4Dm2MJ" width="426" height="460"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d0c8245952112bb86a9be9833eb5b7c4da5ad7c.png" data-download-href="https://discuss.pytorch.org/uploads/default/6d0c8245952112bb86a9be9833eb5b7c4da5ad7c" title="Capture d’écran 2022-05-01 à 17.57.01"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d0c8245952112bb86a9be9833eb5b7c4da5ad7c_2_690x257.png" alt="Capture d’écran 2022-05-01 à 17.57.01" data-base62-sha1="fyGQlWyf5qZs0IHdPJjiajL6uWo" width="690" height="257" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d0c8245952112bb86a9be9833eb5b7c4da5ad7c_2_690x257.png, https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d0c8245952112bb86a9be9833eb5b7c4da5ad7c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d0c8245952112bb86a9be9833eb5b7c4da5ad7c.png 2x" data-dominant-color="1F2D31"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d’écran 2022-05-01 à 17.57.01</span><span class="informations">706×263 42 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Based on the error message it seems that <code>plt.plot(X1, Y, 'b')</code> fails as the two input arrays have a different shape. Could you check the shape of both arrays and explain how <code>plot</code> should visualize them?</p>

chao2022: <p>I’m new to python, I don’t know how to do it.</p>

ptrblck: <p>You can take a look at the <a href="https://matplotlib.org/3.5.0/api/_as_gen/matplotlib.pyplot.plot.html"><code>plt.plot</code> docs</a> to check a few examples on how to use this method and what the expected inputs are. Once you are more familiar with the plotting mechanism, check the shapes of your current inputs via <code>print(X1.shape)</code> etc. and make sure they match and are indeed representing what you want to plot.</p>
L2 distance in pixel space

pietz: <p>I’m trying to create a loss function that measures the euclidean distance of points (not based on coordinates but) based on activated pixels in a 2D map. So, for example:</p>
<pre><code class="lang-auto">a = torch.zeros(1, 1, 5, 5) # (B,C,H,W)
b = torch.zeros(1, 1, 5, 5) # (B,C,H,W)
a[0,0,1,1] = 1.0
b[0,0,3,4] = 1.0
loss(a, b)
# Expected Output: tensor([3.60555])
</code></pre>
<p>I can easily do this calculation with a combination of <code>nonzero()</code> and <code>pow(2).sum().sqrt()</code>. However, I don’t think that I can backpropagate anything, right?</p>
<p>How could I set up this loss function to train a network that minimizes the l2 loss between activated pixels to encourage predictions that overlap with the ground truth binary mask?</p>

Andrei_Cristea: <p>There will always be exactly two pixels that are activated?</p>

pietz: <p>No, but I can “calculate” the assignment in pairs that im interested in.</p>

Andrei_Cristea: <p>Deleted, better answer below.</p>

Andrei_Cristea: <p>How about converting your pixel representation into coordinates as a first step, and then taking the distance. I believe this should work:</p>
<pre><code class="lang-auto">a = torch.zeros(1, 1, 5, 5) # (B,C,H,W)
b = torch.zeros(1, 1, 5, 5) # (B,C,H,W)

a[0, 0, 1, 1] = 1.
b[0, 0, 3, 4] = 1

x_a = (torch.arange(5).float() @ a).sum()
y_a = (torch.arange(5).float() @ a.permute([0, 1, 3, 2])).sum()

x_b = (torch.arange(5).float() @ b).sum()
y_b = (torch.arange(5).float() @ b.permute([0, 1, 3, 2])).sum()

((x_b - x_a).pow(2) + (y_b - y_a).pow(2)).sqrt()

Output:
tensor(3.6056)
</code></pre>

pietz: <p>thank you. this looks good!</p>
Is layer_activation (register_forward_hook) the same as gradient?

Roos_Kraaijveld: <p>I was wondering if the intermediate layer output initialised by register_forward_hook is the same as the gradient of the intermediate output wrt the image?</p>

ptrblck: <p>No, the forward activations are not the gradients unless I’m misunderstanding your question. <a href="https://cs231n.github.io/optimization-2/">This tutorial</a> could be useful as it shows which values are used during the forward and backward pass.</p>

Roos_Kraaijveld: <p>Thanks!</p>
<p>My end goal is to calculate the gradient of layer activations/intermediate layer outputs wrt the input image. However, because of a next step I know that the end result needs to be the same size of my layer activations/intermediate layer outputs array. With the following code I seem to be getting an end result that is the same size as my input image. Hence the confusion with gradients and layer activations. This is the code that I am using:</p>
<pre><code class="lang-auto">
 model.eval()
    tcav = {}
    for ind, (img, label) in enumerate(loader):
        img.requires_grad=True
        img = img.to(device, dtype=torch.float)
        output = model(img)
        
        layer_activation = activation[L] #activation array of layer L
        loss = -torch.mean(layer_activation)
        loss.backward(retain_graph=True)
        
        gradients = tuple_of_tensors_to_tensor(torch.autograd.grad(loss, img))[0]
        grads = normalize(gradients.cpu().detach().numpy().ravel())
        tcav[ind] = layer_activation#grads

</code></pre>
<p>Do you perhaps know what I need to do?</p>
RuntimeError: Expected p_in >= 0 && p_in <= 1 to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)

insung3511: <pre><code class="lang-python">train_dataloader = DataLoader(X_train + y_train,
                              batch_size=BATCH_SIZE,
                              shuffle=True)

test_dataloader = DataLoader(X_test + y_test,
                             batch_size=BATCH_SIZE,
                             shuffle=True)

for _, (data) in enumerate(train_dataloader):
    data = Variable(data.view(-1, BATCH_SIZE))
    print((data))
    sample_data = data.bernoulli()
</code></pre>
<p>I don’t know why doesn’t it works but I got error with create sample_data variable. and errors like this.</p>
<pre><code class="lang-auto">RuntimeError: Expected p_in &gt;= 0 &amp;&amp; p_in &lt;= 1 to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
</code></pre>
<p>Please help me. thank you.</p>

insung3511: <p>Oh, I solved it. I reference this docs. <a href="https://jovian.ai/krrish3398/01-tensor-operations" class="inline-onebox" rel="noopener nofollow ugc">krrish3398/01-tensor-operations - Jovian</a><br>
Sorry guys <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=12" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:" loading="lazy" width="20" height="20"></p>
NaN training loss

promach: <p>Why am I having <a href="https://github.com/buttercutter/gdas/blob/c1ea1779e3e3c2a1aa52df3071256d56f9ed2e03/gdas.py#L50" rel="noopener nofollow ugc">NaN for training loss</a>, <code>ltrain</code> if I change the variable <code>NUM_OF_CELLS</code> from 8 to 16 ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/86e4ab943eb04030238c2f1df82048fda819a5dd.png" data-download-href="https://discuss.pytorch.org/uploads/default/86e4ab943eb04030238c2f1df82048fda819a5dd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86e4ab943eb04030238c2f1df82048fda819a5dd_2_466x500.png" alt="image" data-base62-sha1="jfjU5RhVKy9aL1D4ILc4igxYm3r" width="466" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86e4ab943eb04030238c2f1df82048fda819a5dd_2_466x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86e4ab943eb04030238c2f1df82048fda819a5dd_2_699x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86e4ab943eb04030238c2f1df82048fda819a5dd_2_932x1000.png 2x" data-dominant-color="363738"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">980×1050 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

promach: <p>Problem solved with <a href="https://github.com/buttercutter/gdas/commit/b7f53c1d7d5c51597f1b8ed06aa3a7d329a4ff0d" rel="noopener nofollow ugc">some weight regularization</a></p>
Run two models parallel more than one GPU?

Z_Huang: <p>The following is my code example, but don’t know how to make that happen. I tried to look for torch multiprocessing, but no luck.</p>
<pre><code class="lang-auto">device = 'cuda:0'
net1 = nn.DataParallel(net1, device_ids=[0, 1, 2])
net1 = net1.to(device)
net2 = nn.DataParallel(net2, device_ids=[0, 1, 2])
net2 = net2.to(device)

for epoch in range(start_epoch, 20):   
        train_model(epoch, net1, train_data1, test_data1, optimizer1, scheduler1)
        train_model(epoch, net2, train_data2, test_data2, optimizer2, scheduler2)
</code></pre>
<p>Any help?</p>

Dazitu616: <p>You warp <code>net1</code> with <code>nn.DataParallel</code> but then you reassign it with <code>net1 = wide_net.to(device)</code>? Since <code>device</code> is just one GPU, I guess that’s why <code>net1</code> (and <code>net2</code>) only run on one GPU</p>

Z_Huang: <p>I got a typo, wide_net is net1. I just fixed my post now.</p>

Z_Huang: <p>Should I only need the following code?</p>
<pre><code class="lang-auto">net1 = nn.DataParallel(net1, device_ids=[0, 1, 2])
net2 = nn.DataParallel(net2, device_ids=[0, 1, 2])
</code></pre>
<p>The following code is unnecessary?</p>
<pre><code class="lang-auto">net1 = net1.to(device)
net2 = net2.to(device)
</code></pre>
<p>When I do the following code</p>
<pre><code class="lang-auto">device = 'cuda:0'
net1 = nn.DataParallel(net1, device_ids=[0, 1, 2])
net1 = net1.to(device)
net2 = nn.DataParallel(net2, device_ids=[0, 1, 2])
net2 = net2.to(device)
</code></pre>
<p>They run on multiple GPUs, but not parallel.</p>
<p>I am following this tutorial through: <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Optional: Data Parallelism — PyTorch Tutorials 1.11.0+cu102 documentation</a></p>

Dazitu616: <p>I think your code should work… Usually I just do</p>
<pre><code class="lang-auto">net1 = net1.cuda()
net1 = nn.DataParallel(net1, device_ids=[0, 1, 2])
</code></pre>
<p>What do you mean by parallel? If they run on multiple GPUs, that means it splits the batch data and run a subset of them on each GPU. I think this is exactly <code>parallel</code> in PyTorch means. Data parallel in PyTorch is not the multiprocessing behavior we have in python</p>

Z_Huang: <p>I mean multiple GPUs and multiple models.</p>
When is torch_tensor.to(device) necessary?

robcamp-code: <p>In a lot of tutorials online, you will see the line <code>device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')</code>. Throughout any given deep learning model, there are several examples of doing <code>tensor_a.to(device)</code> and <code>tensor.detatch()</code>. When is it necessary to add a tensor to the cpu and when is it necessary to detach a tensor from the cpu/gpu, and are there any resources where I can read on this topic.</p>

ptrblck: <p>The <code>tensor = tensor.to(device)</code> operation is used to move this tensor to the corresponding <code>device</code> and is needed e.g. if you want to execute the operations on <code>tensor</code> on this <code>device</code> (e.g. the GPU).<br>
<code>tensor.detach()</code> is used to cut the computation graph and make sure that Autograd won’t backpropagate to the previous operations. This is used e.g. if you want to store a specific tensor without its entire computation graph (in e.g. loss stats), in GAN training (e.g. the generator’s output is detached during the training of the discriminator with the real labels).</p>
Autocast - mixed precision training

base-y: <p>I was going through the amp <a href="https://debuggercafe.com/automatic-mixed-precision-training-for-deep-learning-using-pytorch/" rel="noopener nofollow ugc">docs</a> and saw that autocast was applied only during the forward phase and not in backward phase.</p>
<p>Can someone shed some light on why fp16 is not used during backward phase?</p>

ptrblck: <p>The cast operations are differentiable and will thus also be applied in the backward pass. You would however only need to add the <code>autocast</code> usage in the forward pass (unless you want to define a custom <code>backward</code> method).</p>
Implementing DIoU, calculate grad explicitly

unofficial-Jona: <p>Hi,</p>
<p>I’ve implemented the DIoU loss as described in <a href="https://arxiv.org/pdf/1911.08287.pdf" rel="noopener nofollow ugc">this paper</a>. Unfortunately I’m getting the error <code>grad can be implicitly created only for scalar outputs</code>.</p>
<p>Following <a href="https://discuss.pytorch.org/t/loss-backward-raises-error-grad-can-be-implicitly-created-only-for-scalar-outputs/12152">this thread</a> I suppose this is because my returned loss has the shape (B x Loss) (so one loss for each element in Batch). My loss function returns the following <code>1 - self.bbox_iou(y, y_hat) + self.bbox_diou_penalty(y, y_hat)</code>.</p>
<p>I could use something like torch.mean() or torch.sum() to solve this problem. However, this would presumable result in bad predictions. From my rudimentary understanding I guess there is a way to calculate the grad explicitly, using the <code>requires_grad</code> Flag, but how exactly do I do this?</p>
<p>Thanks</p>

ptrblck: <p>If you are reducing the loss and call <code>.backward()</code> on it implicitly a <code>torch.ones(1)</code> gradient will be passed as the <code>gradient</code> argument to <code>backward</code>. Since you are dealing with multiple elements in your loss, you could use e.g. <code>loss.backward(gradient=torch.ones_like(loss))</code> instead.</p>
IntermediateLayerGetter vs create_feature_extractor vs hooks vs timm to extract features

Miguel_Campos: <p>Hello community. I have a lot of doubts of what’s the best way to achieve feature from specific layers of a CNN model.</p>
<p>Checking the  vision documentation, repository and others autors, basically we could say that  <strong>IntermediateLayerGetter</strong>,<strong>feature_extractor.create_feature_extractor()</strong>,   <strong>torch.Tensor.register_hook</strong> or <strong>timm</strong> are the usual way.</p>
<p>Checking the official implementation of vision repository, models with FPN, either create_feature_extractor or torch.Tensor.register_hook are used but IntermediateLayerGetter. So I don’t know why create_feature_extractor exists? …</p>
<p>In fact  I have experimented some problems using create_feature_extractor that are solved if I use timm extraction method (to use with normalizing flow)…</p>
<p>What’s the optimal/best or recommendation in your experience to extract features?.</p>
<p>Thanks</p>
Back propogation through 2 different networks

Sumukha: <p>I have a network rotation_netA and another network netC. I am using rotation_netA to obtain an affine grid (F.affine_grid) which samples the image to give a transformed image. I then feed this image to another network ie., netC.</p>
<p>I want to tune the parameters of  rotation_netA by using the loss from the netC. How is it possible to backpropogate through netC into rotation_netA?</p>
<pre><code class="lang-auto">class rotation_netA(nn.Module):
  def __init__(self):
    super(rotation_netA, self).__init__()
    self.nz=6
    
    
    self.mean = torch.tensor(0)
    self.std = torch.tensor(0)

    self.fc_loc = nn.Sequential(
            nn.Linear(self.nz, 8),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(8, 6  ),
            nn.ReLU(),
            nn.Dropout(0.2),
        )
    self.fc1=nn.Linear(6,4)


  def get_affine_matrix(self,noise):

     identitymatrix = torch.eye(2, 3)
     
     identitymatrix = identitymatrix.unsqueeze(0)
     
     identitymatrix = identitymatrix.repeat(noise.shape[0], 1, 1)
     #print(identitymatrix)
     theta = self.fc_loc(noise)
     theta = self.fc1(theta)


     affinematrix = identitymatrix
     affinematrix[:, 0, 0] = theta[:, 0]
     affinematrix[:, 0, 1] = theta[:, 1]
     affinematrix[:, 1, 0] = theta[:, 2]
     affinematrix[:, 1, 1] = theta[:, 3]
     print(affinematrix)
     return affinematrix


  def forward(self,images):
    if self.mean.device != images.device:
            self.mean = self.mean.to(images.device)
            self.std = self.std.to(images.device)
    bs = images.shape[0]
    self.uniform = Uniform(low=-torch.ones(bs, self.nz), high=torch.ones(bs, self.nz))
   
    noise = self.uniform.rsample()
    
    # get transformation matrix
    
    affinematrix = self.get_affine_matrix(noise)

  
    # compute transformation grid
    grid = F.affine_grid(affinematrix, images.size(), align_corners=True)
    
    # apply transformation
    x = F.grid_sample(images, grid, align_corners=True)
   
    return x


class netC(nn.Module):
  def __init__(self):
    super(netC, self).__init__()


    self.network=nn.Sequential(
            nn.Linear(100, 40),
            nn.ReLU(),
            
            nn.Linear(40, 4 ),
            nn.ReLU(),
            nn.Linear(4,1)
    )


  def forward(self,x):
    y=self.network(x)
    return y




</code></pre>

Matias_Vasquez: <p>Have you tried using your module <code>rot_netA</code> inside of <code>netC</code>?</p>
<pre><code class="lang-python">class netC(nn.Module):
  def __init__(self):
    super(netC, self).__init__()
    self.rot_netA = rotation_netA()    # ←
    self.network=nn.Sequential(
            nn.Linear(100, 40),
            nn.ReLU(),
            
            nn.Linear(40, 4 ),
            nn.ReLU(),
            nn.Linear(4,1)
    )

  def forward(self,x):
    x = self.rot_netA(x)    # ←
    y=self.network(x)
    return y
</code></pre>

Sumukha: <p>Thanks a ton <a class="mention" href="/u/matias_vasquez">@Matias_Vasquez</a> , your suggestion worked out well.</p>
Parallelism not working on ResNet50

neuhart: <p>I am trying to use multiple GPUs while training the ResNet50 on the CIFAR-10 dataset. The code runs on one GPU but doesnt work when Im integrating <code>net = torch.nn.DataParallel(net)</code> . No error raised. When trying to debug I found that evaluating the model on an input (loss = loss_fn(net(x), y)) seems to be the point where the program hangs. Any suggestions? Many thanks in advance!</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose([
            transforms.RandomResizedCrop(224),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                            download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=10,
                                                  shuffle=True, num_workers=0)

if __name__ == "__main__":
    data = trainloader

    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    net = torchvision.models.resnet50()
    if torch.cuda.device_count() &gt; 1:
        print("Let's use", torch.cuda.device_count(), "GPUs!")
        net = torch.nn.DataParallel(net)
    net.to(device)
    loss_fn = torch.nn.CrossEntropyLoss(reduction="mean")

    optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)

    # Train ResNet50 model
    net.train()
    for epoch in range(1, 1 + 1):  # only one epoch for debugging
        train_loss = 0.0
        for x, y in trainloader:
            x, y = x.to(device), y.to(device)
            optimizer.zero_grad()
            loss = loss_fn(net(x), y)
            loss.backward()
            optimizer.step()
            train_loss += loss.item()
</code></pre>
Bounding Box coordinate format

chandlerbing65nm: <p>I’m confused by how bounding boxes are defined.</p>
<p>in detection bbox, a coordinate is formatted as <code>((xmin, ymin), (xmax,ymax))</code>:</p>
<p><code>(xmin, ymin)</code> — indicates top-left corner of bbox.<br>
<code>(xmax,ymax)</code> — indicates bottom-right corner of bbox.</p>
<p>But from plane of a sample box:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2acc4a7877741c5f8add9dfb1c1ebb6ef8f71c82.png" alt="image" data-base62-sha1="66BKbL9D9baxZqmUnioo3QXhKTg" width="318" height="243"></p>
<p>Coordinate of the box is<code> ((2, 3), (4,1))</code>, from this the format should be:</p>
<p><code>((xmin, ymax), (xmax,ymin))</code></p>
<p>What is the logic in the detection bbox format of <code>((xmin, ymin), (xmax,ymax))</code>?</p>

ksmdanl: <p>That is true in the context of standard mathematical expression.</p>
<p>However, image processing packages such as PIL or cv2 determine the top left corner to be the origin. That means the top left corner would be (x,y)=(0,0). As it gets further right-down, it travels to the positive direction. Hence the given coordinate.</p>
Implementing a Fully Connected Layer using nn.Conv2d vs nn.Linear

BharathC: <p>It is possible to implement a fully connected layer either using nn.Linear or by using nn.Conv with the kernel_size equal to the input size. The following piece of code demonstrates that we get identical results using both approaches. Is there any advantage to using one approach over the other? By advantage, I mean number of parameters, memory use, speed etc?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

# Define input
inp = 1000*torch.rand(1,10,3,1)

# Define fc layer using nn.Linear and initialize weights to 1 and biases to 0
fc_lin = nn.Linear(30,4) # 10x3 = 30 input channels
nn.init.ones_(fc_lin.weight)
nn.init.zeros_(fc_lin.bias)
fc_lin.eval();

# Define fc layer using nn.Conv and set kernel_size equal to input dimensions and initialize weights to 1 and biases to 0
fc_conv = nn.Conv2d(10,4,kernel_size=(3,1))
nn.init.ones_(fc_conv.weight)
nn.init.zeros_(fc_conv.bias)
fc_conv.eval();

# Run input through fc linear layers and add dummy dimensions
out_lin = fc_lin(torch.flatten(inp,1))[:,:,None,None]

# Run input through fc conv layer
out_conv = fc_conv(inp)

# Assert outputs are equal
print(torch.equal(out_lin, out_conv))
</code></pre>

eqy: <p>The answer to this will depend on your particular hardware setup and the software libraries available; some conv implementations will dispatch to matmul in which case performance might be similar. However, my guess is that most libraries for convolution e.g., cuDNN will not optimize for this use case as it is somewhat unusual whereas it would be a more typical use case for libraries like cuBLAS.</p>
<p>TLDR; it will ultimately depend on your setup and the shapes, so try them both if you are curious</p>

Dazitu616: <p>Another difference is that, if you want to apply BatchNorm1d to the FC output, using Conv1d is much more convenient than using Linear. This is because Conv1d has output shape [B, C, N] which is the input shape for BN1d, while Linear has output shape [B, N, C]. This is why in many point cloud deep learning papers people use Conv1d to implement FC (e.g. <a href="https://github.com/fxia22/pointnet.pytorch/blob/f0c2430b0b1529e3f76fb5d6cd6ca14be763d975/pointnet/model.py#L91-L93" rel="noopener nofollow ugc">PointNet</a>)</p>
Padding in BERT embedding

hardik_arora: <p>Suppose i have a bert embedding of (32,100,768) and i want to PAD, to make it (32,120,768).<br>
Should i PAD it with torch.zero(1,20,768) ? Where all weights are zero.</p>
<p>I know it can be initially padded in input ids. But i want to know how can i PAD the generated embeddings.</p>
<p>Thank you for any advise in this direction.</p>

AbdulsalamBande: <p>I think you should pad your data to sequence length of 120 before the embedding layer.</p>
Loading DP models in non-DP scenarios?

Qwwq: <p>This is regarding saving and loading of models that are wrapped with DataParallel. Let’s say I wrap a model with DataParallel during training and save the checkpoints at some intervals. After that when I load the saved checkpoints in some other project that does not use DP, I get the error that the state_dict cannot be loaded because there is a mismatch between the saved and the existing state_dicts’ keys. Apparently there is a <code>module.</code> prefix that prohibits the use of the save checkpoint. This is because DP wraps the model by the aforementioned <code>module</code>. Moving the model to cpu before saving also does not resolve this issue. The state_dict still contains the <code>module</code> wrapper. How can I resolve this issue? Ideally I would love to have the model saved without the <code>module.</code> wrapper.</p>

ptrblck: <p>You could save the <code>state_dict</code> via <code>torch.save(model.module.state_dict(), path)</code> as described <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-torch-nn-dataparallel-models">here</a>.</p>
Creating a modifiable MLP of MLPs

ryoukai: <p>Hello,</p>
<p>I’ve created a “BatchMLP” custom nn module <code>class BatchMLP(nn.Module):</code> that takes in a list of MLPs, all of which have the same architecture but different parameters for vectorized inference, and constructs a single new mlp out of them. So the first layer of this BatchMLP object is all the first layers of each mlp in the list of mlps, the second layer of BatchMLP is all the second layers if each mlp in the list of mlps, etc.</p>
<p>During training, I want to be able to update some of the layers of this BatchMLP with new mlps. So for example, if I have a BatchMLP object made out of 100 mlps, and I want to replace the ith one, then I would need to do <code>batch_mlp_object.layers[0].weight[index_i] = new_mlp.layers[0].weight, batch_mlp_object.layers[1].weight[index_i] = new_mlp.layers[1].weight, ...</code></p>
<p>I would like to try to do this without copying the parameters from the new mlp into the BatchMLP object, if possible. So for example, if I store all the mlps inside this object as <code>self.mlps: List[nn.Module] = list_of_mlps</code>, and I used this to construct an <code>nn.Sequential</code> for the BatchMLP object, then updating <code>self.mlps[index_i] = new_mlp</code> would automatically update the sequential layers because they both point to the same tensors in memory.</p>
<p>Is it possible to do this and if so, how?</p>
<p>Thanks!</p>
Smooth-L1 loss equation differentiable?

chandlerbing65nm: <p>The equation for <code>Smooth-L1 loss</code> is stated as:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fea67f579df9793638f32ab1aad78651a0d4f726.png" alt="image" data-base62-sha1="AkJZMUhBgg41dfROuCAHYCobLqS" width="460" height="69"></p>
<p>To implement this equation in <code>PyTorch</code>, we need to use <code>torch.where()</code> which is non-differentiable.</p>
<pre><code class="lang-auto">    diff = torch.abs(pred - target)
    loss = torch.where(diff &lt; beta, 0.5 * diff * diff / beta, diff - 0.5 * beta)
</code></pre>
<p>Why do we use <code>torch.where()</code> for <code>Smooth-L1 loss</code> if it is non-differentiable?</p>

Matias_Vasquez: <p>Hi,</p>
<p>you are correct that <code>torch.where()</code> would make it non-differentiable. However, there might be other ways to achieve this equation while using differentiable operations.</p>
<p>Here is just an example of how it might be done. The actual code is not open source, so I have no idea how they do it, but this is just to show that there are other ways.</p>
<pre><code class="lang-python">target = torch.zeros(16)
pred = torch.arange(0, 2, step=0.125, requires_grad=True)
beta = 1

diff = torch.abs(pred - target)

higher = diff - 0.5 * beta
lower = 0.5 * diff * diff / beta

loss = torch.empty_like(higher)
loss[higher &gt;= beta] = higher[higher &gt;= beta]
loss[higher &lt;  beta] = lower[higher &lt; beta]

print(loss)
</code></pre>
<pre><code class="lang-python"># Output
tensor([0.0000, 0.0078, 0.0312, 0.0703, 0.1250, 0.1953, 0.2812, 0.3828, 0.5000,
        0.6328, 0.7812, 0.9453, 1.0000, 1.1250, 1.2500, 1.3750],
       grad_fn=&lt;IndexPutBackward0&gt;)
</code></pre>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
Fail to run `torch.linalg.svd` because error 11

Maxwell_Albert: <p>I am runing <code>torch.svd_lowrank</code> on cpu and find a error. It shows below.</p>
<pre><code class="lang-auto">torch._C._LinAlgError: linalg.svd: (Batch element 18): The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated singular values (error code: 11).
ok
</code></pre>

Maxwell_Albert: <p>It is because my matircs has nan, solved.</p>
Colab Session Crashes on importing torch_geometric.data

Harsh_Choudhary: <p>My colab session always crashes while tring to import torch_geometric.data module. For reference, the code I am writing is as follows:</p>
<p>import torch</p>
<p>def format_pytorch_version(version):</p>
<p>return version.split(‘+’)[0]</p>
<p>TORCH_version = torch.<strong>version</strong></p>
<p>TORCH = format_pytorch_version(TORCH_version)</p>
<p>def format_cuda_version(version):</p>
<p>return ‘cu’ + version.replace(‘.’, ‘’)</p>
<p>CUDA_version = torch.version.cuda</p>
<p>CUDA = format_cuda_version(CUDA_version)</p>
<p>!pip install torch-scatter     -f <a href="https://pytorch-geometric.com/whl/torch-%7BTORCH%7D+%7BCUDA%7D.html" rel="noopener nofollow ugc">https://pytorch-geometric.com/whl/torch-{TORCH}+{CUDA}.html</a></p>
<p>!pip install torch-sparse      -f <a href="https://pytorch-geometric.com/whl/torch-%7BTORCH%7D+%7BCUDA%7D.html" rel="noopener nofollow ugc">https://pytorch-geometric.com/whl/torch-{TORCH}+{CUDA}.html</a></p>
<p>!pip install torch-cluster     -f <a href="https://pytorch-geometric.com/whl/torch-%7BTORCH%7D+%7BCUDA%7D.html" rel="noopener nofollow ugc">https://pytorch-geometric.com/whl/torch-{TORCH}+{CUDA}.html</a></p>
<p>!pip install torch-spline-conv -f <a href="https://pytorch-geometric.com/whl/torch-%7BTORCH%7D+%7BCUDA%7D.html" rel="noopener nofollow ugc">https://pytorch-geometric.com/whl/torch-{TORCH}+{CUDA}.html</a></p>
<p>!pip install torch-geometric</p>
<p>!pip install torch_geometric</p>
<blockquote>
<blockquote>
<p>from torch_geometric.data import Data &lt;&lt; When the execution reaches this line, the colab session crashes, I have tried switching to GPU and TPU as well, but nothing happens.</p>
</blockquote>
</blockquote>
<p>The logs shows this: WARNING:root:kernel fc8fda2d-f1ce-4808-b78f-a18a55132346 restarted</p>

diegoaichele: <aside class="quote no-group" data-username="Harsh_Choudhary" data-post="1" data-topic="150389">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/harsh_choudhary/48/42064_2.png" class="avatar"> Harsh_Choudhary:</div>
<blockquote>
<p>from torch_geometric.data import Data</p>
</blockquote>
</aside>
<p>Hi,<br>
I did a little change in your code:<br>
this:</p>
<pre><code class="lang-auto">TORCH_version = torch.**version**
</code></pre>
<p>by:</p>
<pre><code class="lang-auto">TORCH_version = torch.__version__
</code></pre>
<p>And it didn’t crash to me in Colab <a href="https://colab.research.google.com/drive/1pl9F02cFbGDCBWUhUcylF9qWj27zNMzv#scrollTo=mc1ptwP5xHwq" rel="noopener nofollow ugc">link</a></p>
<p>I hope I’ve helped</p>
Mixed precision model using more memory in inference(Didn't compare in finetuning)

avijha: <p>I’m comparing GPU memory usage of two models. One of them is trained with half precision while other is full. Even size on disk is close to half for the half precision model but it’s using more GPU memory.<br>
Size of models on disk -<br>
Half: 221743899<br>
Full: 442221694</p>
<p>BatchSize: 1536<br>
Half: max memory used: 11096.01171875<br>
Full: max_memory used: 6763.52685546875</p>
<p>Code to log:</p>
<pre><code class="lang-auto">def start_timer():
    global start_time
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.reset_max_memory_allocated()
    torch.cuda.synchronize()
    start_time = time.time()

def end_timer():
    torch.cuda.synchronize()
    end_time = time.time()
    global start_time, total_time
    total_time += (end_time - start_time)
    print(f"max memory used: {torch.cuda.max_memory_allocated()/(1024**2)}")
</code></pre>
<p>Only difference in inference code is -</p>
<pre><code class="lang-auto">if model_type=='half':
                with torch.autocast('cuda'):
                    output = model(input_ids=input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask)
            else:
                output = model(input_ids=input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask)
</code></pre>
<p>Is it expected behavior, if yes then what operation in autocast is using this memory?</p>

ptrblck: <p>Could you explain the issue a bit more, please?<br>
Your report:</p>
<pre><code class="lang-auto">BatchSize: 1536
Half: max memory used: 6763.52685546875
Full: max_memory used: 11096.01171875
</code></pre>
<p>doesn’t fit the title claiming that mixed-precision is using more memory.</p>

avijha: <p>Aaah my bad!<br>
I pasted it with wrong labels, I’ve edited it now. It still is the half that’s using more memory.</p>