akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to use once Cycle LR properly

talhaanwarch: <p>I have 288 images, and i am using batch size of 4. So I think my steps_per_epoch is 288//4.</p>
<p>I have plotted LR as follow</p>
<pre><code class="lang-auto">import matplotlib.pyplot as plt
opt = torch.optim.Adam(model.parameters(), lr=1e-4,weight_decay=1e-4)
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer,pct_start=0.33, max_lr=1e-3,steps_per_epoch=288//4, epochs=10,last_epoch =-1)
lrs = []
for i in range(10):
    optimizer.step()
    lrs.append(optimizer.param_groups[0]["lr"])
    scheduler.step()
plt.plot(lrs)
</code></pre>
<p>I am getting as follow<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8dd9613a4fd3c019ce6e977afb51d95dc2b7cbb9.png" data-download-href="https://discuss.pytorch.org/uploads/default/8dd9613a4fd3c019ce6e977afb51d95dc2b7cbb9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8dd9613a4fd3c019ce6e977afb51d95dc2b7cbb9_2_690x416.png" alt="image" data-base62-sha1="keR3yDDbNuHdjD8Y3NWBZYmZR0J" width="690" height="416" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8dd9613a4fd3c019ce6e977afb51d95dc2b7cbb9_2_690x416.png, https://discuss.pytorch.org/uploads/default/original/3X/8/d/8dd9613a4fd3c019ce6e977afb51d95dc2b7cbb9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/d/8dd9613a4fd3c019ce6e977afb51d95dc2b7cbb9.png 2x" data-dominant-color="FBFBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">700×423 25.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
but this graph is not showing the lowering of the learning rate</p>

ptrblck: <p>You could see the changing behavior by iterating the scheduler for 720 steps, since you’ve specified <code>steps_per_epoch=72</code> and <code>epochs=10</code>.<br>
From the docs:</p>
<blockquote>
<p>epochs (int): The number of epochs to train for. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided.</p>
</blockquote>
Test a newly trained model, which weights/biases set does the mode use?

Z_Huang: <p>I have a silly question. For example, I trained my model with 100 epochs. At epoch 88, the accuracy rate has reached the highest (92%) level. The 100th epoch only produced an 89% accuracy rate.</p>
<p>Once the train is finished with the 100 epochs (train/validation at the same time), the next step is to test the model. I was wondering which epoch weights/biases do the model use? Is it the 88th (92%) model or the 100th (89%) model?</p>
<p>When I save the model to my local device, which weights/biases set does the model used to store the model file locally?</p>

ptrblck: <aside class="quote no-group" data-username="Z_Huang" data-post="1" data-topic="128487">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/z_huang/48/8201_2.png" class="avatar"> Z_Huang:</div>
<blockquote>
<p>Once the train is finished with the 100 epochs (train/validation at the same time), the next step is to test the model. I was wondering which epoch weights/biases do the model use? Is it the 88th (92%) model or the 100th (89%) model?</p>
</blockquote>
</aside>
<p>Assuming you are calculating the accuracy on the validation dataset, you could use early stopping and store the model giving the best validation accuracy.</p>
<aside class="quote no-group" data-username="Z_Huang" data-post="1" data-topic="128487">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/z_huang/48/8201_2.png" class="avatar"> Z_Huang:</div>
<blockquote>
<p>When I save the model to my local device, which weights/biases set does the model used to store the model file locally?</p>
</blockquote>
</aside>
<p>Whatever the current <code>state_dict</code> returns, i.e. if you’ve continued training the model to 100 epochs and store the <code>model.state_dict()</code> afterwards, the current parameters trained for 100 epoch are stored.</p>
Grad_input_mask layer norm

Trinayan_Baruah: <p>Hi,</p>
<p>What is the grad input mask that is passed in the layer norm kernel call(<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/layer_norm_kernel.cu#L501" class="inline-onebox" rel="noopener nofollow ugc">pytorch/layer_norm_kernel.cu at master · pytorch/pytorch · GitHub</a>). I notice it invokes a bunch of kernels like LayerNormBackward whenever it is set. Can someone tell me what it actually means and when it is used/not used?.</p>
<p>Thanks.</p>

ptrblck: <p>Based on the linked code the mask defines, if specific gradients (<code>dgamma</code> and <code>dbeta</code>) should be initialized and passed to the gradient calculation. It should be used to skip the initialization of these tensors if the gradients are not needed.</p>
Detach hidden state of model

el_youssfi_azeddine: <p>Hello,<br>
I’m working on model to resolve physical PDEs, but during training, the time is getting incresing from an apoch to an other contantly, I don’t understand why, the model is quiet unusual (not classical one) it’s built using pytorch-geometric more specifically ARMA layer,<br>
I read in some discussions here on the forum that we can delete hidden states and that reduce memory usage, but I don’t know how to apply that in my case since my model is composed from two ARMA layers and forward function that implement PDEs.<br>
thanks</p>

ptrblck: <p>Your described issue is often caused by an increasing computation graph, which would not only increase the computation time but also the memory usage.<br>
This could happen, if you are keeping the computation graph alive by using <code>backward(retain_graph=True)</code>, which won’t delete the computation graph after the backward pass.<br>
Often this issue is caused by reusing the same <code>hidden</code> tensors without detaching them. While this could fit your use case, you would have to check “how far” into the past iterations the gradients should be computed.</p>
Using Learning Rate scheduler with Trainer

m_cik: <p>Hello,</p>
<p>I’m currently using Trainer for training models. But I would like to control learning rate over whole training. Can I use lr_scheduler with trainer? I can’t find any information how to use both</p>

ptrblck: <p>Could you give more information about the <code>Trainer</code> and where this class is coming from?<br>
Assuming you are using a higher-level API, I guess its documentation would point to the usage of a learning rate scheduler. If you’ve written the <code>Trainer</code> yourself, could you post its code, please?</p>
Test vs training AUROC metric

arya47: <p>I’m training a model and only after 7 epochs i get a training AUC of 0.98 and cross entropy of 0.16 but my testing auc is 0.55. After training a few more epochs my testing auc dosent show any improvement.</p>
<p>Cant figure out why my model would overfit the training set so early.<br>
What changes should i make to my data or model?</p>
Double Siamese Network?

sergioleno: <p>Hi guys…im working in a “different” kind of network, and i’m pretty new in pytorch.<br>
My idea is to have 2 different image pairs, each one processed by one network. As output i want the depth map of each image pair, and then the mean betwen the two resulting maps…but i have no idea about how to do that. Im starting with DrivingStereo Dataset, did some code but im struggling in the basics, like loading the data</p>
<p>here is my code so far…<br>
<a href="https://colab.research.google.com/drive/107f8365tHXOZDPp6z6O7QxZF-oVCRShn?usp=sharing" rel="noopener nofollow ugc">Colab Notebook</a></p>
<p>I got this diagram from internet. Ignoring the dimensions, it represents what i have in mind.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/497b2776023e7918751e802fa1f371fa190d72c0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/497b2776023e7918751e802fa1f371fa190d72c0" title="Idea"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/497b2776023e7918751e802fa1f371fa190d72c0_2_623x500.jpeg" alt="Idea" data-base62-sha1="au2I8t0DM5ck2jYhsKGTqqn3YnC" width="623" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/497b2776023e7918751e802fa1f371fa190d72c0_2_623x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/497b2776023e7918751e802fa1f371fa190d72c0_2_934x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/497b2776023e7918751e802fa1f371fa190d72c0_2_1246x1000.jpeg 2x" data-dominant-color="E3EBED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Idea</span><span class="informations">1320×1059 142 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And this the data folder structure.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/a/faf6cdb21b41d0c3203f87fa4303fddd3fcfa361.jpeg" alt="Data Structure" data-base62-sha1="zO88MzWpHfIJseqKvz3Szopggq5" width="237" height="193"></p>
<p>When i load the dataset, it came as one “array” with 11104 images…thats why i use the “interval=2776” (i know it’s not how it’s done), because the dataset have left camera, right camera, disparity map and depth map.</p>
<p>the dataset can be found here.<br>
<a href="https://drivingstereo-dataset.github.io/" rel="noopener nofollow ugc">DrivingStereoDataset</a></p>
Class_correct[label] += correct[i].item() IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number

alirazaee: <p>######<span class="hashtag">#load</span> the saved model for testing the trained network######<br>
model.load_state_dict(torch.load(‘model_FER_CAN.pt’))</p>
<pre><code>   # initialize lists to monitor test loss and accuracy
</code></pre>
<p>test_loss = 0.0<br>
class_correct = list(0. for i in range(len(classes)))<br>
class_total = list(0. for i in range(len(classes)))</p>
<p>model.eval() # prep model for evaluation</p>
<p>for data, target in test_loader:<br>
# move tensors to GPU if CUDA is available<br>
if train_on_gpu:<br>
data, target = data.cuda(), target.cuda()<br>
# forward pass: compute predicted outputs by passing inputs to the model<br>
output = model(data)<br>
# calculate the loss<br>
loss = criterion(output, target)<br>
# update test loss<br>
test_loss += loss.item()*data.size(0)<br>
# convert output probabilities to predicted class<br>
_, pred = torch.max(output, 1)<br>
# compare predictions to true label<br>
correct = np.squeeze(pred.eq(target.data.view_as(pred)))</p>
<pre><code>###calculate test accuracy for each object class
print(len(target))
for i in range(len(target)):
    label = target.data[i]
    class_correct[label] += correct[i].item()
    class_total[label] += 1
</code></pre>
<h1>calculate and print avg test loss</h1>
<p>test_loss = test_loss/len(test_loader.sampler)<br>
print(‘Test Loss: {:.6f}\n’.format(test_loss))</p>
<p>for i in range(len(classes)):<br>
if class_total[i] &gt; 0:<br>
print(‘Test Accuracy of %5s: %2d%% (%2d/%2d)’ % (<br>
str(i), 100 * class_correct[i] / class_total[i],<br>
np.sum(class_correct[i]), np.sum(class_total[i])))<br>
else:<br>
print(‘Test Accuracy of %5s: N/A (no training examples)’ % (classes[i]))</p>
<p>print(’\nTest Accuracy (Overall): %2d%% (%2d/%2d)’ % (<br>
100. * np.sum(class_correct) / np.sum(class_total),<br>
np.sum(class_correct), np.sum(class_total)))</p>
<p>error is ==========================================================<br>
class_correct[label] += correct[i].item()<br>
IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number</p>

ptrblck: <p><code>correct</code> seems to be a 0-dim tensor, which you cannot index. This error might be raised, as you are calling <code>np.squeeze</code> to create <code>correct</code>.</p>

111130: <p>Hello, how to solve this problem</p>

ptrblck: <p>The author of the question didn’t follow up so I’m unsure, what the root cause in the posted script was.<br>
However, since you are running into the same issue, check the shapes of all involved tensors and make sure you are not removing dimensions you would like to index later.</p>
Updating elements in-place in one tensor given specific locations at another tensor

MrPositron: <p>Hi all!<br>
I am stuck at one problem. Let say that that I have tensors A and B. Now, I want to update elements in A at locations given in tensor B.<br>
For example, let say there are two tensors A and B with size 3 x 6, and 3 x 2.</p>
<pre><code class="lang-auto">A = [
    [1, 2, 3, 4, 5, 6],
    [2, 3, 4, 5, 6, 1],
    [3, 4, 5, 6, 1, 2],
]
B = [
    [0, 1],
    [0, 3],
    [1, 4],
]
</code></pre>
<p>I wanna do something like this:</p>
<pre><code class="lang-auto">A[B] += 100 # I want the operation to be vectorized.
</code></pre>
<p>So that I will get the following result:</p>
<pre><code class="lang-auto">A = [
    [101, 102, 3, 4, 5, 6],
    [102, 3, 4, 105, 6, 1],
    [3, 104, 5, 6, 101, 2],
]
</code></pre>

KFrank: <p>Hi Nauryzbay!</p>
<aside class="quote no-group" data-username="MrPositron" data-post="1" data-topic="128711" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrpositron/48/8671_2.png" class="avatar"> MrPositron:</div>
<blockquote>
<p>So that I will get the following result:</p>
<pre><code class="lang-auto">A = [
    [101, 102, 3, 4, 5, 6],
    [102, 3, 4, 105, 6, 1],
    [3, 104, 5, 6, 101, 2],
]
</code></pre>
</blockquote>
</aside>
<p>You may use <a href="https://pytorch.org/docs/1.9.0/generated/torch.Tensor.scatter_add_.html" rel="noopener nofollow ugc">scatter_add_()</a> (with pytorch tensors) for this:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; A = torch.tensor ([
...     [1, 2, 3, 4, 5, 6],
...     [2, 3, 4, 5, 6, 1],
...     [3, 4, 5, 6, 1, 2],
... ])
&gt;&gt;&gt; B = torch.tensor ([
...     [0, 1],
...     [0, 3],
...     [1, 4],
... ])
&gt;&gt;&gt; A.scatter_add_ (1, B, torch.tensor ([100]).expand (B.shape))
tensor([[101, 102,   3,   4,   5,   6],
        [102,   3,   4, 105,   6,   1],
        [  3, 104,   5,   6, 101,   2]])
&gt;&gt;&gt; A
tensor([[101, 102,   3,   4,   5,   6],
        [102,   3,   4, 105,   6,   1],
        [  3, 104,   5,   6, 101,   2]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Torch.normal issues error for negative sigma

kirtyvedula: <p>I found this problem recently - it was working fine earlier. When I give</p>
<pre><code class="lang-auto">a = torch.distributions.Normal(0,-1)
</code></pre>
<p>it results in</p>
<blockquote>
<p>ValueError: The parameter scale has invalid values</p>
</blockquote>
<p>This has been raised on Github following some <a href="https://github.com/pytorch/pytorch/issues/49998" rel="noopener nofollow ugc">discussions</a> but it still does not seem to work. Any ideas regarding this error?</p>

KFrank: <p>Hi Kirty!</p>
<aside class="quote no-group" data-username="kirtyvedula" data-post="1" data-topic="128667" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kirtyvedula/48/16795_2.png" class="avatar"> kirtyvedula:</div>
<blockquote>
<p>it was working fine earlier.</p>
</blockquote>
</aside>
<p>Well, I wouldn’t say it was “working fine earlier.”  Rather, <code>Normal</code><br>
was failing to complain about an invalid negative value for <code>scale</code><br>
(sigma) in some situations.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">a = torch.distributions.Normal(0,-1)
</code></pre>
<p>it results in</p>
<blockquote>
<p>ValueError: The parameter scale has invalid values</p>
</blockquote>
</blockquote>
</aside>
<p>Speaking from memory, pytorch used to complain about negative<br>
<code>scale</code> in some, but not all situations, but the error checking has<br>
been tightened up in more recent versions.</p>
<p>Also, from memory, when instantiated with negative <code>scale</code>, <code>Normal</code><br>
behaved (more or less) as if it had been instantiated as:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.distributions.Normal (0, abs (scale))
</code></pre>
<p>You could conceivably use <code>abs()</code> as a work-around to recover the<br>
old behavior, but be cautious about sweeping the likely error under<br>
the rug – you should be figuring out where your negative <code>scale</code> is<br>
coming from and whether it really makes mathematical sense.</p>
<p>Best.</p>
<p>K. Frank</p>
How to multiply each element of a vector row-wise to matrix

samarendra109: <p>I have a 1D tensor of shape [15] and a 2D tensor of shape [15,6].<br>
I want to multiply each member of the vector to the corresponding row in the matrix, i.e. v[0]*m[0,:], v[1]*m[1,:] … v[14]*m[14,:].  So the output is of shape [15,6].</p>
<p>Is there a way to do it in pytorch without looping through the vector and the matrix.</p>

samarendra109: <p>Got the answer,<br>
result = v[:,None]*m</p>
How to run a function only in the slowest process in torch.distributed?

yxchng: <p>Suppose I have 8 GPUs and each running some operations that produce different outputs. After aggregating, I would like to modify a variable self.x with this aggregated results. From what I understand if I write <code>self.x = aggregate</code>, all 8 processes will run this line once, which means in total 8 times. Is that right? If it is, how do I control this to be run only in the slowest process, so that it is only run once.</p>
<p>*It probably doesn’t affect the result, but it just feels “ugly” to me. I am not sure if it is normal for this kind of code to be written when writing distributed code. Hopefully someone can advise.</p>

seungjun: <p>You can synchronize the processes and then perform the variable update on your desired rank process.<br>
Then you will need to distribute the aggregated value to the other ranks.</p>
<pre><code class="lang-python">import torch
import torch.distributed as dist

RANK = 0 #  your target rank
...
dist.barrier() # synchronize (wait for the slowest process)
if dist.get_rank() == RANK:
    self.x = aggregate()
</code></pre>

yxchng: <p>how to distribute to other ranks? i presume all ranks will see the same self.x? or will all process duplicate and keep their own version of self variables?</p>

seungjun: <p>As all processes are independent from each other without communications, <code>self.x</code> should be explicitly handled for all processes with different ranks to have the same value.<br>
You can refer to the collective functions in pytorch <a href="https://pytorch.org/docs/stable/distributed.html" rel="noopener nofollow ugc">distributed docs</a>.</p>
<p>You may choose to <code>broadcast</code> or <code>all_reduce</code>, etc.</p>

yxchng: <p>Ok. A side question.  Do you know if <code>all_gather</code> gathers tensors in the same order? I need to all gather 2 tensors and the they have to be all gathered in the same order so that item i in the first list can be matched to item i in the second list.</p>

seungjun: <p>I haven’t used it but I expect so.<br>
You can try it out and check. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How can I get higher order gradient on GRUCell?

hzhwcmhf: <p>I am working on a network need regularize the gradient, so I must get a second derivative.</p>
<p>But I got an error of</p>
<blockquote>
<p>RuntimeError: the derivative for _cudnn_rnn_backward is not implemented</p>
</blockquote>
<p>I minimize my code to reproduce the error</p>
<pre><code class="lang-auto">cell = nn.GRUCell(10, 10).cuda()
parameters = list(cell.parameters())
x = torch.rand(1, 10).cuda()
y = torch.rand(1, 10).cuda()
incoming = cell(x, torch.zeros(1, 10).cuda())
incoming = cell(y, incoming)
loss = torch.sum(incoming)
grad_all = grad(loss, parameters, retain_graph=True, create_graph=True, only_inputs=True)
print(grad_all[0].requires_grad)
loss2 = torch.sum(torch.cat([v.view(-1) for v in grad_all]))
loss2.backward()
</code></pre>
<p>and come up with another error</p>
<blockquote>
<p>RuntimeError: trying to differentiate twice a function that was markedwith <span class="mention">@once_differentiable</span></p>
</blockquote>
<p>So is there any workaround for me to get the second order gradient? (I’m on pytorch0.4.1)</p>

tom: <p>I’d probably implement a gru cell from nn.Linear.</p>
<p>Best regards</p>
<p>Thomas</p>

hzhwcmhf: <p>Thanks a lot. It works.</p>
<p>So it’s the cuda version of GRUCell marked with once_differentiable. But I hope there is a more precise error, and a simpler way to use the GRUCell rather than implement it myself when I want to get the second order gradient.</p>

tom: <p>I’m all for it and implementing wonderful things for the various RNNs is on my “things I’d like to do when I get to it list”, but I can’t just promise when that will be…</p>
<p>Best regards</p>
<p>Thomas</p>

ramprs: <p><a class="mention" href="/u/hzhwcmhf">@hzhwcmhf</a> , I am stuck with the exact same problem. Would you mind sharing your GRU/LSTM cell implementation with nn.Linear()? Thanks in advance</p>

hzhwcmhf: <p>I think this may help. <a href="https://github.com/emadRad/lstm-gru-pytorch/blob/master/lstm_gru.ipynb" rel="nofollow noopener">https://github.com/emadRad/lstm-gru-pytorch/blob/master/lstm_gru.ipynb</a></p>

ramprs: <p>Thanks <a class="mention" href="/u/hzhwcmhf">@hzhwcmhf</a>. That was very helpful.</p>

Maghoumi: <p>Just letting everyone know that following Thomas’s suggestion in another thread, I whipped up a JIT-based implementation a while back that works much faster than using <code>nn.Linear</code>.</p>
<p>It’s available here: <a href="https://github.com/Maghoumi/JitGRU" rel="nofollow noopener">https://github.com/Maghoumi/JitGRU</a></p>

Tessa_van_der_Heiden: <p>Maybe this is better for you:</p>
<pre><code class="lang-auto">cell = torch.nn.GRU(10, 10)
x = torch.rand(1, 10)
incoming, _ = cell(x.view(1, 1, 10), torch.zeros(1, 1, 10))
loss = torch.sum(incoming)
grad_all=torch.autograd.grad(loss, cell.parameters(), create_graph=True)
loss2 = torch.sum(torch.cat([v.view(-1) for v in grad_all]))
grad2=torch.autograd.grad(loss2, cell.parameters(), create_graph=True)
</code></pre>
Can I use torch.nn.L1Loss to compute L1 Norm between two tensors in the middle of y forward pass?

seyeeet: <p>I wonder if I can use <code>torch.nn.L1Loss</code> as L1 function, in other words</p>
<pre><code class="lang-auto">criterion = torch.nn.L1Loss(reduction=none)
similarity = criterion(x,y)
model(similarity)
</code></pre>
<p>where both <code>x,y</code> have <code>reqires_grad=True</code>, I am not sure if it will automatically make <code>y</code> to have <code>requires_grad=False</code> or not?</p>

KFrank: <p>Hi Seyeeet!</p>
<aside class="quote no-group" data-username="seyeeet" data-post="1" data-topic="128693" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/seyeeet/48/21931_2.png" class="avatar"> seyeeet:</div>
<blockquote>
<p>I am not sure if it will automatically make <code>y</code> to have <code>requires_grad=False</code> or not?</p>
</blockquote>
</aside>
<p>Yes, this will work fine.  <code>L1Loss</code> will propagate gradients back<br>
through both of its arguments without issue.</p>
<p>As an aside, stylistically, I would probably write this as</p>
<pre data-code-wrap="python"><code class="lang-python">model ((x - y).abs())
</code></pre>
<p>To my eye, that’s a little more readable (but it does the same thing).</p>
<p>Best.</p>
<p>K. Frank</p>
Order of the list returned by torch.distributed.all_gather()?

cane95: <p>Hi, I was wondering what is the order of the list returned by <code>torch.distributed.all_gather</code>.<br>
Is the tensor in position <code>i</code> coming from the <code>torch.distributed.get_rank() = i</code>? Thanks!</p>

pritamdamania87: <p>Yes, the ith tensor in the list comes from rank i.</p>

yxchng: <p>is it stated anywhere in the doc? how can i be sure of this? I need to all gather 2 lists and the they have to be all gathered in the same order so that item i in the first list can be matched to item i in the second list.</p>
Resnet: batchnorm2D with in_place=False

James_W: <p>I am looking to modify a pretrained resnet with a non inplace version of batchnorm2D since this is causing me problems when I run in distributed mode (runtimerror saying that grad computation has not been possible due to inplace operation)</p>
<p><code>Error detected in CudnnBatchNormBackward</code></p>
<p>I instantiate the model the vanilla way, no magic:</p>
<p><code>model=torchvision.modlels.resnet152(pretrained=True)</code></p>
<p>Is there a known solution to this problem?</p>

ptrblck: <p>Batchnorm layers don’t have an <code>inplace</code> argument, so could you post an executable code snippet which reproduces this issue for further debugging as well as the output of <code>python -m torch.utils.collect_env</code>?</p>

James_W: <p>thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> I have managed to solve this by setting <code>broadcast_buffers=False</code> It turns out that having it as True is an inplace operation.</p>
<p>but now, I have a massive memory leak issue which I have described here:</p><aside class="quote quote-modified" data-post="1" data-topic="128881">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/james_w/48/14698_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/massive-memory-leak-when-using-distributed/128881">Massive memory leak when using distributed</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    I have a distributed code base I am trying to work with, but, with every epoch I see that my CPU memory increases almost linearly eventually running into OOM on a large 128GB machine :(( 
Without distributed, the code runs fine with no such issues. The isue is exactly describe here: <a href="https://github.com/pytorch/pytorch/issues/13246" class="inline-onebox" rel="noopener nofollow ugc">CPU memory gradually leaks when num_workers &gt; 0 in the DataLoader · Issue #13246 · pytorch/pytorch · GitHub</a> 
I do use num_workers=16 
but the solution posted there, using pyarrow, does not solve my issue - I still ha…
  </blockquote>
</aside>

<p>:((</p>
Why does NumPy (and PyTorch) allow direct multiplicaiton of two row vectors?

oat: <p>The following code does work for direct multiplication btw two row vectors of the same shape. But, it’s conceptually wrong.</p>
<p>Dot product can only be conducted for a row vector and a column vector in the same length, as implemented in the np.dot(), torch.mm(), and torch.matmul() functions.</p>
<p>So, I wonder why NumPy and PyTorch allow this in the software design in the first place which could lead to confusion for beginners.</p>
<pre><code class="lang-auto">import torch

features = torch.randn((1, 5))
weights = torch.randn_like(features)
bias = torch.randn((1, 1))

output_sum = activation(torch.sum(features * weights) + bias)
# output_sum = activation((features * weights).sum() + bias)
</code></pre>
<p>Output:<br>
<code>tensor([[0.1595]])</code></p>

eqy: <p>I think it’s impossible to fully distill the intent of a given numerical libraries interface, since it’s debatable whether NumPy or PyTorch established these conventions or if they were borrowed from predecessors like MATLAB, BLAS, etc…</p>
<p>However, in this case, what is being done is elementwise multiplication (as specified by the * or <code>mul</code> operator). This is usually considered to be distinct from dot product or matrix multiplication.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.randn(1, 5)
&gt;&gt;&gt; b = torch.randn(1, 5)
&gt;&gt;&gt; a * b
tensor([[-0.0809,  0.8302, -0.3525, -0.1582, -0.6963]])
&gt;&gt;&gt; a.mul(b)
tensor([[-0.0809,  0.8302, -0.3525, -0.1582, -0.6963]])
&gt;&gt;&gt; a.dot(b)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: 1D tensors expected, but got 2D and 2D tensors
&gt;&gt;&gt;
</code></pre>

oat: <p>Thanks for the reply.</p>
<p>Yes, it’s doing element-wise multiplication,  and that’s why a sum function is needed to obtain the same output as that from a dot product.</p>
Gradient not present for backpropagation

111119: <p>Hi, I’m struggling to make my deep learning network for A2C model.<br>
My networks can’t be updated bcz of error displayed bottom image.<br>
I can’t understand what is the cause of this problem.<br>
Does anyone know about this problem in detail?<br>
(I already check my Critic-network gradient check by torch.autograd.gradcheck. it returns ‘True’.)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/3/1321e32051ac3310e5b51f6317923c1e65cfa00a.png" data-download-href="https://discuss.pytorch.org/uploads/default/1321e32051ac3310e5b51f6317923c1e65cfa00a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/3/1321e32051ac3310e5b51f6317923c1e65cfa00a.png" alt="image" data-base62-sha1="2JfFQwIATYRb0499kDv8lyeNGBk" width="690" height="243" data-dominant-color="333032"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">879×310 41.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is my network</p>
<pre><code class="lang-auto">class Actor_network(nn.Module):
    """Actor"""
    '''model = nn.Sequential()
    model.add_module('fc1', nn.Linear(STATE_DIM, NODES))
    model.add_module('relu', nn.ReLU())
    model.add_module('fc2', nn.Linear(NODES, 2))
    model.add_module('soft', nn.Softmax(dim=0))
    return model'''
    def __init__(self):
        super(Actor_network, self).__init__()
        self.fc1 = nn.Linear(STATE_DIM, NODES)
        self.fc2 = nn.Linear(NODES, 2)

    def Forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.softmax(self.fc2(x), dim=0)
        return x


class Critic_network(nn.Module):
    """Critic"""
    '''model = nn.Sequential()
    model.add_module('fc1', nn.Linear(STATE_DIM, NODES))
    model.add_module('relu1', nn.ReLU())
    model.add_module('fc2', nn.Linear(NODES, 1))
    return model'''
    def __init__(self):
        super(Critic_network, self).__init__()
        self.fc1 = nn.Linear(STATE_DIM, NODES)
        self.fc2 = nn.Linear(NODES, 1)

    def Forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x
</code></pre>

ptrblck: <p>This error is raised, if you are trying to call <code>backward()</code> on a tensor, which isn’t attached to the computation graph and you can verify it by checking its <code>.grad_fn</code> attribute which would return <code>None</code>.<br>
This might happen, if you’ve detached the tensors manually (e.g. via calling <code>detach()</code> directly, using a 3rd party library such as numpy etc.) or if you’ve used a non-differentiable operation.<br>
In both cases you could check the <code>.grad_fn</code> attribute of the intermediate tensors and narrow down where the tensor is being detached.</p>

111119: <p>I solved it.<br>
Its cause is the code “with torch.no_grad()” at open_source<br>
Thank you!</p>
Tensor size in ResNet trainting

Nil_MSh: <p>Hello! I’m training a ResNet and i get this error:</p>
<blockquote>
<pre><code>out += residual
</code></pre>
<p>RuntimeError: The size of tensor a (16) must match the size of tensor b (32) at non-singleton dimension 3</p>
</blockquote>
<p>this is my code:</p>
<pre><code class="lang-auto">class ResidualBlock(nn.Module):
    expansion = 1
    def __init__(self, in_channels, out_channels, stride=1, downsample=None):
        super(ResidualBlock, self).__init__()
        self.conv1 = conv3x3(in_channels, out_channels, stride)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = conv3x3(out_channels, out_channels)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.downsample = downsample
        self.shortcut = nn.Sequential()
        if stride != 1 or in_channels != self.expansion * out_channels:
            self.shortcut = nn.Sequential(conv3x3(in_channels, out_channels, stride),
                                          nn.BatchNorm2d(self.expansion * out_channels))

    def forward(self, x):
        residual = x
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)
        out = self.conv2(out)
        out = self.bn2(out)
        if self.downsample:
            residual = self.downsample(x)
        out += residual
        out = self.relu(out)
        return out

</code></pre>
<p>I don’t understand what’s wrong.</p>

ptrblck: <p>Based on the error message it seems that the spatial size of the intermediate activation changed compared to the <code>residual</code>, so you might either want to use a downsampling step or change the architecture. I would also recommend to take a look at the <a href="https://github.com/pytorch/vision/blob/3d9d264198e2167030dc04bf453aeca47daa37db/torchvision/models/resnet.py"><code>torchvision</code> implementation</a> in case you get stuck.</p>
Create a GT disparity map

Mathieu_Grosso: <p>Hello,<br>
I have built a model using spatial pyramid attention module. the model is supposed to learn a disparity map, for now i have used as the ground truth a map with only one  value. I want to make the task more complex, thus i would like the GT to be a map of size (B,3,H,W) with different value in the map ranging from -8 to 8 (but only with integer) and I don’t know how to write this matrix with pytorch since i am relatively new with this package. thanks<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Could you explain a bit where you are stuck at the moment?<br>
I guess you are unsure how to construct the map of the desired shape? If so, could you explain how each pixel would get its value?</p>
Autograd.profiler synchronization problem

Seungsu: <p>Hi<br>
I want to know some detail information about backward propagation.</p>
<p>So i use autograd.profiler and see my result trace.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9a63a0e4a6e91ffa106758d3284bf61dcd9618a.png" data-download-href="https://discuss.pytorch.org/uploads/default/e9a63a0e4a6e91ffa106758d3284bf61dcd9618a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9a63a0e4a6e91ffa106758d3284bf61dcd9618a_2_690x154.png" alt="image" data-base62-sha1="xkXoaA2VCUNF1ZIN0LP3nOcKJ6i" width="690" height="154" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9a63a0e4a6e91ffa106758d3284bf61dcd9618a_2_690x154.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9a63a0e4a6e91ffa106758d3284bf61dcd9618a_2_1035x231.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9a63a0e4a6e91ffa106758d3284bf61dcd9618a_2_1380x308.png 2x" data-dominant-color="F0F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1963×440 24.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i found that the empty space until the start of the next iteration is very large.</p>
<p>To find the cause, i checked the time for optimizer.step().</p>
<p>In profiler, optimizer step spent 5ms.<br>
But i write my code,<br>
torch.cuda.synchronize()<br>
start = time.time()<br>
optimizer.step()<br>
torch.cuda.synchronize()<br>
end = time.time()</p>
<p>the results is 100ms.</p>
<p>why profiler does not catch empty space?</p>
I'm getting a dict as a return type from dataloader

Aamir-Lee: <p>Here is my code :<br>
class PlantDataset(Dataset):<br>
def <strong>init</strong>(self,df,transform=None):<br>
self.images = df[‘image’].values<br>
self.labels = df[‘labels’].values<br>
self.transform  = transform</p>
<pre><code>def __len__(self):
    return len(self.images)
    
def __getitem__(self,idx):
    image_id = self.images[idx]
    labels = self.labels[idx]
    image_path = os.path.join(train_img_path , image_id)
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB)
    image = self.transform(image=image)
    return image,labels  
</code></pre>
<p>def get_transform(phase: str):<br>
if phase == ‘train’:<br>
return A.Compose([<br>
A.Resize(224,224),<br>
A.HorizontalFlip(p=0.5),<br>
A.ShiftScaleRotate(p=0.5),<br>
A.RandomBrightnessContrast(p=0.5),<br>
A.Normalize([0.4916,0.4498,0.400],[0.2474,0.2362,0.2322]),<br>
ToTensorV2(),<br>
])<br>
else:<br>
return A.Compose([<br>
A.Resize(224,224),<br>
A.Normalize([0.4916,0.4498,0.400],[0.2474,0.2362,0.2322]),<br>
ToTensorV2(),<br>
])</p>
<p>train = PlantDataset(df_train,transform = get_transform(‘train’))<br>
valid = PlantDataset(df_valid,transform = get_transform(‘valid’))</p>
<p>trainloader = torch.utils.data.DataLoader(train,batch_size=64,shuffle=True)<br>
validloader = torch.utils.data.DataLoader(valid,batch_size=64)<br>
images,labels = next(iter(trainloader))</p>

ptrblck: <p>I don’t know which libraries you are using and cannot reproduce the issue using <code>torchvision</code>, so could you update the code snippet with all missing libs and use random tensors so that we could try to reproduce the issue, please?<br>
PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>
How to change image channel from 3 to 0

ssharma0880: <p>I have a usecase where I want to convert an image from [600, 400, 3] to [600, 400].</p>
<p>Currently, I am saving the image into local directory and while loading I do “img = cv2.imread(img_loc, 0)”.</p>
<p>Although this converts the image channel however, I wonder if there is a better way of doing this. I want to avoid writing to disc if possible.</p>

ssharma0880: <p>NVM, I figured I can just do something like - img_new = img[:,:,0]</p>
Internal implementation of softmax gradient calculation

mustansarsaeed: <p>Hi, Can anyone please give a pointer from where I can get internal implementation? I have implemented my own backward for experimentation purpose but its gradient values goes to NaNs after some rounds.</p>
<p>Thanks.</p>

googlebot: <p>You normally would want to shift softmax inputs by a constant (max(x)), so that vector maximums are 0, for numerical stability. As to your question - aten/src/ATen/native/cpu/SoftMaxKernel.cpp</p>
Output from network - image

Gabriella_m: <p>Hi friends, want to ask you, when I want output from model image, for example 500x500 pixels. What layers should I use ? Can you give me example ?</p>
<p>I want to put image in model , and also output from model should be also image.</p>

tom: <p>A typical/highly successful architecture for image to image networks in U-Nets.</p>
<p>Best regards</p>
<p>Thomas</p>
Freeze classifier training, only train RPN for Faster R-CNN

AnhMinhTran: <p>As per the title mentioned, if I have already pretrained backbone, and I want to train only the RPN instead of the classifier using the Faster R-CNN from torchvision.</p>
<p>Is there any parameters I can pass in to the create_model function or would I stop the classifier training in my train() function?</p>
<p>I’m on mobile so olease excuse my editting</p>
<p>This is my create model function</p>
<h1>
<a name="create-your-backbone-from-timm-1" class="anchor" href="#create-your-backbone-from-timm-1"></a>Create your backbone from timm</h1>
<p>backbone = timm.create_model(<br>
“resnet50”,<br>
pretrained=True,<br>
num_classes=0,  # this is important to remove fc layers<br>
global_pool=""  # this is important to remove fc layers<br>
)</p>
<p>backbone.out_channels = backbone.feature_info[-1][“num_chs”]</p>
<p>anchor_generator = AnchorGenerator(<br>
sizes=((16, 32, 64, 128, 256),), aspect_ratios=((0.25, 0.5, 1.0, 2.0),)<br>
)<br>
roi_pooler = torchvision.ops.MultiScaleRoIAlign(<br>
featmap_names=[“0”], output_size=7, sampling_ratio=2<br>
)<br>
fastercnn_model = FasterRCNN(<br>
backbone=backbone,<br>
num_classes=1000,<br>
rpn_anchor_generator=anchor_generator,<br>
box_roi_pool=roi_pooler,<br>
)</p>
Word level transformer to character level

ShamePooh: <p>Hi,</p>
<p>I’m trying to modify a word-level transformer to a character one to do some spelling corrections. The source code is from:<a href="https://pytorch.org/tutorials/beginner/translation_transformer.html" class="inline-onebox" rel="noopener nofollow ugc">Language Translation with nn.Transformer and torchtext — PyTorch Tutorials 1.9.0+cu102 documentation</a><br>
I shared the embedding weight and projection weight. But my result was terrible, even when I put src and tgt same as gold text. The greedy decoder cannot even form a correct English word. I noticed that my val loss would not reduce after several epochs’ training.<br>
<a href="https://user-images.githubusercontent.com/49515860/128374620-69c46dc8-73b1-427e-8fca-0c465bacdc59.png" rel="noopener nofollow ugc"><img src="https://user-images.githubusercontent.com/49515860/128374620-69c46dc8-73b1-427e-8fca-0c465bacdc59.png" alt="image" width="589" height="500"></a><br>
(This is a model with much less layers and heads)<br>
I tried a lot of parameter combinations but none of them worked.<br>
May I ask is there any key modification should be done in character level transformer? (The vocabulary size of mine is 80+, including caps, not sure does this matter)</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How do I regularize the weights of neural networks?

sze_wen: <p>I have a 102 filters of size 32 x 32. How do I regularize the weights of the neural network to become {0, 1} or { -1, 1} instead of (-1,1) or (0,1) . If weights is more than 0, then w = 1. Else, w = -1 or 0. Is it simply achievable?</p>
BERT Fine Tuning error "ValueError: too many values to unpack (expected 2)" for input_ids

natgolovach: <p>Hello everyone!<br>
I have the same issue that is <a href="https://discuss.huggingface.co/t/errors-when-fine-tuning-t5/3527" class="inline-onebox" rel="noopener nofollow ugc">Errors when fine-tuning T5 - Beginners - Hugging Face Forums</a> but was not able to solve it using the solution in that topic.</p>
<p>I’m finetuning the model using BertForSequenceClassification.from_pretrained(model_name, num_labels=1)</p>
<p>Here is the dataset class:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):

    def __init__(self, df, tokenizer_name=MODEL_name, max_length=1024):
        
        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name, do_lower_case=False)
        self.seqs, self.labels = self.load_dataset(df)
        self.max_length = max_length

    def __len__(self):
        return len(self.labels)
    def load_dataset(self,df):
        seq = list(df['sequence'])
        label = list(df['score'])
        assert len(seq) == len(label)
        return seq, label
        
    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        seq = " ".join("".join(self.seqs[idx].split()))
        seq = re.sub(r"[UZOB]", "X", seq)

        seq_ids = self.tokenizer(seq, truncation=True, padding='max_length', max_length=self.max_length, return_tensors='pt').to(device)

        sample = {key: torch.tensor(val) for key, val in seq_ids.items()}
        sample['labels'] = torch.tensor(self.labels[idx])

        return sample
</code></pre>
<p>Which I then load to Dataloader for batching</p>
<pre><code class="lang-auto">train_seqs_encodings_dataset=MyDataset(df=train_dataset_clean)
 train_loader = DataLoader.DataLoader(
            train_seqs_encodings_dataset,
            batch_size=64,
            shuffle=True
        )
</code></pre>
<p>But in training:</p>
<pre><code class="lang-auto">        loss, predictions = model(b_input_ids, 
                             token_type_ids=b_token_type_ids, 
                             attention_mask=b_input_mask, 
                             labels=b_labels)
</code></pre>
<p>I got the following error:</p>
<blockquote>
<pre><code>923         elif input_ids is not None:
924             input_shape = input_ids.size()
925             batch_size, seq_length = input_shape
926         elif inputs_embeds is not None:
927             input_shape = inputs_embeds.size()[:-1]

ValueError: too many values to unpack (expected 2)
</code></pre>
</blockquote>
<p>So when I print (input_ids.size()) out of (1) batch = torch.Size([64, 1, 1024]), but I believe it expects tuple with (64,1024). The question is why it returns tuple(list) instead of just tuple?</p>
<p>Thank you for your help</p>

ge_yang: <p>same problem here. Trying to work my head around</p>
Why does a cGAN's discriminator use a linear transformation?

dnzsn: <p>Hi Everyone,</p>
<p>I have been trying to dive deeper into cGANs using pytorch and I see across implementations that the discriminator applies a linear transformation to the input data and was wondering why ?<br>
The cGAN paper mentions that D or G can be a linear or non-linear mapping function. So I am assuming <code>self.linear</code>  ensure that D is a linear mapping function, but on to exactly ?</p>
<pre><code class="lang-auto"># Discriminator Model
class Discriminator(nn.Module):
    def __init__(self, ngpu=1):
        super(Discriminator, self).__init__()
        self.label_emb = nn.Embedding(n_class, ndf*16*4)
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # input is (nc) x 64 x 64
            nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf) x 32 x 32
            nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 2),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*2) x 16 x 16
            nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 4),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*4) x 8 x 8
            nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 8),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*8) x 4 x 4
            nn.Conv2d(ndf * 8, ndf * 16, 4, 2, 1, bias=False), 
            nn.BatchNorm2d(ndf * 16),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*16) x 2 x 2
            nn.Flatten()
        )
        self.linear = nn.Sequential(
        nn.Linear(ndf*16*4*2, ndf*16),    
        nn.LeakyReLU(0.2, inplace=True),   
        nn.Linear(ndf*16, 1),
        nn.Sigmoid()    
        )

    def forward(self, input, labels):
        disc_out = self.main(input)
        linear_input = torch.cat((self.label_emb(labels), disc_out), 1)
        linear_output = self.linear(linear_input.squeeze())
        return linear_output.unsqueeze(2).unsqueeze(3)
</code></pre>