akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
ValueError: Expected input batch_size (1) to match target batch_size (40000)

Q_M: <p>i have a trainingdata(torch.size([3,3,200,200]) and i put it in my convolution network,but get error.<br>
how can i fix it?</p>
<pre><code class="lang-auto">class ConNet(nn.Module):
    def __init__(self):
        super(ConNet, self).__init__()
        self.ConNet1 = nn.Sequential(
            nn.Conv2d(
                in_channels=3,
                out_channels=24,
                kernel_size=3,
                stride=1,
                padding=1,
            ),
            nn.ReLU(),
            nn.AvgPool2d(
                kernel_size=2,
                stride=2,
            ),
        )
        self.ConNet2 =nn.Sequential(
            nn.Conv2d(24,12,3,1,0),
            nn.ReLU(),
            nn.AvgPool2d(2,2)
        )
        self.classifier=nn.Sequential(
            nn.Linear(12*49*49,256),
            nn.ReLU(),
            nn.Linear(256,128),
            nn.ReLU(),
            nn.Linear(128,40000)
        )
    def forward(self,x):
        x=self.ConNet1(x)
        x=self.ConNet2(x)
        x=x.view(x.size(0),-1)
        output=self.classifier(x)
        return output
connet=ConNet()
print(connet)
</code></pre>

RaLo4: <p>This has nothing to do with your network. You network is fine I tested it by copying your code and putting <code>torch.rand(3,3,200,200)</code> through it.<br>
This error comes from your loss function. The output of your network seems to be a tensor of size (1, 4000) while the target seem to be of size (4000).<br>
Since you said your input is batchsize 3 none of that makes sense.<br>
Did you maybe use batchsize 1 when this error occurred? If so you can make it work by just unsqueezing the target like this <code>target = torch.unsqueeze(target, 0)</code><br>
Or does it also happen with batchsize higher than 1?<br>
If so can you post your code. Specifically the bit between the output of your net and where your loss function is called.</p>

Q_M: <p>Thank you very much for answering my question. The error dose come form batchsize 1, and i use 'unsqueezing ’ that can solve error. The error doesn’t come from my network. If you were confuesd by my network, i’m very sorry for that.<br>
And thanks again.</p>
How to fine tune pretrained model with BatchNorm

WebberMao: <p>Hi<br>
I am now trying to train a new model with a self-defined classifier in vgg19_bn, I set the features part to eval() mode and requires_grad = False. After training, the loss start from 10 down to about 0.015.<br>
Now I want to fine tune the whole model, the full model was set to train() mode, but got an abnormal loss (about 2.7). This didn’t happen when I did this on vgg19 so I guess the problem is on the BatchNorm layers. Then, I tried to set each BatchNorm layer to eval() mode, the loss became normal (about 0.017).<br>
I want to ask is that right to set each BatchNorm layer to eval() mode when I want to fine tune the whole model or what is the best way to do?</p>
<p>BTW, the batch size is 16, and all the other parameter is default.</p>
Kernel and Bias Regularization in nn.Linear()

mustansarsaeed: <p>Hi, Can anyone please guide how can we add the regularizer in the <code>nn.Linear</code> module same like tensorflow. In <code>Tensorflow</code> we add regularization in the <code>Dense</code> layer as:</p>
<p><code>tf.layers.dense(inputs=pool2_flat, units=2048, name='dense1', kernel_regularizer=tf.contrib.layers.l2_regularizer(0.001), bias_regularizer=tf.contrib.layers.l2_regularizer(0.001))</code></p>
<p>Any thought how can add in nn.Linear?</p>
<p>Thank you.</p>

KFrank: <p>Hi Mustansar!</p>
<aside class="quote no-group" data-username="mustansarsaeed" data-post="1" data-topic="128679" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mustansarsaeed/48/30330_2.png" class="avatar"> mustansarsaeed:</div>
<blockquote>
<p>Hi, Can anyone please guide how can we add the regularizer in the <code>nn.Linear</code> module</p>
</blockquote>
</aside>
<p>Use the <code>weight_decay</code> constructor argument when you instantiate<br>
your optimizer.  For <code>SGD</code> (without momentum), it is the same as<br>
L2 regularization.  For <code>Adam</code>, it is not identical to L2 regularization,<br>
but some people argue that <code>Adam</code>’s <code>weight_decay</code> works better<br>
than L2 regularization.  (Conceptually, they both play the same role.)</p>
<aside class="quote no-group">
<blockquote>
<p>same like tensorflow.</p>
</blockquote>
</aside>
<p>I don’t know the details of what tensorflow does, so I can’t say how<br>
closely this might mimic the tensorflow implementation.</p>
<aside class="quote no-group">
<blockquote>
<p>In <code>Tensorflow</code> we add regularization in the <code>Dense</code> layer as:</p>
</blockquote>
</aside>
<p>If you want to use <code>weight_decay</code> / L2 regularization with just some<br>
layers, you can pass in parameter groups when you instantiate<br>
your optimizer.  Each group can contain different layers and have<br>
different values for <code>weight_decay</code>, including 0.0 (which is its default<br>
value).</p>
<p>Best.</p>
<p>K. Frank</p>
How to solve this optimization problem?

ArshadIram: <p>Hi everyone,</p>
<p>I am a beginner in PyTorch and I am trying to code one of the equations that is mentioned in a research paper.</p>
<p>This equation will solve by using a trained GAN model. Here the z is the latent space of trained gan and x is the generated image.</p>
<p>In terms of coding, how can I do it? How can I have the z of pretrained gan?</p>

KFrank: <p>Hi Iram!</p>
<aside class="quote no-group" data-username="ArshadIram" data-post="1" data-topic="128336" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arshadiram/48/32886_2.png" class="avatar"> ArshadIram:</div>
<blockquote>
<p>This equation will solve by using a trained GAN model. Here the z is the latent space of trained gan and x is the generated image.</p>
</blockquote>
</aside>
<p>The equation you posted an image of means that you are to find<br>
the values of <code>z[i]</code> that minimize <code>MSELoss (x[?], G (z))</code>, while<br>
holding the parameters that define <code>G</code> constant.</p>
<p>(I believe that the use of <code>i</code> as the index on both sides of the<br>
equation is a notational inconsistency – that’s why I used <code>?</code> as<br>
the index for <code>x</code> – but please explain if you believe otherwise.)</p>
<p>To use pytorch to do this, you would use pytorch’s autograd and<br>
gradient-descent-optimization machinery to minimize your “loss”<br>
with respect to <code>z</code>.</p>
<p>To do this you would start with <code>z</code> as a one-dimensional tensor<br>
of length <code>d</code> (shape <code>[d]</code>), and wrap it as a <code>Parameter</code>.  Initialize<br>
<code>z</code> somehow – perhaps to zero or perhaps randomly or perhaps<br>
to an initial guess, if you have one.  Let me assume that <code>G</code> is a<br>
pytorch “model,” that is, some sort of <code>Module</code>.  Go through all of <code>G</code>s<br>
<code>Parameter</code>s, and for each of them set <code>requires_grad = False</code>.<br>
(This is so that you won’t unnecessarily calculate gradients for the<br>
<code>Parameter</code>s of <code>G</code>.)  Instantiate a pytorch optimizer with <code>z</code> as its<br>
<code>Parameter</code>.  Then run an optimization loop who’s iteration is:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = torch.nn.MSELoss() (x, G (z))
opt.zero_grad()
loss.backward()
opt.step()
</code></pre>
<p>Each optimization step moves <code>z</code> in the direction that lowers the<br>
mismatch between <code>G (z)</code> and <code>x</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

ArshadIram: <p>Hi Franks,</p>
<p>The explanation of this equation is:</p>
<p>We first embed x1, x2 into latent space of G. Specifically, we optimize over the latent space to find vectors that produce inputs close to x1 and x2 in l2-norm.</p>
<p>I don’t think using of i on both sides of equation is notational inconsistency. Can you explain why you think this is inconsistent?</p>

ArshadIram: <p>Can you please provide your thoughts on it?</p>
Print(model) layers output vs forward method

arya47: <p>Why is the sequence of layers in print(model) not the same as defined in the forward method?</p>

RaLo4: <p>Because the <code>forward</code> function has no relation to <code>print(model)</code>.<br>
<code>print(model)</code> prints the models attributes defined in the <code>__init__</code> function in the order they were defined.<br>
The result will be the same no matter what you wrote in your <code>forward</code> function. It would even be the same even if your <code>forward</code> function didn’t exist.</p>
Is the weighted sum calculated by transposing the weights associated with a linear layer under the hood in a torch nn model?

oat: <p>In the simple nn module as shown below, the shape of the weights associated with fc1, i.e. <strong>W1</strong>, is <strong>(128 x 784)</strong>.</p>
<p>Assuming the mini batch size is 64, so the shape of the input X is (64, 784).</p>
<p>Does this mean that under the hood the <strong>weighted sum</strong> calculation inside fc1 is carried out as the <strong>dot product</strong> between <strong>input X (shape: 64 x 784)</strong> and <strong>the transpose of W1 (784 x 128)</strong> to obtain <strong>an output for fc1 with a shape of (64 x 128)</strong>?</p>
<pre><code class="lang-auto">from torch import nn
import torch.nn.functional as F 

class myNN(nn.Module):
    def __init__(self):
        super().__init__()
        # the first hidden layer
        self.fc1 = nn.Linear(784, 128)
        # the second hidden layer
        self.fc2 = nn.Linear(128, 64)
        # the output layer 
        self.fc3 = nn.Linear(64, 10)
        
    def forward(self, x):
        # the 1st hidden layer
        x = F.relu(self.fc1(x))
        # the 2nd hidden layer 
        x = F.relu(self.fc2(x))
        # the output layer 
        x = F.softmax(self.fc3(x), dim=1)
        
        return x

# create the nn object
model = myNN()

print('fc1 weights shape:\n', model.fc1.weight.shape)
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">fc1 weights shape:
 torch.Size([128, 784])
</code></pre>

AlphaBetaGamma96: <p>The forward method in <code>nn.Linear</code> is defined as y = xW^T + b, so yes the weight matrix is transposed in the operation. You can read more in the docs,</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=nn%20linear#torch.nn.Linear" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=nn%20linear#torch.nn.Linear</a></p>
Torch.gather from 1D array using 2D indices

Sandeep_Menon: <p>I have an nx1 tensor and an nxm tensor. I want to gather values from the nx1 tensor using the nxm tensor.<br>
For example<br>
for input tensor([1, 2, 3, 4]) and<br>
index tensor([[0, 3], [2, 1],[1, 3], [2,3]])</p>
<p>output should be<br>
tensor([[1, 4], [3, 2], [2,4], [3,4])</p>
<p>How to use torch.gather for this purpose?<br>
My following code gives error</p>
<pre><code class="lang-auto">t = torch.tensor([[1, 2, 3, 4]])
ind = torch.tensor([[0, 3], [2, 1],[1, 3], [2,3]])
torch.gather(t, 0, ind)
</code></pre>
<p><code>RuntimeError: index 2 is out of bounds for dimension 0 with size 1</code></p>
<p><strong>Edit</strong>: You can do simple indexing to achieve this output.</p>
<pre><code class="lang-auto">t[ind]
</code></pre>
<p>Is this the best way to do this. I assume this involves broadcasting the input array.</p>
Tmux session die when DataParallel is used

June30: <p>I have a server with 4 GPUs. When I train a model, I connect to the server using ssh and make a tmux session on the server and train the model on the tmux session. If I use just 1 GPU for training, there’s no problem. But if I use 4 GPUs for training using DataParallel, connection to the server is sometimes disconnected and I cannot connect to the server for 1~2 minutes. After I reconnect to the server, there’s no the tmux session. This disconnection does not appear routinely. The training works well for over 24 hours, but disconnection could appear just in 1 hour.</p>
<p>I think that the session is killed by an unknown reason because the server is not rebooted (no record of reboot).</p>
<p>I tried to connect to the server with ssh except forwarding X11, the disconnection still appears.</p>
<p>I’m using pytorch 1.4 and tmux 2.6.</p>
<p>Is there any idea?</p>

ptrblck: <p>Based on the description I think the server might be indeed rebooting e.g. due to an underpowered PSU if all GPUs are used. Once you are losing the <code>ssh</code> connection you could try to <code>ping</code> the server and should see a timeout until the node reboots.</p>