Training loss is constant since beginning in CNN

Kothari_Gaurav: <p>‘’’<br>
class Orfunction(torch.nn.Module):</p>
<pre><code>def __init__(self, D_in, H, num_classes):
    super().__init__()
    self.D_in = D_in
    self.H = H
    self.D_out = num_classes
    self.conv1 = nn.Conv2d(D_in,H,kernel_size=11, stride=4, padding=2)
    self.mpool1 = nn.MaxPool2d(kernel_size=3, stride=2)
    self.conv2 = nn.Conv2d(64, 192, kernel_size=5, padding=2)
    self.mpool2 = nn.MaxPool2d(kernel_size=3, stride=2)
    self.conv3 = nn.Conv2d(192, 384, kernel_size=3, padding=1)
    self.conv4 = nn.Conv2d(384, 256, kernel_size=3, padding=1)
    self.conv5 = nn.Conv2d(256, 256, kernel_size=3, padding=1)
    self.mpool3 =   nn.MaxPool2d(kernel_size=3, stride=2)
    self.avgpool = nn.AdaptiveAvgPool2d((6, 6))
    self.linear1 = nn.Linear(256 * 6 * 6, 4096)
    self.linear2 = nn.Linear(4096, 4096)
    self.linear3 = nn.Linear(4096, num_classes)

def forward(self, x):
    #print(self.conv1.weight)
    x = self.conv1(x)
    x = F.leaky_relu(x)
    x = self.mpool1(x)
    x = self.conv2(x)
    x = F.leaky_relu(x)
    x = self.mpool2(x)
    x = self.conv3(x)
    x = F.leaky_relu(x)
    x = self.conv4(x)
    x = F.leaky_relu(x)
    x = self.conv5(x)
    x = F.leaky_relu(x)
    x = self.mpool3(x)
    x = self.avgpool(x)
    x = x.view(x.size(0), 256 * 6 * 6)
    x = self.linear1(x)
    x = F.leaky_relu(x)
    x = self.linear2(x)
    x = F.leaky_relu(x)
    x = self.linear3(x)
    x = torch.softmax(x,1)
    #print("s",x.shape)
    #print("pobability distribution is:",x)
    return x
</code></pre>
<p>D_in = 3<br>
H =  64<br>
num_classes = 3</p>
<p>classes = [‘truck’,‘auto’,‘car’]<br>
function_network =  Orfunction(D_in,H,num_classes)</p>
<p>Device = torch.device(‘cpu’)<br>
function_network.to(Device)</p>
<p>loss_function = torch.nn.CrossEntropyLoss()</p>
<p>vls = list(function_network.parameters())</p>
<p>optimizer = optim.Adam(vls, lr=0.01, weight_decay=0.001)</p>
<p>list1 = list()<br>
path2 = ‘/home/prashant/racing_car/google-images-download/downloads/train’<br>
images = os.listdir(path2)<br>
for image in images:<br>
img = os.path.join(path2,image)<br>
list1.append(img)</p>
<p>list2 = list()<br>
for imgs in list1:<br>
img = cv2.imread(imgs)<br>
img1 = cv2.resize(img, (256, 256))<br>
list2.append(img1)</p>
<p>list3 = list()<br>
for j in list2:<br>
img1 = torch.tensor(j, dtype= torch.float,requires_grad=True)<br>
img1 = img1.permute(2,0,1)<br>
img1 = img1.unsqueeze(dim = 0)<br>
list3.append(img1)</p>
<p>batch1 = torch.stack(list3,dim = 1)<br>
batch1 = batch1.squeeze(dim = 0)</p>
<p>y = torch.tensor([0,0,0,0,0,1,1,1,1,1,2,2,2,2,2])<br>
print(y.shape)</p>
<p>epochs = 20</p>
<p>for i in range(epochs):</p>
<pre><code>print('iteration', i)

optimizer.zero_grad()
output = function_network(batch1)
print("o",output.shape)

loss = loss_function(output,y)

loss = Variable(loss,requires_grad=True)

print("loss", loss.item())

print(list(function_network.parameters())[0].grad)
init1 = list(function_network.parameters())[0].clone()

loss.backward()

optimizer.step()

init2 = list(function_network.parameters())[0].clone()

t = torch.equal(init1.data, init2.data)

print('are the same :',t)
</code></pre>
<p>"output is :<br>
loss 1.099882960319519 "<br>
are they same : True<br>
loss 1.099882960319519 "<br>
are they same : True<br>
loss 1.099882960319519 "<br>
are they same : True<br>
loss 1.099882960319519 "<br>
are they same : True<br>
loss 1.099882960319519 "<br>
are they same : True<br>
loss 1.099882960319519 "<br>
are they same : True<br>
loss 1.099882960319519 "<br>
are they same : True<br>
loss 1.099882960319519 "<br>
are they same : True<br>
‘’’</p>

vabh: <aside class="quote no-group" data-username="Kothari_Gaurav" data-post="1" data-topic="45581">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kothari_gaurav/48/12233_2.png" class="avatar"> Kothari_Gaurav:</div>
<blockquote>
<p>loss = Variable(loss,requires_grad=True)</p>
</blockquote>
</aside>
<p>I think this operation is breaking your computation graph.</p>
<p>The order of operations should be:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = loss_function(output,y)
loss.backward()
optimizer.step()
</code></pre>

ptrblck: <p>Additionally to this, you should remove the softmax at the end of your model, since <code>nn.CrossEntropyLoss</code> expects logits and internally applied <code>log_softmax</code> on them.</p>

Kothari_Gaurav: <p>Thank you so much guys for quick reply, it is working now.</p>
<p>Note -  Can you provide any link for understanding.<br>
1- How to decide which optimizer we have to use(Adam vs SGD)<br>
2-  The computation graph.<br>
3-  Which operations breaks the computation graph.<br>
4-  How can I visualise these graphs while building a CNN in PyTorch.</p>

Kothari_Gaurav: <p>Using ADAM:-<br>
loss 1.1901756525039673<br>
iteration 1<br>
loss 400.5993347167969<br>
iteration 2<br>
loss 78.54071807861328<br>
iteration 3<br>
loss 191.35740661621094<br>
iteration 4<br>
loss 153.02627563476562<br>
iteration 5<br>
loss 29.623409271240234<br>
iteration 6<br>
loss 7.654780864715576<br>
iteration 7<br>
loss 4.365676403045654<br>
iteration 8<br>
loss 24.920808792114258<br>
iteration 9<br>
loss 3.465801954269409<br>
iteration 10<br>
loss 1.6431617736816406</p>
<p>using SGD<br>
iteration 0<br>
loss 1.0905007123947144<br>
iteration 1<br>
loss 1.0449053049087524<br>
iteration 2<br>
loss 1.0095171928405762<br>
iteration 3<br>
loss 0.9810587167739868<br>
iteration 4<br>
loss 0.95912104845047<br>
iteration 5<br>
loss 0.950375497341156<br>
iteration 6<br>
loss 0.9360883235931396<br>
iteration 7<br>
loss 0.9562820196151733</p>
<p>Can you explain why in the beginning, loss is increasing so high in ADAM :</p>

vabh: <p>For 2, 3, I think this might be a good starting point, autograd mechanics: <a href="https://pytorch.org/docs/stable/notes/autograd.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/autograd.html</a></p>
<p>For 1, I do not have a good answer. Normally Adam is more forgiving towards hyperparameter choices as compared to SGD. It is useful when you want to iterate quickly on a problem (to see if it works or not)</p>
<p>For 4, I think you can use tensorboard <a href="https://pytorch.org/docs/stable/tensorboard.html?highlight=tensorboard" rel="nofollow noopener">https://pytorch.org/docs/stable/tensorboard.html?highlight=tensorboard</a><br>
or this package: <a href="https://github.com/szagoruyko/pytorchviz" rel="nofollow noopener">https://github.com/szagoruyko/pytorchviz</a></p>