Why my network loss is not decresing? please help

cshy926: <p>Im trying to train a simple model on CIFAR10 and im using the Pytorch tutorial with changing layers from VGG16. can someone please help me and tell me why my loss is constant…am i doing something wrong?</p>
<ul>
<li>
</li>
</ul>
<p>print(’==&gt; Preparing data…’)<br>
transform_train = transforms.Compose([<br>
transforms.RandomCrop(32, padding=4),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.ToTensor(),<br>
transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),<br>
])</p>
<p>transform_test = transforms.Compose([<br>
transforms.ToTensor(),<br>
transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),<br>
])</p>
<p>trainset = torchvision.datasets.CIFAR10(root=’./data’, train=True, download=True, transform=transform_train)<br>
trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, shuffle=True, num_workers=2)</p>
<p>testset = torchvision.datasets.CIFAR10(root=’./data’, train=False, download=True, transform=transform_test)<br>
testloader = torch.utils.data.DataLoader(testset, batch_size=100, shuffle=False, num_workers=2)</p>
<p>print(’==&gt; Building model…’)<br>
net = myvggnet.MyVGGNet16()</p>
<p>net = net.to(device)<br>
if device == ‘cuda’:<br>
cudnn.benchmark = True</p>
<p>criterion = nn.NLLLoss()<br>
softmax = nn.Softmax(1)</p>
<p>lr = 0.05<br>
optimizer = optim.SGD(net.parameters(), lr=lr, momentum=0.9)</p>
<p>def lr_schedule(optimizer, epoch):<br>
if epoch &gt;= 122:<br>
lr = 0.0005<br>
elif epoch &gt;= 81:<br>
lr = 0.005<br>
else:<br>
lr = 0.05<br>
for param_group in optimizer.param_groups:<br>
param_group[‘lr’] = lr</p>
<p>def train(epoch, optimizer):<br>
print(’\nEpoch: %d’ % epoch)<br>
net.train()<br>
train_loss = 0<br>
correct = 0<br>
total = 0</p>
<pre><code>for batch_idx, (inputs, targets) in enumerate(trainloader):
    inputs, targets = inputs.to(device), targets.to(device)
    optimizer.zero_grad()
    out = net(inputs)
    out_sm = softmax(out)
    
    log = torch.log(out_sm)
    
    loss = criterion(log, targets)
    
    loss.backward()
    optimizer.step()

    train_loss += loss.item()
    _, predicted = out_sm.max(1)
    total += targets.size(0)
    correct += predicted.eq(targets).sum().item()

train_loss = train_loss/(batch_idx+1)
train_acc = 100.*correct/total 

return train_loss, train_acc
</code></pre>
<p>def test(epoch, optimizer):<br>
global best_acc<br>
net.eval()<br>
test_loss = 0<br>
correct = 0<br>
total = 0<br>
with torch.no_grad():<br>
for batch_idx, (inputs, targets) in enumerate(testloader):<br>
inputs, targets = inputs.to(device), targets.to(device)<br>
out = net(inputs)<br>
out_sm = softmax(out)<br>
log = torch.log(out_sm)<br>
loss = criterion(log, targets)</p>
<pre><code>        test_loss += loss.item()
        _, predicted = out_sm.max(1)
        total += targets.size(0)
        correct += predicted.eq(targets).sum().item()

test_loss = test_loss/(batch_idx+1)
test_acc = 100.*correct/total 

return test_loss, test_acc
</code></pre>
<p>now = datetime.datetime.now</p>
<p>acc_history = torch.zeros([200,2], dtype=torch.float32, device=‘cuda’)<br>
loss_history = torch.zeros([200,2], dtype=torch.float32, device=‘cuda’)<br>
count = torch.zeros([200,2], dtype=torch.int32, device=‘cuda’)</p>
<p>for epoch in range(start_epoch, start_epoch+200):<br>
t = now()<br>
lr_schedule(optimizer, epoch)</p>
<pre><code>train_loss, train_acc = train(epoch, optimizer)
print("Train Loss : %.3f | Train Acc : %.3f | Training time: %s" % (train_loss, train_acc, now() - t))
test_loss, test_acc = test(epoch, optimizer)
print("Test Loss : %.3f, Test Acc : %.3f" % (test_loss, test_acc))
acc_history[epoch][0] = train_acc
acc_history[epoch][1] = test_acc

loss_history[epoch][0] = train_loss
loss_history[epoch][1] = test_loss
</code></pre>
<ul>
<li>
</li>
</ul>
<p>above is my code, and below is my network</p>
<ul>
<li>
</li>
</ul>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import numpy as np</p>
<p>class MyVGGNet(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(MyVGGNet, self).<strong>init</strong>()<br>
self.features = torch.nn.Sequential(<br>
<span class="hashtag">#conv1</span><br>
torch.nn.Conv2d(3,64,3,padding=1,bias=False),<br>
torch.nn.ReLU(),<br>
torch.nn.Dropout(0.3),<br>
torch.nn.Conv2d(64,64,3,padding=1,bias=False),<br>
torch.nn.ReLU(),<br>
torch.nn.AvgPool2d(2,2),<br>
<span class="hashtag">#conv2</span><br>
torch.nn.Conv2d(64,128,3,padding=1,bias=False),<br>
torch.nn.ReLU(),<br>
torch.nn.Dropout(0.4),<br>
torch.nn.Conv2d(128,128,3,padding=1,bias=False),<br>
torch.nn.ReLU(),<br>
torch.nn.AvgPool2d(2,2),<br>
<span class="hashtag">#conv3</span><br>
torch.nn.Conv2d(128,256,3,padding=1,bias=False),<br>
torch.nn.ReLU(),<br>
torch.nn.Dropout(0.4),<br>
torch.nn.Conv2d(256,256,3,padding=1,bias=False),<br>
torch.nn.ReLU(),<br>
torch.nn.Dropout(0.4),<br>
torch.nn.Conv2d(256,256,3,padding=1,bias=False),<br>
torch.nn.ReLU(),<br>
torch.nn.AvgPool2d(2,2),<br>
<span class="hashtag">#conv4</span><br>
torch.nn.Conv2d(256,512,3,padding=1,bias=False),<br>
torch.nn.ReLU(),<br>
torch.nn.Dropout(0.4),<br>
torch.nn.Conv2d(512,512,3,padding=1,bias=False),<br>
torch.nn.ReLU(),<br>
torch.nn.Dropout(0.4),<br>
torch.nn.Conv2d(512,512,3,padding=1,bias=False),<br>
torch.nn.ReLU(),<br>
torch.nn.AvgPool2d(2,2),<br>
<span class="hashtag">#conv5</span><br>
torch.nn.Conv2d(512,512,3,padding=1,bias=False),<br>
torch.nn.ReLU(),<br>
torch.nn.Dropout(0.4),<br>
torch.nn.Conv2d(512,512,3,padding=1,bias=False),<br>
torch.nn.ReLU(),<br>
torch.nn.Dropout(0.4),<br>
torch.nn.Conv2d(512,512,3,padding=1,bias=False),<br>
torch.nn.ReLU(),<br>
torch.nn.AvgPool2d(2,2)<br>
)<br>
self.classifier = torch.nn.Sequential(<br>
torch.nn.Linear(512<em>1</em>1,512),<br>
torch.nn.ReLU(),<br>
torch.nn.Dropout(0.5),<br>
torch.nn.Linear(512,512),<br>
torch.nn.ReLU(),<br>
torch.nn.Dropout(0.5),<br>
torch.nn.Linear(512,10),<br>
)</p>
<pre><code>def forward(self, x):
    output = self.features(x)
    output = output.view(output.size()[0],-1)
    output = self.classifier(output)
    
    return output
</code></pre>
<p>def MyVGGNet16():<br>
return MyVGGNet()</p>
<ul>
<li>
</li>
</ul>
<p>and after start training, output keep comes out like</p>
<ul>
<li>
</li>
</ul>
<p>Epoch: 0<br>
Train Loss : 2.304 | Train Acc : 10.070 | Training time: 0:00:16.129584<br>
Test Loss : 2.303, Test Acc : 10.000</p>
<p>Epoch: 1<br>
Train Loss : 2.304 | Train Acc : 9.890 | Training time: 0:00:16.121169<br>
Test Loss : 2.304, Test Acc : 10.000</p>
<p>Epoch: 2<br>
Train Loss : 2.303 | Train Acc : 10.288 | Training time: 0:00:16.217818<br>
Test Loss : 2.304, Test Acc : 10.000</p>
<p>Epoch: 3<br>
Train Loss : 2.304 | Train Acc : 10.028 | Training time: 0:00:16.199693<br>
Test Loss : 2.303, Test Acc : 10.000</p>
<p>Epoch: 4<br>
Train Loss : 2.304 | Train Acc : 9.660 | Training time: 0:00:16.242965<br>
Test Loss : 2.303, Test Acc : 10.000</p>
<p>…</p>
<p>loss is 2.303 and not changing<br>
I tried to find out why, and I found out my network is not working<br>
Is anyone can tell me why is this happening?<br>
please let me know</p>

G.M: <p>There r too many dropouts. Especially too many dropouts after convolution. Try removing some of the dropouts.</p>

MSA: <p>I had the same error when I moved from Keras to Pytorch. I had a Keras model with Dropout like you (p=0.4, p=0.5), the model was learning correctly and I switched to Pytorch with the same data, same model and same Dropout value, and the model wasn’t learning, the loss didn’t decrease.<br>
So I removed the dropout and that was working well.</p>
<p>I didn’t seek why there is a huge difference with Dropout.</p>

G.M: <p>I guess it’s because how they implement dropout is different. Like perhaps in Keras a 0.4 Dropout means abandoning 40% but in Pytorch it means abandoning 60%. I’m just guessing, I’m not sure whether they are different or not.</p>

cshy926: <p>I removed all the dropout layer, but it didn’t work…<br>
I’ll try writing network again<br>
thanks for your help</p>

cshy926: <p>I removed all the dropout layer, but it didn’t work<br>
I think network and forward path part is the reason<br>
I’ll write it again<br>
thanks!</p>

G.M: <p>Btw instead of using <code>a.softmax().log()</code>, u should use the “logSoftmax”.</p>