SGD - good results. ADAM - really bad - why?

Moshe_Peleg: <p>Hi,<br>
this is really wired.<br>
i am using a solution code for CIFAR10 using cnn (from Udacity deep learning , public repository).</p>
<p>when i train the network using SGD optimizer - i get accuracy of 72% (lets assume its good)<br>
but when i change to ADAM - i get 10% accuracy, and doing the train the loss dont change.</p>
<p>i have check multiple time that its the correct code, and it is!</p>
<p>do you know the reason? , as known Adam should be <em>better</em> then SGD .</p>
<p>this is the loss for Adam:<br>
Epoch: 1 	Training Loss: 2.305185 	Validation Loss: 2.303153<br>
Epoch: 2 	Training Loss: 2.304046 	Validation Loss: 2.303932<br>
Epoch: 3 	Training Loss: 2.304276 	Validation Loss: 2.304587<br>
Epoch: 4 	Training Loss: 2.304290 	Validation Loss: 2.304771<br>
Epoch: 5 	Training Loss: 2.304313 	Validation Loss: 2.304767<br>
Epoch: 6 	Training Loss: 2.304428 	Validation Loss: 2.305437<br>
Epoch: 7 	Training Loss: 2.304531 	Validation Loss: 2.303925<br>
Epoch: 8 	Training Loss: 2.304736 	Validation Loss: 2.304267</p>
<p>here is loss for Adam<br>
Epoch: 1 	Training Loss: 1.484200 	Validation Loss: 0.292755<br>
Epoch: 2 	Training Loss: 1.128465 	Validation Loss: 0.250197<br>
Epoch: 3 	Training Loss: 0.975305 	Validation Loss: 0.219079<br>
Epoch: 4 	Training Loss: 0.866838 	Validation Loss: 0.219783<br>
Epoch: 5 	Training Loss: 0.779665 	Validation Loss: 0.180415<br>
Epoch: 6 	Training Loss: 0.714491 	Validation Loss: 0.168658<br>
Epoch: 7 	Training Loss: 0.659699 	Validation Loss: 0.160974<br>
Epoch: 8 	Training Loss: 0.613168 	Validation Loss: 0.158714</p>
<p>here is a link to colab if you want: (i had to pot spaces , as new used i am not allowed to add link)</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1gwZChd4C4b7IXTr0HdSscmglqcMlosTW?usp=sharing">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20220310-060047-RC01_433765713" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1gwZChd4C4b7IXTr0HdSscmglqcMlosTW?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1gwZChd4C4b7IXTr0HdSscmglqcMlosTW?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>h t t p s : / / c o l a b . r e s e a r c h . g o o g l e . c o m / d r i v e / 1 g w Z C h d 4 C 4 b 7 I X T r 0 H d S s c m g l q c M l o s T W ? u s p = s h a r i n g</p>
<p>and here is the entire code:</p>
<p>import torch<br>
import numpy as np</p>
<h1>
<a name="check-if-cuda-is-available-1" class="anchor" href="#check-if-cuda-is-available-1"></a>check if CUDA is available</h1>
<p>train_on_gpu = torch.cuda.is_available()</p>
<p>if not train_on_gpu:<br>
print(‘CUDA is not available.  Training on CPU …’)<br>
else:<br>
print(‘CUDA is available!  Training on GPU …’)</p>
<p>from torchvision import datasets<br>
import torchvision.transforms as transforms<br>
from torch.utils.data.sampler import SubsetRandomSampler</p>
<h1>
<a name="number-of-subprocesses-to-use-for-data-loading-2" class="anchor" href="#number-of-subprocesses-to-use-for-data-loading-2"></a>number of subprocesses to use for data loading</h1>
<p>num_workers = 0</p>
<h1>
<a name="how-many-samples-per-batch-to-load-3" class="anchor" href="#how-many-samples-per-batch-to-load-3"></a>how many samples per batch to load</h1>
<p>batch_size = 20</p>
<h1>
<a name="percentage-of-training-set-to-use-as-validation-4" class="anchor" href="#percentage-of-training-set-to-use-as-validation-4"></a>percentage of training set to use as validation</h1>
<p>valid_size = 0.2</p>
<h1>
<a name="convert-data-to-a-normalized-torchfloattensor-5" class="anchor" href="#convert-data-to-a-normalized-torchfloattensor-5"></a>convert data to a normalized torch.FloatTensor</h1>
<p>transform = transforms.Compose([<br>
transforms.ToTensor(),<br>
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))<br>
])</p>
<h1>
<a name="choose-the-training-and-test-datasets-6" class="anchor" href="#choose-the-training-and-test-datasets-6"></a>choose the training and test datasets</h1>
<p>train_data = datasets.CIFAR10(‘data’, train=True,<br>
download=True, transform=transform)<br>
test_data = datasets.CIFAR10(‘data’, train=False,<br>
download=True, transform=transform)</p>
<h1>
<a name="obtain-training-indices-that-will-be-used-for-validation-7" class="anchor" href="#obtain-training-indices-that-will-be-used-for-validation-7"></a>obtain training indices that will be used for validation</h1>
<p>num_train = len(train_data)<br>
indices = list(range(num_train))<br>
np.random.shuffle(indices)<br>
split = int(np.floor(valid_size * num_train))<br>
train_idx, valid_idx = indices[split:], indices[:split]</p>
<h1>
<a name="define-samplers-for-obtaining-training-and-validation-batches-8" class="anchor" href="#define-samplers-for-obtaining-training-and-validation-batches-8"></a>define samplers for obtaining training and validation batches</h1>
<p>train_sampler = SubsetRandomSampler(train_idx)<br>
valid_sampler = SubsetRandomSampler(valid_idx)</p>
<h1>
<a name="prepare-data-loaders-combine-dataset-and-sampler-9" class="anchor" href="#prepare-data-loaders-combine-dataset-and-sampler-9"></a>prepare data loaders (combine dataset and sampler)</h1>
<p>train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size,<br>
sampler=train_sampler, num_workers=num_workers)<br>
valid_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size,<br>
sampler=valid_sampler, num_workers=num_workers)<br>
test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size,<br>
num_workers=num_workers)</p>
<h1>
<a name="specify-the-image-classes-10" class="anchor" href="#specify-the-image-classes-10"></a>specify the image classes</h1>
<p>classes = [‘airplane’, ‘automobile’, ‘bird’, ‘cat’, ‘deer’,<br>
‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’]</p>
<p>“”"### Visualize a Batch of Training Data"""</p>
<h1>
<a name="commented-out-ipython-magic-to-ensure-python-compatibility-11" class="anchor" href="#commented-out-ipython-magic-to-ensure-python-compatibility-11"></a>Commented out IPython magic to ensure Python compatibility.</h1>
<p>import matplotlib.pyplot as plt</p>
<h1>
<a name="matplotlib-inline-12" class="anchor" href="#matplotlib-inline-12"></a>%matplotlib inline</h1>
<h1>
<a name="helper-function-to-un-normalize-and-display-an-image-13" class="anchor" href="#helper-function-to-un-normalize-and-display-an-image-13"></a>helper function to un-normalize and display an image</h1>
<p>def imshow(img):<br>
img = img / 2 + 0.5  # unnormalize<br>
plt.imshow(np.transpose(img, (1, 2, 0)))  # convert from Tensor image</p>
<h1>
<a name="obtain-one-batch-of-training-images-14" class="anchor" href="#obtain-one-batch-of-training-images-14"></a>obtain one batch of training images</h1>
<p>dataiter = iter(train_loader)<br>
images, labels = dataiter.next()<br>
images = images.numpy() # convert images to numpy for display</p>
<h1>
<a name="plot-the-images-in-the-batch-along-with-the-corresponding-labels-15" class="anchor" href="#plot-the-images-in-the-batch-along-with-the-corresponding-labels-15"></a>plot the images in the batch, along with the corresponding labels</h1>
<p>fig = plt.figure(figsize=(25, 4))</p>
<h1>
<a name="display-20-images-16" class="anchor" href="#display-20-images-16"></a>display 20 images</h1>
<p>for idx in np.arange(20):<br>
ax = fig.add_subplot(2, 20/2, idx+1, xticks=[], yticks=[])<br>
imshow(images[idx])<br>
ax.set_title(classes[labels[idx]])</p>
<p>“”"### View an Image in More Detail</p>
<p>Here, we look at the normalized red, green, and blue (RGB) color channels as three separate, grayscale intensity images.<br>
“”"</p>
<p>rgb_img = np.squeeze(images[3])<br>
channels = [‘red channel’, ‘green channel’, ‘blue channel’]</p>
<p>fig = plt.figure(figsize = (36, 36))<br>
for idx in np.arange(rgb_img.shape[0]):<br>
ax = fig.add_subplot(1, 3, idx + 1)<br>
img = rgb_img[idx]<br>
ax.imshow(img, cmap=‘gray’)<br>
ax.set_title(channels[idx])<br>
width, height = img.shape<br>
thresh = img.max()/2.5<br>
for x in range(width):<br>
for y in range(height):<br>
val = round(img[x][y],2) if img[x][y] !=0 else 0<br>
ax.annotate(str(val), xy=(y,x),<br>
horizontalalignment=‘center’,<br>
verticalalignment=‘center’, size=8,<br>
color=‘white’ if img[x][y]&lt;thresh else ‘black’)</p>
<p>import torch.nn as nn<br>
import torch.nn.functional as F</p>
<h1>
<a name="define-the-cnn-architecture-17" class="anchor" href="#define-the-cnn-architecture-17"></a>define the CNN architecture</h1>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
# convolutional layer (sees 32x32x3 image tensor)<br>
self.conv1 = nn.Conv2d(3, 16, 3, padding=1)<br>
# convolutional layer (sees 16x16x16 tensor)<br>
self.conv2 = nn.Conv2d(16, 32, 3, padding=1)<br>
# convolutional layer (sees 8x8x32 tensor)<br>
self.conv3 = nn.Conv2d(32, 64, 3, padding=1)<br>
# max pooling layer<br>
self.pool = nn.MaxPool2d(2, 2)<br>
# linear layer (64 * 4 * 4 → 500)<br>
self.fc1 = nn.Linear(64 * 4 * 4, 500)<br>
# linear layer (500 → 10)<br>
self.fc2 = nn.Linear(500, 10)<br>
# dropout layer (p=0.25)<br>
self.dropout = nn.Dropout(0.25)</p>
<pre><code>def forward(self, x):
    # add sequence of convolutional and max pooling layers
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = self.pool(F.relu(self.conv3(x)))
    # flatten image input
    x = x.view(-1, 64 * 4 * 4)
    # add dropout layer
    x = self.dropout(x)
    # add 1st hidden layer, with relu activation function
    x = F.relu(self.fc1(x))
    # add dropout layer
    x = self.dropout(x)
    # add 2nd hidden layer, with relu activation function
    x = self.fc2(x)
    return x
</code></pre>
<h1>
<a name="create-a-complete-cnn-18" class="anchor" href="#create-a-complete-cnn-18"></a>create a complete CNN</h1>
<p>model = Net()<br>
print(model)</p>
<h1>
<a name="move-tensors-to-gpu-if-cuda-is-available-19" class="anchor" href="#move-tensors-to-gpu-if-cuda-is-available-19"></a>move tensors to GPU if CUDA is available</h1>
<p>if train_on_gpu:<br>
model.cuda()</p>
<p>import torch.optim as optim</p>
<h1>
<a name="specify-loss-function-categorical-cross-entropy-20" class="anchor" href="#specify-loss-function-categorical-cross-entropy-20"></a>specify loss function (categorical cross-entropy)</h1>
<p>criterion = nn.CrossEntropyLoss()</p>
<h1>
<a name="specify-optimizer-21" class="anchor" href="#specify-optimizer-21"></a>specify optimizer</h1>
<p>optimizer = optim.Adam(model.parameters(), lr=0.01)</p>
<h1>
<a name="number-of-epochs-to-train-the-model-22" class="anchor" href="#number-of-epochs-to-train-the-model-22"></a>number of epochs to train the model</h1>
<p>n_epochs = 30</p>
<p>valid_loss_min = np.Inf # track change in validation loss</p>
<p>for epoch in range(1, n_epochs+1):</p>
<pre><code># keep track of training and validation loss
train_loss = 0.0
valid_loss = 0.0

###################
# train the model #
###################
model.train()
for data, target in train_loader:
    # move tensors to GPU if CUDA is available
    if train_on_gpu:
        data, target = data.cuda(), target.cuda()
    # clear the gradients of all optimized variables
    optimizer.zero_grad()
    # forward pass: compute predicted outputs by passing inputs to the model
    output = model(data)
    # calculate the batch loss
    loss = criterion(output, target)
    # backward pass: compute gradient of the loss with respect to model parameters
    loss.backward()
    # perform a single optimization step (parameter update)
    optimizer.step()
    # update training loss
    train_loss += loss.item()*data.size(0)
    
######################    
# validate the model #
######################
model.eval()
for data, target in valid_loader:
    # move tensors to GPU if CUDA is available
    if train_on_gpu:
        data, target = data.cuda(), target.cuda()
    # forward pass: compute predicted outputs by passing inputs to the model
    output = model(data)
    # calculate the batch loss
    loss = criterion(output, target)
    # update average validation loss 
    valid_loss += loss.item()*data.size(0)

# calculate average losses
train_loss = train_loss/len(train_loader.sampler)
valid_loss = valid_loss/len(valid_loader.sampler)
    
# print training/validation statistics 
print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
    epoch, train_loss, valid_loss))

# save model if validation loss has decreased
if valid_loss &lt;= valid_loss_min:
    print('Validation loss decreased ({:.6f} --&gt; {:.6f}).  Saving model ...'.format(
    valid_loss_min,
    valid_loss))
    torch.save(model.state_dict(), 'model_cifar.pt')
    valid_loss_min = valid_loss
</code></pre>
<p>“”"###  Load the Model with the Lowest Validation Loss"""</p>
<p>model.load_state_dict(torch.load(‘model_cifar.pt’))</p>
<p>“”"—</p>
<h2>
<a name="test-the-trained-network-23" class="anchor" href="#test-the-trained-network-23"></a>Test the Trained Network</h2>
<p>Test your trained model on previously unseen data! A “good” result will be a CNN that gets around 70% (or more, try your best!) accuracy on these test images.<br>
“”"</p>
<h1>
<a name="track-test-loss-24" class="anchor" href="#track-test-loss-24"></a>track test loss</h1>
<p>test_loss = 0.0<br>
class_correct = list(0. for i in range(10))<br>
class_total = list(0. for i in range(10))</p>
<p>model.eval()</p>
<h1>
<a name="iterate-over-test-data-25" class="anchor" href="#iterate-over-test-data-25"></a>iterate over test data</h1>
<p>for data, target in test_loader:<br>
# move tensors to GPU if CUDA is available<br>
if train_on_gpu:<br>
data, target = data.cuda(), target.cuda()<br>
# forward pass: compute predicted outputs by passing inputs to the model<br>
output = model(data)<br>
# calculate the batch loss<br>
loss = criterion(output, target)<br>
# update test loss<br>
test_loss += loss.item()*data.size(0)<br>
# convert output probabilities to predicted class<br>
_, pred = torch.max(output, 1)<br>
# compare predictions to true label<br>
correct_tensor = pred.eq(target.data.view_as(pred))<br>
correct = np.squeeze(correct_tensor.numpy()) if not train_on_gpu else np.squeeze(correct_tensor.cpu().numpy())<br>
# calculate test accuracy for each object class<br>
for i in range(batch_size):<br>
label = target.data[i]<br>
class_correct[label] += correct[i].item()<br>
class_total[label] += 1</p>
<h1>
<a name="average-test-loss-26" class="anchor" href="#average-test-loss-26"></a>average test loss</h1>
<p>test_loss = test_loss/len(test_loader.dataset)<br>
print(‘Test Loss: {:.6f}\n’.format(test_loss))</p>
<p>for i in range(10):<br>
if class_total[i] &gt; 0:<br>
print(‘Test Accuracy of %5s: %2d%% (%2d/%2d)’ % (<br>
classes[i], 100 * class_correct[i] / class_total[i],<br>
np.sum(class_correct[i]), np.sum(class_total[i])))<br>
else:<br>
print(‘Test Accuracy of %5s: N/A (no training examples)’ % (classes[i]))</p>
<p>print(’\nTest Accuracy (Overall): %2d%% (%2d/%2d)’ % (<br>
100. * np.sum(class_correct) / np.sum(class_total),<br>
np.sum(class_correct), np.sum(class_total)))</p>
<p>“”"</p>

Moshe_Peleg: <p>about the question .<br>
the SECOND loss is SGD (I accidentally wrote Adam)</p>

mMagmer: <p>you should use way lower <code>lr</code> with adam to work.<br>
try <code>lr = 1e-4</code> or lower</p>
<aside class="quote no-group" data-username="Moshe_Peleg" data-post="1" data-topic="146373">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/moshe_peleg/48/44650_2.png" class="avatar"> Moshe_Peleg:</div>
<blockquote>
<p>as known Adam should be <em>better</em> then SGD</p>
</blockquote>
</aside>
<p>this is not always true.</p>

Moshe_Peleg: <p>Hi <a class="mention" href="/u/mmagmer">@mMagmer</a> ,<br>
i change the lr as you suggested and i see massive improvement. - thank you.<br>
this raise numbers of question:</p>
<ol>
<li>dose Adam optimizer usually need lr in this order of magnitude?</li>
<li>is it a thumb rule what lr i should use in each optimizer?</li>
<li>next time when i build a network and i fail in training it - how can i know if its because bad architecture or because (like in this case) i didn’t choose the right lr ? (there are so many parameters - how could i know were i was wrong?)</li>
</ol>
<p>thank you very much for your king help,<br>
regards,</p>

mMagmer: <p>If you want, I answer your questions, but I might be wrong.</p>
<ol>
<li>yes</li>
<li>I think you can play around with default <code>lr</code> setup in optim method.  i.e. for SGD it’s .1 for Adam it’s 1e-3.</li>
<li>if your model architecture does not require specific optim method, you can use SGD with momentum. It’s very well-behaved.</li>
</ol>
<p>It’s also depended on batch size and how you are averaging the loss.</p>

Moshe_Peleg: <p><a class="mention" href="/u/mmagmer">@mMagmer</a>  thank for your answers</p>
<aside class="quote no-group" data-username="mMagmer" data-post="5" data-topic="146373">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmagmer/48/43283_2.png" class="avatar"> mMagmer:</div>
<blockquote>
<p>It’s also depended on batch size and how you are averaging the loss.</p>
</blockquote>
</aside>
<p>can you please explain more about it?<br>
what should i assume when i use small/big batch size and how the loss avg affect the learning rate</p>

mMagmer: <p>By default, <code>nn.crossentropy</code> and other loss are using <code>reduction='mean'</code>, if set <code>reduction='sum'</code>, you are effectively multiplying LR by batch size.<br>
Even if you’re using ‘mean’, Some of optim methods are requited to update LR after changing batch size.</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/53033556/how-should-the-learning-rate-change-as-the-batch-size-change">
  <header class="source">

      <a href="https://stackoverflow.com/questions/53033556/how-should-the-learning-rate-change-as-the-batch-size-change" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/996260/tanmay" target="_blank" rel="noopener nofollow ugc">
    <img alt="Tanmay" src="https://www.gravatar.com/avatar/8a9df3477c611aeaf2801129c038d4f6?s=256&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/53033556/how-should-the-learning-rate-change-as-the-batch-size-change" target="_blank" rel="noopener nofollow ugc">How should the learning rate change as the batch size change?</a>
</h4>

<div class="tags">
  <strong>machine-learning, deep-learning</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/996260/tanmay" target="_blank" rel="noopener nofollow ugc">
    Tanmay
  </a>
  on <a href="https://stackoverflow.com/questions/53033556/how-should-the-learning-rate-change-as-the-batch-size-change" target="_blank" rel="noopener nofollow ugc">04:17PM - 28 Oct 18 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Moshe_Peleg: <p><a class="mention" href="/u/mmagmer">@mMagmer</a>  thank you for your answer.<br>
i will read about it more</p>