Title: Is Rectified Adam actually *better* than Adam? - PyImageSearch

URL Source: https://pyimagesearch.com/2019/10/07/is-rectified-adam-actually-better-than-adam/

Published Time: 2019-10-07T14:00:16+00:00

Markdown Content:
[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20400'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/rectified_adam_comparison_header.png)

Is the Rectified Adam (RAdam) optimizer actually _better_ than the standard Adam optimizer? According to my 24 experiments, the answer is no, typically not (but there are cases where you _do_ want to use it instead of Adam).

In Liu et al.’s 2018 paper, _On the Variance of the Adaptive Learning Rate and Beyond_, the authors claim that Rectified Adam can obtain:

*   **Better accuracy** (or at least identical accuracy when compared to Adam)
*   And in **fewer epochs** than standard Adam

The authors tested their hypothesis on three different datasets, including one NLP dataset and two computer vision datasets (ImageNet and CIFAR-10).

In each case Rectified Adam outperformed standard Adam…**but _failed_ to outperform standard Stochastic Gradient Descent (SGD)!**

The Rectified Adam optimizer has some strong theoretical justifications — but as a deep learning practitioner, you need more than just theory — **you need to see _empirical results_ applied to a _variety of datasets_.**

And perhaps more importantly, **you need to obtain a mastery level experience operating/driving the optimizer (or a small subset of optimizers) as well.**

Today is part two in our two-part series on the Rectified Adam optimizer:

1.  [_Rectified Adam (RAdam) optimizer with Keras_](https://pyimagesearch.com/2019/09/30/rectified-adam-radam-optimizer-with-keras/) (last week’s post)
2.  _Is Rectified Adam actually \*better\* than Adam_ (today’s tutorial)

If you haven’t yet, go ahead and read part one to ensure you have a good understanding of how the Rectified Adam optimizer works.

From there, read today’s post to help you understand how to design, code, and run experiments used to compare deep learning optimizers.

**To learn how to compare Rectified Adam to standard Adam, _just keep reading!_**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

In the first part of this tutorial, we’ll briefly discuss the Rectified Adam optimizer, including how it works and why it’s interesting to us as deep learning practitioners.

From there, I’ll guide you in designing and planning our set of experiments to compare Rectified Adam to Adam — **you can use this section to learn how you design your own deep learning experiments as well.**

We’ll then review the project structure for this post, including implementing our training and evaluation scripts by hand.

Finally, we’ll run our experiments, collect results, and ultimately decide **is Rectified Adam actually _better_ than Adam?**

### What is the Rectified Adam optimizer?

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20282'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/09/rectified_adam_keras_paper.png)

**Figure 1:** The Rectified Adam (RAdam) deep learning optimizer. Is it better than the standard Adam optimizer? (image source: Figure 6 from Liu et al.)

The Rectified Adam optimizer was proposed by Liu et al. in their 2019 paper, [_On the Variance of the Adaptive Learning Rate and Beyond_](https://arxiv.org/pdf/1908.03265.pdf). In their paper they discussed how their update to the Adam optimizer, called **Rectified Adam**, can:

1.  Obtain a **higher accuracy/more generalizable** deep neural network.
2.  Complete training in **fewer epochs**.

Their work had some strong theoretical justifications as well. They found that adaptive learning rate optimizers (such as Adam) both:

*   Struggle to generalize during the first few batch updates
*   Had _very_ high variance

Liu et al. studied the problem in detail and found that the issue could be rectified (hence the name, Rectified Adam) by:

1.  Applying warm up with a **low initial earning rate**.
2.  Simply **turning off the momentum term** for the first few sets of input training batches.

The authors evaluated their experiments on one NLP dataset and two image classification datasets and found that their **Rectified Adam implementation _outperformed_ standard Adam (but _neither_ optimizer outperformed standard SGD).**

We’ll be continuing Liu et al.’s experiments today and comparing Rectified Adam to standard Adam in 24 separate experiments.

For more details on how the Rectified Adam optimizer works, be sure to review my [previous blog post](https://pyimagesearch.com/2019/09/30/rectified-adam-radam-optimizer-with-keras/).

### Planning our experiments

[![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20401'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/rectified_adam_comparison_plan_experiments.jpg)

**Figure 2:** We will plan our set of experiments to evaluate the performance of the Rectified Adam (RAdam) optimizer using Keras.

To compare Adam to Rectified Adam, **we’ll be training _three_ Convolutional Neural Networks (CNNs)**, including:

1.  ResNet
2.  GoogLeNet
3.  MiniVGGNet

The implementations of these CNNs came directly from my book, [_Deep Learning for Computer Vision with Python_](https://pyimagesearch.com/deep-learning-computer-vision-python-book/).

These networks **will be trained on _four_ datasets:**

1.  MNIST
2.  Fashion MNIST
3.  CIFAR-10
4.  CIFAR-100

For each combination of dataset and CNN architecture, **we’ll apply _two_ optimizers:**

1.  Adam
2.  Rectified Adam

Taking all possible combinations, we end up with _3 x 4 x 2 = 24_ separate training experiments.

We’ll run each of these experiments individually, collect, the results, and then interpret them to determine which optimizer is indeed better.

Whenever you plan your own experiments make sure you take the time to write out the list of model architectures, optimizers, and datasets you intend on applying them to. Additionally, you may want to list the hyperparameters you believe are important and are worth tuning (i.e., learning rate, L2 weight decay strength, etc.).

Considering the 24 experiments we plan to conduct, it makes the most sense to automate the data collection phase. From there, we will be able to work on other tasks while the computation is underway (often requiring days of compute time). Upon completion of the data collection for our 24 experiments, we will then be able to sit down and analyze the plots and classification reports in order to evaluate RAdam on our CNNs, datasets, and optimizers.

### How to design your own deep learning experiments

[![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20400'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/rectified_adam_comparison_plan_lab_notebook.jpg)

**Figure 3:** Designing your own deep learning experiments, requires thought and planning. Consider your typical deep learning workflow and design your initial set of experiments such that a thorough preliminary investigation can be conducted using automation. Planning for automated evaluation now will save you time (and money) down the line.

**Typically, my experiment design workflow goes something like this:**

1.  **Select 2-3 model architectures** that I believe would work well on a particular dataset (i.e., ResNet, VGGNet, etc.).
2.  Decide if I want to **train from scratch** or perform [**transfer learning**](https://pyimagesearch.com/2019/05/20/transfer-learning-with-keras-and-deep-learning/).
3.  Use my [**learning rate finder**](https://pyimagesearch.com/2019/08/05/keras-learning-rate-finder/) to find an acceptable initial learning rate **for the SGD optimizer**.
4.  Train the model on my dataset **using SGD** and **[Keras’ standard decay schedule](https://pyimagesearch.com/2019/07/22/keras-learning-rate-schedules-and-decay/)**.
5.  Look at my results from training, select the architecture that performed best, and **start tuning my hyperparameters**, including model capacity, regularization strength, revisiting the initial learning rate, applying [Cyclical Learning Rates](https://pyimagesearch.com/2019/07/29/cyclical-learning-rates-with-keras-and-deep-learning/), and potentially exploring other optimizers.

You’ll notice that I tend to use SGD in my initial experiments instead of Adam, RMSprop, etc.

Why is that?

To answer that question you’ll need to read the _“You need to obtain mastery level experience operating these three optimizers”_ section below.

_**Note:** For more of my suggestions, tips, and best practices when designing and running your own experiments, be sure to refer to my book, [Deep Learning for Computer Vision with Python](https://pyimagesearch.com/deep-learning-computer-vision-python-book/)._

However, in the context of this tutorial, we’re attempting to _compare_ our results to the work of Liu et al.

We, therefore, need to fix the model architectures, training from scratch, learning rate, and optimizers — **our experiment design now becomes:**

1.  Train ResNet, GoogLeNet, and MiniVGGNet on MNIST, Fashion MNIST, CIFAR-10, and CIFAR-100, respectively.
2.  Train all networks from scratch.
3.  Use the initial, default learning rates for Adam/Rectified Adam (1e-3).
4.  Utilize the Adam and Rectified Adam optimizers for training.
5.  Since these are one-off experiments we’ll not be performing an exhaustive dive on tuning hyperparameters (you can refer to _[Deep Learning for Computer Vision with Python](https://pyimagesearch.com/deep-learning-computer-vision-python-book/)_ if you would like details on how to tune your hyperparameters).

At this point we’ve motivated and planned our set of experiments — now let’s learn how to implement our training and evaluation scripts.

### Project structure

Go ahead and grab the _**“Downloads”**_ and then inspect the project directory with the `tree` command:

$ tree --dirsfirst --filelimit 10
.
├── output \[48 entries\]
├── plots \[12 entries\]
├── pyimagesearch
│   ├── \_\_init\_\_.py
│   ├── minigooglenet.py
│   ├── minivggnet.py
│   └── resnet.py
├── combinations.py
├── experiments.sh
├── plot.py
└── train.py

3 directories, 68 files

Our project consists of two output directories:

*   `output/` : Holds our classification report `.txt` files organized by experiment. Additionally, there is one `.pickle` file per experiment containing the serialized training history data (for plotting purposes).
*   `plots/` : For each CNN/dataset combination, a stacked accuracy/loss curve plot is output so that we can conveniently compare the Adam and RAdam optimizers.

The `pyimagesearch` module contains three Convolutional Neural Networks (CNNs) architectures constructed with Keras. These CNN implementations come directly from [_Deep Learning for Computer Vision with Python_](https://pyimagesearch.com/deep-learning-computer-vision-python-book/).

We will review three Python scripts in today’s tutorial:

*   `train.py` : Our training script accepts a CNN architecture, dataset, and optimizer via command line argument and begins fitting a model accordingly. This script will be invoked automatically for each of our 24 experiments via the `experiments.sh` bash script. Our training script produces two types of output files:
    *   `.txt` : A classification report printout in scikit-learn’s standard format.
    *   `.pickle` : Serialized training history so that it can later be recalled for plotting purposes.
*   `combinations.py` : This script computes all the experiment combinations for which we will train models and collect data. The result of executing this script is a bash/shell script named `experiments.sh` .
*   `plot.py` : Plots accuracy/loss curves for Adam/RAdam using matplotlib directly from the `output/*.pickle` files.

### Implementing the training script

Our training script will be responsible for accepting:

1.  A given model architecture
2.  A dataset
3.  An optimizer

And from there, the script will handle training the specified model, on the supplied dataset, using the specified optimizer.

We’ll use this script to run each of our 24 experiments.

Let’s go ahead and implement the `train.py` script now:

\# import the necessary packages
from pyimagesearch.minigooglenet import MiniGoogLeNet
from pyimagesearch.minivggnet import MiniVGGNet
from pyimagesearch.resnet import ResNet
from sklearn.preprocessing import LabelBinarizer
from sklearn.metrics import classification\_report
from keras.preprocessing.image import ImageDataGenerator
from keras.optimizers import Adam
from keras\_radam import RAdam
from keras.datasets import fashion\_mnist
from keras.datasets import cifar100
from keras.datasets import cifar10
from keras.datasets import mnist
import numpy as np
import argparse
import pickle
import cv2

Imports include our three CNN architectures, four datasets, and two optimizers (`Adam` and `RAdam` ).

Let’s [parse command line arguments](https://pyimagesearch.com/2018/03/12/python-argparse-command-line-arguments/):

\# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-i", "--history", required=True,
	help="path to output training history file")
ap.add\_argument("-r", "--report", required=True,
	help="path to output classification report file")
ap.add\_argument("-d", "--dataset", type=str, default="mnist",
	choices=\["mnist", "fashion\_mnist", "cifar10", "cifar100"\],
	help="dataset name")
ap.add\_argument("-m", "--model", type=str, default="resnet",
	choices=\["resnet", "googlenet", "minivggnet"\],
	help="type of model architecture")
ap.add\_argument("-o", "--optimizer", type=str, default="adam",
	choices=\["adam", "radam"\],
	help="type of optmizer")
args = vars(ap.parse\_args())

Our command line arguments include:

*   `--history` : The path to the output training history `.pickle` file.
*   `--report` : The path to the output classification report `.txt` file.
*   `--dataset` : The dataset to train our model on can be any of the `choices` listed on **Line 26**.
*   `--model` : The deep learning model architecture must be one of the `choices` on **Line 29**.
*   `--optimizer` : Our `adam` or `radam` deep learning optimization method.

Upon providing the command line arguments via the terminal, our training script dynamically sets up and launches the experiment. Output files are named according to the parameters of the experiment.

From here we’ll set two constants and initialize the default number of channels for the dataset:

\# initialize the batch size and number of epochs to train
BATCH\_SIZE = 128
NUM\_EPOCHS = 60

# initialize the number of channels in the dataset
numChans = 1

If our `--dataset` is MNIST or Fashion MNIST, we’ll load the dataset in the following manner:

\# check if we are using either the MNIST or Fashion MNIST dataset
if args\["dataset"\] in ("mnist", "fashion\_mnist"):
	# check if we are using MNIST
	if args\["dataset"\] == "mnist":
		# initialize the label names for the MNIST dataset
		labelNames = \[str(i) for i in range(0, 10)\]

		# load the MNIST dataset
		print("\[INFO\] loading MNIST dataset...")
		((trainX, trainY), (testX, testY)) = mnist.load\_data()

	# otherwise, are are using Fashion MNIST
	else:
		# initialize the label names for the Fashion MNIST dataset
		labelNames = \["top", "trouser", "pullover", "dress", "coat",
			"sandal", "shirt", "sneaker", "bag", "ankle boot"\]

		# load the Fashion MNIST dataset
		print("\[INFO\] loading Fashion MNIST dataset...")
		((trainX, trainY), (testX, testY)) = fashion\_mnist.load\_data()

	# MNIST dataset images are 28x28 but the networks we will be
	# training expect 32x32 images
	trainX = np.array(\[cv2.resize(x, (32, 32)) for x in trainX\])
	testX = np.array(\[cv2.resize(x, (32, 32)) for x in testX\])

	# reshape the data matrices to include a channel dimension which
	# is required for training)
	trainX = trainX.reshape((trainX.shape\[0\], 32, 32, 1))
	testX = testX.reshape((testX.shape\[0\], 32, 32, 1))

Keep in mind that MNIST images are _28×28_ but we need _32×32_ images for our architectures. Thus, **Lines 66 and 67** `resize` all images in the dataset. **Lines 71 and 72** then add the batch dimension.

Otherwise, we have a CIFAR variant `--dataset` to load:

\# otherwise, we must be using a variant of CIFAR
else:
	# update the number of channels in the images
	numChans = 3

	# check if we are using CIFAR-10
	if args\["dataset"\] == "cifar10":
		# initialize the label names for the CIFAR-10 dataset
		labelNames = \["airplane", "automobile", "bird", "cat",
			"deer", "dog", "frog", "horse", "ship", "truck"\]

		# load the CIFAR-10 dataset
		print("\[INFO\] loading CIFAR-10 dataset...")
		((trainX, trainY), (testX, testY)) = cifar10.load\_data()

	# otherwise, we are using CIFAR-100
	else:
		# initialize the label names for the CIFAR-100 dataset
		labelNames = \["apple", "aquarium\_fish", "baby", "bear",
			"beaver", "bed", "bee", "beetle",  "bicycle", "bottle",
			"bowl", "boy", "bridge", "bus", "butterfly", "camel",
			"can", "castle", "caterpillar", "cattle", "chair",
			"chimpanzee", "clock",  "cloud", "cockroach", "couch",
			"crab", "crocodile", "cup", "dinosaur",  "dolphin",
			"elephant", "flatfish", "forest", "fox", "girl",
			"hamster",  "house", "kangaroo", "keyboard", "lamp",
			"lawn\_mower", "leopard", "lion", "lizard", "lobster",
			"man", "maple\_tree", "motorcycle", "mountain", "mouse",
			"mushroom", "oak\_tree", "orange", "orchid", "otter",
			"palm\_tree", "pear", "pickup\_truck", "pine\_tree", "plain",
			"plate", "poppy", "porcupine", "possum", "rabbit",
			"raccoon", "ray", "road", "rocket", "rose", "sea", "seal",
			"shark", "shrew", "skunk", "skyscraper", "snail", "snake",
			"spider", "squirrel", "streetcar", "sunflower",
			"sweet\_pepper", "table", "tank", "telephone", "television",
			"tiger", "tractor", "train", "trout", "tulip", "turtle",
			"wardrobe", "whale", "willow\_tree", "wolf", "woman", "worm"\]

		# load the CIFAR-100 dataset
		print("\[INFO\] loading CIFAR-100 dataset...")
		((trainX, trainY), (testX, testY)) = cifar100.load\_data()

CIFAR datasets contain 3-channel color images (**Line 77**). These datasets are already comprised of _32×32_ images (no resizing is necessary).

From here, we’ll scale our data and determine the total number of classes:

\# scale the data to the range \[0, 1\]
trainX = trainX.astype("float32") / 255.0
testX = testX.astype("float32") / 255.0

# determine the total number of unique classes in the dataset
numClasses = len(np.unique(trainY))
print("\[INFO\] {} classes in dataset".format(numClasses))

Followed by initializing this experiment’s **deep learning optimizer:**

\# check if we are using Adam
if args\["optimizer"\] == "adam":
	# initialize the Adam optimizer
	print("\[INFO\] using Adam optimizer")
	opt = Adam(lr=1e-3)

# otherwise, we are using Rectified Adam
else:
	# initialize the Rectified Adam optimizer
	print("\[INFO\] using Rectified Adam optimizer")
	opt = RAdam(total\_steps=5000, warmup\_proportion=0.1, min\_lr=1e-5)

Either `Adam` or `RAdam` is initialized according to the `--optimizer` command line argument switch.

Our `model` is then built depending upon the `--model` command line argument:

\# check if we are using the ResNet architecture
if args\["model"\] == "resnet":
	# utilize the ResNet architecture
	print("\[INFO\] initializing ResNet...")
	model = ResNet.build(32, 32, numChans, numClasses, (9, 9, 9),
		(64, 64, 128, 256), reg=0.0005)

# check if we are using Tiny GoogLeNet
elif args\["model"\] == "googlenet":
	# utilize the MiniGoogLeNet architecture
	print("\[INFO\] initializing MiniGoogLeNet...")
	model = MiniGoogLeNet.build(width=32, height=32, depth=numChans,
		classes=numClasses)

# otherwise, we must be using MiniVGGNet
else:
	# utilize the MiniVGGNet architecture
	print("\[INFO\] initializing MiniVGGNet...")
	model = MiniVGGNet.build(width=32, height=32, depth=numChans,
		classes=numClasses)

Once either ResNet, GoogLeNet, or MiniVGGNet is built, we’ll binarize our labels and construct our [data augmentation](https://pyimagesearch.com/2019/07/08/keras-imagedatagenerator-and-data-augmentation/) object:

\# convert the labels from integers to vectors
lb = LabelBinarizer()
trainY = lb.fit\_transform(trainY)
testY = lb.transform(testY)

# construct the image generator for data augmentation
aug = ImageDataGenerator(rotation\_range=18, zoom\_range=0.15,
	width\_shift\_range=0.2, height\_shift\_range=0.2, shear\_range=0.15,
	horizontal\_flip=True, fill\_mode="nearest")

Followed by compiling our model and training the network:

\# compile the model and train the network
print("\[INFO\] training network...")
model.compile(loss="categorical\_crossentropy", optimizer=opt,
	metrics=\["accuracy"\])
H = model.fit\_generator(
	aug.flow(trainX, trainY, batch\_size=BATCH\_SIZE),
	validation\_data=(testX, testY),
	steps\_per\_epoch=trainX.shape\[0\] // BATCH\_SIZE,
	epochs=NUM\_EPOCHS,
	verbose=1)

We then evaluate the trained model and dump training history to disk:

\# evaluate the network
print("\[INFO\] evaluating network...")
predictions = model.predict(testX, batch\_size=BATCH\_SIZE)
report = classification\_report(testY.argmax(axis=1),
	predictions.argmax(axis=1), target\_names=labelNames)

# serialize the training history to disk
print("\[INFO\] serializing training history...")
f = open(args\["history"\], "wb")
f.write(pickle.dumps(H.history))
f.close()

# save the classification report to disk
print("\[INFO\] saving classification report...")
f = open(args\["report"\], "w")
f.write(report)
f.close()

Each experiment will contain a classification report `.txt` file along with a serialized training history `.pickle` file.

The classification reports will be inspected manually whereas the training history files will later be opened by operations inside `plot.py` , the training history parsed, and finally plotted.

As you’ve learned, creating a training script that dynamically sets up an experiment is quite straightforward.

### Creating our experiment combinations

At this point, we have our training script which can accept a (1) model architecture, (2) dataset, and (3) optimizer, followed by fitting a model using the respective combination.

That being said, are we going to _manually_ run each and every individual command?

No, not only is that a tedious task, _it’s also prone to human error_.

Instead, let’s create a Python script to **generate a shell script containing the `train.py` command for each experiment we want to run.**

Open up the `combinations.py` file and insert the following code:

\# import the necessary packages
import argparse
import os

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-o", "--output", required=True,
	help="path to output output directory")
ap.add\_argument("-s", "--script", required=True,
	help="path to output shell script")
args = vars(ap.parse\_args())

Our script requires two command line arguments:

*   `--output` : The path to the output directory where the training files will be stored.
*   `--script` : The path to the output shell script which will contain all of our training script commands with command line argument combinations.

Let’s go ahead and open a new file for writing:

\# open the output shell script for writing, then write the header
f = open(args\["script"\], "w")
f.write("#!/bin/sh\\n\\n")

# initialize the list of datasets, models, and optimizers
datasets = \["mnist", "fashion\_mnist", "cifar10", "cifar100"\]
models = \["resnet", "googlenet", "minivggnet"\]
optimizers = \["adam", "radam"\]

**Line 14** opens a shell script file writing. Subsequently, **Line 15** writes the [“shebang”](https://en.wikipedia.org/wiki/Shebang_\(Unix\)) to indicate that this shell script is executable.

**Lines 18-20** then list our `datasets` , `models` , and `optimizers` .

We will form all possible combinations of experiments from these lists in a nested loop:

\# loop over all combinations of datasets, models, and optimizers
for dataset in datasets:
	for model in models:
		for opt in optimizers:
			# build the path to the output training log file
			histFilename = "{}\_{}\_{}.pickle".format(model, opt, dataset)
			historyPath = os.path.sep.join(\[args\["output"\],
				histFilename\])

			# build the path to the output report log file
			reportFilename = "{}\_{}\_{}.txt".format(model, opt, dataset)
			reportPath = os.path.sep.join(\[args\["output"\],
				reportFilename\])

			# construct the command that will be executed to launch
			# the experiment
			cmd = ("python train.py --history {} --report {} "
				   "--dataset {}  --model {} --optimizer {}").format(
						historyPath, reportPath, dataset, model, opt)

			# write the command to disk
			f.write("{}\\n".format(cmd))

# close the shell script file
f.close()

Inside the loop, we:

*   Construct our history file path (**Lines 27-29**).
*   Assemble our report file path (**Lines 32-34**).
*   Concatenate each command per the current loop iteration’s combination and write it to the shell file (**Lines 38-43**).

Finally, we close the shell script file.

_**Note:** I am making the assumption that you are using a Unix machine to run these experiments. If you’re using Windows you should either (1) update this script to generate a batch file instead, or (2) manually execute the `train.py` command for each respective experiment. Note that [I do not support Windows](https://pyimagesearch.com/faqs/single-faq/can-you-help-me-do-___-on-windows) on the PyImageSearch blog so you will be on your own to implement it based on this script._

### Generating the experiment shell script

Go ahead and use the **_“Downloads”_** section of this tutorial to download the source code to the guide.

From there, open up a terminal and execute the `combinations.py` script:

$ python combinations.py --output output --script experiments.sh

After the script has executed you should have a file named `experiments.sh` in your working directory — this file contains the 24 separate experiments we’ll be running to compare Adam to Rectified Adam.

Go ahead and investigate `experiments.sh` now:

#!/bin/sh

python train.py --history output/resnet\_adam\_mnist.pickle --report output/resnet\_adam\_mnist.txt --dataset mnist  --model resnet --optimizer adam
python train.py --history output/resnet\_radam\_mnist.pickle --report output/resnet\_radam\_mnist.txt --dataset mnist  --model resnet --optimizer radam
python train.py --history output/googlenet\_adam\_mnist.pickle --report output/googlenet\_adam\_mnist.txt --dataset mnist  --model googlenet --optimizer adam
python train.py --history output/googlenet\_radam\_mnist.pickle --report output/googlenet\_radam\_mnist.txt --dataset mnist  --model googlenet --optimizer radam
python train.py --history output/minivggnet\_adam\_mnist.pickle --report output/minivggnet\_adam\_mnist.txt --dataset mnist  --model minivggnet --optimizer adam
python train.py --history output/minivggnet\_radam\_mnist.pickle --report output/minivggnet\_radam\_mnist.txt --dataset mnist  --model minivggnet --optimizer radam
python train.py --history output/resnet\_adam\_fashion\_mnist.pickle --report output/resnet\_adam\_fashion\_mnist.txt --dataset fashion\_mnist  --model resnet --optimizer adam
python train.py --history output/resnet\_radam\_fashion\_mnist.pickle --report output/resnet\_radam\_fashion\_mnist.txt --dataset fashion\_mnist  --model resnet --optimizer radam
python train.py --history output/googlenet\_adam\_fashion\_mnist.pickle --report output/googlenet\_adam\_fashion\_mnist.txt --dataset fashion\_mnist  --model googlenet --optimizer adam
python train.py --history output/googlenet\_radam\_fashion\_mnist.pickle --report output/googlenet\_radam\_fashion\_mnist.txt --dataset fashion\_mnist  --model googlenet --optimizer radam
python train.py --history output/minivggnet\_adam\_fashion\_mnist.pickle --report output/minivggnet\_adam\_fashion\_mnist.txt --dataset fashion\_mnist  --model minivggnet --optimizer adam
python train.py --history output/minivggnet\_radam\_fashion\_mnist.pickle --report output/minivggnet\_radam\_fashion\_mnist.txt --dataset fashion\_mnist  --model minivggnet --optimizer radam
python train.py --history output/resnet\_adam\_cifar10.pickle --report output/resnet\_adam\_cifar10.txt --dataset cifar10  --model resnet --optimizer adam
python train.py --history output/resnet\_radam\_cifar10.pickle --report output/resnet\_radam\_cifar10.txt --dataset cifar10  --model resnet --optimizer radam
python train.py --history output/googlenet\_adam\_cifar10.pickle --report output/googlenet\_adam\_cifar10.txt --dataset cifar10  --model googlenet --optimizer adam
python train.py --history output/googlenet\_radam\_cifar10.pickle --report output/googlenet\_radam\_cifar10.txt --dataset cifar10  --model googlenet --optimizer radam
python train.py --history output/minivggnet\_adam\_cifar10.pickle --report output/minivggnet\_adam\_cifar10.txt --dataset cifar10  --model minivggnet --optimizer adam
python train.py --history output/minivggnet\_radam\_cifar10.pickle --report output/minivggnet\_radam\_cifar10.txt --dataset cifar10  --model minivggnet --optimizer radam
python train.py --history output/resnet\_adam\_cifar100.pickle --report output/resnet\_adam\_cifar100.txt --dataset cifar100  --model resnet --optimizer adam
python train.py --history output/resnet\_radam\_cifar100.pickle --report output/resnet\_radam\_cifar100.txt --dataset cifar100  --model resnet --optimizer radam
python train.py --history output/googlenet\_adam\_cifar100.pickle --report output/googlenet\_adam\_cifar100.txt --dataset cifar100  --model googlenet --optimizer adam
python train.py --history output/googlenet\_radam\_cifar100.pickle --report output/googlenet\_radam\_cifar100.txt --dataset cifar100  --model googlenet --optimizer radam
python train.py --history output/minivggnet\_adam\_cifar100.pickle --report output/minivggnet\_adam\_cifar100.txt --dataset cifar100  --model minivggnet --optimizer adam
python train.py --history output/minivggnet\_radam\_cifar100.pickle --report output/minivggnet\_radam\_cifar100.txt --dataset cifar100  --model minivggnet --optimizer radam

_**Note:** Be sure to use the horizontal scroll bar to inspect the entire contents of the `experiments.sh` script. I intentionally did not break up lines or automatically wrap them for better display. You can also refer to **Figure 4** below — I suggest **clicking the image to enlarge + inspect** it._

[![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20238'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/rectified_adam_comparison_experiments.png)

**Figure 4:** The output of our \`combinations.py\` file is a shell script listing the training script commands to run in succession. Click image to enlarge.

Notice how there is a `train.py` call for **each of the 24 possible combinations** of model architecture, dataset, and optimizer. Furthermore, the “shebang” on **Line 1** indicates that this shell script is executable.

### Running our experiments

The next step is to actually perform each of these experiments.

I executed the shell script on an [Amazon EC2 instance](https://pyimagesearch.com/2017/09/20/pre-configured-amazon-aws-deep-learning-ami-with-python/) with an NVIDIA K80 GPU. It took approximately 48 hours to run all the experiments.

To launch the experiments for yourself, just run the following command:

$ ./experiments.sh

After the script has finished running, your `output/` directory should be filled with `.pickle` and `.txt` files:

$ ls -l output/
googlenet\_adam\_cifar10.pickle
googlenet\_adam\_cifar10.txt
googlenet\_adam\_cifar100.pickle
googlenet\_adam\_cifar100.txt
...
resnet\_radam\_fashion\_mnist.pickle
resnet\_radam\_fashion\_mnist.txt
resnet\_radam\_mnist.pickle
resnet\_radam\_mnist.txt

The `.txt` files contain the output of scikit-learn’s `classification_report`, a human-readable output that tells us how well our model performed.

The `.pickle` files contain the training history for the model. We’ll use this `.pickle` file to plot both Adam and Rectified Adam’s performance in the next section.

### Implementing our Adam vs. Rectified Adam plotting script

Our final Python script, `plot.py`, will be used to plot the performance of Adam vs. Rectified Adam, giving us a nice, clear visualization of a given model architecture trained on a specific dataset.

The plot file opens each Adam/RAdam `.pickle` file pair and generates a corresponding plot.

Open up `plot.py` and insert the following code:

\# import the necessary packages
import matplotlib.pyplot as plt
import numpy as np
import argparse
import pickle
import os

def plot\_history(adamHist, rAdamHist, accTitle, lossTitle):
	# determine the total number of epochs used for training, then
	# initialize the figure
	N = np.arange(0, len(adamHist\["loss"\]))
	plt.style.use("ggplot")
	(fig, axs) = plt.subplots(2, 1, figsize=(7, 9))

	# plot the accuracy for Adam vs. Rectified Adam
	axs\[0\].plot(N, adamHist\["acc"\], label="adam\_train\_acc")
	axs\[0\].plot(N, adamHist\["val\_acc"\], label="adam\_val\_acc")
	axs\[0\].plot(N, rAdamHist\["acc"\], label="radam\_train\_acc")
	axs\[0\].plot(N, rAdamHist\["val\_acc"\], label="radam\_val\_acc")
	axs\[0\].set\_title(accTitle)
	axs\[0\].set\_xlabel("Epoch #")
	axs\[0\].set\_ylabel("Accuracy")
	axs\[0\].legend(loc="lower right")

	# plot the loss for Adam vs. Rectified Adam
	axs\[1\].plot(N, adamHist\["loss"\], label="adam\_train\_loss")
	axs\[1\].plot(N, adamHist\["val\_loss"\], label="adam\_val\_loss")
	axs\[1\].plot(N, rAdamHist\["loss"\], label="radam\_train\_loss")
	axs\[1\].plot(N, rAdamHist\["val\_loss"\], label="radam\_val\_loss")
	axs\[1\].set\_title(lossTitle)
	axs\[1\].set\_xlabel("Epoch #")
	axs\[1\].set\_ylabel("Loss")
	axs\[1\].legend(loc="upper right")

	# update the layout of the plot
	plt.tight\_layout()

**Lines 2-6** handle imports, namely the `matplotlib.pyplot` module.

The `plot_history` function is responsible for generating two stacked plots via the subplots feature:

*   Training/validation _**accuracy**_ curves (**Lines 16-23**).
*   Training/validation _**loss**_ curves (**Lines 26-33**).

Both Adam and Rectified Adam training history curves are generated from `adamHist` and `rAdamHist` data passed as parameters to the function.

_**Note:** If you are using TensorFlow 2.0 (i.e., `tf.keras` ) to run this code , you’ll need to change all occurrences of `acc` and `val_acc` to `accuracy` and `val_accuracy` , respectively as TensorFlow 2.0 has made a breaking change to the accuracy name._

Let’s handle parsing command line arguments:

\# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add\_argument("-i", "--input", required=True,
	help="path to input directory of Keras training history files")
ap.add\_argument("-p", "--plots", required=True,
	help="path to output directory of training plots")
args = vars(ap.parse\_args())

# initialize the list of datasets and models
datasets = \["mnist", "fashion\_mnist", "cifar10", "cifar100"\]
models = \["resnet", "googlenet", "minivggnet"\]

Our command line arguments consist of:

*   `--input` : The path to the input directory of training history files to be parsed for plot generation.
*   `--plots` : Our output path where the plots will be stored.

**Lines 47 and 48** list our `datasets` and `models` . We’ll loop over the combinations of datasets and models to generate our plots:

\# loop over all combinations of datasets and models
for dataset in datasets:
	for model in models:
		# construct the path to the Adam output training history files
		adamFilename = "{}\_{}\_{}.pickle".format(model, "adam",
			dataset)
		adamPath = os.path.sep.join(\[args\["input"\], adamFilename\])

		# construct the path to the Rectified Adam output training
		# history files
		rAdamFilename = "{}\_{}\_{}.pickle".format(model, "radam",
			dataset)
		rAdamPath = os.path.sep.join(\[args\["input"\], rAdamFilename\])

		# load the training history files for Adam and Rectified Adam,
		# respectively
		adamHist = pickle.loads(open(adamPath, "rb").read())
		rAdamHist = pickle.loads(open(rAdamPath, "rb").read())

		# plot the accuracy/loss for the current dataset, comparing
		# Adam vs. Rectified Adam
		accTitle = "Adam vs. RAdam for '{}' on '{}' (Accuracy)".format(
			model, dataset)
		lossTitle = "Adam vs. RAdam for '{}' on '{}' (Loss)".format(
			model, dataset)
		plot\_history(adamHist, rAdamHist, accTitle, lossTitle)

		# construct the path to the output plot
		plotFilename = "{}\_{}.png".format(model, dataset)
		plotPath = os.path.sep.join(\[args\["plots"\], plotFilename\])

		# save the plot and clear it
		plt.savefig(plotPath)
		plt.clf()

Inside our nested `datasets` /`models` loop, we:

*   Construct Adam and Rectified Adam’s file paths (**Lines 54-62**).
*   Load serialized training history (**Lines 66 and 67**).
*   Generate the plots using our `plot_history` function (**Lines 71-75**).
*   Export the figures to disk (**Lines 78-83**).

### Plotting Adam vs. Rectified Adam

We are now ready to run the `plot.py` script.

Again, make sure you have used the **_“Downloads”_** section of this tutorial to download the source code.

From there, execute the following command:

$ python plot.py --input output --plots plots

You can then check the `plots/` directory and ensure it has been populated with the training history figures:

$ ls -l plots/
googlenet\_cifar10.png
googlenet\_cifar100.png
googlenet\_fashion\_mnist.png
googlenet\_mnist.png
minivggnet\_cifar10.png
minivggnet\_cifar100.png
minivggnet\_fashion\_mnist.png
minivggnet\_mnist.png
resnet\_cifar10.png
resnet\_cifar100.png
resnet\_fashion\_mnist.png
resnet\_mnist.png

In the next section, we’ll review the results of our experiments.

### Adam vs. Rectified Adam Experiments with MNIST

[![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20560%20112'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2014/06/mnist_sample.jpg)

**Figure 5:** Montage of samples from the MNIST digit dataset.

Our first set of experiments will compare Adam vs. Rectified Adam on the [MNIST dataset](http://yann.lecun.com/exdb/mnist/), a standard benchmark image classification dataset for handwritten digit recognition.

#### MNIST – MiniVGGNet

[![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20771'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/minivggnet_mnist.png)

**Figure 6:** Which is better — Adam or RAdam optimizer using MiniVGGNet on the MNIST dataset?

Our first experiment compares Adam to Rectified Adam when training MiniVGGNet on the MNIST dataset.

Below is the output classification report for the **Adam optimizer:**

              precision    recall  f1-score   support

           0       0.99      1.00      1.00       980
           1       0.99      1.00      0.99      1135
           2       0.98      0.96      0.97      1032
           3       1.00      1.00      1.00      1010
           4       0.99      1.00      0.99       982
           5       0.97      0.98      0.98       892
           6       0.98      0.98      0.98       958
           7       0.99      0.99      0.99      1028
           8       0.99      0.99      0.99       974
           9       1.00      0.99      0.99      1009

   micro avg       0.99      0.99      0.99     10000
   macro avg       0.99      0.99      0.99     10000
weighted avg       0.99      0.99      0.99     10000

As well as the classification report for the **Rectified Adam optimizer:**

              precision    recall  f1-score   support

           0       0.99      1.00      0.99       980
           1       1.00      0.99      1.00      1135
           2       0.97      0.97      0.97      1032
           3       0.99      0.99      0.99      1010
           4       0.99      0.99      0.99       982
           5       0.98      0.97      0.97       892
           6       0.98      0.98      0.98       958
           7       0.99      0.99      0.99      1028
           8       0.99      0.99      0.99       974
           9       0.99      0.99      0.99      1009

   micro avg       0.99      0.99      0.99     10000
   macro avg       0.99      0.99      0.99     10000
weighted avg       0.99      0.99      0.99     10000

As you can see, we’re obtaining **99% accuracy for _both_ experiments.**

Looking at **Figure 6** you can observe the warmup period associated with Rectified Adam:

Loss starts off very high and accuracy very low

After warmup is complete the Rectified Adam optimizer catches up with Adam

What’s interesting to note though is that **Adam obtains lower loss** compared to Rectified Adam — we’ll actually see that trend continue in the rest of the experiments we run (and I’ll explain _why_ this happens as well).

#### MNIST – GoogLeNet

[![Image 9](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20771'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/googlenet_mnist.png)

**Figure 7:** Which deep learning optimizer is actually better — Rectified Adam or Adam? This plot is from my experiment notebook while testing RAdam and Adam using GoogLeNet on the MNIST dataset.

This next experiment compares Adam to Rectified Adam for GoogLeNet trained on the MNIST dataset.

Below follows the output of the **Adam optimizer:**

              precision    recall  f1-score   support

           0       1.00      1.00      1.00       980
           1       1.00      0.99      1.00      1135
           2       0.96      0.99      0.97      1032
           3       0.99      1.00      0.99      1010
           4       0.99      0.99      0.99       982
           5       0.99      0.96      0.98       892
           6       0.98      0.99      0.98       958
           7       0.99      0.99      0.99      1028
           8       1.00      1.00      1.00       974
           9       1.00      0.98      0.99      1009

   micro avg       0.99      0.99      0.99     10000
   macro avg       0.99      0.99      0.99     10000
weighted avg       0.99      0.99      0.99     10000

As well as the output for the **Rectified Adam optimizer:**

              precision    recall  f1-score   support

           0       1.00      1.00      1.00       980
           1       1.00      0.99      1.00      1135
           2       0.98      0.98      0.98      1032
           3       1.00      0.99      1.00      1010
           4       1.00      0.99      1.00       982
           5       0.97      0.99      0.98       892
           6       0.99      0.98      0.99       958
           7       0.99      1.00      0.99      1028
           8       0.99      1.00      1.00       974
           9       1.00      0.99      1.00      1009

   micro avg       0.99      0.99      0.99     10000
   macro avg       0.99      0.99      0.99     10000
weighted avg       0.99      0.99      0.99     10000

Again, **99% accuracy** is obtained for both optimizers.

This time both the training/validation plots are near identical for both accuracy and loss.

#### MNIST – ResNet

[![Image 10](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20771'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/resnet_mnist.png)

**Figure 8:** Training accuracy/loss plot for ResNet on the MNIST dataset using both the RAdam (Rectified Adam) and Adam deep learning optimizers with Keras.

Our final MNIST experiment compares training ResNet using both Adam and Rectified Adam.

Given that MNIST is not a very challenging dataset we obtain **99% accuracy** for the **Adam optimizer:**

              precision    recall  f1-score   support

           0       1.00      1.00      1.00       980
           1       1.00      0.99      1.00      1135
           2       0.98      0.98      0.98      1032
           3       0.99      1.00      1.00      1010
           4       0.99      1.00      0.99       982
           5       0.99      0.98      0.98       892
           6       0.98      0.99      0.99       958
           7       0.99      1.00      0.99      1028
           8       0.99      1.00      1.00       974
           9       1.00      0.98      0.99      1009

   micro avg       0.99      0.99      0.99     10000
   macro avg       0.99      0.99      0.99     10000
weighted avg       0.99      0.99      0.99     10000

As well as the **Rectified Adam optimizer:**

              precision    recall  f1-score   support

           0       1.00      1.00      1.00       980
           1       1.00      1.00      1.00      1135
           2       0.97      0.98      0.98      1032
           3       1.00      1.00      1.00      1010
           4       0.99      1.00      1.00       982
           5       0.99      0.97      0.98       892
           6       0.99      0.98      0.99       958
           7       0.99      1.00      0.99      1028
           8       1.00      1.00      1.00       974
           9       1.00      0.99      1.00      1009

   micro avg       0.99      0.99      0.99     10000
   macro avg       0.99      0.99      0.99     10000
weighted avg       0.99      0.99      0.99     10000

But take a look at **Figure 8** — note how Adam obtains much lower loss than Rectified Adam.

That’s not necessarily a bad thing as it may imply that Rectified Adam is obtaining a **more generalizable model**; however, performance on the testing set is identical so we would need to test on images _outside_ MNIST (which is outside the scope of this blog post).

### Adam vs. Rectified Adam Experiments with Fashion MNIST

[![Image 11](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20338'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/02/fashion_mnist_dataset_sample.png)

**Figure 9:** The Fashion MNIST dataset was created by e-commerce company, [Zalando](https://github.com/zalandoresearch/fashion-mnist), as a drop-in replacement for MNIST Digits. It is a great dataset to practice/experiment with when using Keras for deep learning. ([image source](https://medium.com/deep-learning-turkey/deep-learning-lab-episode-1-fashion-mnist-c7af60029836))

Our next set of experiments evaluate Adam vs. Rectified Adam on the Fashion MNIST dataset, a drop-in replacement for the standard MNIST dataset.

You can read more about [Fashion MNIST here](https://pyimagesearch.com/2019/02/11/fashion-mnist-with-keras-and-deep-learning/).

#### Fashion MNIST – MiniVGGNet

[![Image 12](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20771'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/minivggnet_fashion_mnist.png)

**Figure 10:** Testing optimizers with deep learning, including new ones such as RAdam, requires multiple experiments. Shown in this figure is the MiniVGGNet CNN trained on the Fashion MNIST dataset with both Adam and RAdam optimizers.

Our first experiment evaluates the MiniVGGNet architecture trained on the Fashion MNIST dataset.

Below you can find the output of training with the **Adam optimizer:**

              precision    recall  f1-score   support

         top       0.95      0.71      0.81      1000
     trouser       0.99      0.99      0.99      1000
    pullover       0.94      0.76      0.84      1000
       dress       0.96      0.80      0.87      1000
        coat       0.84      0.90      0.87      1000
      sandal       0.98      0.98      0.98      1000
       shirt       0.59      0.91      0.71      1000
     sneaker       0.96      0.97      0.96      1000
         bag       0.98      0.99      0.99      1000
  ankle boot       0.97      0.97      0.97      1000

   micro avg       0.90      0.90      0.90     10000
   macro avg       0.92      0.90      0.90     10000
weighted avg       0.92      0.90      0.90     10000

As well as the **Rectified Adam optimizer:**

              precision    recall  f1-score   support

         top       0.85      0.85      0.85      1000
     trouser       1.00      0.97      0.99      1000
    pullover       0.89      0.84      0.87      1000
       dress       0.93      0.81      0.87      1000
        coat       0.85      0.80      0.82      1000
      sandal       0.99      0.95      0.97      1000
       shirt       0.62      0.77      0.69      1000
     sneaker       0.92      0.96      0.94      1000
         bag       0.96      0.99      0.97      1000
  ankle boot       0.97      0.95      0.96      1000

   micro avg       0.89      0.89      0.89     10000
   macro avg       0.90      0.89      0.89     10000
weighted avg       0.90      0.89      0.89     10000

Note that the Adam optimizer _outperforms_ Rectified Adam, obtaining **92% accuracy** compared to the 90% accuracy of Rectified Adam.

Furthermore, take a look at the training plot in **Figure 10** — training is very stable with validation loss falling below training loss.

With more aggressive training with Adam, we can likely improve our accuracy further.

#### Fashion MNIST – GoogLeNet

[![Image 13](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20771'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/googlenet_fashion_mnist.png)

**Figure 11:** Is either RAdam or Adam a better deep learning optimizer using GoogLeNet? Using the Fashion MNIST dataset with Adam shows signs of overfitting past epoch 30. RAdam appears more stable in this experiment.

We now evaluate GoogLeNet trained on Fashion MNIST using Adam and Rectified Adam.

Below is the classification report from the **Adam optimizer:**

              precision    recall  f1-score   support

         top       0.84      0.89      0.86      1000
     trouser       1.00      0.99      0.99      1000
    pullover       0.87      0.94      0.90      1000
       dress       0.95      0.88      0.91      1000
        coat       0.95      0.85      0.89      1000
      sandal       0.98      0.99      0.98      1000
       shirt       0.79      0.82      0.81      1000
     sneaker       0.99      0.90      0.94      1000
         bag       0.98      0.99      0.99      1000
  ankle boot       0.91      0.99      0.95      1000

   micro avg       0.92      0.92      0.92     10000
   macro avg       0.93      0.92      0.92     10000
weighted avg       0.93      0.92      0.92     10000

As well as the output from the **Rectified Adam optimizer:**

              precision    recall  f1-score   support

         top       0.91      0.83      0.87      1000
     trouser       0.99      0.99      0.99      1000
    pullover       0.94      0.85      0.89      1000
       dress       0.96      0.86      0.90      1000
        coat       0.90      0.91      0.90      1000
      sandal       0.98      0.98      0.98      1000
       shirt       0.70      0.88      0.78      1000
     sneaker       0.97      0.96      0.96      1000
         bag       0.98      0.99      0.99      1000
  ankle boot       0.97      0.97      0.97      1000

   micro avg       0.92      0.92      0.92     10000
   macro avg       0.93      0.92      0.92     10000
weighted avg       0.93      0.92      0.92     10000

This time both optimizers obtain **93% accuracy**, but what’s more interesting is to take a look at the training history plot in **Figure 11**.

Here we can see that training loss starts to diverge past epoch 30 for the Adam optimizer — this divergence grows wider and wider as we continue training. **At this point, we should start to be concerned about overfitting using Adam.**

On the other hand, Rectified Adam’s performance is _stable_ with no signs of overfitting.

In this particular experiment, it’s clear that Rectified Adam is generalizing better, and had we wished to deploy this model to production, the Rectified Adam optimizer version would be the one to go with.

#### Fashion MNIST – ResNet

[![Image 14](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20771'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/resnet_fashion_mnist.png)

**Figure 12:** Which deep learning optimizer is better — Adam or Rectified Adam (RAdam) — using the ResNet CNN on the Fashion MNIST dataset?

Our final experiment compares Adam vs. Rectified Adam optimizer trained on the Fashion MNIST dataset using ResNet.

Below is the output of the **Adam optimizer:**

              precision    recall  f1-score   support

         top       0.89      0.83      0.86      1000
     trouser       0.99      0.99      0.99      1000
    pullover       0.84      0.93      0.88      1000
       dress       0.94      0.83      0.88      1000
        coat       0.93      0.85      0.89      1000
      sandal       0.99      0.92      0.95      1000
       shirt       0.71      0.85      0.78      1000
     sneaker       0.88      0.99      0.93      1000
         bag       1.00      0.98      0.99      1000
  ankle boot       0.98      0.93      0.95      1000

   micro avg       0.91      0.91      0.91     10000
   macro avg       0.92      0.91      0.91     10000
weighted avg       0.92      0.91      0.91     10000

Here is the output of the **Rectified Adam optimizer:**

              precision    recall  f1-score   support

         top       0.88      0.86      0.87      1000
     trouser       0.99      0.99      0.99      1000
    pullover       0.91      0.87      0.89      1000
       dress       0.96      0.83      0.89      1000
        coat       0.86      0.92      0.89      1000
      sandal       0.98      0.98      0.98      1000
       shirt       0.72      0.80      0.75      1000
     sneaker       0.95      0.96      0.96      1000
         bag       0.98      0.99      0.99      1000
  ankle boot       0.97      0.96      0.96      1000

   micro avg       0.92      0.92      0.92     10000
   macro avg       0.92      0.92      0.92     10000
weighted avg       0.92      0.92      0.92     10000

Both models obtain **92% accuracy**, but take a look at the training history plot in **Figure 12**.

You can observe that Adam optimizer results in lower loss _and_ that the validation loss follows the training curve.

The Rectified Adam loss is arguably more stable with fewer fluctuations (as compared to standard Adam).

Exactly which one is “better” in this experiment would be dependent on how well the model generalizes to images _outside_ the training, validation, and testing set.

Further experiments would be required to mark the winner here, but my gut tells me that it’s Rectified Adam as (1) accuracy on the testing set is identical, and (2) lower loss doesn’t necessarily mean better generalization (in some cases it means that the model may _fail_ to generalize well) — but at the same time, training/validation loss are near identical for Adam. Without further experiments it’s hard to make the call.

### Adam vs. Rectified Adam Experiments with CIFAR-10

[![Image 15](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20488'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2018/10/keras_vs_tensorflow_cifar10.jpg)

**Figure 13:** The CIFAR-10 benchmarking dataset has 10 classes. We will use it for Rectified Adam experimentation to evaluate if RAdam or Adam is the better choice ([image source](https://www.cs.toronto.edu/~kriz/cifar.html)).

In these experiments, we’ll be comparing Adam vs. Rectified Adam performance using MiniVGGNet, GoogLeNet, and ResNet, all trained on the [CIFAR-10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html).

#### CIFAR-10 – MiniVGGNet

[![Image 16](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20771'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/minivggnet_cifar10.png)

**Figure 14:** Is the RAdam or Adam deep learning optimizer better using the MiniVGGNet CNN on the CIFAR-10 dataset?

Our next experiment compares Adam to Rectified Adam by training MiniVGGNet on the CIFAR-10 dataset.

Below is the output of training using the **Adam optimizer:**

              precision    recall  f1-score   support

    airplane       0.90      0.79      0.84      1000
  automobile       0.90      0.93      0.91      1000
        bird       0.90      0.63      0.74      1000
         cat       0.78      0.68      0.73      1000
        deer       0.83      0.79      0.81      1000
         dog       0.81      0.76      0.79      1000
        frog       0.70      0.95      0.81      1000
       horse       0.85      0.91      0.88      1000
        ship       0.93      0.89      0.91      1000
       truck       0.77      0.95      0.85      1000

   micro avg       0.83      0.83      0.83     10000
   macro avg       0.84      0.83      0.83     10000
weighted avg       0.84      0.83      0.83     10000

And here is the output from **Rectified Adam:**

              precision    recall  f1-score   support

    airplane       0.84      0.72      0.78      1000
  automobile       0.89      0.84      0.86      1000
        bird       0.80      0.41      0.54      1000
         cat       0.66      0.43      0.52      1000
        deer       0.66      0.65      0.66      1000
         dog       0.72      0.55      0.62      1000
        frog       0.48      0.96      0.64      1000
       horse       0.84      0.75      0.79      1000
        ship       0.87      0.88      0.88      1000
       truck       0.68      0.95      0.79      1000

   micro avg       0.71      0.71      0.71     10000
   macro avg       0.74      0.71      0.71     10000
weighted avg       0.74      0.71      0.71     10000

Here the Adam optimizer (**84% accuracy**) beats out Rectified Adam (74% accuracy).

Furthermore, validation loss is lower than training loss for the majority of training, implying that we can “train harder” by reducing our regularization strength and potentially increasing model capacity.

#### CIFAR-10 – GoogLeNet

[![Image 17](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20771'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/googlenet_cifar10.png)

**Figure 15:** Which is a better deep learning optimizer with the GoogLeNet CNN? The training accuracy/loss plot shows results from using Adam and RAdam as part of automated deep learning experiment data collection.

Next, let’s check out GoogLeNet trained on CIFAR-10 using Adam and Rectified Adam.

Here is the output of **Adam:**

              precision    recall  f1-score   support

    airplane       0.89      0.92      0.91      1000
  automobile       0.92      0.97      0.94      1000
        bird       0.90      0.87      0.88      1000
         cat       0.79      0.86      0.82      1000
        deer       0.92      0.85      0.89      1000
         dog       0.92      0.81      0.86      1000
        frog       0.87      0.96      0.91      1000
       horse       0.95      0.91      0.93      1000
        ship       0.96      0.92      0.94      1000
       truck       0.90      0.94      0.92      1000

   micro avg       0.90      0.90      0.90     10000
   macro avg       0.90      0.90      0.90     10000
weighted avg       0.90      0.90      0.90     10000

And here is the output of **Rectified Adam:**

              precision    recall  f1-score   support

    airplane       0.88      0.88      0.88      1000
  automobile       0.93      0.95      0.94      1000
        bird       0.84      0.82      0.83      1000
         cat       0.79      0.75      0.77      1000
        deer       0.89      0.82      0.85      1000
         dog       0.89      0.77      0.82      1000
        frog       0.80      0.96      0.87      1000
       horse       0.89      0.92      0.91      1000
        ship       0.95      0.92      0.93      1000
       truck       0.88      0.95      0.91      1000

   micro avg       0.87      0.87      0.87     10000
   macro avg       0.87      0.87      0.87     10000
weighted avg       0.87      0.87      0.87     10000

The Adam optimizer obtains **90% accuracy**, slightly beating out the 87% accuracy of Rectified Adam.

However, **Figure 15** tells an interesting story — past epoch 20 there is quite the divergence between Adam’s training and validation loss.

While the Adam optimized model obtained higher accuracy, there are signs of overfitting as validation loss is essentially stagnant past epoch 30.

Additional experiments would be required to mark a true winner but I imagine it would be Rectified Adam after some additional hyperparameter tuning.

#### CIFAR-10 – ResNet

[![Image 18](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20771'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/resnet_cifar10.png)

**Figure 16:** This Keras deep learning tutorial helps to answer the question: Is Rectified Adam or Adam the better deep learning optimizer? One of the 24 experiments uses the ResNet CNN and CIFAR-10 dataset.

Next, let’s check out ResNet trained using Adam and Rectified Adam on CIFAR-10.

Below you can find the output of the standard **Adam optimizer:**

              precision    recall  f1-score   support

    airplane       0.80      0.92      0.86      1000
  automobile       0.92      0.96      0.94      1000
        bird       0.93      0.74      0.82      1000
         cat       0.93      0.63      0.75      1000
        deer       0.95      0.80      0.87      1000
         dog       0.77      0.88      0.82      1000
        frog       0.75      0.97      0.84      1000
       horse       0.90      0.92      0.91      1000
        ship       0.93      0.93      0.93      1000
       truck       0.91      0.93      0.92      1000

   micro avg       0.87      0.87      0.87     10000
   macro avg       0.88      0.87      0.87     10000
weighted avg       0.88      0.87      0.87     10000

As well as the output from **Rectified Adam:**

              precision    recall  f1-score   support

    airplane       0.86      0.86      0.86      1000
  automobile       0.89      0.95      0.92      1000
        bird       0.85      0.72      0.78      1000
         cat       0.78      0.66      0.71      1000
        deer       0.83      0.81      0.82      1000
         dog       0.82      0.70      0.76      1000
        frog       0.72      0.95      0.82      1000
       horse       0.86      0.90      0.87      1000
        ship       0.94      0.90      0.92      1000
       truck       0.84      0.93      0.88      1000

   micro avg       0.84      0.84      0.84     10000
   macro avg       0.84      0.84      0.83     10000
weighted avg       0.84      0.84      0.83     10000

Adam is the winner here, obtaining **88% accuracy** versus Rectified Adam’s 84%.

### Adam vs. Rectified Adam Experiments with CIFAR-100

[![Image 19](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20450%20450'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/rectified_adam_comparison_cifar100.png)

**Figure 17:** The CIFAR-100 classification dataset is the brother of CIFAR-10 and includes more classes of images. ([image source](https://medium.com/@birdortyedi_23820/deep-learning-lab-episode-5-cifar-100-a557e19219ba))

The [CIFAR-100 dataset](https://www.cs.toronto.edu/~kriz/cifar.html) is the bigger brother of the CIFAR-10 dataset. As the name suggests, CIFAR-100 includes 100 class labels versus the 10 class labels of CIFAR-10.

While there are _more_ class labels in CIFAR-100, there are actually _fewer images per class_ (CIFAR-10 has 6,000 images per class while CIFAR-100 only has 600 images per class).

CIFAR-100 is, therefore, a _more challenging_ dataset than CIFAR-10.

In this section, we’ll investigate Adam vs. Rectified Adam’s performance on the CIFAR-100 dataset.

#### CIFAR-100 – MiniVGGNet

[![Image 20](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20771'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/minivggnet_cifar100.png)

**Figure 18:** Will RAdam stand up to Adam as a preferable deep learning optimizer? How does Rectified Adam stack up to SGD? In this experiment (one of 24), we train MiniVGGNet on the CIFAR-100 dataset and analyze the results.

Let’s apply Adam and Rectified Adam to the MiniVGGNet architecture trained on CIFAR-100.

Below is the output from the **Adam optimizer:**

               precision    recall  f1-score   support

        apple       0.94      0.76      0.84       100
aquarium\_fish       0.69      0.66      0.67       100
         baby       0.56      0.45      0.50       100
         bear       0.45      0.22      0.30       100
       beaver       0.31      0.14      0.19       100
          bed       0.48      0.59      0.53       100
          bee       0.60      0.69      0.64       100
       beetle       0.51      0.49      0.50       100
      bicycle       0.50      0.65      0.57       100
       bottle       0.74      0.63      0.68       100
         bowl       0.51      0.38      0.44       100
          boy       0.45      0.37      0.41       100
       bridge       0.64      0.68      0.66       100
          bus       0.42      0.57      0.49       100
    butterfly       0.52      0.50      0.51       100
        camel       0.61      0.33      0.43       100
          can       0.44      0.68      0.54       100
       castle       0.74      0.71      0.72       100
  caterpillar       0.78      0.40      0.53       100
       cattle       0.58      0.48      0.52       100
        chair       0.72      0.80      0.76       100
   chimpanzee       0.74      0.64      0.68       100
        clock       0.39      0.62      0.48       100
        cloud       0.88      0.46      0.61       100
    cockroach       0.80      0.66      0.73       100
        couch       0.56      0.27      0.36       100
         crab       0.43      0.52      0.47       100
    crocodile       0.34      0.32      0.33       100
          cup       0.74      0.73      0.73       100
..."d" - "t" classes omitted for brevity
     wardrobe       0.67      0.87      0.76       100
        whale       0.67      0.58      0.62       100
  willow\_tree       0.52      0.44      0.48       100
         wolf       0.40      0.48      0.44       100
        woman       0.39      0.19      0.26       100
         worm       0.66      0.56      0.61       100

    micro avg       0.53      0.53      0.53     10000
    macro avg       0.58      0.53      0.53     10000
 weighted avg       0.58      0.53      0.53     10000

And here is the output from **Rectified Adam:**

               precision    recall  f1-score   support

        apple       0.82      0.70      0.76       100
aquarium\_fish       0.57      0.46      0.51       100
         baby       0.55      0.26      0.35       100
         bear       0.22      0.11      0.15       100
       beaver       0.17      0.18      0.17       100
          bed       0.47      0.37      0.42       100
          bee       0.49      0.47      0.48       100
       beetle       0.32      0.52      0.39       100
      bicycle       0.36      0.64      0.46       100
       bottle       0.74      0.40      0.52       100
         bowl       0.47      0.29      0.36       100
          boy       0.54      0.26      0.35       100
       bridge       0.38      0.43      0.40       100
          bus       0.34      0.35      0.34       100
    butterfly       0.40      0.34      0.37       100
        camel       0.37      0.19      0.25       100
          can       0.57      0.45      0.50       100
       castle       0.50      0.57      0.53       100
  caterpillar       0.50      0.21      0.30       100
       cattle       0.47      0.35      0.40       100
        chair       0.54      0.72      0.62       100
   chimpanzee       0.59      0.47      0.53       100
        clock       0.29      0.37      0.33       100
        cloud       0.77      0.60      0.67       100
    cockroach       0.57      0.64      0.60       100
        couch       0.42      0.18      0.25       100
         crab       0.25      0.50      0.33       100
    crocodile       0.30      0.28      0.29       100
          cup       0.71      0.60      0.65       100
..."d" - "t" classes omitted for brevity
     wardrobe       0.61      0.82      0.70       100
        whale       0.57      0.39      0.46       100
  willow\_tree       0.36      0.27      0.31       100
         wolf       0.32      0.39      0.35       100
        woman       0.35      0.09      0.14       100
         worm       0.62      0.32      0.42       100

    micro avg       0.41      0.41      0.41     10000
    macro avg       0.46      0.41      0.41     10000
 weighted avg       0.46      0.41      0.41     10000

The Adam optimizer is the clear winner (**58% accuracy**) over Rectified Adam (46% accuracy).

And just like in our CIFAR-10 experiments, we can likely improve our model performance further by relaxing regularization and increasing model capacity.

#### CIFAR-100 – GoogLeNet

[![Image 21](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20771'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/googlenet_cifar100.png)

**Figure 19:** Adam vs. RAdam optimizer on the CIFAR-100 dataset using GoogLeNet.

Let’s now perform the same experiment, only this time use GoogLeNet.

Here’s the output from the **Adam optimizer:**

               precision    recall  f1-score   support

        apple       0.95      0.80      0.87       100
aquarium\_fish       0.88      0.66      0.75       100
         baby       0.59      0.39      0.47       100
         bear       0.47      0.28      0.35       100
       beaver       0.20      0.53      0.29       100
          bed       0.79      0.56      0.65       100
          bee       0.78      0.69      0.73       100
       beetle       0.56      0.58      0.57       100
      bicycle       0.91      0.63      0.75       100
       bottle       0.80      0.71      0.75       100
         bowl       0.46      0.37      0.41       100
          boy       0.49      0.47      0.48       100
       bridge       0.80      0.61      0.69       100
          bus       0.62      0.60      0.61       100
    butterfly       0.34      0.64      0.44       100
        camel       0.93      0.37      0.53       100
          can       0.42      0.69      0.52       100
       castle       0.94      0.50      0.65       100
  caterpillar       0.28      0.77      0.41       100
       cattle       0.56      0.55      0.55       100
        chair       0.85      0.77      0.81       100
   chimpanzee       0.95      0.58      0.72       100
        clock       0.56      0.62      0.59       100
        cloud       0.88      0.68      0.77       100
    cockroach       0.82      0.74      0.78       100
        couch       0.66      0.40      0.50       100
         crab       0.40      0.72      0.52       100
    crocodile       0.36      0.47      0.41       100
          cup       0.65      0.68      0.66       100
..."d" - "t" classes omitted for brevity
     wardrobe       0.86      0.82      0.84       100
        whale       0.40      0.80      0.53       100
  willow\_tree       0.46      0.62      0.53       100
         wolf       0.86      0.37      0.52       100
        woman       0.56      0.31      0.40       100
         worm       0.79      0.57      0.66       100

    micro avg       0.56      0.56      0.56     10000
    macro avg       0.66      0.56      0.57     10000
 weighted avg       0.66      0.56      0.57     10000

And here is the output from **Rectified Adam:**

               precision    recall  f1-score   support

        apple       0.93      0.76      0.84       100
aquarium\_fish       0.72      0.77      0.74       100
         baby       0.53      0.54      0.53       100
         bear       0.47      0.26      0.34       100
       beaver       0.26      0.22      0.24       100
          bed       0.53      0.49      0.51       100
          bee       0.52      0.62      0.56       100
       beetle       0.50      0.55      0.52       100
      bicycle       0.67      0.79      0.72       100
       bottle       0.78      0.62      0.69       100
         bowl       0.41      0.42      0.41       100
          boy       0.45      0.41      0.43       100
       bridge       0.59      0.72      0.65       100
          bus       0.45      0.53      0.49       100
    butterfly       0.27      0.58      0.37       100
        camel       0.56      0.50      0.53       100
          can       0.58      0.68      0.63       100
       castle       0.81      0.73      0.77       100
  caterpillar       0.51      0.52      0.51       100
       cattle       0.56      0.59      0.58       100
        chair       0.68      0.76      0.72       100
   chimpanzee       0.83      0.73      0.78       100
        clock       0.46      0.56      0.50       100
        cloud       0.88      0.69      0.78       100
    cockroach       0.79      0.68      0.73       100
        couch       0.44      0.39      0.41       100
         crab       0.46      0.47      0.46       100
    crocodile       0.40      0.40      0.40       100
          cup       0.76      0.62      0.68       100
..."d" - "t" classes omitted for brevity
     wardrobe       0.76      0.87      0.81       100
        whale       0.56      0.61      0.59       100
  willow\_tree       0.65      0.30      0.41       100
         wolf       0.61      0.55      0.58       100
        woman       0.39      0.30      0.34       100
         worm       0.62      0.61      0.62       100

    micro avg       0.57      0.57      0.57     10000
    macro avg       0.59      0.57      0.57     10000
 weighted avg       0.59      0.57      0.57     10000

The Adam optimizer obtains **66% accuracy**, better than Rectified Adam’s 59%.

However, looking at **Figure 19** we can see that the validation loss from Adam is quite unstable — towards the end of training validation loss even starts to increase, a sign of overfitting.

#### CIFAR-100 – ResNet

[![Image 22](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20771'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/resnet_cifar100.png)

**Figure 20:** Training a ResNet model on the CIFAR-100 dataset using both RAdam and Adam for comparison. Which deep learning optimizer is actually better for this experiment?

Below we can find the output of training ResNet using **Adam** on the CIFAR-100 dataset:

               precision    recall  f1-score   support

        apple       0.80      0.89      0.84       100
aquarium\_fish       0.86      0.75      0.80       100
         baby       0.75      0.40      0.52       100
         bear       0.71      0.29      0.41       100
       beaver       0.40      0.40      0.40       100
          bed       0.91      0.59      0.72       100
          bee       0.71      0.76      0.73       100
       beetle       0.82      0.42      0.56       100
      bicycle       0.54      0.89      0.67       100
       bottle       0.93      0.62      0.74       100
         bowl       0.75      0.36      0.49       100
          boy       0.43      0.49      0.46       100
       bridge       0.54      0.78      0.64       100
          bus       0.68      0.48      0.56       100
    butterfly       0.34      0.71      0.46       100
        camel       0.72      0.68      0.70       100
          can       0.69      0.60      0.64       100
       castle       0.96      0.69      0.80       100
  caterpillar       0.57      0.62      0.60       100
       cattle       0.91      0.51      0.65       100
        chair       0.79      0.82      0.80       100
   chimpanzee       0.80      0.79      0.79       100
        clock       0.41      0.86      0.55       100
        cloud       0.89      0.74      0.81       100
    cockroach       0.85      0.78      0.81       100
        couch       0.73      0.44      0.55       100
         crab       0.42      0.70      0.53       100
    crocodile       0.47      0.55      0.51       100
          cup       0.88      0.75      0.81       100
..."d" - "t" classes omitted for brevity
     wardrobe       0.79      0.85      0.82       100
        whale       0.58      0.75      0.65       100
  willow\_tree       0.71      0.37      0.49       100
         wolf       0.79      0.64      0.71       100
        woman       0.42      0.49      0.45       100
         worm       0.48      0.80      0.60       100

    micro avg       0.63      0.63      0.63     10000
    macro avg       0.68      0.63      0.63     10000
 weighted avg       0.68      0.63      0.63     10000

And here is the output of **Rectified Adam:**

               precision    recall  f1-score   support

        apple       0.86      0.72      0.78       100
aquarium\_fish       0.56      0.62      0.59       100
         baby       0.49      0.43      0.46       100
         bear       0.36      0.20      0.26       100
       beaver       0.27      0.17      0.21       100
          bed       0.45      0.42      0.43       100
          bee       0.54      0.61      0.57       100
       beetle       0.47      0.55      0.51       100
      bicycle       0.45      0.69      0.54       100
       bottle       0.64      0.54      0.59       100
         bowl       0.39      0.31      0.35       100
          boy       0.43      0.35      0.38       100
       bridge       0.52      0.67      0.59       100
          bus       0.34      0.47      0.40       100
    butterfly       0.33      0.39      0.36       100
        camel       0.47      0.37      0.41       100
          can       0.49      0.55      0.52       100
       castle       0.76      0.67      0.71       100
  caterpillar       0.43      0.43      0.43       100
       cattle       0.56      0.45      0.50       100
        chair       0.63      0.78      0.70       100
   chimpanzee       0.70      0.71      0.71       100
        clock       0.38      0.49      0.43       100
        cloud       0.80      0.61      0.69       100
    cockroach       0.73      0.72      0.73       100
        couch       0.49      0.36      0.42       100
         crab       0.27      0.45      0.34       100
    crocodile       0.32      0.26      0.29       100
          cup       0.63      0.49      0.55       100
..."d" - "t" classes omitted for brevity
     wardrobe       0.68      0.84      0.75       100
        whale       0.53      0.54      0.54       100
  willow\_tree       0.60      0.29      0.39       100
         wolf       0.38      0.35      0.36       100
        woman       0.33      0.29      0.31       100
         worm       0.59      0.63      0.61       100

    micro avg       0.50      0.50      0.50     10000
    macro avg       0.51      0.50      0.49     10000
 weighted avg       0.51      0.50      0.49     10000

The Adam optimizer (**68% accuracy**) crushes Rectified Adam (51% accuracy) here, but we need to be careful of overfitting. As **Figure 20** shows there is quite the divergence between training and validation loss when using the Adam optimizer.

But on the other hand, Rectified Adam really stagnates past epoch 20.

I would be inclined to go with the Adam optimized model here as it obtains significantly higher accuracy; however, I would suggest running some generalization tests using both the Adam and Rectified Adam versions of the model.

### What can we take away from these experiments?

One of the first takeaways comes from looking at the training plots of the experiments — **using the Rectified Adam optimizer can lead to more stable training.**

When training with Rectified Adam we see there are _significantly_ fewer fluctuations, spikes, and drops in validation loss (as compared to standard Adam).

**Furthermore, the Rectified Adam validation loss is much more likely to follow training loss, in some cases near exactly.**

Keep in mind that **raw accuracy isn’t everything** when it comes to training your own custom neural networks — **stability matters as well** as it goes hand-in-hand with **generalization**.

Whenever I’m training a custom CNN I’m not only looking for high accuracy models, I’m also looking for stability. Stability typically implies that a model is converging nicely and will ideally generalize well.

In this regard, **Rectified Adam delivers on its promises** from the Liu et al. paper.

**Secondly, you should note that Adam obtains lower loss than Rectified Adam in _every single experiment_.**

This behavior is not necessarily a bad thing — it _could_ imply that Rectified Adam is generalizing better, but it’s hard to say without running further experiments using images _outside_ the respective training and testing sets.

**Again, keep in mind that lower loss is _not_ necessarily a better model!** When you encounter very low loss (especially loss near zero) your model may be overfitting to your training set.

### You need to obtain mastery level experience operating these three optimizers

[![Image 23](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20400'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2019/10/rectified_adam_comparison_race_car.jpg)

**Figure 21:** Mastering deep learning optimizers is like driving a car. You know your car and you drive it well no matter the road condition. On the other hand, if you get in an unfamiliar car, something doesn’t feel right until you have a few hours cumulatively behind the wheel. Optimizers are no different. I suggest that SGD be your daily driver until you are comfortable trying alternatives. Then you can mix in RMSprop and Adam. Learn how to use them before jumping into the latest deep learning optimizer.

Becoming familiar with a given optimization algorithm is similar to mastering how to drive a car — **you drive your own car better than other people’s cars because you’ve spent so much time driving it;** you understand your car and its intricacies.

Often times, a given optimizer is chosen to train a network on a dataset _not_ because the optimizer itself is better, but because the _driver_ (i.e., you, the deep learning practitioner) is _more familiar with the optimizer_ and understands the “art” behind tuning its respective parameters.

As a deep learning practitioner you should gain experience operating a wide variety of optimizers, **but in my opinion, you should focus your efforts on learning how to train networks using the _three_ following optimizers:**

1.  SGD
2.  RMSprop
3.  Adam

You might be surprised to see SGD is included in this list — isn’t SGD an older, less efficient optimizer than the newer adaptive methods, including Adam, Adagrad, Adadelta, etc.?

Yes, it absolutely is.

**But here’s the thing — nearly _every_ state-of-the-art computer vision model is trained using SGD.**

Consider the ImageNet classification challenge for example:

*   [AlexNet](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf) (there’s no mention in the paper but both the official implementation and CaffeNet used SGD)
*   [VGGNet](https://arxiv.org/pdf/1409.1556) (Section 3.1, _Training_)
*   [ResNet](https://arxiv.org/pdf/1512.03385.pdf) (Section 3.4, _Implementation_)
*   [SqueezeNet](https://arxiv.org/pdf/1602.07360.pdf) (it’s not mentioned in the paper, but SGD was used in their [solver.prototxt](https://github.com/DeepScale/SqueezeNet/blob/master/SqueezeNet_v1.0/solver.prototxt))

**Every single one of those classification networks was trained using SGD.**

Now let’s consider the object detection networks trained on the [COCO dataset](http://cocodataset.org/#home):

*   [Faster R-CNN](https://arxiv.org/pdf/1506.01497.pdf) (Section 3.1.3, _Training RPNs_ and Section 3.2, _Sharing Features for RPN and Fast R-CNN_, as well as [Girshick et al.’s GitHub solver.prototxt](https://github.com/rbgirshick/py-faster-rcnn/blob/master/models/coco/VGG16/fast_rcnn/solver.prototxt))
*   [Single Shot Detectors](https://arxiv.org/pdf/1512.02325.pdf) (Section 3, _Experimental Results_)
*   [RetinaNet](https://arxiv.org/pdf/1708.02002.pdf) (Section 4.1, _Optimization_)
*   [YOLO](https://arxiv.org/pdf/1506.02640.pdf) (I can’t find mention of it in the paper but I believe Joseph Redmon used SGD, correct me if I’m wrong)

**You guessed it — SGD was used to train all of them.**

Yes, SGD may the “old, unsexy” optimizer compared to its younger counterparts, **but here’s the thing, standard SGD _just works_.**

That’s not to say that you shouldn’t learn how to use other optimizers — **you absolutely should!**

But before you go down that rabbit hole, obtain a mastery level of SGD first. From there, start exploring other optimizers — I typically recommend RMSprop and Adam.

**And if you find Adam is working well, consider replacing Adam with Rectified Adam to see if you can get an additional boost in accuracy** (sort of like how replacing ReLUs with ELUs can usually give you a small boost).

Once you understand how to use those optimizers on a variety of datasets, continue your studies and explore other optimizers as well.

All that said, **_if you’re new to deep learning_, don’t immediately try jumping into the more “advanced” optimizers** — you’ll only run into trouble later in your deep learning career.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In this tutorial, we investigated the claims from Liu et al. that **the Rectified Adam optimizer outperforms the standard Adam optimizer** in terms of:

1.  **Better accuracy** (or at least identical accuracy when compared to Adam)
2.  And in **fewer epochs** than standard Adam

To evaluate those claims we trained three CNN models:

1.  ResNet
2.  GoogLeNet
3.  MiniVGGNet

These models were trained on four datasets:

1.  MNIST
2.  Fashion MNIST
3.  CIFAR-10
4.  CIFAR-100

Each combination of the model architecture and dataset were trained using two optimizers:

*   Adam
*   Rectified Adam

In total, we ran _3 x 4 x 2 = 24_ different experiments used to compare standard Adam to Rectified Adam.

The result?

**In each and every experiment Rectified Adam either performed _worse_ or obtained _identical accuracy_ compared to standard Adam.**

That said, **training with Rectified Adam was _more stable_ than standard Adam, likely implying that Rectified Adam _could_ generalize better** (but additional experiments would be required to validate that claim).

Liu et al.’s study of warmup can be utilized in adaptive learning rate optimizers and will likely help future researchers build on their work and create _even better_ optimizers.

**For the time being, my personal opinion is that you’re better off sticking with standard Adam for your initial experiments.** If you find that Adam is working well for your experiments, substitute in Rectified Adam to see if you can improve your accuracy.

You should especially try to use the Rectified Adam optimizer if you notice that Adam is working well, but you need better generalization.

The second takeaway from this guide is that **you should obtain mastery level experience operating these three optimizers:**

1.  SGD
2.  RMSprop
3.  Adam

You should _especially_ learn how to operate SGD.

Yes, SGD is “less sexy” compared to the newer adaptive learning rate methods, **but nearly every computer vision state-of-the-art architecture has been trained using it.**

Learn how to operate these three optimizers _first_.

Once you have a good understanding of how they work and how to tune their respective hyperparameters, _then_ move on to other optimizers.

If you need help learning how to use these optimizers and tune their hyperparameters, be sure to refer to [_Deep Learning for Computer Vision with Python_](https://pyimagesearch.com/deep-learning-computer-vision-python-book/) where I cover my tips, suggestions, and best practices in detail.

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _just enter your email address in the form below!_**

![Image 24](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!