MNIST classification using different activation functions and optimizers with implementation— Accuracy Comparison
Tags: TensorFlow, Multiclass Classification, Mnist Dataset, Keras, Machine

MNIST classification using different activation functions and optimizers with implementation— Accuracy Comparison Marmikpatani Follow Nov 22 · 4 min read

I tried to create a model in Tensorflow version 2.3.1 using keras version 2.4.0 , which was trained on the MNIST dataset. This dataset contains 60000 images of handwritten digits from 0 to 9 each of size 28x28.

I have used 3 different activation function in the last layer (output layer)for this, namely softmax , sigmoid and tanh.

model1 = tf.keras.models.Sequential([

tf.keras.layers.Flatten(input_shape=[28, 28]),

tf.keras.layers.Dense(300, activation=”relu”),

tf.keras.layers.Dense(100, activation=”relu”),

tf.keras.layers.Dense(10, activation=”softmax”)

])

From the code above , the activation function for the last layer is softmax (recommended for multi-class classification)

model2 = tf.keras.models.Sequential([

tf.keras.layers.Flatten(input_shape=[28, 28]),

tf.keras.layers.Dense(300, activation=”relu”),

tf.keras.layers.Dense(100, activation=”relu”),

tf.keras.layers.Dense(10, activation=”sigmoid”)

])

From the code above , the activation function for the last layer is sigmoid (recommended for binary classification)

model3 = tf.keras.models.Sequential([

tf.keras.layers.Flatten(input_shape=[28, 28]),

tf.keras.layers.Dense(300, activation=”relu”),

tf.keras.layers.Dense(100, activation=”relu”),

tf.keras.layers.Dense(10, activation=”tanh”)

])

From the code above , the activation function for the last layer is tanh (recommended for binary classification)

For all the models created above same “sparse_categorical_crossentropy” loss functions and “SGD”(Stochastic gradient descent) optimizer was used.

LOSS_FUNCTION = “sparse_categorical_crossentropy”

OPTIMIZER = “SGD”

METRICS = [“accuracy”]

model1.compile(loss=LOSS_FUNCTION,

optimizer=OPTIMIZER,

metrics=METRICS)

Above code remains the same only ‘model1.compile‘ will change to ‘model2.compile’ and so on.

After doing this the model was trained for 30 epochs

EPOCHS = 30

VALIDATION_SET = (X_valid, y_valid)

history = model1.fit(X_train, y_train, epochs=EPOCHS,

validation_data=VALIDATION_SET)

Similarly ‘model1.fit’ will change

After 30 epochs

model1- loss: 0.0278 — accuracy: 0.9936 — val_loss: 0.0706 — val_accuracy: 0.9800 model2- loss: 0.0317 — accuracy: 0.9926 — val_loss: 0.0712 — val_accuracy: 0.9796 model3- loss: 3.8495 — accuracy: 0.0990 — val_loss: 3.7865 — val_accuracy: 0.0958

This shows that softmax and sigmoid(surprisingly) has performed well but tanh has an accuracy of 0.0990 which means the model has learned almost nothing. When the model was tested on the ‘X_test’ and ‘y_test’ data. Only the model3 (tanh) gave wrong prediction and sigmoid gave low probabilities ,such as

array([[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.47, 0. , 0. ],

[0. , 0. , 0.73, 0. , 0. , 0. , 0. , 0. , 0. , 0. ],

[0. , 0.26, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]],

dtype=float32)

#Here the array has 3 arrays inside it and each array shows the prediction, so the model says that the input data is 47 percent digit 7(based on the position) and so on

and softmax gave high probabilities.

array([[0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],

[0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],

[0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)

and for tahn the probabilities were

array([[ 1., 1., 1., -1., 1., 1., 1., 1., 1., 1.],

[ 1., 1., 1., -1., 1., 1., 1., 1., 1., 1.],

[ 1., 1., 1., -1., 1., 1., 1., 1., 1., 1.]], dtype=float32)

As you can see that model3 has performed bad in the test phase, it has predicted all to be the digit 0 wrongly.

Again the same was done but the optimizer used was ‘Adam’ and softmax gave the same probabilities(predictions) but sigmoid gave better probabilities compared to the previous sigmoid model and tanh showed no improvements

For sigmoid

X_new = X_test[:3]

y_proba = model2.predict(X_new)#where model2 uses sigmoid

y_proba.round(2)

Out[67]:

array([[0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],

[0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],

[0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)

For softmax

X_new = X_test[:3]

y_proba = model1.predict(X_new) # where model uses sofmax

y_proba.round(2)

Out[66]:

array([[0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],

[0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],

[0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)

And for tanh

X_new = X_test[:3]

y_proba = model3.predict(X_new)#where model3 uses tanh

y_proba.round(2)

Out[68]:

array([[ 1., 1., 1., -1., 1., 1., 1., 1., 1., 1.],

[ 1., 1., 1., -1., 1., 1., 1., 1., 1., 1.],

[ 1., 1., 1., -1., 1., 1., 1., 1., 1., 1.]], dtype=float32)

Understandings:

When we use softmax in case of multi-class classification we get a really good accuracy , but when tanh is used we get a poor accuracy , sigmoid and tanh both are used mainly when a problem of binary classification is faced. In this case when sigmoid was used with SGD the probabilities were low but still it could classify and when sigmoid was used with Adam the probabilities were high.(I don’t know how sigmoid works in this case)

Final Points

1. Use softmax in case of multi-class classification

2. Use sigmoid or tanh in case of binary classification