Classifying images with Neural Networks
By: Armando Delgado
Tags: Pytorch, TensorFlow, Image Classification, Comparative, Neural Networks

TensorFlow

We will start with TensorFlow. We start loading the data

from keras import * (x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

As said earlier, this dataset is included in the keras framework. Now, the shape of this loaded tensor is 60000 samples of 28x28, which makes a tridimensional tensor (60000, 28, 28). We may use a convolutional network, but doing it that way it would take a lot of time. Instead, we may convert this tensor to a (60000, 28*28) bidimensional tensor. Also, the data comes as 256 bit data, we are interested in normalizing this data, which we may do simply dividing by 255 (since it goes from 0 to 255).

#Normalizing data

x_train = x_train.astype("float32") / 255

x_test = x_test.astype("float32") / 255

# Reshaping (60000, 28, 28) into (60000, 28*28)

x_train = x_train.reshape(x_train.shape[0], x_train.shape[1]* x_train.shape[1])

x_test = x_test.reshape(x_test.shape[0], x_test.shape[1]*x_test.shape[1])

Now, we convert the ‘y’ part to categorical. Instead of having a single number represented (1, 2, 3…9) we have binary 10-shaped matrix, representing those single numbers using 0s and 1s.

Y shape before categorizing it

num_classes = 10

y_train = keras.utils.to_categorical(y_train, num_classes)

y_test = keras.utils.to_categorical(y_test, num_classes)

Y shape after categorization

Now, as we were saying earlier we use a simple ‘Dense’ based neural network. For this experiment we test different activation functions, optimizers and layer combinations.

First, 3 different neural networks using a 3 layer neural network, using the following combination:

model1 = Sequential()

model1.add(Dense(128, activation=’linear’))

model1.add(Dense(64, activation=’relu’))

model1.add(Dense(10, activation=’softmax’))

For each of these we use a different optimizer. Usually, for many problems I use Adamax or RMSprop as optimizers, this time I add SGD. For the loss measure I use ‘categorical_crossentropy’ and for the metrics on testing I use ‘accuracy’.

model1.compile(optimizer=keras.optimizers.Adamax(lr=0.03), loss=’categorical_crossentropy’, metrics = [‘accuracy’]) model2.compile(optimizer=keras.optimizers.RMSprop(lr=0.03), loss='categorical_crossentropy', metrics = ['accuracy']) model3.compile(optimizer=keras.optimizers.SGD(lr=0.03), loss='categorical_crossentropy', metrics = ['accuracy'])

Also, I try another combination of layers, adding a fourth layer, that would look like this

model61 = Sequential()

model61.add(Dense(256, activation='relu'))

model61.add(Dense(128, activation='linear'))

model61.add(Dense(64, activation='relu'))

model61.add(Dense(10, activation='softmax')) model61.compile(optimizer=keras.optimizers.SGD(lr=0.03), loss='categorical_crossentropy', metrics = ['accuracy'])

On the literature is recommended to use softmax or similar activation functions for classification problem, it is demonstrate during this exercise that softmax activations in the last layer end up getting us better results.

The best result overall would be the following:

model62 = Sequential()

model62.add(Dense(256, activation=’relu’))

model62.add(Dense(128, activation=’relu’))

model62.add(Dense(64, activation=’relu’))

model62.add(Dense(10, activation=’softmax’))

model62.compile(optimizer=keras.optimizers.SGD(lr=0.03), loss=’categorical_crossentropy’, metrics = [‘accuracy’]) model62.fit(x_train, y_train, epochs=5, verbose=1, batch_size=20)

score = model62.evaluate(x_test, y_test, verbose=0)

This model obtained a 97.5% (0.077 cross-entropy) and 98% accuracy respectively on test and train phases. Training and testing this model lasted 1'45", which is a pretty decent result, despite here we only iterate the model 5 times. Using 15 epochs instead of 5 we obtain a 99.8% accuracy on the training, but the testing accuracy only goes up to 98%, so it’s probably an innecessary overfitting.