A Simple CNN: Multi Image Classifier | by Iftekher Mamun | Towards Data Science

Computer vision and neural networks are the hot new IT of machine learning techniques. With advances of neural networks and an ability to read images as pixel density numbers, numerous companies are relying on this technique for more data. For example, speed camera uses computer vision to take pictures of license plate of cars who are going above the speeding limit and match the license plate number with their known database to send the ticket to. Although this is more related to Object Character Recognition than Image Classification, both uses computer vision and neural networks as a base to work.
A more realistic example of image classification would be Facebook tagging algorithm. When you upload an album with people in them and tag them in Facebook, the tag algorithm breaks down the person’s picture pixel location and store it in the database. Because each picture has its own unique pixel location, it is relatively easy for the algorithm to realize who is who based on previous pictures located in the database. Of course the algorithm can make mistake from time to time, but the more you correct it, the better it will be at identifying your friends and automatically tag them for you when you upload. However, the Facebook tag algorithm is built with artificial intelligence in mind. This means that the tagging algorithm is capable of learning based on our input and make better classifications in the future.
We will not focus on the AI aspect, but rather on the simplest way to make an image classification algorithm. The only difference between our model and Facebook’s will be that ours cannot learn from it’s mistake unless we fix it. However, for a simple neural network project, it is sufficient.
Since it is unethical to use pictures of people, we will be using animals to create our model. My friend Vicente and I have already made a project on this, so I will be using that as the example to follow through. The GitHub is linked at the end.
The first step is to gather the data. This in my opinion, will be the most difficult and annoying aspect of the project. Remember that the data must be labeled. Thankfully, Kaggle has labeled images that we can easily download. The set we worked with can be found here: animal-10 dataset. If your dataset is not labeled, this can be be time consuming as you would have to manually create new labels for each categories of images. Another method is to create new labels and only move 100 pictures into their proper labels, and create a classifier like the one we will and have that machine classify the images. This will lead to errors in classification, so you may want to check manually after each run, and this is where it becomes time consuming.
Now that we have our datasets stored safely in our computer or cloud, let’s make sure we have a training data set, a validation data set, and a testing data set. Training data set would contain 85–90% of the total labeled data. This data would be used to train our machine about the different types of images we have. Validation data set would contain 5–10% of the total labeled data. This will test how well our machine performs against known labeled data. The testing data set would contain the rest of the data in an unlabeled format. This testing data will be used to test how well our machine can classify data it has never seen. The testing data can also just contain images from Google that you have downloaded, as long as it make sense to the topic you are classifying.
Let’s import all the necessary libraries first:
import pandas as pdimport numpy as np import itertoolsimport kerasfrom sklearn import metricsfrom sklearn.metrics import confusion_matrixfrom keras.preprocessing.image import ImageDataGenerator, img_to_array, load_img from keras.models import Sequential from keras import optimizersfrom keras.preprocessing import imagefrom keras.layers import Dropout, Flatten, Dense from keras import applications from keras.utils.np_utils import to_categorical import matplotlib.pyplot as plt import matplotlib.image as mpimg%matplotlib inlineimport math import datetimeimport time
Defining Dimensions and locating images:
#Default dimensions we found onlineimg_width, img_height = 224, 224  #Create a bottleneck filetop_model_weights_path = ‘bottleneck_fc_model.h5’# loading up our datasetstrain_data_dir = ‘data/train’ validation_data_dir = ‘data/validation’ test_data_dir = ‘data/test’ # number of epochs to train top model epochs = 7 #this has been changed after multiple model run # batch size used by flow_from_directory and predict_generator batch_size = 50 
In this step, we are defining the dimensions of the image. Depending on your image size, you can change it but we found best that 224, 224 works best. Then we created a bottleneck file system. This will be used to convert all image pixels in to their number (numpy array) correspondent and store it in our storage system. Once we run this, it will take from half hours to several hours depending on the numbers of classifications and how many images per classifications. Then we simply tell our program where each images are located in our storage so the machine knows where is what. Finally, we define the epoch and batch sizes for our machine. For neural networks, this is a key step. We found that this set of pairing was optimal for our machine learning models but again, depending on the number of images that needs to be adjusted.
Importing transfer learning model VGG16:
#Loading vgc16 modelvgg16 = applications.VGG16(include_top=False, weights=’imagenet’)datagen = ImageDataGenerator(rescale=1. / 255) #needed to create the bottleneck .npy files
This is importing the transfer learning aspect of the convolutional neural network. Transfer learning is handy because it comes with pre-made neural networks and other necessary components that we would otherwise have to create. There are many transfer learning model. I particularly like VGG16 as it uses only 11 convolutional layers and pretty easy to work with. However, if you are working with larger image files, it is best to use more layers, so I recommend resnet50, which contains 50 convolutional layers.
For our image classifier, we only worked with 6 classifications so using transfer learning on those images did not take too long, but remember that the more images and classifications, the longer this next step will take. But thankfully since you only need to convert the image pixels to numbers only once, you only have to do the next step for each training, validation and testing only once- unless you have deleted or corrupted the bottleneck file.
Creation of the weights and feature using VGG16:
#__this can take an hour and half to run so only run it once. #once the npy files have been created, no need to run again. Convert this cell to a code cell to run.__start = datetime.datetime.now() generator = datagen.flow_from_directory(     train_data_dir,     target_size=(img_width, img_height),     batch_size=batch_size,     class_mode=None,     shuffle=False)  nb_train_samples = len(generator.filenames) num_classes = len(generator.class_indices)  predict_size_train = int(math.ceil(nb_train_samples / batch_size))  bottleneck_features_train = vgg16.predict_generator(generator, predict_size_train)  np.save(‘bottleneck_features_train.npy’, bottleneck_features_train)end= datetime.datetime.now()elapsed= end-startprint (‘Time: ‘, elapsed)
Since we are making a simple image classifier, there is no need to change the default settings. Just follow the above steps for the training, validation, and testing directory we created above. However, you can add different features such as image rotation, transformation, reflection and distortion.
Once the files have been converted and saved to the bottleneck file, we load them and prepare them for our convolutional neural network. This is also a good way to make sure all your data have been loaded into bottleneck file. Remember to repeat this step for validation and testing set as well.
Creating a bottleneck file for the training data. (Same step for validation and testing):
#training datagenerator_top = datagen.flow_from_directory(    train_data_dir,    target_size=(img_width, img_height),    batch_size=batch_size,    class_mode=’categorical’,    shuffle=False)  nb_train_samples = len(generator_top.filenames) num_classes = len(generator_top.class_indices)  # load the bottleneck features saved earlier train_data = np.load(‘bottleneck_features_train.npy’)  # get the class labels for the training data, in the original order train_labels = generator_top.classes  # convert the training labels to categorical vectors train_labels = to_categorical(train_labels, num_classes=num_classes)
Creating our Convolutional Neural Network code:
#This is the best model we found. For additional models, check out I_notebook.ipynbstart = datetime.datetime.now()model = Sequential() model.add(Flatten(input_shape=train_data.shape[1:])) model.add(Dense(100, activation=keras.layers.LeakyReLU(alpha=0.3))) model.add(Dropout(0.5)) model.add(Dense(50, activation=keras.layers.LeakyReLU(alpha=0.3))) model.add(Dropout(0.3)) model.add(Dense(num_classes, activation=’softmax’))model.compile(loss=’categorical_crossentropy’,   optimizer=optimizers.RMSprop(lr=1e-4),   metrics=[‘acc’])history = model.fit(train_data, train_labels,    epochs=7,   batch_size=batch_size,    validation_data=(validation_data, validation_labels))model.save_weights(top_model_weights_path)(eval_loss, eval_accuracy) = model.evaluate(     validation_data, validation_labels, batch_size=batch_size,     verbose=1)print(“[INFO] accuracy: {:.2f}%”.format(eval_accuracy * 100)) print(“[INFO] Loss: {}”.format(eval_loss)) end= datetime.datetime.now()elapsed= end-startprint (‘Time: ‘, elapsed)
Now we create our model. First step is to initialize the model with Sequential(). After that we flatten our data and add our additional 3 (or more) hidden layers. This step is fully customizable to what you want. We made several different models with different drop out, hidden layers and activation. But since this is a labeled categorical classification, the final activation must always be softmax. It is also best for loss to be categorical crossenthropy but everything else in model.compile can be changed. Then after we have created and compiled our model, we fit our training and validation data to it with the specifications we mentioned earlier. Finally, we create an evaluation step, to check for the accuracy of our model training set versus validation set.
This is our model now training the data and then validating it. An epoch is how many times the model trains on our whole data set. Batch can be explained as taking in small amounts, train and take some more. Each epoch must finish all batch before moving to the next epoch. Training with too little epoch can lead to underfitting the data and too many will lead to overfitting the data. You also want a loss that is as low as possible. The pictures below will show the accuracy and loss of our data set
Code for visualization of the Accuracy and Loss:
#Graphing our training and validationacc = history.history[‘acc’]val_acc = history.history[‘val_acc’]loss = history.history[‘loss’]val_loss = history.history[‘val_loss’]epochs = range(len(acc))plt.plot(epochs, acc, ‘r’, label=’Training acc’)plt.plot(epochs, val_acc, ‘b’, label=’Validation acc’)plt.title(‘Training and validation accuracy’)plt.ylabel(‘accuracy’) plt.xlabel(‘epoch’)plt.legend()plt.figure()plt.plot(epochs, loss, ‘r’, label=’Training loss’)plt.plot(epochs, val_loss, ‘b’, label=’Validation loss’)plt.title(‘Training and validation loss’)plt.ylabel(‘loss’) plt.xlabel(‘epoch’)plt.legend()plt.show()
This picture below shows how well the machine we just made can predict against unseen data. Notice it says that its testing on test_data. Accuracy is the second number. However, this is not the only method of checking how well our machines performed
There are two great methods to see how well your machine can predict or classify. One of them is the classification metrics and the other is the confusion matrix.
To use classification metrics, we had to convert our testing data into a different numpy format, numpy array, to read. That is all the first line of code is doing. The second cell block takes in the converted code and run it through the built in classification metrics to give us a neat result. Please note that unless you manually label your classes here, you will get 0–5 as the classes instead of the animals. The important factors here are precision and f1-score. The higher the score the better your model is. Here is a great blog on medium that explains what each of those are.
Now to make a confusion matrix. There are lots on online tutorial on how to make great confusion matrix. Ours is a variation of some we found online
#Since our data is in dummy format we put the numpy array into a dataframe and call idxmax axis=1 to return the column# label of the maximum value thus creating a categorical variable#Basically, flipping a dummy variable back to it’s categorical variablecategorical_test_labels = pd.DataFrame(test_labels).idxmax(axis=1)categorical_preds = pd.DataFrame(preds).idxmax(axis=1)confusion_matrix= confusion_matrix(categorical_test_labels, categorical_preds)#To get better visual of the confusion matrix:def plot_confusion_matrix(cm, classes,   normalize=False,   title=’Confusion matrix’,   cmap=plt.cm.Blues): #Add Normalization Option ‘’’prints pretty confusion metric with normalization option ‘’’   if normalize:     cm = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]     print(“Normalized confusion matrix”)   else:     print(‘Confusion matrix, without normalization’) # print(cm)    plt.imshow(cm, interpolation=’nearest’, cmap=cmap)   plt.title(title)   plt.colorbar()   tick_marks = np.arange(len(classes))   plt.xticks(tick_marks, classes, rotation=45)   plt.yticks(tick_marks, classes)    fmt = ‘.2f’ if normalize else ‘d’   thresh = cm.max() / 2.   for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):      plt.text(j, i, format(cm[i, j], fmt), horizontalalignment=”center”, color=”white” if cm[i, j] > thresh else “black”)    plt.tight_layout()   plt.ylabel(‘True label’)   plt.xlabel(‘Predicted label’) 
The numpy array we created before is placed inside a dataframe. Confusion matrix works best on dataframes. The 3rd cell block with multiple iterative codes is purely for color visuals. The only important code functionality there would be the ‘if normalize’ line as it standardizes the data.
As we can see in our standardized data, our machine is pretty good at classifying which animal is what. Chickens were misclassified as butterflies most likely due to the many different types of pattern on butterflies. In addition, butterflies was also misclassified as spiders because of probably the same reason. Both elephants and horses are rather big animals, so their pixel distribution may have been similar.
The final phase is testing on images. The cell blocks below will accomplish that:
def read_image(file_path):   print(“[INFO] loading and preprocessing image...”)    image = load_img(file_path, target_size=(224, 224))    image = img_to_array(image)    image = np.expand_dims(image, axis=0)   image /= 255.    return imagedef test_single_image(path):  animals = [‘butterflies’, ‘chickens’, ‘elephants’, ‘horses’, ‘spiders’, ‘squirells’]  images = read_image(path)  time.sleep(.5)  bt_prediction = vgg16.predict(images)   preds = model.predict_proba(bt_prediction)  for idx, animal, x in zip(range(0,6), animals , preds[0]):   print(“ID: {}, Label: {} {}%”.format(idx, animal, round(x*100,2) ))  print(‘Final Decision:’)  time.sleep(.5)  for x in range(3):   print(‘.’*(x+1))   time.sleep(.2)  class_predicted = model.predict_classes(bt_prediction)  class_dictionary = generator_top.class_indices   inv_map = {v: k for k, v in class_dictionary.items()}   print(“ID: {}, Label: {}”.format(class_predicted[0],  inv_map[class_predicted[0]]))   return load_img(path)path = ‘data/test/yourpicturename’test_single_image(path)
The first def function is letting our machine know that it has to load the image, change the size and convert it to an array. Second def function is using transfer learning’s prediction model and an iterative function to help predict the image properly. The path is where we define the image location and finally the test_single_image cell block will print out the final result, depending on the prediction from the second cell block. For this part, I will not post a picture so you can find out your own results. However, the GitHub link will be right below so feel free to download our code and see how well it compares to yours.