Building a Convolutional Neural Network to Recognize Shaved vs UnShaved Faces | by Jacob Tadesse | Towards Data Science

In this tutorial, we will use an image dataset created by scrapping free stock photo sites. The image set contains about 2,000 images of individual people labeled as “shaved” or “unshaved”. We will combine computer vision and machine learning to classify images using a Convolutional Neural Network (CNN).
By the end of this tutorial, you will be able to:
Build a CNN model from scratch and using transfer learningVisualize model structure, hidden layers, and evaluation metricsSave your CNN Model for re-use, and/or deployment
Build a CNN model from scratch and using transfer learning
Visualize model structure, hidden layers, and evaluation metrics
Save your CNN Model for re-use, and/or deployment
To code along, download my image dataset from Google Drive. All image data is stored in a Pandas dataframe, including the Original Image Arrays, as well as the Resized Image Arrays for 700x700, 300x300, 150x150, and 50x50.
We will use pickle to “unpickle” and load the pandas dataframe. We use Keras to build the CNN from scratch and also leverage the VGG16 pre-trained model. We will use Pyplot to visualize the image data and also visualize the intermediate activations within the hidden layers of the CNN.
import pickleimport pandas as pdimport numpy as np   from numpy import asarrayfrom matplotlib import imagefrom matplotlib import pyplotfrom keras.utils import to_categoricalfrom keras.models import Sequentialfrom keras.layers import Dense, Conv2D, MaxPooling2D, Flatten, Dropoutfrom keras.callbacks import EarlyStopping# loading the data frame using pickledata = pickle.load( open( "data_pickle_final.p", "rb" ) )# creating a list of image arrays to previewsize_list = data.iloc(0)[0][-5:]# iterating through list to view imagesfor i in size_list:    pyplot.imshow(i)    pyplot.show();
Here we can see the images data saved in our image set. To ensure the images are not in a stacked order, we will shuffle the order of images before performing a train, test, evaluation split. This is possible using a built in pandas method, .sample().
# preview dataframe informationdisplay(data.info())# preview first 5 rows in dataframedisplay(data.head())# shuffle order of images and preview first 5 imagesresampled_data = data.sample(frac=1)resampled_data.head()
At this juncture we will select the image shape we will use going forward. In your own projects, you may have to scrape photos for your own needs, so please check out my article on Web Scraping Stock Images Using Google Selenium and Python. Okay, we will select images of size 300 x 300, and move right along.
The most common pixel format is the byte image, where this number is stored as an 8-bit integer giving a range of possible values from 0 to 255. Typically zero is taken to be black, and 255 is taken to be white. Values in between make up the different shades of gray. In our example we will be using color photos with that share the scale for red, green and blue, ie: ‘rbg’; where 0 is no light and 255 is maximum light. We will normalize the values by dividing each value by the maximum number in the range, and since we have each image stored as an array, we will reformat the features variables into tensors for our neural network.
# Normalizing the image arraysX = resampled_data.img_data_300X = X/255# Creating an empty tensor to store image arraystensor = np.zeros((X.shape[0],300,300,3))# Iterating through image arrays to update tensorfor idx, i in enumerate(X):    tensor[idx] = i# Checking the tensor shapetensor.shape
Here we will define our target variable. We will first convert our Labels into binary targets for our Neural Network, also known as one-hot encoding.
# creating list to store target codestarget = []# iterating through labels to change strings to numbersfor i in resampled_data['Labels']:  if i == 'shaved':    target.append(0)  elif i == 'unshaved':    target.append(1)
We will add the new binary targets to our dataframe an also define the target variable as ‘y’. Depending on when you download the DataFrame, this may have already been included in the DataFrame labeled, “Target”. If not, no worries. Just follow this script.
# updating dataframe to include targetresampled_data['target'] = target# defining target variabley = resampled_data.target# checking the shapey.shape
Now that we have shuffled our image dataset, defined our target and feature variables; it is time for us to perform our train-test-evaluation split. We will perform this using Sklearn’s train_test_split module. But performing the train-test-split, we will create a hold out set, using the last 100 images, for model evaluation.
# importing train-test-split module from sklearnfrom sklearn.model_selection import train_test_split# Performing train-test split for model trainingX_train, X_test, y_train, y_test = train_test_split(tensor[:-100],                                                     y[:-100],                                                     test_size=0.20,                                                     random_state=42)# check the size of the train/test splitfor i in [X_train, X_test, y_train, y_test]:    print(len(i))# Defining hold out data for evaluation evals_tensors = tensor[-100:]evals_targets = y[-100:]
A Convolutional Neural Network (CNN) is a Deep Learning algorithm used specifically for images, where the algorithm identifies patterns within the image, assigning importance to different patterns/objects in an image, using weights and biases to recognize differences and similarities amongst input images. These CNNs are trained based on labeled images. The job of a CNN is to decrease the pictures into a structure which is simpler to process, without losing weights/bias which are essential tp getting a the best results.
Here we will build the layers to our Convolutional Neural Network. We will start by adding a sequential layer; followed by our 2D convolutional layers, Max pooling layers, Dense layers, and Dropout layers.
Here is the code:
model = Sequential()model.add(Conv2D(32,(4,4),activation='relu',input_shape=(300,300,3)))model.add(Conv2D(32,(4,4),activation='relu', padding='same'))model.add(Conv2D(128,(4,4),activation='relu', padding='same'))model.add(Conv2D(128,(4,4),activation='relu', padding='same'))model.add(MaxPooling2D((2, 2)))model.add(Dropout(.3))model.add(Conv2D(256, (4,4), activation='relu', padding='same'))model.add(Conv2D(256, (4,4), activation='relu', padding='same'))model.add(Conv2D(128, (4,4), activation='relu', padding='same'))model.add(Conv2D(128, (4,4), activation='relu', padding='same'))model.add(MaxPooling2D((2, 2)))model.add(Dropout(.3))model.add(MaxPooling2D((2, 2)))model.add(Dense(1024, activation='relu'))model.add(Dense(512, activation='relu'))model.add(Dropout(.3))model.add(Flatten())model.add(Dense(32, activation='relu'))model.add(Dense(1, activation='sigmoid'))model.summary()
The 2D convolutional layers are what make CNN so powerful in image prediction. This layers uses a kernel/filter (4x4) to examine all pixels for each channel; starting left to right, top to bottom, and front to back; convolving through all pixels (either grayscale, RGB, etc). This filter will perform a dot product multiplication using the pixel data within the feature in order to extract features, such as edges, from the input image.
The first Conv2D layer captures basic features like edges, color, size, and more. As more layers are added to the network, the algorithm identifies complex features from the input image. We used the ReLU, or Rectified Linear Activation Function, to train our deep network, as ReLu tends to converge much more quickly and reliably than training with other activation functions.
The Max Pooling layer reduces the size of the Convolved Feature, decreases computational power required in dimensionality reduction, extracts dominant features, and also suppresses noise.
We will use the sigmoid activation function for our output layer because it exists between (0 to 1), and we want to predict the probability as an output as shaved or unshaved.
Now that we have built our CNN model, we will need to compile, fit and evaluate the model. We will use early stopping to help us identify the global minimum for our binary cross-entropy loss function. We will store the accuracy and loss metrics for each epoch in order to visualize the model performance over time.
# timing the trainingimport timestart_time = time.time()# compiling the modelmodel.compile(loss='binary_crossentropy',              optimizer='adam',              metrics=['accuracy'])# setting upearly stoppinges = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=5)# fitting the model to train/test datahistory = model.fit(X_train, y_train,                      batch_size=128, epochs=1000,                       validation_data=(X_test, y_test),                      callbacks=[es], verbose=1)# printing training timeprint("--- %s seconds ---" % (time.time() - start_time))# evaluating the modeltrain_loss, train_acc = model.evaluate(X_train, y_train)test_loss, test_acc = model.evaluate(X_test, y_test)# checking model performance by accuracy metricsprint('Train: %.3f, Test: %.3f' % (train_acc, test_acc))
Here we will use PyPlot to visualize our model’s performance during training and testing.
# plotting the loss during trainingplt.plot(history.history['loss'], label='train')ply.plot(history.history['val_loss'], label='test')plt.title('Training vs Test Loss')plt.xlabel('Epoch')plt.ylabel('Loss')plt.legend()plt.show()# plotting the accuracy during trainingplt.plot(history.history['accuracy'], label='train')plt.plot(history.history['val_accuracy'], label='test')plt.title('Training vs Test Accuracy')plt.xlabel('Epoch')plt.ylabel('Accuracy')plt.legend()plt.show()
Now that we have our model with pretty decent accuracy, we will use it to make predictions on our hold-out evaluation set.
The model has not seen this data, and should be able to make an accurate prediction. Let’s take a look at the evaluation targets vs the predictions, and also take a look at the images in the evaluation image set.
# making predictions using evaluation featuresres = model.predict(evals_tensors)# iterating through prediction results, and evaluation featuresfor i, j, k in zip(res, evals_tensors, evals_targets):    if i >= .5:      _ = 'Unshaved'      print('Prediction:', round(i[0],2), '-', _) # printing test    else:      _ = 'Shaved'      print('Prediction:', round(i[0],2), '-',_) # printing test    print('Actual:', k)    pyplot.imshow(j)    pyplot.show();
We can also utilize metrics from Sklearn to view the classification report and also view a confusion matrix comparing the predictions to the actual results. We will create both for our test set and hold out evaluation set below.
Here is the code:
from sklearn.metrics import classification_report, confusion_matrix# making predictionsY_pred = model.predict(X_test)Y_pred_train = model.predict(evals_tensors)# creating lists to store predictionsy_preds = []y_preds_train = []# iterating through results to update Test prediction listsfor i in [round(i[0],0) for i in Y_pred]:    if i > 0.5:        y_preds.append(1)    else:        y_preds.append(0)        # iterating through results to update Evaluation prediction listsfor i in [round(i[0],0) for i in Y_pred_train]:    if i > 0.5:        y_preds_train.append(1)    else:        y_preds_train.append(0)
Here is the code to print your confusion matrix and classification report.
# printing the confusion matrixprint('Confusion Matrix - Test Set')print(confusion_matrix(y_test.values.tolist(), y_preds))print('Confusion Matrix - Evaluation Set')print(confusion_matrix(evals_targets.values.tolist(), y_preds_train))# printing the classification reportprint('Classification Report - Test Set')print(classification_report(y_test.values.tolist(), y_preds, target_names=['Shaved','Unshaved']))print('Classification Report - Evaluation Set')print(classification_report(evals_targets.values.tolist(), y_preds_train, target_names=['Shaved','Unshaved']))
With so many hidden layers, it is hard to understand what the model is doing under the hood. While not necessary for the model to perform, it’s a good way to visualize the model structure and also view the intermediate activations.
This code will save your model structure as a PDF file:
# Saving the model structureimport pydotpydot.find_graphviz = lambda: Truefrom keras.utils import plot_modelplot_model(model, show_shapes=True, to_file='model_pdf/{}.pdf'.format('Model_Structure'))
This code will save an image for the intermediate activation layers:
# Saving intermediate activationsfrom keras import modelsimport math# Creating list of layer outputslayer_outputs = [layer.output for layer in model.layers[:9]]activation_model = models.Model(inputs=model.input, outputs=layer_outputs)activations = activation_model.predict(img_tensor)# Extract Layer Names for Labellinglayer_names = []for layer in activation_model.layers[:9]:    layer_names.append(layer.name)# Defining number of images per rowimages_per_row = 16# iterating through layer names and activationsfor layer_name, layer_activation in zip(layer_names, activations):     n_features = layer_activation.shape[-1]     size = layer_activation.shape[1]     n_cols = n_features // images_per_row     display_grid = np.zeros((size * n_cols, images_per_row * size))    for col in range(n_cols): # Tiles each filter into a big horizontal grid        for row in range(images_per_row):            channel_image = layer_activation[0,                                             :, :,                                             col * images_per_row + row]            channel_image -= channel_image.mean() # Post-processes the feature to make it visually palatable            channel_image /= channel_image.std()            channel_image *= 64            channel_image += 128            channel_image = np.clip(channel_image, 0, 255).astype('uint8')            display_grid[col * size : (col + 1) * size, # Displays the grid                         row * size : (row + 1) * size] = channel_image    scale = 1. / size    plt.figure(figsize=(scale * display_grid.shape[1],                        scale * display_grid.shape[0]))    plt.title(layer_name)    plt.grid(False)    plt.imshow(display_grid, aspect='auto', cmap='viridis')plt.savefig(f'intermediate_activation_visualizations/unshaved/{layer_name}')
Here we will leverage the imagenet weights form one of Kera’s pre-trained models, VGG16.
from keras.applications import vgg16from keras.models import Modelimport kerasinput_shape = (300, 300, 3)vgg = vgg16.VGG16(include_top=False, weights='imagenet',                                      input_shape=input_shape)output = vgg.layers[-1].outputoutput = keras.layers.Flatten()(output)vgg_model = Model(vgg.input, output)vgg_model.trainable = Falsefor layer in vgg_model.layers:    layer.trainable = False    import pandas as pdpd.set_option('max_colwidth', -1)layers = [(layer, layer.name, layer.trainable) for layer in vgg_model.layers]pd.DataFrame(layers, columns=['Layer Type', 'Layer Name', 'Layer Trainable'])vgg.summary()
Next, we will re-build our model, using the pre-trained model layers and add additional layers similarly to what we did above.
from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout, InputLayerfrom keras.models import Sequentialfrom keras import optimizersmodel = Sequential()model.add(vgg_model)model.add(Dense(512, activation='relu', input_dim=input_shape))model.add(Dropout(0.3))model.add(Dense(32, activation='relu'))model.add(Dense(1, activation='sigmoid'))model.summary()
Now, we will compile, fit and evaluate our new model.
from keras.callbacks import EarlyStoppingmodel.compile(loss='binary_crossentropy',              optimizer='adam',              metrics=['accuracy'])# simple early stoppinges = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10) ## patience=200history = model.fit(X_train, y_train,                      batch_size=128, epochs=1000,                       validation_data=(X_test, y_test),                      callbacks=[es], verbose=1)train_loss, train_acc = model.evaluate(X_train, y_train)test_loss, test_acc = model.evaluate(X_test, y_test)print('Train: %.3f, Test: %.3f' % (train_acc, test_acc))
Now we can evaluate the metrics from training.
import matplotlib.pyplot as pltplt.plot(history.history['loss'], label='train')plt.plot(history.history['val_loss'], label='test')plt.title('Loss')plt.legend()plt.show()plt.plot(history.history['accuracy'], label='train')plt.plot(history.history['val_accuracy'], label='test')plt.title('Accuracy')plt.legend()plt.show()
We can also evaluate the results of the confusion matrix and classification report to see if our model performs better that the one we built from scratch.
from sklearn.metrics import classification_report, confusion_matrix#Confution Matrix and Classification ReportY_pred = model.predict(X_test)Y_pred_train = model.predict(evals_tensors)y_preds = []y_preds_train = []for i in [round(i[0],0) for i in Y_pred]:    if i > 0.5:        y_preds.append(1)    else:        y_preds.append(0)        for i in [round(i[0],0) for i in Y_pred_train]:    if i > 0.5:        y_preds_train.append(1)    else:        y_preds_train.append(0)print('Confusion Matrix - Test Set')print(confusion_matrix(y_test.values.tolist(), y_preds))print('Confusion Matrix - Evaluation Set')print(confusion_matrix(evals_targets.values.tolist(), y_preds_train))print('Classification Report - Test Set')print(classification_report(y_test.values.tolist(), y_preds, target_names=['Shaved','Unshaved']))print('Classification Report - Evaluation Set')print(classification_report(evals_targets.values.tolist(), y_preds_train, target_names=['Shaved','Unshaved']))
We can see now that this model performs with greater accuracy.
Now that we have a solid model that performs well on unseen images, we can save this model, and the weights. This will allow us to deploy this model, or even share the file with others interested in similar projects.
# Saving whole modelmodel.save('/content/drive/My Drive/Model_Files/Final_Model.h5')# Loading the whole modelfrom keras.models import load_modelloaded_model = load_model('/content/drive/My Drive/Final_Model.h5')loaded_model.compile(loss='binary_crossentropy',                      optimizer='adam', metrics=['accuracy'])# Testing predictions using loaded modelres2 = loaded_model.predict(evals_tensors)results2= []for i in res2:    results2.append(round(i[0],2))compare = zip(results2, evals_targets)# View the comparisonfor i in compare:  print(i)
Since we started with a pickle file, this tutorial would not be complete without saving our model as a pickle file. To save your model as a pickle file, just run the following code:
filename = 'finalized_model.p'pickle.dump(loaded_model, open(filename, 'wb')) # some time later... # load the model from diskloaded_model = pickle.load(open(filename, 'rb'))result = loaded_model.score(X_test, Y_test)
Now let’s review. First, we imported our libraries and image set. We then resampled our data, defined and scaled our features, and stored the images in a tensor. We one-hot encoded our target variables; then split our data into train/test/evaluation sub-sets. We built a CNN model from scratch and also built a model using pre-trained VGG16 and transfer learning; fitting each model to our image set and compiling each model using the adam optimizer, with a binary_crossentropy loss function to improve our accuracy metric during forward and backwards propagation.
After training was complete, we were able to evaluate the metrics, as well as visualize the training results, model structure, and intermediate activations. Finally, we were able to save our whole model as an .h5 file, and also as a pickle file. The next step in our process is to deploy our model for public use. I will use flask to build a python application, and host it on PythonAnywhere, so stay tuned for my next article in this series.
Here is the final product: http://jtadesse.pythonanywhere.com/
You can find my GitHub repository here. Thank you!