Building a multi-output Convolutional Neural Network with Keras | by Rodrigo Bressan | Towards Data Science

In this post, we will be exploring the Keras functional API in order to build a multi-output Deep Learning model. We will show how to train a single model that is capable of predicting three distinct outputs. By using the UTK Face dataset, which is composed of over 20 thousand pictures of people in uncontrolled environments, we will predict the age, gender and sex for each record presented in the dataset, reaching an accuracy of 91% for gender and 78% for race.
The UTKFace dataset is a large dataset composed of over 20 thousand face images with their respective annotations of age, gender and ethnicity. The images are properly cropped into the face region, but display some variations in pose, illumination, resolution, etc.
In order to retrieve the annotations of each record, we need to parse the filenames. Each record is stored in the following format: age_gender_race_date&time.jpg
Where:
age is an integer from 0 to 116
gender is an integer in which 0 represents male and 1 represents female
race is an integer from 0 to 4, denoting white, black, asian, indian and others, respectively
date and time, denoting when the picture was taken
If you want to know more about this dataset, please check their website.
Let’s start by importing some libraries and creating our dictionary to help us on parsing the information from the dataset, along with some other information (dataset location, training split, width and height of the samples).
import numpy as np import pandas as pdimport osimport globimport pandas as pdimport matplotlib.pyplot as pltimport seaborn as snsdataset_folder_name = 'UTKFace'TRAIN_TEST_SPLIT = 0.7IM_WIDTH = IM_HEIGHT = 198dataset_dict = {    'race_id': {        0: 'white',         1: 'black',         2: 'asian',         3: 'indian',         4: 'others'    },    'gender_id': {        0: 'male',        1: 'female'    }}dataset_dict['gender_alias'] = dict((g, i) for i, g in dataset_dict['gender_id'].items())dataset_dict['race_alias'] = dict((r, i) for i, r in dataset_dict['race_id'].items())
Let’s also define a function to help us on extracting the data from our dataset. This function will be used to iterate over each file of the UTK dataset and return a Pandas Dataframe containing all the fields (age, gender and sex) of our records.
def parse_dataset(dataset_path, ext='jpg'):    """    Used to extract information about our dataset. It does iterate over all images and return a DataFrame with    the data (age, gender and sex) of all files.    """    def parse_info_from_file(path):        """        Parse information from a single file        """        try:            filename = os.path.split(path)[1]            filename = os.path.splitext(filename)[0]            age, gender, race, _ = filename.split('_')            return int(age), dataset_dict['gender_id'][int(gender)], dataset_dict['race_id'][int(race)]        except Exception as ex:            return None, None, None            files = glob.glob(os.path.join(dataset_path, "*.%s" % ext))        records = []    for file in files:        info = parse_info_from_file(file)        records.append(info)            df = pd.DataFrame(records)    df['file'] = files    df.columns = ['age', 'gender', 'race', 'file']    df = df.dropna()        return dfdf = parse_dataset(dataset_folder_name)df.head()
As an important step to understand not only the distribution of our dataset, but as well the predictions generated by our model, it is a recommended practice to perform some data visualization process on our dataset. We will start by defining a helper function to generate pie plots based on a given Pandas series:
import plotly.graph_objects as godef plot_distribution(pd_series):    labels = pd_series.value_counts().index.tolist()    counts = pd_series.value_counts().values.tolist()        pie_plot = go.Pie(labels=labels, values=counts, hole=.3)    fig = go.Figure(data=[pie_plot])    fig.update_layout(title_text='Distribution for %s' % pd_series.name)        fig.show()
Let’s start by plotting the race distribution with our predefined plot_distribution method.
plot_distribution(df['race'])
Having a quick glance at this plot, we can see that almost half of the samples are from the white race, so we can expect this group to have a great accuracy. Other races such as black, Indian and Asian also show a good number of samples, probably leading us to good accuracy numbers as well. The race ‘others’ (Hispanics, Latinos, etc) on the other side, show a small number of samples, being more likely to have a small accuracy.
plot_distribution(df['gender'])
For both male and female samples, we have quite a good balanced number of records, so we should have a great accuracy for both classes when using our model.
Let’s also plot how our age feature is distributed over the dataset by using a simple histogram with 20 bins/sectors.
import plotly.express as pxfig = px.histogram(df, x="age", nbins=20)fig.update_layout(title_text='Age distribution')fig.show()
We can also display this same plot in a pie plot. Let’s group the age column into bins and then plot it with a pie chart
bins = [0, 10, 20, 30, 40, 60, 80, np.inf]names = ['<10', '10-20', '20-30', '30-40', '40-60', '60-80', '80+']age_binned = pd.cut(df['age'], bins, labels=names)plot_distribution(age_binned)
We can observe that our dataset is mostly composed of individuals whose age varies between 20 and 30 years, followed by individuals ranging from 30 to 40 years and then 40 to 60 years old. These groups represent around 70% of our dataset, so we can assume that we are going to have good accuracy in predicting individuals in these ranges.
We could also perform some multi-variate analysis on our dataset, but since the scope of this post is to demonstrate the usage of a multi-output model with Keras, we won’t be covering it — maybe another time, if you guys are interested.
In order to input our data to our Keras multi-output model, we will create a helper object to work as a data generator for our dataset. This will be done by generating batches of data, which will be used to feed our multi-output model with both the images and their labels. This step is also done instead of just loading all the dataset into the memory at once, which could lead to an out of memory error.
from keras.utils import to_categoricalfrom PIL import Imageclass UtkFaceDataGenerator():    """    Data generator for the UTKFace dataset. This class should be used when training our Keras multi-output model.    """    def __init__(self, df):        self.df = df            def generate_split_indexes(self):        p = np.random.permutation(len(self.df))        train_up_to = int(len(self.df) * TRAIN_TEST_SPLIT)        train_idx = p[:train_up_to]        test_idx = p[train_up_to:]        train_up_to = int(train_up_to * TRAIN_TEST_SPLIT)        train_idx, valid_idx = train_idx[:train_up_to], train_idx[train_up_to:]                # converts alias to id        self.df['gender_id'] = self.df['gender'].map(lambda gender: dataset_dict['gender_alias'][gender])        self.df['race_id'] = self.df['race'].map(lambda race: dataset_dict['race_alias'][race])        self.max_age = self.df['age'].max()                return train_idx, valid_idx, test_idx        def preprocess_image(self, img_path):        """        Used to perform some minor preprocessing on the image before inputting into the network.        """        im = Image.open(img_path)        im = im.resize((IM_WIDTH, IM_HEIGHT))        im = np.array(im) / 255.0                return im            def generate_images(self, image_idx, is_training, batch_size=16):        """        Used to generate a batch with images when training/testing/validating our Keras model.        """                # arrays to store our batched data        images, ages, races, genders = [], [], [], []        while True:            for idx in image_idx:                person = self.df.iloc[idx]                                age = person['age']                race = person['race_id']                gender = person['gender_id']                file = person['file']                                im = self.preprocess_image(file)                                ages.append(age / self.max_age)                races.append(to_categorical(race, len(dataset_dict['race_id'])))                genders.append(to_categorical(gender, len(dataset_dict['gender_id'])))                images.append(im)                                # yielding condition                if len(images) >= batch_size:                    yield np.array(images), [np.array(ages), np.array(races), np.array(genders)]                    images, ages, races, genders = [], [], [], []                                if not is_training:                break                data_generator = UtkFaceDataGenerator(df)train_idx, valid_idx, test_idx = data_generator.generate_split_indexes() 
In this step, we will define our multi-output Keras model. Our model will be composed of three major branches, one for each available feature: age, gender and race.
The default structure for our convolutional layers is based on a Conv2D layer with a ReLU activation, followed by a BatchNormalization layer, a MaxPooling and then finally a Dropout layer. Each of these layers is then followed by the final Dense layer. This step is repeated for each of the outputs we are trying to predict.
These default layers are defined on the make_default_hidden_layers method, which will be reused on building each of the branches of our model. In the code shown below we will define the class that will be responsible for creating our multi-output model.
from keras.models import Modelfrom keras.layers.normalization import BatchNormalizationfrom keras.layers.convolutional import Conv2Dfrom keras.layers.convolutional import MaxPooling2Dfrom keras.layers.core import Activationfrom keras.layers.core import Dropoutfrom keras.layers.core import Lambdafrom keras.layers.core import Densefrom keras.layers import Flattenfrom keras.layers import Inputimport tensorflow as tfclass UtkMultiOutputModel():    """    Used to generate our multi-output model. This CNN contains three branches, one for age, other for     sex and another for race. Each branch contains a sequence of Convolutional Layers that is defined    on the make_default_hidden_layers method.    """    def make_default_hidden_layers(self, inputs):        """        Used to generate a default set of hidden layers. The structure used in this network is defined as:                Conv2D -> BatchNormalization -> Pooling -> Dropout        """        x = Conv2D(16, (3, 3), padding="same")(inputs)        x = Activation("relu")(x)        x = BatchNormalization(axis=-1)(x)        x = MaxPooling2D(pool_size=(3, 3))(x)        x = Dropout(0.25)(x)        x = Conv2D(32, (3, 3), padding="same")(x)        x = Activation("relu")(x)        x = BatchNormalization(axis=-1)(x)        x = MaxPooling2D(pool_size=(2, 2))(x)        x = Dropout(0.25)(x)        x = Conv2D(32, (3, 3), padding="same")(x)        x = Activation("relu")(x)        x = BatchNormalization(axis=-1)(x)        x = MaxPooling2D(pool_size=(2, 2))(x)        x = Dropout(0.25)(x)        return x    def build_race_branch(self, inputs, num_races):        """        Used to build the race branch of our face recognition network.        This branch is composed of three Conv -> BN -> Pool -> Dropout blocks,         followed by the Dense output layer.        """        x = self.make_default_hidden_layers(inputs)        x = Flatten()(x)        x = Dense(128)(x)        x = Activation("relu")(x)        x = BatchNormalization()(x)        x = Dropout(0.5)(x)        x = Dense(num_races)(x)        x = Activation("softmax", name="race_output")(x)        return x    def build_gender_branch(self, inputs, num_genders=2):        """        Used to build the gender branch of our face recognition network.        This branch is composed of three Conv -> BN -> Pool -> Dropout blocks,         followed by the Dense output layer.        """        x = Lambda(lambda c: tf.image.rgb_to_grayscale(c))(inputs)        x = self.make_default_hidden_layers(inputs)        x = Flatten()(x)        x = Dense(128)(x)        x = Activation("relu")(x)        x = BatchNormalization()(x)        x = Dropout(0.5)(x)        x = Dense(num_genders)(x)        x = Activation("sigmoid", name="gender_output")(x)        return x    def build_age_branch(self, inputs):           """        Used to build the age branch of our face recognition network.        This branch is composed of three Conv -> BN -> Pool -> Dropout blocks,         followed by the Dense output layer.        """        x = self.make_default_hidden_layers(inputs)        x = Flatten()(x)        x = Dense(128)(x)        x = Activation("relu")(x)        x = BatchNormalization()(x)        x = Dropout(0.5)(x)        x = Dense(1)(x)        x = Activation("linear", name="age_output")(x)        return x    def assemble_full_model(self, width, height, num_races):        """        Used to assemble our multi-output model CNN.        """        input_shape = (height, width, 3)        inputs = Input(shape=input_shape)        age_branch = self.build_age_branch(inputs)        race_branch = self.build_race_branch(inputs, num_races)        gender_branch = self.build_gender_branch(inputs)        model = Model(inputs=inputs,                     outputs = [age_branch, race_branch, gender_branch],                     name="face_net")        return model    model = UtkMultiOutputModel().assemble_full_model(IM_WIDTH, IM_HEIGHT, num_races=len(dataset_dict['race_alias']))
Let’s give a look into our model structure, to have a better understanding of what we are building. We can see from it that we have a single input, that in our case is the image we are feeding the CNN, which does decompose into three separated branches, each with their own set of Convolutional layers, followed by their respective Dense layers.
Now it’s time to train our multi-output model, once we have both the data ready to use and the model architecture defined. But before doing this step, we need to compile our model. For this task, we will use a learning rate of 0.0004 and an Adam optimizer — but you can be feel free to try with other hyperparameters as well. We will also use custom loss weights and a custom loss function for each feature. When building our optimizer, let’s use a decay based on the learning rate divided by the number of epochs, so we will slowly be decreasing our learning rate over the epochs.
from keras.optimizers import Adaminit_lr = 1e-4epochs = 100opt = Adam(lr=init_lr, decay=init_lr / epochs)model.compile(optimizer=opt,               loss={                  'age_output': 'mse',                   'race_output': 'categorical_crossentropy',                   'gender_output': 'binary_crossentropy'},              loss_weights={                  'age_output': 4.,                   'race_output': 1.5,                   'gender_output': 0.1},              metrics={                  'age_output': 'mae',                   'race_output': 'accuracy',                  'gender_output': 'accuracy'})
Now let’s train our model with a batch size of 32 for both valid and train sets. We will be using a ModelCheckpoint callback in order to save the model on disk at the end of each epoch.
from keras.callbacks import ModelCheckpointbatch_size = 32valid_batch_size = 32train_gen = data_generator.generate_images(train_idx, is_training=True, batch_size=batch_size)valid_gen = data_generator.generate_images(valid_idx, is_training=True, batch_size=valid_batch_size)callbacks = [    ModelCheckpoint("./model_checkpoint", monitor='val_loss')]history = model.fit_generator(train_gen,                    steps_per_epoch=len(train_idx)//batch_size,                    epochs=epochs,                    callbacks=callbacks,                    validation_data=valid_gen,                    validation_steps=len(valid_idx)//valid_batch_size)
Once we have our model trained, let’s give a better look into how our model performed on both training and validation sets over the epochs:
plt.clf()fig = go.Figure()fig.add_trace(go.Scatter(                    y=history.history['race_output_acc'],                    name='Train'))fig.add_trace(go.Scatter(                    y=history.history['val_race_output_acc'],                    name='Valid'))fig.update_layout(height=500,                   width=700,                  title='Accuracy for race feature',                  xaxis_title='Epoch',                  yaxis_title='Accuracy')fig.show()
We can see that by epoch 50 our model stabilizes itself on the validation set, only increasing on the training one, with an accuracy of approximately 80%.
plt.clf()fig = go.Figure()fig.add_trace(go.Scatter(                    y=history.history['gender_output_acc'],                    name='Train'))fig.add_trace(go.Scatter(                    y=history.history['val_gender_output_acc'],                    name='Valid'))fig.update_layout(height=500,                   width=700,                  title='Accuracy for gender feature',                  xaxis_title='Epoch',                  yaxis_title='Accuracy')fig.show()
Similarly to the race feature, we can see that our model is able to learn most of the patterns to properly predict the gender from a given individual by the 30th epoch, with an accuracy of approximately 90%.
plt.clf()fig = go.Figure()fig.add_trace(go.Scattergl(                    y=history.history['age_output_mean_absolute_error'],                    name='Train'))fig.add_trace(go.Scattergl(                    y=history.history['val_age_output_mean_absolute_error'],                    name='Valid'))fig.update_layout(height=500,                   width=700,                  title='Mean Absolute Error for age feature',                  xaxis_title='Epoch',                  yaxis_title='Mean Absolute Error')fig.show()
In the task of predicting the age feature, we can see that our model takes around 60 epochs to properly stabilize its learning process, with a mean absolute error of 0.09.
fig = go.Figure()fig.add_trace(go.Scattergl(                    y=history.history['loss'],                    name='Train'))fig.add_trace(go.Scattergl(                    y=history.history['val_loss'],                    name='Valid'))fig.update_layout(height=500,                   width=700,                  title='Overall loss',                  xaxis_title='Epoch',                  yaxis_title='Loss')fig.show()
We can notice that by the epoch 50 our model starts to stabilize with a loss value of approximately 1.4. There is also a peak in the loss curve which does appear in the Mean Absolute Error for the age feature, which could explain the influence on the learning of the age feature on the overall loss.
In order to assess how our model performs on the test set, let’s use our UTK data generator class, but this time using the test indexes. We will then call the predict_generator method from our trained model, which will output the predictions for the test set.
test_batch_size = 128test_generator = data_generator.generate_images(test_idx, is_training=False, batch_size=test_batch_size)age_pred, race_pred, gender_pred = model.predict_generator(test_generator,                                                            steps=len(test_idx)//test_batch_size)
Let’s iterate one more time over all our test samples, in order to have their labels into a single list. We will also extract the argmax of each record, in order to retrieve the top predictions and ground truths.
test_generator = data_generator.generate_images(test_idx, is_training=False, batch_size=test_batch_size)samples = 0images, age_true, race_true, gender_true = [], [], [], []for test_batch in test_generator:    image = test_batch[0]    labels = test_batch[1]        images.extend(image)    age_true.extend(labels[0])    race_true.extend(labels[1])    gender_true.extend(labels[2])    age_true = np.array(age_true)race_true = np.array(race_true)gender_true = np.array(gender_true)race_true, gender_true = race_true.argmax(axis=-1), gender_true.argmax(axis=-1)race_pred, gender_pred = race_pred.argmax(axis=-1), gender_pred.argmax(axis=-1)age_true = age_true * data_generator.max_ageage_pred = age_pred * data_generator.max_age
And finally, let’s print the classification reports for each feature on the test set.
from sklearn.metrics import classification_reportcr_race = classification_report(race_true, race_pred, target_names=dataset_dict['race_alias'].keys())print(cr_race)precision    recall  f1-score   support       white       0.80      0.91      0.85      2994       black       0.86      0.82      0.84      1327       asian       0.86      0.79      0.83      1046      indian       0.74      0.74      0.74      1171      others       0.38      0.19      0.25       502    accuracy                           0.80      7040   macro avg       0.73      0.69      0.70      7040weighted avg       0.78      0.80      0.78      7040
From the report above, we can see that our model is excellent at predicting Asian and black individuals, with a precision of 86%, followed by white people 80% and Indian with 74%. The race ‘others’ shows a precision of only 38%, but we need to take into consideration that this group is composed of different races and ethnicities along with a few samples, when compared to the other groups. The weighted accuracy for this classification task is 78%, showing that our classifier was able to properly learn patterns to distinguish different types of races.
cr_gender = classification_report(gender_true, gender_pred, target_names=dataset_dict['gender_alias'].keys())print(cr_gender)precision    recall  f1-score   support        male       0.94      0.87      0.91      3735      female       0.87      0.94      0.90      3305    accuracy                           0.90      7040   macro avg       0.90      0.91      0.90      7040weighted avg       0.91      0.90      0.90      7040
From this report, we can notice that our model is excellent at predicting the gender of a given individual, with a weighted accuracy of 91% for this task.
from sklearn.metrics import r2_scoreprint('R2 score for age: ', r2_score(age_true, age_pred))
R2 score for age: 0.5823979466456328
UTK Face Dataset: http://aicip.eecs.utk.edu/wiki/UTKFace
Keras Multi-output documentation: https://keras.io/getting-started/functional-api-guide/
SanjayaSubedi post on multi-output model: https://sanjayasubedi.com.np/deeplearning/multioutput-keras/
PyImageSearch post on FashionNet: https://www.pyimagesearch.com/2018/06/04/keras-multiple-outputs-and-multiple-losses/