CelebA Attribute Prediction and Clustering with Keras
By: Luca Anzalone
Tags: Keras, Attribute Prediction, Clustering, Computer Vision, Deep Learning

The Model

The model we are going to build is heavily based on the MobileNetV2 architecture, basically is the same model but without the top classification layers (MobileNetV2 is built to output 1000 class probabilities).

First, the model takes one image (3 channels, 224×224 in size) at a time as input, and outputs a vector of probabilities of size n (typically, the size vary according to how much attributes you want to detect). The i-th element of the vector is a real number between 0 (attribute i is not present) and 1 (attribute i is present).

Note that the output vector can contain more than one non-zero elements, meaning that more than one attribute can belong to a face.

Model Architecture

To define the model architecture we have to load the MobileNetV2 architecture without its top layers, set the input size to be 224×224, and add the new top layers. The following code implements all the needed operations:

Then to create the model and display its summary (layers, number of parameters, …), run the following code:

model = build_model(num_features=celeba.num_features)

model.summary()

As you can see, creating the model is simple. Here, we will review the newly added top layers:

The model’s top layers. Image created with Netron.

GlobalAveragePooling2D: uses a kernel big as the last convolutional layers’ kernel to produce 1×1 feature maps. So, if the last convolutional layers outputs n feature maps, each of them wide w and tall h, the global average pooling layer will apply a kernel (w, h) to each feature map to obtain n confidence scores. When the kernel is big as the feature map, the result is a scalar or, equivalently, a 1×1 feature map. Doing so, allows to reduce the number of Fully-Connected (Dense) layers (usually, being at least 2 — thanks to GlobalAveragePooling only one is necessary), thus, reducing the number of parameters, training time, and speed-up inference. The GlobalAveragePooling layer was one of the key innovation of the GoogleLeNet model.

Note that without the GlobalAveragePooling layer, a Flatten layer is necessary. Flattening (w, h, n) feature maps results in a layer with w×h×n units, instead of n resulting from the application of global average pooling (the number of parameter is, thus, considerably less).

Dense: after that, we use a dense layer with 1536 units ReLU -activated, to perform nonlinear combinations of the confidence scores (this increases the model capacity as well its discriminative ability).

after that, we use a dense layer with 1536 units -activated, to perform nonlinear combinations of the confidence scores (this increases the model capacity as well its discriminative ability). BatchNormalization: the model is based on a very deep architecture with hundreds of layers. In order to speed-up training and to reduce the chances for the gradient to vanish or explode, we apply Batch Normalization that normalizes , scales , and shifts the inputs right before applying an activation function. Batch normalization does this by learning four additional parameters. In some cases, batch normalization has regularizing effects.

the model is based on a very deep architecture with hundreds of layers. In order to speed-up training and to reduce the chances for the gradient to vanish or explode, we apply Batch Normalization that , , and the inputs right before applying an activation function. Batch normalization does this by learning four additional parameters. In some cases, batch normalization has regularizing effects. Dropout: randomly deactivates units during training, according to a probability p. In this way, a neuron (unit) is less depended on its neighbor units. This forces the model to generalize better. In addition, dropout can be seen as a simple and efficient (but approximated) method to train an ensemble of many neural networks: at each iteration, different neurons are deactivated resulting in a different network. When training ends, the resulting network performs roughly as averaging all the different networks obtained during training time.

randomly deactivates units during training, according to a probability p. In this way, a neuron (unit) is less depended on its neighbor units. This forces the model to generalize better. In addition, dropout can be seen as a simple and efficient (but approximated) method to train an ensemble of many neural networks: at each iteration, different neurons are deactivated resulting in a different network. When training ends, the resulting network performs roughly as averaging all the different networks obtained during training time. Dense: this is the output layer of the network. Here we use sigmoid as activation function because it allows to have classes (attributes) that are not mutually exclusive (with a softmax layer only one output element can be 1), and more importantly, to directly estimate the probability for each attribute.

Finally, the model is pretty simple to create and is even small: it has only 4.3M parameters, being lightweight and fast, making it also suitable for mobile and web applications.

Training

Sometimes, training a neural network to solve a certain task given a dataset, can be seen as trying to make the network fit the true underlying probability distribution that generates the dataset and the real-world data (the dataset is only a part of the whole data). Usually, the true probability distribution is unknown and can only be partially estimated thanks to the use of a dataset during training.

Trying to capture the data distribution through training-data. Source.

The amount of training data impacts directly the generalization capability of the network we are trying to train. Briefly, the generalization error drops as the amount of training data increases.

The sad news is that, usually, the training data cannot capture the entire probability distribution, because there could be no examples that cover a particular case of the distribution.

Actually, what we can do is to slightly “edit” the training data in order to intentionally introduce some variations (e.g. rotations, shifts, …) — we assume that the real-world data could have such variations.

Example of image augmentations.

Data Augmentation

The practice of Data Augmentation is an effective way to increase the size of the training set. Augmenting the training examples allow the network to “see” more diversified, but still representative, datapoints during training.

The following code defines a set of augmentations for the training-set: rotation, shift, shear, flip, and zoom.

Note that both the validation-set and test-set must not be augmented.

Then we define a couple of data generators: one for training data, and the other for validation data.

When we deal with big datasets that doesn’t fit into memory, we have to find a way to load only the data actually needed.

A data generator is capable of loading the required amount of data (a minibatch of images) directly from the source folder, convert them into training data (fed to the model) and training targets (a vector of attributes — the supervision signal).

For my experiments, I usually set the batch_size = 80 . In general a value between 64 and 128 should work well. Usually you want to increase/decrease the batch size according to computational resources and model’s performances.

Optimizer, Loss Function and Metrics

At this point we select the adadelta optimizer (adam optimizer is fine too, but it seems to perform slightly worse) with default values: like adam, adadelta is an optimizer that requires little tuning because it is capable of automatically adapting the learning rate (slowing down when things go wrong, or moving faster towards promising directions).

The next step, is to chose the loss function: responsible of monitoring the model’s performance and guiding the optimization process.

I find out that a good loss function for attribute prediction task, is a function able to discriminate the single elements of the attribute vector rather than considering the whole vector as a single object. I mean that the loss must understand that the vector [0, 1, 1, 0] is a lot different from this vector [1, 0, 0, 1] (the loss should be high); what is meaningful is the amount of 1s and 0s, and the exact location of the single elements within the input vector.

Loss functions like Mean Absolute Error (MAE), Mean Square Error (MSE) or Root Mean Square Error (RMSE) are not capable of doing this kind of distinction: intuitively, they just count the 1s over a vector, and average on them, consequently loosing information about position.

Instead, loss functions like binary cross-entropy and cosine proximity are the ideal choice. Empirically, the latter seems to perform slightly better.

The last step, is to choose our target performance measure. In this case, we care about maximizing the binary accuracy of the model’s predictions.

Finally, we can compile our model:

model.compile(loss='cosine_proximity',

optimizer='adadelta',

metrics='binary_accuracy')

Fitting

Before training the model is useful to define one or more callbacks. Pretty handy one, are: ModelCheckpoint and EarlyStopping .

ModelCheckpoint: when training requires a lot of time to achieve a good result, often many iterations (or a few costly one) are required. In this case, it is better to save a copy of the best performing model only when an epoch that improves the metrics ends.

when training requires a lot of time to achieve a good result, often many iterations (or a few costly one) are required. In this case, it is better to save a of the best performing model only when an epoch that improves the metrics ends. EarlyStopping: sometimes, during training we can notice that the generalization gap (i.e. the difference between training and validation error) starts to increase, instead of decreasing. This is a symptom of overfitting that can be solved in many ways (reducing model capacity, increasing training data, data augmentation, regularization, dropout, etc). Often, a practical and efficient solution is to stop training when the generalization gap is getting worse.

Early stopping. Source.

Actually, I’ve used only the ModelCheckpoint callback because there was no need to use early stopping. Anyway, if you want to play around by changing the model architecture, hyperparameters, loss and optimizer, the use of early stopping is suggested.

The following code will start training, collecting fitting info into the history variable that provides two learning curve: one for loss, and another for accuracy; which can be both plotted.

Usually, I set num_epochs to be between 10 and 20. Note that the model reaches more or less 89% of accuracy in the first two epochs. But requires longer training to stuck at about 91% (well, at least this is the maximum accuracy I’ve got).

Note: each epoch took around 40 minutes to complete (on Google Colab GPU).

Evaluation

To evaluate the model on the test partition of the dataset, we have to set the test data generator, and let the model make a prediction for each test-data:

The test accuracy I’ve got is about 90.95%. In particular, each facial attribute is detected with the following accuracy: