CelebA Attribute Prediction and Clustering with Keras | by Luca Anzalone | Towards Data Science

In this article, we talk about facial attribute prediction. We will examine the dataset and point out its weaknesses. Furthermore, we will build and train a deep model and finally discuss the overall results.
Facial Attribute prediction is a Computer Vision (CV) task about deducing the set of attributes belonging to a face. Example attributes are the color of hair, hairstyle, age, gender, etc.
In general, facial attribute prediction is a challenging task: it involves face localization first, and then attribute prediction. Moreover, faces are inherently difficult to analyze due to their complex appearance. The appearance of a face can be altered, being even more complex, by face variations. The most common form of face variations are the following:
Occlusions: hairstyle, makeup, glasses (especially sunglasses), hats and other kind of objects can hide meaningful pixels needed by the model to detect a face and its attributes. In case of extreme occlusion, a model may not be able to localize a face at all!
Illumination: extreme lightning or extreme shadowing can make the work of a detection algorithm much harder (if not impossible), like occlusions do.
Expression: emotions can alter the way a face normally appear. If a detection system has never seen faces subject to emotions during training, it can fail to detect them correctly.
Pose: high-degree rotations in terms of roll (x-axis), yaw (z-axis) e pitch (y-axis) can eventually alter both the appearance of a face, and hide its facial features.
Thus, it’s very important that a convolutional model is trained on difficult exemplars of faces in order to generalize well to faces captured in the wild (faces captured under any kind of conditions).
Facial attribute prediction is not only an academic challenge, but also a way to improve existing applications. For example, a photo app can detect the “smiling” attribute in order to decide which is the best photo among a given sequence.
A nice, wide, and diversified dataset to work with is the CelebA dataset. It is a large-scale face attributes dataset with more than 200K celebrity images, covering a large amount of variations, each with 40 attribute annotations.
So, the first thing to do is to download the dataset. It can be found here or on Kaggle, by executing the following code in the terminal:
# install kaggle first: $> pip install kaggle# set your Kaggle API key:os.environ['KAGGLE_USERNAME'] = "YOUR_USERNAME"                       os.environ['KAGGLE_KEY'] = "YOUR_API_KEY"# then download: $> kaggle datasets download -d jessicali9530/celeba-dataset
After downloading and unzipping it, we can load it in our Python environment or Jupyter Notebook. For this, I’ve prepared the CelebA class that does all the boring work, loading the dataset into a nice Pandas DataFrame:
Now, its a piece of cake to load the dataset (I assume that its path is celeba-dataset\) and eventually select a subset of the facial attributes. In the example code, we drop three features.
# load the dataset with 37 out of 40 features:celeba = CelebA(drop_features=[    'Attractive',    'Pale_Skin',    'Blurry',])
After that, we can show some random samples to understand the structure of the celeba dataframe.
# shows five random samplesceleba.attributes.sample(5)
Now we know that every example is associated to a vector of binary labels, where each attribute can be 0 or 1. Therefore, can be very useful to know how much faces an attribute belong to (I omit this but each image of the dataset depicts a single face). By counting the times a 1 occurs for a certain attribute, it is possible to compute its absolute frequency (or even relative frequency by dividing by the total number of samples).
Here, we can clearly observe a data imbalance problem. The frequency of the facial attributes vary a lot, rather than being roughly equal to each other. There are rare attributes (Bald, Mustache, Double_Chin, etc) with a frequency below 10%, and a couple of very common attributes (No_Beard, Young) with a frequency above 70%.
This exhibit another issue: the dataset is biased with respect to young and beardless people.
When the data is imbalanced, the model can easily overfit the datapoints. In this case, it can learn to always output a 1 for every common attributes, and always output a 0 for the rare attributes: the model finds out that this strategy (predicting 0s and 1s) is good (which is not true, because it doesn’t learn patterns). Following this fashion, the model’s output predictions will look like [1, 0, ..., 0, 1] vectors, all the same for every image.
If that happen, the resulting model is quite dumb: it will only waste computational resources, because it can be replaced by simple code (e.g. random guess, or a constant value).
The good news is that the problem can be mitigated by using a proper loss function (further details in the next section).
Finally, the above image shows some images taken from CelebA:
The model we are going to build is heavily based on the MobileNetV2 architecture, basically is the same model but without the top classification layers (MobileNetV2 is built to output 1000 class probabilities).
First, the model takes one image (3 channels, 224×224 in size) at a time as input, and outputs a vector of probabilities of size n (typically, the size vary according to how much attributes you want to detect). The i-th element of the vector is a real number between 0 (attribute i is not present) and 1 (attribute i is present).
Note that the output vector can contain more than one non-zero elements, meaning that more than one attribute can belong to a face.
To define the model architecture we have to load the MobileNetV2 architecture without its top layers, set the input size to be 224×224, and add the new top layers. The following code implements all the needed operations:
Then to create the model and display its summary (layers, number of parameters, ...), run the following code:
model = build_model(num_features=celeba.num_features)model.summary()
As you can see, creating the model is simple. Here, we will review the newly added top layers:
GlobalAveragePooling2D: uses a kernel big as the last convolutional layers’ kernel to produce 1×1 feature maps. So, if the last convolutional layers outputs n feature maps, each of them wide w and tall h, the global average pooling layer will apply a kernel (w, h) to each feature map to obtain n confidence scores. When the kernel is big as the feature map, the result is a scalar or, equivalently, a 1×1 feature map. Doing so, allows to reduce the number of Fully-Connected (Dense) layers (usually, being at least 2 — thanks to GlobalAveragePooling only one is necessary), thus, reducing the number of parameters, training time, and speed-up inference. The GlobalAveragePooling layer was one of the key innovation of the GoogleLeNet model.
Note that without the GlobalAveragePooling layer, a Flatten layer is necessary. Flattening (w, h, n) feature maps results in a layer with w×h×n units, instead of n resulting from the application of global average pooling (the number of parameter is, thus, considerably less).
Dense: after that, we use a dense layer with 1536 units ReLU-activated, to perform nonlinear combinations of the confidence scores (this increases the model capacity as well its discriminative ability).
BatchNormalization: the model is based on a very deep architecture with hundreds of layers. In order to speed-up training and to reduce the chances for the gradient to vanish or explode, we apply Batch Normalization that normalizes, scales, and shifts the inputs right before applying an activation function. Batch normalization does this by learning four additional parameters. In some cases, batch normalization has regularizing effects.
Dropout: randomly deactivates units during training, according to a probability p. In this way, a neuron (unit) is less depended on its neighbor units. This forces the model to generalize better. In addition, dropout can be seen as a simple and efficient (but approximated) method to train an ensemble of many neural networks: at each iteration, different neurons are deactivated resulting in a different network. When training ends, the resulting network performs roughly as averaging all the different networks obtained during training time.
Dense: this is the output layer of the network. Here we use sigmoid as activation function because it allows to have classes (attributes) that are not mutually exclusive (with a softmax layer only one output element can be 1), and more importantly, to directly estimate the probability for each attribute.
Finally, the model is pretty simple to create and is even small: it has only 4.3M parameters, being lightweight and fast, making it also suitable for mobile and web applications.
Sometimes, training a neural network to solve a certain task given a dataset, can be seen as trying to make the network fit the true underlying probability distribution that generates the dataset and the real-world data (the dataset is only a part of the whole data). Usually, the true probability distribution is unknown and can only be partially estimated thanks to the use of a dataset during training.
The amount of training data impacts directly the generalization capability of the network we are trying to train. Briefly, the generalization error drops as the amount of training data increases.
The sad news is that, usually, the training data cannot capture the entire probability distribution, because there could be no examples that cover a particular case of the distribution.
Actually, what we can do is to slightly “edit” the training data in order to intentionally introduce some variations (e.g. rotations, shifts, ...) — we assume that the real-world data could have such variations.
The practice of Data Augmentation is an effective way to increase the size of the training set. Augmenting the training examples allow the network to “see” more diversified, but still representative, datapoints during training.
The following code defines a set of augmentations for the training-set: rotation, shift, shear, flip, and zoom.
Note that both the validation-set and test-set must not be augmented.
Then we define a couple of data generators: one for training data, and the other for validation data.
When we deal with big datasets that doesn’t fit into memory, we have to find a way to load only the data actually needed.
A data generator is capable of loading the required amount of data (a minibatch of images) directly from the source folder, convert them into training data (fed to the model) and training targets (a vector of attributes — the supervision signal).
For my experiments, I usually set the batch_size = 80. In general a value between 64 and 128 should work well. Usually you want to increase/decrease the batch size according to computational resources and model’s performances.
At this point we select the adadelta optimizer (adam optimizer is fine too, but it seems to perform slightly worse) with default values: like adam, adadelta is an optimizer that requires little tuning because it is capable of automatically adapting the learning rate (slowing down when things go wrong, or moving faster towards promising directions).
The next step, is to chose the loss function: responsible of monitoring the model’s performance and guiding the optimization process.
I find out that a good loss function for attribute prediction task, is a function able to discriminate the single elements of the attribute vector rather than considering the whole vector as a single object. I mean that the loss must understand that the vector [0, 1, 1, 0] is a lot different from this vector [1, 0, 0, 1](the loss should be high); what is meaningful is the amount of 1s and 0s, and the exact location of the single elements within the input vector.
Loss functions like Mean Absolute Error (MAE), Mean Square Error (MSE) or Root Mean Square Error (RMSE) are not capable of doing this kind of distinction: intuitively, they just count the 1s over a vector, and average on them, consequently loosing information about position.
Instead, loss functions like binary cross-entropy and cosine proximity are the ideal choice. Empirically, the latter seems to perform slightly better.
The last step, is to choose our target performance measure. In this case, we care about maximizing the binary accuracy of the model’s predictions.
Finally, we can compile our model:
model.compile(loss='cosine_proximity',              optimizer='adadelta',              metrics='binary_accuracy')
Before training the model is useful to define one or more callbacks. Pretty handy one, are: ModelCheckpoint and EarlyStopping.
ModelCheckpoint: when training requires a lot of time to achieve a good result, often many iterations (or a few costly one) are required. In this case, it is better to save a copy of the best performing model only when an epoch that improves the metrics ends.
EarlyStopping: sometimes, during training we can notice that the generalization gap (i.e. the difference between training and validation error) starts to increase, instead of decreasing. This is a symptom of overfitting that can be solved in many ways (reducing model capacity, increasing training data, data augmentation, regularization, dropout, etc). Often, a practical and efficient solution is to stop training when the generalization gap is getting worse.
Actually, I’ve used only the ModelCheckpoint callback because there was no need to use early stopping. Anyway, if you want to play around by changing the model architecture, hyperparameters, loss and optimizer, the use of early stopping is suggested.
The following code will start training, collecting fitting info into the history variable that provides two learning curve: one for loss, and another for accuracy; which can be both plotted.
Usually, I set num_epochs to be between 10 and 20. Note that the model reaches more or less 89% of accuracy in the first two epochs. But requires longer training to stuck at about 91% (well, at least this is the maximum accuracy I’ve got).
Note: each epoch took around 40 minutes to complete (on Google Colab GPU).
To evaluate the model on the test partition of the dataset, we have to set the test data generator, and let the model make a prediction for each test-data:
The test accuracy I’ve got is about 90.95%. In particular, each facial attribute is detected with the following accuracy:
Once we have a working model, we can do stuff like clustering. For this task, the goal of clustering is to group images into clusters where each cluster share the largest possible amount of facial attributes among the images it contains.
In other words, we want to group images whose faces have similar attributes, like having a blonde girls cluster, guys with hats and eyeglasses, etc.
For example, if we want to cluster the following ten attributes:
'Wearing_Lipstick''Smiling''No_Beard'             'Heavy_Makeup''Bald''Male'           'Young''Eyeglasses''Blond_Hair''Wearing_Hat'
we may end up with these clusters:
Here, we can see that each cluster captures one or more facial attributes among the ones that we chosen.
Note: the above clusters are obtained by running standard clustering algorithms like K-Means, on the model’s predictions.
To have a better understanding of the attributes that the cluster have captured, we can summarize it and observe the cluster’s summary, instead.
A cluster can be summarized by extracting its “distinctive features”. Such a thing can be done by computing the principal components describing the cluster (e.g. using PCA). In this case we have clusters of faces, so we can try to compute an eigenface for each cluster, and see what it looks like...
Surprisingly, a cluster’s eigenface is able to summarize the entire cluster by producing a synthetic face (not belonging to the cluster) whose facial attributes are the one captured by the cluster.
In this way, we can see at a glance what are the prominent facial attributes of a set of images, grouped into the same cluster.
In conclusion, well engineered and general applicable models like MobileNetV2 (or Inception and Resnet models), make possible to build effective task-specific models.
It is always a good practice to analyze the dataset and observe some of its sample, in order to understand its structure, data, and eventual flaws.
Attribute prediction is a task that can improve many existing applications and fields (e.g. security, photography, etc).
I omitted the code about the clustering because is a bit long and convoluted. Anyway, you can find all the code at this repository.
I recommend experiment with the Jupyter notebook version of the code.
Hope you enjoyed it, and Thanks for reading it!
Anzalone, Luca, et al. “Transfer Learning for Facial Attributes Prediction and Clustering.” International Conference on Smart City and Informatization. Springer, Singapore, 2019.
Liu, Ziwei, et al. “Deep learning face attributes in the wild.” Proceedings of the IEEE international conference on computer vision. 2015.
MIT 6.S191: Introduction to Deep Learning
Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Szegedy, Christian, et al. “Going deeper with convolutions.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.