# Multi-Task Faces (MTF) Data Set: A Legally and Ethically Compliant Collection of Face Images for Various Classification Tasks

Rami Haffar, David Sánchez, Josep Domingo-Ferrer

<figure><img src="image_1.png" /></figure>

## To cite this version:

Rami Haffar, David Sánchez, Josep Domingo-Ferrer. Multi-Task Faces (MTF) Data Set: A Legally and Ethically Compliant Collection of Face Images for Various Classification Tasks. Pattern Recognition, In press. hal-04268083

HAL Id: hal-04268083

https://hal.science/hal-04268083v1

Submitted on 2 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
Multi-Task Faces (MTF) Data Set: A Legally and
Ethically Compliant Collection of Face Images for
Various Classification Tasks

Rami Haffar<sup>a,*</sup>, David Sánchez<sup>a</sup>, Josep Domingo-Ferrer<sup>a</sup>

<sup>a</sup>Universitat Rovira i Virgili, Department of Computer Engineering and Mathematics, CYBERCAT-Center for Cybersecurity Research of Catalonia, UNESCO Chair in Data Privacy, Av. Paisos Catalans 26, 43007, Tarragona, Catalonia

Abstract

Human facial data hold vast potential for a variety of tasks. However, privacy regulations such as the EU General Data Protection Regulation have motivated the removal or limited the publication of facial data sets from the internet due to inadequate data collection methods. While synthetic data sets have been proposed, they do not faithfully represent the real data distribution. Additionally, most available data sets are task-specific. To address these issues, we present the Multi-Task Faces (MTF) data set, a meticulously curated collection of face images labeled for multi-classification tasks –including face, race, gender, and age classification– that strictly follows ethical collection rules and copyright regulations. In this paper, we detail the data collection and processing procedures, and evaluate data quality across five DL models. The reported results constitute a baseline for further research. The MTF data set is accessible on Github<fnref n="1" />.

Keywords: Face Images, Image data set, Image classification, Deep learning

2000 MSC: 65D19, 68T07

<footnote>*Corresponding author</footnote>

<footnote>Email addresses: rami.haffar@urv.com (Rami Haffar), david.sanchez@urv.cat (David Sánchez), josep.domingo@urv.cat (Josep Domingo-Ferrer)</footnote>

<footnote n="1">¹https://github.com/RamiHaf/MTF_data_set</footnote>
# 1. Introduction

Artificial intelligence (AI) heavily relies on the availability of data, which is the cornerstone for training and evaluating AI models and unlocking their full potential [1]. Since the accuracy and the effectiveness of AI models are directly influenced by the quality of the data those models are trained on, meticulous collection and precise labeling of data are paramount in machine learning.

The procedures for data collection significantly vary depending on the type of data being gathered. In the case of tabular data, collecting information can be done explicitly through surveys and polls, which requires inviting individuals to answer. An alternative procedure is to obtain this type of data from public (e.g., government) administrative records. When it comes to textual data, there are significant limitations due to copyright restrictions. The right to copy, digitize and collect text is heavily curtailed by the copyright holders [2]. Nevertheless, there are valuable sources of text data that are not copyrighted or are available as public-domain material. For example, the Wikipedia provides enormous and curated amounts of text data that can be used for training purposes. The situation is similar for image data, where there are restrictions related to the rights to copy and modify images. Yet, one difference between text and image data is that public-domain sources of images are much more limited.

Images are especially complex data. On the one hand, their redistribution may or may not be authorized depending on the applicable copyright licenses [3]. On the other hand, they may be subject to special regulations according to their contents. For instance, medical images are essential to training computer-aided diagnosis systems but their use requires either explicit consent by the patients or the removal of image traits or metadata that might lead to privacy disclosure [4]. Also, images containing human faces are employed in a wide range of applications—including automated identification of people—which raise profound legal and ethical issues. In particular, their use is restricted by regulations concerning the protection of personally identifiable information, such as the General Data Protection Regulation (GDPR) [5] or the Ethics Guidelines
for Trustworthy AI [6] of the European Union.

More specifically, the GDPR defines personal data in its Article 4² as “any information relating to an identified or identifiable natural person (‘data subject’).” Based on this definition, we can confidently state that facial images are legally considered personal data because they have the capability to easily identify the individuals they depict.

The GDPR also limits the processing of personal data to ensure privacy. In Article 9, it states that “processing of personal data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership, and the processing of genetic data, biometric data for the purpose of uniquely identifying a natural person, data concerning health or data concerning a natural person’s sex life or sexual orientation shall be prohibited.” Facial images, due to their re-identifying nature and the information they contain about an individual’s ethnic origin, race, and biometric data, fall under this category of non-processable data. However, the GDPR indicates exceptions to this prohibition in the same Article, specifically in its second paragraph. One such exception is “(e) processing relates to personal data which are manifestly made public by the data subject.” Hence, processing of personal data on individuals that have chosen to make them public without any restrictions is allowed. This applies to images that have been published under public-domain licenses or with Creative Commons licenses that grant the right to share and modify the content.

Face images are crucial in many AI research areas, such as facial recognition, emotion detection, age estimation, gender classification, facial biometrics analysis, and other related tasks, such as face anonymization. However, the availability of such data has been significantly reduced in recent years.

In particular, several data sets of face images have recently become access-restricted or have been completely removed from the internet due to privacy concerns or to comply with data protection regulations. Furthermore, many

<footnote n="2">²https://gdpr-info.eu/</footnote>
data sets lack clarity regarding the methods employed for image collection,
which makes them susceptible to potential removal in the near future. The
withdrawal of previously public data sets has several shortcomings, and in par-
ticular it hampers the reproducibility of research results obtained on those data.
As an alternative, some data sets solely comprise of synthetic images, whose ap-
plicability is limited, as they tend to deviate from real-world data distributions.

## Contributions and plan

To overcome the limited availability of data sets of human faces, in this
paper we present the Multi-Tasks Faces (MTF) image data set, a meticulously
curated collection of real face images aimed at training and evaluating AI models
for several tasks. The compilation of the images has been done in compliance
with current legal regulations. In particular, all the images are publicly available
and correspond to well-known celebrities, which avoids identity disclosure issues
while keeping data identified. Also, the compiled images have a copyright license
that is either public-domain or Creative Commons with permission to modify,
share, and commercial use. The data set has been compiled under the umbrella
of the SoBigData++ project <fnref n="3" />, and it has been approved by the project's ethical
and legal board. SoBigData++ is a project funded by the European Union that
aims to design and deploy an integrated infrastructure for social mining and big
data analytics, and it adheres to the world's strictest data protection regulations.

Face images in the MTF data set have been manually and simultaneously
labeled according to i) the identity of the 240 individuals the images correspond
to, ii) binary gender labels (male/female), iii) age groups (young/old), and iv)
four distinct racial ethnicities. The data set therefore enables training and
evaluating models for face recognition (FR), gender classification (GC), age
classification (AC), and race classification (RC). The number of images per
celebrity is not uniform, but it reflects the number of available images for each
celebrity that are of good quality and are public-domain or Creative Commons

<footnote n="3">³https://plusplus.sobigdata.eu/</footnote>
with the right to modify, share or use commercially.

In addition to describing the data collection and processing procedures that were carried out, we also report the results of a comprehensive evaluation of the data set on several well-known deep learning (DL) models. These results constitute baselines for future research endeavors employing this data set. Furthermore, we highlight the significance of data processing by comparing the performance of each DL model trained on the MTF data set with the same model trained on raw data crawled from the internet. This justifies the necessity and importance of the different steps undergone in creating the MTF data set.

The remainder of this paper is organized as follows. Section 2 discusses previously published data sets of face images and highlights their limitations. Section 3 details how the MTF data set has been collected, processed and labeled. Section 4 describes the structure of the data set in detail. Section 5 reports baseline results obtained when using the MTF data set with five different DL models. The final section summarizes the key contributions of this work and outlines several lines for further research.

## 2. Related work

CMU Face Images [7] is a small data set consisting of 640 black and white face images corresponding to 20 individuals (32 images per individual). It is labeled according to pose (straight, left, right, up), expression (neutral, happy, sad, angry), eyes (open, sunglasses), and scale (full-resolution, half-resolution, quarter-resolution). No information is available on the type of consent given by the 20 individuals about releasing their images. On the other hand, this data set allows tasks related to pose/emotion classification, but its small size, its lack of race diversity, the low resolution of some images, and the fact that they are black and white limit its applicability.

The Large-scale CelebFaces Attributes (CelebA) data set [8] is a widely employed data set consisting of 202,599 face images featuring 10,177 celebrities.
Each image is labeled according to the corresponding identity, in addition to
40 binary attributes depicting the appearance of the individual (e.g., eyeglasses,
bangs, wearing hats, mustaches), among others.

However, the CelebA data set was compiled by crawling images from the Internet, irrespective of the copyright ownership held by the owners of the images. Consequently, the availability of the CelebA data set is potentially at risk due to privacy concerns and copyright laws, particularly within the European Union [9].

The recently published DigiFace-1M data set [10] comprises 1.22 million synthetic face images. It is divided into two parts: i) 720,000 images featuring 10,000 virtual identities (72 images per identity), and ii) 500,000 images encompassing 100,000 virtual identities (5 images per identity). Although this data set offers a substantial number of images, it lacks information regarding the distribution ratio between males and females or different age groups. It is important to note that the sole purpose of this data set is to train FR models. However, it is worth considering that all the images in this data set are synthetically generated. This fact limits the data set’s representativeness, as it does not accurately reflect real-world data.

The Labeled Faces in the Wild (LFW) data set [11] is another noteworthy data set of face images. It holds 13,233 images corresponding to 5,749 unique identities. The publishers of the data set emphasize that all the data were legally and ethically collected from original images under the Creative Commons copyright licenses. However, LFW presents two significant drawbacks. Firstly, the images are not ‘cleaned’, meaning that the backgrounds of the images contain a substantial amount of information, including sometimes other individuals’ faces. Secondly, each identity in the data set has a very limited number of images, with only 1,680 identities having more than one image and none of the identities having more than three images. These drawbacks limit the applicability of this data set to certain unsupervised biometric identification applications, and render it unsuitable for training supervised models.

Another accessible data set is Megaface [12], which consists of a million face
images that have been ethically collected from images that fall under the Creative Commons copyright licenses. However, Megaface is specifically designed to facilitate the training of *face detection* models. The data set exclusively offers bounding boxes that specify the facial regions within the images, while it does not include identity labels for face recognition purposes. This makes Megaface a valuable resource for advancing face detection algorithms, but it may not be suitable for tasks requiring identity recognition.

Similar to Megaface, other data sets have been released with the specific purpose of facilitating the training of face detection models. Two such data sets are the WIDER FACE data set [13], comprising 32,203 images, and the Multi-Attribute Labeled Faces (MALF) data set [14], comprising 5,250 images. Both of these data sets have been carefully collected, ensuring ethical collection practices, and each image has been labeled to indicate the precise location of faces within the images. However, both data sets are exclusively suitable for face detection tasks. They cannot be directly employed for any other AI training tasks without additional processing and relabeling of the images.

IMDB-WIKI [15] consists of 523,051 face images. These images were obtained from two well-known websites (IMDB and Wikipedia), which provide images from the public domain. Nevertheless, this data set is only labeled for age estimation.

The Flickr-Faces-HQ (FFHQ) data set [16] is an extensive collection of 70,000 high-quality images portraying human faces with remarkable diversity in terms of age, ethnicity, and image backgrounds. These images were directly downloaded from the Flickr website and they are subject to the Creative Common copyright licenses. Additionally, FFHQ lacks any processing or labeling, as its primary purpose is to aid in the development of generative adversarial networks.

On the other hand, several other popular face image data sets have faced challenges regarding their accessibility due to how the images were compiled.

VGGFace2 [17] featured 3.3 million face images captured in real-world conditions, representing more than 9,000 distinct identities. This data set was specifically labeled for FR, with an average of 362 images per identity. How-
ever, the images within the data set were obtained by downloading them from Google image search without taking into account the copyright ownership of the images. As a result, VGGFace2 is no longer publicly available. While the specific reasons for its withdrawal from public access have not been explicitly disclosed, they are likely to be connected to concerns surrounding privacy and legal considerations.

UMDFaces [18] is another significant data set containing 367,888 facial images of 8,277 individuals. The data set is labeled for face recognition, but it also provides information on 21 keypoints that capture essential biometric details. However, the authors have not specified the methodology employed to collect and process the data, thereby raising concerns about the ethical aspects of its creation. Currently, UMDFaces is unavailable for download. The data set's website indicates that the authors are working towards making it accessible again, which implies that they are making efforts to address compliance with relevant regulations.

To our knowledge, the aforementioned data sets represent the only existing collections of human faces specifically intended for training AI models. Unfortunately, they suffer from a variety of drawbacks that diminish their usefulness as benchmarks for FR models. As discussed, some of them have already been removed from the internet due to legal and privacy concerns, while others are at risk of being removed in the near future due to how data have been compiled. Other data sets are entirely synthetic, lack sufficient images to train robust FR models, or are labeled for tasks other than FR.

### 3. Data collection, processing and labeling

The MTF data set was carefully curated to take advantage of the above-mentioned exception in Article 9 of the GDPR, which allows for the collection and processing of personal data that have been voluntarily made public by the data subject/owner. Along this line, our data collection process prioritized privacy by exclusively focusing on publicly known individuals. This ensures the
legal basis needed for publicly releasing the data set and hence its availability
and longevity. Additionally, we sought and obtained approval from the Board
of Operational Ethical and Legality Evaluation (BOEL) of the SoBigData++
H2020 project.

This section provides an overview of the methodologies employed for select-
ing, downloading, processing, and labeling the images in the MTF data set.

3.1. *Data collection*

The initial phase of our data collection process involved selecting the celebri-
ties to be included in the data set. We utilized the IMDB website to search for
celebrity names across different regions. To enhance the data set’s diversity
and inclusivity, we intentionally included four distinct ethnicities, which are
consistent with those employed by the United States Census Bureau<fnref n="4" />: Asian
(Chinese/Korean), Asian (Indian), Black, and White.

In order to maintain a balanced and realistic distribution, we included an
equal number of male and female individuals. This approach ensures fairness
and avoids any gender bias within the collected data. Recognizing age as a
crucial factor, we also sought to include an equal number of young and old
celebrities in the data set. This consideration adds an additional dimension to
the data and enables research related to age-based analysis.

To distinguish between young and old identities within the MTF data set,
we labeled as young those people between ages 18 and 49, and as old those
people aged 50 or above. It is important to note that all images associated with
a particular identity must align with one specific

age group. Any images that did not meet this criterion were removed from
the data set to maintain consistency, such as images of old celebrities in their
youth, or images of young celebrities when they were children.

The search we conducted on the IMDB website resulted in selecting an equal
number of celebrities from each of the four ethnicities under consideration. Our

<footnote n="4">⁴https://www.census.gov/en.html</footnote>
selection process relied on the published lists available on the IMDB website. We conducted searches based on specific criteria and identified celebrities who appeared on these lists. The selection was made according to their presence and ranking on these lists.

The distribution of the selected celebrities from each ethnic group is as follows: 40 old male celebrities, 40 old female celebrities, 40 young male celebrities, and 40 young female celebrities. This totals 160 celebrities selected from each of the four ethnicities, resulting in a grand total of 640 distinct identities. This systematic approach ensures an initial balanced representation within each ethnicity and across different age and gender groups.

However, the process detailed in Section 3.2 to retain only images of sufficient quality reduced the number of identities and the number of images per retained identity, and it altered the above balanced representation. Nonetheless, the data set users can always re-balance the data by taking balanced subsets of the data set.

To download the images of the chosen celebrities, we used the icrawler library. For this purpose, we chose the Bing search engine as a source of the images due to its advanced filtering capabilities by image copyright.

Bing offers six different options: “Creative Commons”, “public domain”, “free to share and use non-commercially”, “free to share and use commercially”, “free to modify, share, and use non-commercially”, and “free to modify, share, and use commercially”. In contrast, other search engines such as Google only provide two options: “Creative Commons” without any additional specifications, and “commercial and others”.

The icrawler library proved to be an excellent resource as it provided us with the ability to select a copyright filter for the downloaded images by employing the filter options offered by the search engine. By leveraging this functionality, we were able to narrow down our image selection to those aligned with our desired copyright requirements. Specifically, we focused on images that fall under the public domain or Creative Commons licenses, granting us the rights to modify, share, and use the images commercially.
To compile a comprehensive data set, we chose not to impose a limit on the number of downloaded images per celebrity due to the limited availability of images with the desired copyright licenses. The image crawling process stopped once no further images were obtainable.

At the end of this phase, the data set consisted of a total of 117,114 images. However, the distribution of these images among the various celebrities was not uniform. This uneven distribution was primarily influenced by two factors: the availability of images on the internet for each celebrity and the individual choices made by celebrities regarding copyright permissions. As a result, the data set exhibited non-iidness (non-identical and independent distribution among celebrities). Nevertheless, this non-iidness accurately reflected the true distribution of data available online for those specific celebrities under the required licenses.

## 3.2. *Data processing*

To ensure accurate training of AI models for facial image classification and identity recognition, it is crucial to have facial images that focus on the faces themselves, while minimizing background details. Therefore, a necessary pre-processing step involves cropping the faces from the crawled images. To this end, each downloaded image was processed through the Haar cascades method [19], a machine learning-based approach that involves training a cascade function with input data to automatically detect the facial regions within the images. For this purpose, we leveraged the pre-trained Haar cascades method available in the OpenCV [20] library. This automated process proved efficient in providing the bounding boxes of the facial areas in the images. These bounding boxes varied in size depending on the appearance of each face in the original images. We then utilized these bounding boxes to crop and isolate the facial regions from the acquired images. As a result, we obtained a collection of 106,438 cropped images, representing 640 distinct identities.

Note that the total number of cropped images was lower than the initially crawled images mainly due to the presence of mismatches caused by the search
engine. Some of the crawled images were not relevant to the desired facial
recognition task and did not even contain any recognizable faces. It is also
worth mentioning that the cropped images had varying resolutions, according
to the size of their respective original images.

In the subsequent stage, three experienced human evaluators were engaged to
visually analyze each cropped image. Their primary objective was to confirm the
presence of the desired celebrity's face in each image. This thorough examination
process encompassed the following steps:

* Firstly, any image that did not contain a complete human face was eliminated. Since the original images were obtained from sources such as public domain or Creative Commons with permissions for modification, sharing, and commercial use, a significant number of them turned out to be random images of art or designs. Consequently, the Haar cascades method mistakenly detected features in these non-face images as faces. Once those inappropriate images were removed, the data set size was reduced to 42,575 images.

* Secondly, the human experts further filtered the data set by eliminating each face image that did not belong to the correct identity. This additional verification step reduced the data set size to 6,453 images, thereby eliminating 36,122 images due to various reasons. For instance, many of the crawled images featured groups of celebrities or celebrities attending festivals with a large number of fans in the background, leading to multiple faces being cropped from the same image. However, only one of those faces belonged to the desired celebrity, which required the removal of the rest. Additionally, search engine results occasionally included images of celebrity relatives such as spouses, children, or coworkers, which were also excluded by the experts during this stage.

* Thirdly, to ensure the suitability of the data set for training robust AI models, the experts implemented several further filtering criteria. Images
that contained obscured or hidden parts of the celebrity's face, such as sunglasses or hands covering the mouth or eyes, were removed. Additionally, any images that were hand-drawn, artificially altered, or generated by AI algorithms were excluded. The data set was further refined by eliminating images where the celebrity's face appeared unnatural due to artistic effects or intrusive movie-related makeup, typically seen in images taken during movie shoots. Furthermore, images with visual disturbances such as heavy pixelation or blurring, which could potentially mislead the AI model, were also excluded from the data set. These filtering criteria ensured the data set consists only of high-quality, natural, and unaltered facial images suitable for training AI models. This process resulted in a further reduction of the data set size to 5,984 images.

* Fourthly, to mitigate the risk of data leakage and avoid any potential additional unnecessary computational cost during AI model training, the experts implemented measures to identify and remove duplicate or highly similar images (e.g., burst shoots) from the data set. They employed three techniques for this purpose. The first technique involved an automated code written using the OpenCV library. This code computed the similarity between images and alerted the experts to potential duplicates. The second technique utilized the Duplicate Photos Fixer<fnref n="5" /> application, which systematically scanned the data set and compiled a list of potentially duplicated or very similar images. Lastly, the experts visually inspected the data set to ensure that no very similar or duplicate images were overlooked. This manual examination served as a final verification step to confirm the absence of any remaining duplicates. As a result of this process, the data set was further reduced to a size of 5,763 images.

* Fifthly, to ensure the usefulness of the data set across the different tasks, images that did not meet the criteria for all desired tasks were eliminated

<footnote n="5">⁵https://www.duplicatephotosfixer.com</footnote>
by the experts. For instance, images depicting outdated representations of
currently old individuals that could impact the age classification task, or
images of the celebrities during their childhood that could affect both face
recognition and age classification, were removed. This selection process
resulted in a reduction of the data set size to 5,320 images.

Such a meticulous filtering process conducted by the human experts ensured
that the remaining images in the data set exclusively represented the facial
images of the intended celebrities. Finally, the experts conducted a thorough
examination of the remaining data to assess their usefulness. During this stage,
the experts counted the number of images for each celebrity that remained
in the data set after undergoing all the aforementioned processes. Individuals
with less than 5 images remaining were excluded from the data set. This step
was essential because a limited number of images would not provide sufficient
data for effective training, validation, and testing in the face recognition task.
Consequently, the data set size was further reduced to 5,246 images. As a
secondary consequence of this stage, the total number of celebrities in the data
set decreased to 240, but each one with a sufficient amount of data available.

This process resulted in a substantial reduction from 117, 114 images to just
5, 246 images, accounting for only 4.47% of the original data set. However, it
ensured that the retained images guarantee the consistency and performance of
the data set for AI training due to the higher image quality. This is empirically
validated in Section 5 across a variety of DL models across the chosen tasks.
It is important to note that this filtering process is a clear advantage of our
data set over other data sets such as Megaface, VGGFace2, and UMDFaces,
which include a large number of systematically crawled internet images that
can introduce significant noise during training.

The remaining images in the MTF data set underwent a final automated
process to standardize their sizes with techniques similar to those used in pre-
vious data sets like LFW, WIDER FACE, and MALF. As a result, all images
in the data set were resized to a uniform resolution of 1024 x 1024 pixels.
### 3.3. Data labeling

After the data processing phase, the same experts labeled each individual image based on the predetermined criteria established during the collection phase. These labels were specifically tailored for the four tasks outlined below:

1. *Face recognition*: Each image was assigned to one of the 240 remaining identities.

2. *Race classification*: Each image was assigned one of the following four labels: Asian (Chinese/Korean), Asian (Indian), Black, or White.

3. *Gender classification*: Each image was assigned one of the two following labels: Male or Female.

4. *Age classification*: To treat this task as a binary classification and not as an age estimation, each of the images was assigned one of the two following labels: Young or Old.

To confirm the correctness of label assignments, two verification phases were undertaken. Firstly, each of the three human experts individually verified the labeling of the entire data set. Secondly, the experts collaboratively conducted a joint verification session.

## 4. The Multi-Task Faces (MTF) data set

After completing the processing and labeling for all the remaining images in the MTF data set, a distribution of celebrities emerged across different labels. While our initial efforts aimed to crawl a balanced number of images across all tasks and labels, the real distribution of the data available online led to an imbalance within the data set. This imbalance can be attributed to various factors: i) celebrities from different regions of the world publish their images at varying rates and under different copyright licenses; ii) young celebrities tend to publish their images more frequently than old celebrities; and iii) old celebrities often have more images from their younger days than images from their current age. Table 1 presents the number of celebrities and images assigned to each label in the four different tasks.
<table><caption>Table 1: Distribution of the celebrities and images among the different tasks of the data set</caption><thead><tr><th>Task</th><th>Label</th><th>Celebrities</th><th>Images</th></tr></thead><tbody><tr><td>Face recognition</td><td>Celebrities</td><td>240</td><td>5,246</td></tr><tr><td rowspan="4">Race classification</td><td>Asian (Chinese / Korean)</td><td>80</td><td>1,715</td></tr><tr><td>Asian (Indian)</td><td>49</td><td>820</td></tr><tr><td>Black</td><td>35</td><td>478</td></tr><tr><td>White</td><td>76</td><td>2,133</td></tr><tr><td rowspan="2">Gender classification</td><td>Males</td><td>130</td><td>2,490</td></tr><tr><td>Females</td><td>110</td><td>2,756</td></tr><tr><td rowspan="2">Age classification</td><td>Young</td><td>190</td><td>4,682</td></tr><tr><td>Old</td><td>50</td><td>514</td></tr></tbody></table>

The four classification tasks are remarkably diverse:

* The FR task involves a classification problem with many categories, as it requires labeling celebrities based on their names. This task encompasses a total of 240 celebrities and includes all the images in the data set.

* The second task is the RC task, which presents a multi-label classification problem with four labels. The distribution of celebrities in this task is imbalanced, as it features two majority groups and two minority groups. The majority groups consist of Asians (Chinese/Korean) and Whites, as celebrities from these regions tend to share their images with our target copyright licenses more frequently than celebrities from the other two regions, namely Asian (Indian) and Black celebrities.

* The GC task represents the first binary classification problem in the MTF data set. The data are relatively well balanced between the two labels, with slightly more male celebrities present. However, both male and female celebrities share their images at a similar rate, although females tend to share a higher number of images overall.

* Finally, the last task offered by the MTF data set is the AC task, which also represents a binary classification problem. In contrast to the GC task, the AC task exhibits an extremely unbalanced data distribution. This disparity is due to the reasons listed above, resulting in the majority of celebrities and images falling under the “young” category, while the “old” category comprises only 514 images belonging to a mere 50 celebrities.
Note that, if the data set users wish to re-balance the distribution within each task, they can do so by selecting balanced subsets of the data set.

Moreover, the MTF data set offers potential for a wide range of tasks that extend beyond the four previously mentioned. Its flexibility allows defining multi-criteria classification tasks involving double or even triple attributes, which provides diverse options for AI model development.

In Table 2, we present the count of celebrities and the respective count of images for the cross-tabulation of race and gender, as well as the cross-tabulation of race and age.

<table><caption>Table 2: Counts of celebrity identities and images when cross-tabulating by race and gender, and by race and age. The first figure in each cell refers to the count of identities, and the second figure to the count of images.</caption><thead><tr><th rowspan="2">Labels</th><th rowspan="2"></th><th colspan="2">Gender labels</th><th colspan="2">Age labels</th></tr><tr><th>Males</th><th>Females</th><th>Young</th><th>Old</th></tr></thead><tbody><tr><td rowspan="4">Race labels</td><td>Asian (Chinese /Korean)</td><td>36/1,020</td><td>44/695</td><td>69/1,612</td><td>11/103</td></tr><tr><td>Asian (Indian)</td><td>22/300</td><td>27/520</td><td>43/749</td><td>6/71</td></tr><tr><td>Black</td><td>18/194</td><td>17/384</td><td>30/533</td><td>5/45</td></tr><tr><td>White</td><td>54/976</td><td>22/1,157</td><td>48/1,788</td><td>28/345</td></tr></tbody></table>

Similarly, Table 3 provides valuable insights by reporting the number of celebrities and their corresponding images for the cross-tabulation of age and gender.

<table><caption>Table 3: Counts of the celebrity identities and images when cross-tabulating by age and gender. The first figure in each cell refers to the count of identities, and the second figure to the count of images.</caption><thead><tr><th rowspan="2">Labels</th><th rowspan="2"></th><th colspan="2">Gender label</th></tr><tr><th>Males</th><th>Females</th></tr></thead><tbody><tr><td rowspan="2">Age labels</td><td>Young</td><td>92/2,049</td><td>98/2,633</td></tr><tr><td>Old</td><td>38/441</td><td>12/123</td></tr></tbody></table>

Tables 2 and 3 illustrate the versatility of the MTF data set, which enables researchers to simultaneously explore and analyze complex classifications based on up to three simultaneous attributes (race/ethnicity, age, and gender). The two tables give a glimpse of the vast opportunities that our data set offers for AI research and development.
4.1. *Data splitting for training and testing*

Once we ensured the suitability of all the images in the data set, we per-
formed two rounds of data splitting meant for training

and testing AI models.

In the first split, we allocated 70% of the data (3,662 images) for training purposes, while the remaining 30% was set aside for validation and testing. Then we took a second split on the 30% portion, which we divided into two subsets: approximately 20% of this portion (332 images) was designated as validation data, to be employed for fine-tuning and optimization of the model. The remaining 80% (1,252 images) was allocated as test data, to be used as an independent evaluation set for assessing the trained model’s generalization and accuracy.

To facilitate a consistent evaluation in future experiments conducted on the
data set, we maintained the same data splitting across all tasks. We did so by
following a methodology that ensured that each subset of the data contained
at least one image representing each label within each task. Specifically, we
made sure to include at least one image per label in each of the validation and
test subsets, as well as a minimum of three images per label in the training
subset. This approach was particularly important for the FR task, where the
data are more spread across the 240 identity labels, which requires a more evenly
distributed data set.

4.2. *Data release*

The data set has been made available under the Creative Commons copy-
right license, granting permissions for sharing, modifying, and commercial us-
age. This respects the copyright of the original images. Users are obligated
to provide proper attribution to the authors of the data set and the owners of
the original images when utilizing the data set. This includes citing the present
paper (the release paper) and specifying the source of the data.

The MTF data set has been made accessible through the European SoBig-
Data++ catalogue. It can be accessed using the following GitHub webpage that
points to the catalogue: https://github.com/RamiHaf/MTF_data_set

The official release of the data set includes the full labeling for each image
by organizing them in an appropriate folder structure. The main folder of the
data set comprises three primary subsets: ‘Test’, ‘Train’, and ‘Val’, which are
used for training and evaluating AI models. Within each subset, there are four
folders corresponding to race classification denoted as ‘Asian_chinese_korean’,
‘Asian_indian’, ‘Black’, and ‘White’. Within each race folder, there are two
additional folders named ‘Males’ and ‘Females’, representing the two labels used
for gender classification. Inside each of the gender folders, there are two more
folders labeled ‘Young’ and ‘Old’, which indicate the two age categories used
for age classification. Within each ‘Age’ folder, there are folders corresponding
to the identities of the celebrities.

With this structure, researchers have the flexibility to rearrange the data
within the data set according to their specific preferences and requirements,
e.g., by defining multi-criteria classification tasks involving several attributes as
mentioned in Tables 2 and 3.

Additional items made available through the above-mentioned GitHub web-
page are as follows:

* To facilitate and support future research on the MTF data set, we have released a second version that is pre-split and organized specifically for users interested in only one of the four tasks. This version includes four separate folders: ‘race_classification’, ‘gender_classification’, ‘face_recognition’, and ‘age_classification’. Each folder contains all the images that are labeled according to the corresponding task.

* We release also the trained models we evaluate in the next section. Since these models provide baseline results for the different tasks supported by the data set, by releasing them we aim at facilitating future investigations conducted on this data set.

* Finally, we release the Python code we used for evaluating the data with the various DL models mentioned in the next section. This code can serve
as a baseline for other researchers to compare their work with the results reported here.

# 5. Experimental results

To assess the efficacy of the MTF data set for training DL models and establish a performance baseline across its supported tasks, we conducted evaluations using five well-known DL classification models. In the following we describe the evaluation metrics employed to assess the performance achieved, the specific models used in the experiments, and the obtained results.

## 5.1. Evaluation metrics

We employed the following standard evaluation metrics to measure the performance of DL models trained on the MTF data set:

*   **Accuracy**: Ratio of correctly predicted instances to the total number of predicted instances in the data set. It provides an overall measure of the classification correctness.

*   **Precision**: Proportion of true positive predictions to the sum of true positive and false positive predictions. It indicates the accuracy of positive predictions.

*   **Recall**: Proportion of true positive predictions to the sum of true positive and false negative predictions. It captures the model's ability to identify positive instances.

*   **F1 Score**: Harmonic mean of precision and recall. It offers a balanced measure of the model's accuracy.

## 5.2. DL models and hyperparameters

To establish a baseline for evaluating the performance of the MTF data set, we conducted assessments using five well-known DL classification models. These models vary in their depth and complexity. Table 4 provides an overview of the
chosen models, including their number of parameters and the year they were published. Furthermore, the table summarizes the main purpose behind their design and highlights their advantages at time of their release.

<table><caption>Table 4: DL models used to evaluate MTF data set</caption><thead><tr><th>Models</th><th>Parameters</th><th>Year</th><th>Purpose</th></tr></thead><tbody><tr><td>MobileNetV3 [21]</td><td>4,509,472</td><td>2019</td><td>To run in embedded systems<br>Reduced number of parameters</td></tr><tr><td>AlexNet [22]</td><td>57,987,120</td><td>2012</td><td>Extremely capable DL model at that time,<br>to be trained on the ImageNet [23] data set</td></tr><tr><td>ResNet50 [24]</td><td>23,999,792</td><td>2016</td><td>Reduced number of parameters and matrix multiplications<br>Enables much faster training of each layer<br>Uses a stack of three layers rather than two</td></tr><tr><td>Vgg16 [25]</td><td>135,243,824</td><td>2014</td><td>Presents the small (3 × 3) convolution filters<br>Introduces the deep network with 16 layers</td></tr><tr><td>ConvNeXT [26]</td><td>87,812,464</td><td>2022</td><td>Introduces the Vision Transformer into image processing<br>through depthwise convolution.</td></tr></tbody></table>

To ensure a fair evaluation across the different models, we used the same hyperparameters during both training and evaluation. However, a minor exception was made for the batch size. Due to resource limitations, we adjusted the batch size to be smaller for the models with a large number of parameters, allowing them to run efficiently on the available hardware. Conversely, for the smaller models, we increased the batch size to optimize time efficiency. This approach enabled us to conduct a comprehensive and equitable assessment of the performance of the models.

Specifically, the chosen hyperparameters included the cross-entropy loss function [27], the Adam optimizer [28] and an automated learning rate scheduler [29]. The initial learning rate was set at 0.001, and it decreased by a factor of 0.1 every 20 epochs. The training process was limited to a maximum of 100 epochs, with early stopping implemented if the training loss did not improve for five consecutive epochs.

The batch size for the three smaller models was set to 128 images, while for the two larger models it was reduced to 32 images.

Experiments were conducted on a machine equipped with an AMD Ryzen 5 3600 CPU running at a base speed of 3.6 GHz, 32 GB of RAM, and an NVIDIA GeForce RTX 3060 GPU with 12 GB of dedicated RAM.
## 5.3. Results

We next report the performance of the five DL models on the MTF data set. All the experiments were conducted using the pre-defined models available in the PyTorch [30] library. For each of the four tasks of the data set, we report the results obtained from random guessing, the performance of the five models when trained from scratch on the data set, and the performance of the five models when fine-tuned on the MTF data set using pre-trained weights provided by the PyTorch library. These pre-trained weights were obtained by training the model using the ImageNet data set [23].

### 5.3.1. Face recognition task

Table 5 reports the performance of DL models on the MTF data set for the face recognition task, which comprises 240 labels.

<table><caption>Table 5: Performance of the DL models on the MTF data set for the face recognition task. Boldface figures are the best in each column.</caption><thead><tr><th>Model</th><th>Training</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1 score</th><th>Number of epochs</th><th>Training time (minutes)</th></tr></thead><tbody><tr><td>Random guess</td><td>N/A</td><td>0.16%</td><td>0.00%</td><td>0.42%</td><td>0.00%</td><td>N/A</td><td>N/A</td></tr><tr><td rowspan="2">MobileNet_v3</td><td>From scratch</td><td>2.88%</td><td>0.01%</td><td>0.42%</td><td>0.02%</td><td>70</td><td><strong>94.77</strong></td></tr><tr><td>Pre-trained</td><td>57.35%</td><td>49.77%</td><td>46.16%</td><td>43.88%</td><td>71</td><td>94.99</td></tr><tr><td rowspan="2">AlexNet</td><td>From scratch</td><td>18.93%</td><td>8.71%</td><td>10.76%</td><td>7.97%</td><td>76</td><td>95.56</td></tr><tr><td>Pre-trained</td><td>36.66%</td><td>27.34%</td><td>25.81%</td><td>23.94%</td><td>82</td><td>101.96</td></tr><tr><td rowspan="2">ResNet 50</td><td>From scratch</td><td>9.58%</td><td>3.13%</td><td>4.30%</td><td>3.13%</td><td>100</td><td>168.24</td></tr><tr><td>Pre-trained</td><td>58.39%</td><td>46.94%</td><td>46.01%</td><td>44.61%</td><td>94</td><td>151.95</td></tr><tr><td rowspan="2">VGG16</td><td>From scratch</td><td>23.96%</td><td>17.89%</td><td>16.08%</td><td>14.57%</td><td>70</td><td>184.13</td></tr><tr><td>Pre-trained</td><td>34.66%</td><td>24.21%</td><td>23.63%</td><td>21.23%</td><td>74</td><td>156.77</td></tr><tr><td rowspan="2">ConvNeXT</td><td>From scratch</td><td>13.26%</td><td>5.13%</td><td>6.60%</td><td>4.60%</td><td>71</td><td>216.9</td></tr><tr><td>Pre-trained</td><td><strong>79.87%</strong></td><td><strong>76.29%</strong></td><td><strong>73.31%</strong></td><td><strong>74.07%</strong></td><td>76</td><td>201.74</td></tr></tbody></table>

As expected, all trained models achieved better results than the random guess. However, the MobileNet_v3 model trained from scratch exhibited subpar performance, just slightly above the random baseline (accuracy 2.88% and F1 score 0.02%). Remarkably, its low recall of just 0.42% indicates that this model predicted all images to a single label, suggesting a failure to accurately extract image features for precise predictions. This outcome for MobileNet_v3 was unsurprising, because this model was designed for simple and fast training in embedded systems. The other models trained from scratch exhibited better
performance than the aforementioned model, with higher recall scores implying predictions across multiple labels, and potentially benefiting from increased training epochs and higher learning rates.

As anticipated, all fine-tuned models with pre-trained weights outperformed their scratch-trained counterparts. This can be attributed to the pre-trained layers of the models, which are adept at identifying important image features and passing them to the model's classifier. The recall scores of the pre-trained models indicate successful predictions across multiple correct labels. Considering the uneven distribution of the data, the ConvNeXT model stood out with high results, with an accuracy of 79.78% and an F1 score of 74.07% due to its novelty and ability to tackle complex tasks using the Vision Transformer. Despite this model showing high performance in the FR task, the complexity of this highly multidimensional classification challenge presents significant difficulties for all other models, including the ConvNeXT model when trained from scratch. As a result, the MTF data set emerges as a valuable asset to encourage and support future research in this area.

When examining the runtime required to train the models, we observe that each pre-trained model exhibits a very similar training duration if compared to its counterpart trained from scratch. The slight discrepancy in training times can be attributed to the variation in the number of epochs required to achieve the final model. The training time is contingent on the model's complexity and the number of parameters. The longer training times of ResNet50 and ConvNeXT are due to the former using residual layers and the latter employing Vision Transformers, which contribute to their higher complexity. Furthermore, the smaller batch size used in the training of VGG16 and ConvNeXT played a role in the extended training time required by these models.

### 5.3.2. Race classification

The race classification task within the presented MTF data set involves a four-label classification objective. The relative differences across models follow what was observed in the previous FR task.
<table><caption>Table 6: Performance of the DL models on the MTF data set for the race classification task. Boldface figures are the best in each column.</caption><thead><tr><th>Model</th><th>Training</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1 score</th><th>Number of epochs</th><th>Training time (minutes)</th></tr></thead><tbody><tr><td>Random guess</td><td>N/A</td><td>15.34%</td><td>3.83%</td><td>25.00%</td><td>6.65%</td><td>N/A</td><td>N/A</td></tr><tr><td rowspan="2">MobileNet_v3</td><td>From scratch</td><td>40.97%</td><td>10.24%</td><td>25.00%</td><td>14.53%</td><td>70</td><td><b>92.4</b></td></tr><tr><td>Pre-trained</td><td>85.62%</td><td>83.12%</td><td>81.09%</td><td>82.01%</td><td>72</td><td>98.97</td></tr><tr><td rowspan="2">AlexNet</td><td>From scratch</td><td>67.01%</td><td>60.13%</td><td>55.31%</td><td>56.23%</td><td>73</td><td>92.97</td></tr><tr><td>Pre-trained</td><td>87.22%</td><td>85.82%</td><td>84.49%</td><td>85.08%</td><td>78</td><td>94.22</td></tr><tr><td rowspan="2">ResNet 50</td><td>From scratch</td><td>60.06%</td><td>44.24%</td><td>45.46%</td><td>43.39%</td><td>74</td><td>106.84</td></tr><tr><td>Pre-trained</td><td>90.18%</td><td>88.95%</td><td>87.16%</td><td>87.98%</td><td>71</td><td>103.1</td></tr><tr><td rowspan="2">VGG16</td><td>From scratch</td><td>78.04%</td><td>74.65%</td><td>71.36%</td><td>72.70%</td><td>70</td><td>134.27</td></tr><tr><td>Pre-trained</td><td>92.33%</td><td>90.66%</td><td>90.95%</td><td>90.78%</td><td>74</td><td>177.13</td></tr><tr><td rowspan="2">ConvNeXT</td><td>From scratch</td><td>61.50%</td><td>51.40%</td><td>48.43%</td><td>48.16%</td><td>82</td><td>233.86</td></tr><tr><td>Pre-trained</td><td><b>95.77%</b></td><td><b>94.41%</b></td><td><b>94.94%</b></td><td><b>94.65%</b></td><td>70</td><td>207.07</td></tr></tbody></table>

As shown in Table 6, all models demonstrated better performance than the random guess by the end of their training, whether pre-trained or trained from scratch. The training times to reach their final form were similar to those shown in Table 5. The slight differences in training times among tasks were due to varying numbers of training epochs, after which all models reached the stopping condition of non-improved loss for five successive epochs.

The data set exhibits an imbalanced nature for this task, with two majority labels: Asian (Chinese/Korean) accounting for 32.38% of the data, and White comprising 40.97% of the data. On the other hand, two minority labels are present: Asian (Indian) making up 15.63% of the data, and Black representing 11.02% of the data. This imbalance severely affected the MobileNet_v3 model when trained from scratch: its recall was just 25%, which indicates that all images were predicted to belong to a single label. That behavior could be attributed to the simplicity of the model and the low number of parameters, that prevented the model from handling such complex data. Moreover, the other models trained from scratch (ResNet_50, AlexNet, and ConvNeXT) were also impacted by the data distribution, with accuracies lower than 70% and recall scores hovering around 50%, which suggest they were primarily classifying images into two labels. The exception was the VGG16 model, which achieved a relatively high accuracy of 78.04% and a recall score of 71.36%, therefore indicating a successful multi-label image classification.
If we examine the pre-trained models, all of them outperformed their trained-from-scratch counterparts, boasting recall scores over 81%. This demonstrates their ability to correctly classify images into the four labels and their resilience to the data set imbalance. The most complex models, ConvNeXT and VGG16, achieved impressively high accuracy and F1 scores, both exceeding 90%. Their accuracies were 95.77% and 92.33%, respectively, with F1 scores of 94.65% and 90.78%.

### 5.3.3. Gender classification

The gender classification task in the MTF data set involves a binary classification problem, where the data distribution is well balanced: approximately 47.12% of the images are labeled as males, and 52.88% are labeled as females in both the training and the test sets. Thus, GC should be an 'easy' problem for most DL models, as the results reported in Table 7 suggest.

<table><caption>Table 7: Performance of the DL models on the MTF data set for the gender classification task. Boldface figures are the best in each column.</caption><thead><tr><th>Model</th><th>Training</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1 score</th><th>Number of epochs</th><th>Training time (minutes)</th></tr></thead><tbody><tr><td>Random guess</td><td>N/A</td><td>45.61%</td><td>40.60%</td><td>43.88%</td><td>39.18%</td><td>N/A</td><td>N/A</td></tr><tr><td rowspan="2">MobileNet_v3</td><td>From scratch</td><td>52.88%</td><td>26.44%</td><td>50.00%</td><td>34.59%</td><td>59</td><td><strong>80.3</strong></td></tr><tr><td>Pre-trained</td><td>76.60%</td><td>77.94%</td><td>75.89%</td><td>75.93%</td><td>74</td><td>100.74</td></tr><tr><td rowspan="2">AlexNet</td><td>From scratch</td><td>70.13%</td><td>71.35%</td><td>69.28%</td><td>69.05%</td><td>71</td><td>90.33</td></tr><tr><td>Pre-trained</td><td>88.18%</td><td>88.16%</td><td>88.11%</td><td>88.13%</td><td>73</td><td>92.99</td></tr><tr><td rowspan="2">ResNet 50</td><td>From scratch</td><td>70.85%</td><td>70.80%</td><td>70.57%</td><td>70.62%</td><td>78</td><td>142.16</td></tr><tr><td>Pre-trained</td><td>97.60%</td><td>97.59%</td><td>97.61%</td><td>97.60%</td><td>71</td><td>127.24</td></tr><tr><td rowspan="2">VGG16</td><td>From scratch</td><td>90.65%</td><td>90.61%</td><td>90.67%</td><td>90.63%</td><td>76</td><td>181.9</td></tr><tr><td>Pre-trained</td><td>97.28%</td><td>97.29%</td><td>97.26%</td><td>97.27%</td><td>80</td><td>211.4</td></tr><tr><td rowspan="2">ConvNeXT</td><td>From scratch</td><td>73.80%</td><td>73.92%</td><td>73.97%</td><td>73.80%</td><td>71</td><td>209.88</td></tr><tr><td>Pre-trained</td><td><strong>98.88%</strong></td><td><strong>98.86%</strong></td><td><strong>98.90%</strong></td><td><strong>98.88%</strong></td><td>74</td><td>219.22</td></tr></tbody></table>

When MobileNet_v3 was trained from scratch, it exhibited low performance, as it did in the previous tasks: it achieved accuracy 52.88%, and recall 50%, indicating that it labeled all images as females. Thus, the proposed MTF data set contains images more complicated than the MobileNet_v3 model can handle even for the easiest task.

However, the remaining trained-from-scratch models performed better, with accuracies exceeding 70% and recall scores over 69%. Among them, VGG16 stood out, achieving impressive results with an accuracy 90.65% and a recall
90.67%. The complexity of the VGG16 network and its extensive parameter count allowed it to accurately classify the images.

On the other hand, models fine-tuned with pre-trained weights demonstrated superior performance compared to the previously discussed models. Three of these models achieved accuracies over 97%. Ranked from best to worst accuracy, these models are ConvNeXT, ResNet 50, and VGG16, with accuracies 98.88%, 97.60%, and 97.28% respectively, and F1 scores of 98.88%, 97.60%, and 97.27%, respectively. These outstanding results can be attributed to the capabilities of these models, including their large number of parameters and their use of novel layers such as residual and transformer layers.

Regarding the training runtimes, they are comparable to those of the previous tasks for each respective model, taking into consideration the number of training epochs needed for each model to converge, as well as the differences in batch sizes used.

### 5.3.4. Age classification

The age classification task in the MTF data set is also a binary classification challenge. However, unlike the GC task, the data distribution for this task exhibits a significant bias, with 89.94% of the data labeled as 'young' and only 10.06% labeled as 'old' in both the training and test sets.

<table><caption>Table 8: Performance of the DL models on the MTF data set for the age classification task. Bold figures are the best in each column.</caption><thead><tr><th>Model</th><th>Training</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1 score</th><th>Number of epochs</th><th>Training time (minutes)</th></tr></thead><tbody><tr><td>Random guess</td><td>N/A</td><td>39.06%</td><td>51.80%</td><td>54.49%</td><td>35.27%</td><td>N/A</td><td>N/A</td></tr><tr><td rowspan="2">MobileNet_v3</td><td>From scratch</td><td>90.01%</td><td>44.97%</td><td>50.00%</td><td>47.35%</td><td>72</td><td>101.19</td></tr><tr><td>Pre-trained</td><td>94.81%</td><td>91.06%</td><td>78.08%</td><td>83.03%</td><td>70</td><td>96.67</td></tr><tr><td rowspan="2">AlexNet</td><td>From scratch</td><td>90.50%</td><td>86.20%</td><td>53.48%</td><td>54.06%</td><td>65</td><td><strong>94.67</strong></td></tr><tr><td>Pre-trained</td><td>95.93%</td><td>90.71%</td><td>85.75%</td><td>88.03%</td><td>67</td><td>96.86</td></tr><tr><td rowspan="2">ResNet 50</td><td>From scratch</td><td>90.02%</td><td>78.37%</td><td>50.75%</td><td>48.92%</td><td>70</td><td>110.96</td></tr><tr><td>Pre-trained</td><td>95.85%</td><td>89.14%</td><td>87.47%</td><td>88.28%</td><td>77</td><td>140</td></tr><tr><td rowspan="2">VGG16</td><td>From scratch</td><td>92.81%</td><td>90.26%</td><td>66.40%</td><td>72.21%</td><td>78</td><td>168.09</td></tr><tr><td>Pre-trained</td><td>96.33%</td><td>91.82%</td><td>87.03%</td><td>89.24%</td><td>72</td><td>149.55</td></tr><tr><td rowspan="2">ConvNeXT</td><td>From scratch</td><td>90.50%</td><td>86.20%</td><td>53.48%</td><td>54.06%</td><td>75</td><td>211.75</td></tr><tr><td>Pre-trained</td><td><strong>97.60%</strong></td><td><strong>95.13%</strong></td><td><strong>91.27%</strong></td><td><strong>93.09%</strong></td><td>73</td><td>220.09</td></tr></tbody></table>

Interestingly, the high accuracy levels reached (above 90%) did not guarantee accurate predictions, as the data imbalance led to biased models. This is evident
when analyzing the precision, recall, and F1 score, where biased models had low
scores hovering around 50%, which reveal the impact of imbalanced predictions.
During training from scratch, only the VGG16 model appeared unbiased. This
observation suggests that smaller models and models with complex novel layers
are more susceptible to bias when trained from scratch.

On the other hand, fine-tuned models with pre-trained weights exhibited
better resistance to bias, as evidenced by the recall scores. All pre-trained
models reported recall rates above 78%, indicating the accurate classification of
images into the two labels. Notably, ConvNeXT stood out with an impressive
accuracy of 97.60% and an F1 score of 93.09%, making it the best-performing
model among those we tried.

The reported training times were again consistent with those observed during
the training of other tasks. This consistency is attributed to using the same
number of images and models across all tasks.

5.4. *The role of data processing*

As discussed in Section 3.2, the MTF data underwent a profound manual
processing whereby a large amount of low quality, inappropriate or noisy images
were filtered out. To illustrate the benefits that this manual processing brings to
the data set and the subsequent model training, in this section we compare the
performance of the best-performing DL model (ConvNeXT with pre-training)
on the four tasks when trained on two distinct sets of data: the raw, unprocessed
collection of 106,438 cropped images we crawled from the Internet, which form
a data set similar to other renowned published unprocessed data sets such as
Megaface and LFW, and the meticulously processed MTF data set comprising
5,246 images.

The unprocessed data were only subjected to automatic cropping and then
assigned the corresponding task labels employed to crawl them from the internet.
Then, these unprocessed images were split into a training set encompassing 70%
of the data, and a validation set comprising the remaining 30%.

The results of this experiment are presented in Table 9.
<table><caption>Table 9: Comparison of ConvNeXT with pre-training on MTF data vs. unprocessed cropped data</caption><thead><tr><th>Task</th><th>Training data</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1 score</th><th>Number of epochs</th><th>Training time (minutes)</th></tr></thead><tbody><tr><td rowspan="2">FR</td><td>Unprocessed</td><td>10.07%</td><td>8.85%</td><td>8.09%</td><td>7.01%</td><td>94</td><td>4089.72</td></tr><tr><td>MTF</td><td>79.87%</td><td>76.29%</td><td>73.31%</td><td>74.07%</td><td>76</td><td>201.74</td></tr><tr><td rowspan="2">RC</td><td>Unprocessed</td><td>65.38%</td><td>59.89%</td><td>58.67%</td><td>59.18%</td><td>87</td><td>3809.95</td></tr><tr><td>MTF</td><td>95.77%</td><td>94.41%</td><td>94.94%</td><td>94.65%</td><td>70</td><td>207.07</td></tr><tr><td rowspan="2">GC</td><td>Unprocessed</td><td>69.46%</td><td>69.38%</td><td>69.32%</td><td>69.34%</td><td>93</td><td>4094.41</td></tr><tr><td>MTF</td><td>98.88%</td><td>98.86%</td><td>98.90%</td><td>98.88%</td><td>74</td><td>219.22</td></tr><tr><td rowspan="2">AC</td><td>Unprocessed</td><td>62.57%</td><td>61.70%</td><td>61.57%</td><td>61.62%</td><td>89</td><td>3911.01</td></tr><tr><td>MTF</td><td>97.60%</td><td>95.13%</td><td>91.27%</td><td>93.09%</td><td>73</td><td>220.09</td></tr></tbody></table>

For the FR task, to enable a fair comparison, training on the raw data set only considered the images of the 240 identities in the MTF data set. The ConvNeXT model trained on the raw data set achieved a mere 10% accuracy. In contrast, the model trained on the processed MTF data set exhibited an impressive 79.87% accuracy in 240-label classification. The disparity in performance is evident from the F1 score, where the model trained on the MTF data set outperformed the model trained on the raw data by an order of magnitude. The observed discrepancy in performance was particularly expected in the FR task, given the key role of data processing to discard a large number of crawled images that did not correspond to the right celebrity, and given the substantial amount of noise in the raw data. These factors explain the large gap in results between the two data sets.

For the remaining three tasks (race, gender, and age classification), similar results were observed. The model trained on the raw data reported accuracy levels ranging from 62% to 69%, while the model trained on the MTF data set reported accuracies higher than 95%. Although these three tasks were comparatively easier for the models to learn, and the occurrence of incorrect images crawled from the search engine was low, the impact of manual processing by human experts to enhance the model performance was evident. This implies that, while models can learn some patterns from raw data, these raw data cannot be trusted to train accurate and robust models.

Training runtimes were also proportional to the size of the data. In this
case, the raw data set resulted in around 20 times higher training cost than the MTF data, and all this to produce poorly performing models. This illustrates the importance of data quality over data quantity, and the benefits of smaller but high-quality data sets like MTF against larger but noisier data sets such as Megaface and LFW.

## 6. Conclusions and future work

We have introduced the Multi-Task Faces (MTF) image data set, which encompasses face images suitable for four distinct tasks: face recognition, race classification, gender classification, and age classification. Contrary to similar data sets consisting of human faces which are at risk of being removed from the internet for privacy reasons, the MTF data set was compiled with a view to longevity thanks to compliance with the legal requirements of the GDPR and strict ethical considerations. MTF exclusively focuses on publicly recognized celebrities' images that were either public-domain or published under Creative Commons licenses granting permission for modification, sharing, and commercial use. In this paper, we have elaborated on the data preparation processes, while also presenting comprehensive statistical information that aids to better understand the quality of our data set.

Furthermore, the distribution of the data across the different tasks reflects the real distribution of the data available online. This distribution offers the researchers four distinct scenarios: a classification problem with hundreds of unbalanced labels in the face recognition task, a classification problem with four unbalanced labels in the race classification task, a binary classification problem with balanced labels in the gender classification task, and finally a binary classification problem with extremely unbalanced labels in age classification.

Our evaluation of the performance of five DL models on the four tasks of the MTF data set shows that the models with pre-trained weights have better performance than those trained from scratch, due to their prior training in extracting the important features in the images. The recently proposed Con-
vNeXT model with the Vision Transformer technique outperformed all other models in the four tasks. Additionally, to facilitate the reproducibility of the results we provide a repository with the data, the trained models, and the code used in the experiments of this paper.

Furthermore, we proved the importance of the data set processing by comparing the performance of the ConvNeXT model on the MTF data set against its performance on raw data directly cropped from the crawled images. The results of this experiment show that the processing of the data significantly improves their ability to train DL models: the models trained on the MTF data set for the various tasks greatly outperformed the models trained on the raw data.

As a future work, we plan to apply and test MTF on other tasks, such as face anonymization. In particular, the FR task in the MTF data set can be used to measure the residual re-identification risk of image anonymization algorithms, whereas the accuracy achieved for the other three tasks on the anonymized images would quantify the degree of utility preserved.

## Declaration of interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Data availability

The authors confirm that the MTF dataset, along with the source code and the trained models, has been made accessible through the following link https://github.com/RamiHaf/MTF_data_set, and is available under the Creative Commons copyright license.
Acknowledgement

We would like to acknowledge the effort and generosity of the owners of
the images in making these public-domain or releasing them under the most
flexible Creative Commons license. Image owners are acknowledged in a CSV
file named "URL of original images" and included in the zipfile of the MTF
data set, that links all the original sources of the images. This research was par-
tially supported by MCIN/AEI/ 10.13039/501100011033 (PhD grant PRE2019-
089210 and project PID2021-123637NB-I00 "CURLING"), INCIBE (project
"HERMES" and INCIBE-URV cybersecurity chair), the European Commission
(projects H2020-871042 "SoBigData++" and H2020-101006879 "MobiDataLab"),
and the Government of Catalonia (ICREA Acadèmia Prizes to J. Domingo-
Ferrer and to D. Sánchez, and grant 2021 SGR 00115). The authors are with
the UNESCO Chair in Data Privacy, but the views in this paper are their own
and are not necessarily shared by UNESCO.

References

[1] A. Jain, H. Patel, L. Nagalapatti, N. Gupta, S. Mehta, S. Guttula, S. Mu-
jumdar, S. Afzal, R. Sharma Mittal, V. Munigala, Overview and impor-
tance of data quality for machine learning tasks, in: Proceedings of the
26th ACM SIGKDD international conference on knowledge discovery &
data mining, 2020, pp. 3561–3562.

[2] R. Ducato, A. Strowel, Limitations to text and data mining and consumer empowerment: making the case for a right to “machine legibility”, IIC-International Review of Intellectual Property and Competition Law 50 (6) (2019) 649–684.

[3] G. Voyatzis, I. Pitas, Protecting digital-image copyrights: A framework, IEEE Computer Graphics and Applications 19 (1) (1999) 18–24. doi:10.1109/38.736465.
[4] K. A. Philbrick, A. D. Weston, Z. Akkus, T. L. Kline, P. Korfiatis, T. Sakinis, P. M. Kostandy, A. Boonrod, A. Zeinoddini, N. Takahashi, B. J. Erickson, RIL-contour: a medical imaging dataset annotation tool for and with deep learning, J. Digit. Imaging 32 (4) (2019) 571–581. doi:10.1007/s10278-019-00232-0.

[5] P. Voigt, A. Von dem Bussche, The EU General Data Protection Regulation (GDPR), A Practical Guide, 1st Ed., Cham: Springer International Publishing 10 (3152676) (2017) 10–5555.

[6] N. A. Smuha, The EU approach to ethics guidelines for trustworthy artificial intelligence, Computer Law Review International 20 (4) (2019) 97–106.

[7] T. Mitchell, CMU face images, http://archive.ics.uci.edu/dataset/124/cmu+face+images, accessed: November 2, 2023 (Jun. 1999). doi:10.24432/C5JC79.
URL https://doi.org/10.24432/C5JC79

[8] Z. Liu, P. Luo, X. Wang, X. Tang, Deep learning face attributes in the wild, in: Proceedings of International Conference on Computer Vision (ICCV), 2015.

[9] E. Rosati, Copyright and the Court of Justice of the European Union, Oxford University Press, Oxford, 2019.

[10] G. Bae, M. de La Gorce, T. Baltrušaitis, C. Hewitt, D. Chen, J. Valentin, R. Cipolla, J. Shen, DigiFace-1M: 1 million digital face images for face recognition, in: 2023 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, 2023.

[11] G. B. Huang, M. Ramesh, T. Berg, E. Learned-Miller, Labeled faces in the wild: A database for studying face recognition in unconstrained environments, Tech. Rep. 07-49, University of Massachusetts, Amherst (October 2007).
[12] I. Kemelmacher-Shlizerman, S. M. Seitz, D. Miller, E. Brossard, The Megaface benchmark: 1 million faces for recognition at scale, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4873-4882.

[13] S. Yang, P. Luo, C. C. Loy, X. Tang, WIDER FACE: A face detection benchmark, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

[14] B. Yang, J. Yan, Z. Lei, S. Z. Li, Fine-grained evaluation on face detection in the wild, in: Automatic Face and Gesture Recognition (FG), 11th IEEE International Conference on, IEEE, 2015.

[15] R. Rothe, R. Timofte, L. V. Gool, Deep expectation of real and apparent age from a single image without facial landmarks, International Journal of Computer Vision 126 (2-4) (2018) 144-157.

[16] T. Karras, S. Laine, T. Aila, A style-based generator architecture for generative adversarial networks, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4401-4410, https://github.com/NVlabs/ffhq-dataset.

[17] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, A. Zisserman, VGGFace2: A dataset for recognising faces across pose and age, in: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), IEEE, 2018, pp. 67-74.

[18] A. Bansal, A. Nanduri, C. Castillo, R. Ranjan, R. Chellappa, UMDFaces: An annotated face dataset for training deep networks, in: 2017 IEEE International Joint Conference on Biometrics (IJCB), IEEE, 2017, pp. 464-473.

[19] P. Viola, M. Jones, Rapid object detection using a boosted cascade of simple features, in: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001, Vol. 1, IEEE, 2001, pp. I-I.
[20] A. Mordvintsev, A. K. OpenCV-Python Tutorials Documentation: Release 1, https:// opencv24-python-tutorials.readthedocs.io/_/ downloads/en/stable/pdf/ (2017).

[21] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, et al., Searching for mobilenetv3, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1314–1324.

[22] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: P. L. Bartlett, F. C. N. Pereira, C. J. C. Burges, L. Bottou, K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, 2012, pp. 1106–1114.

[23] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, ImageNet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255.

[24] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.

[25] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, in: Y. Bengio, Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1409.1556

[26] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, S. Xie, A ConvNet for the 2020s, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11976–11986.
[27] Y. Ho, S. Wookey, The real-world-weight cross-entropy loss function: Mod-
eling the costs of mislabeling, IEEE access 8 (2019) 4806–4813.

[28] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: Y. Bengio, Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980

[29] C. Kim, S. Kim, J. Kim, D. Lee, S. Kim, Automated learning rate scheduler for large-batch training, CoRR abs/2107.05855 (2021). arXiv:2107.05855. URL https://arxiv.org/abs/2107.05855

[30] S. Imambi, K. B. Prakash, G. Kanagachidambaresan, Pytorch, Programming with TensorFlow: Solution for Edge Computing Applications (2021) 87–104.