# ViTFER: Facial Emotion Recognition with Vision Transformers
## 3 Data Integration/Augmentation and Analysis
### Datasets  
**FER-2013:** FER-2013 is a dataset composed of 35.953 images in seven classes (fear, disgust, sad, happy, neutral, surprise, angry). As seen in Figure 1, Images are 48 \(\times\) 48 pixels in size with a grey-scaled color palette. The classes' variations and feature distributions are helpful in the merging phase for other classes to obtain a good distribution and normalize the amount of data variation. According to the final classification, the contempt class was missed for this type of data. Its composition already divides its original samples into training and validation sets using different folders with the function of labeling. We cannot use this structure because it is necessary to merge it with other datasets and proceed with the splitting into three different subfolders (training, testing, and validation). Finally, we can analyze some features of the sample distribution separately between training and validation folders. Figure 2 represents the graph distribution of various classes in the FER2013 dataset, where red bars indicate values below the mean and green indicates values above the mean, where means are equal to 4100 and 1100 for training and validation sets, respectively. FER-2013 does not have many samples for the disgust class. This observation provides excellent motivation to merge it with other datasets and perform the data augmentation phase. Furthermore, it is unbalanced, and we must manage it in data augmentation proportionally or data integration with some samples from other datasets to obtain a result set with the same number of samples for each class. * **CK+48:** CK+48 is a small dataset composed of 981 images in seven classes (fear, disgust, sad, happy, neutral, surprise, angry). Images are 48 \(\times\) 48 in size with a grey-scaled color palette as shown in Figure 3. The classes' variations and feature distributions are helpful in the merging phase for other classes to obtain a good distribution and  
Figure 1: Samples from the FER-2013 dataset. Figure 2: Distribution of classes in the FER-2013 dataset. Graph represents the mean values of green and red bars. normalize the amount of data variation. Generally, images taken from video frames did not have much variation, and the total number of elements is negligible compared to other datasets. Compared with the FER-2013, images are in frontal view with a clean pattern for facial expression. CK+48 contained less than 1000 images and did not have a splitting into the training and validation sets as FER-2013. We will use it as part of the final training set for each dataset integration in our experiments. Figure 4 represents the graph distribution of various classes in the CK+48 dataset, where red bars indicate values below the mean and green indicates values above the mean, where the mean is equal to 142. According to the final classification, we do not have any samples for the disgust class, and contempt has fewer samples; pixel values are in the total range (0.255) for a single channel. Images are in PNG format; we will convert them to JPG to normalize images under a single format. The algorithms used for compression are the primary distinction between JPG and PNG. To make the file smaller, JPG employs a lossy compression method that discards some of the image data. PNG, in contrast, employs a lossless technique that preserves all the data; however, given the great number of files that we will generate from data augmentation, we decided to have lower-quality resolution images with reduced size compared to high-quality ones because of the small amount of data in PNG; the image noise applied on their conversion did not have a proportionally huge impact on the final dataset, and some added noise can increase the resistance of the model. Figure 4: Distribution of the CK+48 dataset with the classification of classes describing their mean values. Figure 3: Samples from the CK+48 dataset. * **AffectNet:** The AffectNet dataset [28] has samples of different sizes and high-quality images in gray-scale or color in the RGB range.