SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuhang Zhang, Yaqi Li, Lixiong Qin, Xuannan Liu, Weihong Deng  
Beijing University of Posts and Telecommunications  
{zyhzyh, yaqili, lixiongqin, xuannanliu, whdeng}@bupt.edu.cn
Facial expression data is characterized by a significant imbalance, with most collected data showing happy or neutral expressions and fewer instances of fear or disgust. This imbalance poses challenges to facial expression recognition (FER) models, hindering their ability to fully understand various human emotional states. Existing FER methods typically report overall accuracy on highly imbalanced test sets but exhibit low performance in terms of the mean accuracy across all expression classes. In this paper, our aim is to address the imbalanced FER problem. Existing methods primarily focus on learning knowledge of minor classes solely from minor-class samples. However, we propose a novel approach to extract extra knowledge related to the minor classes from both major and minor class samples. Our motivation stems from the belief that FER resembles a distribution learning task, wherein a sample may contain information about multiple classes. For instance, a sample from the major class surprise might also contain useful features of the minor class fear. Inspired by that, we propose a novel method that leverages re-balanced attention maps to regularize the model, enabling it to extract transformation invariant information about the minor classes from all training samples. Additionally, we introduce re-balanced smooth labels to regulate the cross-entropy loss, guiding the model to pay more attention to the minor classes by utilizing the extra information regarding the label distribution of the imbalanced training data. Extensive experiments on different datasets and backbones show that the two proposed modules work together to regularize the model and achieve state-of-the-art performance under the imbalanced FER task. Code is available at https://github.com/zyh-uaiaaaa.
Facial expression recognition (FER) plays a crucial role in enabling machines to understand human emotional states, thus facilitating the realization of machine intelligence [18, 11, 25]. Recent research efforts [42, 35, 49] have employed in-the-wild datasets to train FER models, resulting in impressive performance in terms of overall classification accuracy. However, these FER datasets suffer from a significant imbalance, with a higher abundance of happy and neutral expression faces, which are easily obtainable from the internet and daily life, compared to faces displaying negative expressions such as fear or disgust [6, 32]. This imbalance can have adverse effects on human-computer interaction when FER models misinterpret infrequently occurring negative emotions as frequently occurring positive emotions.
In this paper, our objective is to investigate imbalanced learning in the context of FER. We observe that existing FER methods [34, 43, 42, 29] yield relatively low performance in terms of the mean accuracy across all classes, mainly due to the highly imbalanced nature of the training data. While imbalanced (long-tailed) learning methods in the image classification domain have shown some
improvements [52, 5], they provide limited enhancements in the mean accuracy within the FER field. This can be attributed to several factors. Firstly, FER datasets typically contain a small number of classes, resulting in marginal improvements in the mean accuracy of all classes. Secondly, the increase in accuracy for minor classes often comes at the expense of decreased accuracy for major classes. This imbalance disproportionately affects the overall accuracy, particularly when evaluating on imbalanced test sets.
To address the aforementioned problem, our objective is to design a method that can maintain high performance on major classes while improving the performance on minor classes to the greatest extent possible. We refrain from modifying the dataset structure through under-sampling or over-sampling, as these approaches may enhance the performance of minor classes at the expense of degrading the performance on major classes. Instead, we aim to leverage each sample fully to extract additional knowledge relevant to minor classes. Our motivation is that FER resembles a label distribution learning task, implying that samples from major classes may contain information pertaining to minor classes as well. For instance, the major class "surprise" shares certain similarities with the minor class "fear".
Inspired by that, we propose a novel method that mines extra knowledge regarding minor classes from samples belonging to both major and minor classes. This extra information enhances the recognition performance on minor classes without significantly compromising the performance on major classes. Specifically, we employ the concept of attention map consistency [7], which is utilized in the EAC method [50] to prevent FER models from memorizing noisy labels. We observe that EAC employs attention maps from all expression classes, rather than just the labeled class, to regularize a specific sample, resembling the idea of label distribution learning. Building upon this insight, we introduce a re-balanced attention consistency module to address the imbalanced FER task for the first time. In our approach, attention maps for all expression classes can be extracted from a given FER sample. We introduce re-balanced weights, following the design of [5], which are set inversely proportional to the effective number of samples belonging to each class. These weights facilitate the model in extracting additional knowledge related to minor classes from all training samples. Furthermore, as the classification loss may still be affected by the imbalanced data, we propose a re-balanced smooth labels module that utilizes the acquired re-balanced weights and the extra knowledge of the imbalanced label distribution. This module guides the model to focus more on minor classes during the decision-making process.
To demonstrate the effectiveness of our proposed method, we conducted extensive experiments on different FER datasets, including datasets with varying imbalance factors [5]. Ablation studies confirmed that each proposed module contributes to the enhancement of mean accuracy, and the two modules synergistically achieve state-of-the-art performance. Moreover, we evaluated the generalization ability of our method by combining it with different backbones including transformers. The experimental results showcased that our method is lightweight, easy to implement, and seamlessly compatible with various backbones, thus significantly improving their performance.
The main contributions of our work are listed as follows:
- We highlight the imbalanced learning problem in FER and find existing methods in the large-scale image classification field might fall short in the FER field, we benchmark existing FER methods utilizing both overall accuracy and mean accuracy on all classes.
- We propose a novel method consisting of two modules: re-balanced attention consistency (RAC) and re-balanced smooth labels (RSL). RAC mines extra knowledge pertaining to minor classes from both major and minor samples, thereby enhancing performance on minor classes without compromising performance on major classes. RSL leverages the imbalanced label distribution to further regularize the classification loss and prioritize the decision-making process for minor classes.
- Our proposed method is easy to implement, lightweight, and seamlessly compatible with different backbone architectures including transformers. Through extensive experiments, we validate that our proposed method achieves state-of-the-art performance in terms of both overall accuracy and mean accuracy of all classes across different FER datasets and backbone architectures.
Facial expression recognition methods in recent years primarily focus on extracting effective features from in-the-wild datasets using deep learning models [22, 46, 13, 16, 20, 24, 21, 23]. In-the-wild datasets [26, 6] pose greater challenges and are more prone to noise due to label inaccuracies, difficult poses, occlusions, and low-quality images compared to laboratory collected datasets [30, 14, 40]. These factors adversely affect the recognition performance of facial expression recognition (FER) models. Several works have addressed the real-world FER task. SCN [42] introduces a learnable temperature and a relabel module to weight FER samples and suppress noise. DMUE [37] learns sub-modules for each class to handle noisy labels. RUL [49] treats expression uncertainty as a relative concept and learns uncertainty values for images through comparisons, aiding overall feature learning. TransFER [45] pioneers the use of vision transformers for in-the-wild FER. EAC [50] employs erasing attention map consistency to prevent FER models from memorizing noisy labels in real-world FER datasets.
While these approaches achieve impressive overall accuracy on test sets, they tend to exhibit lower performance in terms of mean accuracy of all classes, particularly for minor classes. In fact, previous works primarily focus on mitigating noise in in-the-wild datasets, often overlooking the imbalanced nature of FER datasets. In this work, our objective is to address imbalanced learning in in-the-wild FER datasets and provide an orthogonal supplement for previous works.
Imbalanced learning in real-life datasets, where certain classes have a majority of samples while others have very few, has been addressed through three perspectives: data pre-processing, loss re-weighting, and model ensemble. This paper primarily focuses on the first two categories, as model ensemble methods [44, 48] are computationally intensive. Data pre-processing methods commonly involve data re-sampling [31, 38]. However, studies [15, 52] have shown that data re-sampling can negatively impact representation learning. Over-sampling may introduce duplicated samples, leading to the increased risk of overfitting, while under-sampling may discard valuable examples. Another technique is data augmentation [4, 53], which applies predefined transformations to augment the dataset, particularly for minority classes. However, finding effective augmentation methods for facial expression recognition (FER) data, which contain specific local features related to expressions, can be challenging. Loss re-weighting methods assign different weights to classes or instances during training, known as loss re-weighting [27, 39, 41, 5]. The aim is to propagate appropriate gradient values for all classes during training.
In this section, we present a novel method for imbalanced facial expression recognition. Our approach focuses on extracting extra information of minor classes from both minor and major class samples, rather than solely relying on minor class samples. We introduce two modules, namely re-balanced attention consistency (RAC) and re-balanced smooth labels (RSL), to guide the model to extract balanced information from the entire training set.
3.1 Re-balanced attention consistency
We first introduce attention map consistency [7], which regularizes the attention maps to follow the same spatial transform if the input images are spatially transformed, which can help the model to learn transformation invariant features. EAC designs an imbalanced framework to utilize attention map consistency to prevent the models from memorizing the noisy labels. We find that instead of only extracting attention maps of the latent truth like utilizing GradCAM [36], EAC uses attention maps of all classes to regularize the FER model. We speculate that the reason lies in that utilizing attention maps of all classes is similar to label distribution learning [3, 51, 19], which mines information of several classes from one training sample. Inspired by that, we adapt attention map consistency to solve the imbalanced learning problem of FER for the first time. To be more specific, given a sample, the attention maps of all expression classes should follow the same spatial transform if the given sample is spatially transformed. Thus, the model could mine useful minor-class information from all samples instead of only the samples from the minor classes. Furthermore, we introduce the re-balanced attention consistency to regularize the model to focus more on the minor classes to achieve balanced learning. We set different weights to the attention maps of different classes
Figure 1: The illustration of re-balanced attention consistency. We propose re-balanced attention consistency to facilitate the model to mine extra transformation invariant knowledge of minor classes from both major and minor-class samples, which boosts the classification accuracy on minor classes while do not degrade the high accuracy on major classes.
and guide the model to learn more invariant information related to minor classes. Inspired by [5], which designs a re-weighting scheme that uses the effective number of samples of each class to re-balance the classification loss, we propose to use the effective number of samples to inversely the weigh attention maps of different classes. In the following, we formulate the re-balanced attention consistency module in detail.
Given images  $\mathbf{x}$ , we first flip them to get their flipped counterparts  $\widetilde{\mathbf{x}}$ . The features of  $\mathbf{x}$  and  $\widetilde{\mathbf{x}}$  are extracted from the last convolutional layer, denoted as  $\mathbf{F} \in \mathbb{R}^{N \times C \times H \times W}$  and  $\widetilde{\mathbf{F}} \in \mathbb{R}^{N \times C \times H \times W}$ , where  $N, C, H$  and  $W$  respectively represent the number of images, channels, height and width. We input  $\mathbf{F}$  and  $\widetilde{\mathbf{F}}$  to the global average pooling (GAP) layer to get features  $\mathbf{f} \in \mathbb{R}^{N \times C \times 1 \times 1}$  and  $\widetilde{\mathbf{f}} \in \mathbb{R}^{N \times C \times 1 \times 1}$ , and then resize them to  $N \times C$ . The classification loss is computed according to
$$
l _ {c l s} = - \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\log \left(\frac {e \mathbf {W} _ {y _ {i}} \mathbf {f} _ {i}}{\sum_ {l = 1} ^ {L} e \mathbf {W} _ {l} \mathbf {f} _ {i}}\right) + \log \left(\frac {e \mathbf {W} _ {y _ {i}} \tilde {\mathbf {f}} _ {i}}{\sum_ {l = 1} ^ {L} e \mathbf {W} _ {l} \tilde {\mathbf {f}} _ {i}}\right)\right), \tag {1}
$$
where  $\mathbf{W}_{y_i}$  is the  $y_{i}$ -th weight of the fully connected (FC) layer and  $y_{i}$  is the label of  $\mathbf{x}_i$ ,  $L$  is the total number of expression classes. CAM [53] is utilized to compute attention maps  $\mathbf{A} \in \mathbb{R}^{N \times L \times H \times W}$  and  $\widetilde{\mathbf{A}} \in \mathbb{R}^{N \times L \times H \times W}$  for  $\mathbf{x}$  and  $\widetilde{\mathbf{x}}$  following
$$
A (i, l, h, w) = \sum_ {c = 1} ^ {C} W (l, c) F (i, c, h, w), \tag {2}
$$
where  $i, l, c, h, w$  represent the sample, expression class, channel, height and width number. We then re-balance the attention maps  $\mathbf{A}$  and  $\widetilde{\mathbf{A}}$  through weight  $\mathbf{B} \in \mathbb{R}^L$  to get the re-balanced attention maps  $\mathbf{M}$  and  $\widetilde{\mathbf{M}}$  following
$$
M (i, l, h, w) = B _ {l} \cdot A (i, l, h, w). \tag {3}
$$
The balance weight  $\mathbf{B} \in \mathbb{R}^{L}$  is enlightened by the re-balanced weight based on the effective number in [5], which is computed following
$$
B _ {l} = \frac {1 - \beta}{1 - \beta^ {n _ {l}}}, \tag {4}
$$
where  $n_l$  is the number of training samples in class  $l$ ,  $\beta \in [0,1)$  is the hyperparameter controlling the re-balanced weight, a larger value of  $\beta$  emphasizes more of the minor samples, following [5], we set the  $\beta$  as 0.9999 across all our experiments. As the attention map before and after the flip transformation should be consistent with each other, we compute the consistency loss between  $\mathbf{M}$  and  $Flip(\widetilde{\mathbf{M}})$  following
$$
l _ {c o n s} = \frac {1}{N L H W} \sum_ {i = 1} ^ {N} \sum_ {l = 1} ^ {L} \sum_ {h = 1} ^ {H} \sum_ {w = 1} ^ {W} | | M (i, l, h, w) - F l i p (\widetilde {M}) (i, l, h, w) | | _ {2}. \tag {5}
$$
The training loss is computed as
$$
l _ {\text {t r a i n}} = l _ {\text {c l s}} + \lambda l _ {\text {c o n s}}, \tag {6}
$$
where  $\lambda$  is the weight of the attention consistency loss, which determines the relative importance of the consistency loss compared to the classification loss in the overall training objective.
Figure 2: The illustration of re-balanced smooth labels. We propose re-balanced smooth labels to utilize the existing prior knowledge about the label distribution of the training set to guide the model towards placing more emphasis on minor classes during decision-making, while maintaining high performance on major classes.
3.2 Re-balanced smooth labels
In Section 3.1, we introduced the re-balanced attention consistency module, which enables the model to extract balanced and transformation invariant knowledge of minor classes from all training samples, resulting in improved classification accuracy for those classes. However, the imbalanced training set can still negatively impact the classification loss, denoted as  $l_{cls}$ . To address this, we present the re-balanced smooth labels module in this section. This module aims to regulate the classification loss and promote balanced learning.
We denote the prediction of the FER model towards  $\mathbf{x}_i$  as  $\mathbf{p}_i$ , where  $\mathbf{p}_i$  is the likelihood the model assigns to the  $i$ -th given sample  $\mathbf{x}_i$ , the one-hot label of  $\mathbf{x}_i$  is denoted as  $\mathbf{y}_i$ . For simplicity, we re-write the classification loss towards  $\mathbf{x}_i$  as Eq. 7 and things are the same with  $\widetilde{\mathbf{x}}_i$ .
$$
l _ {c l s} = - \frac {1}{N} \sum_ {i = 1} ^ {N} H \left(\mathbf {y} _ {i}, \mathbf {p} _ {i}\right) = - \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {l = 1} ^ {L} y (i, l) \log (p (i, l)), \tag {7}
$$
where  $y(i,l)$  is "1" for the correct class of  $\mathbf{x}_i$  and "0" for the rest. For a FER model trained with a re-balanced smooth label of parameter  $\alpha$ , we minimize instead the cross-entropy between the re-balanced targets  $\widetilde{\mathbf{y}}_i$  and the model's outputs  $\mathbf{p}_i$ , where
$$
\widetilde {y} (i, l) = (1 - \alpha) y (i, l) + \alpha B _ {l} / L, \tag {8}
$$
$B_{l}$  is the re-balanced weight of the class  $l$  given in Section 3.1.
3.3 Discussion with existing works
Our most related work is EAC. Though both methods utilize attention map consistency, EAC aims to address the noisy label learning task, while our method focuses on imbalanced learning. Furthermore, the motivations of the two works are different. EAC utilizes the difference of attention maps before and after the transformation to prevent the model to memorize noisy labels, while we propose the re-balanced attention consistency to guide the model to extract useful features related to minor classes from both major and minor samples. We novelly solve the imbalanced FER task through a label distribution learning perspective. We also propose a re-balanced smooth labels module to further regularize the classification loss from the negative effect of the imbalanced training set. We use the extra knowledge of label distribution to facilitate the model to focus more on the prediction of minor classes. The motivation of effectively utilizing all training samples to deal with imbalanced FER and increase the performance of minor classes led to the title of our paper: leave no stone unturned.
Table 1: Comparison with different methods on RAF-DB using pre-trained ResNet-18 as backbone. * denotes that we copied the accuracy from the original paper. We arrange the expression classes according to their sample numbers and observe that they exhibit varying levels of difficulty. For instance, despite having a small number of training samples, all methods achieve relatively high performance on the anger class. On the other hand, the disgust and fear classes prove to be the most challenging, and our method achieves the highest accuracy on these two classes. We report the last epoch accuracy of all methods, we also display the best accuracy of our method for reference.
Method	Conference	Happiness	Neutral	Sadness	Surprise	Disgust	Anger	Fear	Overall	Mean
Baseline	-	95.44	88.53	85.56	83.59	58.75	78.40	59.46	87.42	78.53
CB [5]	CVPR'19	95.11	90.74	84.73	86.93	64.38	73.46	59.46	88.04	79.26
SCN [42]	CVPR'20	94.77	90.29	80.33	86.93	60.00	76.54	45.95	86.73	76.40
BBN [52]	CVPR'20	93.59	91.62	84.94	84.80	61.88	77.78	52.70	87.39	78.19
PT* [12]	TAFFC'21	96.00	92.00	87.00	87.00	55.00	81.00	54.00	88.80	78.86
RUL [49]	NeurIPS'21	95.78	87.06	86.19	89.36	65.00	83.33	64.86	88.66	81.66
EAC [50]	ECCV'22	95.95	92.06	87.03	88.15	66.88	85.80	58.11	89.90	82.00
Ours	NeurIPS'23	96.37	89.56	89.33	87.84	66.89	80.86	66.22	89.77	82.44
Ours (best)	NeurIPS'23	96.03	87.79	89.33	87.23	73.13	84.57	70.27	89.80	84.05
RAF-DB [26] has 30,000 facial expression images, which are annotated with basic or compound expressions by 40 trained annotators. In our experiment, we only utilize the 7 basic expressions and these include 12,271 images for training and 3,068 images for testing. We report the mean accuracy on all expressions to evaluate the imbalanced learning performance of different methods.
FERPlus [1] is extended from FER2013 [6] with finer labels. It is collected by the Google search engine with ten crowd-sourced annotators labeling each image. The most voting category is used as the annotation for a fair comparison with other FER methods [43, 37]. We utilize the same 7 classes with RAF-DB, which results in 24,941 images for training and 3,137 images for testing.
AffectNet [32] is a large-scale FER dataset, which contains eight expressions (7 basic expressions and contempt). There are a total of 286,564 training images and 4,000 test images. We carry out experiments with both 7 basic classes and 8 classes. As the test set of AffectNet is balanced, the mean accuracy on all expressions is the same with the overall accuracy.
To make fair comparisons with other methods, we compare all methods under the same backbone of MS-Celeb-1M [8] pre-trained ResNet-18. We also test the effectiveness of our method under different backbones including MobileNet [10], ResNet-50 [9] and Swin Transformer [28]. The facial expression images are detected and aligned using MTCNN [47]. The image size is  $224 \times 224$ , the learning rate is set to 0.0001. We use Adam [17] optimizer with weight decay of 0.0001 and ExponentialLR learning rate scheduler with a gamma of 0.9. The max training epoch  $T_{max}$  is set to 60. The consistency weight  $\lambda$  is set to 2 and the smooth parameter  $\alpha$  is set to 0.1 according to the ablation study in section 4.7. All experiments are conducted on 4 NVIDIA RTX 2080Ti.
4.3 Experiments on imbalanced FER datasets
We conduct experiments on the RAF-DB dataset to evaluate the performance of different methods in imbalanced learning. We report the classification accuracy of each class and the overall and mean accuracy to assess their effectiveness. The baseline method involves training with the pre-trained ResNet-18 without additional modules. We compare our method with imbalanced learning methods CB, BBN, and state-of-the-art FER methods SCN, PT, RUL, and EAC. To obtain the mean accuracy, we re-implement these methods based on their open-source code. The results in Table 1 demonstrate that our method achieves the highest overall accuracy of  $89.77\%$  and mean accuracy of  $82.44\%$  on the RAF-DB dataset using ResNet-18 as the backbone.