SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Comparison with different methods on AffectNet using pre-trained ResNet-18 as backbone. We carry out experiments with both 7 and 8 classes.
Method	Conference	Happiness	Neutral	Sadness	Anger	Surprise	Fear	Disgust	Contempt	Mean
SCN [42]	CVPR'20	95.20	82.70	44.20	56.30	35.80	38.00	20.90	-	53.30
BBN [52]	CVPR'20	87.00	57.10	66.80	58.30	54.90	71.10	30.10	-	60.76
RUL [49]	NeurIPS'21	90.50	62.40	64.70	69.30	60.80	49.00	34.20	-	61.56
EAC [50]	ECCV'22	91.40	64.50	65.70	66.30	61.60	60.90	45.80	-	65.17
Ours	NeurIPS'23	86.20	59.00	64.20	66.50	57.80	64.50	61.90	-	65.73
SCN [42]	CVPR'20	94.60	74.90	58.20	63.80	40.90	43.20	30.80	2.20	51.08
BBN [52]	CVPR'20	78.40	58.40	60.60	67.70	59.40	55.00	37.00	46.70	57.90
RUL [49]	NeurIPS'21	71.00	63.40	46.60	54.90	53.70	58.60	44.70	47.70	55.08
EAC [50]	ECCV'22	84.00	58.80	65.00	65.90	62.20	60.30	46.10	41.90	60.53
Ours	NeurIPS'23	78.60	54.30	63.80	59.50	57.60	64.10	59.40	60.00	62.16
Based on the accuracy of each class, which is not commonly reported in previous works, we observe varying levels of difficulty among the expression classes. For instance, despite having very few training samples, all methods achieve relatively high performance on the anger class. We speculate that the distinct features associated with anger make it easier for FER models to detect. On the other hand, the disgust and fear classes prove to be the most challenging, with all methods exhibiting low accuracy. However, our method outperforms others on these two classes, achieving accuracies of  $66.89\%$  and  $66.22\%$  respectively. This highlights the effectiveness of our approach in extracting extra knowledge from both major and minor-class samples to address the imbalanced learning problem.
We further carry out experiments on AffectNet, which is one of the largest and most imbalanced datasets available for FER. From the results in Table 2, we could draw the conclusion that our method achieves the best mean accuracy on the test set under both 7 or 8 classes. Besides, we notice that our method improves existing methods on the minor classes of fear (Fea), disgust (Dis), contempt (Con) by remarkable margins, which illustrates that our method is more suitable for the imbalanced learning of FER task. Our method even achieves  $60.00\%$  accuracy on the contempt class under 8 classes. Furthermore, our method clearly decreases the test accuracy gap between happy  $(78.60\%)$  and contempt  $(60.00\%)$  compared with other methods under 8 classes, which means our method is fairer and achieves a more balanced test accuracy.
4.4 Experiments with different imbalance factors
Following existing imbalanced learning methods [5, 33, 2] in the image classification field, we also construct imbalanced FER datasets with different imbalance factors. The definition of the imbalance factor of a dataset is following [5] as the number of training samples in the largest class divided by the smallest. Given the imbalance factor, the imbalanced FER datasets are created by reducing the number of training samples per class according to an exponential function  $n = n_l \mu^l$ , where  $l$  is the class index,  $n_l$  is the original number of training images and  $\mu \in (0,1)$ . Due to space limitation, the sample number of each class is summarized in the supplementary material. We evaluate our method on RAF-DB and FERPlus datasets with imbalance factors ranging from 50 to 150, as shown in Table 3 and Table 4. The results demonstrate the superior performance of our method across different imbalance factors. Specifically, compared to the state-of-the-art FER method EAC, our method consistently improves upon it with  $4.09\%$ ,  $3.26\%$ ,  $1.67\%$  and  $2.13\%$ ,  $2.62\%$ ,  $4.15\%$  regrading mean accuracy of all classes on RAF-DB and FERPlus respectively.
4.5 Different backbones
We evaluate the generalization ability of our method by combining it with four different backbones: MobileNet, ResNet-18, ResNet-50, and Tiny Swin Transformer, on RAF-DB. The results consistently demonstrate the improvement in imbalanced learning performance across different backbones. We also observe that different backbones have a notable effect on the performance of imbalanced learning. Notably, when combined with Tiny Swin Transformer, our method achieves state-of-the-art performance with accuracy of  $71.62\%$  and  $85.00\%$  on the most difficult expression classes of fear and disgust, respectively, as well as an overall accuracy of  $92.31\%$  and a mean accuracy of  $87.71\%$ .
Table 3: Comparison with other methods on RAF-DB with different imbalance factors. Disgust and fear are the most difficult classes. Our method achieves the highest accuracy on the overall, mean accuracy and the accuracy on the most difficult classes under different imbalance factors.
Method	Imbalance	Happiness	Neutral	Sadness	Surprise	Disgust	Anger	Fear	Overall	Mean
Baseline	50	95.95	87.35	79.08	84.19	39.38	64.20	2.70	83.28	64.69
BBN	50	93.59	91.91	81.80	82.98	41.25	71.60	37.84	85.01	71.57
EAC	50	95.53	93.82	82.01	89.06	50.00	70.99	29.73	87.09	73.02
Ours	50	96.37	90.00	85.36	85.41	53.75	73.46	55.41	87.65	77.11
Baseline	100	97.72	87.94	73.85	81.76	10.63	54.94	0.00	80.96	58.12
BBN	100	94.94	93.38	71.34	82.37	36.88	65.43	31.08	83.44	67.92
EAC	100	95.27	92.06	83.68	89.97	36.88	62.35	28.38	85.79	69.80
Ours	100	96.37	91.18	82.85	86.63	44.38	65.43	44.59	86.47	73.06
Baseline	150	95.86	90.29	75.73	77.51	9.38	46.91	0.00	80.11	56.53
BBN	150	94.85	93.53	74.69	81.46	30.00	55.56	28.38	82.92	65.49
EAC	150	96.20	91.62	77.82	79.64	36.25	59.88	39.19	84.13	68.66
Ours	150	96.62	91.91	79.29	83.89	36.25	61.11	43.24	85.20	70.33
Table 4: Comparison with other methods on FERPlus with different imbalance factors. Our method achieves the highest accuracy on the overall, mean accuracy and the accuracy on the most difficult classes (fear and disgust) under different imbalance factors.
Method	Imbalance	Neutral	Happiness	Surprise	Sadness	Anger	Fear	Disgust	Overall	Mean
Baseline	50	86.42	93.17	89.14	76.56	83.88	46.99	22.22	85.85	71.20
BBN	50	84.31	91.38	93.18	77.60	84.98	54.22	33.33	85.59	74.14
EAC	50	90.09	95.63	90.15	76.30	84.62	49.40	33.33	88.11	74.22
Ours	50	91.19	94.06	91.67	79.95	82.05	56.63	38.89	88.68	76.35
Baseline	100	90.32	93.56	87.63	66.80	78.21	46.99	22.22	85.43	69.39
BBN	100	88.62	91.71	93.18	74.74	81.32	51.81	38.89	86.48	74.32
EAC	100	90.73	95.30	90.66	74.48	81.32	53.45	27.78	87.87	73.39
Ours	100	91.56	94.85	92.42	77.34	82.78	54.22	38.89	88.81	76.01
Baseline	150	91.28	93.84	89.14	63.02	78.75	44.58	11.11	85.49	67.39
BBN	150	90.09	92.61	93.43	67.19	79.12	46.99	33.33	86.01	71.82
EAC	150	93.12	94.74	89.65	71.35	78.75	39.76	22.22	87.41	69.94
Ours	150	93.94	94.51	90.40	71.88	79.12	55.42	33.33	88.30	74.09
Table 5: The performance of our method under different backbones. We find that backbones have a significant influence on the accuracy. Our method consistently improves the performance under different backbones and achieves the state-of-the-art overall accuracy of  $92.31\%$  and mean accuracy of  $87.71\%$  with Tiny Swin Transformer (Swin-T).
Backbone	Happiness	Neutral	Sadness	Surprise	Disgust	Anger	Fear	Overall	Mean
MobileNet	93.84	83.09	77.62	88.75	45.00	75.31	56.76	83.96	74.34
MobileNet+Ours	94.26	88.82	81.38	83.28	59.38	74.07	59.46	86.15	77.24
ResNet-18	95.44	88.53	85.56	83.59	58.75	78.40	59.46	87.42	78.53
ResNet-18+Ours	96.37	89.56	89.33	87.84	66.89	80.86	66.22	89.77	82.44
ResNet-50	94.77	87.79	87.03	85.71	68.75	84.57	60.81	88.33	81.35
ResNet-50+Ours	95.95	87.65	89.75	88.75	80.63	85.19	66.22	90.29	84.88
Swin-T	97.05	91.62	87.87	90.27	78.75	86.42	60.81	91.30	84.68
Swin-T+Ours	96.96	92.06	88.28	92.40	85.00	87.65	71.62	92.31	87.71
Table 6: Ablation study of our proposed two modules re-balanced attention consistency (RAC) and re-balanced smooth labels (RSL). Both of the two modules can improve the performance based on the baseline, while they can cooperate to achieve the best performance.
RAC	RLS	Happiness	Neutral	Sadness	Surprise	Disgust	Anger	Fear	Overall	Mean
95.44	88.53	85.56	83.59	58.75	78.40	59.46	87.42	78.53
✓	96.29	89.26	87.87	88.75	65.63	76.54	59.46	89.08	80.54
✓	94.60	90.44	85.15	82.07	57.50	80.86	64.86	87.48	79.35
✓	✓	96.37	89.56	89.33	87.84	66.89	80.86	66.22	89.77	82.44
Figure 3: The hyperparameter study of the consistency weight  $\lambda$  and the smooth parameter  $\alpha$ .
4.6 Ablation study
We conduct an ablation study on RAF-DB to evaluate the contribution of each proposed module. The results in Table 6 demonstrate the effectiveness of both the re-balanced attention consistency (RAC) and re-balanced smooth labels (RSL) in improving the baseline method's performance. Interestingly, we observe that utilizing only the RAC module achieves superior performance compared to using only the RSL module. This could be attributed to the additional information provided by the re-balanced attention map consistency. Moreover, combining both RAC and RSL modules results in even better performance, indicating their effective collaboration in addressing the imbalanced FER task.
4.7 Hyperparameter study
We carry out experiments on RAF-DB to study the effect of different hyperparameters to our method. We plot the results in Figure 3.
Consistency weight  $\lambda$  The results demonstrate that our method exhibits low sensitivity to the consistency weight  $\lambda$ , with both the mean accuracy and overall accuracy varying within a small range of  $1\%$  as  $\lambda$  changes from 0.05 to 4. Notably, the optimal value for  $\lambda$  in our method is found to be 2, indicating that larger values may excessively prioritize consistency loss over classification loss, potentially leading to a decrease in classification accuracy. Conversely, smaller values of  $\lambda$  may fail to effectively regulate the model in extracting additional knowledge related to minor classes from all samples, thus negatively impacting accuracy.
Smooth parameter  $\alpha$  The smooth parameter, ranging from 0 to 1, determines the strength of the latent truth (set as  $1 - \alpha$ ). We evaluate different values of  $\alpha$  from 0.05 to 0.4 and find that the optimal value is  $\alpha = 0.1$ . A larger  $\alpha$  negatively impacts performance as the excessive smooth effect hampers the model's ability to learn useful information. On the other hand, a smaller  $\alpha$  fails to effectively utilize the prior knowledge of label distribution to prioritize minor classes.
4.8 Visualization results
We provide visualization results to illustrate the effectiveness of our proposed method. The learned attention maps, as shown in Figure 4, reveal three key observations. First, our method consistently learns attention maps that are more consistent across different transformations, enabling the FER model to capture transformation invariant information for various expression classes. Second, our method effectively extracts additional knowledge related to minor classes (e.g., disgust and fear) from samples of major classes (e.g., sadness and surprise), as there are shared features between major
Figure 4: The attention maps corresponding to different classes learned by different methods. We utilize the attention map of a certain class to mine transformation invariant information of that class. The attention maps of the labels are marked by red. Attention maps learned by our method are more consistent before and after the flip transformation across all different classes. Furthermore, shown in the last two columns, our method can mine extra knowledge related to the minor classes like fear (the open mouth feature) from the samples of major classes of surprise.
Table 7: Other transformation methods for re-balanced attention consistency.
Method	Happiness	Neutral	Sadness	Surprise	Disgust	Anger	Fear	Overall	Mean
Intensity	95.02	86.47	82.01	82.98	63.13	72.84	56.76	86.05	77.03
Scaling	95.78	91.91	84.31	85.41	75.00	82.10	60.81	89.37	82.19
Ours	96.37	89.56	89.33	87.84	66.89	80.86	66.22	89.77	82.44
and minor classes in FER. For instance, the first column in the right part of Figure 4 demonstrates how our method captures the mouth corner feature associated with disgust from a sample labeled as sadness. Furthermore, the last two columns in the figure show that our method identifies the open mouth feature shared by fear and surprise, allowing us to extract fear-related features from surprise samples. More results in the Supp. material. Third, our method produces non-overlapping attention maps for different classes, in contrast to the baseline method. For example, for the sample labeled as sadness, the attention maps for happiness and sadness learned by our method do not overlap, while they overlap in the baseline method. This indicates that the attention maps learned by our method are more meaningful and distinct.
4.9 Other transformations
Flipping of the images is shown to introduce the notion of re-balanced attention consistency. In this section, we investigate whether some other transformations (e.g., scaling, intensity attenuation, or gain) work well under the imbalanced FER task. The results on RAF-DB in Table 7 illustrate that intensity transformation performs poorly, while scaling performs well, which almost surpasses our method. The reason lies in that attention map consistency regularizes the model to focus on the same regions before and after the transformation, which incorporates spatial information as the attention map in our method has height and width dimensions. Thus, the transformation should be spatial-related transformation to maximize the function of the method.
In this paper, we investigate the imbalanced learning problem in facial expression recognition (FER). We observe that existing imbalanced learning methods tend to improve performance on minor classes at the expense of major classes. Motivated by the label distribution learning characteristic of FER, we propose a novel approach to extract additional knowledge about minor classes from both major and minor class samples. This allows us to enhance the performance on minor classes while maintaining high performance on major classes. Our method consists of two modules: re-balanced attention consistency and re-balanced smooth labels, which regulate attention maps and classification loss, respectively. Instead of relying on traditional over-sampling or under-sampling techniques, our method effectively utilizes all training samples and incorporates prior knowledge of the imbalanced data distribution to prioritize minor classes. Through extensive experiments on various imbalanced FER datasets and with different backbones, we validate the effectiveness of our proposed method.
We sincerely thank all the reviewers who have given us lots of valuable suggestions for the improvement of our paper. This work was supported in part by the BUPT Excellent Ph.D. Students Foundation No.CX2023111 and in part by scholarships from China Scholarship Council (CSC) under Grant CSC No.202206470048.