Title: Leave No Stone Unturned: Mine Extra Knowledge for Imbalanced Facial Expression Recognition
Venue: NeurIPS 2023
Accepted: yes
Keywords: Facial expression recognition

imbalanced learning
Abstract: Facial expression data is characterized by a significant imbalance, with most collected data showing happy or neutral expressions and fewer instances of fear or disgust. This imbalance poses challenges to facial expression recognition (FER) models, hindering their ability to fully understand various human emotional states. Existing FER methods typically report overall accuracy on highly imbalanced test sets but exhibit low performance in terms of the mean accuracy across all expression classes. In this paper, our aim is to address the imbalanced FER problem. Existing methods primarily focus on learning knowledge of minor classes solely from minor-class samples. However, we propose a novel approach to extract extra knowledge related to the minor classes from both major and minor class samples. Our motivation stems from the belief that FER resembles a distribution learning task, wherein a sample may contain information about multiple classes. For instance, a sample from the major class surprise might also contain useful features of the minor class fear. Inspired by that, we propose a novel method that leverages re-balanced attention maps to regularize the model, enabling it to extract transformation invariant information about the minor classes from all training samples. Additionally, we introduce re-balanced smooth labels to regulate the cross-entropy loss, guiding the model to pay more attention to the minor classes by utilizing the extra information regarding the label distribution of the imbalanced training data. Extensive experiments on different datasets and backbones show that the two proposed modules work together to regularize the model and achieve state-of-the-art performance under the imbalanced FER task. Code is available at https://github.com/zyh-uaiaaaa.

Leave No Stone Unturned: Mine Extra Knowledge for Imbalanced Facial Expression Recognition

Abstract

Facial expression data is characterized by a significant imbalance, with most collected data showing happy or neutral expressions and fewer instances of fear or disgust. This imbalance poses challenges to facial expression recognition (FER) models, hindering their ability to fully understand various human emotional states. Existing FER methods typically report overall accuracy on highly imbalanced test sets but exhibit low performance in terms of the mean accuracy across all expression classes. In this paper, our aim is to address the imbalanced FER problem. Existing methods primarily focus on learning knowledge of minor classes solely from minor-class samples. However, we propose a novel approach to extract extra knowledge related to the minor classes from both major and minor class samples. Our motivation stems from the belief that FER resembles a distribution learning task, wherein a sample may contain information about multiple classes. For instance, a sample from the major class surprise might also contain useful features of the minor class fear. Inspired by that, we propose a novel method that leverages re-balanced attention maps to regularize the model, enabling it to extract transformation invariant information about the minor classes from all training samples. Additionally, we introduce re-balanced smooth labels to regulate the cross-entropy loss, guiding the model to pay more attention to the minor classes by utilizing the extra information regarding the label distribution of the imbalanced training data. Extensive experiments on different datasets and backbones show that the two proposed modules work together to regularize the model and achieve state-of-the-art performance under the imbalanced FER task. Code is available at https://github.com/zyh-uaiaaaa .

1 Introduction

Facial expression recognition (FER) plays a crucial role in enabling machines to understand human emotional states, thus facilitating the realization of machine intelligence [18, 11, 25]. Recent research efforts [42, 35, 49] have employed in-the-wild datasets to train FER models, resulting in impressive performance in terms of overall classification accuracy. However, these FER datasets suffer from a significant imbalance, with a higher abundance of happy and neutral expression faces, which are easily obtainable from the internet and daily life, compared to faces displaying negative expressions such as fear or disgust [6, 32]. This imbalance can have adverse effects on human-computer interaction when FER models misinterpret infrequently occurring negative emotions as frequently occurring positive emotions.

In this paper, our objective is to investigate imbalanced learning in the context of FER. We observe that existing FER methods [34, 43, 42, 29] yield relatively low performance in terms of the mean accuracy across all classes, mainly due to the highly imbalanced nature of the training data. While imbalanced (long-tailed) learning methods in the image classification domain have shown some

improvements [52, 5], they provide limited enhancements in the mean accuracy within the FER field. This can be attributed to several factors. Firstly, FER datasets typically contain a small number of classes, resulting in marginal improvements in the mean accuracy of all classes. Secondly, the increase in accuracy for minor classes often comes at the expense of decreased accuracy for major classes. This imbalance disproportionately affects the overall accuracy, particularly when evaluating on imbalanced test sets.

To address the aforementioned problem, our objective is to design a method that can maintain high performance on major classes while improving the performance on minor classes to the greatest extent possible. We refrain from modifying the dataset structure through under-sampling or over-sampling, as these approaches may enhance the performance of minor classes at the expense of degrading the performance on major classes. Instead, we aim to leverage each sample fully to extract additional knowledge relevant to minor classes. Our motivation is that FER resembles a label distribution learning task, implying that samples from major classes may contain information pertaining to minor classes as well. For instance, the major class "surprise" shares certain similarities with the minor class "fear".

Inspired by that, we propose a novel method that mines extra knowledge regarding minor classes from samples belonging to both major and minor classes. This extra information enhances the recognition performance on minor classes without significantly compromising the performance on major classes. Specifically, we employ the concept of attention map consistency [7], which is utilized in the EAC method [50] to prevent FER models from memorizing noisy labels. We observe that EAC employs attention maps from all expression classes, rather than just the labeled class, to regularize a specific sample, resembling the idea of label distribution learning. Building upon this insight, we introduce a re-balanced attention consistency module to address the imbalanced FER task for the first time. In our approach, attention maps for all expression classes can be extracted from a given FER sample. We introduce re-balanced weights, following the design of [5], which are set inversely proportional to the effective number of samples belonging to each class. These weights facilitate the model in extracting additional knowledge related to minor classes from all training samples. Furthermore, as the classification loss may still be affected by the imbalanced data, we propose a re-balanced smooth labels module that utilizes the acquired re-balanced weights and the extra knowledge of the imbalanced label distribution. This module guides the model to focus more on minor classes during the decision-making process.

To demonstrate the effectiveness of our proposed method, we conducted extensive experiments on different FER datasets, including datasets with varying imbalance factors [5]. Ablation studies confirmed that each proposed module contributes to the enhancement of mean accuracy, and the two modules synergistically achieve state-of-the-art performance. Moreover, we evaluated the generalization ability of our method by combining it with different backbones including transformers. The experimental results showcased that our method is lightweight, easy to implement, and seamlessly compatible with various backbones, thus significantly improving their performance.

The main contributions of our work are listed as follows:

We highlight the imbalanced learning problem in FER and find existing methods in the large-scale image classification field might fall short in the FER field, we benchmark existing FER methods utilizing both overall accuracy and mean accuracy on all classes.

We propose a novel method consisting of two modules: re-balanced attention consistency (RAC) and re-balanced smooth labels (RSL). RAC mines extra knowledge pertaining to minor classes from both major and minor samples, thereby enhancing performance on minor classes without compromising performance on major classes. RSL leverages the imbalanced label distribution to further regularize the classification loss and prioritize the decision-making process for minor classes.

Our proposed method is easy to implement, lightweight, and seamlessly compatible with different backbone architectures including transformers. Through extensive experiments, we validate that our proposed method achieves state-of-the-art performance in terms of both overall accuracy and mean accuracy of all classes across different FER datasets and backbone architectures.

2 Related work

Facial expression recognition methods in recent years primarily focus on extracting effective features from in-the-wild datasets using deep learning models [22, 46, 13, 16, 20, 24, 21, 23]. In-the-wild datasets [26, 6] pose greater challenges and are more prone to noise due to label inaccuracies, difficult poses, occlusions, and low-quality images compared to laboratory collected datasets [30, 14, 40]. These factors adversely affect the recognition performance of facial expression recognition (FER) models. Several works have addressed the real-world FER task. SCN [42] introduces a learnable temperature and a relabel module to weight FER samples and suppress noise. DMUE [37] learns sub-modules for each class to handle noisy labels. RUL [49] treats expression uncertainty as a relative concept and learns uncertainty values for images through comparisons, aiding overall feature learning. TransFER [45] pioneers the use of vision transformers for in-the-wild FER. EAC [50] employs erasing attention map consistency to prevent FER models from memorizing noisy labels in real-world FER datasets.

While these approaches achieve impressive overall accuracy on test sets, they tend to exhibit lower performance in terms of mean accuracy of all classes, particularly for minor classes. In fact, previous works primarily focus on mitigating noise in in-the-wild datasets, often overlooking the imbalanced nature of FER datasets. In this work, our objective is to address imbalanced learning in in-the-wild FER datasets and provide an orthogonal supplement for previous works.

Imbalanced learning in real-life datasets, where certain classes have a majority of samples while others have very few, has been addressed through three perspectives: data pre-processing, loss re-weighting, and model ensemble. This paper primarily focuses on the first two categories, as model ensemble methods [44, 48] are computationally intensive. Data pre-processing methods commonly involve data re-sampling [31, 38]. However, studies [15, 52] have shown that data re-sampling can negatively impact representation learning. Over-sampling may introduce duplicated samples, leading to the increased risk of overfitting, while under-sampling may discard valuable examples. Another technique is data augmentation [4, 53], which applies predefined transformations to augment the dataset, particularly for minority classes. However, finding effective augmentation methods for facial expression recognition (FER) data, which contain specific local features related to expressions, can be challenging. Loss re-weighting methods assign different weights to classes or instances during training, known as loss re-weighting [27, 39, 41, 5]. The aim is to propagate appropriate gradient values for all classes during training.

3 Method

In this section, we present a novel method for imbalanced facial expression recognition. Our approach focuses on extracting extra information of minor classes from both minor and major class samples, rather than solely relying on minor class samples. We introduce two modules, namely re-balanced attention consistency (RAC) and re-balanced smooth labels (RSL), to guide the model to extract balanced information from the entire training set.

3.1 Re-balanced attention consistency

We first introduce attention map consistency [7], which regularizes the attention maps to follow the same spatial transform if the input images are spatially transformed, which can help the model to learn transformation invariant features. EAC designs an imbalanced framework to utilize attention map consistency to prevent the models from memorizing the noisy labels. We find that instead of only extracting attention maps of the latent truth like utilizing GradCAM [36], EAC uses attention maps of all classes to regularize the FER model. We speculate that the reason lies in that utilizing attention maps of all classes is similar to label distribution learning [3, 51, 19], which mines information of several classes from one training sample. Inspired by that, we adapt attention map consistency to solve the imbalanced learning problem of FER for the first time. To be more specific, given a sample, the attention maps of all expression classes should follow the same spatial transform if the given sample is spatially transformed. Thus, the model could mine useful minor-class information from all samples instead of only the samples from the minor classes. Furthermore, we introduce the re-balanced attention consistency to regularize the model to focus more on the minor classes to achieve balanced learning. We set different weights to the attention maps of different classes

Original

x

F

Mean

M

FC

class 1

classL

GAP

Re-balanced

Re-balanced

Consistency

loss

weight

loss

Flipped

M

x

Mean

Flip

and guide the model to learn more invariant information related to minor classes. Inspired by [5], which designs a re-weighting scheme that uses the effective number of samples of each class to re-balance the classification loss, we propose to use the effective number of samples to inversely the weigh attention maps of different classes. In the following, we formulate the re-balanced attention consistency module in detail.

Given images x , we first flip them to get their flipped counterparts ˜ x . The features of x and ˜ x are extracted from the last convolutional layer, denoted as F ∈ R N × C × H × W and ˜ F ∈ R N × C × H × W , where N , C , H and W respectively represent the number of images, channels, height and width. We input F and ˜ F to the global average pooling (GAP) layer to get features f ∈ R N × C × 1 × 1 and ˜ f ∈ R N × C × 1 × 1 , and then resize them to N × C . The classification loss is computed according to

where W y i is the y i -th weight of the fully connected (FC) layer and y i is the label of x i , L is the total number of expression classes. CAM [53] is utilized to compute attention maps A ∈ R N × L × H × W and ˜ A ∈ R N × L × H × W for x and ˜ x following

where i , l , c , h , w represent the sample, expression class, channel, height and width number. We then re-balance the attention maps A and ˜ A through weight B ∈ R L to get the re-balanced attention maps M and ˜ M following

The balance weight B ∈ R L is enlightened by the re-balanced weight based on the effective number in [5], which is computed following

where n l is the number of training samples in class l , β ∈ [0 , 1) is the hyperparameter controlling the re-balanced weight, a larger value of β emphasizes more of the minor samples, following [5], we set the β as 0.9999 across all our experiments. As the attention map before and after the flip transformation should be consistent with each other, we compute the consistency loss between M and Flip ( ˜ M ) following

The training loss is computed as

where λ is the weight of the attention consistency loss, which determines the relative importance of the consistency loss compared to the classification loss in the overall training objective.

0.6

0.6

sample

number

surprise

fear

disgusthappiness sadness

anger

surprise

fear

disgusthappinesssadness

anger

reverse

0.4/6

0.4/6

0.4/6

0.4/6

0.4/6

0.4/6

smoothlabel

re-balancedsmoothlabel

3.2 Re-balanced smooth labels

In Section 3.1, we introduced the re-balanced attention consistency module, which enables the model to extract balanced and transformation invariant knowledge of minor classes from all training samples, resulting in improved classification accuracy for those classes. However, the imbalanced training set can still negatively impact the classification loss, denoted as l cls . To address this, we present the re-balanced smooth labels module in this section. This module aims to regulate the classification loss and promote balanced learning.

We denote the prediction of the FER model towards x i as p i , where p i is the likelihood the model assigns to the i -th given sample x i , the one-hot label of x i is denoted as y i . For simplicity, we re-write the classification loss towards x i as Eq. 7 and things are the same with ˜ x i .

where y ( i, l ) is "1" for the correct class of x i and "0" for the rest. For a FER model trained with a re-balanced smooth label of parameter α , we minimize instead the cross-entropy between the re-balanced targets ˜ y i and the model's outputs p i , where

B l is the re-balanced weight of the class l given in Section 3.1.

3.3 Discussion with existing works

Our most related work is EAC. Though both methods utilize attention map consistency, EAC aims to address the noisy label learning task, while our method focuses on imbalanced learning. Furthermore, the motivations of the two works are different. EAC utilizes the difference of attention maps before and after the transformation to prevent the model to memorize noisy labels, while we propose the re-balanced attention consistency to guide the model to extract useful features related to minor classes from both major and minor samples. We novelly solve the imbalanced FER task through a label distribution learning perspective. We also propose a re-balanced smooth labels module to further regularize the classification loss from the negative effect of the imbalanced training set. We use the extra knowledge of label distribution to facilitate the model to focus more on the prediction of minor classes. The motivation of effectively utilizing all training samples to deal with imbalanced FER and increase the performance of minor classes led to the title of our paper: leave no stone unturned.

4 Experiments

4.1 Datasets

RAF-DB [26] has 30,000 facial expression images, which are annotated with basic or compound expressions by 40 trained annotators. In our experiment, we only utilize the 7 basic expressions and these include 12,271 images for training and 3,068 images for testing. We report the mean accuracy on all expressions to evaluate the imbalanced learning performance of different methods.

FERPlus [1] is extended from FER2013 [6] with finer labels. It is collected by the Google search engine with ten crowd-sourced annotators labeling each image. The most voting category is used as the annotation for a fair comparison with other FER methods [43, 37]. We utilize the same 7 classes with RAF-DB, which results in 24,941 images for training and 3,137 images for testing.

AffectNet [32] is a large-scale FER dataset, which contains eight expressions (7 basic expressions and contempt). There are a total of 286,564 training images and 4,000 test images. We carry out experiments with both 7 basic classes and 8 classes. As the test set of AffectNet is balanced, the mean accuracy on all expressions is the same with the overall accuracy.

4.2 Implementation details

To make fair comparisons with other methods, we compare all methods under the same backbone of MS-Celeb-1M [8] pre-trained ResNet-18. We also test the effectiveness of our method under different backbones including MobileNet [10], ResNet-50 [9] and Swin Transformer [28]. The facial expression images are detected and aligned using MTCNN [47]. The image size is 224 × 224 , the learning rate is set to 0 . 0001 . We use Adam [17] optimizer with weight decay of 0 . 0001 and ExponentialLR learning rate scheduler with a gamma of 0.9. The max training epoch T max is set to 60. The consistency weight λ is set to 2 and the smooth parameter α is set to 0.1 according to the ablation study in section 4.7. All experiments are conducted on 4 NVIDIA RTX 2080Ti.

4.3 Experiments on imbalanced FER datasets

We conduct experiments on the RAF-DB dataset to evaluate the performance of different methods in imbalanced learning. We report the classification accuracy of each class and the overall and mean accuracy to assess their effectiveness. The baseline method involves training with the pre-trained ResNet-18 without additional modules. We compare our method with imbalanced learning methods CB, BBN, and state-of-the-art FER methods SCN, PT, RUL, and EAC. To obtain the mean accuracy, we re-implement these methods based on their open-source code. The results in Table 1 demonstrate that our method achieves the highest overall accuracy of 89 . 77% and mean accuracy of 82 . 44% on the RAF-DB dataset using ResNet-18 as the backbone.

Based on the accuracy of each class, which is not commonly reported in previous works, we observe varying levels of difficulty among the expression classes. For instance, despite having very few training samples, all methods achieve relatively high performance on the anger class. We speculate that the distinct features associated with anger make it easier for FER models to detect. On the other hand, the disgust and fear classes prove to be the most challenging, with all methods exhibiting low accuracy. However, our method outperforms others on these two classes, achieving accuracies of 66 . 89% and 66 . 22% respectively. This highlights the effectiveness of our approach in extracting extra knowledge from both major and minor-class samples to address the imbalanced learning problem.

We further carry out experiments on AffectNet, which is one of the largest and most imbalanced datasets available for FER. From the results in Table 2, we could draw the conclusion that our method achieves the best mean accuracy on the test set under both 7 or 8 classes. Besides, we notice that our method improves existing methods on the minor classes of fear (Fea), disgust (Dis), contempt (Con) by remarkable margins, which illustrates that our method is more suitable for the imbalanced learning of FER task. Our method even achieves 60.00% accuracy on the contempt class under 8 classes. Furthermore, our method clearly decreases the test accuracy gap between happy (78.60%) and contempt (60.00%) compared with other methods under 8 classes, which means our method is fairer and achieves a more balanced test accuracy.

4.4 Experiments with different imbalance factors

Following existing imbalanced learning methods [5, 33, 2] in the image classification field, we also construct imbalanced FER datasets with different imbalance factors. The definition of the imbalance factor of a dataset is following [5] as the number of training samples in the largest class divided by the smallest. Given the imbalance factor, the imbalanced FER datasets are created by reducing the number of training samples per class according to an exponential function n = n l µ l , where l is the class index, n l is the original number of training images and µ ∈ (0 , 1) . Due to space limitation, the sample number of each class is summarized in the supplementary material. We evaluate our method on RAF-DB and FERPlus datasets with imbalance factors ranging from 50 to 150, as shown in Table 3 and Table 4. The results demonstrate the superior performance of our method across different imbalance factors. Specifically, compared to the state-of-the-art FER method EAC, our method consistently improves upon it with 4 . 09% , 3 . 26% , 1 . 67% and 2 . 13% , 2 . 62% , 4 . 15% regrading mean accuracy of all classes on RAF-DB and FERPlus respectively.

4.5 Different backbones

We evaluate the generalization ability of our method by combining it with four different backbones: MobileNet, ResNet-18, ResNet-50, and Tiny Swin Transformer, on RAF-DB. The results consistently demonstrate the improvement in imbalanced learning performance across different backbones. We also observe that different backbones have a notable effect on the performance of imbalanced learning. Notably, when combined with Tiny Swin Transformer, our method achieves state-of-the-art performance with accuracy of 71 . 62% and 85 . 00% on the most difficult expression classes of fear and disgust, respectively, as well as an overall accuracy of 92 . 31% and a mean accuracy of 87 . 71% .

82.5

82.4

89.7

89.5

mean accuracy

82.2

89.6

overall accuracy

mean accuracy

82.0

88.5

accuracy

82.0

89.5

81.5

87.5

overalla

81.8

89.4

81.0

86.5

81.6

mean

89.3

mean

overall

overall

80.5

85.5

0.05

1.00

2.00

3.00

4.00

0.05

0.10

0.20

0.30

0.40

consistency weight 入

smooth parameter α

4.6 Ablation study

We conduct an ablation study on RAF-DB to evaluate the contribution of each proposed module. The results in Table 6 demonstrate the effectiveness of both the re-balanced attention consistency (RAC) and re-balanced smooth labels (RSL) in improving the baseline method's performance. Interestingly, we observe that utilizing only the RAC module achieves superior performance compared to using only the RSL module. This could be attributed to the additional information provided by the re-balanced attention map consistency. Moreover, combining both RAC and RSL modules results in even better performance, indicating their effective collaboration in addressing the imbalanced FER task.

4.7 Hyperparameter study

We carry out experiments on RAF-DB to study the effect of different hyperparameters to our method. We plot the results in Figure 3.

Consistency weight λ The results demonstrate that our method exhibits low sensitivity to the consistency weight λ , with both the mean accuracy and overall accuracy varying within a small range of 1% as λ changes from 0 . 05 to 4 . Notably, the optimal value for λ in our method is found to be 2 , indicating that larger values may excessively prioritize consistency loss over classification loss, potentially leading to a decrease in classification accuracy. Conversely, smaller values of λ may fail to effectively regulate the model in extracting additional knowledge related to minor classes from all samples, thus negatively impacting accuracy.

Smooth parameter α The smooth parameter, ranging from 0 to 1 , determines the strength of the latent truth (set as 1 -α ). We evaluate different values of α from 0 . 05 to 0 . 4 and find that the optimal value is α = 0 . 1 . A larger α negatively impacts performance as the excessive smooth effect hampers the model's ability to learn useful information. On the other hand, a smaller α fails to effectively utilize the prior knowledge of label distribution to prioritize minor classes.

4.8 Visualization results

We provide visualization results to illustrate the effectiveness of our proposed method. The learned attention maps, as shown in Figure 4, reveal three key observations. First, our method consistently learns attention maps that are more consistent across different transformations, enabling the FER model to capture transformation invariant information for various expression classes. Second, our method effectively extracts additional knowledge related to minor classes (e.g., disgust and fear) from samples of major classes (e.g., sadness and surprise), as there are shared features between major

original

disgust

happiness

sadness

fear

surprise

disgust

happiness

sadness

fear

surprise

baseline

ours

and minor classes in FER. For instance, the first column in the right part of Figure 4 demonstrates how our method captures the mouth corner feature associated with disgust from a sample labeled as sadness. Furthermore, the last two columns in the figure show that our method identifies the open mouth feature shared by fear and surprise, allowing us to extract fear-related features from surprise samples. More results in the Supp. material. Third, our method produces non-overlapping attention maps for different classes, in contrast to the baseline method. For example, for the sample labeled as sadness, the attention maps for happiness and sadness learned by our method do not overlap, while they overlap in the baseline method. This indicates that the attention maps learned by our method are more meaningful and distinct.

4.9 Other transformations

Flipping of the images is shown to introduce the notion of re-balanced attention consistency. In this section, we investigate whether some other transformations (e.g., scaling, intensity attenuation, or gain) work well under the imbalanced FER task. The results on RAF-DB in Table 7 illustrate that intensity transformation performs poorly, while scaling performs well, which almost surpasses our method. The reason lies in that attention map consistency regularizes the model to focus on the same regions before and after the transformation, which incorporates spatial information as the attention map in our method has height and width dimensions. Thus, the transformation should be spatial-related transformation to maximize the function of the method.

5 Conclusion

In this paper, we investigate the imbalanced learning problem in facial expression recognition (FER). We observe that existing imbalanced learning methods tend to improve performance on minor classes at the expense of major classes. Motivated by the label distribution learning characteristic of FER, we propose a novel approach to extract additional knowledge about minor classes from both major and minor class samples. This allows us to enhance the performance on minor classes while maintaining high performance on major classes. Our method consists of two modules: re-balanced attention consistency and re-balanced smooth labels, which regulate attention maps and classification loss, respectively. Instead of relying on traditional over-sampling or under-sampling techniques, our method effectively utilizes all training samples and incorporates prior knowledge of the imbalanced data distribution to prioritize minor classes. Through extensive experiments on various imbalanced FER datasets and with different backbones, we validate the effectiveness of our proposed method.

Acknowledgments and Disclosure of Funding

We sincerely thank all the reviewers who have given us lots of valuable suggestions for the improvement of our paper. This work was supported in part by the BUPT Excellent Ph.D. Students Foundation No.CX2023111 and in part by scholarships from China Scholarship Council (CSC) under Grant CSC No.202206470048.

References

E. Barsoum, C. Zhang, C. C. Ferrer, and Z. Zhang. Training deep networks for facial expression recognition with crowd-sourced label distribution. In Proceedings of the 18th ACM international conference on multimodal interaction , pages 279-283, 2016.

K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma. Learning imbalanced datasets with label-distributionaware margin loss. Advances in neural information processing systems , 32, 2019.

S. Chen, J. Wang, Y. Chen, Z. Shi, X. Geng, and Y. Rui. Label distribution learning on auxiliary label space graphs for facial expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

H.-P. Chou, S.-C. Chang, J.-Y. Pan, W. Wei, and D.-C. Juan. Remix: rebalanced mixup. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16 , pages 95-110. Springer, 2020.

Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9268-9277, 2019.

I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee, et al. Challenges in representation learning: A report on three machine learning contests. In Neural Information Processing: 20th International Conference, ICONIP 2013, Daegu, Korea, November 3-7, 2013. Proceedings, Part III 20 , pages 117-124. Springer, 2013.

H. Guo, K. Zheng, X. Fan, H. Yu, and S. Wang. Visual attention consistency under image transforms for multi-label image classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 729-739, 2019.

Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14 , pages 87-102. Springer, 2016.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017.

Y. Huang, F. Chen, S. Lv, and X. Wang. Facial expression recognition: A survey. Symmetry , 11(10):1189, 2019.

J. Jiang and W. Deng. Boosting facial expression recognition by a semi-supervised progressive teacher. IEEE Transactions on Affective Computing , 2021.

X. Jin, Z. Lai, and Z. Jin. Learning dynamic relationships for facial expression recognition based on graph convolutional network. IEEE Transactions on Image Processing , 30:7143-7155, 2021.

T. Kanade, J. F. Cohn, and Y. Tian. Comprehensive database for facial expression analysis. In Proceedings fourth IEEE international conference on automatic face and gesture recognition (cat. No. PR00580) , pages 46-53. IEEE, 2000.

B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09217 , 2019.

D. Kim and B. C. Song. Emotion-aware multi-view contrastive learning for facial emotion recognition. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIII , pages 178-195. Springer, 2022.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

J. Kumari, R. Rajesh, and K. Pooja. Facial expression recognition: A survey. Procedia computer science , 58:486-491, 2015.

N. Le, K. Nguyen, Q. Tran, E. Tjiputra, B. Le, and A. Nguyen. Uncertainty-aware label distribution learning for facial expression recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6088-6097, 2023.

H. Li, N. Wang, X. Ding, X. Yang, and X. Gao. Adaptively learning facial expression representation via cf labels and distillation. IEEE Transactions on Image Processing , 30:2016-2028, 2021.

H. Li, N. Wang, X. Yang, and X. Gao. Crs-cont: a well-trained general encoder for facial expression analysis. IEEE Transactions on Image Processing , 31:4637-4650, 2022.

H. Li, N. Wang, X. Yang, X. Wang, and X. Gao. Towards semi-supervised deep facial expression recognition with an adaptive confidence margin. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4166-4175, 2022.

H. Li, N. Wang, X. Yang, X. Wang, and X. Gao. Unconstrained facial expression recognition with no-reference de-elements learning. IEEE Transactions on Affective Computing , 2023.

H. Li, N. Wang, Y. Yu, X. Yang, and X. Gao. Lban-il: A novel method of high discriminative representation for facial expression recognition. Neurocomputing , 432:159-169, 2021.

S. Li and W. Deng. Deep facial expression recognition: A survey. IEEE transactions on affective computing , 13(3):1195-1215, 2020.

S. Li, W. Deng, and J. Du. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2852-2861, 2017.

T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Z. Liu, Y. Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

T. Lukov, N. Zhao, G. H. Lee, and S.-N. Lim. Teaching with soft label smoothing for mitigating noisy labels in facial expressions. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XII , pages 648-665. Springer, 2022.

M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba. Coding facial expressions with gabor wavelets. In Proceedings Third IEEE international conference on automatic face and gesture recognition , pages 200-205. IEEE, 1998.

D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV) , pages 181-196, 2018.

A. Mollahosseini, B. Hasani, and M. H. Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing , 10(1):18-31, 2017.

K. Oksuz, B. C. Cam, S. Kalkan, and E. Akbas. Imbalance problems in object detection: A review. IEEE transactions on pattern analysis and machine intelligence , 43(10):3388-3415, 2020.

A. Psaroudakis and D. Kollias. Mixaugment & mixup: Augmentation methods for facial expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2367-2375, 2022.

D. Ruan, Y. Yan, S. Lai, Z. Chai, C. Shen, and H. Wang. Feature decomposition and reconstruction learning for effective facial expression recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7660-7669, 2021.

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618-626, 2017.

J. She, Y. Hu, H. Shi, J. Wang, Q. Shen, and T. Mei. Dive into ambiguity: Latent distribution mining and pairwise uncertainty estimation for facial expression recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6248-6257, 2021.

L. Shen, Z. Lin, and Q. Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14 , pages 467-482. Springer, 2016.

J. Tan, C. Wang, B. Li, Q. Li, W. Ouyang, C. Yin, and J. Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11662-11671, 2020.

M. Valstar, M. Pantic, et al. Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In Proc. 3rd Intern. Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect , page 65. Paris, France., 2010.

G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8769-8778, 2018.

K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao. Suppressing uncertainties for large-scale facial expression recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6897-6906, 2020.

K. Wang, X. Peng, J. Yang, D. Meng, and Y. Qiao. Region attention networks for pose and occlusion robust facial expression recognition. IEEE Transactions on Image Processing , 29:4057-4069, 2020.

X. Wang, L. Lian, Z. Miao, Z. Liu, and S. X. Yu. Long-tailed recognition by routing diverse distributionaware experts. arXiv preprint arXiv:2010.01809 , 2020.

F. Xue, Q. Wang, and G. Guo. Transfer: Learning relation-aware facial expression representations with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3601-3610, 2021.

D. Zeng, Z. Lin, X. Yan, Y. Liu, F. Wang, and B. Tang. Face2exp: Combating data biases for facial expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20291-20300, 2022.

K. Zhang, Z. Zhang, Z. Li, and Y. Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters , 23(10):1499-1503, 2016.

Y. Zhang, B. Hooi, L. Hong, and J. Feng. Test-agnostic long-tailed recognition by test-time aggregating diverse experts with self-supervision. arXiv e-prints , pages arXiv-2107, 2021.

Y. Zhang, C. Wang, and W. Deng. Relative uncertainty learning for facial expression recognition. Advances in Neural Information Processing Systems , 34:17616-17627, 2021.

Y. Zhang, C. Wang, X. Ling, and W. Deng. Learn from all: Erasing attention consistency for noisy label facial expression recognition. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI , pages 418-434. Springer, 2022.

Z. Zhao, Q. Liu, and F. Zhou. Robust lightweight facial expression recognition network with label distribution training. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 3510-3519, 2021.

B. Zhou, Q. Cui, X.-S. Wei, and Z.-M. Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9719-9728, 2020.

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2921-2929, 2016.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with the topic of facial  expression recognition (FER). In particular, authors point out the problem of imbalanced class data since most FER data sets will have many more neutral or happy face images than images with other facial expressions.  Authors propose an approach to address this problem building on two ideas. First is that there may be information to be learnt about the minor classes even from the samples from the major classes.   Based on this observation, authors propose a novel attention map rebalancing to regularize the model. As the second idea, they  introduce a label smoothing approach that weights the minor classes more than the major classes. Results  on RAF-DB and FERPlus facial expression image data sets show that the proposed approach is able to outperform state of the art FER approaches.\n\nSTRENGTHS:\nThe main idea of learning about the minor classes from training samples in both major and minor classes appears to be novel and is one of the strengths of this paper. The idea of weighting the minor classes more heavily or using smoothed labels have been used in other applications before, but perhaps not in FER.\n\nAnother strength of this paper is the  set of numerical results provided. Both data sets being used are state of the art data sets and results clearly indicate that the proposed method outperforms competing methods for FER.\n\nAnother strength of the paper is that it is relatively well-written although authors should have more clearly explained the difference between \"overall accuracy\" and \"mean accuracy\".\n\nWEAKNESSES:\nIn Fig. 1, flipping of the images is shown to introduce the notion of re-balanced attention consistency. It is not clear if flipping is the only transformation that makes sense for the FER or if some other transformations (e.g., scaling, intensity attenuation or gain) make sense. That needs to be discussed in more detail.\n\nAnother weakness is that the proposed method is somewhat similar to the EAC approach although authors do a good job of clarifying the similarities and differences between their approach and EAC.\n\nThe main assertion that there is information to be gained about minor classes from samples in major classes seems to mainly backed up by pointing to the similarities in corresponding attention maps (e.g., open mouth regions for fear and surprise categories). This evidence is subjective and not entirely convincing.\n\nQUESTIONS:\n1. What is the difference between mean accuracy and overall accuracy?\n\n2. Line 106: How do we know that the two types of information are \"orthogonal\"? Perhaps, characterizing them as \"complementary\" might be more appropriate.\n\n3. Fig. 1 caption: \"--- while do not degrade the high accuracy on major classes\" should be \"--- while not degrading the high accuracy on major classes\"\n\n4. Fig. 2: I assume that the bar corresponding to 0.6 probability is for neutral images --- that needs to be clearly indicated in the figure.\n\n5. Eq. (7): Is the tilde on y() here being used to denote the flipped samples or something else?\n\n6. Shouldn't there be bold-face numbers in every column of Table 1 and other tables?\n\nLIMITATIONS:\nThere is no discussion of  limitations in this submission. \n\nAuthors may want to add some discussion of the  performances of the proposed approaches based on gender, race and/or age considerations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly focuses on solving the imbalanced problem in Facial Expression Recognition (FER). The goal of this paper is to enhance the performance on minor classes without compromising the performance on major classes. The contribution is two fold. Re-balanced attention consistency (RAC) module is proposed to mines extra knowledge pertaining to minor classes from both major and minor samples. Re-balanced smooth labels (RSL) module is proposed to regulate the classification loss and promote balanced learning. Extensive experiments on different datasets and backbones validate the effectiveness of the proposed method.\n\nSTRENGTHS:\n(1) This paper adapts attention map consistency to solve the imbalanced learning problem of FER for the first time, which proves that solving imbalanced FER through re-balanced strategy is promising.\n\n(2) The experiment including various backbones, imbalance and attention visualization is sufficient.\n\nWEAKNESSES:\n(1) The purpose of re-balanced attention consistency (RAC) is to extract balanced and transformation invariant knowledge of minor classes from all training samples. The word \"knowledge\" is unclear and should be more specific. Does it refer to high response values for specific regions on the facial feature map or something else?\n\n(2) The authors' explanation about the role of re-balanced smooth labels (RSL) is to improve performance to regulate the classification loss and promote balanced learning. This explanation is too abstract and difficult for readers to understand. Why is it designed as Eq.(8)? What do the two terms of Eq.(8) represent and why fusing these two terms works? I think the author should provide a more detailed explanation.\n\n(3) The proposed re-balanced attention consistency (RAC) is very similar to the attention consistency used in EAC. The difference is that an additional balance weight is used to re-weight the attention maps. I would like to know how much performance has been improved due to the addition of balance weight. I have looked at the ablation study in Table 5 and can only know the performance improvement of the entire RAC module.\n\n(4) The comparison in Table 1 is questionable. The overall accuracy of EAC is 88.01%, which does not match the number of 89.99% in the original EAC paper (See Table 6 in [46]).\n\n(5) In line 283, the authors think non-overlapping attention maps is better than overlapping ones. Are there any references supporting this viewpoint?\n\nOverall I think the paper is tackling the right problem. However, the unclear description of the proposed modules weakens the contribution of the paper.\n\nQUESTIONS:\nSame as the weakness section.\n\nLIMITATIONS:\nThe authors did not discuss the limitation in this paper. I suggest that the authors provide the \"maximum capability\" of the proposed method. In other words, what degree of imbalance will cause the proposed method to fail.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the imbalanced learning problem in facial expression recognition (FER). Unlike existing works in imbalanced learning for image classification, the proposed method addresses imbalanced FER from a novel perspective of label distribution learning. Specifically, the proposed method is motivated by the observation that certain major classes in FER may contain valuable features for the minor classes. Building upon this observation, a re-balanced attention map consistency module is introduced, which encourages the model to extract useful information about the minor classes from both minor and major-class samples through attention map consistency. Additionally, a re-balanced label smooth module is incorporated to leverage prior knowledge of the training data distribution and regularize the classification loss. Experimental results on FER datasets with varying levels of imbalance and different backbone architectures demonstrate the effectiveness of the proposed method, particularly in improving the performance of the minor classes.\n\nSTRENGTHS:\n1. This paper provides an interesting label distribution learning perspective to deal with the imbalanced FER, which is novel to me. It proposes two modules to mine extra information of minor-class samples from both major and minor-class samples. From my understanding, this method is specifically designed for FER and technical novelties are sound.\n\n2. The experimental demonstration well supports the claims. The authors have curated different FER datasets with various imbalance factors and the performance improvement on these imbalanced FER datasets is non-trivial compared with other SOTA methods, especially on the minor classes like fear and disgust. Ablation studies presented in Table 5 indicate that each of the proposed modules contributes to the overall performance. I also noticed that in Table 4, combined with Swin-T, the proposed method achieves remarkable results of 92.31% overall accuracy and 87.71% mean accuracy on the widely used RAF-DB dataset.\n\n3. The proposed method is easy to implement and the paper is written in an easy-to-understand manner.\n\nWEAKNESSES:\n1. Lack of experiments. As illustrated by the authors, they utilize the weight (i.e., a hyper-parameter of 0.9999) introduced by Ref. [5] as the re-balanced weight for both the re-balanced attention maps and re-balanced smooth label. However, I think there should be some studies regards of this hyperparameter, and what if we utilize different values for the two proposed modules?\n\n2. Some claims were not supported by experiments. The authors claimed that \"EAC employs attention maps from all expression classes, rather than just the labeled class, which is similar to label distribution learning\". However, there is a lack of experiments to validate this claim. I believe the authors should include experiments to validate their point of label distribution learning, i.e., what is the performance if we only utilize the attention map of the labeled class instead of all classes?\n\n3. Comparative techniques were not sufficiently performed. For example, what if we use GradCAM instead of CAM to implement attention consistency? What if we utilize the basic label smooth instead of the proposed re-balanced label smooth?\n\nQUESTIONS:\nI have some questions regarding the weight design of the proposed method. Since the weight is utilized by both of the proposed modules simultaneously, I think it is very important for the performance of the proposed method. Therefore, I am curious what the performance would be if we simply set the weight as the inverse of the sample numbers of each class? There should be more discussions regarding the weight design of this method.\n\nLIMITATIONS:\nThe limitations of this paper should be discussed. This method is designed specifically for imbalanced learning in FER as FER has the label distribution learning characteristic. However, things might be different in other imbalanced image classification tasks. For example, minor classes in imbalanced CIFAR-100 might contain little common feature with major classes in imbalanced CIFAR-100.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper, the authors tackle the issue of imbalanced learning in the facial expression recognition task by introducing a fresh approach. Their proposed method revolves around the concept of attention consistency under spatial transforms, aiming to extract information about multiple classes from each sample. This approach effectively enhances the performance of under-represented classes. The authors evaluate their approach on two datasets and successfully achieve state-of-the-art performance with a transformer backbone.\n\nSTRENGTHS:\n(+) They achieve state-of-the-art (SOTA) performance on the RAF-DB dataset with the Swin backbone.\n\n(+) Table 4 clearly demonstrates the performance improvement brought by the proposed approach. It highlights the significant increase in performance for both minor classes and the overall score.\n\n(+) Despite attention consistency being proposed in previous work, the authors cleverly and novelly employ a modified version of it incorporating rebalanced smooth labeling. They provide a reasonable hypothesis, stating that information for minor classes can be captured from all training samples.\n\n(+) The visualization results presented in both section 4.7 and the appendix provide further evidence that the proposed approach functions effectively.\n\nWEAKNESSES:\n(-) To further support the idea, I suggest conducting evaluations on multi-label classification using other facial expression recognition (FER) datasets, such as BP4D. If the hypothesis is correct, the proposed approach should also work effectively in a different type of FER task. Testing it in this manner would provide a good opportunity for validation.\n\n(-) The two methods proposed in the paper, in terms of their structure, are not entirely novel. However, there is potential for further refinement and innovation in their design.\n\nQUESTIONS:\n- What is the purpose of utilizing Global Average Pooling to extract features? Doesn't it result in a loss of information?\n\n- Based on the findings presented in Table 4, it appears that the performance improvement for minor classes surpasses that of other classes. Regarding the fairness aspect (although it may not be the primary focus of the paper), what can be said about the trained models? Do you believe that your method also facilitates the development of fairer models?\n\nLIMITATIONS:\nGiven that it is a facial expression recognition task, privacy could potentially be a concern.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an approach to tackle the imbalance problem in facial expression recognition. The approach makes use of information from the majority class to help improve performance of the minority classes. The proposed approach is evaluated on two public FER datasets. Different backbone networks are evaluated, along with comparisons to state of the art and ablation studies.\n\nSTRENGTHS:\nThe paper investigates a challenging problem. and the proposed approach using attention map consistency is interesting. Encouraging results are shown on two public datasets. Ablation study is generally well conducted.\n\nWEAKNESSES:\nExperimental design is lacking. FER2013 is an old dataset; newer FER datasets should be evalauted (e.g., AffectNet - Mollahosseini, Ali, Behzad Hasani, and Mohammad H. Mahoor. \"Affectnet: A database for facial expression, valence, and arousal computing in the wild.\" IEEE Transactions on Affective Computing 10.1 (2017): 18-31.). Along with being newer, AffectNet needs to be evaluated as the approach is on imbalanced FER datasets. The paper conducts experiments under different imbalance factors, however, AffectNet is one of the largest and most imbalanced datasets available for FER. For example, in AffectNet, Happy is ~46% of the data and contempt is ~1% of the data. How does the proposed approach handle these extreme imbalance issues?\n\nThe comparison to CB [5] in Table 1 is not clear. This paper is not on FER and does not evaluate RAF-DB. What is this?\n\nPaper states the accuracy of each class is not commonly shown in FER papers. This is not accurate. If it is not in a table, this is often shown through confusion matrices (e.g., Farzaneh, Amir Hossein, and Xiaojun Qi. \"Facial expression recognition in the wild via deep attentive center loss.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.)\n\nQUESTIONS:\nHow does the proposed approach handle extreme imbalance such as in AffectNet (e.g., Happy vs. Contempt)?\n\nWhat is reference [5] in Table 1? This is not a FER paper that is being compared to.\n\nLIMITATIONS:\nThe limitations of the proposed work are not addressed. How does the approach work on extremely imbalanced data? What kind of societal impact can this have? There are always ethical concerns with FER systems.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}