Leave No Stone Unturned: Mine Extra Knowledge for Imbalanced Facial Expression Recognition
Authors: Yuhang Zhang
Yaqi Li
lixiong Qin
Xuannan Liu
Weihong Deng
Venue: NeurIPS 2023 poster (2023)
Keywords: Facial expression recognition
imbalanced learning
TL;DR: Mine extra knowledge related to the minor classes from both minor and major-class samples

Abstract: Facial expression data is characterized by a significant imbalance, with most collected data showing happy or neutral expressions and fewer instances of fear or disgust. This imbalance poses challenges to facial expression recognition (FER) models, hindering their ability to fully understand various human emotional states. Existing FER methods typically report overall accuracy on highly imbalanced test sets but exhibit low performance in terms of the mean accuracy across all expression classes. In this paper, our aim is to address the imbalanced FER problem. Existing methods primarily focus on learning knowledge of minor classes solely from minor-class samples. However, we propose a novel approach to extract extra knowledge related to the minor classes from both major and minor class samples. Our motivation stems from the belief that FER resembles a distribution learning task, wherein a sample may contain information about multiple classes. For instance, a sample from the major class surprise might also contain useful features of the minor class fear. Inspired by that, we propose a novel method that leverages re-balanced attention maps to regularize the model, enabling it to extract transformation invariant information about the minor classes from all training samples. Additionally, we introduce re-balanced smooth labels to regulate the cross-entropy loss, guiding the model to pay more attention to the minor classes by utilizing the extra information regarding the label distribution of the imbalanced training data. Extensive experiments on different datasets and backbones show that the two proposed modules work together to regularize the model and achieve state-of-the-art performance under the imbalanced FER task. Code is available at https://github.com/zyh-uaiaaaa.

Leave No Stone Unturned: Mine Extra Knowledge for Imbalanced Facial Expression Recognition
Yuhang Zhang, Yaqi Li, Lixiong Qin, Xuannan Liu, Weihong Deng Beijing University of Posts and Telecommunications (zyhzyh, yaqili, lixiongqin, xuannanliu, whdeng) edu, on
Abstract
Facial expression data is characterized by a significant imbalance, with most collected data showing happy neutral expressions and fewer instances of fear disgust. This imbalance poses challenges to facial expression recognition (FER) models, hindering their ability to fully understand various human emotional states. Existing FER methods typically report overall accuracy on highly imbalanced test sets but exhibit low performance in terms of the mean accuracy across all expression classes. In this paper, our aim is to address the imbalanced FER problem. Existing methods primarily focus learning knowledge of minor classes solely from minor-class samples. However, we propose novel approach to extract extra knowledge related the minor classes from both major and minor class samples. Our motivation stems from the belief that FER resembles a distribution learning task, wherein sample may contain information about multiple classes. For instance, a sample from the major class surprise might also contain useful features of the minor class fear. Inspired by that, we propose a novel method that leverages re-balanced attention maps to regularize the model, enabling it to extract transformation invariant information about the minor classes from all training samples. Additionally, we introduce re- re-balanced smooth labels to regulate the cross-entropy loss, guiding the model pay more attention to the minor classes by utilizing the extra information regarding the label distribution of the imbalanced training data. Extensive experiments on different datasets and backbones show that the two proposed modules work together to regularize the model and achieve state-of-the-art performance under the imbalanced FER task. Code is available at btttps///githue www.co////////oo htt//yyyuuiaaaa
Introduction
Facial expression recognition (FER) plays crucial role in enabling machines to understand human emotional states, thus facilitating the realization of machine intelligence [18 [1] 25). Recent research efforts 42 35 49 have employed in-the-wild datasets to train FER models, resulting in impressive performance in terms of overall classification accuracy. However, these FER datasets suffer from a significant imbalance, with higher abundance of happy and neutral expression faces, which easily obtainable from the internet and daily life, compared to faces displaying negative expressions such as fear disgust 6 32). This imbalance can have adverse effects buman-computer interaction when FER models misinterpret infrequently occurring negative emotions as frequently occurring positive emotions.
In this paper, our objective is to investigate imbalanced learning in the context of FER. We observe that existing FER methods [34 43 42 29
yield relatively low performance in terms of the mean accuracy across all classes, mainly due to the highly imbalanced nature of the training data. While imbalanced (long-tailed) learning methods in the image classification domain have shown some
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

improvements [52 55, they provide limited enhancements in the mean accuracy within the FER field. This can be attributed several factors. Firstly, FER datasets typically contain small number of classes, resulting in marginal improvements in the mean accuracy of all classes. Secondly, the increase in accuracy for minor classes often comes at the expense of decreased accuracy for major classes. This imbalance disproportionately affects the overall accuracy, particularly when evaluating on imbalanced test sets.
To address the aforementioned problem, our objective is to design a method that can maintain high performance on major classes while improving the performance on minor classes to the greatest extent possible. We refrain from modifying the dataset structure through under- mader-sampling or over-sampling as these approaches may enhance the performance of minor classes at the expense of degrading the performance on major classes. Instead, we aim to leverage each sample fully to extract additional ledge relevant to minor classes. Our motivation is that FER resembles a label distribution learning task, implying that samples from major classes may contain information pertaining to minor classes as well. For instance, the major class "surprise" shares certain similarities with the minor class "fear".
Inspired by that, we propose novel method that mines extra knowledge regarding minor classes from samples belonging to both major and minor classes. This extra information enhances the recognition performance on minor classes without significantly compromising the performance on major classes. Specifically, we employ the concept of attention map consistency [7]. which is utilized in the EAC method [50] prevent FER models from memorizing noisy labels. We observe that EAC employs attention maps from all expression classes, rather than just the labeled class, to regularize a specific sample, resembling the idea of label distribution learning. Building upon this insight, we introduce a re-balanced attention consistency module to address the imbalanced FER task for the first time. In our approach, attention maps for all expression classes can be extracted from a given FER sample. We introduce re-balanced weights, following the design of [5), which are set inversely proportional to the effective number of samples belonging to each class. These weights facilitate the model in extracting additional knowledge related to minor classes from all training samples. Furthermore, as the classification loss may still be affected by the imbalanced data, we propose a re- balanced smooth labels module that utilizes the acquired re-balanced weights and the extra knowledge of the imbalanced label distribution. This module guides the model to focus more on minor classes during the docisiinnmmking process.
To demonstrate the effectiveness of our proposed method, we conducted extensive experiments on different FER datasets, including datasets with varying imbalance factors [5). Ablation studies confirmed that each proposed module contributes to the enhancement of mean accuracy, and the two modules synergistically achieve state-of-the-art performance. Moreover, we evaluated the generalization ability of our method by combining it with different backbones including transformers. The experimental results showcased that method is lightweight, easy to implement, and seamlessly compatible with various backbones, thus significantly improving their performance.
The main contributions of our work are listed as follows:
We highlight the imbalanced learning problem in FER and find existing methods in the large-scale image classification field might fall short in the FER field, we benchmark existing FER methods utilizing both overall accuracy and mean accuracy on all classes.
We propose a novel method consisting of two modules: re-balanced attention consistency (RAC) and re-balanced smooth labels (RSI... RAC mines extra knowledge pertaining to minor classes from both major and minor samples, thereby enhancing performance on minor classes without compromising performance on major classes RSL leverages the imbalanced label distribution to further regularize the classification loss and prioritize the catision-making process for minor classes.
Our proposed method is easy to implement, lightweight, and seamlessly compatible with different backbone architectures including transformers. Through extensive experiments, we validate that our proposed method achieves state-of-the-art performance in terms of both overall accuracy and mean accuracy of all classes across different FER datasets and backbone architectures.

---

2 Related work
Facial expression recognition methods in recent years primarily focus on extracting effective features
from in-the-wild datasets using deep learning models [22] 46 13 T6
20 24 21 23]. In-the-wild
datasets [26] [ pose greater challenges and are more prone to noise due to label inaccuracies, difficult
poses, occlusions, and low-quality images compared to laboratory collected datasets 30) 14 40)
These factors adversely affect the recognition performance of facial expression recognition (FER)
models. Several works have addressed the real-world FER task. SCN [42] introduces a learnable temperature and relabel module to weight FER samples and suppress noise. DMUE [37] Iearns sub-modules for each class to handle noisy labels. RUL 49 treats expression uncertainty as
relative concept and learns uncertainty values for images through comparisons, aiding overall feature learning. TransFER [45] pioneers the use of vision transformers for in-the-wild FER. EAC [50)
employs erasing attention map consistency to prevent FER models from memorizing noisy labels in real-world FER datasets.
While these approaches achieve impressive overall accuracy on test sets, they tend to exhibit lower performance in terms of mean accuracy of all classes, particularly for minor classes. In fact, previous works primarily focus on mitigating noise in in- in-the-wild datasets, often overlooking the imbalanced nature of FER datasets. In this work, our objective is to address imbalanced learning in in-the-wild FER datasets and provide an orthogonal supplement for previous works.
Imbalanced learning in real-life datasets, where certain classes have majority of samples while others have very few, has been addressed through three perspectives: data pre-proes pre-processing loss re- weightiing, and model ensemble. This paper primarily focuses the first two categories, model ensemble methods [44] 48) are computationally intensive. Data pre-processing methods commonly have shown that data re-sampling can negatively impact representation learning. Over-sampling may introduce duplicated samples, leading to the increased risk of overfitting, while suder-sampling may discard valuable examples. Another technique is data augmentation [3).. which applies predefined transformations to augment the dataset, particularly for minority classes. However, finding effective augmentation methods for facial expression recognition (FER) data, which contain specific local features related to expressions, can be challenging. Loss re- weighting methods assign different weights classes or instances during The aim is to propagate appropriate gradient
involve data re-sampling [31 38. However, studies [5] 52
training, known as loss re-weighting (27 39 41 5
values for all classes during training.
3 Method
In this section, we present novel method for imbalanced facial expression recognition. Our approach focuses on extracting extra information of minor classes from both minor and major class samples, rather than solely relying minor class samples. We introduce two modules, namely re-balanced attention consistency (RAC) and re-balanced smooth labels (RSL), to guide the model to extract balanced information from the entire training set.
3.1 Re-balanced attention consistency
We first introduce attention map consistency 77. which regularizes the attention maps to follow the same spatial transform if the input images are spatially transformed, which can help the model to learn transformation invariant features. EAC designs imbalanced framework to utilize attention map consistency to prevent the models from memorizing the noisy labels. We find that instead of only extracting attention maps of the latent truth like utilizing GradCAM (36], EAC uses attention maps of all classes to regularize the FER model. We speculate that the reason lies in that utilizing attention maps of all classes is similar label distribution learning 3 51 19, which mines information of several classes from one training sample. Inspired by that, we adapt attention map consistency to solve the imbalanced learning problem of FER for the first time. To be more specific, given sample, the attention maps of all expression classes should follow the same spatial transform if the given sample spatially transformed. Thus, the model could mine useful minor-class information from all samples instead of only the samples from the minor classes. Furthermore, we introduce the ce-balanced attention consistency to regularize the model to focus more on the minor classes to achieve balanced learning. We set different weights to the attention maps of different classes
3

---

Original
F
Mean
M
class
class
GAP
Re Re-balanced loss
Re Re-balanced weight
Consistency
loss
Flipped
Mean
Flip
The illustration of ce-balanced attention consistency. We propose re-balanced attention consistency to facilitate the model to mine extra transformation invariant knowledge of minor classes from both major and minor-class samples, which boosts the classification accuracy on minor classes while do not degrade the high accuracy major classes.
Figure 1:
and guide the model learn more invariant information related minor classes. Inspired by [5]. which designs a re-weighting scheme that uses the effective number of samples of each class to re-balance classification loss, we propose to use the effective number of samples to inversely the weigh attention maps of different classes. In the following, we formulate the re-balanced attention consistency module in detail.
Given images we first flip them to get their flipped counterparts X. The features of and x are extracted from the last convolutional layer, denoted as F E R-NxCxH>W and F E xx where N. C, H and W respectively represent the number of images, channels, height and width. We input F and F to the global average pooling (GAP) layer to get features E IR Nxxxx1 and f Nxxxxx and then resize them to N x C. The classification loss is computed according to
Wxf
W.
1
lcls
EE (log(
Σ W.f. log(
N
EL
W.f.
where W d1 is the y4-th weight of the fully connected (FC) layer and yi is the label of x,, L is the total number of expression classes. CAM 5 is utilized to compute attention maps A E TR Xx
and A R NxLXHX for and ğ following
A(i, w) Σ ((L.e))(i.ch.....
(2)
c=1
where i, 1, c, h, w represent the sample, expression class, channel, height and width number. We then re-balance the attention maps A and A through weight B E R² to get the re-balanced attention maps M and M following
M(i, h, w) B, A(i,1, w).
The balance weight B _ is enlightened by the re-balanced weight based on the effective number in (5), which is computed following
B 1
gm
where nı is the number of training samples in class I, B E [0, 1) is the hyperparameter controlling the re-balanced weight, a larger value of emphasizes more of the minor samples, following [5], we set the 3 as 0.9999 across all our experiments. As the attention map before and after the flip transformation should be consistent with each other, compute the consistency loss between M
and (l following
ALE W N ΣΣΣΣ
Lcons NLHW 1=1
[M(i,L,h,,)) - (lip(M)(C) w)ll
The training loss is computed as
Iraii = Lats Xloons
(6)
where À is the weight of the attention consistency loss, which determines the relative importance of the consistency loss compared to the classification loss in the overall training objective.

---

0.6
0.6
sample number
surptise
disgust
happiness sadness
anger
sarprise
disgust happiness
anger
reverse
0.4/6
0.4/6
0.4/6
0.4/6
0.4/6
0.4/6
smooth label
re-balanced smooth label
Figure 2: The illustration of bablllncedd smooth labels. We propose re-balanced smooth labels to utilize the existing prior knowledge about the label distribution of the training set to guide the model towards placing more emphasis on minor classes during decision-makinn while maintaining high performance on major classes.
3.2 Re-balanced smooth labels
In Section 3.1
introduced the re-balanced attention consistency module, which enables the model to extract balanced and transformation invariant knowledge of minor classes from all training samples, resulting in improved classification accuracy for those classes. However, the imbalanced training set can still negatively impact the classification loss, denoted as las. To address this, we present the re-balanced smooth labels module in this section. This module aims regulate the classification loss and promote balanced learning.
We denote the prediction of the FER model towards P,. where P; is the likelihood the model assigns to the i-th given sample x,, the one-hot label of x; denoted as yz- For simplicity, we re-write the classification loss towards x; Eq. 7 and things are the same with X,.
N
N L
lets = N
Σ
HLy P.)
Σ
yi. ()lgg(p(t,)),
N
where (i, is "1" for the correct class of x; and "0" for the rest. For FER model trained with a re-balanced smooth label of parameter a, we minimize instead the cross-entropy between the re-balanced targets y, and the model's outputs P,, where
((, = (1- aB,LL,
(8)
B, is the re-balanced weight of the class given in Section 3.1
3.3 Discussion with existing works
Our most related work is EAC. Though both methods utilize attention map consistency. EAC aims to address the noisy label learning task, while our method focuses on imbalanced learning. Furthermore, the motivations of the two works are different. EAC utilizes the difference of attention maps before and after the transformation to prevent the model to memorize noisy labels, while propose the re- be-balanced attention consistency to guide the model to extract useful features related to minor classes from both major and minor samples. We novelly solve the imbalanced FER task through label distribution learning perspective. We also propose re- re-balanced smooth labels module to further regularize the classification loss from the negative effect of the imbalanced training set. use extra knowledge of label distribution to facilitate the model to focus more the prediction of minor classes. The motivation effectively utilizing all training samples to deal with imbalanced FER and increase the performance of minor classes led to the title of our paper: leave stone unturned.
5

---

Table 1: Comparison with different methods on RAF-DB using pre-trained ResNect-8 as backbone. * denotes that we copied the accuracy from the original paper. We arrange the expression classes according to their sample numbers and observe that they exhibit varying levels of difficulty. For instance, despite having a small number of training samples, all methods achieve relatively high performance on the anger class. On the other hand, the disgust and fear classes prove to be the most challenging. and our method achieves the highest accuracy on these two classes, We report the last epoch accuracy of all methods, we also display the best accuracy of our method for reference.
Method
Conference
Happiness Neutral Sadness Surprise Disgust Anger Fear Overall Mean
Baseline
95,44
88.53
85.56
83.59
58.75
78.40
59.46
87.42
78.53
CB
[5]
CVPR'19
95.11
90.74
84.73
86.93
64.38
73.46 59.46
88.04
79.26
SCN 42
CVPR 20
94.77
90.29
80.33
86.93
60.00
76.54
45.95
86.73
76.40
BBN 122
CVPR 20
93.59
91.62
84.94
84.80
61.88
77.78
52.70
87.39
78.19
PT
12
21
96.00
92.00
87.00
87.00
55.00
81.00 54.00
88.80
78.86
RUL
49
NeurlPS'21
95.78
87.06
86.19
89.36
65.00
83.33 64.86
88.66
81.66
EAC
50
2CCV'22
95.95
92.06
87.03
88.15
66.88
85.80
58.11
89.90
82.00
Ours
NeurlPS'23
96.37
89.56
89.33
87.84
66.89
80.86
66.22
89.77
82.44
Ours (best) NeurlPS'22 23
96.03
87.79
89.33
87.23
73.13
84.57
70.27
89.80
84.05
Experiments
4.1 Datasets
RAF-DB 26 has 30,000 facial expression images, which are annotated with basic or compound expressions by 40 trained annotators. In our experiment, we only utilize the 7 basic expressions and these include 12,271 images for training and 3,068 images for testing. We report the mean accuracy on all expressions to evaluate the imbalanced learning performance of different methods.
FERPlus [1] is extended from FER2013 [6] with finer labels. It is collected by the Google search engine with ten crowd-sourced annotators labeling each image. The most voting category is used as the annotation for fair comparison with other FER methods 37 We utilize the same 7 classes with RAF-DB, which results in 24,941 images for training and 3,137 images for testing.
AffectNet 33 is a large-scale FER dataset, which contains eight expressions (7 basic expressions and contempt). There are a total of 286,564 training images and 4,000 test images. We carry out experiments with both 7 basic classes and 8 classes. As the test set of AffectNet is balanced, the mean accuracy on all expressions is the same with the overall accuracy.
4.2
Implementation details
To make fair comparisons with other methods, we compare all methods under the same backbone of MS-Celeb-111 1M (8] pre-trained ResNet-18. We also test the effectiveness of our method under
different backbones including MobileNet [10], ResNet-50 ( and Swin Transformer [28]. The facial expression images
detected and aligned using MTCNN (7). The image size is 224 x 224, the learning rate is set to 0.0001. We use Adam [17] optimizer with weight decay of 0.0001 and ExponentialLR learning rate scheduler with a gamma of 0.9. The max training epoch Tman
to 60. The consistency weight À is set to 2 and the smooth parameter is set to 0. according to the ablation study in section 4.7 All experiments are conducted on NVIDIA RTX 2080Ti.
4.3 Experiments on imbalanced FER datasets
We conduct experiments on the RAF-DB dataset evaluate the performance of different methods in imbalanced learning. We report the classification accuracy of each class and the overall and mean accuracy to assess their effectiveness. The baseline method involves training with the pre-trained ResNet-18 18 without additional modules. We compare our method with imbalanced learning methods CB, BBN, and saite-of-the-aad FER methods SCN, PT, RUL, and EAC. To obtain the mean accuracy, we re-implement these methods based on their open-source code. The results in Table [ demonstrate that our method achieves the highest overall accuracy of 89.77% and mean accuracy of 82.44% on the RAF-DB dataset using ResNet-18 18 as the backbone.

---

Table 2: Comparison with different methods on AffectNet using pre-trained ResNet-18 18 as backbone. We carry out experiments with both 7 and 8 classes.
Method
Conference
Happiness Neutral Sadness
Anger
Surprise
Fear
Disgust Contempt Mean
SCN
42
CVPR'20
95.20
82.70
44.20
56.30
35.80
38.00
20.90
53.30
BBN
52
CVPR 20
87.00
57.10
66.80
58.30
54.90
71.10
30.10
60.76
RUL
49
21
90.50
62.40
64.70
69.30
60.80
49.00
34.20
61.56
EAC
50
ECCV'22
91.40
64.50
65.70
66.30
61.60
60.90
45.80
65.17
Ours
NeurlPS 23
86.20
59.00
64.20
66.50
57.80
64.50
61.90
65.73
SCN
42
CVPR 20
94.60
74.90
58,20
63.80
40.90
43.20
30.80
2.20
51.08
BBN
52
CVPR'20 20
78.40
58.40
60.60
67.70
59.40
55.00
37.00
46.70
57.90
RUL
49
NeurlPS'21
71.00
63.40
46.60
54.90
53.70
58.60
44.70
47.70
55.08
BAC
50
ECCV'22
84.00
58.80
65.00
65.90
62.20
60.30
46.10
41.90
60.53
Ours
NeurIPS'23
78.60
54.30
63.80
59.50
57.60
64.10
59.40
60.00
62.16
Based on the accuracy of each class, which is not commonly reported in previous works, we observe varying levels of difficulty among the expression classes. For instance, despite having very few training samples, all methods achieve relatively high performance on the anger class. We speculate that the distinct features associated with anger make easier for FER models to detect. On the other hand, the disgust and fear classes prove to be the most challenging, with all methods exhibiting low accuracy. However, our method outperforms others on these two classes, achieving accuracies of 66.89% and 66. 66.22% respectively. This highlights the effectiveness of our approach in extracting extra knowledge from both major and minor-class samples to address the imbalanced learning problem.
We further carry out experiments on AffectNet, which is one of the largest and most imbalanced datasets available for FER. From the results in Table 2 we could draw the conclusion that our method achieves the best mean accuracy on the test set under both 7 or 8 classes. Besides, we notice that our method improves existing methods on the minor classes of fear (Fea), disgust (Dis), contempt (Con) by remarkable margins, which illustrates that our method is more suitable for the imbalanced learning of FER task. Our method even achieves 60.00% accuracy on the contempt class under classes. Furthermore, and contempt (60.00%) compared with other methods under classes, which means our method fairer and achieves more balanced test accuracy.
method clearly decreases the test accuracy gap between happy (78.60%)
4.4 Experiments with different imbalance factors
Following existing imbalanced learning methods [5] 33 2 in the image classification field, we also construct imbalanced FER datasets with different imbalance factors. The definition of the imbalance factor of a dataset following [5] as the number of training samples in the largest class divided by the smallest. Given the imbalance factor, the imbalanced FER datasets are created by reducing the number of training samples per class according to an exponential function n nupu, where is the class index, Tu is the original number of training images and μ E (0, 1). Due space limitation, the sample number of each class is summarized in the supplementary material. We evaluate our method on RAF-DB and FERPlus datasets with imbalance factors ranging from 50 150, shown in Table g and Table 4 The results demonstrate the superior performance of our method across different imbalance factors. Specifically, compared to the state-of-the-arr FER method EAC, our method consistently improves upon it with 4.09%. 3.26%. 1.67% and 2.13%, 2.62%, 4.15% regrading mean accuracy of all classes RAF-DB and FERPlus respectively.
4.5 Different backbones
We evaluate the generalization ability of our method by combining it with four different backbones: MobileNet, ResNet- 18, ResNet-50, and Tiny Swin Transformer, on RAF-DB. The results consistently demonstrate the improvement in imbalanced learning performance across different backbones. We also observe that different backbones have a notable effect on the performance of imbalanced learning. Notably, when combined with Tiny Swin Transformer, our method achieves state-of-the-art performance with accuracy of 71. 1..2% and 85.00% on the most difficult expression classes of fear and disgust, respectively, as well overall accuracy of 92.31% 31% and a mean accuracy of 87. 87.71%.
7

---

Table 3: Comparison with other methods on RAF-DB with different imbalance factors. Disgust and fear are the most difficult classes. Our method achieves the highest accuracy on the overall, mean accuracy and the accuracy on the most difficult classes under different imbalance factors.
Method
Imbalance Happiness
Neutral
Sadness
Surprise
Disgust
Anger
Fear
Overall
Mean
Baseline
50
95.95
87.35
79.08
84.19
39.38
64.20
2.70
83.28
64.69
BBN
50
93.59
91.91
81.80
82.98
41.25
71.60
37.84
85.01
71.57
EAC
50
95.53
93.82
82.01
89.06
50.00
70.99
29.73
87.09
73.02
Ours
50
96.37
90.00
85.36
85.41
53.75
73.46
55.41
87.65
77.11
Baseline
100
97.72
87.94
73.85
81.76
10.63
54.94
0.00
80.96
58.12
BBN
100
94.94
93.38
71.34
82.37
36.88
65.43
31.08
83.44
67.92
EAC
100
95.27
92.06
83.68
89.97
36.88
62.35
28.38
85.79
69.80
Ours
100
96.37
91.18
82.85
86.63
44.38
65.43
44.59
86.47
73.06
Baseline
150
95.86
90.29
75.73
77.51
9.38
46.91
0.00
80.11
56.53
BBN
150
94.85
93.53
74.69
81.46
30.00
55.56
28.38
82.92
65.49
EAC
150
96.20
91.62
77.82
79.64
36.25
59.88
39.19
84.13
68.66
Ours
150
96.62
91.91
79.29
83.89
36.25
61.11
43.24
85.20
70.33
Table 4: Comparison with other methods FERPlus with different imbalance factors. Our method achieves the highest accuracy on the overall, mean accuracy and the accuracy on the most difficult classes (fear and disgust) under different imbalance factors.
Method Imbalance
Neutral
Happiness
Surprise Sadness
Anger
Fear
Disgust
Overall
Mean
Baseline
50
86,42
93.17
89.14
76.56
83.88
46.99
22.22
85,85
71.20
BBN
50
84.31
91.38
93.18
77.60
84.98
54.22
33.33
85.59
74.14
EAC
50
90.09
95.63
90.15
76.30
84.62
49.40
33.33
88.11
74.22
Ours
50
91.19
94.06
91.67
79.95
82.05
56.63
38.89
88.68
76.35
Baseline
100
90.32
93.56
87.63
66.80
78.21
46.99
22.22
85,43
69.39
BBN
100
88.62
91.71
93.18
74.74
81.32
51.81
38.89
86.48
74.32
EAC
100
90.73
95.30
90.66
74.48
81.32
53.45
27.78
87.87
73.39
Ours
100
91.56
94.85
92.42
77.34
82.78
54.22
38.89
88.81
76.01
Baseline
150
91.28
93.84
89.14
63.02
78.75
44.58
11.11
85,49
67.39
BBN
150
90.09
92.61
93.43
67.19
79.12
46.99
33.33
86.01
71.82
EAC
150
93.12
94.74
89.65
71.35
78.75
39.76
22.22
87.41
69.94
Ours
150
93.94
94.51
90.40
71.88
79.12
55.42
33.33
88.30
74.09
Table 5: The performance of our method under different backbones. We find that backbones have a significant influence on the accuracy. Our method consistently improves the performance under different backbones and achieves the state-of-the-ane overall accuracy of 92.31% and mean accuracy of 87.71% with Tiny Swin Transformer (Swin-T)..
Backbone
Happiness Neutral
Sadness
Surprise
Disgust
Anger
Fear
Overall
Mean
MobileNet
93.84
83.09
77.62
88.75
45.00
75.31
56.76
83.96
74.34
MobileNet+Ours
94.26
88.82
81.38
83.28
59.38
74.07
59.46
86.15
77.24
ResNet-18
95.44
88.53
85.56
83.59
58.75
78.40
59.46
87.42
78.53
ResNee-18+Ours
96.37
89.56
89.33
87.84
66.89
80.86
66.22
89.77
82.44
ResNet-50
94.77
87.79
87.03
85.71
68.75
84.57
60.81
88.33
81.35
ResNet-50+Ours
95.95
87.65
89.75
88.75
80.63
85.19
66.22
90.29
84.88
Swin-T
97.05
91.62
87.87
90.27
78.75
86.42
60.81
91.30
84.68
Swin-T+Ours
96.96
92.06
88.28
92.40
85.00
87.65
71.62
92.31
87.71

---

Table 6: Ablation study of our proposed two modules re- re-balanced attention consistency (RAC) and ce-balanced smooth labels (RSL). Both of the two modules can improve the performance based on the baseline, while they can cooperate to achieve the best performance.
RAC
RSL Happiness
Neutral
Sadness
Surprise
Disgust
Anger
Fear
Overall
Mean
95.44
88.53
85.56
83.59
58.75
78.40
59.46
87.42
78.53
96.29
89.26
87.87
88.75
65.63
76.54
59.46
89.08
80.54
94.60
90.44
85.15
82.07
57.50
80.86
64.86
87.4
79.35
96.37
89.56
89.33
87.84
66.89
80.86
66.22
89.77
82.44
mean accuracy (%)
89.7
Speraa Recuraes 1%
meun accuracy (%)
89.5
(%) Aatmmoe dyeall
02:2
82.0
80.6
88.5
02.0
09.5
81.3
81.8
B9.4
Ovee
81.0
86.5
81.6
meeal
89.5
IDcan
80.5
85.5
0.05
1.00
0.40
consistency weight X
smooth parameter a
Figure 3: The hyperparameter study of the consistency weight À and the smooth parameter a.
4.6
Ablation study
We conduct an ablation study on RAF-DB to evaluate the contribution of each proposed module. The results in Table demonsstate the effectiveness of both the cebbalaced attention consistency (RAC) and re-balanced smooth labels (RSL) in improving the baseline method's 's performance. Interestingly, we observe that utilizing only the RAC module achieves superior performance compared to using only the RSL module. This could be attributed to the additional information provided by the re-bal re-balanced attention map consistency. Moreover, combining both RAC and RSL modules results in even better performance, indicating their effective collaboration in addressing the imbalanced FER task.
4.7 Hyperparameter study
We carry out experiments on RAF-DB to study the effect of different hyperparameters to our method. We plot the results in Figure 3
Consistency weight A The results demonstrate that our method exhibits low sensitivity to the consistency weight X, with both the mean accuracy and overall accuracy varying within small range of 1% as À changes from 0.05 to 4. Notably, the optimal value for X in our method is found to be indicating that larger values may excessively prioritize consistency loss over classification loss, potentially leading to decrease in classification accuracy. Conversely, smaller values of À may fail to effectively regulate the model in extracting additional knowledge related to minor classes from all samples, thus negatively impacting accuracy.
Smooth parameter a The smooth parameter, ranging from 0 to 1, determines the strength of the latent truth (set as - a). We evaluate different values of a from 0.05 to 0.4 and find that the optimal value is a 0.1. A larger negatively impacts performance as the excessive smooth effect hampers the model's 's ability to learn useful information. On the other hand, smaller a fails to effectively utilize the prior knowledge of label distribution to prioritize minor classes.
4.8 Visualization results
We provide visualization results to illustrate the effectiveness of our proposed method. The learned attention maps, as shown in Figure 4 reveal three key observations. First, our method consistently learns attention maps that are more consistent across different transformations, enabling the FER model to capture transformation invariant information for various expression classes. Second, our method effectively extracts additional knowledge related to minor classes (e.g., disgust and fear) from samples of major classes (c.g., sadness and surprise), as there are shared features between major

---

origital
disgust
happinees
sadness
surprise
disgust
happiness
Sii
baseline
OurS
Figure 4: The attention maps corresponding to different classes learned by different methods. We utilize the attention map of certain class to mine transformation invariant information of that class. The attention maps of the labels are marked by red. Attention maps learned by our method are more consistent before and after the flip transformation across all different classes. Furthermore, shown in the last two columns, method mine extra knowledge related to the minor classes like fear (the open mouth feature) from the samples of major classes of surprise.
Table 7: Other transformation methods for re- re-balanced attention consistency.
Method
Happiness
Neutral
Sadness
Surprise
Disgust
Anger
Fear
Overall
Mean
Intensity
95.02
86.47
82.01
82.98
63.13
72.84
56.76
86.05
77.03
Scaling 95.78
91.91
84.31
85.41
75.00
82.10
60.81
89.37
82.19
Ours 96.37
89.56
89.33
87.84
66.89
80.86
66.22
89.77
82.44
and minor classes in FER. For instance, the first column in the right part of Figure demonstrates how our method captures the mouth comner feature associated with disgust from sample labeled as sadness. Furthermore, the last two columns the figure show that our method identifies the open mouth feature shared by fear and surprise, allowing to extract fear-related features from surprise samples. More results in the Supp. material. Third, our method produces non-overlapping attention maps for different classes, in contrast to the baseline method. For example, for the sample labeled as sadness, the attention maps for happiness and sadness learned by our method do not overlap, while they overlap in the baseline method. This indicates that the attention maps learned by method more meaningful and distinct.
4.9 Other transformations
Flipping of the images is shown introduce the notion of re-balanced attention consistency. In this section, we investigate whether some other transformations (e.g.. scaling, intensity attenuation, gain) work well under the imbalanced FER task. The results on RAF-DB in Table / illustrate that intensity transformation performs poorly, while scaling performs well, which almost surpasses method. The reason lies in that attention map consistency regularizes the model to focus the same regions before and after the transformation, which incorporates spatial information as the attention map in method has height and width dimensions. Thus, the transformation should be spatial-related transformation to maximize the function of the method.
5 Conclusion
In this paper, we investigate the imbalanced learning problem in facial expression recognition (FER). We observe that existing imbalanced learning methods tend improve performance on minor classes at the expense of major classes. Motivated by the label distribution learning characteristic of FER, we propose a novel approach extract additional knowledge about minor classes from both major and minor class samples. This allows us to enhance the performance on minor classes while maintaining high performance on major classes. Our method consists of two modules: re-balanced attention consistency and re- balanced smooth labels, which regulate attention maps and classification loss, respectively. Instead of relying on traditional over-sampling or under-sampling techniques, our method effectively utilizes all training samples and incorporates prior knowledge of the imbalanced data distribution prioritize minor classes. Through extensive experiments on various imbalanced FER datasets and with different backbones, validate the effectiveness of our proposed method.
10

---

Acknowledgments and Disclosure of Funding
We sincerely thank all the reviewers who have given us lots of valuable suggestions for the im- provement of our paper. This work was supported in part by the BUPT Excellent Ph.D. Students Foundation No. 1X203113111 and in part by scholarships from China Scholarship Council (CSC) under Grant CSC No.202206470048.
References
[1] F. Barsoum, C. Zhang, C. C. Ferrer, and 7. Zhang. Training deep networks for facial expression recognition with crowd-sourced label distribution. In Proceedings of the 18th ACM international conference on multimodal interaction, pages 279-283, 2016.
[2] K. Cao, Wei, A. Gaidon, N. Arechiga, and Ma. Learning imbalanced datasets with -ibe-diistiitution aware margin loss. Advances neural information processing systems, 32, 2019.
S. Chen, J. Wang. Y. Chen, Z. Shi, X. Geng, and Y. Rui. Label distribution learning on auxiliary label space graphs for facial expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[4] H.-P. Chou, S.-C. Chang, J.-Y. Pan, W. Wei, and D.-C. Juan. Remix: rebalanced mixup. Computer Vision-ECCC 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pages 95-1 10. Springer, 2020.
[5] Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie. Class balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9268-9277, 2019.
[6] I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee, al. Challenges representation learning: report on three machine learning contests. In Neural Information Processing: 20th International Conference, ICONIP 2013, Duegu, Korea, November 3-7, 2013. Proceedings, Part III 20, pages 117-124, Springer, 2013.
[7] H. Guo, K. Zheng, X. Fan, H. Yu, and S. Wang. Visual attention consistency under image transforms for multi-label image classification. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 729-739, 2019.
[8] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao. Ms-celeb-Im: A dataset and benchmark for large-scale face recognition. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 87-102. Springer, 2016.
[9] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[10] A. G. Howard, M. 7hu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andrectto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.0466, 2017.
[11] Y. Huang, F. Chen, Lx, and X. Wang. Facial expression recognition: A survey. Symmetry, 11(10):1189, 2019.
[12] Jiang and W. Deng. Boosting facial expression recogmition by semi-supervised progressive teacher. IEEE Transactions on Affective Computing, 2021.
[13] X. Jin, Lai, Jin. Learning dynamic relationships for facial expression recognition based on graph convolutional network. IEFE Transactions on Image Processing, 30:7 143-7155, 2021.
[14] T. Kanade, J. Cohn, and Y. Tian. Comprehensive database for facial expression analysis. Proceedings fourth IEEE international conference on automatic face and gesture recognition (cat. No. PR00580), pages 46 53, IEEE, 2000.
[15] B. Kang. S. Xie, M. Rohubach, Z. Yan, A. Gordo, J. Feng. and Y. Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv: 1910.09217, 2019.
[16] D. Kim and B. C. Song. Emotion-awee aware multi-view contrastive learning for facial emotion recognition. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 27, 2022, Proceedings, Part XIII, pages 178 195. Springer, 2022.
11

---

[17] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprini arXiv:1412.6980. 2014.
[18] J. Kumari, R. Rajesh, and K. Pooja. Facial expression recognition: A survey, Procedia computer science, 58:486 491, 2015.
[19] N. Le, K. Nguyen, Q. Tran, E. Tjiputra, B. Le, and A. Nguyen. Uncertainty-wware label distribution learning for facial expression recognition. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 6088 6097, 2023.
[20] H. Li, N. Wang. X. Ding, X. Yang, and X. Gao. Adaptively learning facial expression representation via cf labels and distillation. IEEE Transactions Image Processing, 30:2016-2028, 2021.
[21] H. Li, N. Wang. X. Yang, and X. Gao. Crs-cont: well-trained general encoder for facial expression analysis. IEEE Transactions on Image Processing, 31:4637-466 4650, 2022.
[22] H. Li, N. Wang, X. Yang, X. Wang, and X. Gao. Towards cemi-supervised deep facial expression recognition with adaptive confidence margin. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4166-4175, 2022.
[23] H. Li, N. Wang, X. Yang, X. Wang, and X. Gao. Unconstrained facial expression recognition with no-reference de-clements learning. IEEE Transactions on Affective Compuring. 2023.
[24] H. Li, N. Wang, Y. Yu, X. Yang, and X. Gao. Lban-il: A novel method of high discriminative representation for facial expression recognition. Neurocomputing, 432:159-169, 2021.
[25] S. Li and W. Deng. Deep facial expression recognition: survey. IEEE transactions on affective computing, 13():1195-1215, 2020.
[26] S. Li, W. Deng, and J. Du. Reliable crowdsoureing and deep locality-preserving learning for expression recognition the wild. Proceedings of the IEFE conference on computer vision and pattern recognition, pages 2852-2861, 2017.
2017.
[27] T.-Y. Lin, Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pages 1980-2988 2017.
[28] Z. Liu, Lin, Y. Cao, H. Hu, Wei, Zhang, S. Lin, and Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[29] T. Lukov, N. Zhao, G. H. Lee, and S.-N. Lim. Teaching with soft label smoothing for mitigating noisy labels in facial expressions. In Computer Vision 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XII, pages 648-665. Springer, 2022.
[30] M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba. Coding facial expressions with gabor wavelets. In Proceedings Third IEEE international conference automatic face and gesture recognition, pages 200-205. IEEE, 1998.
[31] D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Li, A. Bharambe, and L. Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pages 181-196, 2018
[32] A. Mollahosseini, B. Hasani, and M. H. Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1):18-31, 2017.
[33] K. Oksuz, B. C. Cam, S. Kalkan, and E. Akbas. Imbalance problems in object detection: A review. JEEE transactions on pattern analysis and machine intelligence, 43(10):33883445, 2020.
[34] A. Psaroudakis and D. Kollias. Mixaugment & mixup: Augmentation methods for facial expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 2367-2375, 2022.
[35] D. Ruan, Y. Yan. Lai, Z. Chai, C. Shen, and H. Wang. Feature decomposition and reconstruction learning for effective facial expression recognition. In Proceedings of the IEEE/CVF conference on computer vision and puttern recugnition, pages 7660-7669, 2021.
[36] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-basee based localization. In Proceedings of the IEEE international conference computer vision, pages 618 6266, 2017.
12

---

[37] J. She. Y. Hu, H. Shi, J. Wang, Q. Shen, and T. Mei. Dive into ambiguity: Latent distribution mining and pairwise uncertainty estimation for facial expression recognition. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 6248-6257 6257. 2021.
[38] L. Shen, Z. Lin, and Q. Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In Computer Vision- ECCo 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part VII 14, pages 467-482 Springer, 2016.
[39] Tan, C. Wang, B. Li, Q. Li, W. Ouyang. Yin, and Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 11662-11661 11671,111121 2020.
[40] M. Valstar, M. Pantic, et al. Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In Proc. 3rd Intern. Workshop on EMOTION (satellite of LREC): Corpora for Research Emotion and Affect, page 65. Paris, France., 2010.
[41] G. Van Horn, O. Mac Aodha, Y. Song. Y. Cui. C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 8769-8778, 2018.
[42] K. Wang, X. Peng. J. Yang. S. Lu, and Y. Qiao. Suppressing uncertainties for large-scat facial expression recognition. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6897-6906, 2020.
[43] K. Wang, X. Peng. J. Yang, D. Meng, and Y. Qiao. Region attention networks for pose and occlusion robust facial expression recognition. IEEE Transactions on Image Processing, 29:4057-444 -460077446,,,,, 2020.
[44] X. Wang, L. Lian. Z. Miao, Liu, and S. Yu. Long-tailed recognition by routing diverse distribution- aware experts. arXiv preprint arXiv:20/0.01809, 2020.
[45] F. Xue, Q Wang, and G. Guo. Transfer: Learning Itlation-awan facial expression representations with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3601-3610, 2021.
[46] D. Zeng, Z. Lin, X. Yan, Y. Liu, F. Wang, and B. Tang. Face2exp: Combating data biases for facial expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20291-20382 2022.
[47] K. Zhang, 7. Zhang, 7. Li, and Y. Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEFF signal processing letters, 1490-1139,,,,,,,, 2016.
[48] Y. Zhang, B. Hooi, L. Hong, and J. Feng. Test-agnostic long-tailed tailed recognition by test-time aggregating diverse experts with superviinn. arXiv e-prints, pages arXiv-2100 2021.
[49] Y. Zhang, C. Wang, and W. Deng. Relative uncertainty lcarning for facial expression recognition. Advances in Neural Information Processing Systems, 34:17616 2021.
[50] Y. Zhang, C. Wang. X. Ling, and W. Deng. Learn from all: Erasing attention consistency for noisy label facial expression recognition. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, pages 418-434. Springer, 2022.
[51] Z. Zhao, Q Liu, and F. Zhou. Robust lightweight facial expression recognition network with label distribution training. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 3510-3519, 2021.
[52] B. Zhou, Cui, X.-S Wei, and Z.-M. Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the TEEE/CVF conference on computer vision and pattern recognition, pages 9719-9728, 2020.
[53] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2016.
13


Reviews:
[{"review_id": "DjFR5HhQqQ", "reviewer": "Reviewer_Ey1e", "summary": "The paper presents an approach to tackle the imbalance problem in facial expression recognition. The approach makes use of information from the majority class to help improve performance of the minority classes. The proposed approach is evaluated on two public FER datasets. Different backbone networks are evaluated, along with comparisons to state of the art and ablation studies.", "questions": "How does the proposed approach handle extreme imbalance such as in AffectNet (e.g., Happy vs. Contempt)?\n\nWhat is reference [5] in Table 1? This is not a FER paper that is being compared to.", "limitations": "The limitations of the proposed work are not addressed. How does the approach work on extremely imbalanced data? What kind of societal impact can this have? There are always ethical concerns with FER systems.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper investigates a challenging problem. and the proposed approach using attention map consistency is interesting. Encouraging results are shown on two public datasets. Ablation study is generally well conducted.", "weaknesses": "Experimental design is lacking. FER2013 is an old dataset; newer FER datasets should be evalauted (e.g., AffectNet - Mollahosseini, Ali, Behzad Hasani, and Mohammad H. Mahoor. \"Affectnet: A database for facial expression, valence, and arousal computing in the wild.\" IEEE Transactions on Affective Computing 10.1 (2017): 18-31.). Along with being newer, AffectNet needs to be evaluated as the approach is on imbalanced FER datasets. The paper conducts experiments under different imbalance factors, however, AffectNet is one of the largest and most imbalanced datasets available for FER. For example, in AffectNet, Happy is ~46% of the data and contempt is ~1% of the data. How does the proposed approach handle these extreme imbalance issues?\n\nThe comparison to CB [5] in Table 1 is not clear. This paper is not on FER and does not evaluate RAF-DB. What is this?\n\nPaper states the accuracy of each class is not commonly shown in FER papers. This is not accurate. If it is not in a table, this is often shown through confusion matrices (e.g., Farzaneh, Amir Hossein, and Xiaojun Qi. \"Facial expression recognition in the wild via deep attentive center loss.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful review, which has assisted us in enhancing the quality of our paper.\n\n**Weaknesses:**\n\n**1.** Thanks for your valuable suggestion. We have conducted experiments on AffectNet with both 7 and 8 expression classes. Due to the time consumption, we use ResNet-18 as the backbone for all methods. As AffectNet has a balanced test set with 500 samples of each class, the mean accuracy equals the overall accuracy.\n\n|Method | Hap        | Neu       | Sad      | Ang       | Sur      | Fea     | Dis      | Mean Acc  | Overall Acc|\n|-------| :-:        | :-:       |  :-:     |   :-:     |   :-:    |   :-:   |  :-:     |  :-:      | :-:        |\n| SCN   | **95.20**  |**82.70**  | 44.20    | 56.30     | 35.80    | 38.00   | 20.90    | 53.30     | 53.30      |\n| BBN   |  87.00     | 57.10     | **66.80**| 58.30     | 54.90    | 71.10   | 30.10    | 60.76     | 60.76      |\n| RUL   | 90.50      | 62.40     | 64.70    | **69.30** | 60.80    | 49.00   | 34.20    |  61.56      | 61.56      |\n| EAC   | 91.40      | 64.50     | 65.70    | 66.30     |**61.60** | 60.90   | 45.80    | 65.17     | 65.17      |\n|Ours   | 86.20      | 59.00     | 64.20    | 66.50     | 57.80    |**64.50**|**61.90** | **65.73** |**65.73**   |\n\n  \n|Method | Hap        | Neu       | Sad     | Ang       | Sur      | Fea     | Dis     | Con     | Mean Acc  |Overall Acc|\n|-------| :-:        | :-:       |  :-:    |   :-:     |   :-:    |   :-:   |  :-:    | :-:     | :-:       | :-:       |\n| SCN   | **94.60**  | **74.90** | 58.20   | 63.80     | 40.90    | 43.20   | 30.80   |2.20     |  51.08    | 51.08     |\n| BBN   | 78.40      | 58.40     | 60.60   | **67.70** | 59.40    | 55.00   | 37.00   |46.70    |  57.90    | 57.90     |\n| RUL   | 71.00      | 63.40     | 46.60   | 54.90     | 53.70    | 58.60   | 44.70   |47.70    |  55.08    |  55.08    |\n| EAC   | 84.00      | 58.80     |**65.00**| 65.90     | **62.20**| 60.30   | 46.10   |41.90    | 60.53     | 60.53     |\n|Ours   | 78.60      | 54.30     | 63.80   | 59.50     | 57.60    |**64.10**|**59.40**|**60.00**| **62.16** |**62.16**  |\n\nFrom the results, we could draw the conclusion that our method achieves the best mean and overall accuracy on the test set under both 7 or 8 classes. Besides, we notice that our method improves existing methods on the minor classes of fear (Fea), disgust (Dis), contempt (Con) by remarkable margins, which illustrates that our method is more suitable for the imbalanced learning of FER task. Our method even achieves 60.00% accuracy on the contempt class under 8 classes. Furthermore, though \"Happy is ~46% of the data, and contempt is ~1% of the data,\" our method clearly decreases the test accuracy gap between happy (78.60%) and contempt (60.00%) compared with other methods under 8 classes, which means our method is fairer and achieves a more balanced test accuracy.\n\n**2.** The re-balanced weight design of our method is inspired by CB [5], which is a widely used method in imbalanced learning of image classification. We carry out experiments using [5] to show that simply weighing the cross-entropy loss cannot bring much improvement in the FER task, which illustrates the superiority of our method using re-balanced attention map consistency to mine extra minor-class information from all FER samples.\n\n**3.** We report the accuracy of each class and arrange them according to the sample number of each class instead of plotting confusion matrices due to space limitations and visualization simplicity. As suggested, we display the main results of our paper using confusion matrices in the uploaded PDF in the above \"global response\" for further reference.\n\n**Questions:**\n\nPlease refer to 1. and 2. in the weaknesses part.\n\n**Limitations:**\n\nThanks for your valuable suggestion. We study our method under extremely imbalanced data. For example, on RAF-DB, we only keep 10 training samples for both fear and disgust and keep other classes the same, leading to a extremely imbalanced train set summarized as below.\n\n|Class  | Hap   | Neu   | Sad     | Sur    | Ang   | Dis    | Fea  |\n|------ | :-:   | :-:   |  :-:    |   :-:  |   :-: |   :-:  |  :-: |\n|Number |4772   | 2524  | 1982    | 1290   | 705   | 10     | 10   |\n\nThe test accuracy on the original test set of RAF-DB is shown below.\n\n|Method   | Hap   | Neu   | Sad  | Sur     | Ang   | Dis    | Fea  | Overall Acc| Mean Acc|\n|------   | :-:   | :-:   |  :-: |   :-:   |   :-: |   :-:  |  :-: |  :-:       |  :-:    |\n|Baseline |96.03|**91.03**|87.66 | 88.15   |82.10  | 0.00   |0.00  |84.71       |   63.57 |\n|EAC      |96.54 |89.71|**90.79**|**90.58**|87.04  | 0.00   |0.00  |85.63       |   64.95 |\n|Ours     |**96.79**|89.71|90.17 | 90.27   |**87.65**|**7.50**|**1.35**|**86.05**|**66.21**|\n\nThe results illustrate that our method still outperforms the SOTA method EAC on extremely imbalanced data, mainly in the minor classes accuracy and the mean accuracy. However, the extreme case with happy having 4772 samples, which is around 500 times more than the samples for disgust and fear (only 10 samples each), also leads to a very low test accuracy of our method for disgust and fear. This should be considered a limitation of our method. As for the ethical concerns, we could further combine our method with the technology of differential privacy [1] to help preserve the privacy during recognition.\n\n[1] Privacy Preserving Face Recognition Utilizing Differential Privacy."}, {"review_id": "N8CcMqjQO6", "reviewer": "Reviewer_31BV", "summary": "\nIn the paper, the authors tackle the issue of imbalanced learning in the facial expression recognition task by introducing a fresh approach. Their proposed method revolves around the concept of attention consistency under spatial transforms, aiming to extract information about multiple classes from each sample. This approach effectively enhances the performance of under-represented classes. The authors evaluate their approach on two datasets and successfully achieve state-of-the-art performance with a transformer backbone.", "questions": "- What is the purpose of utilizing Global Average Pooling to extract features? Doesn't it result in a loss of information?\n\n- Based on the findings presented in Table 4, it appears that the performance improvement for minor classes surpasses that of other classes. Regarding the fairness aspect (although it may not be the primary focus of the paper), what can be said about the trained models? Do you believe that your method also facilitates the development of fairer models?", "limitations": "Given that it is a facial expression recognition task, privacy could potentially be a concern.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "(+) They achieve state-of-the-art (SOTA) performance on the RAF-DB dataset with the Swin backbone.\n\n(+) Table 4 clearly demonstrates the performance improvement brought by the proposed approach. It highlights the significant increase in performance for both minor classes and the overall score.\n\n(+) Despite attention consistency being proposed in previous work, the authors cleverly and novelly employ a modified version of it incorporating rebalanced smooth labeling. They provide a reasonable hypothesis, stating that information for minor classes can be captured from all training samples.\n\n(+) The visualization results presented in both section 4.7 and the appendix provide further evidence that the proposed approach functions effectively.", "weaknesses": "(-) To further support the idea, I suggest conducting evaluations on multi-label classification using other facial expression recognition (FER) datasets, such as BP4D. If the hypothesis is correct, the proposed approach should also work effectively in a different type of FER task. Testing it in this manner would provide a good opportunity for validation.\n\n(-) The two methods proposed in the paper, in terms of their structure, are not entirely novel. However, there is potential for further refinement and innovation in their design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful review. The strengths you highlighted align perfectly with the core contributions of our paper, and they indeed encapsulate what we take great pride in. Your review is undoubtedly one of the most exhilarating ones we have received in over a year. Thank you very much.\n\n**Weaknesses:**\n\n**1.** Thanks for your valuable suggestion. Though we wanted to carry out experiments on BP4D, we emailed the authors of BP4D and they told us they could not enter into an agreement for the dataset at this time and we could not download it. As suggested, we conduct experiments on the multi-label FER task. We utilize a widely used multi-label FER dataset EmotioNet [1, 2], which is published in CVPR and provides 6 basic expressions and 10 compound expressions. There are a total of 2,478 images and we utilize 80% of them for training and the rest for testing. The results of different methods are summarized below.\n\n|Method | Hap    | Hap Dis | Sad Dis | Sad   | Hap Sur  | Ang   | Sur  | Ang Sur  | Awe     | \n|-------| :-:    | :-:     |  :-:    | :-:   |    :-:   | :-:   | :-:  |  :-:     | :-:     |\n|SCN     | 82.24 |**71.74**|**80.00**|**66.67**| 0.00   | 35.71 | 30.77| 15.38    |**61.54**|\n|BBN     | 77.63 | 69.57   | 65.71   | 62.96 | 35.29    |**64.29**| 53.85| 7.69     | 53.85   |\n|RUL     | 75.66 | 62.32   | 62.86   | 59.26 | 35.29    |50.00  |**69.23**|46.15|53.85 |\n|EAC     | 78.29 | 69.57   | 68.57   |**66.67**|**70.59**| 57.14 | 46.15 | 46.15    | 46.15   |\n|Ours    |**86.18**| 65.94 | 68.57   | 55.56 |**70.59**| 57.14 | 46.15   |**53.85**| 46.15   |\n\n\n|Method | Dis     | Ang Dis | App   | Fea Ang  | Sad Ang | Fea  | Fea Sur| Overall Acc| Mean Acc|\n|-------| :-:    | :-:     |  :-:    | :-:   |    :-:   | :-:     | :-:  |  :-:     |:-:     |\n|SCN      | 8.33    | 0.00    | 0.00  |**30.00**|  0.00   |  0.00| 0.00 |  59.43   |  30.15 |\n|BBN      | 8.33    | 18.18   | 0.00  |**30.00**|**30.00**| 50.00| 0.00|  60.45   |  39.21 |\n|RUL      | 50.00   |**45.45**| 0.00  |**30.00**| 20.00   | 30.00| 0.00 |  59.43   |  43.13 |\n|EAC      |**66.67**| 27.27   | 0.00  |**30.00**| 20.00 | 40.00|**50.00**|  64.71   |  48.95 |\n|Ours     |**66.67**| 36.36   |**10.00**|**30.00**|**30.00**|**60.00**|**50.00**|**66.73**|  **52.07**| \n\nWe have organized the 16 different expression classes in [1, 2] into two tables based on the number of their training samples, arranged from largest to smallest.  There are instances of similar test accuracy scores when the test sample numbers for certain compound expressions are small, which can be attributed to the challenges associated with collecting samples for compound expressions. \n\nThe results demonstrate that our method consistently achieves the best overall and mean accuracy in the context of the multi-label FER task. Additionally, our method excels in terms of performance within the minor classes in the second table. The multi-label FER task is also aligned with our motivation, as the definition of compound expressions implies that a sample could encompass features from multiple basic expression classes. For instance, a sample in the fearfully angry (Fea Ang) class contains features from both the fear and angry classes.\n\n**2.** Thanks for bringing that up. Though the structure of our RAC and RSL modules is not entirely novel, their motivation is. We propose RAC to guide the model to mine extra information related to minor classes from all training samples for the first time. RSL acts as a complementary module for RAC and utilizes the extra information regarding the label distribution of the imbalanced training data to weigh minor classes more heavily. Both of the two modules learn existing extra knowledge to improve the imbalanced learning of FER, which aligns with the title of our paper.\n \n**Questions:**\n\n**1.** The features used for attention map consistency are extracted before the Global Average Pooling (GAP) layer to prevent information loss, as we employ F (features before GAP) instead of f (features after GAP) in Eq.(2) of our paper. The success of our method hinges on preserving the spatial information related to expression features since the attention map consistency also extends across the spatial dimensions.\n\n**2.** Yes, we believe our method achieves a fairer FER model. Though in Table 4, the performance improvement for minor classes sometimes surpasses that of other classes, the test accuracy of minor classes is still lower than the major classes. We believe a fair FER model should achieve similar accuracy on different expression classes. Based on the results in Table 4 and the results on AffectNet in the rebuttal for reviewer Ey1e, the gap between the accuracy of major and minor classes is distinctly smaller using our method. Thus, we believe our method achieves a fairer FER model.\n\n**Limitations:**\n\nThanks for your suggestion. We have added the limitation discussion in our revised manuscript. As there might be a privacy problem in FER, we could further combine our method with the technology of differential privacy [3] to help preserve privacy during FER.\n\n[1] Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild.\n\n[2] Emotionet challenge: Recognition of facial expressions of emotion in the wild.\n\n[3] Privacy Preserving Face Recognition Utilizing Differential Privacy."}, {"review_id": "uFYPNnu8XA", "reviewer": "Reviewer_Ux1i", "summary": "This paper focuses on the imbalanced learning problem in facial expression recognition (FER). Unlike existing works in imbalanced learning for image classification, the proposed method addresses imbalanced FER from a novel perspective of label distribution learning. Specifically, the proposed method is motivated by the observation that certain major classes in FER may contain valuable features for the minor classes. Building upon this observation, a re-balanced attention map consistency module is introduced, which encourages the model to extract useful information about the minor classes from both minor and major-class samples through attention map consistency. Additionally, a re-balanced label smooth module is incorporated to leverage prior knowledge of the training data distribution and regularize the classification loss. Experimental results on FER datasets with varying levels of imbalance and different backbone architectures demonstrate the effectiveness of the proposed method, particularly in improving the performance of the minor classes.", "questions": "I have some questions regarding the weight design of the proposed method. Since the weight is utilized by both of the proposed modules simultaneously, I think it is very important for the performance of the proposed method. Therefore, I am curious what the performance would be if we simply set the weight as the inverse of the sample numbers of each class? There should be more discussions regarding the weight design of this method.", "limitations": "The limitations of this paper should be discussed. This method is designed specifically for imbalanced learning in FER as FER has the label distribution learning characteristic. However, things might be different in other imbalanced image classification tasks. For example, minor classes in imbalanced CIFAR-100 might contain little common feature with major classes in imbalanced CIFAR-100.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper provides an interesting label distribution learning perspective to deal with the imbalanced FER, which is novel to me. It proposes two modules to mine extra information of minor-class samples from both major and minor-class samples. From my understanding, this method is specifically designed for FER and technical novelties are sound.\n\n2. The experimental demonstration well supports the claims. The authors have curated different FER datasets with various imbalance factors and the performance improvement on these imbalanced FER datasets is non-trivial compared with other SOTA methods, especially on the minor classes like fear and disgust. Ablation studies presented in Table 5 indicate that each of the proposed modules contributes to the overall performance. I also noticed that in Table 4, combined with Swin-T, the proposed method achieves remarkable results of 92.31% overall accuracy and 87.71% mean accuracy on the widely used RAF-DB dataset.\n\n3. The proposed method is easy to implement and the paper is written in an easy-to-understand manner. ", "weaknesses": "1. Lack of experiments. As illustrated by the authors, they utilize the weight (i.e., a hyper-parameter of 0.9999) introduced by Ref. [5] as the re-balanced weight for both the re-balanced attention maps and re-balanced smooth label. However, I think there should be some studies regards of this hyperparameter, and what if we utilize different values for the two proposed modules?\n\n2. Some claims were not supported by experiments. The authors claimed that \"EAC employs attention maps from all expression classes, rather than just the labeled class, which is similar to label distribution learning\". However, there is a lack of experiments to validate this claim. I believe the authors should include experiments to validate their point of label distribution learning, i.e., what is the performance if we only utilize the attention map of the labeled class instead of all classes?\n\n3. Comparative techniques were not sufficiently performed. For example, what if we use GradCAM instead of CAM to implement attention consistency? What if we utilize the basic label smooth instead of the proposed re-balanced label smooth?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thrilled to receive your review. We sincerely thank you for acknowledging the strengths of our work.\n\n**Weaknesses:**\n\n**1.** The re-balanced weight is inspired by the effective number of [5]. We visualize the re-balanced weight on the original RAF-DB according to different $\\beta$ values. The re-balanced weight is normalized, and the sum of the weight is 1. The standard deviation (Std) can reflect the distribution of the re-balanced weight. From the results in the Table, we can summarize that on  RAF-DB, when $\\beta$ = 0.9, the re-balanced weight follows the uniform distribution, while when $\\beta$ = 0.9999, the re-balanced weight has a large standard deviation and is similar to the inverse of the sample number. Notice that when $\\beta$ = 0.9, our method degrades to EAC plus traditional label smoothing. Thus, EAC plus label smoothing could be viewed as a special form of our method which is suitable for balanced train sets, while our method is a more general form with our introduced RAC and RSL modules. \n\n|$\\beta$| Sur        | Fea    | Dis    | Hap    | Sad    | Ang    | Neu    | Std    |\n|----------|------------|--------| -------|  ------|  ------|  ------| -------|------- |\n|0.9| 0.1429     | 0.1429 | 0.1429 | 0.1429 | 0.1429 | 0.1429 | 0.1429 | 0.0000 |\n|0.99| 0.1415     | 0.1505 | 0.1417 | 0.1415 | 0.1415 | 0.1417 | 0.1415 | 0.0031 |\n|0.999| 0.1091     | 0.3227 | 0.1545 | 0.0798 | 0.0917 | 0.1563 | 0.0860 | 0.0789 |\n|0.9999| 0.0959     | 0.4188 | 0.1677 | 0.0306 | 0.0645 | 0.1705 | 0.0520 | 0.1235 |\n|inverse| 0.0939     | 0.4310 | 0.1689 | 0.0254 | 0.0611 | 0.1718 | 0.0480 | 0.1290 |\n\nTo solve the imbalanced FER, following [5], we utilize a $\\beta$ value of 0.9999 across all our experiments. We also carry out experiments with different $\\beta$ values as suggested, and the results are shown below. The results show that using $\\beta$ as 0.9999 achieves the best results. Note that simply setting the weight as the inverse of the sample number can also yield good results, as its re-balanced weight is similar to the case when $\\beta$ = 0.9999. \n\n|Acc|0.9|0.99|0.999|0.9999|inverse| \n|--|--|--|--|--|--| \n|Overall|88.95|89.50|89.28|**89.77**|89.50|\n|Mean|81.18|81.24|81.75|**82.44**|82.02| \n\nWe utilize different values for the two modules as suggested. The first value is for the RAC and the second value is for the RSL. From the results, we observe a trend that with the increase of $\\beta$ from 0.9 to 0.9999, both the overall and mean accuracy increase. \n\n|Acc|0.9+0.9999|0.99+0.9999|0.999+0.9999|0.9999+0.9999|0.9999+0.999|0.9999+0.99|0.9999+0.9|\n|--|--|--|--|--|--|--|--| \n|Overall|88.92|89.41|89.57|**89.77**|89.67|89.63|89.47| \n|Mean|81.21|81.68|82.24|**82.44**|82.06|81.93|81.72|\n\n**2.** We carry out experiments with only the attention map of the labeled class instead of all classes to validate our claim of label distribution learning. The results are listed in the following table.\n\n|Method|Hap|Neu|Sad|Sur|Dis|Ang|Fea|Overall Acc|Mean Acc|\n|--|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|One map|95.02|**90.00**|87.66|85.71|57.50|**80.86**|63.51|88.30|80.04|\n|Ours|**96.37**|89.56|**89.33**|**87.84**|**66.89**|**80.86**|**66.22**|**89.77**|**82.44**|\n\nOur method outperforms the use of only one attention map of the label class on almost all cases. The results validate our claim that FER resembles label distribution learning as utilizing attention map consistency only on the label class fails to learn extra information from other classes on a given sample. \n\n**3.** We conduct experiments utilizing GradCAM instead of CAM to implement attention map consistency. The results are shown in the following table.\n\n|Method|Hap|Neu|Sad|Sur|Dis|Ang|Fea|Overall Acc|Mean Acc|\n|--|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|GradCAM|**96.46**|87.65|87.03|84.19|59.38|**82.72**|59.46|88.17|79.56|\n|Ours|96.37|**89.56**|**89.33**|**87.84**|**66.89**|80.86|**66.22**|**89.77**|**82.44**|\n\nThe performance of utilizing GradCAM is worse than using CAM. As stated in the GradCAM paper, GradCAM contains a ReLU function. We speculate that the ReLU function might diminish some useful information and thus degrades the regularization effect of the attention maps. Furthermore, if we want to get the attention maps of all classes using GradCAM, we need to backward the gradient many times and concatenate the attention map responding to each class, which is not as efficient as using CAM. Based on the better performance and the easier implementation, we choose to use CAM instead of GradCAM in our method.\n\nThe basic label smooth means the $\\beta$ for the label smooth is 0.9 and the $\\beta$ for attention map consistency is 0.9999 based on the above discussion. Thus, the overall accuracy is 89.47% and the mean accuracy is 81.72%, which are lower than our 89.77% and 82.44%, which illustrates the improvement of our RSL upon the basic label smooth. \n\n**Questions:**\n\nPlease refer to the 1. in the weaknesses part.\n\n**Limitations:**\n\nThanks for your suggestion. As our method is specifically designed for FER, it might not work well under datasets without the label distribution learning characteristic, we have added it to the limitation discussion of our revised manuscript."}, {"review_id": "S5yDhB203x", "reviewer": "Reviewer_N9Q5", "summary": "This paper mainly focuses on solving the imbalanced problem in Facial Expression Recognition (FER). The goal of this paper is to enhance the performance on minor classes without compromising the performance on major classes. The contribution is two fold. Re-balanced attention consistency (RAC) module is proposed to mines extra knowledge pertaining to minor classes from both major and minor samples. Re-balanced smooth labels (RSL) module is proposed to regulate the classification loss and promote balanced learning. Extensive experiments on different datasets and backbones validate the effectiveness of the proposed method.", "questions": "Same as the weakness section. ", "limitations": "The authors did not discuss the limitation in this paper. I suggest that the authors provide the \"maximum capability\" of the proposed method. In other words, what degree of imbalance will cause the proposed method to fail.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) This paper adapts attention map consistency to solve the imbalanced learning problem of FER for the first time, which proves that solving imbalanced FER through re-balanced strategy is promising.\n\n(2) The experiment including various backbones, imbalance and attention visualization is sufficient.", "weaknesses": "(1) The purpose of re-balanced attention consistency (RAC) is to extract balanced and transformation invariant knowledge of minor classes from all training samples. The word \"knowledge\" is unclear and should be more specific. Does it refer to high response values for specific regions on the facial feature map or something else?\n\n(2) The authors' explanation about the role of re-balanced smooth labels (RSL) is to improve performance to regulate the classification loss and promote balanced learning. This explanation is too abstract and difficult for readers to understand. Why is it designed as Eq.(8)? What do the two terms of Eq.(8) represent and why fusing these two terms works? I think the author should provide a more detailed explanation.\n\n(3) The proposed re-balanced attention consistency (RAC) is very similar to the attention consistency used in EAC. The difference is that an additional balance weight is used to re-weight the attention maps. I would like to know how much performance has been improved due to the addition of balance weight. I have looked at the ablation study in Table 5 and can only know the performance improvement of the entire RAC module.\n\n(4) The comparison in Table 1 is questionable. The overall accuracy of EAC is 88.01%, which does not match the number of 89.99% in the original EAC paper (See Table 6 in [46]).\n\n(5) In line 283, the authors think non-overlapping attention maps is better than overlapping ones. Are there any references supporting this viewpoint?\n\nOverall I think the paper is tackling the right problem. However, the unclear description of the proposed modules weakens the contribution of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your thorough review, which helps us a lot to improve our paper.\n\n**Weaknesses:**\n\n**1.** Yes, the knowledge pertains to attention regions on the feature map. The transformation invariant knowledge ensures the FER model to focus on the same region before and after the transformation, which improves accuracy according to [7, 46]. The novelty and motivation of our paper are rooted in our idea of extracting useful minor-class features across all training samples, instead of simply introducing attention map consistency for the imbalanced FER task.  \n\n**2.** The motivation for RSL is shown in Fig.2. By setting the smooth label inversely proportional to the sample number of each class, we create an imbalanced smooth label with larger smooth labels for minor classes compared to major classes. To achieve minimum loss during training, the FER model will predict more logits into the minor classes. As stated by Reviewer ss3S, this is akin to weighing the minor classes more heavily during training to improve the model's performance on them. The computation of Eq.(8) follows Fig.2 and is derived from Eq.(3) of label smooth in paper [1]. In Eq.(8), $\\tilde{y}(i,l)$ is the $l$-th class value of the re-balanced one-hot label of $x_{i}$. The first item represents the original value of the one-hot label with a weight of $(1-\\alpha)$. The second item is the smooth label introduced by our method, where $(\\alpha/L)$ is the weight for normalization. $B_{l}$ is our re-balanced weight. Eq.(8) does not represent a fusion of two terms; it is the computation of our introduced RSL. \n\n**3.** While the technical implementation of RAC is similar to EAC, the motivation and the resolved task are quite distinct. EAC focuses on preventing the model from remembering part of the features to mitigate label noise, while our RAC aims to mine extra information of minor classes from all samples to promote balanced learning. As we introduce the RSL module in our paper, to solely evaluate the effectiveness of the balance weight, we combine EAC with our RSL module as the comparison group. The results on RAF-DB show that our method outperforms EAC+RSL on both overall and mean accuracy.\n\n|Method|Hap|Neu|Sad|Sur|Dis|Ang|Fea|Overall Acc|Mean Acc|\n|--|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|EAC+RSL|95.95|88.24|88.91|86.63|65.63|79.63|63.51|88.92|81.21|\n|Ours|**96.37**|**89.56**|**89.33**|**87.84**|**66.89**|**80.86**|**66.22**|**89.77**|**82.44**|\n\nWe further show the results when the imbalance factor is 150. The results illustrate that our method also outperforms EAC+RSL in most cases.\n\n|Method|Hap|Neu|Sad|Sur|Dis|Ang|Fea|Overall Acc|Mean Acc|\n|--|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|EAC+RSL|**97.05**|**93.24**|77.20|78.72|**38.75**|59.88|29.73|84.52|67.80|\n|Ours|96.62|91.91|**79.29**|**83.89**|36.25|**61.11**|**43.24**|**85.20**|**70.33**|\n\n**4.** As EAC does not report the accuracy of each class and the mean accuracy, we could not cite their overall accuracy directly. We re-implemented their method strictly using their code and achieved 89.99% accuracy using their code. However, we noticed that their code is based on ResNet-50, while our Table 1 reports all results of different methods using ResNet-18 as the backbone. To ensure a fair comparison, we modified the pretrained ResNet-50 to pretrained ResNet-18. The best overall accuracy using ResNet-18 reached 88.85%, and the overall accuracy of the last epoch was reported as 88.01%. We believe reporting the best epoch accuracy may be considered cherry-picking, so we reported all the results using the accuracy of the last epoch. We have summarized both the last epoch accuracy and the best epoch overall accuracy in the following table. The results in the table show that our method outperforms EAC under almost all cases. Note that the best overall accuracy does not necessarily mean that the model achieves the best mean accuracy in the same epoch. In fact, the best mean accuracy of our proposed method reached 83.18% under ResNet-18.\n\n|Method|Hap|Neu|Sad|Sur|Dis|Ang|Fea|Overall Acc|Mean Acc|\n|--|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|EAC last|95.70|86.91|87.87|87.23|59.38|80.25|58.11|88.01|79.35|\n|Ours last|**96.37**|**89.56**|**89.33**|**87.84**|**66.89**|**80.86**|**66.22**|**89.77**|**82.44**|\n|EAC best|96.03|88.68|**90.17**|86.02|61.25|79.63|59.46|88.85|80.18|\n|Ours best|**96.54**|**89.26**|89.75|**88.15**|**65.63**|**81.48**|**64.86**|**89.80**|**82.24**|\n\n**5.** In Fig. 4, the non-overlapping attention maps are from the happy and sad. These classes shouldn't exhibit similar features. However, other classes might be different, where the rationale comes from the definition of compound expressions [2-3]. For instance, the compound expression \"fearfully surprised\" exists, leading to similar features between these two classes and resulting in overlapping attention maps.\n\n**Limitations:**\n\nWe carry out experiments with an extreme case of only 10 samples of both disgust and fear classes. We refer the reviewer to reviewer Ey1e for the summary of training sample numbers and baseline performance due to space limitations. The results illustrate that our method outperforms the SOTA method EAC, mainly in the minor-class accuracy and the mean accuracy. Note that happy (4772 samples) is around 500 times more than the samples of disgust and fear, which leads to low test accuracy for disgust and fear. We think this degree of imbalance might be considered a limitation.\n\n|Method|Hap|Neu|Sad|Sur|Ang|Dis|Fea|Overall Acc|Mean Acc|\n|--|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|EAC|96.54|**89.71**|**90.79**|**90.58**|87.04|0.00|0.00|85.63|64.95|\n|Ours|**96.79**|**89.71**|90.17|90.27|**87.65**|**7.50**|**1.35**|**86.05**|**66.21**|\n\n[1] The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration. In CVPR.\n\n[2] Sixteen facial expressions occur in similar contexts worldwide. In Nature.\n\n[3] Multi-Label Compound Expression Recognition: C-EXPR Database & Network. In CVPR. "}, {"review_id": "xP7RpZEMOo", "reviewer": "Reviewer_ss3S", "summary": "This paper deals with the topic of facial  expression recognition (FER). In particular, authors point out the problem of imbalanced class data since most FER data sets will have many more neutral or happy face images than images with other facial expressions.  Authors propose an approach to address this problem building on two ideas. First is that there may be information to be learnt about the minor classes even from the samples from the major classes.   Based on this observation, authors propose a novel attention map rebalancing to regularize the model. As the second idea, they  introduce a label smoothing approach that weights the minor classes more than the major classes. Results  on RAF-DB and FERPlus facial expression image data sets show that the proposed approach is able to outperform state of the art FER approaches.", "questions": "1. What is the difference between mean accuracy and overall accuracy?\n\n2. Line 106: How do we know that the two types of information are \"orthogonal\"? Perhaps, characterizing them as \"complementary\" might be more appropriate.\n\n3. Fig. 1 caption: \"--- while do not degrade the high accuracy on major classes\" should be \"--- while not degrading the high accuracy on major classes\"\n\n4. Fig. 2: I assume that the bar corresponding to 0.6 probability is for neutral images --- that needs to be clearly indicated in the figure.\n\n5. Eq. (7): Is the tilde on y() here being used to denote the flipped samples or something else?\n\n6. Shouldn't there be bold-face numbers in every column of Table 1 and other tables?", "limitations": "There is no discussion of  limitations in this submission. \n\nAuthors may want to add some discussion of the  performances of the proposed approaches based on gender, race and/or age considerations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main idea of learning about the minor classes from training samples in both major and minor classes appears to be novel and is one of the strengths of this paper. The idea of weighting the minor classes more heavily or using smoothed labels have been used in other applications before, but perhaps not in FER.\n\nAnother strength of this paper is the  set of numerical results provided. Both data sets being used are state of the art data sets and results clearly indicate that the proposed method outperforms competing methods for FER.\n\nAnother strength of the paper is that it is relatively well-written although authors should have more clearly explained the difference between \"overall accuracy\" and \"mean accuracy\".", "weaknesses": "In Fig. 1, flipping of the images is shown to introduce the notion of re-balanced attention consistency. It is not clear if flipping is the only transformation that makes sense for the FER or if some other transformations (e.g., scaling, intensity attenuation or gain) make sense. That needs to be discussed in more detail.\n\nAnother weakness is that the proposed method is somewhat similar to the EAC approach although authors do a good job of clarifying the similarities and differences between their approach and EAC.\n\nThe main assertion that there is information to be gained about minor classes from samples in major classes seems to mainly backed up by pointing to the similarities in corresponding attention maps (e.g., open mouth regions for fear and surprise categories). This evidence is subjective and not entirely convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very exhilarated to receive your review. We sincerely thank you for your professional review, which gives us many instructions for polishing our paper. Thanks very much.\n\n**Weaknesses:**\n\n**1.** Thanks for your valuable suggestion. We carried out experiments to study the performance of using other transformations. The results are displayed below.\n\n|Method|Hap|Neu|Sad|Sur|Dis|Ang|Fea|Overall Acc|Mean Acc|\n|--|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Intensity|95.02|86.47|82.01|82.98|63.13|72.84|56.76|86.05|77.03|\n|Scaling|95.78|**91.91**|84.31|85.41|**75.00**|**82.10**|60.81|89.37|82.19|\n|Ours|**96.37**|89.56|**89.33**|**87.84**|66.89|80.86|**66.22**|**89.77**|**82.44**|\n\nThe results illustrate that intensity transformation performs poorly, while scaling performs well, which almost surpasses our method. The reason lies in that attention map consistency regularizes the model to focus on the same regions before and after the transformation, which incorporates spatial information as the attention map in our method has height and width dimensions. Thus, the transformation should be spatial-related transformation to maximize the function of the method. \n\n**2.** Although the technical details of our RAC module are similar to EAC, the motivation and the task are different. EAC aims to prevent the model from memorizing part of the features to handle label noises, while our RAC aims to mine extra information related to minor classes from all training samples to solve imbalanced learning. We also introduce an RSL module to further enhance performance. Experiment results show that our method distinctly outperforms EAC under the imbalanced learning task.\n\n**3.** Thanks for bringing that up. The evidence is somewhat subjective since facial expression recognition can be subjective and context-dependent, unlike other classification tasks with clear-out boundaries like CIFAR-10. For instance, various annotators of facial expression datasets might yield diverse results for the same image. Our assertion is also supported by [1-3], all of which validate that expression is a continuous statement, implying that a sample could contain information from several classes. Notably, the dataset in [3] comprises expressions such as sadly fearful, sadly disgusted, fearfully surprised, and more. This aligns with our research, where we strive to extract extra information related to minor classes like fear and disgust from major classes like surprise and sadness. Moreover, our Supp. material contains additional visualization results to provide a more intuitive comprehension. We conducted experiments to support our claim. During training, we employed attention map consistency solely on the label class, rather than all classes, for each sample. The results are presented below.\n \n|Method|Hap|Neu|Sad|Sur|Dis|Ang|Fea|Overall Acc|Mean Acc|\n|--|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|One map|95.02|**90.00**|87.66|85.71|57.50|**80.86**|63.51|88.30|80.04|\n|Ours|**96.37**|89.56|**89.33**|**87.84**|**66.89**|**80.86**|**66.22**|**89.77**|**82.44**|\n\nOur method outperforms the comparison group in almost all cases. The results validate our claim that FER resembles label distribution learning, as solely utilizing attention map consistency on the label class fails to capture additional information from other classes within each given sample. \n\n**Questions:**\n\n**1.** Mean accuracy is calculated as the average value of the test accuracy for each class. Overall accuracy is calculated as the test accuracy for the entire test set. Mean accuracy equals overall accuracy when the test set is balanced, meaning each class has the same number of samples. Both RAF-DB and FERPlus have imbalanced test sets, implying the model could easily achieve high overall accuracy by correctly classifying major classes such as happy. This is unsatisfactory as understanding the minor classes like fear and disgust with negative human emotions is equally crucial.\n\n**2.** Thanks for your valuable advice. We use the term \"orthogonal supplement\" to convey that previous works primarily address the noise in FER datasets. However, label noises are not connected to the varying sample numbers of each class, which causes the imbalanced learning problem. Nevertheless, your advice to use \"complementary\" is indeed a superior choice, and we have revised our manuscript.\n\n**3.** We appreciate your suggestion. We have corrected the grammar mistake and conducted proofreading to polish our writing.\n\n**4.** Yes, you're right. We have indicated the neutral class in the new version of Fig. 2 and added it to our manuscript.\n\n**5.** The tilde on y() signifies the re-balanced smooth label, distinct from the original label y(). To enhance clarity, we have replaced tilde y() with y_smooth().\n\n**6.** As we mainly focus on the mean accuracy and the accuracy of minor classes, we did not use bold-face numbers in every column. According to your advice, we used bold-face numbers in every column of our tables and marked the mean accuracy and the accuracy of minor classes using a different color.\n\n**Limitations:**\n\nWe added the limitation discussion regarding gender, race, and age to our revised manuscript. Societal and cultural factors might lead to distinct expression patterns, which could impact the model's ability to generalize across genders and races. Things are similar regarding the age factor as facial expressions evolve over time due to muscle tone changes and aging effects.\n\n[1] Sixteen facial expressions occur in similar contexts worldwide. In Nature.\n\n[2] Multi-Dimensional, Nuanced and Subjective-Measuring the Perception of Facial Expressions. In CVPR.\n\n[3] Multi-Label Compound Expression Recognition: C-EXPR Database & Network. In CVPR. "}]

Author rebuttal:
We sincerely thank all reviewers for their time and effort in reviewing our paper, as well as for their insightful feedback!

We are encouraged that Reviewers 31BV, Ux1i, and ss3S all find our main idea of mining extra knowledge about minor classes from training samples of both major and minor classes to be novel. Reviewer 31BV thinks that we "cleverly and innovatively" designed our method, and Reviewer Ux1i also believes that our method is specifically designed for FER and the technical innovations are well-founded. Reviewers Ey1e and N9Q5 assert that we are addressing a challenging problem, and that our method is interesting and holds promise in solving imbalanced FER through a re-balanced strategy. Moreover, Reviewers ss3S, Ux1i, and 31BV all concur that our method achieved state-of-the-art performance on the RAF-DB dataset with the Swin backbone, and our results "clearly indicate that the proposed method outperforms competing methods" under the imbalanced learning FER task, particularly in terms of the accuracy of minor classes. Reviewers Ux1i and N9Q5 also note the thoroughness of the experiments in our paper. Reviewers ss3S and Ux1i remark that Table 4 clearly demonstrates the performance improvement brought about by the proposed approach, and that our paper is well-written.


Addressing the weaknesses of our paper, Reviewer Ey1e suggested that we add experiments on AffectNet. Following this suggestion, we conducted experiments and found that our method also outperforms other FER methods on this extremely imbalanced dataset. Reviewer 31BV suggested evaluating our method under multi-label FER task. We conducted experiments as suggested and found that our method still performs well. Multi-label FER task aligns with our statement that a sample might contain features from several expression classes. For instance, the class "fearfully angry" combines features from both the fear and angry classes. Reviewer Ux1i primarily raised concerns about the hyper-parameter study, prompting us to add the corresponding experiments. Reviewer N9Q5 pointed out some unclear descriptions regarding our proposed modules, so we added more detailed discussions of our RAC and RSL modules. We greatly appreciate the extensive feedback provided by Reviewer ss3S, which has significantly aided in improving the quality of our paper. Additionally, Reviewer ss3S suggested exploring other transformations, leading us to include corresponding experiments as suggested.

We address specific questions below and respond to each review with a separate response in detail. We have incorporated all feedback into the revised version of our manuscript and supplementary material. With our responses, we hope that we have addressed the reviewers' concerns. Please let us know if there are any further questions or comments. We are eager to do our utmost to address them.

Meta-review: While attention consistency and label smoothing (Hinton et al.) have been widely studied in the literature (e.g. in the EAC work), this paper has introduced some novel modifications, such as a re-balanced weight based on the effective number in [5]. The modified techniques, when combined as a whole, performs effectively for facial expression recognition tasks on both major and minor classes. All reviewers are positive on this submission. The AC agrees with them and suggests accepting this submission.
Decision: Accept (poster)