Optimal Transport-based Identity Matching for Identity-invariant Facial Expression Recognition
Authors: Daeha Kim
Byung Cheol Song
Venue: NeurIPS 2022 Accept (2022)
Keywords: Facial Expression Recognition
optimal transport
identity matching
identity-invarient

Abstract: Identity-invariant facial expression recognition (FER) has been one of the challenging computer vision tasks. Since conventional FER schemes do not explicitly address the inter-identity variation of facial expressions, their neural network models still operate depending on facial identity. This paper proposes to quantify the inter-identity variation by utilizing pairs of similar expressions explored through a specific matching process. We formulate the identity matching process as an Optimal Transport (OT) problem. Specifically, to find pairs of similar expressions from different identities, we define the inter-feature similarity as a transportation cost. Then, optimal identity matching to find the optimal flow with minimum transportation cost is performed by Sinkhorn-Knopp iteration. The proposed matching method is not only easy to plug in to other models, but also requires only acceptable computational overhead. Extensive simulations prove that the proposed FER method improves the PCC/CCC performance by up to 10% or more compared to the runner-up on wild datasets. The source code and software demo are available at https://github.com/kdhht2334/ELIM_FER.

Optimal Transport-based Identity Matching for Identity-inee innariint Facial Expression Recognition
Daeha Kim Inha University kdhht:50200maiil.. com
Byung Cheol Song
Inha University besong@inha. ac kr
Abstract
(dentity-invarian facial expression recognition (FER) has been one the chal- lenging computer vision tasks. Since conventional FER schemes do not explicitly address the inter-identity variation of facial expressions, their neural network mod- els still operate depending facial identity. This paper proposes to quantify the inter-identity variation by utilizing pairs of similar expressions explored through specific matching process, We formulate the identity matching process Optimal Transport (OT) problem. Specifically, to find pairs of similar expressions from different identities, define the inter-feature similarity cost. Then, optimal identity matching to find the optimal flow with minimum trans- portation cost performed by Sinkhorn-Knopp iteration. The proposed matching method is not only easy to plug in to other models, but also requires only accept- able computational overhead. Extensive simulations prove that the proposed FER method improves the PCC/CCC performance by up to 10% or more compared to the runner-up on wild datasets. The source code and software demo are available https: / thub. com cwm/kdhh w233h2 111/33//EIII BEEIMEIII._EEEREEE
transportation
Introduction
For sophisticated human-computer interaction (HCI), fa- cial expression recognition (FER) must be able accu- rately predict the emotions of people with different per- sonalities or expressive styles. On the other hand, to ex- press the type and intensity of emotions in detail, VA FER, which annotates facial expressions in continuous domain based on two axes of Valence (V) and Arousal (A), is at- tracting attention [36]. Here, valence and arousal indicate the positive/negative of emotions and the degree of activa- tion, respectively. Recently, many techniques for analyzing complex fine-grad fine-grained expressions have been developed to improve the regression or prediction performance of VA FER [16, 27, 17]. However, it is still challenging properly learn facial expressions that differ from person to person due to heterogeneity in identity (ID) attributes such as age, gender, and culture.
Weeight for FER learning
Identity
matching
ID pairs
Samples ID 1
Samples of ID 2
learning
Figure 1: Conceptual illustration of ELIM. ELIM utilizes identity pairs that advantageous for FER learning.
Thus, so-called ID-aware FER is receiving lot of attention nowadays. For example, [22, 2] learned emotional factors inherent strong facial expression samples through adversarial learning, and generated ID samples favorable to FER through GAN [14]. However, it is observed that conventional models have difficulty learning the unique properties of facial expressions inherent in each ID, and above all, they are still dependent on the ID change.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

We believe that the inter-ID ID variation of facial expressions is the key to solving this problem. To analyze the inter-ID variation, we propose a novel FER learning based on pair-wise similarity by finding pairs of similar expressions from different IDs. This idea was inspired by cognitive research [6, 48, 18] that analyzes facial expressions through comparison between IDs. This paper realizes our idea through a novel framework named Expression Learning with Identity Matching (ELIM). Specifically, ELIM finds ID pairs with similar expressions through ID matching based on the optimal transport (OT) problem [8], and assigns relevance weights according to the similarity of ID pairs (see Fig. 1). Next, based on the relevance weights, we compute and compare the tendency of differences in facial expressions between IDs, ie.. the ID shift of facial expressions. As discussed in Sec. 4, ID-invariant representations are generated by ID shift vectors quantified from ID pairs. Finally, FER learning is performed through risk minimization [43].
Contributions of this paper are summarized as follows: (I) To realize ID-invariant FER, we propose an FER framework ELIM that adaptively considers inter-ID variation of facial expressions. Specifically, ELIM learns facial expressions based on the similarity (or dissimilarity) between IDs obtained through ID matching (see Fig. 4). (II) The state-of-the thte--r-t (SOTA) performance of ELIM is verified through extensive experiments on various real-world datasets. Especially, ELIM works more accurately compared to prior arts even for samples in which inconsistency between facial expressions and the emotion label predictions exists (see Fig. 3).
2 Related Work
Valence-Arousal (VA) FER is approach based on the circumplex model [36] in which emotions are distributed in a two-dimensional (2D) circular space. VA FER has a wide research spectrum, from formulating mapping between features and continuous VA labels [17] to mapping complex and high-dimensional facial expression information on VA space [27. 37J. Recently, some interesting studies have been reported the so-called ID-invariant FER, which is designed to be robust to even unseen IDs. For example, [2] introduced a generative model to generate and analyze similar expression examples from different IDs. However, [2] is fundamentally dependent the performance of the generative model, and its performance is limited because only a single pair is compared to analyze the expression difference between IDs. From another point of view, several FER methods tried to learn intra-identity discrepancy of samples with different facial expressions [47, 39J. However, since they do not explicitly consider inter-ID variations, they inevitably suffer from performance degradation for IDs that was never learned [41]. [5] proposed personalized affective module based on an adversarial autoencoder to learn features and generate samples having different expressions. However, like [47, 391, this VA FER method also shows limited performance for unseen target ID. On the other hand, since ELIM reflects the relative traits of expression between IDs in model learning. it can achieve reliable performance even on the test dataset including unseen IDs.
Distribution shift has been treated very important not only in FER but also in various machine learning tasks. For example, distribution shifts between set of source domains and unseen target domains [32, 49] are seriously considered in the field of out-of-domain (OoD) generalization [34, 15]. In particular. normalization methods [52, 12] that generate domain-invariant representations by quantifying distribution shifts through domain-specific statistic are being actively studied in the OoD generalization field. ELIM is differentiated from conventional normalization methods [52, 12] in that defines statistics from the OT problem that automatically matches pairs of similar facial expressions. Note that ELIM is the first case applying the distribution shift concept to the field of VA FER (cf. Problem 1). Specifically, ELIM regards IID a domain and quantifies ID shift vectors. By normalizing features with the corresponding ID vectors, ID-invariant representations can be generated.
3 Preliminaries
This section describes the problem formulation of ELIM and briefly reviews the OT problem respon- sible for optimal resource allocation.
3.1 Problem Formulation with Nomenclature
This paper focuses on the regression task to estimate VA labels y(E R² C y from features =(E Rd) C. Z. Here, y and Z stand for the label and feature spaces, respectively, The VA FER task

---

generally assumes that labels depend on facial expressions rather than IDs. However, this assumption does not always hold due ID shift, i... a phenomenon in which (marginal) distributions of IDs are shifted by their attributes even though the label prediction mechanism is the same: p² (y) 2) 1² (y)=) and p² (2) & (z)) Let p¹ and p² denote distributions of two different IDs. We aim find a predictor f: 2 + y that works robustly even when label predictions are inconsistent with the corresponding facial expressions. Formally,
Problem 1. Let Z Jecsuppte))) be the training-puu purppose feature set collected from multiple IDs.
The corresponding VA label set Y defined similarly. Eir an index set of training environments and supp(-) stands for its support set. The cardinality of Etr is N. Then, the risk of predictor f for ID is formulated as follows:
arg min max eEsupp(Err)
ES (C(((ZZ))
where L is loss function based mean-squared error (MSE).
Note that the goal of Problem is to minimize the empirical risk [43] of f from as many IDs as possible. To minmmize the risk of Eq. derived to generalize ID shifts, we design a model to learn representations of facial expressions invariant to ID shifts (ef. Sec. 4.2).
3.2 Review of Optimal Transport
This section briefly review the OT problem that is being applied to various real-world applications such as speech signal processing and biomedical technology [30, 46]. In this paper, the OT problem is applied to the VA FER task for a similar purpose to the above studies. OT is known as tool for measuring the optimal distance between two distribution sets. For example, suppose you are transporting product from a set of suppliers m]. Also, assume that the i-th supplier a, has some unit(s) of products, and = (b = 1,2,
[a 1,2, ... n) to set of demanders
the j-th demander by requires some unit(s) of products. Here, a, and b mean the number of product(s) in double sense. The cost per unit transported from supplier to demander is Ci, E RTx" and the number of units transported is Xi, RRxm The objective of the OT problem is to search the cheapest flow X = /i = 1, ..... n, = 1, m) that can transport all products from suppliers to demanders at the minimum cost:
OT(a, b) =min Σ
Cij
X
W
(2)
s.L.
Xij =bj, Σ
Xij
0i
In order find the solution X* of the above problem, can employ linear programming tool. However, since this tool does not necessarily provide a unique solution and it requires near- cubic complexity [1], it is difficult to apply through a fast iteration solution, i.e., Sinkhorn Knopp [8] (cf. Appendix A). Sinkhorn -nopp method based on the matrix scaling algorithm can obtain the unique optimal solution X with pear-square complexity [3]. Since this method computes X by aggregating the importance between all features, can analyze both global and local distributions of data. In this paper, we define a; and by from the features of each ID to capture the global representation of facial expressions, and perform the ID matching process through the Sinkhorn- -unhorr-KKoop method.
nigh-dimensional features. So, we address this issue
Proposed Method: ELIM
This section proposes Expression Learning with Identity Matching (ELIM) for VA FER. ELIM consists of Munkoom-Knopp-basee based ID matching and ID-invariant expression learning.
Pre-processing, Since ELIM regards ID as a domain, samples in the mini-batch need to be grouped randomly selected among N IDs in the mini-batch and the remaining N IDs are our target. Then, feature set Ze is constructed by the backbone (Jo) [29, 19, 40] on an ID basis. Next, ID matching process is performed between each target ID and the reference ID, thereby ID pairs corresponding the target ID are constructed.
according to ID. One reference
3

---

Reference ID having samples
Identity matching via Sinkhorn
Feature of ref. ID
R1
Feature target ID Sinkhorm-Knopp
%
Inference
I
Weights sharing
Loss
Valence
f
Arousal
Total N IDs
Target ID having m samples
b
by
Prns
Figure 2: Overview of the ELIM. Among the total N IDs in the mini-batch, reference ID, i.c.. e1 randomly chosen. The rest become target IDs e2,....N .N. For convenience, only one target ID e2 is considered here. First, fo encodes feature sets. The ID matching cost is defined from the feature sets, and then node weights a; and by are computed. Next, optimal matching flows having the minimum matching cost are generated by Sinkhorn- Knopp iteration. Finally, the ID shift vector µ corresponding to the target ID is derived. is learned generate ID-invariant representations by receiving n and 2 as inputs. During inference, only fo and fo are used without additional cost.
4.1 Identity Matching via Sinkhorn-Knopp Iteration
Early ID matching approach focused on calculating differences between features based on distance metries such as L2-norm and contrastive loss [20, 22]. However, since conventional methods based on inter-sample matching can quantify even differences in attributes (e.g., hair, skin, etc.) unrelated to facial expressions, they cannot be free from the influence of ID. In other words, previous works are not robust to expression differences between IDs, i.e., inter-ID variation. To overcome this drawback, even inter-ID variation should be learned. To derive the information about inter-ID variation, reliable ID pairs are required. To acquire reliable ID pairs, we match samples from different IDs by using Sinkhorn Kinkhorn-Knopp iteration [8] which can consider both global and local distributions of samples.
As in Fig. 2, this section uses only a reference ID e and target ID as an example for convenience. Let their feature sets be ze Rdxx and z E Rdxm, respectively. Here, the cardinality of Ze: and = is n and m, respectively. First, matching costs and relevance weights are generated from 2e and n The cost matrix C defined based on cosine similarity that a pair of features with (="1)²z² 22
similar expression has a smaller matching cost: Ct.j
2 Here, ||-|| indicates 1.2-norm.
Then, relevance weights a, and by are computed capture the global representation inherent each ID) (cf. Sec. 4.3). Using the relevance weights, the Sinkhorn- Knopp iteration computes the optimal flow matrix X* (ef. Appendix A). As result, based on C' and X", the ID shift of the target ID) calculated as follows:
n
CiyXig j= 1,2, ,, Σ
(3)
n is a weighted cosine distance with the optimal transportation plan X* between the j-th target sample and the reference samples weight. ID shift vectors ~1² ....... n of all target IDs computed in the same way are used for feature normalization process of the next section.
4.2 Identity-Invariant Expression Learning
This section describes the process of generating and learning the ID-invariant representation of the target feature set Z using the ID shift vector fº. First, inspired by several normalization techniques [52, 12] that perform domain-specific adaptation generalization of features, we define ID-invariant representation by using fr8 and af as shifting and scaling parameters, respectively:
2e
Ve E {ez,... .,n),
(4)
Ed ( j 1.... m. Since μt and 8º captured from a specific ID are used, the normalization of Eq. 4 can remove ID ID-specific information from the model. Next, the normalized
where of

---

d-dimensional feature set 2 is mapped onto the 2D VA space:
y fo(2))=>'
3, Ve ( cNN]:
(5)
where 0 = ( E Rdx2 3 ]2 Affine parameter is trained to adapt to the normalized features of target IDs in fo of Eq. 5, which behaves similarly to adaptive instance normalization [21]. That is, a given model is encouraged to learn an ID-invariant representation [31, 45, 12]. Finally, the risk loss [43] between ÿ'e and the VA label set Ye is calculated:
== min 0.00
Σ (Es)
) (fo(29) ye)
Note that Eq. 6, which regresses predictions and labels in terms of MSE, calculates empirical risk loss through Monte-Carlo sampling.
Algorithm 1 Training Procedure of ELIM
Require: # IDs N, learning rate Ir, initialize Lmain to zero, initialize parameters (Ø, 0) to Normal
1: while not converge (ø, 8) do
2: 2 + ID grouping and feature encoding via fø. Ve E {ei, ; ,e
3: for = 2....... .N do
4:
_ 4- ID matching and ID shift vector calculation (Eq. 3)
5:
za: "yy t Normalization and mapping into VA space (Eqs. 4 and 5)
6:
Lmain := Lmain + MSE(Y y + Update the main loss (Eq. 6)
7:
end for
8:
Liotel = Lmain + Lee + Calculate the total loss
9:
(ø, 0) == (tb, 0) V (0.0) Litatat f- Update learnable model parameters (d) 0)
10:
end while
Training procedure. Algorithm 1 describes the details of the losses of ELIM calculated every step. Reference feature set ze and target feature 7* quantify ID shift vectors p = through ID matching. ID-invariant representations are generated from these vectors and 2e2..., and then the empirical risk loss Lmain is calculated in the VA space. With ev [50] calculated by MSE between unnormalized feature set 2 and label Y", parameters (ó, f) are updated until convergence.
4.3 Relevance Weight Design
Since the matching flow X is determined based relevance weights, i.e., and b, the design of the relevance weights is important for effective ID matching. This section describes the design process with the aforementioned n dxx and n Rdxm examples.
Sampling reliable features. a reference ID has weak facial expression, the difference in facial expression with other IDs will not be noticeable, which indicates the difficulty in ID) matching. In other words, the feature such a reference ID), i.e., = is not suitable for matching. Therefore, we set the norm 1y of the label corresponding to z" to the sampling score of =t i = 1, ................................ n. = corresponding to top elements terms of π are sampled. However, since the sampling of z1 is non-differentiable, we employ re-parameterized gradient estimator with the Gumbel-topk [26]:
1I/(, !ly I),
g g=log Gumbel 1)
Retrieving the k({ n)) largest values topk(g1, 9n.) will give i samples without replacement. Finally, Ze, Z _e Raxk' retrieved in the same way used calculate relevance weights.
Computing relevance weights. If weights are generated only from features of either reference ID target ID, it will be difficult to capture the global representation of facial expression. Instead, if aggregated information to find ID pairs that are advantageous for FER. This be proven experimentally (cf. Table 4). Inspired by the feature representation of a few-shot classification task [38], we use the average of z support of "1 to calculate the reference ID's relevance weight
features between IDs used for weight calculation, it will become casier
à
172
a;
8.1. à
2
i= 1,2,
,,,,
(8)
a
m
5

---

where F4, indicates the clipping of negative value(s). Another relevance weight b of the target ID is calculated similarly. The effect of the proposed relevance weight will be analyzed Sec. 5.4.
Experiments
5.1 Datasets and Implementation Details
Datasets. We adopted public databases only for research purposes, and informed consent was obtained if necessary. AFEW-VA [28] derived from the AFEW dataset [10] consists of about 600 short video clips annotated with VA labels frame by frame. Evaluation was performed through cross validation at ratio of 5:1. Aff- wild [50] is the first large-scale in -n-the-wild VA dataset that collects the reactions of people who watch movies or TV shows. Aff-wild with 298 videos amounts to over 30 hours in length, whose number of frames is about 1.18M. Since the test data of Aff-wild was not disclosed, this paper adopted the sampled train set for evaluation purpose in the same way as previous works [17, 22]. Aff-wild2, which added videos of spontaneous facial behaviors
Aff-wild, was also released [25]. This database consists of about 1,4IM frames. Since subjects of different ethnicity and age groups newly added, Aff-wild2 can further enhance the reliability of FER performance.
Configurations. All models were implemented in PyTorc, and the following experiments were performed on Intel Xeon CPU and NVIDIA RTX A6000 GPU. Each experiment five times. The light- weighte AlexNet (AL-tuned) [29]. ResNet18 (R18) [19], and MIp-Mixer (MMx) [40] were chosen fa. respectively. The mini-bateh size for those models was set to 512, 256, and 128, respectively (cf. Appendix B for model details). Learnable parameters (Ø, 0) were optimized with Adam optimizer [23]. The learning rate (LR) of Φ and @ set 5e-5 and Ie-5, respectively. LR decreases 0.8-fold at the initial 5K iterations and 0.8-fold every 20K iterations. c of Eq. 4 is le-6. At each iteration, the number of ID, i.e., N set 10 by default. The size d of z is 64, and the size k of Gumbel-basee bbaee sampling in Sec. 4.3 is 10. Note that the cardinality of Z always greater than equal to k. For face detection, deep face detector [51] used, and the detected facial regions were resized to 224 x 224 through random cropping (center cropping when evaluating). ID grouping performed based on folders the database without separate clustering tools. Inference is performed through backbone (ft) and projector (fe) without Eq. 3 and Eq. 4.
repeated
Evaluation metrics. Root mean- euean-s error (RMSE) and sign agreement (SAGR) were used to measure the point-wise difference and overall degree of expressions. addition, Pearson correlation coefficient (PCC) and concordance correlation coefficient (CCC) were employed to measure the expression tendency (cf. Appendix C for metric details). Since the objective of FER learning is to simultaneously achieve the minimization of RMSE and the maximization of PCCICCC [22, 27],
Lrcc and LCcc are used together to learn Lea, where Le(/)o0= =1 (L/))CC,+U//)CCC
Arousal
Arouzal
0.5
0.0
0.0
0.0
ELMM
LUK
CAF
CAF
CAF
-1.0
Baseline
-1.0
Bascline
-1.0
Beseline
-10
1.0
GT: -08,0.8
-100
1.0
GT:-0.2,001
-05
as
GT: 0.0,0.4
(a) Angry area VA space
(b) Sad area VA space
(c) Surprise area space
Figure 3: Verification experiments of ID shift when (y) = p" (3/2) but p²(x) * pp)()) Predictions of the two images are located in VA space and +, respectively. Best viewed in color.
5.2 Idea Verification of ELIM
To explain the mechanism of ELIM more clearly, we present experimental answers to the most curious questions: Q1. Is ELIM robust when there is a discrepancy between facial expression and its label prediction? Q2. Is ELIM successful in learning identity (ID)-invariant features? For a fair experiment, compared the proposed method with CAF [22] of SOTA performance as well as the baseline

---

0066
0.4
0.2
0.0
4
(a) ELIM (ours)
(b) CAF
Figure 4: Verification of ID-invariant representation ability between ELIM and CAF. To effectively examine the ID dependency, we compared the distances between an anchor expression sample (e.g.. ①)) and two different expression samples the same ID (e.g.. 2 and 3) Here, the color bar and black lines indicate the range of arousal values and arousal differences, respectively. The larger the difference, the thicker the line. Ten IDs randomly sampled from the test set were used for this experiment.
composed of only CNN and FC layers [29]. All methods were trained with AL-tuned backbone on AFEW-VA dataset under the same experimental condition as ELIM.
A1. Analysis of inconsistency cases. The main goal of ELIM is to accomplish robust performance even in cases where expressions and corresponding label predictions are inconsistent. To verify this, we implement an ID shift case (cf. Sec. 3.1). In detail, we examine how robust each method is to ID shifts for facial images with the same GT but different IDs. Fig. illustrates the prediction results in the VA space. Compared with CAF and baseline, ELIM not only estimates GT precisely, but also outputs relatively consistent prediction results. For more examples, refer Appendix D.
A2. Visualization of ID-invariant representation ability. We visualized feature distributions of ELIM and CAF through t-SNE [42]. Fig. 4 shows the result of comparing the distance between an anchor expression sample and two different expression samples with the same ID. Note that in the case of ELIM, samples with different IDs but similar expressions are located closer in the feature space (see (-2 and 4 in Fig. 4aa)). Mcanwhile, in CAF, samples with the same ID tend to be located close each other (see 2 and 5- Fig. 4(b)). ELIM adaptively considers inter-ID variations of facial expressions through the OT problem, but CAF seems to have an ID-dependent feature distribution because it has means to explicitly consider them. For visualization results and qualitative experiments of other feature distributions, refer to Appendix D.
Table 1: Experimental results on Aff-wild dataset. was evaluated on Aff-wild's test set using ResNet50 backbone. 'tuned' refers to the parameter lightweight version.
RMSE (L)
PCC (f)
CCC (t)
Methods
Backbone
# of params
(V)
(A)
(V)
(A)
(V)
(A)
Hasani et al. [16]
Reseption-ResNett
~56M
0.27
0.36
0.44
0.26
0.36
0.19
BreG-NeXt [17]
ResNeXt50
3.1M
0.26
0.31
0.42
0.40
0.37
0.31
MIMAMO [9]"
ResNet50+GRU
225.6
0.58
0.52
CAF [22]
AL-tuned
2.7M
0.24
0.21
0.55
0.57
0.54
0.56
CAF [22]
ResNet18
11M
0.22
0.20
0.57
0.57
0.55
0.56
AL-tuned
2.6M
0.228
0.199
0.634
0.638
0.628
0.611
ELIM (ours)
ResNet18
11.7M
0.217
0.197
0.635
0.602
0.609
0.570
Mip- Mip-Mixer
59.1M
0.214
0.197
0.645
0.684
0.611
0.631
5.3 Comparison with State-of-the-art Methods
Table I compares ELIM and several existing methods in terms of various evaluation metrics for the Aff-wild dataset. Note that ELIM shows better CCC performance while having smaller parameter size than other methods including CAF, For example, ELIM with AL-tuned has about 2.1 times smaller
7

---

parameter size than Hasani er al. [16] but improves CCC(V) by about 26%. Furthermore, ELIM shows 8.8% higher CCC(V) than a prior CAF [22]. In addition, an experiment was conducted using the recently popular Transformer MIp-Mixer [40] as backbone. Although the parameter size increased, noticeable performance improvement was observed. ELIM with MMx showed about 11% better PCC(A) than CAF, and its RMSE and CCC were also improved significantly.
Table 2 shows the spontaneous expression recognition performance of short video clips through a total of four evaluation metrics. Note that ELIM, which achieved SOTA performance, and CAF, which took 2nd place, are both FER methods designed be robust against changes in identity's expression. As a result, this suggests that it is more meaningful to extract ID-invariant features than to focus only on changes in the expression itself. And, ELIM showed better performance than CAF in terms of CCC metric, which is important for expression tendency analysis. For example, ELIM with R18 showed CCC higher than CAF with R18 by about 2% and 7% in terms of Valence and Arousal, respectively. Considering that the Arousal axis indicating the degree of activation has higher prediction difficulty than the Valence axis indicating positive and negative emotions, such a performance improvement is sufficiently significant. In addition, even in the most difficult Aff-wildd ELIM showed better CCC (mean) than the latest FER method AP [37] (cf. Table 3). These results suggest that further performance improvement is expected if temporal information is applied to static image-based D-invariant methods.
Table 2: Experimental results on AFEW-VA dataset.
RMSE (U)
SAGR (f)
PCC (f)
CCC (?)
Case
Methods
(V)
(A)
(V)
(A)
(V)
(A)
(V)
(A)
Mitenkova er al. [33]
0.40
0.41
0.33
0.42
0.33
0.40
Kossaifi et al. [27]
0.24
0.24
0.64
0.77
0.55
0.57
0.55
0.52
CAF (AL-tuned) [22]
0.20
0.20
0.66
0.83
0.67
0.63
0.58
0.57
Static
CAF (R18) [22]
0.17
0.68
0.87
0.60
0.59
0.54
ELIM (AL-tuned)
0.186
0.198
0.692
0.815
0.680
0.645
0.602
0.381
ELIM (R18)
0.168
0.170
0.723
0.876
0.651
0.679
0.615
0.614
ELIM (MMx)
0.167
0.164
0.747
0.877
0.704
0.707
0.643
0.598
Kollias et al. [24]
0.51
0.58
0.52
0.56
Temporal
HO-Conv [27]
0.28
0.19
0.53
0.75
0.12
0.23
0.11
0.15
HO-Con C0-Conv/B
0.20
0.21
0.67
0.79
0.64
0.62
0.57
0.56
Table 3: Results on the validation set of Aff- wild2.
Table 4: Effectiveness of OT problem different settings.
Methods
CCC (V)
CCC (A)
Mcan
ConvGRU [7]
0.398
0.503
0.450
Weighting methods
Sampling methods
Elapsed time (sec.)
CCC
Self-Attention [44]
0.419
0.505
0,462
(mean)
AP [37]
0.438
0.498
0.468
QPTH Sinkhorn
Self-side
None
1.97
0.580
BLIM (AL)
0.451
0.478
0,465
Relevance
2.44
0.587
ELIM (R18)
0.449
0.496
0.473
Self-side
0.26
0.583
Gumbel
ELIM (MMx)
0.498
0.493
0.495
Relevance
0.28
0.592
Self-side
0.97
0.586
Gumbel
Relevance
1.05
0.589
5.4 Ablation Studies and Discussions
Our ablation studies aim to answer: Q3. Is relevance weight effective? Q4. Is ID matching valid? Q5. Is the number of training IDs the larger the better? Q6. What other factors can be analyzed for ID shift? All experiments in this section were performed under the same setting as AL-tuned of Table 2.
A3. Effectiveness analysis on relevance weights. Sinkhorn- Kankhorn-Knopp [8] depends on the size and number of relevance weights. Table shows the effectiveness of this fast iteration tool three settings. First, let's examine the effect of the optimization way on the OT problem. As benchmark group of Sinkhorn-Knoo interior point method of factorized KKT matrix. Looking at the 3rd and 5th rows of Table 4, Sinkhorn
we chose QPTH [4] that a representative optimization tool based on the

---

Reference
Targets
ID-54
ID-515
ID-510
ID-512
10-909
10-315 0.111
Je 10-5100 0.193
# 20-512, 0.233
# 1D-500, 0.499
ID-528
ID-525
ID-531
ID-520
ID-533
ID-526 0.002
10-531
0.176
p DD-533
Figure 5: Verification of ID matching on the validation split of AFEW-VA.
provides approximately 3.7 times faster operation speed than QPTH while showing almost equivalent CCC performance. This proves that Sinkhorn, which has been often used to solve transportation problems of high-dimensional features, has the advantage of fast operation speed because it is mainly based on iterative updates of vectors. Next, that the relevance weight according to Eq. 8 shows better performance improvement than the self-side (non-relevance) weight despite slight increase in computation. Here, 'self-side' indicates determining the weights with the L2-norm of the individual features of each ID. Looking at the 3rd and 4th rows of Table 4, the relevance weight provides significant CCC performance improvement of about 1%. This suggests that using relevance weights is valuable enough. Finally, Gumbel- based sampling has processing speed of about 8.7 times faster than non non- con-aapling. while providing higher CCC (mean) by 0.5% (see 2nd and 4th rows). Therefore, we can argue that when calculating ID shifts, it is very important to select samples showing clear differences facial expressions as in ELIM. See Appendix D for additional results from Sinkhorn.
A4. Validation of ID matching. When training our model, we adopted a sufficiently large sized mini-batch. As a result, we did not observe any extreme case where the facial expressions of all samples between IDs are totally different each other. If the valence signs of facial expressions between ID groups are all different, ID-dependent features can be generated through Eq. 4 due to the matching between samples of different facial expressions. However, occurs very rarely. Also, since ELIM explores solution through iterative training, such a case has little effect on the overall performance of our model. Fig. 5 shows the ID shift value (nr between reference and target samples. For example, when ID-518 is set to a reference, sample with similar expression such as ID-526 has a small ID shift value, whereas sample with opposite valence sign (i.e., ID-533) has relatively large ID shift value. For convenience, only examples of one sample per ID shown in this figure.
mentioned above, this phenomenon
A5. Influence of the number of training IDs. ELIM, which performs matching of ID pairs through the OT problem, assumes sufficient number IDs.
To analyze the effect of the number of training IDs,
60-
we trained ELIM with or without Lea. Fig. shows that the performance of the two models increases almost the same rate as the number training IDs increases. This prove that sufficient number of training IDs required, and that ELIM with more training IDs has the potential to bring about higher performance improvement.
CCC(mean)
59
88
w/o Cvp
57
56
IDs for training
Figure 6: Influence of the number of train- ing ID. case of 'w/o Lea only Lmain and LoUPICe were used for model training.
A6. Other factors that can influence FER learning. Since inter-ID variation can be sufficiently shifted by change of age or gender [13, 11], age, gender, race, etc. can also be used as indicators to measure inter-ID variation. As fragmentary example, there a technical report that analyzes the tendency differences in facial expressions by domain of the age group of people [35]. Thus, starting with study, various emotional change factors that can be

---

obtained from humans will be grafted into deep learning models for inter- ID variation analysis for FER. Refer to Appendix for the expression lendency experiments according to demographies.
Conclusion and Others
To analyze the inter-identity variation of facial expressions, newly propose identity matching strategy based on optimization theory. The proposed method, which finds and utilizes optimal ID pairs based similarity useful for facial expression learning, succeeded in explicitly dealing with identity shifts, that is, the main cause of performance gap in the test set. The results of verifying identity shifts and visualizing them from t-SNE perspective support the assumption of this paper that is effective to analyze inter-identity variation by considering identity as a domain.
Limitations. The proposed method interacts with multiple target IDs based on single reference ID. Thus, model learning may depend on selection
reference ID. order to overcome this limitation, dynamic connection methods of ID pairs along with more advanced relevance weight calculation methods should be studied in the future.
Potential negative impacts. FER may face with various privacy concerns. For example, someone can estimate and monitor an individual's emotional response social media through this FER system without permission. Although only an authorized database with informed consent is used in this research field, someone's 's facial images are likely be used for learning of the FER system without permission. In order to solve this problem, the privacy-awars face data processing and expression learning framework should be urgently studied and introduced.
Acknowledgments and Disclosure of Funding
The authors greatly thank the three anonymous reviewers for their kind comments which have really helped improve the quality of our paper. This work was supported by IITP grants funded by the Korea government (MSIT) (No. 2021-0-02068, Al Innovation Hub and 15-2002--0015591 Artificial Intelligence Convergence Research Center (Inha University)), and was supported by the NRF grant funded by the Korea government (MSIT) (No. 2022R 1022R1A2C201009 and No. 2022R A4A 1033549).
References
Ravindra K Ahuja, Thomas Magnanti, and James B Orlin. Network flows. 1988.
[2] Kamran Ali and Charles E Hughes. Facial expression recognition by using disentangled identity-invarian expression representation. In 2020 25th International Conference Pattern Recognition (ICPR), pages 9460-9467. IEEE, 2021.
pages
[3] Jason Altschuler, Jonathan Niles Weed, and Philippe Rigollet. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. Advances neural information processing systems, 30, 2017.
Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization layer in neural networks. In International Conference on Machine Learning, pages 136-145. PMLR, 2017.
Pablo Barros, German Parisi, and Stefan Wermter, personalized affective memory model for improving emotion recognition. International Conference on Machine Learning, pages 485 485-494, 2019.
Andrew J Calder and Andrew W Young. Understanding the recognition of facial identity and facial expression. Nature Reviews Neuroscience, ((8):641-51) 2005.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau. Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 2014, Doha, Qatar, pages 1724 1734. ACL, 2014.
[8] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[9] Didan Deng, Zhaokang Chen, Yuqian Zhou, and Bertram Shi. Mimamo net: Integrating micro-and macro- motion for video emotion recognition. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 3621-2628, 2020.
10

---

[10] Abhinav Dhall, Amanjot Kaur, Roland Goecke, and Tom Gedeon. Emotiw 2018: Audio-video, student engagement and group-level affect prediction. In Proceedings the 20th ACM International Conference on Multimodal Interuction, pages 653-656, 2018
[11] Ulf Dimberg and Olov Lundquist. Gender differences in facial reactions facial expressions. Biological psychology, 30(2):151-15,, 1990.
[12] Xinjie Fan, Qifei Wang, Junjie Ke, Feng Yang. Boqing Gong, and Mingyuan Zhou. Adversarially adaptive normalization for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8208-8217, 2021.
[13] Mara Fölster, Ursula Hess, and Katja Werheid. Facial age affects emotional expression decoding. Frontiers in psychology. 5:30, 2014.
[14] Ian Goodfellow. Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[15] Ishaan Guirajani and David Lopez-Paz. search of lost domain generalization. In 9th International Con- ference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[16] Behzad Hasani and Mohammad H Mahoor. Facial affect estimation in the wild using deep residual and convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 9-16, 2017.
[17] Behzad Hasani, Pooran Singh Negi, and Mohammad Mahoor. Breg-next: Facial affect computing using adaptive residual networks with bounded gradient. IEEE Transactions Affective Computing, 2020.
[18] James V Haxby, Elizabeth A Hoffman, and M Ida Gobbini. The distributed human neural system for face perception. Trends in cognitive sciences, 4(6):223 233, 2000.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Wei Huang, Siyuan Zhang, Peng Zhang. Yufci Zha, Yuming Fang, and Yanning Zhang. Identity-aware facial expression recognition via deep metric learning based on synthesized images. IEEE Transactions on Multimedia, 2021.
[21] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages 1501-1510, 2017.
[22] Dae Ha Kim and Byung Cheol Song. Contrastive adversarial learning for person independent facial emotion recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5948 5956, 2021.
[23] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[24] Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, Bjöm Schuller, Irene Kotsia. and Stefanos Zafeiriou. Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures. and beyond. International Journal of Computer Vision, (2766-7),,,7992,, 2019.
[25] Dimitrios Kollias and Stefanos Zafeiriou. Expression, affect, action unit recognition: Aff-wild2, multi- task learning and areface. In 30th British Machine Vision Conference 2019, BMVC 2019, Cardiff. UK, September 9-12, 2019, page 297, 2019.
[26] Wouter Kool, Herke Van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel- top-k trick for sampling sequences without replacement. In International Conference on Machine Learning, pages 3499-3508. PMLR, 2019.
[27] Jean Kossaifi, Antoine Toisoul, Adrian Bulat, Yannis Panagakis, Timothy M Hospedales, and Maja Pantic. Factorized hideer-order cnns with application to spatio-temporal emotion estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6060-6069, 2020.
[28] Jean Kossaifi, Georgios Tzimiropoulos. Sinisa Todorovic, and Maja Pantic. Afew-va database for valence and arousal estimation in in-the-wild. Image and Vision Computing, 65:23-36 2017.
11

---

[29] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
[30] Hsin- Yi Lin, Huan-Hsin Tseng, Xugang Lu, and Yu Tsao. Unsupervised noise adaptive speech enhancement by discriminator-consteri 2021.
optimal transport. Advances in Neural Information Processing Systems, 34,
[31] Massimiliano Mancini, Lorenzo Porzi, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Boosting domain adaptation by discovering latent domains. In Proceedings of the IEEE Conference Computer Vision and Patiern Recognition, pages 3771-3780, 2018
[32] Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, 2019.
[33] Anna Mitenkova, Jean Kossaifi, Yannis Panagakis, and Maja Pantic. Valence and arousal estimation in-the-wild with tensor methods. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pages 1-7, 2019.
[34] Vaishnavh Nagarajan Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of out duitt--disttibution generalization. In International Conference on Learning Representations, 2020.
[35] Rafael Poyiadzi, Jie Shen, Stavros Petridis, Yujiang Wang, and Maja Pantic. Domain generalisation for apparent emotional facial expression recognition across age-groups. arXiv preprint 4rXiv:2110.09168; 2021.
[36] James A Russell. circumplex model of affect. Journal personality and social psychology. 1961)1 1980.
[37] Enrique Sanchez, Mani Kumar Tellamekala, Michel Valstar, and Georgios Tzimiropoulos. Affective processes: stochastic modelling of temporal context for emotion and facial expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9074-9084, 2021.
[38] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[39] Byung Cheol Song, Min Kyu Lee, and Dong Yoon Choi. Facial expression recognition via relation- based conditional generative adversarial network. In 2019 International Conference on Multimodal Interaction, pages 35-39, 2019.
[40] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, al. Mlp-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Syxtems, 34, 2021.
[41] Antonio Torralba and Alexci Efros. Unbiased look dataset bias. In CVPR 2011, pages 1521 1521-1528. IEEE, 2011.
[42] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of machine learning research, 9(11), 2008
[43] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, volume 30, pages 5998-6008. Curran Associates, Inc., 2017.
[45] Ximei Wang, Ying Jin, Mingsheng Long, Jianmin Wang, and Michael I Jordan. Transferable normalization: Towards improving transferability of deep neural networks. Advances in neural information processing systems, 32, 2019.
[46] Fang Wu, Nicolas Courty, Zhang Qiang, Ziqing Li, et al. Metric learning-chhanced optimal transport for biochemical regression domain adaptation. arXiv preprint arXiv:2202.066 06208, 2022.
[47] Huiyuan Yang. Umur Ciftci, and Lijun Yin. Facial expression recognition by de- ex-exssession residue learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2168 2177. 2018.
12

---

[48] Alla Yankouskaya, Glyn W Humphreys, and Pia Rotshtein. The processing of facial identity and expression is interactive, but dependent task and experience. Frontiers Human Neuroscience, 8:920, 2014.
[49] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
[50] Stefanos Zafeiriou, Dimitrios Kollias, Mihalis A Nicolaou, Athanasios Papaioannou. Guoying Zhao, and Irene Kotsia. Aff-wild: Valence and arousal in-the-wild chaalleed In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 34-41, 2017.
[51] Kaipeng Zhang, Zhanpeng Zhang. Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10):1499-1500 2016.
[52] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event Austria, May 3-7, 2021. OpenReview.net, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section 1.
(b) Did you describe the limitations of your work? [Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.1.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 3.1 and Section 5.2.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Algorithm section in the supplemental material. link to the full source code is added to the camera-ready version.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No] Instead of error bars, we use the average values of multiple experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 5.1.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 5.1.
(b) Did you mention the license of the assets? [Yes] See link for source code and software demo.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] See link for source code and software demo or Python scripts in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See Section 5.1.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Section 6.
5. If you used crowdsourcing conducted research with human subjects...
13

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "jt1H1KhpCkl", "reviewer": "Reviewer_zNEr", "summary": "The paper proposes an approach to identity invariant facial expression recognition. Identity matching is formulated as an optimal transport problem. The optimal flow for this problem is found using Sinkhorn-Knopp. Evaluations are conducted on three public FER datasets. ", "questions": "How can the proposed approach handle shift due to demographics such as age, gender, or race?", "limitations": "Negative impact has been addressed. Limitation regarding the reference ID is detailed, however, limitations on the scope of demographics and the proposed approach are not detailed. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nPaper proposes a solution to a challenging problem in FER. Problem is well formulated in paper including needed background information to setup proposed solution. State of the art results are detailed on multiple public datasets. Results show the proposed approach can easily be integrated into different models. Ablation study on the relevance weights, and impact of number of IDs is well conducted. \n\nWeaknesses\n\nAs the paper focuses on identity invariant FER, other factors such as age, gender, and race could have a large impact on the ID matching and FER learning. While this is briefly discussed in Section 5.4 (A5), the discussion is largely superficial with a tech report being cited that talk about differences in facial expression across age. It is not clear how the proposed approach can handle these changes across different demographics. Handling this type of shift is important for generalized FER.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8Cb8xqZjxXG", "reviewer": "Reviewer_U2SY", "summary": "The paper proposes a method for facial expression recognition (by means of valence and arousal) which aims to remove the factors that relate to identity from the backbone features, so as to make the classifier be robust against identity-specific features. In particular, a method is proposed to remove from the feature representation the identity shift, by computing the optimal transport between the batch-specific features in a way to represent the image-specific domain shift from the other images in the batch. The features are then normalized according to the computing shifting mean and scale and passed through the final classifier which regresses the values of valence and arousal. The use of optimal transport to assign the image-specific mean vector is grounded on the idea that the batch-specific features can be rearranged, while preserving the total mass distribution, so as to remove the identity-based similarities encountered in the batch. The method is validated in a set of challenging datasets achieving competitive results. ", "questions": "As stated above, my main concerns or questions can be summarized as:\n1) Exploring or comparing against of attention-based methods for residual learning. \n\n2) Inference\n\n3) Use of SEWA or AffectNet to compare against CAF\n\n4) Different number of parameters for same networks in Table 1.\n\n5) Different losses and comparison against existing methods.\n\n6) Explanation of Figure 4\n", "limitations": "The limitations are properly addressed in the paper and hence I have no further questions in this regard.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is technically novel and achieves competitive results and hence has the merits to be accepted at NeurIPS. The use of OT to disentangle the factors of identity is novel, the motivation is clear and the experiments are compelling. The paper is well accompanied by algorithms and descriptions that help understanding the derivation, with further pseudo-code for the crucial parts in the Supplementary Material. The paper could benefit from a better clarity of presentation and elaboration on the significance and meaning of the learned shift, and thus I would like to particularly ask the authors to improve the manuscript’s presentation and writing substantially for better clarity.\n\nMy main comments and concerns are listed below:\n\n- Digging under the hood of Equations 3 and 4 as well as how the cost matrix is computed based on similarity, I can spot that there are some substantial similarities and differences between using the OT assignment and using cross-attention to attenuate the influence of the global features. Plugging the cost matrix into Eqn 3 and observing the “residual” behaviour of Eqn. 4 can derive to expressing the new features as a linear combination of the other images’ features, given by a weighted similarity between them. This resembles to me the use of cross-attention with X being the attention weights. Obviously, the authors propose a completely different way to assign these attention weights, but after reading the full derivation I would like to ask the authors if a small transformer encoder instead to compute the mean features in Equation (3) would also enforce the learned features to be identity-invariant. I think that such exploration is important and the paper would gain significant strength such comparison would be given. \n\n- It is not mentioned in the paper (or I missed that), but there is no reference to the process of inference. My understanding is that at test time no shift is obtained because the features are already expected to be “centered”, but some clarification in this regard would be appreciated.\n\n- I think the paper would gain strength by including the performance of the proposed approach on AffectNet and SEWA, while AffWild2 and AffWild are challenging, they are a subset of each other, and AFEW-VA has recently given way to the aforementioned datasets.\n\n- What makes the number of parameters on Table 1 be different for the proposed method and that of CAF for the AL-tuned and ResNet-18 backbones? \n\n- I am a bit confused about the results shown in Figure 5. First, the behaviour of the curves is a bit counterintuitive as e.g. there is a drop in two points of CCC between using 18 and 19 subjects, which is recovered after using 20 subjects. What is this behaviour due to? In addition, it is not clear to me what does w/o L_va mean, considering that this is the main training loss. The blue curve, does it correspond to a training using the PCC + CCC losses? Please do clarify. \n\n- It is important also to understand the contribution of the training losses in the method. How does this compare against competing methods? If they had been trained without L_ccc or P_ccc then I believe the authors should also consider a fair evaluation using their method trained with the same losses. This will indicate the contribution of the losses in the performance gain.\n\n- I didn’t get quite well what Figure 4 attempts to represent, so I would like to request the authors to elaborate on it.\n\n- Finally, as mentioned before, I believe the paper would greatly benefit from better writing and presentation. The readability of the paper is somewhat poor affecting its reach. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6XSC0PiI8Jv", "reviewer": "Reviewer_yjnn", "summary": "The paper proposes to quantify the inter-identify variation by utilizing pairs of similar expressions for identify-invariant facial expression recognition. To find pairs of similar expressions from different identities, the authors define the inter-feature similarity as a transportation cost. Then, they perform optimal identity matching to find the optimal flow with minimum transportation cost by Sinkhorn-Knopp iteration. ", "questions": "1. Please give reasonable explanations for the above weaknesses.\n2. In line 169, I do not find the definition of $L_{va}$ in the cited paper [50]. Maybe the author made a mistake.", "limitations": "Please see the Questions.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors propose a method to quantify the inter-identify variation, which is different from the previous works in a non-quantitative way.\n2. The experimental results exceed most existing works\nWeaknesses:\n1. As mentioned in the paper, inter-identify variation is quantified by pairs of similar expressions. However, when the expressions of individuals with different IDs differ greatly in a batch, that is, the precondition fails. The methods may not work.\n2. The variations between reference and targets are calculated in a batch, but different references will be chosen in different batches. The variations among references are not considered.\n3. Although the authors describe the training process in detail, they do not mention the testing process, which I believe is a non-trivial process.\n4. The authors only verify that the proposed method performs better in VA prediction, but it is lack experiments on expression and AU prediction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept