Learning with Alignments: Tackling the Inter- and Intra-domain Shifts for Cross-multidomain Facial Expression Recognition

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 13:53:00

Abstract: Facial Expression Recognition (FER) holds significant importance in human-computer interactions. Existing cross-domain FER methods often transfer knowledge solely from a single labeled source domain to an unlabeled target domain, neglecting the comprehensive information across multiple sources. Nevertheless, cross-multidomain FER (CMFER) is very challenging for (i) the inherent inter-domain shifts across multiple domains and (ii) the intra-domain shifts stemming from the ambiguous expressions and low inter-class distinctions. In this paper, we propose a novel Learning with Alignments CMFER framework, named LA-CMFER, to handle both inter- and intra-domain shifts. Specifically, LA-CMFER is constructed with a global branch and a local branch to extract features from the full images and local subtle expressions, respectively. Based on this, LA-CMFER presents a dual-level inter-domain alignment method to force the model to prioritize hard-to-align samples in knowledge transfer at a sample level while gradually generating a well-clustered feature space with the guidance of class attributes at a cluster level, thus narrowing the inter-domain shifts. To address the intra-domain shifts, LA-CMFER introduces a multi-view intra-domain alignment method with a multi-view clustering consistency constraint where a prediction similarity matrix is built to pursue consistency between the global and local views, thus refining pseudo labels and eliminating latent noise. Extensive experiments on six benchmark datasets have validated the superiority of our LA-CMFER.

Main paper: 
 introduction: 
 Facial Expression Recognition (FER) endeavors to discern human expressions and emotional states, playing a pivotal role in humancomputer interactions [1] . Nowadays, FER has gained notable promotion thanks to diverse deep learning (DL) algorithms [2, 3, 4, 5, 46, 47, 48] and well-annotated FER datasets [6, 7, 8] . However, these FER works [9, 10] typically operate under the assumption that both training and testing samples come from the same dataset (domain) and inherently share an identical data distribution. In practice, their classification accuracy often drops sharply due to the great discrepancy in data distribution (i.e., inter-domain shifts) [11, 12] when applied in different scenes, making them incapable of tackling the cross-domain problem settings.To alleviate the inter-domain shifts, unsupervised Cross-Domain FER (CDFER) [13, 14, 49] has been introduced, aiming to extract domain-invariant features from a single labeled source domain to accurately classify the samples in an unlabeled target domain. Various techniques, such as adversarial learning [11, 15, 16] and metric learning [12, 17, 18] , have been explored to reduce the distribution disparity between the two domains. However, these methods primarily focus on leveraging data information from a single source, while overlooking the valuable facial knowledge buried in multiple source domains. In reality, there are multiple labeled source datasets collected with diverse conditions, such as different acquisition environments, ethnic characteristics, etc. Leveraging these sources can significantly increase the number of training samples and further provide more comprehensive and discriminative feature representations of facial expressions, thus contributing to better generalization ability. Subsequently, unsupervised Cross-Multidomain FER (CMFER) [19] is motivated to effectively transfer richer facial knowledge from multiple labeled source domains to the unlabeled target domain.A simple solution to CMFER tasks is to regard different sources as a single source and directly apply CDFER methods. Regretfully, as the number of sources increases, these approaches often exhibit sub-optimal performance or even degradation owing to the growing complexity of data distributions [19, 20] . Therefore, developing a well-designed CMFER method is quite imperative. Delving deeply into CMFER, there are two main challenges. The first one is the above-mentioned inter-domain shifts across different domains due to their inherent data distribution difference. Specifically, as shown in Fig. 1 , with different shooting conditions, AffectNet [6] (i.e., 𝑇𝑇) contains both gray and color images, while FER-2013 [21] (i.e., 𝑆𝑆 𝑚𝑚 ) only includes gray images. Besides, JAFFE [22] (i.e., 𝑆𝑆 𝑛𝑛 ) has notable racial characteristics where all the facial images come from Japanese women. These severe inter-domain shifts disrupt the effective extraction of domain-invariant facial expression representations and hinder smooth knowledge transfer from source domains to the target domain. The second challenge for CMFER is the intra-domain shifts that occur within the target domain. As shown in Fig. 1 , the subtle differences and intrinsic ambiguities in human expressions (marked by red borders) may lead to lower inter-class distinctions. Therefore, the intra-domain shifts are likely to cause the model to produce uncertain category predictions and acquire inaccurate semantic information, severely damaging the prediction robustness.In this paper, we propose a novel Learning with Alignments for Cross-Multidomain Facial Expression Recognition (LA-CMFER) framework to simultaneously tackle the inter-and intra-domain shifts. In LA-CMFER, we consider both essential global and local features for facial expressions and employ a shared dual-branch structure across multiple source domains and the target domain. Then, to address the inter-domain shifts, a dual-level (i.e., samplelevel and cluster-level) inter-domain alignment is developed to encourage the model to pay more attention to the hard-to-align samples with higher uncertainty and generate a more well-clustered feature space based on the predicted labels. As for the intra-domain shifts, a multi-view intra-domain alignment is devised with a mutiview clustering consistency constraint to form a discriminative target feature space, and a multi-view voting scheme to gain the high-quality pseudo labels for an effective information interaction between the local and global branches, respectively. Both inter-and intra-domain alignments act with united strength to comprehensively extract discriminative facial representations from multiple source domains and promote the classification in the target domain. The main contributions of this paper are three-fold: We present a Learning with Alignments for Cross-Multidomain Facial Expression Recognition (LA-CMFER) framework to tackle both inter-and intra-domain shifts and utilize beneficial knowledge from multiple source domains to address the challenging CMFER task.  To mitigate the inter-domain shift, we introduce a duallevel inter-domain alignment method that prioritizes hardto-align samples and fosters a well-clustered feature space.As for the intra-domain shift, we design a muti-view clustering consistency constraint between the global and local branches to gain a concentrated target feature space for alleviating the interference of noisy samples.  Extensive experiments on six commonly used FER benchmark datasets verify the superior performance of our method compared to other state-of-the-art approaches. 
cross-domain fer: 
 CDFER follows the fundamental principles of Unsupervised Domain Adaptation (UDA) [23] , aiming to apply the classifier trained with a single labeled source domain to classify samples in an unlabeled target domain. Numerous studies have focused on CDFER tasks, which can be categorized into two groups: metricbased approaches vs. adversarial-based approaches. Concretely, metric-based approaches [12, 13 17, 18, 24] construct proper distance functions to extract more discriminative facial features for reducing cross-domain distribution variation. For instance, Ni et al. [24] combined metric learning with dictionary learning to alleviate the transfer facial expression recognition issue. Zhang et al. [13] presented a local-global discriminative subspace transfer learning (LGDSTL) method where a local-global graph is used as distance metric. Besides, adversarial-based approaches [11, 15, 16, 25] mine the domain-invariant facial features with adversarial training. To illustrate, Xie et al. [15] embedded a graph representation propagation with adversarial learning and presented an adversarial graph representation adaptation framework. Considering the important local facial features, Ji et al. [11] proposed a region attention-enhanced CDFER approach to emphasize the local features for minimizing domain discrepancies. Despite their promising results, these methods deal with only one single source and are not competitive in addressing the complex data distributions encountered in cross-multidomain scenarios.Multi-source Domain Adaptation (MDA) [26] holds the assumption that data can be gathered from diverse source domains with different distributions, which is a more practical but challenging task compared to single-source domain adaptation.Pioneer studies [27, 28, 50] have theoretically confirmed that the target distribution can be represented as a weighted combination of source distributions. To further ensure effective adaptation, the key lies in overcoming the inter-domain shifts caused by diverse data distributions across domains. Along with this philosophy, Zhao et al. [29] used adversarial training to align the target and source distributions. Zhu et al. [30] developed an MDA framework with multiple domain-specific classifiers to mine the domain-invariant features and use a maximum mean discrepancy (MMD) loss [31] to ease the inter-domain shift. Li et al. [32] devised a feature filtration network to selectively align features across domains. Unlike existing MDA works in natural image classification, the CMFER tasks are more difficult. Besides the inter-domain shifts, the subtle differences between facial expressions also bring about intra-domain shifts, e.g., low inter-class distinctions in the decision boundaries. In the emerging CMFER, only Liu et al. [19] proposed a domain-uncertain mutual learning (DUML) network based on adversarial learning to consider both inter-domain and intradomain uncertainty. In this paper, we present the Learning with Alignments framework with dual-level inter-domain alignment and multi-view intra-domain alignment to adeptly address two kinds of domain shifts with a simpler and lighter architecture. and supervise the predictions of the two branches in turn, facilitating the knowledge interaction of the two branches. 
global and local branches: 
 Given the notable importance of both global and local features, LA-CMFER adopts a dual-branch architecture. As shown in Fig. processing )). These two branches make predictions from the complementary global-local views for both source and target samples, thus better detecting the beneficial expression knowledge.To tackle the inter-domain shifts between source and target distributions, LA-CMFER presents the dual-level (sample-and cluster-level) inter-domain alignment to comprehensively extract the domain-invariant facial features. This is gained by prioritizing hard-to-align samples with higher uncertainty and grouping samples based on their category information. Sample-level Inter-Domain Alignment. Existing CDFER works usually use the discrepancy-based constraints (e.g., MMD [30, 31, 52] , triplet loss [4] , and adversarial loss [11, 25] ) to minimize the inter-domain shift. Concretely, given the source distribution 𝒫𝒫 𝑠𝑠 𝑛𝑛 (𝑥𝑥, 𝑦𝑦) and target distribution 𝒫𝒫 𝑇𝑇 (𝑥𝑥, 𝑦𝑦), the traditional MMD loss [31] can be formulated as follows:EQUATIONEQUATIONwhere 𝜙𝜙 𝑖𝑖 𝑎𝑎 denotes the average feature mappings of input samples within the batch 𝐵𝐵 𝑎𝑎 to the reproducing Kernel Hilbert Space 𝜘𝜘 and 𝑎𝑎 represents the target domain (𝑇𝑇) or the 𝑛𝑛-th source (𝑆𝑆 𝑛𝑛 ). Despite its satisfactory performance, traditional MMD usually averages the distances among samples with uniform weights, thus neglecting the varying importance of different samples in the alignment. In the CMFER task, some hard-to-align samples, exhibiting higher uncertainty and located near decision boundaries, may offer greater insights into the domain alignment and knowledge transfer, thus warranting increased attention. Thus, we propose the sample-level inter-domain (SID) alignment with a hardness-aware weighting function. Specifically, given an image 𝑥𝑥 𝑖𝑖 𝑎𝑎 and its prediction 𝑃𝑃 𝑀𝑀,𝑖𝑖 𝑎𝑎 with 𝐽𝐽 probability values {𝑃𝑃 𝑀𝑀,𝑖𝑖 𝑎𝑎,𝑗𝑗 } 𝑗𝑗=1, where 𝑀𝑀 = 𝐺𝐺 𝑜𝑜𝑜𝑜 𝐿𝐿 denotes different branches and 𝐽𝐽 denotes the number of classes, we use a measurement term 𝛺𝛺 𝑀𝑀 (•) to adaptively evaluate the instantaneous hardness of 𝑥𝑥 𝑖𝑖 𝑎𝑎 in the current iteration:EQUATIONwhere Finally, 𝐻𝐻 𝑀𝑀 (•) is utilized to modify the traditional MMD loss as our sample-level inter-domain alignment loss ℒ 𝑠𝑠𝑖𝑖𝑖𝑖 𝑀𝑀 :EQUATIONEQUATIONNotably, 𝐻𝐻(•) can be seamlessly integrated into other discrepancy-based constraints in a plug-and-play way for providing useful perceptions about sample importance. Cluster-level Inter-domain Alignment. Existing discrepancybased inter-domain alignments [30, 31] may blindly align samples with different classes closer, potentially impeding the model from learning discriminative features. Thus, we introduce a cluster-level inter-domain alignment to better group samples based on their categories. We use the real label 𝑦𝑦 𝑖𝑖 𝑠𝑠 𝑛𝑛 =𝑘𝑘 and pseudo label 𝑦𝑦 � 𝑖𝑖 𝑇𝑇 =𝑑𝑑 ̂ to denote the category attributes of the source sample 𝑥𝑥 𝑖𝑖 𝑠𝑠 𝑛𝑛 and target sample 𝑥𝑥 𝑖𝑖 𝑇𝑇 , respectively. Then, we minimize the distances between source-target samples of the same category (𝑑𝑑 ̂= 𝑘𝑘) and maximize them for samples of different categories (𝑑𝑑 ̂≠ 𝑘𝑘). This process is summarized as a cluster-level inter-domain alignment loss ℒ 𝑚𝑚𝑖𝑖𝑖𝑖 :EQUATIONEQUATIONwhere the hardness-aware function 𝐻𝐻(•) is retained to dynamically balance the importance of each sample. As ℒ 𝑚𝑚𝑖𝑖𝑖𝑖 optimizes, we can achieve a more fine-grained alignment to foster an intra-class convergence while promoting inter-class divergence.Finally, the dual-level inter-domain alignment is performed in both global and local branches at the same time, thus deriving the following inter-domain loss ℒ 𝑖𝑖𝑛𝑛𝑖𝑖𝑖𝑖𝑖𝑖 :EQUATIONwhere 𝜆𝜆 is a hyper-parameter for balancing these two terms.Notably, for all domains, we include only those samples whose maximum prediction probability exceeds a predefined reliability threshold 𝜖𝜖, thus preventing unreliable knowledge transfer from excessively uncertain samples. Drawing on the cluster assumption [33] , we consider target samples in 𝐵𝐵 𝑇𝑇 as a specific class (i.e., class 𝑟𝑟 ) and their corresponding prediction probabilities in 𝐵𝐵 𝑇𝑇 as a batch-wise cluster representation for class 𝑟𝑟, which is captured by the 𝑟𝑟-th column in 𝑃𝑃 𝐺𝐺,𝐵𝐵 𝑇𝑇 or 𝑃𝑃 𝐿𝐿,𝐵𝐵 𝑇𝑇 . To reach the intra-domain alignment, we enhance the consistency among cluster representations for identical class assignments (i.e., higher diagonal values in MPC) and inconsistency among them with different class assignments (i.e., lower off-diagonal values in MPC). This process can be represented as the following multi-view clustering consistency loss ℒ 𝑚𝑚𝑚𝑚𝑚𝑚 :EQUATIONwhere 𝑆𝑆(•) calculates the sum of absolute values across all matrix elements, 𝐼𝐼 ∈ ℝ 𝐾𝐾×𝐾𝐾 represents an identity matrix, and ‖•‖ denotes a normalization function [34] . Minimizing ℒ 𝑚𝑚𝑚𝑚𝑚𝑚 facilitates the well-separated clusters of different expressions in the target feature space and encourages consistent predictions from dual branches. where 𝑀𝑀𝑀𝑀(•) is a function to determine whether the two voting conditions are simultaneously met. By incorporating ℒ 𝑚𝑚𝑚𝑚𝑚𝑚 and ℒ 𝑚𝑚𝑚𝑚𝑚𝑚 , LA-CMFER is guided to achieve a meaningful knowledge interaction between the two branches, promoting mutual enhancement and facilitating the development of a more robust target feature space.The whole training objective contains five parts: (1) supervised loss ℒ 𝑠𝑠𝑠𝑠𝑠𝑠 for labeled source samples, (2) inter-domain loss ℒ 𝑖𝑖𝑛𝑛𝑖𝑖𝑖𝑖𝑖𝑖 , (3) multi-view clustering consistency loss ℒ 𝑚𝑚𝑚𝑚𝑚𝑚 , and (4) multi-view voting loss ℒ 𝑚𝑚𝑚𝑚𝑚𝑚 for unlabeled target samples.The supervised loss ℒ 𝑠𝑠𝑠𝑠𝑠𝑠 can be represented as follows:EQUATIONwhere 𝐶𝐶𝐸𝐸(•) denotes the cross-entropy loss. Finally, the training objective of LA-CMFER is formulated as:EQUATIONwhere 𝛼𝛼, 𝛽𝛽, and 𝛾𝛾 are three weighted hyper-parameters.The training stage of LA-CMFER is summarized in Algorithm 1. In the inference stage, we take the prediction with a higher probability from the two branches as the final result [51] . 
experimental setup: 
 Datasets. To verify the performance of the proposed method, we follow the experiment settings in [19] and involve six commonly used FER datasets, including three lab-controlled datasets CK+ [7] , JAFFE [22] , and Oulu-CASIA [35] and three Internet-collected large-scale field datasets AffectNet [6] , RAF-DB [8] , and FER-2013 [21] . Specifically, CK+ provides 593 annotated video samples from 123 subjects and uses the last three frames with six basic emotions (i.e., anger, disgust, fear, surprise, happy, and sad) and neutral emotions, finally gaining a total of 1,236 images. JAFFE includes 213 facial expression images from 10 Japanese women. Oulu-CASIA (Oulu) captures 2988 face images by choosing the last three frames with normal lighting for basic expressions and the first frame for neutral expressions. AffectNet stands as the largest facial expression dataset to date. Following [19] , we curate up to 5,000 facial images for each emotion category, resulting in a total of 33,793 training images and 3,500 testing images. RAF-DB is a real-world FER dataset that has 12,271 and 3,068 images for training and testing, respectively. FER-2013, compiled via the Google image search engine, contains 35,887 where 𝑀𝑀 = 𝐺𝐺 𝑜𝑜𝑜𝑜 𝐿𝐿 with Eq. ( 5) 9:Compute the cluster-level inter-domain alignment loss ℒ 𝑚𝑚𝑖𝑖𝑖𝑖 𝑀𝑀 where 𝑀𝑀 = 𝐺𝐺 𝑜𝑜𝑜𝑜 𝐿𝐿 with Eq. ( 7) 10: Compute the total inter-domain loss ℒ 𝑖𝑖𝑛𝑛𝑖𝑖𝑖𝑖𝑖𝑖 with Eq. ( 9) 11: Compute the multi-view clustering consistency loss ℒ 𝑚𝑚𝑚𝑚𝑚𝑚 with Eq. ( 11) 12:if 𝟙𝟙�𝑀𝑀𝑀𝑀(𝑃𝑃 𝐺𝐺,𝑖𝑖 𝑇𝑇 , 𝑃𝑃 𝐿𝐿,𝑖𝑖 𝑇𝑇 )� 13:Compute 𝑦𝑦 � 𝑖𝑖 𝑇𝑇 and multi-view voting loss ℒ 𝑚𝑚𝑚𝑚𝑚𝑚 with Eq. ( 12) 14:Compute the supervised loss ℒ 𝑠𝑠𝑠𝑠𝑠𝑠 with Eq. ( 13) 15:Update {𝜃𝜃 𝐸𝐸 , 𝜃𝜃 𝐴𝐴𝐺𝐺 , 𝜃𝜃 𝐴𝐴𝐿𝐿 , 𝜃𝜃 𝐶𝐶𝐺𝐺 , 𝜃𝜃 𝐶𝐶𝐿𝐿 } by minimizing Eq. ( 14) 16: end for Fig. 3 . Examples from the six FER datasets. images of different facial expressions and uses 28,709 images for training and 3,589 images for testing. Intuitively, some samples from each dataset are given in Fig. 3 .For label settings, we select the subset with six basic and neutral emotion labels from each dataset, and the training and testing sets of the lab-controlled datasets are the same. In our experiments, each dataset is sequentially regarded as a domain and we utilize the symbol '→A' to represent that dataset A is the target domain while other datasets serve as source domains. Implementation Details. We conduct all our experiments on two GeForce RTX 3090 GPUs with the PyTorch toolbox. Same as [19] , face images are first detected and aligned by the RetinaFace [36] and further resized to 224 ×224. We use the conv1∼conv3 layers of ResNet-18 pre-trained on ImageNet [37] as the shared encoder 𝐸𝐸 . The details of the global and local branches are in Supplementary Materials. For all datasets, we train the whole framework for 20,000 iterations with a learning rate of 0.01 and a batch size of 128. The setting of the optimizer and learning schedule are kept the same with [38] . In our experiments, we maintain a fixed random seed over 3 runs and report the mean results. The hyper-parameter 𝜆𝜆 is empirically set as 0.02. For the hyper-parameters in Eq. ( 14), based on our trial studies, we set 𝛼𝛼 as 0.4 for FER-2013 and AffectNet and 0.1 for the rest. Besides, we set 𝛽𝛽 as 0.5 and 𝛾𝛾 as 0.1. The reliability threshold 𝜖𝜖 is set as 0.4. To select reliable pseudo labels, 𝜏𝜏 is set as 0.9 [39, 40] .To verify the effectiveness of the proposed method, we compare our method with state-of-the-art (SOTA) Single-source DA methods (SDA) and MDA methods. For SDA methods, two protocols are adopted: (1) Single Best, which reports the best result among all source domains, and (2) Source Combine, which naively combines all source domains and then uses the SDA methods. Specifically, Source-Only refers to directly transferring the model trained in source domains to the target domain. We select (i) the traditional SDA methods: DAN [4] , DANN [41] , ADDA [42] , and CD-FER method AGLRLS [14] with both protocols. (ii) The CD-FER methods: LGDSTL [13] , DMSRL [43] , and RANDA [11] are chosen as additional comparison methods for Single Best. For MDA methods, we choose (iii) the typical MDA methods MDAN [29] , M 3 SDA [44] , and MJD [45] ; and (iv) the latest CMFER method DUML [19] . To ensure a fair comparison, the results of these methods are obtained either from their respective papers or by reimplemented using their released code.The comparison results on different FER datasets are reported in Tab.1 where our LA-CMFER gains the best average accuracy of 69.32% and largely surpasses the second-best DUML by 1.59%. In particular, all Source Combine methods are relatively unpromising, as the inter-domain shifts impair effective knowledge transfer. On the '→CK+' and '→FER-2013' tasks, the SOTA CDFER method AGLRLS experiences considerable performance degradation (i.e., 62.40% average accuracy) when adapting from single-source to multi-source scenarios. This underscores the impracticality of solely relying on the Source Combine strategy to address CMFER tasks. Besides, by tackling the inter-domain shifts, MDA methods gain better performance where MJD obtains 62.59% average accuracy and reaches 68.29% and 63.45% on the '→RAF-DB' and '→Oulu' task, respectively. As a well-designed SOTA CMFER method, DUML achieves the second-best overall performance. Compared to it, our LA-CMFER still keeps its leading accuracy. Especially, on the challenging '→RAF-DB' task, LA-CMFER remarkably surpasses DUML by 4.62%. These experimental results have verified the superior performance of our LA-CMFER.To study the contributions of key components in our LA-CMFER, we progressively conduct ablation experiments on all FER datasets. For clarity, 'Baseline' means directly transferring the dual-branch model trained in the source domains to the target domain with only supervised loss ℒ 𝑠𝑠𝑠𝑠𝑠𝑠 . 'MMD' indicates using the traditional MMD loss to assist the network to alleviate the inter-domain shift. The experimental better fosters consistency between two branches, thus decreasing the prediction uncertainty and promoting intra-domain alignments.In the intra-domain alignment, LA-CMFER designs a multi-view voting scheme to provide additional supervision (ℒ 𝑚𝑚𝑚𝑚𝑚𝑚 ) with the high-quality pseudo labels which are consistently generated by two branches and at least one branch achieves a higher confidence than the threshold 𝜏𝜏 . Here, we compare our scheme with a separate pseudo labeling (SPL) strategy where each branch is independently constrained with high-confidence pseudo labels within the branch. The experimental results are given in Tab.6. On the '→ CK+' and '→RAF-DB' tasks, our multi-view voting scheme gains 5.05% and 3.46% notable accuracy enhancements, respectively, thus verifying its superiority for global-local knowledge interaction. Hyper-parameter Sensitivity Tests: In Eq. ( 14), we utilize hyperparameters 𝛼𝛼 , 𝛽𝛽 , and 𝛾𝛾 to balance our four losses. Here, we perform hyper-parameter sensitivity tests to determine the optimal values of the three parameters, and the results are shown in Fig. 6 .For the '→CK+' task, when 𝛼𝛼, 𝛽𝛽, and 𝛾𝛾 are respectively set as 0.1, 0.5, and 0.1, we obtain the highest accuracies. And when 𝛼𝛼, 𝛽𝛽, and 𝛾𝛾 are respectively set as 0.4, 0.5, and 0.1, the best results are gained on the '→FER-2013' task. Thus, we set 𝛼𝛼 as 0.4 for the hard dataset FER-2013 and AffectNet while 0.1 for the rest. Besides, we set 𝛽𝛽 and 𝛾𝛾 as 0.5 and 0.1, respectively. The hyper-parameter 𝜆𝜆 in Eq. ( 9) balances the sample-and cluster-level inter-domain alignment, and we try to find its optimal value. Since the cluster-level alignment only finetunes the feature space with label information, we select the value of 𝜆𝜆 from {0.01, 0.02, 0.04, 0.06, 0.08, 0.10}. As seen in Tab. 7, our model is robust to 𝜆𝜆 and when 𝜆𝜆 is set as 0.02, the best accuracies are gained. Feature visualizations: To assess the transferability of our model, we employ t-SNE visualizations to visualize the feature embeddings of different models on the '→CK+' task. As illustrated in Fig. 7 , the feature embeddings of the Baseline present a noticeable mismatch with the source domain. With domainuncertainty estimations, DUML better aligns the source domains to the target one but exhibits relatively poor intra-class compactness where the decision boundaries are not clear enough. In contrast, our LA-CMFER surpasses both Baseline and DUML, as evidenced by the formation of clusters with more distinct boundaries. This signifies the superior transferability of our approach which can effectively maintain the strong discrimination capability. Notably, more additional experiments including 'Selections of Reliability Threshold ϵ', 'Number of Source Domains', and so on are given in the Supplementary Materials. 
conclusion: 
 In this paper, we present the LA-CMFER framework to achieve the CMFER task by addressing both inter-and intra-domain shifts. We first propose a dual-level (i.e., sample-and cluster-level) interdomain alignment method to narrow the inter-domain shifts by prioritizing hard-to-align samples and sufficiently using the class attributes. Then, to tackle the intra-domain shifts, a multi-view intra-domain alignment method is introduced to promote consistency between the global and local branches. Extensive experiments have verified the superiority of our LA-CMFER.  
 
 Appendix:  


Decision: The authors made a strong case in the rebuttal and most reviewers are satisfied with the responses and clarifications provided. From the technical standpoint, the work introduces significant strides into cross-multidomain facial expression recognition by tackling the problem of domain shift from across multiple domains and internally within specific domains $-$ this provides useful insights to the research community working on this topic. The validation of the proposed method was very comprehensive and was carried out on six benchmark datasets, showcasing its robustness and generalizability. The ablation studies provided were also comprehensive, but appear quite mechanical (as with many papers these days). There wasn't a sufficient investigation into the intricacies of the alignment process, which was reduced to merely loss functions that are optimized together with other loss functions. I thought the number of source domains would be something that warrants attention in the main content of the paper but was relegated to the supplementary material. It is crucial to understand how this alignment would be impacted when the number of domains increases since optimizing across domains will be more challenging. All things considered, this is a borderline case but if possible, I would recommend accepting this paper as a poster at ACM MM.

Two points to note:
* Despite some reviewers indicating that this work is "multimodal" in nature, I disagree. The notion of "domain" is not equivalent to a "modality". It is obvious that the modality of interest in this work is image, and _only_ image (note: CK+ is from video but individual extracted frames are typically used). These images can be obtained from a variety of "domains" (separately collected datasets), as what this paper looks into. As the theme indicates priority for truly multi-modal works in ACM MM, this work may be slightly disadvantaged despite the strong reviews received. As this work is also suitable for affective computing or machine learning venues, I am not strongly against not accepting this paper.
* The original submitted format of the paper is not ACM's and the authors should have it formatted to the standard ACM paper format. It is a very simple instruction to follow.

Reviews: review: paper_summary: paper_summary: This paper proposes a Learning with Alignments for Cross-Multidomain Facial Expression Recognition (LA-CMFER) framework to tackle the inter - and intra -domain shifts. To this end, authors use a dual-level alignment and a multi-view alignment for inter-domain and intra-domain shifts, respectively.
strength_weakness: strength_weakness: 1. The paper is well-organized and clearly written.
2. The experiments are extensive and helpful to validate the effectiveness of the model.
limitations: limitations: 1. How do you ensure that the manipulation of emphasizing hard sample alignment are optimally performed to maintain a balance between removing noise from multiple labeled source domains and preserving relevant information according to decision boundaries.
2. To display the shapes and colors in the Fig.2 more clearly, the authors are suggested to add the descriptions.
3. The conclusion section is a little short, which should more explicitly summarize the main contributions.
paper_summary: paper_summary: This paper proposed an method for multi-source domain adaptive facial expression recognition. In this paper, inter-domain shift means the diverse between a source dataset and a target dataset, while intra-domain shift means the diverse between different classes in the target dataset. The proposed method designed sample- and cluster-level alignment loss to handle the inter-domain shift, and a multi-view (local and global features) voting scheme for minimizing the intra-domain shift. The results are SOTA.
strength_weakness: strength_weakness: 1. This paper is well organized and easy to follow.
limitations: limitations: 1.The words ‘cross-multidomain’ may be not accurate enough. This paper tries to transfer the emotional Knowledge from multiple source domains (datasets) to a single target domain (dataset). In domain adaptation area, this task is called multi-source domain adaptive FER.

2.The definition of domain is not clear. The meaning of 'domain' in this paper is more more like 'dataset'. In this paper, inter-domain shift is actually cross-dataset domain shift, and intra-domain shift is intra-dataset domain shift. Moreover, The images in AffectNet is diverse, which may contain data from different domains.  It will be better if the author describe this point more accurate. 

3.It is common to concurrently use the local and global features for (cross-domain) FER. However, what is the motivation of using both of them in this paper? and what is new compared to published methods that based on local and global features?

4.The idea of ‘sample-level inter-domain alignment’ is very similar with that of ECAN [1], which also reweight samples and use MMD.

5.I find the ‘intra-domain’ shift is handled on the target dataset. Is there ‘intra-domain’ shift in a source dataset? 

[1] A Deeper Look at Facial Expression Dataset Bia, IEEE TAFFC, 2020.
paper_summary: paper_summary: This paper studied the cross-multidomain problems with inter- and intra-domain shifts in facial expression recognition. They introduced a dual-level inter-domain alignment method to prioritize hard-to-align samples and foster a well-clustered feature space. Meanwhile, they also designed a muti-view clustering consistency constraint between the global and local branches to gain a concentrated target feature space for alleviating the interference of noisy samples.
strength_weakness: strength_weakness: This paper provides a novel Learning with Alignments for Cross-Multidomain Facial Expression Recognition (LA-CMFER) framework to simultaneously tackle inter- and intra-domain shifts. To address the inter-domain shifts, they use the scaling strategy to pay more attention and give higher weight to those hard-to-align samples with higher uncertainty between different domains and generate a more well-clustered feature space. As for the intra-domain shifts, they utilized a multi-view voting scheme to gain high-quality pseudo labels for an effective information interaction in the dual-level branches. The methodology in this paper looks professional, it can be seen that the theory of this paper is very solid.
limitations: limitations: There are some problems, which must be solved before it is considered for publication. If the following problems are well-addressed, the reviewer believes that the essential contribution of this paper is important for facial expression recognition.

1.  **Inconsistent Data**: In **Tabel 1**, the experimental data of **DMUL** is not consistent in their original paper[1]. It is not known whether this inconsistency is due to an error while copying or other reasons. However, DUML is a method to solve the same problem in CMFER, and we **suggest** that the results must keep correct and objective.


2.  **Inproper tuning**: In Equation (14), we can find that the final loss function consists of four parts, related to three hyper-parameters (i.e. $\alpha$, $\beta$, and $\gamma$). However,  in **Section 4.3** , we find that the sum of hyper-parameters(i.e. $\alpha$, $\beta$, and $\gamma$) of the loss function is not consistent for different target datasets (for the '→CK+' task, $\alpha$ + $\beta$ + $\gamma$ = 0.7, while for the '→FER- 2013' task, $\alpha$ + $\beta$ + $\gamma$ = 1.0). It seems that the authors did not consider the influence of these loss functions among each other, but tuned them separately. It is hard to convince us that the parameter-sensitivity experiments is proper and can strongly support your methods. The **suggestion** is that you need to consider the effects of different loss with each other and reset the hyper-parameter ablation experiments.


3.  **Complexity Analysis**: Following the paper's work, this structure may have some effectiveness in solving the cross-multidomain FER, but it also increases the complexity of the framework compared with traditional MDA methods in FER. We **advise** you to add the comparison analysis on computational costs and training time with other SOTA methods.


* * *
**Reference:**

[1] Liu H, Cai H, Lin Q, et al. Learning from More: Combating Uncertainty Cross-multidomain for Facial Expression Recognition[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 5889-5898.
paper_summary: paper_summary: This paper proposes the LA-CMFER method to address cross-multidomain in facial expressions. By handling both inter-domain and intra-domain transitions, it achieves effective CMFER. Experiments validate the effectiveness of the method.
strength_weakness: strength_weakness: 1. The proposed LA-CMFER method effectively addresses the cross-multidomain challenges in facial expression recognition.
2. Extensive experiments have validated its effectiveness.
limitations: limitations: 1. In FER, capturing subtle changes in facial expressions is crucial. The authors' focus on both local and global aspects of the face is reasonable. However, why does dividing the face into 4 fixed regions ensure the capture of subtle facial movements? Generally, facial images have different poses, making it difficult for 4 fixed local patches to consistently focus on the same facial subregion.
2. How does MPC work? The paper does not provide a detailed explanation of Eq.10.
3. Given that the proposed LA-CMFER and DUML (ACM MM’23) both address the CMFER task, please clarify the differences between the two methods.
4. Please provide a deeper discussion of the proposed method, including its 'limitations'.
5. Open-source the code.
6. The paper's descriptions need serious improvement, especially the explanation of equations such as Eq.1 and Eq.3.