Amanda: Adaptively Modality-Balanced Domain Adaptation for Multimodal Emotion Recognition (2024)

Abstract: This paper investigates unsupervised multimodal domain adaptation for multimodal emotion recognition, which is a solution for data scarcity yet remains under studied. Due to the varying distribution discrepancies of different modalities between source and target domains, the primary challenge lies in how to balance the domain alignment across modalities to guarantee they are all well aligned. To achieve this, we first develop our model based on the information bottleneck theory to learn optimal representation for each modality independently. Then, we align the domains via matching the label distributions and the representations. In order to balance the representation alignment, we propose to minimize a surrogate of the alignment losses, which is equivalent to adaptively adjusting the weights of the modalities throughout training, thus achieving balanced domain alignment across modalities. Overall, the proposed approach features Adaptively modality-balanced domain adaptation, dubbed Amanda, for multimodal emotion recognition. Extensive empirical results on commonly used benchmark datasets demonstrate that Amanda significantly outperforms competing approaches. The code is available at https://github.com/1emonx/Amanda.

heading: 1 Introduction
text: Emotion recognition has gained increasing attention in recent years in a wide spectrum of applications, including emotional support (Tu et al., 2022), conversation system (Shi and Huang, 2023) and healthcare (Zanwar et al., 2023). Multimodal emotion recognition which takes advantage of heterogeneous and complementary signals, such as acoustic, visual, lexical information, has demonstrated superior performance to its unimodal counterpart (Zhu et al., 2022; Zhang and Li, 2023). Nevertheless, one of the notable drawbacks of multimodal
*Corresponding author: Jun Sun (sunjun16sj@gmail.com).
learning is that collecting and annotating data of multiple modalities is much more expensive than one single modality (Lian et al., 2023). Thus, the importance of the ability of a model to transfer knowledge from annotated datasets to unannotated but related ones is manifested in the context of multimodal emotion recognition.
In this regard, unsupervised domain adaptation techniques are popular for promoting the generalization capability of a model from a labeled source domain to an unlabeled target domain. Domain adaptation typically fills the model’s performance gap between the target and source domains via matching the data distributions of the two domains with sample-based, feature-based and inferencebased approaches (Kouw and Loog, 2021). Accordingly, numerous schemes have been developed for various tasks in the fields of computer vision (Li et al., 2021; Liu et al., 2023) and natural language processing (Calderon et al., 2022; Dua et al., 2023). In contrast, domain adaptation in multmodal learning settings remains relatively less researched, not to mention in multimodal emotion recognition.
The previous literature on multimodal domain adaptation broadly falls into two categories, multiple visual modalities and general multiple modalities. The former tackles multimodal computer vision tasks, where different modalities correspond to RGB and optical flows (Munro and Damen, 2020), CT and MRI images (Kruse et al., 2021), or 2D image and 3D point cloud (Xing et al., 2023). In these scenarios, the modalities are similar and share the same environment, suggesting a close distribution gap between source and target domains of different modalities. Therefore, no specific effort is required to address modality differences when aligning the domains. The latter category focuses on more general multimodal domain adaptation approaches, applicable to text/image and video/audio applications. However, in these studies, the source and target domains of different modalities are aligned uniformly
14448
without recognizing modality disparity (Qi et al., 2018), or they are not considered jointly, leading to the situation where some modalities are well aligned while others not (Yuan et al., 2022).
In multimodal emotion recognition tasks, the commonly utilized modalities—linguistic, visual and acoustic, exhibit high heterogeneity. Moreover, these modalities live in decoupled spaces, as opposed to the visual modalities mentioned above. Consequently, from the source to target domains, different modalities experience varying degree of distribution shift. For example, consider a shift in the working scene of a multimodal emotion recognition system from the day (source domain) to the night (target domain). In this scenario, the distribution of visual features noticeably shifts due to the variation of illumination conditions, while that of acoustic features remains relatively unchanged.
Henceforth, directly applying existing domain adaptation approaches to multimodal emotion recognition might result in an imbalanced alignment of different modalities. The model may then rely heavily on the well aligned modalities in the target domain and under-utilize others; in other words, well aligned modalities dominate others, causing the latter to be under-trained.
With the above analysis, in this paper, we advocate modality independence (Sun et al., 2023; Qu et al., 2021) and align the source and target domains of different modalities, taking their varying distribution gaps into consideration. To be specific, we design our model based on the information bottleneck (IB) theory (Saxe et al., 2019; Kawaguchi et al., 2023), which enforces each modality to perform label prediction, thereby encouraging each to obtain its optimal representation independently. As for the domain alignment, we first introduce label distribution alignment under the practical assumption that the label distributions remain consistent across the source and target domains. We then employ correlation alignment (Sun et al., 2016; Sun and Saenko, 2016) to match the optimal representations in the two domains for each modality.
To balance the representation alignment, we minimize a surrogate of the alignment losses rather than minimizing a weighted sum of the losses with fixed weights. Via judiciously devising the surrogate function, minimizing it is tantamount to minimizing the weighted sum of the losses with the weights being adaptively tuned throughout the training progress. Concretely, the modalities with larger (resp. smaller) losses receive proportionally larger
(resp. smaller) weights, which achieves dynamically balanced domain alignment across modalities.
In summary, our work features Adaptively modality-balanced domain adaptation (abbreviated as Amanda) for multimodal emotion recognition. The contributions are primarily threefold.
• We develop a mulitmodal emotion recognition model which learns the representations of modalities independently and aligns the source and target domain via matching the representations and the labels.
• We propose a paradigm for alignment loss surrogate function design, which adaptively balances all modalities during training.
• Empirical results verify the effectiveness of the proposed method, and demonstrate that Amanda outperforms the compared schemes.

heading: 2 Related Works

heading: 2.1 Domain adaptation
text: There are an enormous number of prior works on domain adaptation, for which interested readers can refer to survey papers (Wang and Deng, 2018; Kouw and Loog, 2021; Yu et al., 2023) and references therein. We only cover the most relevant works, which can be classified into two branches, i.e., the adversarial learning methods and moment matching methods. Starting from the pioneering work DANN (Ganin et al., 2016), a vast amount of adversarial learning methods emerge. MDAN (Zhao et al., 2018) investigates domain adaptation with multiple source domains and devises two versions of optimization strategies. CDAN (Long et al., 2018), MADA (Pei et al., 2018) and CAN (Wu et al., 2021) introduce label prediction information as conditioning for domain alignment. DALN (Chen et al., 2022) proposes a novel Nuclear-norm 1-Wasserstein discrepancy (NWD) and constructs a discriminator-free adversarial model via reusing the task-specific classifier as a discriminator. CDA (Yadav et al., 2023) incorporates contrastive learning into domain adaptation to achieve class-level alignment.
As for the moment matching branch, maximum mean discrepancies (MMD) (Tzeng et al., 2014) and its variants, such as MK-MMD (Long et al., 2015), RTN (Long et al., 2016) are typical first order moment approaches which match the mean of the representations. Coral (Sun et al., 2016; Sun and Saenko, 2016) and JDDA (Chen et al., 2019)
represent second moment approaches, matching the covariance of the representations.

heading: 2.2 Balanced multimodal learning
text: Another line of relevant studies that inspire this work are devoted to balancing the convergence of different modalities to prevent some modalities being overfitting while others being underfitting. The learning rates of different modalities are dynamically regulated via tracking the label prediction losses of all modalities in studies (Sun et al., 2021; Peng et al., 2022). Work (Wu et al., 2022) proposes a scheme to estimate the model’s dependence on each modality, based on which an algorithm to balance the learning speeds of all modalities is introduced. In study (Wang et al., 2020), the overfitting behaviors of the modalities are evaluated, and accordingly, an optimal blending of gradients is computed for model updates. The relative advantage of each modality is defined during model training, with which a bi-level optimization problem is formulated to re-weight the loss terms of all modalities in work (Sun et al., 2024). These approaches are effective yet involve a delicate heuristic design in the light of some observations during training.
In our work, we adopt the second moment matching method for representation alignment, thus focusing on balancing modalities and circumventing the difficulty in balancing the competing generative and discriminative components in adversarialbased methods. We propose a paradigm for alignment loss surrogate function design, enabling adaptive balancing of alignment losses across modalities without extra effort to tune the learning rates.

heading: 3 Method: Amanda
text: Prior to delving into our method, Amanda, we introduce the notations and assumptions below. Notations: Suppose the multimodal training dataset contains N samples, each with M modalities. For ease of expression, let us define an auxiliary modality as a union of all modalities, and thus the total number of modalities is M + 1. Let [P ] for any positive integer P denote the set {1, 2, · · · , P}. The training samples are denoted by ({xn,m}m∈[M ], {yn,m}m∈[M+1]), where n ∈ [N ] indexes the samples, xn,m ∈ Rdm represents the dm-dimensional feature vector (the feature can also be vector sequence) of modality m,∀m ∈ [M ], and yn,m represents the label corresponding to modality m,∀m ∈ [M + 1] (for
datasets where all modalities share a common label, yn,1 = yn,2 = · · · = yn,M+1 holds). Suppose the number of emotion categories is C; then the label yn,m can be a one-hot vector or a scalar in [C], and we adopt either of these two forms when necessary in the rest of the paper. For the consistency of expression, we use xn,M+1 := [xn,1;xn,2; · · · ;xn,M ] to collect all features of sample n.
Let Xm and Ym represent general feature and label random variables for all m ∈ [M + 1], with xn,m and yn,m as their realizations. Let vector Zm ∈ Rd, a map of Xm , denote the representation of modalitym, and zn,m is a realization ofZm (for brevity, we assume the representations of all modalities are d-dimensional vectors). We use superscript s and t to distinguish variables of source and target domains. For instance, Xsm and X t m denote the features of modality m from source and target domains, respectively. Assumptions: In this paper we consider unsupervised domain adaptation problem for multimodal learning, for which the following assumptions are satisfied: 1) the label target domain data is inaccessible; 2) the feature distributions shift with the domains, yet the label distributions remain unchanged, meaning that p(Xtm) ̸= p(Xsm) and p(Y tm) = p(Y s m) hold for anym ∈ [M+1], where p(·) represents the distribution of a random variable. The second assumption holds true when the domain changes the feature but is not a causal factor of the considered event. For example, although the illumination (the feature) of the vision system varies between the day and the night (different domains), one’s emotion (the label) distribution remains relatively stable with the day and the night.

heading: 3.1 Model design

heading: A. Overview of model framework
text: In the considered multimodal emotion recognition task, three modalities are utilized, and thus we have M = 3. The model takes as input the features Xm,m ∈ [M ], which are extracted from the raw data of the three modalities with corresponding pretrained models. Specifically, the widely used language and audio models, BERT-base (Devlin et al., 2018) and wav2vec2 (Baevski et al., 2020), are employed to obtain lexical and acoustic features, respectively. For the visual feature extraction, we utilize a variant of Vision Transformer (ViT) (Dosovitskiy et al., 2020), named APViT (Xue et al., 2022), which emphasizes the most discrim-
inative features and discards noisy backgrounds. The adopted APViT is pretrained on the RAF-DB (Li et al., 2017) database, a specialized dataset for facial emotion recognition, aligning with the task considered in this paper. More details about the feature extraction are presented in Section 4.
Figure 1(a) visualizes the architecture of our model, Amanda, in an example with two modalities (M = 2) for simplicity. To map the feature Xm to the representation Zm, we employ a deterministic encoder fm(·;θfm) : Rdm → Rd with model parameter θfm, which means Zm = fm(Xm;θ f m), ∀m ∈ [M ]. For different modalities, fm(·) can take different forms; for instance, in our model framework, we utilize TextCNN for the acoustic and lexical modalities, and LSTM for the visual modality. Let ZM+1 := [Z1;Z2; · · · ;ZM ] concatenate the representation of all modalities. For each modalitym ∈ [M+1], an MLP gm(·;θgm) is adopted to predict the label using the corresponding representation, that is, Ŷm = gm(Zm;θ g m). The multimodal prediction ŶM+1 is admitted as the final predicted label. For ease of expression, we use θ = {θgM+1} ∪ {θ f m,θ g m}m∈[M ] to collect all model parameters.
B. IB based representation learning With the above model framework, the information chain followsXm → Zm → Ym,∀m ∈ [M + 1], as is shown in Figure 1(b). A model with good generalization performance should be able to generate representation Zm which maintains task relevant information and discards the rest in Xm. To achieve this, the labeled source domain data is uti-
lized to minimize the following information bottleneck (IB) loss:
LIB(θ) := ∑
m∈[M+1] γI(Xsm,Z s m)−I(Zsm,Y sm), (1)
where I(·, ·) represents the mutual information of two random variables, and γ is a coefficient balancing the two terms.
From the perspective of information theory, it is evident that minimizing LIB(θ) leads to a representation Zsm that retains minimal information from the original featureXsm while capturing the maximal information of the label Y sm. Henceforth, Zsm is an optimal representation in the sense of information bottleneck theory (Saxe et al., 2019; Kawaguchi et al., 2023). Moreover, not only is the joint modality M + 1 enforced to learn the task relevant information I(ZsM+1,Y s M+1), but each individual modality m, for all m ∈ [M ], is also required to maximize their corresponding I(Zsm,Y s m) even if all modalities share a common label. This promotes modality independence and prevents some weak modalities from being ’lazy’ and being dominated by strong modalities.
Next, we elaborate on how the two information terms in Eq. (1) are calculated.
I(Xsm,Z s m)= H(Z s m)−H(Zsm|Xsm)
=H(Zsm)=EZsm [− log p(Zsm)], (2)
where H(·) denotes entropy, and H(Zsm|Xsm) = 0 since Zm = fm(Xm;θ f m) is a deterministic function. Upon assuming that p(Zsm) follows Gaussian distribution N (µsm,Σsm) (µsm ∈ Rd, and Σsm ∈ Rd×d is a diagonal matrix), we can estimate µsm
and Σsm with the representations z s n,m, n ∈ [N s]. The the entropy of H(Zsm) is
H(Zsm) = 1
2 log |Σsm|+
d 2 (1 + log(2π)), (3)
where |Σsm| represents the determinant of Σsm. Similarly, I(Zsm,Y s m) can written as:
I(Zsm,Y s m) = H(Y s m)−H(Y sm|Zsm)
= HsY,m −H(Y sm|Zsm)
= HsY,m+ 1
N s
Ns∑
n=1
log p(ysn,m|zsn,m), (4)
where we use the fact that H(Y sm) = H s Y,m is a constant independent from parameter θ. Combining Eqs. (1), (2), (3) and (4), we obtain the information bottleneck loss as follows (with constant terms omitted).
LIB(θ)= M+1∑
m=1
[γ 2 log |Σsm|
− 1 N s
∑
n∈[Ns] log p(ysn,m|zsn,m)
] .
(5)

heading: C. Label alignment
text: As assumption 2) states, the label distributions of the target and source domains remain the same. We capitalize on this assumption to exploit the unlabeled target data. For the target domain sample Xtm, we can obtain its label Ŷ t M+1 = p(Y tM+1|XtM+1;θ). Although no label can be used as supervision signal for each individual target domain sample, the label distributions of the target and source domains can be aligned. The label distribution of source domain can be immediately computed from the labels as following:
p(Y sM+1) = ȳ s M+1 =
1
N s
∑
n∈[Ns] ysn,M+1. (6)
The predicted label distribution of target domain is
ˆ̄ytM+1 = 1
N t
∑
n∈[Nt] ŷtn,M+1. (7)
Label alignment (LA) is achieved by minimizing the following cross entropy loss between the target and source label distributions:
LLA(θ) = −EȳsM+1∼p(Y sM+1)[logˆ̄y t M+1]
= ∑
c∈[C] −(ȳsM+1)clog(ˆ̄ytM+1)c, (8)
where (·)c is the c-th element of the vector.

heading: D. Modality-wise representation alignment
text: We align the optimal representations of different
modalities across the target and source domains, following the idea of matching the distributions by aligning the second order statistics. In specific, we first calculate the variance of Zsm and Z t m, and denote them asCsm andC t m,∀m ∈ [M+1], respectively. Then, the representation is aligned by minimizing the following correlation alignment (CA) loss (Sun et al., 2016):
LCAm (θ) = ||Ctm −Csm||2F , (9) where || · ||F represents the Frobenius norm.
However, as mentioned above, directly applying correlation alignment to multimodal domain adaptation faces the difficulty in balancing the modalities, since the gaps between target and source distributions of different modalities vary. To this end, we propose to minimize a surrogate function of LCA(θ) := [LCA1 (θ),LCA2 (θ), · · · ,LCAM+1(θ)], h(·) : RM+1 → R. The goal of minimizing h(LCA(θ)) is to dynamically balance different modalities during the optimization procedure. The details of how to determine h(·) are postponed to the next subsection.
With the above model and loss functions, the overall model training loss follows:
L(θ)=LIB(θ)+α1LLA(θ)+α2h(LCA(θ)), (10) where α1 and α2 are the constant coefficients weighting the three loss terms which are also shown in Figure 1. In the sequel, we present our approach for the design of the surrogate function h(·).

heading: 3.2 Adaptive modality balancing
text: In this subsection, we develop a surrogate function — modality balanced alignment loss (MBAL) function h(a(θ)),∀a(θ) = [a1(θ), a2(θ), · · · , aM+1(θ)], am(θ) ≥ 0, ∀m ∈ [M+1], such that minimizing h(a(θ)) can adaptively balance the minimization of all elements of a(θ). Note that here we use a(θ) for brevity and generality, and substituting a(θ) with LCA(θ) in h(a(θ)) directly gives the alignment loss term in Eq. (10). A. A general design of the MBAL function We first propose that h(a(θ)) in general takes the following form:
h(a(θ)) = ϕ−1 ( ∑
m∈[M+1] ϕ(am(θ))
) , (11)
where ϕ(·) is a convex and monotonically increasing function, and ϕ−1(·) denotes the inverse function of ϕ(·).
Applying the chain rule of derivative, the gradi-
ent of h(a(θ)) is derived: ∇θh(a(θ))= ∑M+1 m=1 ϕ ′(am(θ)) · ∇θam(θ)
ϕ′ ( ϕ−1( ∑M+1 m=1 ϕ(am(θ))) )
= M+1∑
m=1
ψm(θ) · ∇θam(θ), (12)
where ϕ′(·) is the derivative of function ϕ(·), and ψm(θ) is defined as:
ψm(θ) = ϕ′(am(θ))
ϕ′ ( ϕ−1( ∑M+1 m=1 ϕ(am(θ))) ) . (13)
From Eq. (12), it is obvious that the gradient ∇θh(a(θ)) corresponds to the weighted sum of ∇θam(θ),m ∈ [M +1] with weight coefficient ψm(θ). For the brevity of expression, we drop the variable θ when no ambiguity occurs.
Next, we analyze the properties of ψm with ϕ(·) elaborated as broadly used convex functions. B. Two families of MBAL functions We will show that when ϕ(·) takes the form of power and exponential functions, the corresponding surrogate function h(a) is consolidated as norm and log-exp functions, respectively. The weights ψm,m ∈ [M+1] are then properly bounded and positively correlated to am, ∀am ≥ 0. This implies that with properly chosen learning rate, the convergence of the learning can be guaranteed, and meanwhile larger losses enjoy larger weights. Norm functions For any p ≥ 1, choosing ϕ(a) = ap immediately gives that h(a) = ||a||p := ( ∑M+1
m=1 a p m)1/p, which means h(a) is the p-norm
of a. Then, ψm can be attained as:
ψm = ap−1m
(∑M+1 m=1 a p m ) p−1 p
, ∀m ∈ [M+1]. (14)
Three cases come in order based on the value of p. 1) p = 1: h(a) is a direct summation of am, and ψm = 1,m ∈ [M+1] hold. This case corresponds to the imbalanced version of Amanda. 2) 1 < p < +∞: Eqs. (14) and (12) indicate that the gradient ∇θam(θ) associated with larger am is highlighted with larger weight ψm. This implies that during training, the equivalent alignment loss weights of different modalities are adaptively regulated according to the corresponding losses, which pays more attention to larger losses. 3) p = +∞: Eq. (14) reduces to ψm = 1, if m = argmaxm∈[M+1]am; otherwise, ψm = 0. Consequently, only the largest alignment loss among all modalities counts during training in terms of the gradient in Eq. (12). Log-exp functions For any t > 0, choosing
ϕ(a) = exp(ta) leads to log-exp function: h(a) = 1 t ln( ∑M+1 m=1 exp(tam)). The weight ψm writes as:
ψm = exp(tam)∑M
m=1 exp(tam) , ∀m ∈ [M+1]. (15)
Similarly, two cases follow: 1) 0 < t < +∞: Similar to the analysis of case 2) in the above norm function part, conclusion can be drawn by combining Eq. (12) and Eq. (14) that gradient-based training algorithms will "take more care of" the larger alignment losses. 2) t = +∞: This case is exactly the same as case 3 in the above norm function part. C. Theoretical properties and insights Now we present theoretical properties of the weight ψ:= [ψ1,ψ2, · · · ,ψM+1] and MBAL function h(a). Lemma 1. The norm of the weightψ satisfies (p ≥ 1, and 1/p+ 1/q = 1):
||ψ||q = 1, if h(a) = ||a||p; (16a)
||ψ||1 = 1, if h(a)= 1
t ln(
M+1∑
m=1
exp(tam). (16b)
Eqs. (16a) and (16b) can be verified via calculating the q-norm and 1-norm of ψ using ψm in Eqs. (14) and (15), respectively.
Theorem 1. The MBAL function h(a) is an upper bound of the weighted sum of am with weights ψm,m ∈ [M+1], which translates to the following inequalities: M+1∑
m=1
ψmam≤ ||a||p = h(a); (17a)
M+1∑
m=1
ψmam≤ 1
t ln(
M+1∑
m=1
exp(tam))=h(a). (17b)
Proof. For any p ≥ 1, and 1/p + 1/q = 1, the inequality (i.e., Eq. (17a)) below follows from Hölder’s inequality and Eq. (16a).
M+1∑
m=1
ψmam=ψ Ta≤||ψ||q · ||a||p = ||a||p.
Since ln(·) is a concave function, the following inequality (i.e., Eq. (17b)) is a result of Jensen’s inequality and Eq. (16b).
1 t ln(
M+1∑
m=1
ψmexp(tam)) ≥ 1
t
M+1∑
m=1
ψmln(exp(tam))
=
M+1∑
m=1
ψmam,
which finishes the proof.
To sum up, we propose a paradigm for the alignment loss surrogate function design, under which two families of surrogate functions, norm functions and log-exp functions are analyzed. Theoretical results show that with the developed approach, the representation alignment losses of different modalities are adaptively balanced during training using gradient-based algorithms. Furthermore, minimizing the surrogate function boils down to minimizing the upper bound of the weighted sum of the alignment losses, where the bounded weights always correlate positively to the losses in the training progress.

heading: 4 Numerical Results
text: Benchmark datasets: We assess our method on four widely used benchmark multimodal emotion recognition datasets, IEMOCAP (Busso et al., 2008), MELD (Poria et al., 2019), CMU-MOSEI (Zadeh et al., 2018), and MSP-IMPROV (Busso et al., 2016), which all contain acoustic, visual and lexical modalities. IEMOCAP and MSP-IMPROV are composed of dyadic conversations collected in the laboratory setting, and the latter is of higher recording quality. CMU-MOSEI gathers monologue videos from more than 1000 speakers on YouTube over various topics. MELD consists of fragments from the TV series "Friends", which contains multi-party conversations with over two participants. These datasets are collected from different scenarios and exhibit different characteristics, and hence represent different domains. Following work (Zhao et al., 2021), we select samples in the four classes— neutral, happy, sad and angry, to construct datasets for our experiments. Feature extraction: For the visual modality, we first sample each video uniformly to obtain 64 frames. Then, the frames are processed with S3FD (Zhang et al., 2017) to attain the speaker’s faces which are then fed into vision model APViT (Xue et al., 2022), resulting in 64 × 768 sequential feature. BERT-base (Devlin et al., 2018) and Wav2Vec2 (Baevski et al., 2020) are employed to extract lexical and acoustic features, respectively. To retain the feature of different levels, the outputs from the 1st, 7th, and 12th transformer blocks are concatenated as the final feature. The generated feature sequences are of dimension 2304, and their lengths are determined by the lengths of the text and audio, respectively. Baseline models: We compare our model,
Amanda, with DANN (Ganin et al., 2016), CDAN (Long et al., 2018), MADA (Pei et al., 2018), DALN (Chen et al., 2022), and CDAN-E, of which the first four are introduced in the Related Works sections, and CDAN+E is an extension of CDAN with the incorporation of entropy-aware reweighting for the domain discrimination loss. Implementation details: The multimodal emotion recognition model involves three modalities, in which we employ one-layer LSTM for visual modality, and TextCNN for acoustic and lexical modalities as study (Zhao et al., 2021). The dimension of the representations is chosen as 128. We adopt optimizer Adam with learning rate 5× 10−4, momentum coefficient (0.9, 0.999) and batch size 128 for model training. The parameter settings are γ = 5 × 10−4, α1 = 0.08; and α2 = 0.1 is selected for the comparison studies, and we will investigate how α2 impacts the model performance in the ablation studies. More details of the implementation can be found from the code in the supplementary material. Throughout this section, we use weighted F1 score as model performance metric. The reported F1 scores are obtained by averaging results from 3 repeated experiments, conducted on 2 Nvidia A100 GPUs with 40GB memory.

heading: 4.1 Comparison studies
text: We denote Amanda with 1-norm, 2-norm and ∞- norm, log-exp(t = 1) surrogate functions by AN(p = 1), A-N(p = 2), A-N(p = ∞) and AL(t = 1), respectively. Table 1 reports the F1 scores of the baseline models and different versions of Amanda. In this table, direct transfer means the target domain data is not used for training, and directly be tested with the model trained on source domain. The results indicate that Amanda with pnorm (p > 1) log-exp (t > 0) surrogate functions performs on par with the baseline models on dataset MSP → IEMOCAP; and for all other datasets, it achieves substantial improvement.
Note that some baseline models (Ganin et al., 2016; Long et al., 2018; Pei et al., 2018) are originally devised for unimodal settings, we directly conduct domain adaptation for each modality. And the multimodal feature classifier is reused as a domain discriminator in DALN (Chen et al., 2022). Therefore, the comparison results underscore that multimodal domain adaptation requires dedicated design rather than simply employing techniques from the unimodal context.

heading: 4.2 Ablation studies
text: In this part, we conduct ablative studies on the two critical designs in Amanda, the label alignment and balanced representation alignment. Figure 2 illustrates the model performance with varying weight (α2) of the surrogate functions. Comparing Amanda without correlation alignment (A. w/o CA) to direct transfer (D.T.), it is clear that label alignment enhances the knowledge transferring capability of the model significantly. For weight α2 ≤ 0.2, Amanda with p-norm (p > 1) and log-exp surrogate functions can further improve the model performance over Amanda without CA (A. w/o CA). Particularly, when weight α2 grows larger than 0.2, the balanced versions of Amanda, A-N(p = 2), A-N(p = ∞) and A-L(t = 1), experience less performance drop compared to its imbalanced counterpart, A-N(p = 1). The above results corroborate that both the label alignment and the adaptive domain alignment contribute to the success of Amanda.
In order to demonstrate that the proposed p-norm surrogate functions (p > 1) are able to balance
the domain alignment of different modalities, we show the normalized alignment losses (L̃CAm (θ) := LCAm (θ)/ ∑M m=1 LCAm (θ)) in Figure 3, where the
target and source datasets are IEMOCAP and MSP, respectively. Consistent with our analysis in section 3.2, the losses are not balanced with p = 1, and hence the losses exhibit large discrepancy among modalities throughout the training, as illustrated in Figure 3(a). In contrast, with p = 2 as shown in Figure 3(b), the losses are adaptively balanced, leading to closer gaps among modalities (when the three normalized losses are all 1/3, perfect balance is achieved). Figure 3(c) displays the case of p = +∞, where the losses are also more balanced than that of the case p = 1. These results verify that the proposed surrogate functions succeed in balancing the domain alignment of different modalities.

heading: 5 Conclusions
text: In this work, we devise a multimodal domain adaptation approach for multimodal emotion recognition. In order to close the gap between the target and source domains, we propose to match the label
distributions of the two domains and to align the optimal representations for different modalities. Towards the objective of balancing the representation alignment, a general alignment loss surrogate function design paradigm is developed. Furthermore, we present the theoretical analysis of two families of surrogate functions which achieve adaptively modality-balanced domain adaptation. The effectiveness of the proposed approach is corroborated by extensive comparison and ablation studies.

heading: 6 Limitations
text: In light of the future work, the limitations of the present work are mainly twofold. 1) Although our method is applicable to more general multimodal supervised learning problems, we only validate it on emotion recognition tasks. 2) We have not established the theoretical upper bound of the target domain risk for the proposed approach.

heading: Acknowledgements
text: The work by Jun Sun is supported by the National Natural Science Foundation of China (Grant No. 62306289).

heading: Future Work
text: In light of the future work, the limitations of the present work are mainly twofold. 1) Although our method is applicable to more general multimodal supervised learning problems, we only validate it on emotion recognition tasks. 2) We have not established the theoretical upper bound of the target domain risk for the proposed approach