Learning in Order!   A Sequential Strategy to Learn Invariant Features for Multimodal Sentiment Analysis

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 13:59:59

Abstract: This work proposes a novel and simple sequential learning strategy to train models on videos and texts for multimodal sentiment analysis. To estimate sentiment polarities on unseen out-of-distribution data, we introduce a multimodal model that is trained either in a single source domain or multiple source domains using our learning strategy. This strategy starts with learning domain invariant features in text, followed by learning sparse domain-agnostic features in videos, assisted by the selected features learned in text. Our experimental results demonstrate that our model achieves significantly superior performance than the state-of-the-art approaches in both single-source and multi-source settings. Our feature selection procedure favors the features that are independent to each other and are strongly correlated with their polarity labels. To facilitate research on this topic, the source code of this work will be publicly available upon acceptance.

Main paper: 
 introduction: 
 Multimodal Sentiment Analysis (MSA) is concerned with understanding people's attitudes or opinions based on information from more than one modalities, such as videos and texts. It finds rich applications in both industry and research communities, such as understanding spoken reviews of target products posted on YouTube and developing multimodal AI assistants for mental health support. Prior MSA approaches make an impractical assumption that training and test data comprise independent identically distributed samples [58, 64, 76, [79] [80] [81] . However, training datasets are available only for a handful of applications that satisfy that assumption. Therefore, this work aims to remove the assumption such that MSA models trained on a single domain or multiple source domains can work robustly on unseen out-of-distribution (OOD) data, without leveraging any target domain data.To enable models to work robustly across domains, a key idea is to exploit domain invariant sparse representations, which serve as causes of target labels from a causal perspective [63, 67] . In contrast, spurious correlations, which do not indicate causal relations, impede the generalization capability of pre-trained foundation models [6, 25] . Existing MSA models heavily rely on jointly learned multimodal features for sentiment analysis [27] . However, the spurious features of the visual modality may adversely affect the features of the text modality, leading to inaccurate prediction outcomes [23, 24, 80] . Therefore, it would be interesting to investigate i) how to automatically identify domain invariant representations for MSA, and ii) what are the key characteristics of domain invariant features in a multimodal setting.To answer the above research questions, as illustrated in Figure 1 , we propose a Sequential Strategy to Learn Invariant Features (S 2 LIF) for building a domain generalization (DG) MSA model based on videos and texts. Instead of learning domain-agnostic features simultaneously from all modalities, our technique first leverages the sparse masking technique [34] to select invariant hidden features from texts, followed by learning the invariant features from videos, conditioned on the selected textual features. To the best of our knowledge, it is the first time to report the importance of feature learning order for domain generalization. We conduct extensive experiments to i) demonstrate the superiority of our approach in comparison with the competitive baselines in both single source domain and multi-source domain settings, and ii) investigate key characteristics of selected features using our approach. Our key contributions are summarized as follows:• We introduce a novel domain generalization MSA model, which explicitly learns domain-invariant features and mitigates spurious domain-specific features by adopting a sparse masking technique. • We propose a new sequential multimodal learning strategy, which extracts the domain-invariant features from texts first, followed by employing them to identify the features relevant to labels from videos using the sparsity technique. • We demonstrate empirically that i) our sequential multimodal learning strategy prefer selecting the domain invariant features in the visual modality, which are independent of the selected features in the text modality and strongly correlate with labels; and ii) it is important to adhere to the learning order of our approach to mitigate spurious correlations, because it is evident that the alternative learning order or learning all modalities simultaneously using the same sparsity technique leads to inferior performance. 
related work 2.1 multimodal sentiment analysis: 
 MSA methods can be roughly divided into two categories: 1) Multimodal Representation Learning aims to learn fine-grained multimodal representation, which provides rich decision evidence for multimodal sentiment prediction. They employ a disentangled technique to learn modality-common and modality-specific representations to mitigate the heterogeneity of multimodal representations [28, 65, 71, 74] . 2) Multimodal Fusion aims to learn cross-modal information transfer by designing complex cross-modal interactive networks. The development of multimodal fusion methods has evolved from multi-modal tensor fusion [76] to cross-modal attention [44, 45, 64, 73, 77, [79] [80] [81] [82] . The current MSA methods only train and test on a specific domain, and do not consider the generalization ability of the model. They suffer performance degradation when tests on out-of-distribution data, so learning robust MSA models is essential.Domain generalization aims to design a deep neural network model that learns domain-invariant features and is able to maintain stable performance in both the source domain and multiple unseen target domains. Numerous domain generalization methods have been proposed to learn domain-invariant features for single-source or multi-source domain generalization [5, 12-18, 22, 26, 30, 31, 37, 38, 40, 48, 51, 54, 59, 72] . We roughly divide current methods of domain generalization into three categories: 1) Learning invariant features aims to capture the domain-generalized features to reduce the dependence of features on specific domains and to achieve high performance on unseen domains [48] . 2) Optimize algorithm aims to learn domain-invariant features and remove domain-specific features [5, 10, 22, 38, 52, 59] , such as adversarial training and metalearning, through tailored designed network structures. 3) Data augmentation aims to generate new data to improve the generalization performance of the model, and these generated new data are out-ofdistribution samples different from the source domain [66, 68, 69] .From the perspective of data generation, causal representation learning considers that raw data is entangled with two parts of features: correlated features with label (domain-invariant features) and spuriously correlated features with the label (domain-specific features). The goal is to disentangle domain-invariant features and domain-specific features. Domain-invariant features guarantee stable performance in different test environments [1, 49] . Based on this assumption, numerous methods attempt to learn domain invariant features [2, 3, 11, 32, 56] . Following previous work, our proposed approach aims to learn the domain-invariant features (i.e., the features correlated with the label), while removing the features domain-specific (i.e., the spuriously correlated features with the label). Concretely, we adopt sparse techniques to remove spuriously correlated features with the label [36, 43, 47, 60] . , where X 𝑖 {𝑡,𝑣 } ∈ R 𝑑 , 𝑦 ∈ R 𝐾 , while D 𝑇 is a dataset in a target domain. The goal is to learn multimodal domain-invariant features for sentiment polarity prediction in unseen domains without using target domain data for training.Model Overview. Our work is motivated by the functional lottery ticket hypothesis [41] suggesting that there is a subnetwork that can achieve better out-of-distribution performance than the original network. Hence, We employ the sparse masking techniques to identify a subset of hidden features in the multimodal setting. The findings of our empirical studies indicate the importance of the learning order between modalities for domain generalization performance.The architecture of our model is illustrated in Figure 2 . Given a text and a sequence of video frames X {𝑡,𝑣 } , we employ a pre-trained encoders ELECTRA [19] and VGGFace2 with a 1-layer Transformer encoder [21] to map them to respective hidden representations 𝑥 𝑡 and 𝑥 𝑣 . To achieve sparsity in hidden representations, our model generates a mask vector 𝑚 {𝑡,𝑣 } with the mask function 𝑓 𝑚𝑎𝑠𝑘 to select domain-invariant features 𝑥 𝑐 {𝑡,𝑣 } from X {𝑡,𝑣 } . The mask function is characterized by the learnable parameter 𝑟 {𝑡,𝑣 } and threshold 𝑠 {𝑡,𝑣 } . The feature selection in a modality is achieved by computing the dot-product between the mask vectors and the corresponding hidden representations. We empirically find that text is the superior modality in comparison with videos based on their performance in each modality. Our further studies show that conditioning on the strong text features reduces the selection of visual features that correlate with those text features. On the one hand, reduction of statistical dependencies between features leads to improvement of generalization performance. On the other hand, selection of features adhere to the functional lottery ticket hypothesis. Therefore, our text classifier 𝑔 𝑡 first selects the key features using the masking technique, followed by learning sparse video representations for the visual classifier 𝑔 𝑣 to predict sentiment polarity conditioned on the selected text features. In addition, our model leverages prior information from video frames to eliminate redundant frames.Keyframe-aware Masking. Given that there is a large amount of frames in a video clip, which contains redundant information [47, 60] . The frame sequence x𝑣 of a video clip contains rich priors, which explicitly correspond to neighboring frames. We can easily obtain the motion of the video frame sequence to guide the masking of redundant frames according to the temporal difference. We employ global and local neighbor frames to select informative frames 𝑥 𝑣 and constrain the semantic invariance of video frames by reconstructing losses L 𝑟𝑒𝑐𝑜𝑛 . Note that this part is not our main contribution, and it is an extension of previous work [47] . See supplementary materials for details.Step 1: Visual frame features𝑥 # ∈ 𝑅 "" Mask Vector m Selected visual frame features 𝑥 # $ ∈ 𝑅 "" Learnable Threshold s Learnable Mask r 𝑔 ) Attention … Keyframe-aware Mask Global & Local Motion Dynamic Frame Mask ℒ %&*+, 𝑔 ! ℒ ! "#$%"& ℒ ' "#$%"& 𝑃(𝑌|𝑥 ! $ # ) 𝑃(𝑌|𝑥 # $ # , 𝑥 ! $ # ) ℒ ! () ℒ ' () 𝑓 )𝑓 ! : Text feature extractor.: Visual feature extractor.ℒ ) "#$%"& + ℒ ) '( + ℒ %&*+,Step 2:Learning Objective: Sequential Multimodal Learning. The selection of domain invariant features is also motivated from a causal perspective. The logit of the classifier is computed as the product between the features 𝑥 and the weights 𝑊 of the label 𝑘 from the classification layer 𝑔.ℒ ! "#$%"& + ℒ ! '( + ℒ ) "#$%"& + ℒ ) '( +ℒ ) %&*+, Learning Sparse Text Features Conditional Learning Sparse Visual Features 𝑔 ) 𝑔 ! : Text Classifier. : Visual Classifier. 𝑥 ) 𝑥̅ ) Y Y 𝑥 # $ # ∈ 𝑅 "" Mask Mask Mask Mask C Condition (domain-invariant text features) 𝑓 !"#$ (. ) 𝑓 !"#$ (. )EQUATIONwhere the subscripts 𝑗 and 𝑘 denote 𝑗-th feature and 𝑘-th class respectively. For all polarity labels with both modalities, we obtain a matrix 𝑅 as follows, where each elementEQUATIONBy analyzing the matrix 𝑅, we conclude that 1) 𝑌 is the result of feature 𝑥 estimated via a classifier. From the causal perspective, the selected features can be seen as the causes of 𝑌 subjecting to independent noise [29, 50, 62] :EQUATIONwhere the notation 𝑃𝑎(𝑌 ) denotes the features of direct causal effects with Y, where 𝑃𝑎(𝑌 ) is a subset of 𝑥. The function 𝑔 represents the classifier. The multimodal features 𝑥 are divided into two subsets, domain-specific features 𝑥 𝑠 (spurious correlated features with the label across domain) and domain-invariant features 𝑥 𝑐 (correlated features with the label across domain) [55] . We use three features {𝑥 1 , 𝑥 2 , 𝑥 3 } to explain the causal relationship between 𝑥 and 𝑌 . As shown in Figure 3 (a), the outcome 𝑌 is specified as 𝑌 = 𝑔(𝑥 1 , 𝑥 3 ) + 𝜖, {𝑥 1 , 𝑥 3 } ⊆ 𝑥 𝑐 . The feature 𝑥 3 is the subset of 𝑥 𝑠 . There exist two distinct relationships between the feature sets 𝑥 𝑠 and 𝑥 𝑐 : a) there is no direct causal relationship between 𝑥 3 and 𝑥 1 . b) there is a direct causal relationship between 𝑥 3 and 𝑥 2 . We remove the edge between 𝑥 2 and 𝑥 3 to eliminate the impact of 𝑥 3 on 𝑥 2 . Therefore, our goal is to identify the features 𝑥 𝑐 and remove the features 𝑥 𝑠 . Formally, we expectEQUATIONwhere the features {𝑥 𝑐 𝑖 , 𝑥 𝑐 𝑗 } ⊆ 𝑥 𝑐 are selected mutual independent domain-invariant features [57] . We design learnable masks 𝑚 and learnable threshold 𝑠 in Section 3 to set the values of domain-specific features in 𝑥 𝑠 𝑘 to 0. Removing the features 𝑥 𝑠 𝑘 ⊆ 𝑥 𝑠 eliminates its direct causal effects on (𝑥 𝑐 𝑖 , 𝑥 𝑐 𝑗 ) and the outcome 𝑌 . 2) simultaneously optimizing such entangled features 𝑥 = {𝑥 𝑐 , 𝑥 𝑠 } for both text and visual modalities (i.e., imbalanced multimodal features) poses a special challenge for the classifier [23, 24] . 𝑥 ! 𝑥 " 𝑥 # U ∈ Y 𝑥 ! 𝑥 " 𝑥 # U ∈ Y ✕ ✕ (a) (b)Figure 3 : (a): The causal structure of the data generation process involves direct causal effects from 𝑥 1 and 𝑥 2 to 𝑌 . There exists a causal relationship between 𝑥 2 and 𝑥 3 . 𝜖 represents independent noise. The latent variable 𝑈 serves as a confounder for 𝑥 1 and 𝑥 3 . (b) Severing the edge between 𝑥 2 and 𝑥 3 and eliminating the causal relationship.Our sequential learning strategy is also motivated by curriculum learning [8, 46, 84] that we learn the features first, which perform well on the target tasks, followed by more challenging ones.By analyzing the causal relationship and removing spurious correlation features using multimodal learnable masks, we can obtain a new evidence matrix 𝑅 𝑀 . The form of the new evidence matrix 𝑅 𝑀 for the classifier is as follows: R 𝑀 =                           𝑤 𝑡 {1,1} 𝑥 𝑡 1 𝑚 𝑡 1 . . . 𝑤 𝑡 {1,𝐾 } 𝑥 𝑡 1 𝑚 𝑡 1 𝑤 𝑡 {2,1} 𝑥 𝑡 2 𝑚 𝑡 2 . . . 𝑤 𝑡 {2,𝐾 } 𝑥 𝑡 2 𝑚 𝑡 2 h h h h h 𝑤 𝑡 {3,1} 𝑥 𝑡 3 𝑚 𝑡 𝑖 . . . h h h h h 𝑤 𝑡 {3,EQUATIONwhere {𝑚 𝑡 , 𝑚 𝑣 } denotes mask vector in Section 3. The red notation 𝑚 𝑡 𝑖 and 𝑚 𝑣 𝑗 represent learnable mask to select domain-invariant feature with two stages in the above equations. By analyzing the evidence matrix 𝑅 𝑀 of the classifier 𝑔 and the direct causal effect with outcome 𝑌 , we can utilize the learnable mask and threshold to sequential select domain-invariant features 𝑥 𝑐 and remove domainspecific features 𝑥 𝑠 .Multimodal Learnable Masks. Regarding how to automatically identify domain invariant representations for MSA, we design multimodal learnable masks to select features. Specifically, to remove domain-specific features, we tailor a function, denoted as 𝑓 𝑚𝑎𝑠𝑘 . The inputs of 𝑓 𝑚𝑎𝑠𝑘 consists of the features 𝑥 from a modality, a learnable parameter 𝑟 , and a dynamic threshold 𝑠. The output is domain-invariant features 𝑥 𝑐 .EQUATIONwhere we apply the mask vector 𝑚 ∈ R 𝑑 (consisting of zero and non-zero value) on the feature 𝑥 ∈ R 𝑑 . The mask vector 𝑚 is obtained by utilizing a trainable pruning threshold 𝑠 ∈ R 𝑑 and a learnable parameter 𝑟 ∈ R 𝑑 . Given a set of features 𝑥, our method can dynamically select features using mask vector 𝑚. We utilize the unit step function F (•) to produce mask vector, which takes the learnable parameters 𝑟 and thresholds 𝑠 as input and output the binary masks 𝑝. Formally,EQUATIONwhere the binary mask 𝑝 and mask vector 𝑚 are obtained by:EQUATIONEQUATIONEQUATIONwhere 𝑥 𝑐 represents domain-invariant features, which remove spuriously correlated features and retain the correlated features with the label 𝑦 in training stage. It was unable to complete end-toend training during model training. The reason is that the binary mask produced by our unit step function is non-differentiable. To overcome this issue, previous works [33, 61, 83] based on straightthrough estimator (STE) [7] to estimate derivatives and design binarization function that can be back-propagatation. [70] give more approximate estimates than STE to handle non-differentiable scenarios. Using this derivative estimate to approximate the unit step function allows the model to train end-to-end.EQUATIONTo encourage the model to learn sparse features, we add a sparse regularization term [34] to the threshold as one of the training objectives. Formally,EQUATIONwhere the regular term 𝑒𝑥𝑝 (-𝑠 * 𝑖 ) raises the value of the dynamic threshold 𝑠, so that a few feature values can exceed the threshold to learn more sparse features. We utilize the function 𝑓 𝑚𝑎𝑠𝑘 to obtain domain-invariant features of textual and visual tokens. Formally,EQUATIONwhere the definitions of 𝑓 𝑚𝑎𝑠𝑘 , 𝑟 * , and 𝑠 * are specified in Equation 6∼ 10.Apart from learning the domain-invariant features of each token, we also calculate the similarity between each token and the learnable mask to learn domain-invariant tokens. Formally,EQUATION)EQUATIONwhere 𝑥 EQUATIONwhere ';' denotes concatenation along the feature dimension.Learning Objective. Initially, we employ the classifier to learn domain-invariant features from the text modality (i.e. text modality). Formally,EQUATIONSubsequently, we utilize the domain-invariant features from the text modality to assist in selecting domain-invariant features from the visual modality. Formally,EQUATION)where the symbol L 𝐶𝐸 * denotes Cross-Entropy loss and 𝛼 is hyperparameter. Accordingly, the overall learning objective is:L = L 𝑡 + L 𝑣 (19) 
experiments 4.1 datasets: 
 We select three typical MSA benchmark datasets: CMU-MOSI [78] , CMU-MOSEI [79] and MELD [53] . The detailed partition of the dataset is included in the supplementary materials.We employ text pre-trained language model Electra [19] and visual pre-trained model VGG Face2 [9] , extracting features from both textual content and video frames. We use a multilayer perceptron to unify the multimodal feature dimensions and a 1-layer Transformer encoder [21] to model the multimodal data of the sequence. The batch size and epoch are set to 16 and 200, and the learning rate is configured to 7e-5. Warm up epoch is 3. Our implementation is executed using the PyTorch framework with Adam optimizer [35] on the V100 GPU.We select the state-of-the-art model in the field of MSA, MLLM and DG (OOD) as the Baseline.MULT [65] designs a Multimodal Transformer to align multi-modal sequential data and capture cross-modal information interaction.ALMT [80] employs non-verbal modalities to reinforce the features of the text modality several times and dismisses the non-verbal information after completing the reinforcement process. MAD [54] designs a two-stage learning strategy, learning domainspecific and domain-invariant features respectively to constrain the two features by regular terms. RIDG [17] aligns the labels of each class with the classified evidence to ensure the domain generalization of the model. MLLM. We selected five multimodal large language models, including Blip-2 [39] , InstructBlip [20] , [42] and Qwen-VL [4] with excellent performance from the benchmark [75] of the multimodal large model as the baseline.The distribution of the dataset is approximately balanced. We evaluate the model performance using a 3-class accuracy metric, specifically [Positive, Neutral, Negative]. Overall Comparisons. To justify the effectiveness of our proposed S 2 LIF model, we compareed the model with the following state-ofthe-art baseline in the filed of MSA and DG. Models that focus on capturing cross-modal dependencies, called MulT and ALMT. Models that aims to learn domain-invariant features, namely, MAD and RIDG. Tables 1 and 2 show the results of the comparison. By analyzing these two tables, we draw the following conclusions: i) The MSA method shows visual performance in the unseen domain. With the addition of our multimodal learnable masks, the traditional models also gain the ability of DG. The fact demonstrates the effectiveness of sparse mask in DG. ii) Our model significantly outperforms the multimodal large model in 4 of the 6 Settings. We speculate that there is contamination from emotional datasets during the training phase of the multimodal large language model. In the two settings with better performance, the logits of InstructBlip for correctly predicted samples exceed 0.93, significantly higher than the logits generated by other multimodal large language models, which are around 0.65. iii) The model performance in sequential multimodal learning is better than that in non-sequential multimodal learning when we distinguish text and visual modalities. This demonstrates the effectiveness of the sequential multimodal learning strategy.Existence of Domain-invariant Features. An essential assumption in our study is the presence of domain-invariant features in crossdomain multimodal data. To gain insight into this assumption, we visualized the selected and removed features for each domain using a heatmap. We marked positions with '1' where the features are consistently selected across domains. From Figure 4 , we could conclude that there is a presence of domain-invariant features across multiple domains, and our proposed model can automatically select the domain-invariant features. Moreover, we visualized the proportion of features retained during the training phase. Figure 5 Table 1 : The performance (accuracy of 3-classification) of single-source domain generalization. The symbols 𝑉 ,𝑇 , and 𝑀 denote using visual, textual, and multimodal features, respectively. The symbols 𝑇 → 𝑉 and 𝑉 → 𝑇 indicate the multimodal learning order. 𝑇 &𝑉 denotes simultaneous learning. 'Frozen' and 'Fine tuning' represents freezing and fine-tuning the parameter of pre-trained language model. We train the model on the source domain and infer on both the source and target domains. Cross-modal Feature Correlation Analysis. Apart from the the superior performance, the key advantage of our proposed model compared to other models is that its sequential multimodal learning.from the text modality. The features of these visual modalities prefer mutually independent from the features of the text modality, allowing the information learned from the visual modalities to complement that of the text modality. For each domain-invariant feature 𝑥 𝐶 ′ 𝑡 𝑖 ⊂ 𝑥 𝑐 ′ 𝑡 from the text modality, we employed Fisher's z-test to calculate the ratio of features in the domain-invariant feature set 𝑥 𝑐 ′ 𝑣 of the visual modality that are independent and dependent of that specific feature 𝑥 𝑐 ′ 𝑡 𝑖 . From Figure 6 , we could see that, conditioning on the text modality, the model exhibits a higher proportion of independence among domain-invariant features across modalities. These results demonstrate that our proposed sequential multimodal learning strategy in Equation ( 4) is capable of learning more effective, sparse, and independent cross-modal features.Intra-modal Feature Correlation Analysis. To valid the independence among the learned domain-invariant features, we conducted Fisher's z-test on the features in the Multi-source setting A with intra-modality. Specially, we selected 𝑥 𝑐 ′ 𝑗 from the domain-invariant feature set 𝑥 𝑐 ′ and computed the ratio of features in the set that are independent and dependent of 𝑥 𝑐 ′ 𝑗 . From Figure 7 , we could be observed that, for the learned domain-invariant feature set, the proportion of features that are independent of any other feature in the set is significantly higher than the proportion of features that are dependent. This observation substantiates our assumption in Equation ( 4) that the combination of the learnable mask and classifier can effectively learn sparse and independent features.Correlation Analysis Between Features and Label. To demonstrate the effectiveness of sequential multimodal learning, we also employed Fisher's z-test to analyze the correlation between the learned domain-invariant features and labels. From Figure 8 , we could observe that sequential multimodal learning is capable of capturing more features that are dependent with labels. The removed features exhibit independent with the labels. This experimental result validates the efficacy of sparse masks for feature selection and the effectiveness of sequential multimodal learning (as described in Equation ( 3) and (4).Ablation Studies. To gain the insights into our sequential multimodal learning strategy, We compare our model with the following variants: 1) Reordering sequence learning, including 𝑇 → 𝑉 , 𝑉 → 𝑇 , 𝑇 &𝑉 , where they respectively denote sequential multimodal learning with textual modality as the condition, with visual modality as the condition, and simultaneous learning of textual and visual modality. 2) Add Noise, introducing noise by replacing domain-invariant features with random noise as evidence for the classifier. 3) Using DS, utilizing domain-specific features as evidence for the classifier. 4) w/o key-frame mask, eliminating the Anonymous Authors Submission Id 2340The correlation between each text domain-invariant features and visual domain-invariant feature set by 𝑇 → 𝑉 strategy.The correlation between each text domain-invariant features and visual domain-invariant feature set by 𝑇&𝑉 strategy.The correlation between each text domain-invariant features and visual domain-invariant feature set by 𝑉 → 𝑇 strategy.Ratio % Ratio % Ratio %  
conclusion: 
 In this paper, we design a sequential multimodal learning strategy to learn cross-domain invariant features for MSA. Specifically, we first employ learnable masks and classifiers to learn the invariant features from texts, and then select the invariant features of videos, conditioned on the selected text features. The experiment demonstrates the efficacy of our model in both single-domain and multi-source domain settings. Based on extensive experiments, we conclude that i) the learning order between modalities is important for domain generalization performance, and ii) our learning strategy prefers the selection of features that are statistically independent to each other, in particular between modalities.In the future, we will consider including more modalities, such as audio modality, to analyze the correlation between cross-modal invariant features in cross-domain scenarios.  
 
 Appendix:  


Decision: The paper received mixed scores. Some reviewers incresased the evaluation to BA after rebuttal as some concerns have been cleared. The average score is on the positive side, however there still are concerns.

Reviews: review: paper_summary: paper_summary: The paper titled "Learning in Order! A Sequential Strategy to Learn Invariant Features for Multimodal Sentiment Analysis" introduces a novel and straightforward sequential learning strategy for training models on videos and texts for multimodal sentiment analysis (MSA). This approach aims to estimate sentiment polarities on unseen out-of-distribution data by training a multimodal model in either a single source domain or multiple source domains. The strategy involves first learning domain-invariant features from text, followed by learning sparse domain-agnostic features from videos, guided by the features learned from text. The experimental results show that this model significantly outperforms state-of-the-art methods in both single-source and multi-source settings. Additionally, the paper highlights a feature selection procedure that favors features independent of each other and strongly correlated with their polarity labels. The authors also commit to making the source code publicly available upon acceptance of the paper.
strength_weakness: strength_weakness: This paper is notable for its innovative sequential learning strategy, which emphasizes the importance of the learning order between modalities for domain generalization performance. The theoretical approach is sound, as it effectively integrates domain-invariant feature learning from text with domain-agnostic feature learning from videos, leading to a more robust MSA model. The comprehensive evaluation across single-source and multi-source settings demonstrates the model's effectiveness and generalizability. The clarity of the methodology, supported by extensive experimental validation, underscores the technical correctness and practical applicability of the approach. Furthermore, the paper provides a detailed feature selection process that enhances the model's performance by focusing on statistically independent features, which is a novel contribution to the field. The commitment to open-source the code further adds to the paper's value by facilitating future research and development.
limitations: limitations: Despite the strengths, the paper has several limitations that need addressing. Firstly, the presentation of the content is somewhat difficult to follow, making it challenging for readers to grasp the core contributions and methodology. Figures 4, 6, 7, 8, and 9 are too small and nearly illegible in the printed version, which hampers the reader's ability to understand the experimental results fully. Moreover, while the bibliography is extensive, spanning two pages for a ten-page paper seems excessive and could be streamlined for better readability. There is also a typo at the beginning of paragraph 4.5 ("we compared"), which needs correction. Lastly, although the paper is interesting and makes a valuable contribution, it would benefit from a more thorough discussion of its limitations, such as potential scalability issues or the impact of different modalities not yet explored, like audio. Given these points, the article is accepted with revisions as outlined above.
paper_summary: paper_summary: The authors address a very interesting task with an innovative architecture. In detail, it proposes a new sequential learning strategy for multimodal sentiment analysis, training models on videos and texts, with better performance than state-of-the-art methods, using domain-invariant and sparse domain-agnostic features.
strength_weakness: strength_weakness: The proposed architecture seems innovative, applies special ‘attention’ mechanisms and succeeds in achieving excellent results in a multimodal context
limitations: limitations: The structure of the paper must be improved and some details of the supplementary material must necessarily be added to the main paper. Below are some notes to improve the paper:
- To get an overview of how the paper is organised, the structure of the paper could be added at the end of the introduction section.
- The section on the method requires several changes and improvements in terms of description in order to understand well the proposed approach. For example, what ‘n’ (not ‘N’) represents on page 2, line 183 (method section). Check and improve the whole document in this respect.
- The details of the dataset and its partition information (trainset (%), validation set (%) and test set (%)) should be added in the main document!
paper_summary: paper_summary: The paper proposes a sequential learning strategy to train models on video and texts for multimodal sentiment analysis.
strength_weakness: strength_weakness: The article is clear and well written.
limitations: limitations: The concept of multimodal sentiment analysis leveraging sequential transfer learning is well-established in the field, and numerous studies have already explored this approach. Given this context, the primary shortcoming of the presented paper lies in its lack of novelty. While the paper might offer some incremental improvements or variations on existing methods, it does not introduce a fundamentally new concept or approach that significantly advances the state-of-the-art in multimodal sentiment analysis. Moreover, the reliance on sequential transfer learning, while effective, has been extensively documented and utilized in prior research, diminishing the impact of its use in this context. In terms of experimental results, Table 1 shows better results for single-source settings A and single-source settings B, but not in single-source settings C where other method return best experimental results. In Table 2 in multi-source settings C there are evident better performances but in multi-source settings A and B other methods compared with the one proposed in the paper, return better results in the source domain. Overall, the experimental results demonstrate good performance, making the article acceptable despite its lack of novelty. The findings suggest that the methods employed are effective and yield satisfactory outcomes, contributing positively to the body of research in multimodal sentiment analysis. While the approach itself may not be groundbreaking, the good results indicate that the techniques used are robust and reliable. This level of performance ensures that the paper holds value and relevance within the field, justifying its acceptance.
paper_summary: paper_summary: This work proposes a novel and simple sequential learning strategy to train models on videos and texts for multimodal sentiment analysis.
strength_weakness: strength_weakness: I'm very sorry, but I don't have enough time to review your article in detail. I think the article is well written, so I give it a higher rating.
limitations: limitations: I'm very sorry, but I don't have enough time to review your article in detail. I think the article is well written, so I give it a higher rating.
paper_summary: paper_summary: The paper proposes a novel and simple sequential learning strategy to train models on videos and texts for multimodal sentiment analysis. This strategy starts with learning domain invariant features from texts, followed by learning sparse domain-agnostic features from videos, assisted by the selected features learned in texts. This paper introduces a multimodal model that is trained either in a single source domain or multiple source domains using our learning strategy. The experimental results demonstrate that the multimodal model achieves significantly better performance than the state-of-the-art approaches on average in both single-source and multi-source settings.
strength_weakness: strength_weakness: The advantage of this article is that it proposes a learning strategy to learn domain-invariant text features and visual features separately to improve domain generalization. The idea is simple and intuitive.  Experimental results verify the performance of the method.
limitations: limitations: This idea is more like a specific strategy, and some loss functions are also designed to improve performance. The key idea of this idea is to first learn the invariant features from texts, and then select the invariant features of videos, conditioned on the selected text features. It emphasizes that the learning order between modalities is important (Title: Learning in Order! A Sequential Strategy to Learn Invariant Features for Multimodal Sentiment Analysis) and learning all modalities simultaneously leads to inferior performance, which is very confusing and has no experimental evidence. The article states in the conclusion that the proposed learning strategy tends to select statistically independent features, especially between modalities, and there is no reasonable explanation for this. Therefore, the plausibility of this article is unclear.

Detailed comments:

1.	In Section 3 (Method), the role of each learning objective is not well explained.
2.	The various strategies in Section 3 (Method) are a bit messy, and the reading does not clearly reflect which strategy improves which part of the model performance.