Cross-lingual Multimodal Sentiment Analysis for Low-Resource Languages via Language Family Disentanglement and Rethinking Transfer
Authors: Long Chen, Shuoyu Guan, Xiaohua Huang, Wen-Jing Wang, Cai Xu, Ziyu Guan, Wei Zhao
Accepted at: acl25

Cross-lingual Multimodal Sentiment Analysis for Low-Resource Languages via Language Family Disentanglement and Rethinking Transfer

Long Chen 1†, Shuoyu Guan 1*, Xiaohua Huang2, Wen-Jing Wang1, Cai Xu3, Ziyu Guan3, Wei Zhao3
1Shaanxi Key Laboratory of Information Communication Network and Security, Xi’an University of Posts and Telecommunications, Xi’an, China
2School of Information Science and Technology, Northwest University, Xi’an, China
3School of Computer Science and Technology, Xidian University, Xi’an, China
Correspondence: {chenlong@, shuoyuguan@stu., wjing@ }xupt.edu.cn,
huangxiaohua@stumail.nwu.edu.cn, {cxu@, zyguan@, yzhao@mail.}xidian.edu.cn

Abstract

Existing multimodal sentiment analysis (MSA) methods have achieved significant success, leveraging cross-modal large-scale models (LLMs) and extensive pre-training data. However, these methods struggle to handle MSA tasks in low-resource languages. While multilingual LLMs enable cross-lingual transfer, they are limited to textual data and cannot address multimodal scenarios. To achieve MSA in low-resource languages, we propose a novel transfer learning framework named Language Family Disentanglement and Rethinking Transfer (LFD-RT). During pre-training, we establish cross-lingual and cross-modal alignments, followed by a language family disentanglement module that enhances the sharing of language universals within families while reducing noise from cross-family alignments. We propose a rethinking strategy for unsupervised fine-tuning that adapts the pre-trained model to MSA tasks in low-resource languages. Experimental results demonstrate the superiority of our method and its strong language-transfer capability on target low-resource languages. Code and models are available at https://github.com/ShuoyuGuan/LFD-RT.

1 Introduction

Sentiment analysis aims to detect the sentiment orientations of opinion data. It can be applied to various domains, including market analysis, social media monitoring, and finance and investment. Current advanced social platforms offer diverse multimodal opinion data, thereby extending sentiment analysis beyond its traditional reliance on textual data to encompass multimodal sentiment analysis (MSA). Early research in multimodal sentiment analysis primarily centers on the fusion of multimodal representations, with the progression ranging from basic feature-level fusion (Poria et al., 2018; Zadeh et al., 2018) to more sophisticated decision-level (Chen and Yang, 2020) and hybrid approaches (Han et al., 2021). Later, the research roadmap was dispersed into various fine-grained tasks such as multimodal pre-training models (Lu et al., 2019), modality-invariant representation learning (Hazarika et al., 2020), sentiment analysis under missing uncertain modalities (Zeng et al., 2022) and few-shot learning (Yang et al., 2023). However, none of these works can be utilized for multimodal sentiment analysis in low-resource languages. Multilingual models can handle this problem. Early studies focused on leveraging multilingual LLMs (e.g., mBERT (Pires, 2019), XLM (Conneau and Lample, 2019)) and adversarial training (Dong et al., 2020) for cross-lingual transfer. Subsequently, multi-view-based methods (Fei and Li, 2020) and contrastive learning (Lin et al., 2023) have emerged for MSA. However, these multilingual methods were designed solely for textual data. Thakkar et al. (Thakkar et al., 2024) first attempted to conduct the multimodal and multilingual sentiment analysis task. Nonetheless, the proposed model is overly simplistic and fails to account for the cross-linguistic alignment and cross-lingual noise resistance.

In summary, the key problem we aim to address in this paper is that current multimodal methods struggle to adapt to data in low-resource languages due to their dependence on large-scale training data that is typically available from high-resource languages. Multilingual models can address the challenge of data scarcity; however, they are primarily applied to single-modal textual data.

Inspired by the disentanglement technique in (Ge et al., 2024) and the language family concept in linguistics, we propose a transfer learning framework called Language Family Disentanglement and Rethinking Transfer. During the pre-training phase, we conduct cross-lingual and cross-modal alignments. For aligned cross-lingual features, we
develop a module for language family disentanglement to share language universals within the same family, while reducing the noise generated by cross-family alignments. Then, we use a fusion operation to perform a linguistic-to-modal transfer. We further establish a masked language task to recalibrate the fused representation and conduct the sentiment semantic learning task using double classifiers. During the fine-tuning stage, we propose a novel rethinking strategy to achieve unsupervised fine-tuning of the pre-trained classifiers. The two classifiers conduct a double check on the predictions from the input data. If they reach an agreement, the predicted label serves as a pseudo-label for prediction and backpropagation. If there is inconsistency, the data are sent back as input to conduct a masked prediction task and update the model parameters. In this manner, we finally achieve unsupervised fine-tuning, enabling the pre-trained model to adapt to downstream MSA in low-resource languages. The experimental results demonstrate the efficacy of our methodology and its robust cross-linguistic transferability to low-resource target languages.

2 Related Works

As most of the existing work has been reviewed in the Introduction, we briefly supplement some of the latest representative research.

Multimodal Sentiment Analysis. Li et al. (Li et al., 2024) introduced sample-level contrast and category-guided prototype extraction to handle uncertain missing modalities. Zhu et al. (Zhu et al., 2024) fused video information into text semantics via a text-based cross-modal approach, while Zhao et al. (Zhao et al., 2025) utilized pseudo-data generation for MSA.

Multilingual Sentiment Analysis. Kanayama et al. (Kanayama et al., 2024) enhanced generative methods by incorporating sentiment extractors and external polarity data. Miah et al. (Miah et al., 2024) performed sentiment analysis by translating foreign languages into English.

3 Methodology

3.1 Problem Definition

Given a text-image tuple \((T, I)\), our goal is to develop a model that can accurately predict the sentiment label \(y\). This model supports diverse sentiment classification tasks, including but not limited to the Ekman Emotion Classification Task (Anger, Disgust, Fear, Joy, Sadness, Surprise, Neutral).

3.2 Overall Framework

As shown in Figure 1, we employ a pre-training and unsupervised fine-tuning paradigm as our transfer learning framework. In the pre-training phase, we first perform *cross-linguistic and cross-modal alignments*. Then, we use *language family disentanglement* to mitigate noises induced by linguistic characteristics in cross-lingual alignment, such as lexical, syntactic, and morphological differences. Finally, we train multiple sentiment classifiers on aligned disentangled representations to capture sentiment semantics. After pre-training, the entire pre-trained model is transferred. During the unsupervised fine-tuning stage, we design a *rethinking fine-tuning* strategy to adapt the multiple sentiment classifiers without supervised signals while keeping the other components of the pre-trained model frozen.

3.3 Pre-training

We utilize two data sets for pre-training. One set, denoted by \(D_{cm} = \{T_i, I_i, y_i\}_{1 \leq i \leq N}\), is employed for cross-modal process. Here, \(T_i\) represents an opinion text in the anchor language (a high-resource language, such as English), \(I_i\) denotes its corresponding image, and \(y_i\) is the ground truth label for \((T_i, I_i)\). Another set, denoted as \(D_{cl} = \{\hat{T}_j, T_j^{ts}\}_{1 \leq j \leq M}\), is used for the cross-lingual computations. In this set, \(\hat{T}_j\) is the anchor text, and \(T_j^{ts}\) represents its translation in another low-resource language. Note that \(T_i\) and \(\hat{T}_j\) are different; the former are opinion texts while the latter is not.

Cross-lingual Alignment. This component is designed for cross-lingual transfer. We employ a pre-trained Multilingual BERT (mBERT) as the encoder:
\[
\mathbf{x}_j^{an} = mBERT(\hat{T}_j) \\
\mathbf{x}_j^{ts} = mBERT(T_j^{ts})
\]
A contrastive learning strategy is then applied to the feature vectors, enabling the model to explore the similarities and differences between texts in diverse languages, thereby achieving cross-lingual alignment. Considering that cross-lingual knowledge is transferred to cross-modal representations in the following module, we employ unified contrastive
loss (Li et al., 2023b) for cross-lingual alignment because it can simultaneously align cross-lingual texts and image-text pairs. The loss function is:

\[
\mathcal{L}_{CLA} = \mathcal{L}_{ucl}(\mathbf{x}_j^{an}, \mathbf{x}_j^{ts}) =
-\mathbb{E}_{\sim D_{cl}}[\log \frac{\exp(s(\mathbf{x}_j^{an}, \mathbf{x}_j^{ts}))/\tau)}{\sum\limits_{j=1}^M \exp(s(\mathbf{x}_j^{an}, \mathbf{x}_{-j}^{ts}))/\tau)} \\
+ \log \frac{\exp(s(\mathbf{x}_j^{an}, \mathbf{x}_j^{ts}))/\tau)}{\sum\limits_{j=1}^M \exp(s(\mathbf{x}_j^{an}, \mathbf{x}_{-j}^{ts}))/\tau)}
\]

where \( \tau \) is a temperature hyperparameter.

Cross-modal Alignment. This component performs a cross-modal alignment using a method similar to the cross-lingual alignment:

\[
\mathbf{x}_i^{an} = mBERT(T_i) \\
\mathbf{x}_i^{im} = CLIP(I_i)
\]

The only difference is that we use a CLIP (Radford et al., 2021) as the encoder for the input images.

Language Family Disentanglement. After cross-lingual alignment, the aligned features are fed into a language family disentanglement module. We use this module to mitigate the negative impacts of noises induced by linguistic characteristics while maximizing the retention of beneficial transfer knowledge. Cross-lingual transfer learning involves leveraging the structural, syntactic, semantic, and cultural similarities and differences between languages. Closely related languages, such as English and Frisian, exhibit significant overlaps in grammatical structures, vocabulary semantics, and linguistic features. For instance, both languages share similar noun inflections for gender, number, and case, as well as comparable verb tense and voice systems. These shared linguistic properties can serve as inductive biases, facilitating more effective cross-lingual knowledge transfer by reducing the domain gap between source and target languages. However, for distantly related languages, such as English and Chinese, these similarities diminish, and differences become more pronounced, introducing noise that can hinder transfer performance. The challenge, therefore, lies in designing models that can effectively capture and preserve shared linguistic universals while minimizing the interference caused by language-specific variations.

Coincidentally, language families serve as a robust indicator of linguistic similarities and differences. Languages within the same family share common origins and exhibit similar syntactic, morphological, and phonological patterns, and vice versa. The concept of a language family offers a valuable strategy, enabling the retention of relevant features within the same family while filtering out noise from different families. To this
end, we design a language detangler as shown in Figure 1. We first use 13 family disentanglers \( f_{k:1 \leq k \leq K} : x_j^{ts} \rightarrow f_j^k \) to generate the family-related features . The disentangler function is:

\[
f_j^k = W^{k} x_j^{ts}
\]

where \( W^{k:1 \leq k \leq K} \) are parameter matrices for \( K \) family disentanglers. Then, family-related features \( F_j = [f_j^1 f_j^2 \cdots f_j^K] \) are fed into the cross-attention mechanism to calculate the correlations with features of the aligned anchor language features as follows:

\[
Q_j = W_q F_j \\
K_j = W_k X_j^{an} \\
V_j = W_v X_j^{an} \\
U_j = \text{softmax} \left( \frac{Q_j K_j^T}{\sqrt{d_k}} \right) V_j
\]

where \( W_q, W_k \) and \( W_v \) are parameter matrices, \( U = [u_j^1 u_j^2 \cdots u_j^K] \) are anchor-correlation features, and we input them into linear projection to obtain the final predictions \( P_j = [p_j^1 p_j^2 \cdots p_j^K] \) as following:

\[
P_j = W_l U_j
\]

We implement a masked language task to ensure that samples within the same language family (i.e., anchor family) are more closely grouped, coupled with a KL divergence regularization task to promote a uniform distribution between samples from different families. The loss function for the masked language task is:

\[
\mathcal{L}_{mask_1} = \mathcal{L}_{MASK}(T_j^{tr}, p_j) =
-\frac{1}{M} \sum_{j=1}^M \sum_{p=1}^{T_j^{tr}} \log \left( p_{jp}^{k=anchor} \right)
\]

where \( w_p^{T_j^{tr}} \) represents the vocabulary probability distribution for masked words in position \( p \), and \( p_{jp}^{k=anchor} \) signifies that the predicted word probability in position \( p \) of the translated text belongs to the same family as the anchor language. \( r \) represents the masking probability, and \( |T_j^{tr}| \) denotes the total number of words in the text \( T_j^{tr} \). We employ the same masking strategy as BERT (Kenton and Toutanova, 2019). The KL divergence regularization loss is:

\[
\mathcal{L}_{KL} = \frac{1}{M} \sum_{j=1}^M P_j^{k \neq anchor} \log \frac{P_j^{k \neq anchor}}{P_{uniform}}
\]

where \( P_j^{k \neq anchor} \) denotes the probability distribution of translated texts originating from language families distinct from that of the anchor language, and \( P_{uni} \) signifies a uniform distribution. We regulate the samples from other language families to achieve a uniform distribution with a high degree of chaos, indicating significant information disorder. This is in line with the objective of alleviating the noise effects of other language families. Combining the two tasks mentioned above, the language family disentanglement loss function is:

\[
\mathcal{L}_{LFD} = (\alpha_1 \mathcal{L}_{mask_1} + \alpha_2 \mathcal{L}_{KL})
\]

where \( \alpha_1 \) and \( \alpha_2 \) are hyperparameters.

**Cross-modal Fusion.** In the cross-modal interaction pipeline, we employ a cross-attention mechanism, enhanced through language family disentanglement, to process aligned multimodal representations. This approach promotes knowledge transfer from linguistic to visual modalities, while enabling cross-modal fusion. The computations are:

\[
q_i = W_q x_i^{im} \\
k_i = W_k x_i^{an} \\
v_i = W_v x_i^{an} \\
z_i = \text{softmax} \left( \frac{q_i k_i^T}{\sqrt{d_k}} \right) v_i
\]

Note that the parameter matrices \( W_q, W_k \) and \( W_v \) are shared by the cross-attention of the language family disentanglement module. Next, we fed fused representation into two multilayer perceptions (\( MLP_1 \) and \( MLP_2 \)) for predicting the sentiment orientations:

\[
o_i^{1v2} = MLP_{1v2}(z_i)
\]
\[
\mathcal{L}_{MLP_{1v2}} = \mathcal{L}_{CE}(y_i, o_i^{1v2})
\]
\[
= -\frac{1}{N} \sum_{i=1}^N y_i \log \left( o_i^{1v2} \right)
\]

Both perceptions have one hidden layer, utilize ReLU as the activation function, and share the same cross-entropy loss function. To mitigate the semantic shift caused by the fusion operation, we also incorporate a masked language loss to recalibrate the semantics of the fused representation. We employ another projection layer along with the same masked language strategy used in the language family disentanglement module. The computational equations are:

\[
P_i = W_l Z_i
\]

K = 13 is the total number of language families covering all the languages in our dataset.
Figure 2: An intuitive explanation of rethinking unsupervised fine-tuning.

\[
\mathcal{L}_{mask_2} = \mathcal{L}_{MASK}(T_i, p_i)
\] (15)

Thus, the combined loss function for sentiment analysis is:

\[
\mathcal{L}_{SA} = (\beta_1 \mathcal{L}_{MLP_1} + \beta_2 \mathcal{L}_{MLP_2} + \beta_3 \mathcal{L}_{mask_2})
\] (16)

where \( \beta_1 \) and \( \beta_2 \) are weight coefficients. By combining the loss functions from the above modules, we arrive at the final optimization loss for the pre-training part:

\[
Loss_{PT} = (\mathcal{L}_{Align} + \mathcal{L}_{FCD} + \mathcal{L}_{SA})
\] (17)

Note that we only use the labeled multimodal data in the high-resource language for pre-training.

3.4 Unsupervised Rethinking Fine-tuning

<table>
  <tr>
    <th>Meld</th>
    <th colspan="3">CH SIMS</th>
  </tr>
  <tr>
    <th>Train</th>
    <th>Val</th>
    <th>Test</th>
    <th>Train</th>
    <th>Val</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>5548</td>
    <td>2710</td>
    <td>2511</td>
    <td>1462</td>
    <td>665</td>
    <td>655</td>
  </tr>
</table>

Table 1: MSA data statistics

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="6">Languages</th>
  </tr>
  <tr>
    <th>en</th>
    <th>eo</th>
    <th>es</th>
    <th>et</th>
    <th>eu</th>
    <th>fi</th>
    <th>fr</th>
    <th>he</th>
    <th>hu</th>
    <th>id</th>
    <th>io</th>
    <th>ka</th>
    <th>ar</th>
    <th>mg</th>
    <th>ml</th>
    <th>oc</th>
    <th>sk</th>
    <th>sw</th>
    <th>ta</th>
    <th>te</th>
    <th>tg</th>
    <th>tl</th>
    <th>tr</th>
  </tr>
  <tr>
    <th>Meld</th>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <th>CH SIMS</th>
    <th>zh</th>
    <th>ja</th>
    <th>jv</th>
    <th>ko</th>
    <th>tt</th>
  </tr>
</table>

Table 2: Languages in Meld and CH SIMS dataset.

Fine-tuning is a crucial step in adapting the pre-trained model for low-resource languages. In real-world scenarios, we commonly encounter the challenge of limited data resources in minority languages. To address this challenge, we design an unsupervised learning method called rethinking fine-tuning. This method draws inspiration from the answer consistency check with multiperson cross-validation, as illustrated in Figure 2. Specifically, during the fine-tuning phase, the pre-trained models are used as initialization. We fine-tune the two MLPs and fixed the other components of the model. The dataset for fine-tuning is denoted as \( D_{low} = \{T_s^{low}, I_s^{low}\}_{1 \leq s \leq S} \), where \( T_s^{low} \) is the opinion text in the target low-resource language, and its corresponding image is represented as \( I_s^{low} \). The forward computation process is identical to that of the pre-training (note that input text is processed by the disentangler of its corresponding language family, i.e., \( F_x \)). For the two pre-trained MLP classifiers, we develop an unsupervised rethinking strategy to tune their parameters: If both classifiers yield the same label, we deem this label a pseudo-label for backpropagation; if they produce different labels, it indicates a divergence in their understanding of the sentiment semantics of the input data. In this case, the input data pairs are returned and used as model inputs for masked language training. This approach is the same as the training method that addresses the semantic shift problem during the pretraining process. The loss functions for rethinking fine-tuning are:

\[
Loss_{FT} =
\begin{cases}
\text{if } \hat{y}_s^1 = \hat{y}_s^2 = \hat{y}_s : \\
\quad \mathcal{L}_{SC} = \lambda_1 \mathcal{L}_{MLP_1} + \lambda_2 \mathcal{L}_{MLP_2}, \\
\text{if } \hat{y}_s^1 \neq \hat{y}_s^2 : \\
\quad \mathcal{L}_{MASK}(T_s^{low}, p_s),
\end{cases}
\] (18)

where \( \mathcal{L}_{MLP_{1/2}} = \mathcal{L}_{CE}(\hat{y}_s, o_s^{1/2}) \). \( \hat{y}_s^1 \) and \( \hat{y}_s^2 \) are the predicted one-hot labels of the MLP1 and MLP2 respectively. \( \hat{y}_s \) is the pseudo-label. The objectives of the masked language task in the pre-training and fine-tuning stages differ; the former aims to solve the semantic shift problem of fused features, whereas the latter aims to rethink the sentiment semantics of the input data. Therefore, we reconstruct a new linear projection layer to generate \( p_s \) for the masked language task in the fine-tuning stage. By employing a rethinking strategy, we conduct an unsupervised fine-tuning. We use double MLP checkers because the experimental results indicate that adding more checkers does not significantly enhance performance; rather, it increases the model complexity and the risk of conflicting
3.5 Inference
During the inference phase, the final predicted label is determined through agreement between the two classifiers. In cases of disagreement, the class with the highest predicted probability is assigned as the output.

4 Experiments

4.1 Datasets and Language Families
We employ two MSA datasets with different granularities for evaluation: 1) a fine-grained 7-level Ekman MSA dataset called Meld. Collected from the TV series "Friends", it contains 1,433 conversations, 13,708 utterances, and 304 speakers. Each utterance is labeled as anger, disgust, sadness, joy, surprise, fear, or neutral. 2) A coarse-grained 5-level MSA dataset called CH SIMS: A Chinese multimodal sentiment analysis dataset with 2,281 video clips. Each utterance is labeled as positive, weakly positive, negative, weakly negative, or neutral. The video-text pairs are used for training. We adhere to the data splitting procedure outlined in Table 1. To create the validation and test sets, we translate English and Chinese texts into 22 and 4 other languages, respectively, using Google Translate. The complete set of 28 language abbreviations that follow the ISO 639 standard is presented in Table 2. The bold abbreviations are English (en) and Chinese (zh), which serve as the anchor languages. Note that our primary emphasis is on multilingual learning. Therefore, we exclude the audio data, as it pertains to a single language, and instead focus solely on texts and images.

For the cross-lingual task in the pretraining stage, we select 124,037 text pairs from WikiMatrix. The corpus is a large-scale multilingual parallel dataset released by Facebook, comprising 135 million parallel sentences. The collected pairs feature 28 languages, including Esperanto, Tamil, and others, with English serving as the anchor language. Similarly, we also collect 26,965 text pairs in Chinese (anchor language) and other 4 languages from WikiMatrix. The two gathered datasets are used for pretraining in the Meld and CH SIMS tasks. Table 3 illustrates the language family affiliations of all languages used in the experiments.

<table>
  <tr>
    <th>Language</th>
    <th>Family</th>
    <th>Language</th>
    <th>Family</th>
  </tr>
  <tr>
    <td>eo</td>
    <td>-</td>
    <td>ko</td>
    <td>KR</td>
  </tr>
  <tr>
    <td>es</td>
    <td>IE</td>
    <td>ar</td>
    <td>AA</td>
  </tr>
  <tr>
    <td>et</td>
    <td>U</td>
    <td>mg</td>
    <td>AN</td>
  </tr>
  <tr>
    <td>eu</td>
    <td>-</td>
    <td>ml</td>
    <td>Drav.</td>
  </tr>
  <tr>
    <td>fi</td>
    <td>U</td>
    <td>oc</td>
    <td>IE</td>
  </tr>
  <tr>
    <td>fr</td>
    <td>IE</td>
    <td>sk</td>
    <td>IE</td>
  </tr>
  <tr>
    <td>gl</td>
    <td>IE</td>
    <td>sw</td>
    <td>NC</td>
  </tr>
  <tr>
    <td>he</td>
    <td>AA</td>
    <td>ta</td>
    <td>Drav.</td>
  </tr>
  <tr>
    <td>hu</td>
    <td>U</td>
    <td>te</td>
    <td>Drav.</td>
  </tr>
  <tr>
    <td>id</td>
    <td>AN</td>
    <td>tg</td>
    <td>IE</td>
  </tr>
  <tr>
    <td>io</td>
    <td>-</td>
    <td>tl</td>
    <td>AN</td>
  </tr>
  <tr>
    <td>ja</td>
    <td>JR</td>
    <td>tr</td>
    <td>Trk.</td>
  </tr>
  <tr>
    <td>jv</td>
    <td>AN</td>
    <td>zh</td>
    <td>ST</td>
  </tr>
  <tr>
    <td>ka</td>
    <td>K</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 3: Language-family affiliation.

4.2 Implementation details
We optimize the hyperparameters on the validation set through a sequential tuning strategy, where each hyperparameter is individually adjusted while the others remain fixed. The final configuration is as follows: the temperature hyperparameter in the contrastive loss is set to \( \tau = 0.07 \), and the masking probability is \( r = 15\% \). The weighted hyperparameters of the loss functions are configured as: \( \alpha_1 = 0.3, \alpha_2 = 0.1, \beta_1 = \beta_2 = 1, \beta_3 = 0.3 \), and \( \lambda_1 = \lambda_1 = 0.5 \). We use the Adam optimizer with a learning rate of \( 5 \times 10^{-6} \).

All experiments were performed on an NVIDIA RTX 3090 GPU, which required 20 hours of training time. Our hyperparameter selection was guided by preliminary grid search and empirical validation: temperature \( \tau \in (0, 0.1) \) with 0.01 increments, and task weights \( \alpha_1, \alpha_2 \in (0, 1) \) with 0.05 increments. The parameter sensitivity analysis revealed an inverted U-shaped performance curve, where excessive \( \alpha_1 (> 0.3) \) caused sentiment-task dominance degradation, while \( \alpha_2 > 0.1 \) induced random masking behavior. These observations align with the theoretical expectations. Optimal parameters were determined as \( \tau = 0.07, \alpha_1 = 0.3, \alpha_2 = 0.1 \), with the masking ratio fixed at BERT's standard 15%.

4.3 Baselines
Large language models (LLMs) have exhibited remarkable proficiency in MSA tasks, prompting us to adopt the top-performing LLMs as our baseline methodologies. The baselines can be categorized into three groups: 1) multilingual baselines,
including mBERT (Pires, 2019), XLM-R (Conneau, 2019), and Llama 3.2 (Dubey et al., 2024); 2) multimodal baselines, comprising CLIP (Radford et al., 2021), BLIP-2 (Li et al., 2023a) and GPT-4o (2024); and 3) hybrid baselines combining multilingual and multimodal LLMs. L+C: We use Llama 3.2 and CLIP as text and image encoders, respectively. We then employ concatenation for multimodal feature fusion. We fix CLIP and use Lora to fine-tune Llama. M2SA (Thakkar et al., 2024): This is the current state-of-the-art method for multimodal and multilingual sentiment analysis. It employs XLMR-SM and CLIP encoders along with concatenation for feature fusion.

To ensure fairness, we use the same labeled MSA training data for both the multimodal and hybrid baselines. We remove images and rely solely on the labeled texts to train the multilingual baselines. We use accuracy and the weighted average F1 score as evaluation metrics.

4.4 Result Analysis

Main comparisons. Table 4 shows the comparison results on Meld. For the overall classification results, multilingual baselines perform worse than the multimodal baselines. This indicates that additional modal data, such as visual modality, can enhance the performance of single-modal models using textual data. L + C and M2SA perform significantly worse than GPT-4o on the F1 measure, demonstrating that merely combining multilingual and multimodal LLMs does not yield performance gains. Conversely, this may lead to negative effects due to the noise introduced by cross-language fusion and the semantic shift resulting from linguistic-modality fusion. Our method, which benefits from custom-designed modules for cross-lingual noise and semantic shift, performs better than the baseline methods. The slightly lower performance of our method compared to GPT-4o in handling emotions, such as Disgust and Fear, may be attributed to the relatively small number of pre-training samples for these categories. This data imbalance likely confers GPT-4o with a comparative advantage in addressing such scenarios, as its extensive pre-training on diverse datasets enhances its robustness to minority categories.

Cross-lingual performances. Table 5 presents the F1 scores across different languages. English, Spanish, and French belong to the Indo-European language family. The transfer performance from English to French is superior to that from English to Spanish, which can be attributed to the closer linguistic relationship between English and French, characterized by greater similarities in language structure and vocabulary. Notably, the model demonstrates exceptional transfer performance for several low-resource languages, including Occitan and Tajik. When combined with its robust average performance across 23 languages, these results strongly validate the efficacy of language family disentanglement in enhancing cross-lingual transfer capabilities. Furthermore, the model’s consistent performance across various low-resource languages from diverse language families underscores its resilience to cross-lingual noise and highlights its strong generalization capabilities beyond linguistic boundaries.

Generalization ability. Table 6 shows the model’s overall and cross-language performance on the CH-SIMS dataset. The overall trends align with Table 4, with our method outperforming baselines. The balanced dataset enables the Llama and CLIP combination to fully use their complementary strengths. The model achieves higher F1 scores for pictographic languages (Chinese, Korean, and Japanese), while performance is slightly lower for alphabetic languages like Javanese and Tatar. Nonetheless, the results underscore the model’s robust capability in cross-lingual sentiment analysis.

4.5 Ablation Study

To evaluate the contribution of each component in LFD-RT, we conduct an ablation study, with results shown in Table 7. Removing the image modality (w/o CLIP) or removing the text modality (w/o mBERT) results in the most significant performance degradation, highlighting the importance of multimodal information for performance improvement. Eliminating the language family disentangler (w/o LFD) or double cross-alignments (w/o DSA) also leads to notable performance drops, demonstrating their equal importance. Additionally, removing the rethinking fine-tuning module (w/o RFT) degrades the model performance, as this module enables self-adaption to downstream MSA tasks in low-resource languages.

5 Conclusion

In this paper, we propose a multilingual multi-modal sentiment analysis framework leveraging language family disentanglement and rethinking transfer. During pre-training, we first perform
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Anger</th>
    <th colspan="2">Disgust</th>
    <th colspan="2">Fear</th>
    <th colspan="2">Joy</th>
    <th colspan="2">Neutral</th>
    <th colspan="2">Sadness</th>
    <th colspan="2">Surprise</th>
    <th colspan="2">All</th>
  </tr>
  <tr>
    <th>Acc.</th><th>F1</th>
    <th>Acc.</th><th>F1</th>
    <th>Acc.</th><th>F1</th>
    <th>Acc.</th><th>F1</th>
    <th>Acc.</th><th>F1</th>
    <th>Acc.</th><th>F1</th>
    <th>Acc.</th><th>F1</th>
    <th>Acc.</th><th>F1</th>
  </tr>
  <tr>
    <td>mBERT</td><td>15.7</td><td>19.1</td><td>5.8</td><td>8.9</td><td>10.9</td><td>14.1</td><td>28.0</td><td>26.3</td><td>75.6</td><td>65.1</td><td>12.0</td><td>11.6</td><td>21.5</td><td>22.4</td><td>46.5</td><td>32.9</td>
  </tr>
  <tr>
    <td>XLM-R</td><td>16.3</td><td>19.7</td><td>5.4</td><td>10.1</td><td>6.7</td><td>9.4</td><td>21.8</td><td>15.3</td><td>80.7</td><td>65.5</td><td>5.2</td><td>8.3</td><td>25.0</td><td>24.0</td><td>47.7</td><td>36.3</td>
  </tr>
  <tr>
    <td>Llama 3.2</td><td>18.1</td><td>24.5</td><td>20.2</td><td>15.5</td><td>11.6</td><td>8.7</td><td>48.2</td><td>43.4</td><td>73.1</td><td>69.4</td><td>10.3</td><td>13.1</td><td>33.6</td><td>34.7</td><td>50.3</td><td>46.9</td>
  </tr>
  <tr>
    <td>CLIP</td><td>12.0</td><td>17.8</td><td>7.2</td><td>12.4</td><td>9.7</td><td>11.3</td><td>30.3</td><td>35.3</td><td>81.4</td><td>68.3</td><td>1.1</td><td>2.3</td><td>22.0</td><td>29.8</td><td>48.6</td><td>42.5</td>
  </tr>
  <tr>
    <td>BLIP2</td><td>15.3</td><td>16.1</td><td>8.4</td><td>10.5</td><td>9.3</td><td>10.7</td><td>29.1</td><td>32.1</td><td>80.5</td><td>66.5</td><td>10.3</td><td>12.8</td><td>21.7</td><td>27.4</td><td>48.0</td><td>44.7</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>21.2</td><td>28.1</td><td>11.7</td><td>16.5</td><td>13.0</td><td>15.5</td><td>25.7</td><td>32.5</td><td>81.8</td><td>66.7</td><td>14.2</td><td>19.4</td><td>24.1</td><td>28.5</td><td>50.8</td><td>48.6</td>
  </tr>
  <tr>
    <td>L + C</td><td>20.6</td><td>25.2</td><td>17.3</td><td>20.8</td><td>11.5</td><td>13.1</td><td>51.0</td><td>40.0</td><td>81.1</td><td>70.1</td><td>7.4</td><td>10.6</td><td>34.6</td><td>35.2</td><td>51.6</td><td>44.1</td>
  </tr>
  <tr>
    <td>M2SA</td><td>20.1</td><td>23.8</td><td>21.3</td><td>17.5</td><td>12.4</td><td>19.6</td><td>38.6</td><td>43.8</td><td>82.3</td><td>70.7</td><td>14.6</td><td>21.1</td><td>24.7</td><td>27.0</td><td>53.5</td><td>43.8</td>
  </tr>
  <tr>
    <th>LFD-RT</th><th>21.7</th><th>27.7</th><th>22.0</th><th>24.1</th><th>12.1</th><th>17.2</th><th>49.1</th><th>45.3</th><th>83.7</th><th>71.1</th><th>15.2</th><th>22.4</th><th>35.1</th><th>37.8</th><th>54.7</th><th>49.1</th>
  </tr>
</table>

Table 4: Comparison results on Meld.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="6">Meld</th>
    <th colspan="2">Avg.</th>
  </tr>
  <tr>
    <th>English</th><th>Spanish</th><th>French</th><th>Occitan</th><th>Tamil</th><th>Luxembourgish</th>
    <th>Tajik</th><th>Avg.</th>
  </tr>
  <tr>
    <td>mBERT</td><td>42.3</td><td>33.8</td><td>33.1</td><td>32.2</td><td>34.3</td><td>35.1</td><td>32.4</td><td>32.9</td>
  </tr>
  <tr>
    <td>XLM-R</td><td>42.6</td><td>35.9</td><td>34.2</td><td>30.7</td><td>32.6</td><td>36.0</td><td>33.9</td><td>36.3</td>
  </tr>
  <tr>
    <td>Llama</td><td>56.1</td><td>44.6</td><td>55.4</td><td>44.8</td><td>36.0</td><td>25.7</td><td>43.9</td><td>46.9</td>
  </tr>
  <tr>
    <td>CLIP</td><td>51.8</td><td>41.5</td><td>47.7</td><td>40.9</td><td>39.7</td><td>39.4</td><td>43.6</td><td>42.5</td>
  </tr>
  <tr>
    <td>BLIP2</td><td>52.2</td><td>40.1</td><td>43.5</td><td>40.6</td><td>36.1</td><td>37.1</td><td>42.8</td><td>44.7</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>55.6</td><td>34.9</td><td>61.5</td><td>35.7</td><td>41.3</td><td>43.8</td><td>44.0</td><td>48.6</td>
  </tr>
  <tr>
    <td>L + C</td><td>44.3</td><td>40.0</td><td>41.7</td><td>39.4</td><td>38.2</td><td>44.0</td><td>46.5</td><td>44.1</td>
  </tr>
  <tr>
    <td>M2SA</td><td>43.7</td><td>44.9</td><td>41.2</td><td>40.0</td><td>40.4</td><td>41.9</td><td>41.3</td><td>43.8</td>
  </tr>
  <tr>
    <th>LFD-RT</th><th>57.0</th><th>46.6</th><th>52.7</th><th>46.1</th><th>43.5</th><th>44.6</th><th>47.3</th><th>49.1</th>
  </tr>
</table>

Table 5: Comparison results on different languages.

<table>
  <tr>
    <th colspan="2">CH SIMS</th>
    <th colspan="6">CH SIMS</th>
  </tr>
  <tr>
    <th>Method</th><th>Acc.</th><th>F1</th>
    <th>Method</th><th>Chinese</th><th>Japanese</th><th>Korean</th><th>Javanese</th><th>Tatar</th><th>Avg.</th>
  </tr>
  <tr>
    <td>mBERT</td><td>34.3</td><td>32.1</td>
    <td>mBERT</td><td>36.9</td><td>40.0</td><td>28.3</td><td>25.2</td><td>25.5</td><td>32.1</td>
  </tr>
  <tr>
    <td>XLM-R</td><td>34.8</td><td>30.9</td>
    <td>XLM-R</td><td>38.1</td><td>38.6</td><td>29.7</td><td>23.5</td><td>24.7</td><td>30.9</td>
  </tr>
  <tr>
    <td>Llama 3.2</td><td>35.1</td><td>32.2</td>
    <td>Llama</td><td>37.8</td><td>39.0</td><td>30.6</td><td>28.9</td><td>24.8</td><td>32.2</td>
  </tr>
  <tr>
    <td>CLIP</td><td>31.1</td><td>24.0</td>
    <td>CLIP</td><td>30.2</td><td>34.4</td><td>16.4</td><td>18.7</td><td>17.7</td><td>24.0</td>
  </tr>
  <tr>
    <td>BLIP2</td><td>37.4</td><td>32.1</td>
    <td>BLIP2</td><td>37.6</td><td>39.2</td><td>28.3</td><td>28.6</td><td>26.8</td><td>32.1</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>21.5</td><td>23.8</td>
    <td>GPT-4o</td><td>32.0</td><td>24.3</td><td>26.7</td><td>18.3</td><td>17.7</td><td>23.8</td>
  </tr>
  <tr>
    <td>L + C</td><td>38.8</td><td>34.1</td>
    <td>L + C</td><td>40.3</td><td>39.7</td><td>35.2</td><td>27.4</td><td>28.1</td><td>34.1</td>
  </tr>
  <tr>
    <td>M2SA</td><td>37.1</td><td>33.7</td>
    <td>M2SA</td><td>39.8</td><td>41.0</td><td>32.7</td><td>28.9</td><td>26.3</td><td>33.7</td>
  </tr>
  <tr>
    <th>LFD-RT</th><th>40.1</th><th>34.3</th>
    <th>LFD-RT</th><th>41.0</th><th>40.3</th><th>33.3</th><th>29.7</th><th>27.0</th><th>34.3</th>
  </tr>
</table>

Table 6: Comparison results on CH SIMS.

cross-lingual and cross-modal alignment, followed by disentangling language families to facilitate the sharing of language universals within the same family while minimizing noise from cross-lingual alignment across different families. To address low-resource language challenges, we introduce a novel double-check rethinking strategy for unsupervised fine-tuning. Experimental results demonstrate the superiority of our approach and highlight the effectiveness of language family-based transfer learning.

Limitations

Although our approach demonstrates promising results, it is subject to two primary limitations. First,
<table>
  <tr>
    <th>Method</th>
    <th>Acc</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>w/o CLIP</td>
    <td>48.1</td>
    <td>42.3</td>
  </tr>
  <tr>
    <td>w/o mBERT</td>
    <td>48.9</td>
    <td>43.4</td>
  </tr>
  <tr>
    <td>w/o LFD</td>
    <td>51.4</td>
    <td>47.1</td>
  </tr>
  <tr>
    <td>w/o DSA</td>
    <td>51.3</td>
    <td>47.4</td>
  </tr>
  <tr>
    <td>w/o RFT</td>
    <td>53.8</td>
    <td>47.6</td>
  </tr>
  <tr>
    <td>LFD-RT</td>
    <td>54.7</td>
    <td>49.1</td>
  </tr>
</table>

Table 7: Ablation results of DFD-RT on Meld.

constrained by the limited scale of pre-training data, our model’s performance on minority-class emotions has hindered its overall effectiveness. Second, we relied solely on coarse-grained language family information, overlooking the fine-grained hierarchical structure of language families, groups, branches, and individual languages. Consequently, the model’s ability to differentiate between languages within the same family remains insufficient.

Acknowledgments

This paper is supported by the National Natural Science Foundation of China (62203354, 62472340, 6230336, 62133012, 62425605, 62306235), the Key Research and Development Program of Shaanxi (Program Nos. 2024GX-YBXM-122, 22025CY-YBXM-041, 2024CY2-GJHX-15, 2022ZDLGY01-10), the Natural Science Basic Research Program of Shaanxi (Program Nos. 2025JC-QYXQ-040, 2025JC-YBQN-900, 2025CY-YBXM-067), Guangdong Basic and Applied Basic Research Foundation (Program No. 2025A1515010267), the Postdoctoral Research Project Sustentation Fund of Shaanxi (No. 2024BSHSDZZ112), and the Key Scientific Research Projects of the Shaanxi Provincial Education Department (No. 24JR152).

References

Jiaao Chen and Diyi Yang. 2020. Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106–4118.

A Conneau. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116.

Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. Advances in neural information processing systems, 32.

Xin Dong, Yaxin Zhu, Yupeng Zhang, Zuohui Fu, Dongkuan Xu, Sen Yang, and Gerard De Melo. 2020. Leveraging adversarial training in self-learning for cross-lingual text classification. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1541–1544.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Hongliang Fei and Ping Li. 2020. Cross-lingual unsupervised sentiment classification with multi-view transfer learning. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5759–5771.

Ling Ge, Chunming Hu, Guanghui Ma, Jihong Liu, and Hong Zhang. 2024. Da-net: A disentangled and adaptive network for multi-source cross-lingual transfer learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18047–18055.

Wei Han, Hui Chen, and Soujanya Poria. 2021. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9180–9192.

Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM international conference on multimedia, pages 1122–1131.

Hiroshi Kanayama, Yang Zhao, Ran Iwamoto, and Takuya Ohko. 2024. Incorporating syntax and lexical knowledge to multilingual sentiment classification on large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 4810–4817.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacl-HLT, volume 1. Minneapolis, Minnesota.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.

Mingcheng Li, Dingkang Yang, Xiao Zhao, Shuaibing Wang, Yan Wang, Kun Yang, Mingyang Sun, Dongliang Kou, Ziyun Qian, and Lihua Zhang. 2024. Correlation-decoupled knowledge distillation for multimodal sentiment analysis with incomplete modalities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12458–12468.
Zejun Li, Zhihao Fan, Jingjing Chen, Qi Zhang, Xuan-Jing Huang, and Zhongyu Wei. 2023b. Unifying cross-lingual and cross-modal modeling towards weakly supervised multilingual vision-language pre-training. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5939–5958.

Nankai Lin, Yingwen Fu, Xiaotian Lin, Dong Zhou, Aimin Yang, and Shengyi Jiang. 2023. CI-xabsa: Contrastive learning for cross-lingual aspect-based sentiment analysis. IEEE/ACM Transactions on Audio, Speech, and Language Processing.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32.

Md Saef Ullah Miah, Md Mohsin Kabir, Talha Bin Sarwar, Mejdl Safran, Sultan Alfarhood, and MF Mridha. 2024. A multimodal approach to cross-lingual sentiment analysis with ensemble of transformer and Ilm. Scientific Reports, 14(1):9603.

T Pires. 2019. How multilingual is multilingual bert. arXiv preprint arXiv:1906.01502.

Soujanya Poria, Navonil Majumder, Devamanyu Hazarika, Erik Cambria, Alexander Gelbukh, and Amir Hussain. 2018. Multimodal sentiment analysis: Addressing key issues and setting up the baselines. IEEE Intelligent Systems, 33(6):17–25.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Gaurish Thalkar, Sherzod Hakimov, and Marko Tadić. 2024. M2sa: Multimodal and multilingual model for sentiment analysis of tweets. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10833–10845.

Xiaocui Yang, Shi Feng, Daling Wang, Yifei Zhang, and Soujanya Poria. 2023. Few-shot multimodal sentiment analysis based on multimodal probabilistic fusion prompts. In Proceedings of the 31st ACM international conference on multimedia, pages 6045–6053.

AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236–2246.

Jiandan Zeng, Tianyi Liu, and Jiantao Zhou. 2022. Tag-assisted multimodal sentiment analysis under uncertain missing modalities. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1545–1554.

Zheyu Zhao, Zhongqing Wang, Shichen Li, Hongling Wang, and Guodong Zhou. 2025. Bridging modality gap for effective multimodal sentiment analysis in fashion-related social media. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1813–1823.

Aoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang, Yiming Tang, and Fuji Ren. 2024. Kebr: Knowledge enhanced self-supervised balanced representation for multimodal sentiment analysis. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 5732–5741.

Reviewer paper summaries: This paper proposes LFD-RT that tackles multimodal sentiment analysis for low-resource languages. The authors use cross-lingual alignment (anchor vs. translated text) and cross-modal alignment (text vs. image) during pretraining. They also Introduce Language Family Disentanglement (LFD): each language is assigned to one of 13 broad families, encouraging “same-family” features to cluster and reducing noise for “different-family” alignments. 

Additionally, they propose a “rethinking” unsupervised fine-tuning approach that uses two classifiers to cross-check predictions, thereby refining the model with unlabeled data for "unsupervised" fine-tuning (This feels like having an additional checker/annotator). They test on two datasets—Meld (7-category emotions, originally in English) and CH SIMS (5-category sentiment, originally in Chinese)—translated into 28 languages grouped into 13 families. Results show consistent improvements over baselines (e.g., XLM-R, mBERT, Llama 3.2, M2SA), underscoring the synergy of family-level alignment and cross-modal fusion. 

However, practical and scalability concerns weigh against it – specifically the reliance on parallel data or translations in truly low-resource scenarios, limited clarity on hyperparameter tuning, and the absence of some key baselines (e.g., GPT-4 Vision)
Summary of strengths: The paper has three significant contributions to cross-lingual MSA: 

1. Language-family disentanglement module -- Authors introduce a novel approach to leveraging linguistic typology for noise reduction in cross-lingual transfer,  grouping the 28 languages into 13 families effectively reduces cross-lingual “noise,” showing clear gains in ablation (w/o LFD lowers performance).
2. Innovative Multimodal + Cross-Lingual Focus --The paper extends disentanglement ideas from text-only settings to image–text data, with a special emphasis on low-resource languages that lack large-scale labeled data.
3.  In-depth Experimentation: Analyzing 28 languages from 13 families across two sentiment tasks (7-class Ekman emotions and 5-level intensity) offers solid proof of cross-family generalization. Remarkably, the framework performs well in pictographic languages (Chinese: F1 41.0, Japanese: F1 40.3), surpassing Llama 3.2 by 3.2 to 4.3 points while relying entirely on translated data.
Summary of weaknesses: 1. The paper’s approach to language family disentanglement does not seem well grounded in linguistic theory. It simplifies complex phylogenetic relationships into 13 fixed categories but does not consider variations within subfamilies. For example, Indo-European languages are treated as one group, even though Germanic and Romance languages have major differences. This simplification may explain inconsistent performance across languages, such as the 6.3-point F1 difference between French (52.7) and Spanish (46.6) as the paper groups them both to the same language family (IE). Also, recent work by Kanayama et al., 2024 (acl findings) shows that external lexical-syntactic knowledge can effectively improve multilingual sentiment classification. Though they do not explicitly address language families, their method highlights the value of language-tailored enhancements. Subdividing families (e.g., Romance vs. Germanic) could further reduce cross-lingual noise and align well with the idea of leveraging structural cues across languages.

2. Dependency on machine-translated data for low-resource language evaluation -- This raises concerns. While the authors use Google Translate to create test sets for languages like Luxembourgish and Tatar, real-world low-resource scenarios often involve code-mixing, dialectal variations, and non-standard syntax that machine translation systems handle poorly. This methodological choice risks inflating performance metrics, as translated text may retain structural properties of the source language (English/Chinese) rather than reflecting authentic usage patterns.

3. Comparisons to Purely Image-Only or Larger Multimodal LLMs -- The ablation study includes a "w/o CLIP" (text-only) baseline, showing a performance drop, but does not provide an image-only baseline. Without this, it is unclear how much each modality—text and image—individually contributes to the model’s performance. Additionally, a comparison with a state-of-the-art vision-language model (e.g., GPT-4 Vision or BLIP-2) could help assess whether the proposed approach offers a meaningful advantage. While the authors compare against GPT-4o (text-only), this does not serve as the strongest multimodal LLM baseline.

4. The proposed framework shows strong empirical improvements, but the paper offers little justification for its hyperparameter choices (e.g., temperature τ, masking ratio, α₁, α₂) and does not outline a systematic tuning strategy. Providing a brief sensitivity analysis or explaining the rationale behind these selections would help assess the robustness of the approach under different settings. 

Also regarding computation efficiency (may be relevant to industry) - The pre-training phase requires simultaneous optimization of four loss components (alignment, disentanglement, sentiment analysis, masked language tasks), which likely necessitates substantial GPU resources, it would be beneficial to add an appendix section discussing this and point number 4.
Comments, suggestions and typos: “Overll Framework” → “Overall Framework” (typo in heading 3.2).

inconsistent references to “LFD-RT” vs. “DFD-RT” in the text or table captions (if it occurs). Double-check for uniform naming.

Line 231: ‘matices’ should be corrected to ‘matrices,’ e.g., ‘where Wₖ:1≤k≤K are parameter matrices for K family disentanglers.’”


Question to Authors:  How does the model perform on authentic low-resource language data containing code-mixing or dialectal variations, as opposed to machine-translated text? Could the translation process introduce syntactic biases that inflate cross-lingual transfer metrics
Overall assessment scores: 3