## 4 Methodology
### Contrastive Learning Based on Multimodal Data Completion  
**Sample division.** For each data instance, we assess the presence or absence of modalities, categorizing the data into two sets: one consisting of complete modalities and the other of incomplete modalities. The data with complete modalities are utilized for training and testing the learning framework, while the trained model is employed to complete the missing modalities. Let the dataset with complete modalities contain \(N\) samples, each featuring three modalities: text, audio, and image. We denote the states of the \(i\)th sample as \(T_{i}\), \(A_{i}\), and \(I_{i}\), respectively. Each sample is assigned a synthetic emotion label \(Y_{i}\in 1,2,\ldots,K\), where \(K\) represents the number of emotion categories [48]. Assuming that the image modality is absent [49], our objective is to complement the labels of the image modality using a contrastive learning approach [50].  
**Base learner.** For each emotion \(k\in{1,2,\ldots,K}\), we define a binary classifier \(f_{k}(T,A)\), where the input consists of text and audio modalities, and the output provides a probability indicating whether the image modality corresponds to emotion \(k\)[51]. For each sample \((T_{i},A_{i},Y_{i})\), if \(Y_{i}=k\), we set \(Z_{ik}=1\); otherwise, we set \(Z_{ik}=0\). This results in a binary label matrix \(\mathbf{Z}\in{0,1^{N\times K}}\) for the image modality [52]. In this context, positive examples are defined as data that share the same modalities and synthetic sentiments as the current instance but differ in the missing modalities. For the case where the image modality is absent, we consider data whose images correspond to emotion A as positive examples for classifier A, while those whose images do not correspond to A are treated as negative examples.  
**Contrastive loss function.** The goal of contrastive learning is to maximize the similarity between positive examples and minimize the similarity between negative examples. We utilize cosine similarity to evaluate this similarity. For each emotion \(k\), we define the contrastive loss function \(L_{k}\) as follows:  
\[L_{k}=-\frac{1}{N}\sum_{i=1}^{N}Z_{ik}\log\frac{\exp(\text{sim}(v_{k}(T_{i},A_{ i}),v_{k}(T_{+},A_{+})))}{\sum_{j=1}^{N}\exp(\text{sim}(v_{k}(T_{i},A_{i}),v_{k}(T_{ j},A_{j})))} \tag{2}\]  
where \((T_{+},A_{+})\) is a positive sample that shares the same modalities and synthetic emotions as \((T_{i},A_{i})\) but differs in the image modality. The purpose of this loss function is to maximize the similarity between positive examples and minimize it between negative examples. To train the classifier \(f_{k}(T,A)\), we minimize the contrastive loss function \(L_{k}\) using stochastic gradient descent. After training, the classifier \(f_{k}(T,A)\) can be applied to predict whether any missing image modality data \((T,A)\) correspond to emotion \(k\). Specifically, we compute the output probability \(p_{k}(T,A)\) of \(f_{k}(T,A)\). This allows us to derive the emotional distribution \(p_{1}(T,A),p_{2}(T,A),\ldots,p_{K}(T,A)\) for the image modality. The emotion associated with the maximum probability is taken as the predicted emotion for the missing data, represented as  
\[\hat{Y}(T,A)=\text{argmax}_{k\in{1,2,\ldots,K}}p_{k}(T,A) \tag{3}\]  
The predicted label is concatenated as an additional dimension in the feature vector, thus ensuring that all modalities of each data instance are complete and enabling effective multimodal alignment.