SlowGuess/NAACL_2022_OCR huggingface dataset:
CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection
Zhen Li, Bing Xu\*, Conghui Zhu, Tiejun Zhao
linklizhen@163.com,{hitxb,conghui,tjzhao}@hit.edu.cn
Compared with unimodal data, multimodal data can provide more features to help the model analyze the sentiment of data. Previous research works rarely consider token-level feature fusion, and few works explore learning the common features related to sentiment in multimodal data to help the model fuse multimodal features. In this paper, we propose a Contrastive Learning and Multi-Layer Fusion (CLMLF) method for multimodal sentiment detection. Specifically, we first encode text and image to obtain hidden representations, and then use a multi-layer fusion module to align and fuse the token-level features of text and image. In addition to the sentiment analysis task, we also designed two contrastive learning tasks, label based contrastive learning and data based contrastive learning tasks, which will help the model learn common features related to sentiment in multimodal data. Extensive experiments conducted on three publicly available multimodal datasets demonstrate the effectiveness of our approach for multimodal sentiment detection compared with existing methods. The codes are available for use at https://github.com/Link-Li/CLMLF
With the development of social networking platforms which have become the main platform for people to share their personal opinions. How to extract and analyze sentiments in social media data efficiently and correctly has broad applications. Therefore, it has attracted attention from both academic and industrial communities (Zhang et al., 2018a; Yue et al., 2019). At the same time, with the increasing use of mobile internet and smartphones, more and more users are willing to post multimodal data (e.g., text, image, and video) about different topics to convey their feelings and sentiments. So multimodal sentiment analysis has become a popular research topic (Kaur and Kautish, 2019).
(a) Heathrow. Fly early to-morrow morning. (positive)
(c) Ridge Avenue is closed after a partial building collapse and electrical fire Saturday night. (negative)
Figure 1: Examples of multimodal sentiment tweets
(b) Blue Jays game with the fam! Let's go! (positive)
(d) Flexible spinal cord implants will let paralyzed people walk. (neutral)
As for multimodal data, the complementarity between text and image can help the model analyze the real sentiment of the multimodal data. As shown in Figure 1, detecting sentiment with only text modality or image modality may not be certain of the true intention of the tweet. Such as Figure 1a, if we only analyze the text modality, we will find that this is a declarative sentence that does not express sentiment. In fact, the girl's smile in the image shows that the sentiment of this tweet is positive. At the same time, in Figure 1c, we can find that the ruins in the image which deepen the expression of negative sentiment in the text.
For multimodal sentiment analysis, we focus on text-image sentiment analysis in social media data. In existing works, some models try to concatenate different modal feature vectors to fuse the multimodal features, such as MultiSentiNet (Xu and Mao, 2017) and HSAN (Xu, 2017). Kumar and Vepa (2020) proposes to use gating mechanism and attention to obtain deep multimodal contextual feature vectors. Multi-view Attentional Network (MVAN) is proposed by Yang et al. (2020) which
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2282 - 2294
introduces memory networks to realize the interaction between modalities. Although the above mentioned models are relatively better than unimodal models, the inputs with different modalities are in different vector spaces. Therefore, it is difficult to fuse multimodal data with a simple concatenation strategy, so the improvement is also limited. Furthermore, the gating mechanism and memory network are essentially not designed for multimodal fusion. Although they can help the model analyzes the sentiment in the multimodal data by storing and filtering the features in the data, it is obvious that these methods are difficult to align and fuse the features of text and image. Since Transformers have achieved great success in many fields, such as natural language processing and computer vision (Lin et al., 2021; Khan et al., 2021), we propose Multi-Layer Fusion (MLF) module based on Transformer-Encoder. Benefiting from the multi-headed self-attention in Transformer, which can capture the internal correlation of data vectors. Therefore, text tokens and image patches with explicit and implicit relationships will have higher attention weight allocation to each other which means the MLF module can help align and fuse the token-level text and image features better. And MLF is a multi-layer encoder, which can help improve the abstraction ability of the model and obtain deep features in multimodal data.
Some previous work has explored the application of contrastive learning in the multimodal field. Huang et al. (2021) proposes the application of contrastive learning in multilingual text-to-video search, and Yuan et al. (2021) applies contrastive learning to learn visual representations that embraces multimodal data. However, there is little work to study the application of contrastive learning in multimodal sentiment analysis, so we propose two contrastive learning tasks, Label Based Contrastive Learning (LBCL) and Data Based Contrastive Learning (DBCL), which will help the model learn common features related to sentiment in multimodal data. For example, as shown in Figure 1a and Figure 1b. We can find that both tweets show positive sentiment. And we also can find there are smiling expressions in the image of the two tweets which is a common feature of those tweets. If the model can learn common features related to sentiment, it will greatly improve the performance of the model.
In this paper, we propose a Contrastive Learning
and Multi-Layer Fusion (CLMLF) method for multimodal sentiment analysis based on text and image modalities. For evaluation, CLMLF is verified on three multimodal sentiment datasets, namely MVSA-Single, MVSA-Multiple (Niu et al., 2016) and HFM (Cai et al., 2019). CLMLF achieves better performance compared to several baseline models in all three datasets. Through a comprehensive set of ablation experiments, case study, and visualizations, we demonstrate the advantages of CLMLF for multimodal fusion<sup>1</sup>. Our main contributions are summarized as follows:
- We propose a multi-layer fusion module based on Transformer-Encoder that multi-headed self-attention can help align and fuse token-level features of text and image, and it can also benefit from the depth of MLF which improves model abstraction ability. Experiments show that the proposed architecture of MLF is simple but effective.
- We propose two contrastive learning tasks based on label and data, which leverages sentiment label features and data augmentation. Those two contrastive learning tasks can help the model learn common features related to sentiment in multimodal data, which improve the performance of the model.
In this section, we will introduce CLMLF. Figure 2 illustrates the overall architecture of CLMLF model for multimodal sentiment detection that consists of two modules: multi-layer fusion module and multi-task learning module. Specifically, the multi-layer fusion module is on the right in Figure 2, it includes a text-image encoder, image Transformer layer, and text-image Transformer fusion layer modules. The multi-task learning module is on the left in Figure 2, it includes three tasks, sentiment classification, label based contrastive learning and data based contrastive learning tasks.
2.2 Multi-Layer Fusion Module
We use Multi-Layer Fusion module to align and fuse the token-level features of text and image. As shown on the right of Figure 2. First, we
There are also the experimental results and analysis of CLMLF in aspect based multimodal sentiment analysis task, which can refer to Appendix B
Figure 2: The framework of the proposed CLMLF model
use BERT (Devlin et al., 2019) and ResNet (He et al., 2015) to encode the text and image to obtain the hidden representation of the text  $T = \{t_{C}, t_{1}, t_{2}, \dots, t_{S}\}$ ,  $T \in \mathbb{R}^{n_{t} \times d_{t}}$  and the hidden representation of the image  $I_{c}^{\prime} \in \mathbb{R}^{p_{i} \times p_{i} \times d_{i}}$ , and  $I_{c}^{\prime}$  is the image feature map output by the last layer of convolution layer of ResNet. We transform the hidden representation dimension of  $I_{c}^{\prime}$  into the same dimension as the  $T$ . And we can get the sequence feature representation of the image  $I^{\prime}$  as follows:
$$
I ^ {\prime} = \text {f l a t t e n} \left(I _ {c} ^ {\prime} W _ {I} + b _ {I}\right) \tag {1}
$$
Where  $I^{\prime} = \{i_{1}^{\prime},i_{2}^{\prime},\dots,i_{n_{i}}^{\prime}\}, I^{\prime}\in \mathbb{R}^{n_{i}\times d_{t}}, n_{i} = p_{i}\times p_{i}$ . And the function of flatten means flatten the input vector by reshaping the first two-dimensions into a one-dimensional.
After that, we will encode the image sequence features  $I^{\prime}$ . Here we use the vanilla TransformerEncoder proposed by Vaswani et al. (2017). Input  $I^{\prime}$  into the image Transformer layer which is based on a multi-layer Transformer-Encoder to obtain the final encoding of image sequence features  $I$ .
$$
\left\{i _ {1}, i _ {2}, \dots , i _ {n _ {i}} \right\} = T E _ {I} \left(\left\{i _ {1} ^ {\prime}, i _ {2} ^ {\prime}, \dots , i _ {n _ {i}} ^ {\prime} \right\}\right) \tag {2}
$$
$$
I = \left\{i _ {1}, i _ {2}, \dots , i _ {n _ {i}} \right\} \tag {3}
$$
Where  $TE_{I}$  means the vanilla TransformerEncoder of image.
In order to align and fuse the features of text and images, we concatenate the features of the text  $T$  and the image sequence features  $I$ . We use a new multi-layer Transformer-Encoder as a text-image fusion layer which will align and fuse multimodal features. Then the fusion sequence features of text and image can be obtained. It is as follows:
$$
\left\{f _ {1}, f _ {2}, \dots , f _ {n _ {t} + n _ {i}} \right\} = T E _ {M} (\text {c o n c a t} (T, I)) \tag {4}
$$
$$
F = \left\{f _ {1}, f _ {2}, \dots , f _ {n _ {t} + n _ {i}} \right\} \tag {5}
$$
Where  $T E_{M}$  means the vanilla Transformer-Encoder of multimodal data.
Now, we obtain the sequence features of text and image fusion, but it is obvious that the sequence features can not be used in the classification task. So we use a simple attention layer to get the multimodal representation  $R$ .
$$
\tilde {q} _ {i} = G E L U \left(f _ {i} W _ {1} + b _ {1}\right) W _ {2} + b _ {2} \tag {6}
$$
$$
q _ {i} = \exp \left(\frac {\tilde {q} _ {i}}{\sum_ {j = 1} ^ {n _ {t} + n _ {i}} \tilde {q} _ {j}}\right) \tag {7}
$$
$$
\tilde {R} = \sum_ {i = 1} ^ {n _ {t} + n _ {i}} q _ {i} f _ {i} \tag {8}
$$
$$
R = G E L U \left(\tilde {R} W _ {R} + b _ {R}\right) \tag {9}
$$
where  $GELU$  is the activation function.  $R \in \mathbb{R}^{d^t}$
2.3 Sentiment Classification
As shown in the SC task in Figure 2, we feed the above multimodal representation  $R$  into the fully connected layer and employ the softmax function for sentiment detection. We use the cross-entropy loss as the classification loss and it is as follows:
$$
L _ {s c} = \text {C r o s s - E n t r o p y} (G E L U \left(R W _ {s c} + b _ {s c}\right)) \tag {10}
$$
2.4 Label Based Contrastive Learning
In order to let the model learn the sentiment related features in the multimodal data, we use label based contrastive learning task to help the model extract the sentiment related features while MLF module fuses text and image data. As shown in the LBCL
task in Figure 2, we divide the data in each batch into positive and negative examples according to its sentiment label. For example, in Figure 2, for a negative label of multimodal data, the data in the batch with the same negative labels as positive examples (the square of pink color), and the data with no negative labels are taken as negative examples (the square of gray color).
The specific step can refer to Algorithm 1. The meanings of specific functions in the algorithm are as follows: *einsum* means Einstein summation convention, *gather* means gathers values along with an index, and *τ* represents the contrastive learning's temperature. The algorithm consists of two main steps: the first step is to generate the unmask label  $L_{t}$  according to the data labels in the batch; the second step is to calculate the loss matrix  $l_{pn}$ , and use the unmask label  $L_{t}$  and the loss matrix  $l_{pn}$  to get the final loss  $L_{l - cl}$ , which are the water-red elements in LBCL task on the left in Figure 2.
Algorithm 1 LBCL Algorithm
Require: The sentiment label is  $L$ , which is a list of all data in the batch, assuming that the sentiment is divided into three categories: positive (0), neutral (1) and negative (2); The MultiLayer Fusion Model of MLF; the texts are  $T$ ; the images are  $I$ ;  $C$  denotes length of  $L_{c}$ ;  $S$  denotes length of  $L$ .
Ensure: Label contrastive learning loss  $L_{l - cl}$
1: initialize  $L_{c} = [L - 0, L - 1, L - 2]$  and  $L_{t} = list()$
2: for  $i = 1$ ;  $i <= C$ ;  $i + + \mathbf{d}\mathbf{o}$
3: initialize  $\tilde{L}_t = list()$
4: for  $l = 1$ ;  $l <= T$ ;  $l + + \mathbf{d}$
5: if  $L_{c}[i][j]$  equals 0 then
6:  $L_{t}$  .append(j)
7: end if
8: end for
9:  $L_{t}$  .append  $(\tilde{L}_t)$
10: end for
11:  $R = MLF(T,I)$
12:  $\tilde{l}_{pn} = \text{einsum}(nc, ck - > nk, [R, R^T])$
13:  $l_{pn} = \text{LogSoftmax}(l_{pn} / \tau)$ .view(-1)
14:  $L_{cl} = L_t[L[1]]$
15: for  $q = 2$ ;  $q <   = S,q + + \mathbf{do}$
16:  $L_{cl} = \text{concat}(L_{cl}, L_t[L[q]] + q \times T)$
17: end for
18:  $L_{lbcl} = \text{gather}(l_{pn}, \text{index} = L_{cl}) / T$
19: return  $L_{lbcl}$
2.5 Data Based Contrastive Learning
In order to strengthen the robustness of the model to the data and enhance the learning ability of the model to the invariant features in the data. We add a contrastive learning task based on data augmentation which is DBCL task in Figure 2. Considering the flexible expression of text and images. It may cause the model to be too sensitive to the surface features of data, rather than focus on fusing the invariant features in text and images, that is, effective features. Sentiment related features should exist in these effective features, because the true meaning of the meaning user wants to express should not change with the changes in text and images. For example, both "I had ice cream today. I was very happy" and "I'm very happy today because I ate ice cream" express positive sentiment. The keyword "happy" has not changed which means the happy is an effective feature, but some other words have changed greatly. The data based contrastive learning can force the model learning the effective features in the data, which is more conducive to the model to learn the features related to sentiment in the data. Algorithm 2 describes the process of data based contrastive learning.
Specifically, as for text, we use a data augmentation method called back-translation (Sennrich et al., 2016; Edunov et al., 2018; Xie et al., 2020), which refers to the procedure of translating an existing text  $x$  in language  $E$  into another language  $C$  and then translating it back into  $E$  to obtain an augmented text  $x$ . As observed by Yu et al. (2018), back-translation can generate diverse paraphrases while preserving the semantics of the original sentences. So we use back-translation to construct positive examples of text in contrastive learning.
For image augmentation, we use a method called RandAugment (Cubuk et al., 2020), which is inspired by AutoAugment (Cubuk et al., 2018). AutoAugment uses a search method to combine all transformations to find a good augmentation strategy. In RandAugment, it does not use search, but instead uniformly samples from the same set of augmentation transformations. In other words, RandAugment is simpler and requires no labeled data as there is no need to search for optimal policies.
2.6 Model Training
The label contrastive loss or data contrastive loss can be simply added to the total loss as a regularization. Can be combined like follows:
Algorithm 2 DBCL Algorithm
Require: The Multi-Layer Fusion Model of  $MLF$ ; the texts are  $T$ ; the images are  $I$ ;  $BT$  means back-translation and  $RA$  means RandAugment;  $T$  denotes of batch size.
Ensure: Data contrastive learning loss  $L_{d - cl}$
1:  $R = MLF(T,I)$
2:  $R_{au} = MLF(BT(T),RA(I))$
3:  $l_{pn} = \text{einsum}(nc, ck - > nk, [R, R_{au}^{T}])$
4: cl_label = arange(T)
5:  $L_{dbcl} = \text{Cross-Entropy}(l_{pn} / \tau, \text{cl_label})$
6: return  $L_{dbcl}$
$$
L = L _ {s c} + \lambda_ {l b c l} L _ {l b c l} + \lambda_ {d b c l} L _ {d b c l} \tag {11}
$$
where  $\lambda_{lbcl}$  and  $\lambda_{dbcl}$  are coefficients to balance the different training losses.
We demonstrate the effectiveness of our method on three public datasets which are MVSA-Single, MVSA-Multiple $^2$  (Niu et al., 2016) and HFM $^3$  (Cai et al., 2019). Both datasets collect data from Twitter, each text-image pair is labeled by a single sentiment. For a fair comparison, we process the original two MVSA datasets in the same way used in Xu and Mao (2017), as for HFM, we adopt the same data preprocessing method as that of Cai et al. (2019). We randomly split the MVSA datasets into train set, validation set, and test set by using the split ratio 8:1:1. The statistics of these datasets are given in Table 2. The detailed statistics of these datasets are given in Appendix A.
For the experiments of CLMLF, we use the Pytorch and HuggingFace Transformers (Wolf et al., 2020) as the implementation of baselines and our method. We use the Bert-base and ResNet-5 as the text and image encoder in Multi-Layer Fusion module. The batch size is set to 32, 64 and 128 for MVSA-Single, MVSA-Multiple and HFM. We
use AdamW optimizer. The  $\epsilon$  is 1e-8 and  $\beta$  is (0.9, 0.999). The learning rate is set to 2e-5. Both  $\lambda_{lbcl}$  and  $\lambda_{dbcl}$  are set to 1.0 in Equation 11 during training. For the number of layers of MLF, please refer to Section 4.3. And all the experiments are done on four NVIDIA 3090 GPUs.
3.3 Compared Methods
We compare our model with the unimodal sentiment models and the multimodal baseline models.
Unimodal Baselines: For text modality, CNN (Kim, 2014) and Bi-LSTM (Zhou et al., 2016) are well-known models for text classification tasks. TGNN (Huang et al., 2019) is a text-level graph neural network for text classification. BERT (Devlin et al., 2019) is a pre-trained model for text, and we fine-tuned on the text only. For image modality, OSDA (Yang et al., 2020) is an image sentiment analysis model based on multiple views. ResNet (He et al., 2015) is pre-trained and fine-tuned on the image only.
Multimodal Baselines: MultiSentiNet (Xu and Mao, 2017) is a deep semantic network with attention for multimodal sentiment analysis. HSAN (Xu, 2017) is a hierarchical semantic attentional network based on image captions for multimodal sentiment analysis. Co-MN-Hop6 (Xu et al., 2018) is a co-memory network for iteratively modeling the interactions between multiple modalities. MGNNS (Yang et al., 2021) is a multi-channel graph neural networks with sentiment-awareness for image-text sentiment detection. Schifanella et al. (2016) concatenates different feature vectors of different modalities as multimodal feature representation. Concat(2) means concatenating text features and image features, while Concat(3) has one more image attribute features. MMSD (Cai et al., 2019) fuses text, image, and image attributes with a multimodal hierarchical fusion model. Xu et al. (2020) proposes the D&R Net to fuse text, image, and image attributes by constructing the Decomposition and Relation Network.
4.1 Overall Result
Table 1 illustrates the performance comparison of our CLMLF model with the baseline methods. We use Weighted-F1 and ACC as the evaluation metrics for MVSA-Single and MVSA-Multiple which is the same as Yang et al. (2021) and use Macro-F1 and ACC as the evaluation metrics for HFM. we
$^{2}$ http://mcrlab.net/research/mvsa-sentiment-analysis-on-multi-view-social-data/
<sup>3</sup>https://github.com/headacheboy/data-of-multimodal-sarcasm-detection
4https://pytorch.org/
$^{5}$ https://github.com/huggingface/transformers
<sup>6</sup>https://huggingface.co/bert-base-uncased
<sup>7</sup>https://pytorch.org/vision/stable/models.html
Modality	Model	MVSA-Single	MVSA-Multiple	Model	HFM
Acc	F1	Acc	F1	Acc	F1
Text	CNN	0.6819	0.5590	0.6564	0.5766	CNN	0.8003	0.7532
BiLSTM	0.7012	0.6506	0.6790	0.6790	BiLSTM	0.8190	0.7753
BERT	0.7111	0.6970	0.6759	0.6624	BERT	0.8389	0.8326
TGNN	0.7034	0.6594	0.6967	0.6180
Image	ResNet-50	0.6467	0.6155	0.6188	0.6098	ResNet-50	0.7277	0.7138
OSDA	0.6675	0.6651	0.6662	0.6623	ResNet-101	0.7248	0.7122
Multimodal	MultiSentiNet	0.6984	0.6984	0.6886	0.6811	Concat(2)	0.8103	0.7799
HSAN	0.6988	0.6690	0.6796	0.6776	Concat(3)	0.8174	0.7874
Co-MN-Hop6	0.7051	0.7001	0.6892	0.6883	MMSD	0.8344	0.8018
MGNNS	0.7377	0.7270	0.7249	0.6934	D&R Net	0.8402	0.8060
CLMLF	0.7533	0.7346	0.7200	0.6983	CLMLF	0.8543	0.8487
Table 1: Experimental results of different models on MVSA-Single, MVSA-Multiple and HFM datasets
Dataset	Train	Val	Test	Total
MVSA-S	3611	450	450	4511
MVSA-M	13624	1700	1700	17024
HFM	19816	2410	2409	24635
Table 2: Statistics of the three datasets
have the following observations. First of all, our model is competitive with the other strong baseline models on the three datasets. Second, the multimodal models perform better than the unimodal models on all three datasets. What is more, we found the sentiment analysis on the image modality gets the worst results, this may be that the sentimental features in the image is too sparse and noisy, which makes it difficult for the model to obtain effective features for sentiment analysis. At last, for simple tasks, the performance improvement of multimodal models is limited. For example, on HFM dataset, the improvement of CLMLF relative to BERT is less than MVSA-Single dataset that because HFM is a binary classification task, while MVSA-Single is a three classification task.
We also try to apply CLMLF to aspect based multimodal sentiment analysis task which can refer to Appendix B for details.
We further evaluate the influence of multi-layer fusion module, label based contrastive learning, and data based contrastive learning. The evaluation results are listed in Table 3. The Result shows that the whole CLMLF model achieves the best performance among all models. We can see multi-layer fusion module can improve the performance, which
shows that a multi-layer fusion module can fuse the multimodal data. On this foundation, adding the label and data based contrastive learning can improve the model performance more, which means contrastive learning can lead the model to learn common features about sentiment and lead different sentiment data away from each other.
4.3 Influence of MLF Layer
We explored the effects of different layers of Transformer-Encoder on the results. As shown in Figure 3a, fix the image transformer layer and set the text-image transformer fusion layer from 1 to 6. As shown in Figure 3b, fix the text-image transformer fusion layer and set the image transformer layer from 1 to 3. Finally, we selected different combinations of 3-2 (which means three layers of text-image transformer fusion layer and two layers of image transformer layer), 4-2, and 5-1 for the three datasets. This also proves that the contribution of text and images in the dataset is different. It can be seen from Table 1 that CLMLF gains more from the text than images in HFM dataset. Therefore, in MLF module, the layers of transformer related to text are more than images.