Sentiment Knowledge Enhanced Self-supervised Learning for Multimodal Sentiment Analysis
Year: 2023

Abstract:
Multimodal Sentiment Analysis (MSA) has made great progress that benefits from extraordinary fusion scheme. However, there is a lack of labeled data, resulting in severe overfitting and poor generalization for supervised models applied in this field. In this paper, we propose Sentiment Knowledge Enhanced Selfsupervised Learning (SKESL) to capture common sentimental patterns in unlabeled videos, which facilitates further learning on limited labeled data. Specifically, with the help of sentiment knowledge and non-verbal behavior, SKESL conducts sentiment word masking and predicts fine-grained word sentiment intensity, so as to embed sentiment information at the word level into pre-trained multimodal representation. In addition, a non-verbal injection method is also proposed to integrate non-verbal information into the word semantics. Experiments on two standard benchmarks of MSA clearly show that SKESL significantly outperforms the baseline, and achieves new State-OfThe-Art (SOTA) results.

heading: 1 Introduction
text: Multimodal Sentiment Analysis (MSA) is a rapidly developing research field, which extends conventional text sentiment analysis to a multimodal setup where three modalities are present: text, audio and visual (Morency et al., 2011). With the abundance of user-generated opinion videos, MSA has a wide range of applications in e-commerce, intelligent customer service, human-computer interaction, etc.
In MSA, the construction of the training dataset relies on artificial perceptual evaluation for the sentiment of opinion videos, which is a very timeconsuming and labor-intensive task, that is why video data with sentimental annotation is insufficient. As a result, supervised models applied in this field suffer from severe overfitting and poor
† Corresponding author
generalization (Dai et al., 2021). Although previous studies have used several methods to alleviate the overfitting, most of them are based on general approaches such as multi-task learning (Dai et al., 2021; Akhtar et al., 2019; Chauhan et al., 2020; Yu et al., 2020), parameter regularization (Liang et al., 2019; Mai et al., 2020) and data augmentation (Liu et al., 2022), which neglect to consider the large number of unlabeled opinion videos that naturally exist on the Internet.
These opinion videos contain common sentimental patterns or compositional sentiment semantics about how the three modalities in the video are fused to express the overall sentiment, which can be leveraged to learn better sentiment representations. Inspired by recent knowledge-enhanced pretraining models on text sentiment analysis (Tian et al., 2020; Yin et al., 2020; Ke et al., 2020; Zhao et al., 2022), we argue that pre-training models enriched with the sentiment knowledge of words and non-verbal behavior will facilitate the characteriza-
12966
tion of the sentimental patterns in videos, thereby resulting in better performance on multimodal sentiment analysis.
In this paper, we propose a Sentiment Knowledge Enhanced Self-supervised Learning (SKESL) method, which uses contextual and non-verbal information to predict the fine-grained sentiment intensity of a word to learn the common sentimental patterns in opinion videos, as shown in Figure 1. Specifically, given a speaker video without sentiment annotation, we first use the Automatic Speech Recognition (ASR) technology to obtain the transcribed text and then mask the most sentimentally salient words in the text according to the pre-specified sentiment lexicon. A pre-trained language representation model is utilized to acquire the sequence representation of the processed text. To integrate non-verbal information into the text representations, we further propose a non-verbal information aggregation method based on the crossmodal attention mechanism to derive non-verbal information-enhanced text representations. Finally, the masked word representations are exploited to predict the sentiment intensity itself.
After the SKESL is completed, we transfer the pre-trained model to the task of multimodal sentiment analysis, and adopt a small amount of sentiment-annotated data to fine-tune the model. To evaluate the effectiveness of SKESL, we test on two benchmark datasets: CMU-MOSI (Zadeh et al., 2016) and CMU-MOSEI (Zadeh and Pu, 2018). Experimental results demonstrate that our model outperforms both the baseline and the current State-of-the-Art (SOTA) approach.
The main contributions can be summarized as follows:
• To the best of our knowledge, this paper is the first self-supervised learning method for multimodal sentiment analysis that leverages sentiment knowledge from large-scale unlabeled videos to facilitate improved sentiment representation learning.
• This paper proposes a novel non-verbal information aggregation method for obtaining text sequence representations enhanced by audio and visual information.
• The proposed SKESL method not only surpasses the baseline in experimental performance, but also achieves SOTA in the field of multimodal sentiment analysis.

heading: 2 Related Work
text: Pre-training Language Models In NLP, it has become a paradigm to pre-train language models on the large scale unlabeled data in an autoencoding (Devlin et al., 2019) or auto-regressive manner (Radford et al., 2018, 2019), and fine-tune the pre-trained models on the downstream tasks using task-specific labeled data.
Recently, more pre-trained language models have been proposed, which can be roughly divided into four categories (Ke et al., 2020): 1) Knowledge enhancement: Introducing domain-specific knowledge in the process of pre-training language representation models has been shown to be effective. A representative model is ERNIE (Zhang et al., 2019), which explicitly introduces knowledge graph to pretrained language models. 2) Transferability: On the basis of the general pre-training language model, further post-training is performed on more specific auxiliary tasks (Li et al., 2021). 3) Model compression: The compressed pre-trained language model can be widely applied in resource-constrained devices and tasks requiring real-time capability. The commonly used model compression methods include knowledge distillation (Sanh et al., 2019; Jiao et al., 2020), quantization (Shen et al., 2020) and pruning (Gordon et al., 2020). 4) Pre-training objectives: Aiming at rich and variable text expressions, many studies have further improved the text feature on basis of general Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) objective (Devlin et al., 2019). For instance, SpanBERT (Joshi et al., 2020) masks consecutive spans randomly instead of individual tokens, while BERT-WWM (Cui et al., 2021) utilizes the Whole Word Mask (WWM) strategy to impose the model to learn complete semantics.
Knowledge Enhanced Pre-training Language Models Incorporating external knowledge into pretraining language models has become prevalent and has been shown to be significant. Such external knowledge includes commonsense knowledge for tasks such as entity typing and relation classification (Zhang et al., 2019; Peters et al., 2019; Liu et al., 2020; Xiong et al., 2020), sentiment knowledge for sentiment analysis (Tian et al., 2020; Yin et al., 2020; Ke et al., 2020), word sense knowledge for word sense disambiguation (Levine et al., 2020), commonsense knowledge for commonsense reasoning and sarcasm generation (Klein and Nabi, 2020; Chakrabarty et al., 2020), legal knowledge
for legal element extraction (Zhong et al., 2020), and biomedical knowledge for health question answering and medical inference (He et al., 2020).
Knowledge Enhanced Pre-training Models for Text Sentiment Analysis Some research (Tian et al., 2020; Yin et al., 2020; Ke et al., 2020; Zhao et al., 2022) integrates the sentiment knowledge into the pre-training process which includes sentiment words, word polarity and aspect-sentiment pairs. The learned representation would be more sentiment-specific and appropriate for text sentiment analysis.
Knowledge Enhanced Models for Multimodal Sentiment Analysis In MSA, some works consider sentiment knowledge with explicit supervision. For example, SWAFN (Chen and Li, 2020) designs a sentimental words prediction objective as an auxiliary task to incorporate sentimental words knowledge. MAGCN (Xiao et al., 2022) also incorporates sentiment knowledge into inter-modality learning.

heading: 3 Methodology
text: In this section, we describe our proposed Sentiment Knowledge Enhanced Self-supervised Learning (SKESL) framework for multimodal sentiment analysis, as shown in Figure 2. The framework contains Sentiment Word Masking (SWM), text representation learning, non-verbal information injection (a.k.a., multimodal fusion), and Sentiment Intensity Prediction (SIP) modules. In the subsequent subsections, we will detail the four modules.

heading: 3.1 Formulation
text: Our task is defined as follows: given a set of three modalitiesM={T (Text),A(Audio),V (Visual)}, an opinion video, i.e., multimodal sequence, can be represented as Xm = {xm1 ,xm2 , ...,xmTm} ∈ RTm∗dm where m ∈ M−{T}, xmi ∈ Rdm denotes the extracted sentiment feature corresponding to modality m, dm is the dimension of the feature, and Tm is the length of sequence of modality m. Our goal is to predict the sentiment intensity y ∈ R or polarity y ∈ {positive, neutral, negative} of the whole video.

heading: 3.2 Sentiment Word Masking
text: Sentiment Word Masking (SWM) aims to construct a corrupted version for each input sequence where sentiment information is masked. For a speaker video without sentiment annotation, a good ASR technique is first exploited to transcribe the speech
to text S = {w1, w2, ..., wN}. As sentiment words in the text, especially those with the most salient sentiment, are the most essential clues in the textual modality for detecting sentiment, we employ a sentiment lexicon to search and then mask them, i.e., use special tokens in their place. The sentiment lexicon (Hutto and Gilbert, 2014) consists of explicit sentiment intensity scores for each sentiment word, thus we can easily find the sentiment words with the highest sentiment intensity. Meanwhile, the score yMASK of the highest sentiment intensity is chosen to act as a label for guiding SKESL. The corrupted sentence is represented as S′ = {w1, w2, ..., wMASK, ..., wN} where wMASK denotes the masked word.
It is worth noting that a sentence with sentiment tendencies does not necessarily have sentiment words. To cope with this situation, we adopt a random masking strategy and assign a label with the sentiment intensity “0.0” to the masked word. The motivation is that the pre-training model is induced to distinguish whether the masked position holds a word without any sentiment based on contextual and non-verbal information. In this way, the model has a stronger sentimental semantic cognition of the words in the sentence and can learn better sentimental multimodal representations.

heading: 3.3 Text representation learning
text: After getting the corrupted sentence S′, we need to encode it into a sequence of word representations for subsequent processing. Given the outstanding language representation capabilities and widespread use of BERT, we opted to utilize it as the text encoder and input the corrupted sentence S′ to it,
XT := {xT1 ,xT2 , ...,xTN} = fθLM ( S′ ) (1)
where θLM represents the parameters of BERT, xTi ∈ RdT denotes the encoded word representation, and dT is the dimension of the representation.

heading: 3.4 Non-verbal information injection
text: Unlike knowledge enhanced pre-training models towards text sentiment analysis, we emphasize that our SKESL deals with opinion videos that contain multiple modalities rather than just text. For the same word, there exists different sentiment with different non-verbal accompaniments. Therefore, the exact sentiment semantics of a word is determined by the word itself and the accompanied nonverbal behavior (Wang et al., 2019). Without the
help of non-verbal information, it is difficult for the pre-trained language model to determine the masked word and infer the sentiment intensity of the word. Therefore, to integrate the non-verbal information into word representations, inspired by Multimodal Transformer (MulT) (Tsai et al., 2019) which provides a latent cross-modal adaptation that fuses multimodal information by directly attending to low-level features in other modalities, we propose a new non-verbal information injection (a.k.a., multimodal fusion) method as shown in Figure 3.
The method repeatedly reinforces the text representations with the low-level features from audio and visual modalities by learning the attention across the features of two modalities. The low-level features benefit the model to preserve the original sentiment semantics for non-verbal behavior and learn the text-centric multimodal representations. Formally, we first define XT0 = X T , XV0 = X V and XA0 = X A to represent the text, visual and audio feature sequences before multimodal fusion, respectively. The Queries, Keys and Values sequences for Cross-Modal Attention (CMA) is computed by linear transformation as follows,
QmT = LN ( XTl−1 ) ·WmQ (2)
Km = LN(Xm0 ) ·WmK (3)
Vm = LN(Xm0 ) ·WmV (4)
where m ∈ M − {T}, LN(·) denotes the Layer Normalization, θCMA = {WmQ ∈ RdT×dT ,WmK ∈ Rdm×dT ,WmV ∈ Rdm×dT } are weights.
After obtaining Queries, Keys and Values sequences, we utilize the CMA to inject audio and visual information into text representations,
Yml = CMA ( QmT ,Km,Vm )
= softmax ( QmT ·Km√
dT
) ·Vm
(5)
where m ∈ M − {T}, Yml ∈ RN×dT denotes the text sequence enhanced by modality m. In this way, each word receives information from all the elements across audio and visual feature sequences. Then, the enhanced text representations Yml along with the previous text representation XTl−1 are aggregated together,
Yl = Y A l + LN ( XTl−1 ) +YVl (6)
Similar to the structure of vanilla Transformer (Vaswani et al., 2017), Yl ∈ RN×dT then goes through layer normalization, FeedForward Neural Network (FFNN) and residual connection,
XTl = fθFF (LN (Yl)) +Yl (7)
where θFF represents the parameters of FFNN, XTl ∈ RN×dT is the output of block l.

heading: 3.5 Sentiment Intensity Prediction
text: After L blocks, the refined text representations are XTL ∈ RN×dT , in which xTMASK,L is the refined representation of masked word. We simply use a 2-layer fully connected network with non-linear activation function to predict the sentiment intensity
of masked words.
ypred = fθFC ( xTMASK,L ) (8)
where θFC represents the parameters of the fully connected network, ypred is the predicted sentiment intensity. We define θ = {θLM, θCMA, θFF, θFC}, therefore the objective of the model is as follows,
θ∗ = argmin θ L(ypred, yMASK) (9)
where L is chosen as Mean Absolute Error (MAE) loss function. The model is pre-trained in an endto-end way.

heading: 3.6 Fine-tuning
text: We verify the effectiveness of SKESL on multimodal sentiment analysis task. On top of the pre-trained language model and multimodal fusion module, an output layer is added to perform taskspecific prediction. The neural network is then fine-tuned on labeled multimodal data.

heading: 4 Experiments

heading: 4.1 Datasets
text: We pre-train our models on two speaker video datasets: VoxCeleb1 (Nagrani et al., 2017) and
VoxCeleb2 (Chung et al., 2018) due to rich sentimental information within two datasets (Albanie et al., 2018). VoxCeleb1 and VoxCeleb2 contain over 100,000 video clips for 1,200+ speakers and over 1 million video clips for 6,000+ speakers collected from open-source media, respectively. Both datasets are approximately gender balanced, with speakers spanning a wide range of different ethnicities, accents, professions and ages. After a rough screening, we removed video clips that were not in English, and selected 132,708 video clips for 1,105 speakers from VoxCeleb1 and 947,726 video clips for 5,256 speakers from VoxCeleb2.
In addition, two multimodal sentiment datasets are used for fine-tuning and testing: CMU-MOSI (Zadeh et al., 2016), and CMU-MOSEI (Zadeh and Pu, 2018). The CMU-MOSI and CMUMOSEI datasets consist of 2,199 and 22,846 opinion video clips from YouTube movie reviews, respectively. Each video clip has been scored between −3 (strong negative) and +3 (strong positive). Following previous works (Tsai et al., 2019; Rahman et al., 2020; Qian et al., 2022), for CMUMOSI dataset, we utilize 1,284 segments for training, 229 segments for validation, and 686 segments for testing. For CMU-MOSEI dataset, we use 16,326 segments for training, 1,861 segments for validation, and 4,659 segments for testing. Table 1 and 2 show the statistics of all datasets.

heading: 4.2 Sentiment Features
text: We extract sentiment-related features for nonverbal modalities.
Audio: The library librosa (McFee et al., 2015) is used to extract frame-level acoustic features which include 1-dimensional logarithmic fundamental frequency (log F0), 20-dimensional MelFrequency Cepstral Coefficients (MFCCs) and 12-dimensional Constant-Q chromatogram (CQT). These features are related to emotions and tone of speech according to (Yu et al., 2020).
Visual: The MultiComp OpenFace2.0 toolkit (Baltrusaitis et al., 2018) is used to extract a set of visual features including 340-dimensional facial landmarks, 35-dimensional facial action units, 6-dimensional head pose and orientation, 40- dimensional rigid and non-rigid shape parameters, and 288-dimensional eye gaze 1.
1For more details, you can see https://github. com/TadasBaltrusaitis/OpenFace/wiki/ Output-Format

heading: 4.3 Experimental Design
text: Sentiment Lexicon We use the VADER 2 sentiment lexicon (Hutto and Gilbert, 2014) to search and mask sentiment words. The VADER sentiment lexicon is sensitive to both the polarity and the intensity of sentiments expressed in social media contexts. It contains rich sentiment words with explicit sentiment scores from −4 to +4. In order to be consistent with the CMU-MOSI and CMU-MOSEI datasets, we linearly scale the score to [−3,+3].
ASR We utilize the widely recognized Google Cloud Speech API 3 to acquire transcripts for the pretraining datasets. Considering that we do not have access to the actual transcripts, it is not possible to calculate the precise ASR word error rate. However, we can confidently state that improved ASR performance would result in better outcomes. This is because a low-performing ASR system may inaccurately identify sentiment words, potentially leading to flawed results.
Training Details All models are built on the Pytorch (Paszke et al., 2019) toolbox with the NVIDIA RTX 3090 GPUs. The Adam (Kingma and Ba, 2014) optimizer is adopted for both pretraining and fine-tuning. The initial learning rate is set to 5e - 6 for BERT and 1e - 4 for other parameters. The batch size is 32. The number of epoch is 200. All our experiments were done with the exact same random seed. The models use the designated validation set of CMU-MOSI and CMU-MOSEI for finding best hyper-parameters 4. You can refer to Appendices A for more details.
2You can find it at https://github.com/ cjhutto/vaderSentiment
3https://cloud.google.com/ speech-to-text
4Our codes are publicly available at https://github. com/qianfan1996/SKESL.git

heading: 4.4 Evaluation Metrics
text: Following previous works (Tsai et al., 2019; Yu et al., 2021), we record our experimental results in two forms: classification and regression. For classification, we report the weighted F1 score and binary accuracy. For regression, we report Mean Absolute Error (MAE) and Pearson correlation (Corr). Except for MAE, higher values denote better performance for all metrics.

heading: 4.5 Baseline Models
text: Our model does not require manual alignment of language words with visual and audio, since the unlabeled video data has no explicit word timestamps. We perform a comprehensive comparative study against SKESL by considering various baselines and state-of-the-art models in either aligned or unaligned settings as detailed below.

heading: 4.5.1 Aligned Setting
text: MARN (Zadeh et al., 2018b) models intra-modal and cross-modal interactions by designing the Long-short Time Hybrid Memory and Multiattention Block. MFN (Zadeh et al., 2018a) focuses on continuously modeling the view-specific and cross-view interactions, and aggregating them through time with a Multi-view Gated Memory. RMFN (Liang et al., 2018) decomposes the modeling process into multi-stage fusion, with each stage specifically targeting a subset of multimodal signals. RAVEN (Wang et al., 2019) considers the fine-grained structure of non-verbal subword sequences, and dynamically adjusts the word representations based on these non-verbal cues. MCTN (Pham et al., 2019) learns joint representations by cyclically translating from source to target modalities while ensures robustness even in the presence of noisy or missing target modalities. MISA (Hazarika et al., 2020) incorporates a combination of losses including distributional similarity, orthogonal loss, reconstruction loss and task prediction loss to learn both modality-invariant and modalityspecific representations. MAG-BERT (Rahman et al., 2020) is an improvement over RAVEN on aligned data with applying multimodal adaptation gate at different layers of the BERT backbone.

heading: 4.5.2 Unaligned Setting
text: MulT (Tsai et al., 2019) employs a cross-modal attention mechanism that enables a latent crossmodal adaptation, merging multimodal information by directly attending to low-level features in
other modalities. PMR (Lv et al., 2021) introduces a message hub to explore three-way interactions across all involved modalities within the context of multimodal fusion in unaligned multimodal sequences. LMR-CBT (Fu et al., 2021) achieves complementary learning of different modalities by incorporating three effective components: local temporal learning, cross-modal feature fusion and global self-attention representations. Self-MM (Yu et al., 2021) designs a label generation module to obtain independent unimodal supervisions, effectively balancing the learning progress across different sub-tasks.

heading: 5 Results and Analysis
text: In this section, we make a detailed analysis and discussion about our experimental results.

heading: 5.1 Quantitative Results
text: As shown in Table 3, our model achieves stateof-the-art performance on all the metrics on both datasets. Specifically, consistently significant improvement is observed compared to the previous unaligned models. Even comparing with aligned models, our model still achieves competitive or better results.
To investigate the impact of the amount of the unlabeled video data, we pre-train on VoxCeleb1 (132K) and VoxCeleb2 (947K) datasets and results are shown as Table 4 and 5. It is evident that a larger amount of pre-training data leads to more
significant performance improvements. Furthermore, we find that the performance improvement on the CMU-MOSEI dataset is larger than that on the CMU-MOSI dataset. For example, the accuracy is relatively improved by 0.39% and 1.07%, respectively. The most likely reasons are that the CMU-MOSI dataset is too small and contains noisy labels. Therefore, we boldly guess that it will be difficult to improve the performance on the CMUMOSI dataset in the future.
In addition, to study the effect of different sizes of backbone language models, we use bert-base and bert-large models with 110M parameters and 340M parameters, respectively. Results are shown in Table 6 and 7. We observed that the perfor-
mance improves as the pre-trained language model has more parameters and stronger expressiveness. This fits with our intuition and previous conclusion (Brown et al., 2020) that scaling up language models can greatly improve performance.

heading: 5.2 Ablation Study
text: To further explore the contributions of different components, we conduct an ablation study on CMU-MOSI dataset and the results are shown in Table 8. Without SKESL, the model’s accuracy and F1 score dropped by 1.40% and 1.46%, respectively. This suggests that it is indeed useful to transfer sentimental knowledge mined from unlabeled video data to downstream prediction tasks. Further, if the audio and visual modality is not used, i.e., only the BERT language model is used for the sentiment prediction task, the performance will be further degraded. This fact aligns with the observations in prior work (Tsai et al., 2019; Rahman et al., 2020; Qian et al., 2022) that multimodal sentiment
analysis is better than text-only sentiment analysis.

heading: 5.3 Case Study
text: To visually validate the reliability of our model, we present some examples shown in Figure 4. The examples are from the first three speaker videos of the test set of the CMU-MOSI dataset. The prediction results demonstrate that pre-training models enriched with the sentiment knowledge of words and non-verbal behavior will facilitate the characterization of the sentimental patterns in videos, thereby resulting in better performance on multimodal sentiment analysis.

heading: 6 Conclusion
text: In this paper, we highlighted the sentiment knowledge enhanced self-supervised learning in MSA. We find that mining sentimental prior information from unlabeled video data can lead to better predictions on labeled data. The larger the amount of unlabeled video data and the stronger the language modeling ability, the better the performance that can be achieved.

heading: 7 Limitations
text: We note that there are several limitations with such a sentiment knowledge enhanced self-supervised
learning approach. First, the preprocessing of massive videos is time-consuming and laborious. Second, the pre-training of our model has relatively large requirements on the GPU resources. Finally, we argue that there should not be too many videos without sentimental words, so as to avoid the model having a large bias and not learning any sentiment knowledge.

heading: A Appendices
text: A.1 Hyper-parameters Setting
A.2 Model Efficiency
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section 7
7 A2. Did you discuss any potential risks of your work? Our work does not have potential risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? section 4.5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendices A.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section 4.3 and Appendices A.1
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? As with the original paper for other work (e.g., MAG-BERT, MulT, Self-MM) in MSA, we do not report error bars. All experiments were done with the exact same random seed. We report the max after a grid search of hyperparameter using the validation sets.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 4.2, 4.3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Therefore, we boldly guess that it will be difficult to improve the performance on the CMUMOSI dataset in the future. In addition, to study the effect of different sizes of backbone language models, we use bert-base and bert-large models with 110M parameters and 340M parameters, respectively. Results are shown in Table 6 and 7. We observed that the perfor-
mance improves as the pre-trained language model has more parameters and stronger expressiveness