Venue: EMNLP 2023

M2DF: Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysis

Fei Zhao\({}^{1}\) Chunhui Li\({}^{1}\) Zhen Wu\({}^{1}\) Yawen Ouyang\({}^{2}\)

Jianbing Zhang\({}^{1}\) Xinyu Dai\({}^{1}\)

\({}^{1}\)National Key Laboratory for Novel Software Technology, Nanjing University, China

\({}^{2}\)Institute for AI Industry Research (AIR), Tsinghua University

{zhaof, lich}@smail.nju.edu.cn, ouyangyawen@air.tinghua.edu.cn

Corresponding author.

###### Abstract

Multimodal Aspect-based Sentiment Analysis (MABSA) is a fine-grained Sentiment Analysis task, which has attracted growing research interests recently. Existing work mainly utilizes image information to improve the performance of MABSA task. However, most of the studies overestimate the importance of images since there are many noisy images unrelated to the text in the dataset, which will have a negative impact on model learning. Although some work attempts to filter low-quality noisy images by setting thresholds, relying on thresholds will inevitably filter out a lot of useful image information. Therefore, in this work, we focus on whether the negative impact of noisy images can be reduced without filtering the data. To achieve this goal, we borrow the idea of Curriculum Learning and propose a Multi-grained Multi-curriculum Denoising Framework (M2DF), which can achieve denoising by adjusting the order of training data. Extensive experimental results show that our framework consistently outperforms state-of-the-art work on three sub-tasks of MABSA. Our code and datasets are available at [https://github.com/grandchicken/M2DF](https://github.com/grandchicken/M2DF).

+
Footnote †: Corresponding author.

+
Footnote †: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9057–9070

December 6-10, 2023 ©2023 Association for Computational Linguistics

## 1 Introduction

Multimodal Aspect-based Sentiment Analysis (MABSA) is a fine-grained Sentiment Analysis task Liu et al. (2012); Pontiki et al. (2014), which has received extensive research attention in the past few years. MABSA has derived a number of subtasks, which all revolve around predicting several sentiment elements, i.e., aspect term(a), sentiment polarity(p), or their combinations. For example, given a pair of sentence and image, Multimodal Aspect Term Extraction (MATE) aims to extract all the aspect terms mentioned in a sentence Wu et al. (2020), Multimodal Aspect-Oriented Sentiment Classification (MASC) aims to identify the sentiment polarity of a given aspect term in a sentence Yu and Jiang (2019), whereas the goal of Joint Multimodal Aspect-Sentiment Analysis (JMASA) is to extract all aspect terms and their corresponding sentiment polarities simultaneously Ju et al. (2021). We illustrate the definitions of all the sub-tasks with a specific example in Table 1.

The assumption behind these sub-tasks is that the image information can help the text content identify the sentiment elements Yu and Jiang (2019); Ju et al. (2021), so how to make use of image information lies at the heart of the MABSA task. To this end, a lot of efforts have been devoted, roughly divided into three categories: (1) transforming the image into a global feature vector and utilizing attention mechanisms to sift through visual content pertinent to the text Moon et al. (2018); Xu et al. (2019); (2) partitioning the image into multiple vi

\begin{table}
\begin{tabular}{l c c}  & **Sub-tasks** & **Input** & **Output** \\ \hline Multimodal Aspect Term & \multirow{2}{*}{\(\mathbf{S}+\mathbf{I}\)} & \multirow{2}{*}{\(a_{1}\), \(a_{2}\)} \\ Extraction (MATE) & & \\ \cline{2-3} Multimodal Aspect-Oriented & & \(\mathbf{S}+\mathbf{I}+a_{1}\) & \(p_{1}\) \\ \cline{2-3} Sentiment Classification (MASC) & & \(\mathbf{S}+\mathbf{I}+a_{2}\) & \(p_{2}\) \\ \hline Joint Multimodal Aspect-Sentiment & & \(\left(a_{1}\text{, }p_{1}\right)\), \\ Analysis (JMASA) & & \(\mathbf{S}+\mathbf{I}\) & \(\left(a_{2}\text{, }p_{2}\right)\) \\ \end{tabular}
\end{table}
Table 1: The definitions of all MABSA subtasks. Aspect terms and their corresponding sentiment polarities are highlighted in the sentence \(\mathbf{S}\).

 

sual regions averagely, and subsequently establishing connections between text sequences and visual regions (Yu and Jiang, 2019; Yu et al., 2020; Ju et al., 2021); (3) retaining exclusively the regions containing visual objects in the image and enabling dynamic interactions with the text sequences (Wu et al., 2020, 2020; Zhang et al., 2021; Yu et al., 2022; Ling et al., 2022; Yang et al., 2022).

Despite their promising results, most of the studies overestimate the importance of images since there are many noisy images unrelated to the text in the dataset, which will have a negative impact on model learning. In the MABSA task, Sun et al. (2020, 2021), Ju et al. (2021), Xu et al. (2022) and Yu et al. (2022) developed the cross-modal relation detection module that can filter low-quality noisy images by setting thresholds. However, relying on thresholds will inevitably filter out a lot of useful image information. Thus, in this work, we focus on another aspect: _can we reduce the negative impact of noisy images without filtering the data?_

To answer the above question, we borrow the idea of Curriculum Learning (CL) (Bengio et al., 2009), because CL can achieve denoising by adjusting the order of training data (Wang et al., 2022). In other words, CL no longer presents training data in a completely random order during training, but encourages training more time on clean data and less time on noisy data, which reveals the denoising efficacy of CL on noisy data. The key to applying CL is how to define clean/noisy examples and how to determine a proper denoising scheme. By analyzing the characteristics of the task, we tailor-design a **M**ulti-grained **M**ulti-curriculum **D**enoising **F**ramework (M2DF). M2DF first evaluates the degree of noisy images contained in each training instance from multiple granularities, and then the denoising curriculum gradually exposes the data containing noisy images to the model starting from clean data during training. In this way, M2DF theoretically assigns a larger learning weight to clean data (Gong et al., 2016), thereby achieving the effect of noise reduction.

Our contributions are summarized as follows: (1) To the best of our knowledge, we provide a new perspective to reduce the negative impact of noisy images in the MABSA task; (2) We propose a novel Multi-grained Multi-curriculum Denoising Framework, which is agnostic to the choice of base models; (3) We evaluate our denoising framework on some representative models, including the current state-of-the-art. Extensive experimental results show that our denoising framework achieves competitive performance on three sub-tasks.

## 2 Related Work

In this section, we review the existing studies on Multimodal Aspect-based Sentiment Analysis (MABSA) and Curriculum Learning (CL).

### Multimodal Aspect-based Sentiment Analysis

As an important sentiment analysis task, various neural networks have been proposed to deal with the three sub-tasks of MABSA, i.e., Multimodal Aspect Term Extraction (MATE), Multimodal Aspect-Oriented Sentiment Classification (MASC) and Joint Multimodal Aspect-Sentiment Analysis (JMASA). According to the different ways of utilizing image information, their work can be divided into three categories:

(1) Transforming the image into a global feature vector and utilizing well-designed attention mechanisms to sift through visual content pertinent to the text. Moon et al. (2018) used an attention mechanism to extract primary information from word embedding, character embedding, and global image vector. Xu et al. (2019) proposed a multi-interactive memory network to capture the multiple correlations in multimodal data.

(2) Partitioning the image into multiple visual regions averagely, and subsequently establishing connections between text sequences and visual regions. Yu and Jiang (2019) used the pre-trained model BERT (Devlin et al., 2019) and ResNet (He et al., 2016) to extract aspect and visual regions features, and devised an attention mechanism to obtain aspect-sensitive visual representations. Yu et al. (2020) proposed an entity span detection module to alleviate the visual bias. Sun et al. (2020), Sun et al. (2021) and Ju et al. (2021) introduced cross-modal relation detection module to decide whether the image feature is useful. Zhao et al. (2022a) proposed to utilize the external matching relations between different (text, image) pairs to improve the performance. Zhao et al. (2022b) developed a Knowledge-enhanced Framework to improve the visual attention capability.

(3) Retaining exclusively the regions containing visual objects in the image and enabling dynamic interactions with the text sequences. Wu et al. (2020, 2020) and Zheng et al. (2021) employed Faster-RCNN (Ren et al., 2015) or Mask-RCNN (He et al., 2020) to extract object features, and then interact 

with textual features via a co-attention network. Zhang et al. (2021) utilized a multi-modal graph to capture various semantic relationships between words and visual objects in a (text, image) pair.

Despite their success, most studies overlooked the fact that there are many noisy images in the dataset that are unrelated to the text, which negatively impacts model learning. While some efforts Sun et al. (2020, 2021); Yu et al. (2022); Ju et al. (2021) have been made to develop cross-modal relation detection modules to filter out low-quality noisy images, these approaches are threshold-dependent and inevitably filter out lots of useful image information. Therefore, in this work, we focus on whether the negative impact of noisy images can be reduced without data filtering.

Curriculum Learning.The concept of Curriculum Learning (CL) is initially introduced by Bengio et al. (2009), and has since manifested its multifaceted advantages across a diverse array of machine learning tasks Platanios et al. (2019); Wang et al. (2019); Liu et al. (2020); Xu et al. (2020); Lu and Zhang (2021). One of the advantages of CL is denoising, which is achieved by training more time on clean data and less time on noisy data Wang et al. (2022). In this work, we borrow the idea of CL to reduce the negative effect of noisy images. To the best of our knowledge, this is the first attempt to use CL for denoising in the MABSA task.

## 3 Methodology

### Overview

As shown in Figure 1, we propose a Multi-grained Multi-curriculum Denoising Framework (M2DF) for the MABSA task, which consists of two modules: _Noise Metrics_ and _Denoising Curriculums_. To be specific, when the image and text are unrelated, it is likely that the image is noise. In such cases, the similarity between the image and text is usually low, and the aspect terms in the text will generally not appear in the image. Based on this, we severally define a _coarse-grained (sentence-level) noise metric_ and a _fine-grained (aspect-level) noise metric_ to measure the degree of noisy images contained in each training instance.

Subsequently, we design a _single denoising curriculum_ for each noise metric, which first ranks all training instances according to the size of noise metric, and then gradually exposes the training data containing noisy images to the model starting from clean data during training. In this manner, the model trains more time on the clean data and less time on the noisy data, so as to reduce the negative impact of noisy images. Given that the two noise metrics measure the noise level of the image from different granularities, a natural idea is to combine them together to produce better denoising effects. To this end, we extend the _single denoising curriculum_ into _multiple denoising curriculum_.

### Noise Metrics

In this section, we define the _Noise Metrics_ used by M2DF from two granularities.

#### 3.2.1 Coarse-grained Noise Metric

Empirically, when the image and text are relevant, the similarity between them tends to be relatively high. Conversely, when the image and text are irrelevant, the similarity between them is generally low. Thus, it can be inferred that the lower the similarity between the image and the text, the more likely the image is noise.

Figure 1: The overview of our M2DF framework. The solid line 1 indicates the coarse-grained denoising curriculum is selected at the current training step, and the dashed line 2 indicates it is not selected.

 

Based on the above intuition, we measure the degree of noisy images contained in each training instance by calculating the similarity between the image and the text. Specifically, given a sentence-image pair (\(S_{i}\), \(I_{i}\)) in the training dataset \(\mathcal{D}\), we first input each modality into different encoders of pre-trained model CLIP Radford et al. (2021) to obtain the textual feature \(H^{i}_{S}\) and visual feature \(V^{i}_{I}\). Then the similarity is calculated as:

\[\text{sim}(H^{i}_{S},V^{i}_{I})=\text{cos}(H^{i}_{S},V^{i}_{I}), \tag{1}\]

where cos(\(\cdot\)) is a cosine function. After that, considering the lower the similarity between the image and the text, the more likely the image is noise, we define a coarse-grained (sentence-level) noise metric as follows:

\[d_{c}=1.0-\frac{\text{sim}(H^{i}_{S},V^{i}_{I})}{\max_{(H^{k}_{S},V^{k}_{I}) \in\mathcal{D}}\text{sim}(H^{k}_{S},V^{k}_{I})}, \tag{2}\]

where \(d_{c}\) is normalized to [0.0, 1.0]. Here, a lower score indicates the pair (\(S_{i}\), \(I_{i}\)) is clean data.

#### 3.2.2 Fine-grained Noise Metric

Except for sentence-level noise metric, we also measure the noise level from the perspective of aspect, because the goal of MABSA is to identify the aspect terms or predict the sentiment of aspect terms. Specifically, when the image and text are relevant, the aspect terms in the sentence will appear in the image with high probability. In contrast, when the image and text are irrelevant, the aspect terms in the sentence will generally not appear in the image. Thus, we can judge the relevance of the image and the text by calculating the similarity between the aspect terms in the sentence and the visual objects in the image. The lower the similarity is, the image is more likely to be noise.

Based on the above intuition, we measure the degree of noisy images by calculating the similarity between the aspect terms in the sentence and the visual objects in the image. However, except for the MASC task, the aspect terms of other subtasks are unknown. Considering that aspect terms are mostly noun phrases Hu and Liu (2004), we employ noun phrases extracted by the Stanford parser1 as aspect terms within sentences for the MATE and JMASA tasks. Meanwhile, we use the object detection model Mask-RCNN to extract the visual objects in the image. The number of aspect terms and visual objects extracted from the sentence \(S_{i}\) and the image \(I_{i}\) are as follows:

Footnote 1: [https://nlp.stanford.edu/software/lex-parser.shtml](https://nlp.stanford.edu/software/lex-parser.shtml)

\[d_{aspect}(S_{i}) \triangleq N_{S_{i}}, \tag{3}\] \[d_{object}(I_{i}) \triangleq N_{I_{i}}, \tag{4}\]

Then, we input aspect terms in sentence \(S_{i}\) and visual objects in image \(I_{i}\) into different encoders of pre-trained model CLIP to obtain a group of aspect features \(\left\{H^{ij}_{a}\right\}_{j=1}^{N_{S_{i}}}\) and object features \(\left\{V^{ij}_{o}\right\}_{j=1}^{N_{I_{i}}}\). After that, we average all these aspect features and object features to obtain the final aspect representation and object representation. The cross-modal similarity between them is as follows:

\[H^{S_{i}}_{a}=\frac{1}{N_{S_{i}}}\sum_{j=1}^{N_{S_{i}}}H^{ij}_{a}, \tag{5}\]

\[V^{I_{i}}_{o}=\frac{1}{N_{I_{i}}}\sum_{j=1}^{N_{I_{i}}}V^{ij}_{o}, \tag{6}\]

\[\text{sim}(H^{S_{i}}_{a},V^{I_{i}}_{o})=\text{cos}(H^{S_{i}}_{a},V^{I_{i}}_{o}), \tag{7}\]

Finally, considering that the lower the similarity between the aspect terms and the visual objects, the more likely the image is noise, we define a fine-grained (aspect-level) noise metric as follows:

\[d_{f}=1.0-\frac{\text{sim}(H^{S_{i}}_{a},V^{I_{i}}_{o})}{\max_{(H^{S_{k}}_{a}, V^{I_{k}}_{o})\in\mathcal{D}}\text{sim}(H^{S_{k}}_{a},V^{I_{k}}_{o})}, \tag{8}\]

where \(d_{f}\) is normalized to [0.0, 1.0]. Similarly, a lower score indicates the pair (\(S_{i}\), \(I_{i}\)) is clean data.

### Denoising Curriculums

In this section, we first introduce the _single denoising curriculum_ for each noise metric. Then we extend it into the _multiple denoising curriculum_.

Figure 2: Illustration of the Single Denoising Curriculum for coarse-grained noise metric \(d_{c}\). At each step, \(p(t)\) is computed based on the current step \(t\), and a batch of data is uniformly sampled from the training instances whose coarse-grained noise metric \(d_{c}\) is lower than \(p(t)\) (blue region in the example).

 



#### 3.3.1 Single Denoising Curriculum

In training, we first sort each instance in the training dataset \({\cal D}\) based on a single coarse-grained noise metric \(d_{c}\) or fine-grained noise metric \(d_{f}\). Then inspired by Platanios et al. (2019), we define a simple denoising function \(p(t)\), which controls the pace of learning from clean data to noise instances. The specific function form is as follows:

\[p(t)=\begin{cases}\sqrt{t\frac{1-p(0)^{2}}{T}+p(0)^{2}}&\text{if $t\leq T$},\\ 1.0&\text{otherwise}.\end{cases} \tag{9}\]

where \(p(0)\) is a predefined initial value and is usually set to 0.01, \(T\) is the duration of the denoising curriculum. At time step \(t\), \(p(t)\) means the upper limit of noise metric, and the model is only allowed to use the training instances whose noise metric \(d_{c}\) or \(d_{f}\) is lower than \(p(t)\). In this way, the model no longer presents training data in completely random order during training, but trains more time on clean data and less time on noisy data, which reveals the denoising efficacy of CL on noisy data Wang et al. (2022). After \(p(t)\) becomes 1.0, the single denoising curriculum is completed and the model can then freely access the entire dataset. In Figure 2, we give an illustration of the Single Denoising Curriculum for coarse-grained noise metric.

#### 3.3.2 Multiple Denoising Curriculum

As mentioned above, each single denoising curriculum uses only one noise metric for denoising. Considering that the two noise metrics measure the noise level of the image from different granularities, a natural idea is to combine them together to produce better denoising effects. To this end, we present an extension to the single denoising curriculum: multiple denoising curriculum.

Intuitively, there are a few readily conceivable approaches, including directly merging two noise metrics into one (i.e., \(d_{c}+d_{f}\)), as well as randomly or sequentially selecting curricula to train the model (i.e., \(d_{c}\to d_{f}\to d_{c}\to d_{f}\)). However, a more impactful strategy lies in dynamically selecting the most appropriate denoising curricula at each training step. To achieve this goal, we use the ratio of two consecutive performances on a validation set to measure the effectiveness of each denoising curriculum as follows:

\[s_{\theta} =\frac{\chi^{\theta}}{\chi^{\theta_{prev}}}, \tag{10}\] \[\chi^{\theta} =\frac{2*Pre*Rec}{Pre+Rec}. \tag{11}\]

where \(\chi^{\theta}\) is the F1-score computed at the current validation turn, \(\chi^{\theta_{prev}}\) is computed at the previous validation turn, \(Pre\) and \(Rec\) denote Precision and Recall. A higher value of \(s_{\theta}\) indicates greater progress achieved by the current denoising curriculum, which should lead to a larger sampling probability to achieve better denoising. Based on this, at each training step, we can dynamically select the most suitable denoising curriculum by comparing the values of \(s_{\theta}\) between the two denoising curriculums. The details of multiple denoising curriculum are presented in Algorithm 1 and Figure 1.

```
Input:The training set: \({\cal D}\); Coarse-grained Noise Metric: \(d_{c}\); Fine-grained Noise Metric: \(d_{f}\). Output:A model with multiple denoising curriculum.
1 Compute two noise metrics \(d_{c}\) and \(d_{f}\) for each training sample in \({\cal D}\); Sort \({\cal D}\) based on two noise metrics, resulting in \({\cal D}_{c}\) and \({\cal D}_{f}\);// Initialize\(s_{\theta}^{c}\), \(s_{\theta}^{f}\)for\({\cal D}_{c}\)and\({\cal D}_{f}\), respectively;
2for\({\cal D}_{i}\)in\({\cal D}_{c},{\cal D}_{f}\)do
3 Initialize the \(p_{i}(0)\);
4 Initialize the training step \(t_{i}=0\);
5while\(t_{i}<=1\)do
6 Uniformly sample one batch \(B_{i}(t_{i})\) from \({\cal D}_{i}\), such that \(d_{i}\leq p_{i}(t_{i})\);
7 Training the model with \(B_{i}(t_{i})\);
8\(t_{i}\gets t_{i}+1\);
9 end while
10Compute the ratio \(s_{\theta}^{b}\) of two consecutive performances on a validation set;
11 end if
12//Dynamicallyselectthemostsuitabledenoisingcurriculum;
13while not converged do
14\(j=\underset{i\in\left\{c:f\right\}}{\operatorname{arg\,max}}(s_{\theta}^{b})\);//\(j\)denotes\(c\)or\(f\)
15\(t_{j}\leftarrow t_{j}+1\);
16 Uniformly sample one batch \(B_{j}(t_{j})\) from \({\cal D}_{j}\), such that \(d_{j}\leq p_{j}(t_{j})\);
17 Training the model with \(B_{j}(t_{j})\);
18 Compute the ratio \(s_{\theta}^{b}\) of two consecutive performances on a validation set;
19 Update \(s_{\theta}^{b}\);
20 end while
```

**Algorithm 1**Multiple Denoising Curriculum.

## 4 Experiments

### Experimental Settings

Datasets.To evaluate the effect of the M2DF Framework, we carry out experiments on two public multimodal datasets TWITTER-15 and

TWITTER-17 from Yu and Jiang (2019), which include user tweets posted during 2014-2015 and 2016- 2017, respectively. General information for both datasets is presented in Table 2.

Implementation Details.During training, we train each model for a fixed 50 epochs, and then select the model with the best F1 score on the validation set. Finally, we evaluate its performance on the test set. We implement all the models with the PyTorch2 framework, and run experiments on an RTX3090 GPU. More details about the hyper-parameters can be found in Appendix A.

Footnote 2: [https://pytorch.org/](https://pytorch.org/)

Evaluation Metrics and Significance Test.As with the previous methods Ju et al. (2021); Yu and Jiang (2019), for MATE and JMASA tasks, we adopt Precision (Pre), Recall (Rec) and Micro-F1 (F1) as the evaluation metrics. For the MASC task, we use Accuracy (Acc) and Macro-F1 as evaluation metrics. Finally, we report the average performance and standard deviation over 5 runs with random initialization. Besides, the paired \(t\)-test is conducted to test the significance of different methods.

### Compared Methods

In this section, we introduce some representative baselines for each sub-task of MABSA: (1) **Baselines for Joint Multimodal Aspect-Sentiment Analysis (JMASA):** UMT-collapse Yu et al. (2020), OSCGA-collapse Wu et al. (2020), RPBERT Sun et al. (2021), RDS Xu et al. (2022), JML Ju et al. (2021), VLP-MABSA Ling et al. (2022); (2) **Baselines for Multimodal Aspect Term Extraction (MATE):** UMT Yu et al. (2020), OSCGA Wu et al. (2020), JML Ju et al. (2021), VLP-MABSA Ling et al. (2022); (3) **Baselines for Multimodal Aspect Sentiment Classification (MASC):** TomBERT Yu and Jiang (2019), CapTrBERT Khan and Fu (2021), JML Ju et al. (2021), ITMYu et al. (2022), FITE Yang et al. (2022), VLP-MABSA Ling et al. (2022).

To verify the generalization of our framework, we apply M2DF to all baselines of three sub-tasks. It should be noted that RpBERT, RDS, JMT, and ITM use a cross-modal relation detection module to filter out noise images. In order to better compare their denoising effects with M2DF, we replace the cross-modal relation detection module in RpBERT, RDS, JMT, and ITM with our M2DF framework.

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|c|c|c|c|} \hline  & \multicolumn{8}{c|}{**TWITTER-15**} & \multicolumn{8}{c|}{**TWITTER-17**} \\ \cline{2-13}  & **Pos** & **Neg** & **Neu** & **Total** & **AT** & **Words** & **AL** & **Pos** & **Neu** & **Neg** & **Total** & **AT** & **Words** & **AL** \\ \hline Train & 928 & 368 & 1883 & 3179 & 1.348 & 9023 & 16.72 & 1508 & 416 & 1638 & 3562 & 1.410 & 6027 & 16.21 \\ Dev & 303 & 149 & 670 & 1122 & 1.336 & 4238 & 16.74 & 515 & 144 & 517 & 1176 & 1.439 & 2922 & 16.37 \\ Test & 317 & 113 & 607 & 1037 & 1.354 & 3919 & 17.05 & 493 & 168 & 573 & 1234 & 1.450 & 3013 & 16.38 \\ \hline \end{tabular}
\end{table}
Table 2: The basic statistics of our two multimodal Twitter datasets. Pos: Positive, Neg: Negative, Neu: Neutral, AT: Avg. Targets, AL: Avg. Length

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|c|} \hline \multirow{2}{*}{**Methods**} & \multicolumn{4}{c|}{**TWITTER-15**} & \multicolumn{4}{c|}{**TWITTER-17**} \\ \cline{2-9}  & **Pre** & **Rec** & **F1** & **Pre** & **Rec** & **F1** \\ \hline UMT-collapse Yu et al. (2020) & 60.4 & 61.6 & 61.0 & 60.0 & 61.7 & 60.8 \\ + M2DF & 61.1\(\pm 0.40\) & 63.4\(\pm 0.57\) & 62.2\(\pm 0.10\) & 60.9\(\pm 0.28\) & 62.0\(\pm 0.52\) & 61.4\(\pm 0.13\) \\ \hline OSCGA-collapse Wu et al. (2020) & 63.1 & 63.7 & 63.2 & 63.5 & 63.5 & 63.5 \\ + M2DF & 64.4\(\pm 0.37\) & 64.6\(\pm 0.45\) & 64.5\(\pm 0.13\) & 64.1\(\pm 0.11\) & 63.9\(\pm 0.16\) & 64.0\(\pm 0.12\) \\ \hline RpBERT Sun et al. (2021) & 49.3 & 46.9 & 48.0 & 57.0 & 55.4 & 56.2 \\ + M2DF & 49.3\(\pm 0.20\) & 49.0\(\pm 0.25\) & 49.2\(\pm 0.15\) & 56.9\(\pm 0.34\) & 56.5\(\pm 0.38\) & 56.7\(\pm 0.22\) \\ \hline RDS\({}^{\blacktriangle}\)Xu et al. (2022) & 60.8 & 61.7 & 61.2 & 61.8 & 62.9 & 62.3 \\ + M2DF & 61.2\(\pm 0.12\) & 63.0\(\pm 0.35\) & 62.1\(\pm 0.15\) & 62.4\(\pm 0.16\) & 63.6\(\pm 0.12\) & 63.0\(\pm 0.08\) \\ \hline JML\({}^{\blacktriangle}\)Ju et al. (2021) & 64.8 & 63.6 & 64.0 & 65.6 & 66.1 & 65.9 \\ + M2DF & 64.9\(\pm 0.36\) & 65.3\(\pm 0.16\) & 65.1\(\pm 0.25\) & 67.7\(\pm 0.30\) & 67.0\(\pm 0.08\) & 67.3\(\pm 0.16\) \\ \hline VLP-MABSA Ling et al. (2022) & 64.1 & **68.6** & 66.3 & 65.8 & 67.9 & 66.9 \\ + M2DF & **67.0\(\pm 0.20\)** & 68.3\(\pm 0.26\) & **67.6\(\pm 0.18\)** & **67.9\(\pm 0.10\)** & **68.8\(\pm 0.37\)** & **68.3\({}^{\dagger}\pm 0.18\)** \\ \hline \end{tabular}
\end{table}
Table 3: Test results on the TWITTER–15 and TWITTER–17 datasets for JMASA task (%). For the baseline models, the results with \({}^{\blacktriangledown}\) are obtained by running the code released by the authors, and the other results without symbols are retrieved from Ju et al. (2021) and Ling et al. (2022). Best results are in bold. The marker \({}^{\dagger}\) refers to significant test p-value \(<\) 0.05 when comparing with other multimodal methods.

 

For other baselines, we directly integrate M2DF into them to obtain the final model.

## 5 Results and Analysis

### Main Results

In this section, we analyze the results of different methods on three sub-tasks of MABSA.

Results of JMASA.The main experiment results of JMASA are shown in Table 3. Based on these results, we can make a couple of observations: (1) _VLP-MABSA_ performs better than other multimodal baselines. A possible reason is that it develops a series of useful pre-training tasks; (2) Compared to the base model _UMT-collapse_ and _OSCGA-collapse_, _UMT-collapse + M2DF_ and _OSCGA-collapse + M2DF_ achieve better results on both datasets. This outcome reveals the effective denoising effect of our framework; (3) It is worth mentioning that _RpBERT + M2DF_ and _RDS + M2DF_ obtain good improvements in Micro-F1 over _RpBERT_ and _ROS_. Similarly, _JMT + M2DF_ achieves a boost of 1.1% and 1.4% in Micro-F1 compared to _JMT_. These observations indicate that the denoising effect of M2DF is better than that of the cross-modal relation detection module; (4) The performance boost of _VLP-MABSA + M2DF_ over _VLP-MABSA_ by 1.3% and 1.4%. These results affirm the robustness of the M2DF, showcasing its ability to consistently generate substantial enhancements even when applied to the current state-of-the-art model.

Results of MATE and MASC.Table 4 and Table 5 show the main experiment results of MATE and MASC. As we can see, incorporating M2DF into the baselines of MATE and MATC tasks can obtain competitive performance. To be specific, for the MATE task, _VLP-MABSA + M2DF_ obtain 1.2% and 1.1% improvements in terms of the F1-score. For the MAC task, _VLP-MABSA + M2DF_ can improve the F1-score by up to 1.9% and 1.6%. These results indicate that M2DF is applicable to three subtasks simultaneously, showcasing its remarkable capacity for generalization.

### Ablation Study

Without loss of generality, we choose _VLP-MABSA + M2DF_ model for the ablation study to investigate the effects of different modules in M2DF.

Effects of the Noise Metrics.As we mentioned in Section 3.3.1, we develop a single denoising curriculum for each noise metric. Here, we discuss the contribution of each noise metric in our framework. From Table 6, we can observe the fol

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|} \hline \multirow{2}{*}{**Methods**} & \multicolumn{3}{c|}{**TWITTER-15**} & \multicolumn{3}{c|}{**TWITTER-17**} \\ \cline{2-8}  & **Pre** & **Rec** & **F1** & **Pre** & **Rec** & **F1** \\ \hline UMT [20] & 77.8 & 81.7 & 79.7 & 86.7 & 86.8 & 86.7 \\   + M2DF & 79.1\(\pm\)0.14 & 81.5\(\pm\)0.33 & 80.3\(\pm\)0.12 & 87.4\(\pm\)0.18 & 87.5\(\pm\)0.22 & 87.5\(\pm\)0.15 \\ \hline OSCGA [21] & 81.7 & 82.1 & 81.9 & 90.2 & 90.7 & 90.4 \\   + M2DF & 82.0\(\pm\)0.10 & 82.8\(\pm\)0.31 & 82.4\(\pm\)0.13 & 90.3\(\pm\)0.15 & 91.5\(\pm\)0.17 & 90.9\(\pm\)0.07 \\ \hline JML\({}^{\blacktriangle}\)[21] & 82.9 & 81.2 & 82.0 & 90.2 & 90.9 & 90.5 \\   + M2DF & 84.0\(\pm\)0.26 & 82.3\(\pm\)0.12 & 83.1\(\pm\)0.14 & 91.1\(\pm\)0.11 & 90.9\(\pm\)0.18 & 91.0\(\pm\)0.12 \\ \hline VLP-MABSA\({}^{\blacktriangle}\)[19] & 82.2 & **88.2** & 85.1 & 89.9 & 92.5 & 91.3 \\   + M2DF & **85.2\(\pm\)**0.24 & 87.4\(\pm\)0.20 & **86.3\({}^{\dagger}\pm\)**0.15 & **91.5\(\pm\)**0.25 & **93.2\(\pm\)**0.23 & **92.4\({}^{\dagger}\pm\)**0.14 \\ \hline \end{tabular}
\end{table}
Table 4: Test results on the TWITTER-15 and TWITTER-17 datasets for MATE task (%). For the baseline models, the results with \({}^{\blacktriangle}\) are obtained by running the code released by the authors, and the other results without symbols are retrieved from [19]. Best results are in bold.

\begin{table}
\begin{tabular}{|l|c|c|c|c|} \hline \multirow{2}{*}{**Methods**} & \multicolumn{2}{c|}{**TWITTER-15**} & \multicolumn{2}{c|}{**TWITTER-17**} \\ \cline{2-5}  & **Acc** & **F1** & **Acc** & **F1** \\ \hline TomBERT & 77.2 & 71.8 & 70.5 & 68.0 \\   + M2DF & 77.9\(\pm\)0.11 & 73.2\(\pm\)0.11 & 71.0\(\pm\)0.14 & 68.7\(\pm\)0.20 \\ \hline CapTIBERT & 78.0 & 73.2 & 72.3 & 70.2 \\   + M2DF & 78.4\(\pm\)0.12 & 74.0\(\pm\)0.08 & 73.0\(\pm\)0.08 & 71.3\(\pm\)0.07 \\ \hline PITE & 78.5 & 73.9 & 70.9 & 68.7 \\   + M2DF & 78.9\(\pm\)0.07 & 74.2\(\pm\)0.08 & 71.5\(\pm\)0.11 & 69.4\(\pm\)0.12 \\ \hline ITM & 78.3 & 74.2 & 72.6 & 72.0 \\   + M2DF & **78.9\(\pm\)**0.05 & **75.9\(\pm\)**0.07 & 73.2\(\pm\)0.10 & **73.0\({}^{\ddagger}\pm\)**0.08 \\ \hline JML\({}^{\blacktriangle}\)[21] & 78.1 & - & 72.7 & - \\   + M2DF & 78.8\(\pm\)0.15 & - & 74.0\(\pm\)0.12 & - \\ \hline VLP-MABSA\({}^{\blacklozenge}\) & 77.2 & 72.9 & 73.2 & 71.4 \\   + M2DF & **78.9\(\pm\)**0.15 & 74.8\(\pm\)0.24 & **74.3\({}^{\ddagger}\pm\)**0.15 & **73.0\({}^{\ddagger}\pm\)**0.16 \\ \hline \end{tabular}
\end{table}
Table 5: Test results on two Twitter datasets for MASC task(%). For the baselines, the results with \({}^{\blacktriangle}\) are obtained by running the code released by the authors, and the other results are retrieved from [19], [20] and [21]. For a fair comparison, we follow the setting of [21] and [20], i.e., JML only evaluates the aspects correctly predicted on the MATE task while the other methods evaluate on all the golden aspects.

 

lowing: (1) Settings (a-b) reveal that every noise metric can boost the performance of base model _VLP-MABSA_, which validates the rationality of our designed noise metrics; (2) The fine-grained noise metric achieves better performance than the coarse-grained noise metric in most tasks, which indicates that fine-grained noise metric is more accurate in measuring whether noisy images are included in each training instance; (3) When incorporate two noise metric together, the performance is further improved (see setting c), which demonstrates that two noise metrics improve the performance of the MABSA task from different perspectives.

Effects of the Denoising CurriculumsAs we mentioned in Section 3.3.2, three easily thought of strategies to implement the multiple denoising curriculum are Merge, Randomly, and Sequentially. Table 6 (settings d-f) shows the results of the three strategies. As we can see, (1) All three strategies achieve stable improvements on the MABSA tasks, which demonstrates the effectiveness and robustness of our M2DF framework; (2) The results of the Merge strategy are inferior to Randomly and Sequentially. A possible reason is that directly merging two noise metrics is unable to give full play to their respective advantages; (3) All of them perform worse than our final approach (setting c), which verifies the superiority of the dynamic learning strategy at each training step.

Performance on Different SubsetsTo gain a deeper insight into the role of M2DF, we took an additional step by partitioning the TWITTER-17 test set into three subsets of equal size. This division was based on the values of a coarse-grained noise metric \(d_{c}\), spanning from low to high. By employing this approach, we aimed to examine the impact of M2DF across various levels of noise present in the dataset. The results are shown in Table 7. We can observe that compared with _VLP-MABAS_, _VLP-MABSA + M2DF_ achieves more performance improvement on the subset with the higher noise level. This suggests that M2DF can indeed mitigate the negative impact of noisy images.

### Training Time

Curriculum Learning can effectively steer the learning process away from poor local optima, making it converge faster Bengio et al. (2009); Platanios et al. (2019). As a result, the training time for the denoising phase is reduced compared to regular training. In **Appendix C**, we display the training times for VLP-MABSA and VLP-MABSA+M2DF on the JMASA task.

### Case Study

As we mentioned in section 5.2, M2DF performs well when the multimodal instance contains noisy images unrelated to the text, here we conduct the case study to better understand the advantages of

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|} \hline \multirow{2}{*}{**Settings**} & **Coarse-grained** & **Fine-grained** & **Denoising** & \multicolumn{4}{c|}{**TWITTER-15**} & \multicolumn{4}{c|}{**TWITTER-17**} \\  & **Noise Metric** & **Noise Metric** & **Curriculum** & **MATE** & **MASC** & **JMASA** & **MTE** & **MSC** & **JMASA** \\ \hline VLP-MABSA & ✗ & ✗ & ✗ & 85.1 & 72.9 & 66.3 & 91.3 & 71.4 & 66.9 \\ \hline (a) & ✓ & ✗ & Single & 85.8 & 73.6 & 66.8 & 91.6 & 72.1 & 67.4 \\ (b) & ✗ & ✔ & Single & 85.9 & 73.9 & 67.0 & 91.6 & 72.3 & 67.5 \\ (c) & ✓ & ✔ & Multiple & 86.3 & 74.8 & 67.6 & 92.4 & 73.0 & 68.3 \\ \hline (d) & ✓ & ✔ & Merge & 85.8 & 73.8 & 66.9 & 91.7 & 72.1 & 67.9 \\ (e) & ✓ & ✔ & Randomly & 86.1 & 74.2 & 67.3 & 91.9 & 72.5 & 68.0 \\ (f) & ✓ & ✔ & Sequentially & 86.2 & 74.5 & 67.3 & 92.1 & 72.6 & 68.2 \\ \hline \end{tabular}
\end{table}
Table 6: Ablation study of our denoising framework(%). We use the current state-of-the-art model VLP-MABSA Ling et al. (2022) as the base model to conduct the analysis. The setting (c) denotes our full proposed approach. We evaluate three tasks JMASA, MATE, and MASC in terms of F1.

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|} \hline \multirow{2}{*}{**Methods**} & \multicolumn{3}{c|}{**Level-1**} & \multicolumn{3}{c|}{**Level-2**} & \multicolumn{3}{c|}{**Level-3**} \\ \cline{2-11}  & **MATE** & **MASC** & **JMASA** & **MATE** & **MASC** & **JMASA** & **MATE** & **MASC** & **JMASA** \\ \hline VLP-MABSA & 92.7 & 73.4 & 67.2 & 91.6 & 73.0 & 69.8 & 89.6 & 68.0 & 63.4 \\ VLP-MABSA + M2DF & 92.9 & 73.6 & 67.8 & 92.1 & 73.8 & 71.2 & 91.6 & 71.3 & 65.6 \\ \hline \(\Delta\) & +0.2 & +0.2 & +0.6 & +0.5 & +0.8 & +1.4 & +2.0 & +3.3 & +2.2 \\ \hline \end{tabular}
\end{table}
Table 7: Model performance on the divided TWITTER-17 test sets with different noise levels (%). We evaluate three tasks JMASA, MATE, and MASC in terms of F1. \(\Delta\) represents the difference between the performance of VLP-MABSA and VLP-MABSA + M2DF at different noise level.

 

M2DF. Figure 3 presents the predict results of three noise examples on the JMASA task by _JML + M2DF_ and _VLP-MBASA + M2DF_, as well as two representative baselines _JML_ and _VLP-MABSA_. It is clear that both _JML_ and _VLP-MABSA_ give the incorrect prediction when faced with noisy images. The reason behind this may be that: (1) although _JML_ develops a cross-modal relation detection module to filter out noisy images, this module relies on thresholds, so using it to filter noisy images is not reliable enough; (2) _VLP-MABSA_ treats the image equally and ignores the negative effects of noisy images. In contrast, we observe that _JML + M2DF_ and _VLP-MABSA + M2DF_ can obtain all correct aspect terms and aspect-dependent sentiment on three noise samples, suggesting that our framework M2DF can achieve better denoising.

## 6 Conclusion

In this paper, we propose a novel Multi-grained Multi-curriculum Denoising Framework (M2DF) for the MABSA task. Specifically, we first define a coarse-grained noise metric and a fine-grained noise metric to measure the degree of noisy images contained in each training instance. Then, in order to reduce the negative impact of noisy images, we design a single denoising curriculum and a multiple denoising curriculum. Results from numerous experiments indicate that our denoising framework achieves better performance than other state-of-the-art methods. Further analysis also validates the superiority of our denoising framework.

### Limitations

The current study has two limitations that warrant further attention. First, while we have developed two noise metrics to evaluate the degree of noisy images contained in the dataset, there may be other methods to achieve this goal, which requires us to do further research and exploration. Second, we have not examined other CL training schedules, such as self-paced learning Kumar et al. (2010), which may be worth considering in the future. We view our study as a starting point for future research, with the goal of propelling the field forward and leading to even better model performance.

## Acknowledgements

We would like to thank the anonymous reviewers for their insightful comments. Fei Zhao would like to thank Siyu Long for his constructive suggestions. This work is supported by the National Natural Science Foundation of China (No. 62206126, 61936012 and 61976114).

Figure 3: Three examples of the predictions by JML, VLP-MABSA, JML + M2DF, and VLP-MABSA + M2DF. POS, NEU, and NEG are short for Positive, Neutral, and Negative, respectively.

 



## References

* J. Bengio, J. Louradour, R. Collobert, and J. Weston (2009)Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009, Vol. 382 of ACM International Conference Proceeding Series, pp. 41-48. External Links: ISBN 978-3-540-3871-4 Cited by: SS1.
* G. Chen, Y. Tian, and Y. Song (2020)Joint aspect extraction and sentiment analysis with directional graph convolutional networks. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online), pp. 272-279. External Links: Link, Document Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL, pp. 4171-4186. Cited by: SS1.
* T. Gong, Q. Zhao, D. Meng, and Z. Xu (2016)Why curriculum learning & self-paced learning work in big/noisy data: a theoretical perspective. Big Data & Information Analytics1 (1), pp. 111. Cited by: SS1.
* K. He, G. Gkioxari, P. Dollar, and R. B. Girshick (2020)Mask r-cnn. IEEE Trans. Pattern Anal. Mach. Intell.42 (2), pp. 386-397. External Links: ISSN 1050-321X Cited by: SS1.
* K. He, X. Zhang, S. Ren, and J. Sun (2016)Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. External Links: Link, Document Cited by: SS1.
* M. Hu, Y. Peng, Z. Huang, D. Li, and Y. Lv (2019)Open-domain targeted sentiment analysis via span-based extraction and classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 537-546. External Links: Link, Document Cited by: SS1.
* M. Hu and B. Liu (2004)Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, Washington, USA, August 22-25, 2004, pp. 168-177. External Links: Link, Document Cited by: SS1.
* X. Ju, D. Zhang, R. Xiao, J. Li, S. Li, M. Zhang, and G. Zhou (2021)Joint multi-modal aspect-sentiment analysis with auxiliary cross-modal relation detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 4395-4405. External Links: Link, Document Cited by: SS1.
* 24, 2021, pp. 3034-3042. External Links: Link, Document Cited by: SS1.
* M. Kumar, B. Packer, and D. Koller (2010)Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada, pp. 1189-1197. External Links: Link, Document Cited by: SS1.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020)BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7871-7880. External Links: Link, Document Cited by: SS1.
* Y. Ling, J. Yu, and R. Xia (2022)Vision-language pre-training for multimodal aspect-based sentiment analysis. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 2149-2159. External Links: Link, Document Cited by: SS1.
* B. Liu (2012)Sentiment Analysis and Opinion Mining. Synthesis Lectures on Human Language Technologies Morgan & Claypool Publishers. External Links: Link Cited by: SS1.
* X. Liu, H. Lai, D. F. Wong, and L. S. Chao (2020)Norm-based curriculum learning for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 427-436. External Links: Link, Document Cited by: SS1.
* J. Lu, D. Batra, D. Parikh, and S. Lee (2019)Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13-23. External Links: Link, Document Cited by: SS1.
* J. Lu and J. Zhang (2021)Exploiting curriculum learning in unsupervised neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pp. 924-934. External Links: Link, Document Cited by: SS1.
* S. Moon, L. Neves, and V. Carvalho (2018)Multimodal named entity recognition for short social media posts. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NASALL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume

(Long Papers), pp. 852-860. Cited by: SS1.
* E. Platanios, O. Stretcu, G. Neubig, B. Poczos, and T. M. Mitchell (2019)Competence-based curriculum learning for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Vol. , pp. 1162-1172. External Links: Document, Link Cited by: SS3.4.
* M. Pontiki, D. Galanis, J. Pavlopoulos, H. Papageorgiou, I. Androutsopoulos, and S. Manandhar (2014)Semeval-2014 task 4: aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation, Cited by: SS1.
* A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021)Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, Y. Bengio et al. (Eds.), Vol. , pp. 8748-8763. External Links: ISSN 1536-860X, Link, Document Cited by: SS1.
* S. Ren, K. He, R. B. Girshick, and J. Sun (2015)Faster r-cnn: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 91-99. External Links: ISBN 978-3-319-1939-19 

Jianfei Yu, Jieming Wang, Rui Xia, and Junjie Li. 2022. Targeted multimodal sentiment classification based on coarse-to-fine grained image-target matching. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_, pages 4482-4488. International Joint Conferences on Artificial Intelligence Organization. Main Track.
* Zhang et al. (2021) Dong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021. Multimodal graph fusion for named entity recognition with targeted visual guidance. In _AAAI 2021, Virtual Event, February 2-9, 2021_, pages 14347-14355.
* Zhao et al. (2022a) Fei Zhao, Chunhui Li, Zhen Wu, Shangyu Xing, and Xinyu Dai. 2022a. Learning from different text-image pairs: A relation-enhanced graph convolutional network for multimodal ner. In _Proceedings of the 30th ACM International Conference on Multimedia_, MM '22, page 3983-3992, New York, NY, USA. Association for Computing Machinery.
* Zhao et al. (2022b) Fei Zhao, Zhen Wu, Siyu Long, Xinyu Dai, Shujian Huang, and Jiajun Chen. 2022b. Learning from adjective-noun pairs: A knowledge-enhanced framework for target-oriented multimodal sentiment classification. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 6784-6794, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Zheng et al. (2021) Changmeng Zheng, Zhiwei Wu, Tao Wang, Yi Cai, and Qing Li. 2021. Object-aware multimodal named entity recognition in social media posts with adversarial learning. _IEEE Trans. Multim._, 23:2520-2532.

## Appendix A Implementation Details

As we mentioned in section 4.2, we chose some representative models as the foundations of our work for three sub-tasks. Therefore, when applying our framework, we keep the parameters the same as the baseline methods in Table 8. We implement all the models with the PyTorch framework, and run experiments on an RTX3090 GPU.

## B Additional Analysis

The Image Information is UsefulExcept for multimodal baselines, we also compare the M2DF with pure textual ABSA models, i.e., SPAN Hu et al. (2019), D-GCN Chen et al. (2020) and BART Lewis et al. (2020), the results of the JMASA task are shown in Table 9. For text-based methods, it is clear that BART consistently outperforms all the baselines, we attribute this to the fact that the encoder-decoder structure of BART can grasp the overall meaning of the whole sentence in advance, which is important for predicting label sequences. VLP-MABSA is built upon the foundation of BART, and the multimodal methods VLP-MABSA and VLP-MABSA + M2DF both outperform BART, which indicates that image information is useful for the MABSA task. This aligns with the conclusions of previous methods.

Effect of Different Multimodal Encoder and Image EncoderWe utilize CLIP for multimodal feature extraction, due to its training on a vast collection of text-image pairs. This allows it to excel in capturing multimodal semantics better. Besides, we also try using ViLBERT Lu et al. (2019) to extract multimodal features. For object detection, we try using Faster-RCNN to replace Mask-RCNN. The results of these two methods on the JMASA task are shown in Table 10. We can observe that the F1-score of VLP-MABSA + M2DF (ViLBERT) is lower than that of VLP-MABSA + M2DF (Ours). This is because CLIP benefits from a larger-scale and more diverse training dataset, which enables the model to capture multimodal features more accurately. Thus, CLIP is a better version to extract multimodal features. Moreover, the F1-score of VLP-MABSA + M2DF (Faster-RCNN) is slightly

Figure 4: Training times for VLP-MABSA and VLP-MABSA+M2DF on the JMASA task.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Methods** & **Backbone** & **Learning Rate** \\ \hline UMT-collapse & BERT+ResNet & 5e-5 \\ UMT & BERT+ResNet & 5e-5 \\ OSCGA-collapse & Glove+Mask RCNN & 0.008 \\ OSCGA & Glove+Mask RCNN & 0.008 \\ RpBERT & BERT+ResNet & 1e-6 \\ TomBERT & BERT+ResNet & 5e-5 \\ CapTriBERT & BERT+ResNet & 5e-5 \\ FTE & BERT+ResNet & 5e-5 \\ JML & BERT+ResNet & 2e-5 \\ VLP-MABSA & BART+Faster R-CNN & 5e-5 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Hyperparameters of baseline methods.

 

lower than that of VLP-MABSA + M2DF (Ours), which indicates that Mask-RCNN is a better version for object detection in the MABSA task. Certainly, Faster-RCNN is also a good choice.

## Appendix C Training Time

Figure 4 lists the training times for VLP-MABSA and VLP-MABSA+M2DF on the JMASA task.

## Appendix D Error Analysis

We randomly sample 100 error cases of _VLP-MABSA + M2DF_ in the JMASA task, and then divide them into three error categories. Figure 5 shows the proportions and some representative examples for each category. The top category is annotation bias. As shown in Figure 5(a), the sentiment of aspect term "Orange Carpet" is annotated as "NEU" (Neutral), but it actually expresses the sentiment of "POS" (Positive). This type of error presents a significant challenge for our model to provide accurate predictions. The second category is lacking background knowledge. In Figure 5(b), the aspect term "Orange is the New Black" refers to a well-known comedy, the model typically can only extract part of the aspect term without the help of background knowledge. The third category is insufficient information. As shown in Figure 5(c), both the sentence and the image content are quite simple, which is unable to provide sufficient information for our model to accurately identify aspect terms and their corresponding sentiment polarities.

There is a need for the development of more advanced natural language processing techniques to address the above problems.

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|} \hline \multirow{2}{*}{**Methods**} & \multicolumn{3}{c|}{**TWITTER-15**} & \multicolumn{3}{c|}{**TWITTER-17**} \\ \cline{2-8}  & **Pre** & **Rec** & **F1** & **Pre** & **Rec** & **F1** \\ \hline VLP-MABSA + M2DF (Ours) & 67.0\(\pm 0.20\) & 68.3\(\pm 0.26\) & 67.6\(\pm 0.18\) & 67.9\(\pm 0.10\) & 68.8\(\pm 0.37\) & 68.3\(\pm 0.18\) \\ VLP-MABSA + M2DF (ViLBERT) & 66.6\(\pm 0.09\) & 67.9\(\pm 0.25\) & 67.2\(\pm 0.11\) & 67.3\(\pm 0.29\) & 68.2\(\pm 0.13\) & 67.7\(\pm 0.17\) \\ VLP-MABSA + M2DF (Faster-RCNN) & 66.9\(\pm 0.40\) & 68.1\(\pm 0.18\) & 67.5\(\pm 0.11\) & 67.7\(\pm 0.18\) & 68.6\(\pm 0.18\) & 68.1\(\pm 0.09\) \\ \hline \end{tabular}
\end{table}
Table 10: Effect of different image encoders (%).

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|} \hline \multirow{2}{*}{**Methods**} & \multicolumn{3}{c|}{**TWITTER-15**} & \multicolumn{3}{c|}{**TWITTER-17**} \\ \cline{2-8}  & **Pre** & **Rec** & **F1** & **Pre** & **Rec** & **F1** \\ \hline SPAN Hu et al. (2019) & 53.7 & 53.9 & 53.8 & 59.6 & 61.7 & 60.6 \\ D-GCN Chen et al. (2020) & 58.3 & 58.8 & 58.5 & 64.2 & 64.1 & 64.1 \\ BART Lewis et al. (2020) & 62.9 & 65.0 & 63.9 & 65.2 & 65.6 & 65.4 \\ \hline VLP-MABSA Ling et al. (2022) & 64.1 & 68.6 & 66.3 & 65.8 & 67.9 & 66.9 \\ VLP-MABSA + M2DF & 67.0\(\pm 0.20\) & 68.3\(\pm 0.26\) & 67.6\(\pm 0.18\) & 67.9\(\pm 0.10\) & 68.8\(\pm 0.37\) & 68.3\(\pm 0.18\) \\ \hline \end{tabular}
\end{table}
Table 9: Test results on the TWITTER-15 and TWITTER-17 datasets for the JMASA task (%). The results of text-based methods (i.e., SPAN, D-GCN, and BART) are retrieved from Ling et al. (2022).

 

* [14] M. C. Bender, J. R. Friedman, and A. K. G. Hinton. The classification of deep neural networks with applications to image recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 257-260, June 2009.
* [15] D. P. Kingma and S. Ba. Adam: a method for stochastic optimization. _arXiv preprint arXiv:1312.8810_, 2013.
* [16] T. L. Bishop, A. Vedaldi, and Y. Bengio. Generative adversarial nets. In _Advances in Neural Information Processing Systems (NIPS)_, volume 1, pages 1-10, 2015.

[MISSING_PAGE_POST]