M2DF: Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysis
Authors: Fei Zhao
Chunhui Li
Zhen Wu
Yawen Ouyang
Jianbing Zhang
Xinyu Dai
Venue: EMNLP 2023 Main (2023)
Keywords: aspect-based sentiment analysis

Abstract: Multimodal Aspect-based Sentiment Analysis (MABSA) is a fine-grained Sentiment Analysis task, which has attracted growing research interests recently. Existing work mainly utilizes image information to improve the performance of MABSA task. However, most of the studies overestimate the importance of images since there are many noise images unrelated to the text in the dataset, which will have a negative impact on model learning. Although some work attempts to filter low-quality noise images by setting thresholds, relying on thresholds will inevitably filter out a lot of useful image information. Therefore, in this work, we focus on whether the negative impact of noisy images can be reduced without modifying the data. To achieve this goal, we borrow the idea of Curriculum Learning and propose a Multi-grained Multi-curriculum Denoising Framework (M2DF), which can achieve denoising by adjusting the order of training data. Extensive experimental results show that our framework consistently outperforms state-of-the-art work on three sub-tasks of MABSA.

M2DF: Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysis
Fei 'hao' Chunhui Li Zhen Wult Yawen Ouyang"
Jianbing Zhang' Xinyu Dai'
National Key Laboratory for Novel Software Technology, Nanjing University, China Institute for Al Industry Research (AIR), Tsinghua University ( (zhaof, lich) (achh/@mmil..m. .nju edu, en, ouyangyawen@air: tinghua. edu. cn (wuz, z zjb, daixinyu/ @nju edu. chu.cn. en
Abstract
S: [Lady Gagale) posative out and about in [NYC] pameuurrl ( (May, 2 nd
Multimodal Aspect-based Sentiment Analysis (MABSA) is fine-grained Sentiment Analy- sis task, which has attracted growing research interests recently. Existing work mainly uti- lizes image information to improve the perfor- mance of MABSA task. However, most of the studies overestimate the importance of images since there many noisy images unrelated to the text in the dataset, which will have negative impact on model learning. Although work attempts filter low-quality noisy images by setting thresholds. relying on thresh- olds will inevitably filter out lot useful image information. Therefore, in this work, we focus on whether the negative impact of noisy images can be reduced without filter- ing the data. To achieve this goal, we borrow the idea of Curriculum Learning and propose a Multi-grained Multi-curriculum Denoising Framework (M2DF), which can achieve de- noising by adjusting the order training data. Extensive experimental results show that framework consistently outperforms state-of- the-art work on three sub-tasks MABSA. Our code and datasets available https: Wggihhu..com/gaandcciikkn//////
I:
Sub-tasks
Input
Output
Multimodal Aspect Term Extraction (MATE)
S+I
aj.
Multimodal Oriente Sentiment Classification (MASC)
S S+I+
S+
Pa
Joint Multimodal Aspect-Sentiment Analysis (IMASA)
(0), PI). (0g, P2)
S+I
Table I:
The definitions MABSA subtasks. Aspect terms and their corresponding sentiment polarities are highlighted in the sentence S.
sentiment polarity of given aspect term in a sentence (Yu and Jiang, 2019), whereas the goal of Joint Multimodal Aspect-Sentimee Analysis (JMASA) is extract all aspect terms and their cor- responding sentiment polarities simultaneously (Ju et al., 2021). We illustrate the definitions of all the tasks with a specific example in Table
1 Introduction
Multimodal Aspect-based Sentiment Analysis (MABSA) is fine-grained Sentiment Analysis task (Liu, 2012; Pontiki et al., 2014), which has received extensive research attention in the past few years. MABSA has derived number of sub- tasks, which all revolve around predicting several sentiment elements, i.e., aspect term(a), sentiment polarity(p), or their combinations. For example, given a pair of sentence and image, Multimodal Aspect Term Extraction (MATE) aims to extract all the aspect terms mentioned in sentence (Wu al., 2020a), Multimodal Aspect- Oriented Senti- ment Classification (MASC) aims identify the
The assumption behind these sub-tasks that the image information can help the text content iden- tify the sentiment elements (Yu and Jiang, 2019; Ju et al., 2021), so how to make use of image in- formation lies at the heart of the MABSA task. To this end, a lot of efforts have been devoted, roughly divided into three categories: (1) transforming the image into global feature vector and utilizing tention mechanisms sift through visual content pertinent to the text (Moon et al., 2018; Xu et al., 2019); (2) partitioning the image into multiple vi-
Equal contributions. Corresponding author.

---

sual regions averagely, and subsequently establish- ing connections between lext sequences and visual regions (Yu and Jiang, 2019; Yu et al., 2020; Ju et al., 2021); (3) retaining exclusively the regions containing visual objects in the image and enabling dynamic interactions with the text sequences ( (Wu et al., b:20a,b; Zhang et al., 2021: Yu et al., 2022; Ling et al., 2022; Yang al., 2022).
rent state-of-the-art, Extensive experimental results show that our denoising framework achieves com- petitive performance on three sub-tasks.
Related Work
In this section, review the existing studies on Multimodal Aspect-based Sentiment Analysis (MABSA) and Curriculum Learning (CL).
Despite their promising results, most of the stud- ies overestimate the importance of images since there are many noisy images unrelated to the text in the dataset, which will have a negative impact on model learning. In the MABSA task, Sun et al. (2020, 2021), Ju et al. (2021), Xu et al. (2022) and Yu et al. (2022) developed the moosl relation detection module that can filter low-quality noisy images by setting thresholds. However, relying on thresholds will inevitably filter out lot of useful image information. Thus, in this work, we focus on another aspect: can we reduce the negative impact of noisy images without filtering the data?
Multimodal Aspect-based Sentiment Analysis. As an important sentiment analysis task, vari- ous neural networks have been proposed to deal with the three sub-tasks of MABSA, i.c., Multi- modal Aspect Term Extraction (MATE), Multi- modal Aspect-Oriented Sentiment Classification (MASC) and Joint Multimodal Aspect-Sentiment Analysis (JMASA). According to the different ways of utilizing image information, their work can be divided into three categories:
(1) Transforming the image into global fea- ture vector and utilizing well-designed attention mechanisms to sift through visual content pertinent to the text. Moon al. (2018) used an attention mechanism extract primary information from word embedding, character embedding. and global image vector. Xu et al. (2019) proposed a multi- interactive memory network to capture the multiple correlations in multimodal data.
To answer the above question, borrow the idea of Curriculum Learning (CL) (Bengio al.. 2009), because CL can achieve denoising by adjust- ing the order of training data (Wang et al., 2022). other words, CL no longer presents training data in a completely random order during train- ing, but encourages training more time on clean data and less time on noisy data, which reveals the denoising efficacy of CL on noisy data. The key to applying CL. is how define clean/noisy examples and how to determine proper denois- ing scheme. By analyzing the characteristics of the task, we tailor-design a Multi-grained Multi- curriculum Denoising Framework (M2DF). M2DF first evaluates the degree of noisy images contained in each training instance from multiple granulari- ties, and then the denoising curriculum gradually exposes the data containing noisy images to the model starting from clean data during training. In this way, M2DF theoretically assigns larger learn- ing weight to clean data (Gong et al., 2016), thereby achieving the effect of noise reduction.
(2) Partitioning the image into multiple visual regions averagely, and subsequently establishing connections between text sequences and visual re- gions. Yu and Jiang (2019) used the pre-trained model BERT (Devlin al., 2019) and ResNet (He et al., 2016) to extract aspect and visual regions fea- tures, and devised an attention mechanism to obtain aspect-sensitive visual representations. Yu et al. (2020) proposed an entity span detection module to alleviate the visual bias. Sun et al. (2020), Sun et al. (2021) and Ju et al. (2021) introduced cross- modal relation detection module to decide whether the image feature is useful. Zhao et al. (2022a) proposed to utilize the external matching relations between different (text, image) pairs to improve the performance. Zhao et al. (2022b) developed a Knowledge-enhance Framework to improve the visual attention capability.
Our contributions are summarized as follows: (1) To the best of our knowledge, we provide a new perspective to reduce the negative impact of noisy images in the MABSA task; (2) We propose a novel Multi-grained Multi-curriclum Denoising Framework, which is agnostic to the choice of base models; (3) We evaluate our denoising framework on some representative models, including the cur-
(3) Retaining exclusively the regions containing visual objects in the image and enabling dynamic interactions with the text sequences. Wu et al. (2020a.b) and Zheng et al. (2021) employed Faster-
RCNN (Ren et al., 2015) or RCCN- (He et al., 2020) to extract object features, and then interact

---

Multiple Denoising Curriculum
Single Denoising Curriculum
Ranking by Coarse-grained Noise Metric de
Batch Sampled from clean data to noisy data
Training the Model
Compute the si the validation set
Training Dataset
$b Argmax(s s())
Single Denoising Curriculum
Ranking by Fine-grained Noise Metric dz
Batch Sampled from clean data to noisy data
Training the Model
Compute the s/ on the validation set
The overview of our M2DF framework. The solid line @ indicates the coarse-grained denoising curriculum is selected at the current training step, and the dashed line @ indicates it is not selected.
Figure I:
with textual features via a co-attention network. Zhang et al. (2021) utilized multi-modal graph capture various semantic relationships between words and visual objects in (text, image) pair.
ules: Noise Meirics and Denoising Curriculums. To be specifie, when the image and text unre- lated, it is likely that the image is noise. In such cases, the similarity between the image and text is usually low, and the aspect terms in the text will generally not appear the image. Based on this, we severally define a coarse-grained (sentence- level) noise metric and fine tite-graind (aspect-level) noise metric measure the degree of noisy images contained in each training instance.
Despite their success, most studies overlooked the fact that there are many noisy images in the dataset that are unrelated to the text, which neg- atively impacts model learning. While some ef- forts (Sun et al., 2020, 2021; Yu et al., 2022; Ju al., 2021) have been made develop cross- modal relation detection modules to filter out qualiuylity noisy images, these approaches are deeeedeld-deeeddent and inevitably filter out lots of useful image information. Therefore, in this work, we focus whether the negative impact of noisy images can be reduced without data filtering.
Subsequently, we design a single denoising cur- riculum for each noise metric, which first ranks all training instances according to the size of noise metric, and then gradually exposes the training data containing noisy images to the model starting from clean data during training. In this manner, the model trains more time on the clean data and less time the noisy data, tive impact of noisy images. Given that the two noise metries measure the noise level of the im- age from different granularities, a natural idea combine them together to produce better denoising effects. To this end, we extend the single denoising curriculum into multiple denoising curriculum.
Curriculum Learning. The concept of Currieu- lum Learning (CL) is initially introduced by (Ben- gio al., 2009), and has since manifested its mul- tifaceted advantages across a diverse array ma- chine learning tasks (Platanios et al., 2019; Wang et al., 2019; Liu et al., 2020; Xu et al., 2020; Lu and Zhang, 2021). One of the advantages of CL. is denoising, which is achieved by training more time clean data and less time noisy data (Wang al., 2022). In this work, we borrow the idea of CL reduce the negative effect of noisy images. To the best of our knowledge, this the first attempt to use CL for denoising in the MABSA task.
to reduce the nega-
3.2 Noise Metrics
In this section, we define the Noise Metrics used by M2DF from two granularities.
3.2.1 Coarse-grained Noise Metric
Empirically, when the image and text are relevant, the similarity between them tends to be relatively high. Conversely, when the image and text are irrelevant, the similarity between them is generally low. Thus, it can be inferred that the lower the similarity between the image and the text, the more likely the image is noise.
3 Methodology
3.1 Overview
As shown in Figure 1, we propose Multi-grained multi-curriculmm Denoising Framework (M2DF) for the MABSA task, which consists of two mod-

---

Based on the above intuition, we measure the degree of noisy images contained in each training instance by calculating the similarity between the image and the text. Specifically, given a sentence- image pair (St, 13) in the training dataset D, we first imput each modality into different encoders of pre-trained model CLIP (Radford et al., 2021) to obtain the textual feature It's and visual feature Vj. Then the similarity is calculated as:
Coerse-greined Noise Metric
Denoising function pit)
Noise Metric
0.
22. Pt) current step
Sample data unifermiy
region
0%
3o0n
10000 Training
Training dataset
Figure Illustration of the Single Denoising Curricu- lum for coarse grrseegraiinddd noise metric de- At each step. p(t) is computed based on the current step t. and a batch of data uniformly sampled from the training instances whose coarse- coarse-grained noise metric de is lower than p(t) (blue region in the example).
sim H3, v/) = cos H's Vi),
(1)
where cos(.) is a cosine function. After that, con- sidering the lower the similarity between the image and the text, the more likely the image is noise, we define coarse-grained (sentence-level) noise metric as follows:
and visual objects extracted from the sentence S and the image I, are follows:
daspect (S:) & Ns
(3)
Hm's Vi) max (HE sim S v/y
d. = 1.0
(2)
dodijeed (Is) & Ni.,
(4)
Then, input aspect terms in sentence S₃ and visual objects in image I; into different encoders of pre-trained model CLIP to obtain a group of aspect Ns; features (H22) }= and object features {V.*) j= Nr;
where de is normalized to [0.0, 1.0]. Here, lower score indicates the pair (S,, Is) is clean data.
3.2.2 Fine-grained Noise Metric
After that, we average all these aspect features and object features to obtain the final aspect represen- tation and object representation. The cross- modal similarity between them is as follows:
Except for sentence-level noise metric, we also measure the noise level from the perspective of as- pect, because the goal of MABSA is to identify the aspect termss or predict the sentiment of aspect terms. Specifically, when the image and text are relevant, the aspect terms in the sentence will ap- pear in the image with high probability. In contrast, when the image and text are irrelevant, the aspect terms in the sentence will generally not appear in the image. Thus, we can judge the relevance of the image and the text by calculating the similarity between the aspect terms in the sentence and the vi- sual objects in the image. The lower the similarity is, the image is more likely to be noise.
HS.
H
(5)
Ns.
N1.
Vl
V
(6)
N1.
j=1
sim(H a V.le) = co V!:),
(7)
Finally, considering that the lower the similarity between the aspect terms and the visual objects, the more likely the image is noise, define fine-grained (aspect-level) noise metric follows:
Based on the above intuition, we measure the de- gree of noisy images by calculating the similarity between the aspect terms the sentence and the visual objects in the image. However, except for the MASC task, the aspect terms other subtasks are unknown. Considering that aspect terms are mostly noun phrases (Hu and Liu, 2004), we em-- ploy noun phrases extracted by the Stanford parser' aspect terms within sentences for the MATE and JMASA tasks. Meanwhile, we use the object de- extract the visual
sim sim V
dy = 1.0 max, (H .v/ v JeD sim H.Sh Vola)*
(8)
where dy is normalized [0.0, 1.0]. Similarly, a lower score indicates the pair (S,, I2) is clean data.
3.3 Denoising Curriculums
tection model objects in the image. The number of aspect terms
In this section, we first introduce the single denois- ing curriculum for each noise metric. Then we extend it into the multiple denoising curriculum.

---

Algorithm 1: Multiple Denoising Curricu-
data and less time noisy data, which reveals the denoising efficacy of CL on noisy data (Wang al., 2022). After p(t) becomes 1.0, the single denoising curriculum completed and the model can then freely access the entire dataset. In Figure 2, we give an illustration of the Single Denoising Curriculum for coarse-grained noise metric.
lum.
The training set: D; Coarse-ge grainedg Noise Metrie: dei Fine-grained Noise Metrie: dy. Output: A model with multiple denoising curriculum.
Input:
Compute two noise metrics de and d for each training sample in D;
Sort D based on two noise metrics, resulting in D. and Dy;
3.3.2 Multiple Denoising Curriculum As mentioned above, each single denoising cur- riculum uses only one noise metric for denoising. Considering that the two noise metrics measure the noise level of the image from different granular- ities, a natural idea is to combine them together to produce better denoising effects. To this end, we present an extension to the single denoising curriculum: multiple denoising curriculum.
// Initializa f0r sg for De and Dfr
respectives ively;
for D, De. D, do
Initialize the p();
Initialize the training step 4 0: while 4 <= 1 do
Uniformly sample one batch (((4:) from DD such that d; S (ta);; Training the model with (t)):
4- f, + 1;
end
11
Compute the ratio si of two consecutive performances on a validation set;
Intuitively, there are a few readily conceivable approaches, including directly merging two noise metrics into one (i.e., de + dy), as well as ran- domly or sequentially selecting curricula to train the model (i.e., de + dj -> de > dd). However, a more impactful strategy lies in dynamically select- ing the most appropriate denoising curricula at each training step. To achieve this goal, use the ratio of two consecutive performances on validation measure the effectiveness of each denoising curriculum as follows:
12 end
/ synamically select the most sui t dencising curriculung
13
while not converged do
14
j arg max(s)); // denotes (E(c.))
15
6g +;
16
Uniformly sample one batch (t,),) from Dj, such that d, S (t,);
17
Training the model with (2
Compute the ratio oi two consecutive performances on a validation set;
18
Update of;
19
end
98
(10)
X Sprea
3.3.1 Single Denoising Curriculum
X 8 = Pre Rec
2* * Pre * Rec
(11)
In training, we first sort each instance in the training dataset 'D based on a single coarse marse-grained noise metric de or fine- fine-grained noise metric dy. Then inspired by (Platanios et al., 2019), we define simple denoising function p(t), which controls the pace of learning from clean data noise instances. The specific function form is follows:
where x" is the Fl-score computed at the current validation turn, X is computed the previ- ous validation turn, Pre and Rec denote Precision and Recall. A higher value of sy indicates greater progress achieved by the current denoising curricu- lum, which should lead to a larger sampling proba- bility to achieve better denoising. Based on this, at each training step, we can dynamically select the most suitable denoising curriculum by comparing the values of sø between the two denoising curricu- lums. The details of multiple denoising curriculum are presented in Algorithm and Figure 1.
P(O)
((02 IT
p(t)
1.0
otherwise. (9)
where p(0) is predefined initial value and is usu- ally set to 0.01, T is the duration of the denoising curriculum. At time step t, p(t) means the upper limit of noise metric, and the model is only allowed use the training instances whose noise metric de or d is lower than ptt). In this way, the model no longer presents training data in completely random order during training, but trains more time on clean
Experiments
4.1 Experimental Settings
Datasets. To evaluate the effect of the M2DF Framework, we carry out experiments on two public multimodal datasets TWITTER-15 and

---

TWITTER-15
TWITTER-17
Pos
Neg
Neu
Total
AT
Words
Pos
Neu
Neg
Total
Words
AL
Train
368
3179
1.348
9023
16.72
1508
416
1638
3562
1.410
6027
16.21
Dev
303
149
670
1122
1.336
4238
16.74
515
144
517
1176
1.439
2922
16.37
Test
317
113
607
1037
1.354
3919
17.05
493
168
573
1234
1,450
3013
16.38
Table 2: The basic statistics of our two multimodal Twitter datasets. Pos: Positive, Neg: Negative, Neu: Neutral, AT: Avg. Targets, AL: Avg. Length
TWITTER-15
TWITTER-17
Methods
Pre
Rec
UMT-collapse (Yu al., 2020)
60.4
61.6
61.0
60.0
61.7
60.8
M.DF
61.1:0.44
63,411.57
62.2:1010
60.910.28
62.0:10.52
61.410.13
OSCGA-collae CooG--collapse (Wu et al., 2020h)
63.1
63.7
63.2
63.5
63.5
63.5
M2DF
64.410037
64.610044
64.5:10.33
64.11
63.9110.16
64.0.10.12
RpBERT (Sun et al., 2021)
49.3
46.9
48.0
57.0
55.4
56.2
M2DF
49.3+0.20
49.0+1..5
09.2+40115
56.940.34
56.5:+0.38
56.7::0.22
RDS
(Xu et al., 2022)
60.8
61.7
61.2
61.8
62.9
62.3
+ M2DF
61.240.12
63.0110.35
62.1:0.115
62.4:0.16
63.6:10:12
63.0:10.08
JMI.* (Ju al., 2021)
64.8
63.6
64.0
65.6
66.1
65.9
M2DF
64.910.36
65.310,16
65, 65.110..25
67.710.30
67.0110.08
67.3:10.16
VIP-MARSA (Ling al., 2022)
64.1
68.6
66.3
65.8
67.9
66.9
M2DF
67.0.1..0
68.3:10.26
67.6* 100.18
67.9110.10
68,8110.77
68.31
1:018
Table 3:
Test results on the IWITTER-15 and TWITTER-17 datasets for JMASA task (%). For the baseline models, the results with & are obtained by running the code released by the authors, and the other results without symbols are retrieved from (Ju al., 2021) and (Ling et al.. 2022). Best results in bold. The marker" refers significant test p-value < 0.05 when comparing with other multimodal methods.
TWITTER-11 from (Yu and Jiang, 2019), which include user tweets posted during 2014-2015 and 2016- 2017, respectively. General information for both datasets is presented in Table 2.
4.2 Compared Methods
In this section, we introduce some representative baselines for each sub-task of MABSA: (1) Base- lines for Joint Multimodal Aspect-Sentiment Analysis (JMASA): UMT-collapse (Yu et al., 2020), OSCGA-collapse (Wu et al., 2020b), Rp- BERT (Sun et al., 2021), RDS (Xu et al., 2022), JML (Ju et al., 2021), VLP-MABSA (Ling et al., 2022); (2) Baselines for Multimodal Aspect Term Extraction (MATE): UMT (Yu et al., 2020), OS- CGA (Wu et al., 2020b), JML (Ju et al., 2021). VLP-MABSA (Ling et al., 2022); (3) Baselines for Multimodal Aspect Sentiment Classification (MASC): TomBERT (Yu and Jiang, 2019), Cap'T- BERT (Khan and Fu, 2021), ), JML (Ju et al., 2021), ITM (Yu et al., 2022), FITE (Yang et al., 2022), VLP-MABSA (Ling et al., 2022).
Implementation Details. During training, we train each model for a fixed 50 epochs, and then select the model with the best Fl score on the val- idation set. Finally, we evaluate its performance on the test set. We implement all the models with the PyTorch? framework, and run experiments RTX3090 GPU. More details about the hyper- parameters can be found in Appendix A.
Evaluation Metries and Significance Test. As
with the previous methods (Ju et al., 2021; Yu and Jiang, 2019), for MATE and JMASA tasks, adopt Precision (Pre), Recall (Rec) and Micro-F1 (FI) as the evaluation metrics. For the MASC task, we use Accuracy (Acc) and Macro-FI as evaluation metrics. Finally, we report the average performance and standard deviation over 5 runs with random ini- tialization. Besides, the paired t-test is conducted test the significance of different methods.
To verify the generalization of our framework, we apply M2DF to all baselines of three sub-tasks. It should be noted that RpBERT, RDS, JMT, and ITM a cross-modal relation detection module filter out noise images. In order to better compare their denoising effects with M2DF, we replace the cross-modal relation detection module in RpBERT, RDS, JMT, and ITM with our M2DF framework.
thttss//pytorch.orr/

---

TWITTER-13
TWITTER-17
Methods
Pre
Rec
Pre
Rec
UMT (Yu al., 2020)
77.8
81.7
86.7
86.8
86.7
+
M22F
79.1+0.11
81.5++0333
80.3+0012
87.4+0.18
87.5+0.22
87.5++0.15
OSCGA (Wu et al., 2020b)
81.7
82,1
81.9
90.2
90.7
90.4
M2DF
82.0:10.10
82.8:0.31
82.4:10.13
90.3+0.15
91.540.17
90.9:0.007
JML (Ju et al., 2021)
82.9
81.2
82.0
90.2
90.9
90.5
+ M2DF
84.0+0.26
82.3+0012
83.14-0114
91.1+0011
90.9++018
91.0+0.12
VLP-MABSA* (Ling al., 2022)
82.2
88.2
85.1
89.9
92.5
91.3
+ M2DF
85.2+0.24
87,4+0020
86.3) +0.15
91.5+0.25
93.2+0.23
92.4' +0.14
Table 4: Test results on the TWITTER-15 and TWITTER-17 datasets for MATE task (%) For the baseline models. the results with are obtained by rumning the code released by the authors, and the other results without symbols are retrieved from (Ling et al., 2022). Best results are in bold.
Methods
TWITTER-15
TWITTER-17
on both datasets. This outcome reveals the effec- tive denoising effect of our framework; (3) is
Ace
TomRERT
77.2
71,8
70.5
68.0
M2DF
77.910.11
73.2.10.11
71.010.14
68.7100.00
worth mentioning that RpBERT + M2DF and RDS
CaRT:ERRT
78.0
73.2
72.3
70.2
+
M2DF obtain good improvements in Micro-F1 over RpBERT and RDS. Similarly, JMT JMT + M2DF achieves a boost of 1.1% and 1.4% in Micro-F1 compared to JMT. These observations indicate that the denoising effect of M2DF is better than that of the cross-modal relation detection module; (4) The performance boost of VLP-MABSA + M2DF over VLP-MABSA by 1.3% and 1.4%. These re- sults affirm the robustness of the M2DF, showcas- ing its ability to consistently generate substantial enhancements even when applied to the current state-of-the-an art model.
M2DF
78.420.12
74.0=20.8
73.0:0.08
71.340.07
HTTE
78.5
73.9
70.9
68.7
789+0.07
74.2+00.8
71.5+0.11
69.440.12
ITM
78.3
74.2
72.0
78910.05
75.010.07
73.2100.00
73,010.08
IMI
78.1
72.7
M2DF
78.8:0.15
74.0:10.12
VLP MABSA
77.2
72.9
73.2
71.4
M2DF
78.9
74.8 10021
74.3/
73.0' 10.16
Table 5: Test results on two Twitter datasets MASC ta))(%). For the baselines, the results with are ob- tained by running the code released by the authors, and the other results are retrieved from (Ling et al., 2022), (Yu et al.. 2022) and (Yang et al., 2022). For fair com- parison, we follow the setting of (Ju et al., 2021) and (Ling et al., 2022), i.e,, JML only evaluates the aspects correctly predicted on the MATE task while the other methods evaluate on all the golden aspects.
Results of MATE and MASC. Table and Ta- ble show the main experiment results of MATE and MASC. As we can see, incorporating M2DF into the baselines of MATE and MATC tasks can obtain competitive performance. To be specific, for the MATE task, VLP-MABSA + M2DF obtain 1.2% and 1.1% improvements in terms of the F1- For the MASC task, VLP-MABSA + M2DF can improve the Fi-score by up 1.9% and 1.6%. These results indicate that M2DF applicable three subtasks simultaneously, showcasing its re- markable capacity for generalization.
For other baselines, we directly integrate M2DF into them to obtain the final model.
5 Results and Analysis
5.1 Main Results
this section, we analyze the results of different methods on three sub-tasks of MABSA.
5.2 Ablation Study
Results of JMASA. The main experiment re- sults of JMASA are shown in Table 3. Based on these results, we can make couple of observa- tions: (1) VLP-MABBS performs better than other multimodal baselines. A possible reason is that it develops a series of useful pre-training tasks; (2) Compared to the base model UMT-collapse and OSCGA collapse, UMT-collapse + M2DF and OSCGA OSCGA-collapse + M2DF achieve better results
Without loss of generality, we choose VIPP-MBSAA + M2DF model for the ablation study to investigate the effects of different modules in M2DF.
Effects of the Noise Metrics. As we mentioned in Section 3.3.1, we develop single denoising curriculum for each noise metric. Here, we dis- cuss the contribution each noise metric in our framework. From Table 6, we can observe the fol-

---

Coarse-grained Noise Metrie
Fine-grained Noise Metric
Denoising Curriculum
TWITTER-15
TWITTER-17
Settings
MATE
MASC
JMASA
MATE
MASC
JMASA
VLP-MABSA
X
85.1
72.9
66.3
91.
71.4
66.9
(a)
X
Single
85.8
73.6
66.8
91.6
72.1
67.4
(b)
X
Single
85.9
73.9
67.0
91.6
72.3
67.5
(c)
Multiple
86.3
74.8
67.6
92.4
73.0
68.3
(d)
Merge
85.8
73.8
66.9
91.7
72.1
67.9
(c)
Randomly
86.1
74.2
67.3
91.9
72.5
68.0
Sequentially
86.2
74.5
67.3
92.1
Table 6:
Ablation study of denoising framework(%)). We the current state-of-the-art model VL.P- MABSA (Ling et al., 2022) as the base model to conduct the analysis. The setting (c) denotes our full proposed approach. We evaluate three tasks JMASA, MATE, and MASC terms
Level-1
Level-2
Level-3
Methods
MATE
MASC
JMASA
MATE
MASC
JMASA
MATE
MASC
JMASA
VLP-MABSA
92.7
73.4
67.2
91.6
73.0
69.8
89.6
68.0
63.4
VLP-MABSA M2DF
92.9
73.6
67.8
92.1
73.8
71.2
91.6
71.3
65.6
+0,2
+0.2
+0.6
+0,5
+0.8
+1.4
+2.0
+3.3
+2.2
Table 7: Model performance the divided TWITTER-11 test sets with different noise levels (%). We evaluate three tasks JMASA, MATE, and MASC terms of F1. represents the difference between the performance of VLP-MABSA and VLP-MABSA M2DF at different noise level.
lowing: (1) Settings (a-b) reveal that every noise metric can boost the performance of base model VLP-MABSA, which validates the rationality of our designed noise metrics; (2) The fine-grained noise metric achieves better performance than the coarse- grained noise metric in most tasks, which indicates that üine-graines noise metric is more accurate in measuring whether noisy images are included in each training instance; (3) When incorporate two noise metric together, the performance is further improved (see setting c), which demonstrates that two noise metrics improve the performance of the MABSA task from different perspectives.
Performance on Different Subsets. To gain a deeper insight into the role of M2DF, we took an ad- ditional step by partitioning the TWITTER- 17 test set into three subsets of equal size. This division was based the values of coarse-grained noise metric de spanning from low to high. By employ- ing this approach, we aimed examine the impact of M2DF across various levels of noise present in the datasct. The results are shown in Table 7. We can observe that compared with VLP-MABSA, VLP-MABSA M2DF achieves more performance improvement on the subset with the higher noise level. This suggests that M2DF can indeed mitigate the negative impact of noisy images.
Effects of the Denoising Curriculums. As we mentioned in Section 3.3.2, three easily thought of strategies to implement the multiple denoising curriculum are Merge, Randomly, and Sequentially. Table 6 (settings d-f) shows the results of the three strategies. As we can see, (1) All three strategies achieve stable improvements on the MABSA tasks, which demonstrates the effectiveness and robust- ness of our M2DF framework; (2) The results of the Merge strategy are inferior to Randomly and Sequentially. A possible reason is that directly merging two noise metrics is unable give full play their respective advantages; (3) All of them perform worse than our final approach (setting c), which verifies the superiority of the dynamic learn- ing strategy at each training step.
5.3 Training Time
Curriculum Learning can effectively steer the learn- ing process away from poor local optima, making converge faster (Bengio et al., 2009; Platanios et al., 2019). As a result, the training time for the denoising phase is reduced compared to regular training. In Appendix C, we display the training times for VLP-MABSA and VLP-MABSA+M2DD the JMASA task.
5.4
Case Study
As we mentioned in section 5.2, M2DF performs well when the multimodal instance contains noisy images unrelated to the text, here we conduct the case study to better understand the advantages

---

(a)
(c)
Th
Day
Image
RT jmsclee And when Alexander saw the breadth of his domain, he wept
Benicia High baseball
Montreal Candiens Pet dog jersey shirt ( sizes
Text
clinches playoff berth with
10 win over Fairfield
NEW
(Benicia High, POS)
(Montreal Candiens, NEU)
Golden
(Alexander, NEU)
(Fairfield, NEU)
(NHL, NEU)
JML
(Alexander, POS) X
(Benicia, NEG)
(Montreal, POS)
X
(Fairfield, NEG)
(NHL, NEG)
(Benicia High, NEU) X
VLP-MABSA
(Alexander, POS) X
(Fairfield, NEG)
(Montreal Candiens NHL, POS)
(Benicia High, POS)
(Montreal Candiens, NEU) (NHL, NEU) (Montreal Candiens, NEU) (NHL, NEU)
JML M2DF
(Alexander, NEU)
(Fairfield, NEU)
VLP-MABSA
(Benicia High, POS) (Fairfield, NEU)
(Alexander, NEU)
M2DF
Figure 3: Three examples of the predictions by JML, YLP-MABSA, JML M2DF, and VLP-MABSA + M2DF. POS, NEU, and NEG are short for Positive, Neutral, Negative, respectively.
M2DF. Figure 3 presents the predict results of three noise examples on the JMASA task by JML M2DF and VLP-MABSA M2DF, as well as representative baselines JMI. and VLP-MABSA. It clear that both JML and VLP-MABSA give the incorrect prediction when faced with noisy im- ages. The reason behind this may be that: (1) al- though JML develops a cross-modal relation detec- tion module to filter out noisy images, this module relies on thresholds, so using it to filter noisy im- ages is not reliable enough; (2) VLP-MABSA treats the image equally and ignores the negative effects of noisy images. In contrast, observe that JML M2DF and VLP-MABSA + M2DF can obtain all correct aspect terms and aspect-dependent senti- ment on three noise samples, suggesting that our framework M2DF can achieve better denoising.
plc denoising curriculum. Results from numerous experiments indicate that our denoising framework achieves better performance than other state-of- the- art methods. Further analysis also validates the superiority of denoising framework.
Limitations
The current study has two limitations that warrant further attention. First, while we have developed two noise metrics to evaluate the degree of noisy images contained in the dataset, there may be other methods to achieve this goal, which requires us to do further research and exploration. Second, we have not examined other CL. training schedules, such as self-paced learning (Kumar et al., 2010), which may worth considering in the future. We view our study as starting point for future re- search, with the goal of propelling the field forward and leading to even better model performance.
6 Conclusion
In this paper, we propose a novel Multi-grained Multi-curicclum Denoising Framework (M2DF) first define
Acknowledgements
for the MABSA task. Specifically, coarse-grained noise metric and fine-grained noise metric to measure the degree of noisy images contained in each training instance. Then, in order to reduce the negative impact of noisy images, we design single denoising curriculum and a multi-
We would like to thank the anonymous reviewers for their insightful comments. Fei Zhao would like to thank Siyu Long for his constructive sugges- tions. This work is supported by the National Nat- ural Science Foundation of China (No. 62206126. 61936012 12 and 61 61976114)

---

References
Conference, Virtual Event, China, October 20 24. 2021, pages 0034-3042.
Yoshua Bengio, Jéröme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Con- ference on Machine Learning. ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009, volume 382 of ACM International Conference Proceeding Series, pages 41-48, ACM.
M. Pawan Kumar, Benjamin Packer, and Daphne Koller. 2010. ceff-paced learning for latent variable mod- els. In Advances Neural Information Processing Systems 23: 24th Annual Conference on Neural Infor- mation Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada, pages 1189--1977 Curran Asso- ciates, Inc.
Guimin Chen, Yuanhe Tian, and Yan Song. 2020. Joint aspect extraction and sentiment analysis with direc- tional graph convolutional networks. In Proceedings of the 28th International Conference on Compula- tional Linguistics, pages 272-279, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising peuueece-to-sqqeeeee for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguisties.
pre-training
Jacob Devlin, Ming-Wei, Wei Chang, Kenton Lee, and
Kristina Toutamova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proc. of NAACI, pages 4171-4188
Yan Ling, Jianfei Yu, and Rui Xia. 2022. Vision- language pre-training for multimodal aspect-based sentiment analysis. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics ( (Volume I: Lonng Papers), pages 21 2149- 2159, Dublin, Ireland. Association for Computational Linguistics.
Tieliang Gong, Qian Zhao, Deyu Meng, and Zong- ben Xu. 2016. Why curriculum learning & self- paced learning work in big/noisy data: A theoret- ical perspective. Big Data & Information Analytiss 1(1):111.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. 2020. Mask R-CNN. IEEE Trans. Pattern Anal. Mach. Intell., 32(2))386-397
Bing Liu. 2012. Sentiment Analysis and Opinion Min- ing. Synthesis Lectures on Human I Language Tech- nologies. Morgan & Claypool Publishers.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Xuebo Liu, Houtim Lai, Derek F. Wong, and Lidia S. Chao. 2020. Norm-based curriculum learning for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 427-436. Association for Computational Linguistics.
Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, and Yiwei Lv. 2019. Open-domain targeted senti- ment analysis via span- based extraction and classifi- cation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 537-546, Florence, Italy, Association for Computa- tional Linguistics.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-an and and-langaaee tasks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8- 14. 2019, Vancouver. BC, Canada, pages 13-23.
Minqing Hu and Bing Liu. 2004. Mining and summa- rizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference Knowl- edge Discovery and Data Mining, Seattle, Washing- ton, USA, August 22-25, 2004, pages 168-177. ACM.
Jinliang Lu and Jiajun Zhang. 2021. Exploiting curricu- lum learning in unsupervised neural machine trans- lation. In Findings of the Association for Compu- tational Linguistics: EMNLP 2021, Virtual Evens / Punta Cana, Dominican Republic, 16-20 November, 2021. pages 924 934. Association for Computational Linguistics.
Xincheng Ju, Dong Zhang, Rong Xiao, Junhui Li, Shoushan Li, Min Zhang, and Guodong Zhou. 2021. Joint multi- modal aspect-sentiment analysis with aux- iliary cross-modal relation detection. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 4395-4405, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Scungwhan Moon, Leonardo Neves, and Vitor Carvalho. 2018. Multimodal named entity recognition for short social media posts. In Proceedings of the 2018 Con- ference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies. NAACL-HLT 2018 New Or- leans, Louisiana, USA, June 1-6, 2018, Volume
Zaid Khan and Yun Fu. 2021. Exploiting BERT for multimodal target sentiment classification through input space translation, In MM '21: ACM Multimedia

---

(Long Papers), pages 852-860. Association for Com- putational Linguistics.
Xin Wang, Yudong Chen, and Wenwu Zhu. 2022. A survey on curriculum learning. IEEE Trans. Pattern Anal. Mach. Intell., 449):4555-4576
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabás Póczos, and Tom M. Mitchell. 2019. Competence-hased curriculum learning for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLI 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 1162-1172. 1172. Associa- tion for Computational Linguisties.
Hanqian Wu, Siliang Cheng, Jingjing Wang. Shoushan Li, and Lian Chi. 2020a. Multimodal aspect extrac- tion with region- region-aware aware alignment network. In Nat- ural Language Processing and Chinese Computing 9th CCF International Conference, NLPCC 2020, Zhengzhou. China, October 14-18, 2020, Proceed- ings, Pari I, volume 12430 of Lecture Notes in Com- puter Science, pages 145 1566. Springer.
Zhiwei Wu, Changmeng Zheng, Yi Cai, Junying Chen, Ho-fung Leung, and Qing Li. 2020b. Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts. In Proceedings of the 28th ACM International Con- ference on Multimedia, MM '20, page 1038-1046, New York, NY, USA. Association for Computing Machinery.
Maria Pontíki, Dimitris Galanis, John Pavlopoulos, Har- ris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014, Semeval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Viriual Event, volume 139 of Proceedings of Machine Learning Research, pages 9748-8763.
Benfeng Xu, Licheng Zhang. Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. 2020. Curriculum learning for natural language understand- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020. Online, July 5-10, 2020, pages 0095-6104. Association for Computational Linguisties.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster R-CNN: towards real-time ob- ject detection with region proposal networks. In Ad- vances in Neural Information Processing Systems 28: Annual Conference on Neural Information Process- ing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 91-99.
Bo Xu, Shizhou Huang, Ming Du Hongya Wang, Hui Song, Chaofeng Sha, and Yanghua Xiao. 2022. Dir- ferent data, different modalities! reinforced data split- ting for effective multimodal information extraction from social media posts. In Proceedings of the 29th International Conference on Computational Linguis- rics, pages 1855-1864. International Committee on Computational Linguistics.
Lin Sun. Jiquan Wang. Yindu Su. Fangsheng Weng. Yuxuan Sun. Zengwei Zheng, and Yuanyi Chen. 2020. RIVA: A pre-trained tweet multimodal model based on text-image relation for multimodal NER. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1852-1862, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Nan Xu, Wenji Mao, and Guandan Chen. 2019, Multi- interactive memory network for aspect based multi- modal sentiment analysis. In Proc. of AAAI, pages 371-378.
Hao Yang, Yanyan Zhao, and Bing Qin. 2022. Face- sensitive image-to- songe-tt--emotion w-----mooo---ceeee cross-modal trans- lation for multimodal aspect-based sentiment analy- sis. In Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing, pages 3324-3335, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, and Fang- sheng Weng. 2021. Rpbert: A text-image relation propagation- based BERT model for multimodal NER. In Thirty-Fifth AAAI Conference on Artificial Intel- ligence, AAAI 2021, Thirty-Third Conference on In- novative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence. EAAI 2021. Virtual Event, February 2-9. 2021, pages 13860-13868.
Jianfei Yu and Jing Jiang. 2019. Adapting bert for target-oriented multimodal sentiment classification. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. 19, pages
International Joint Conferences on Artificial Intelligence Organization.
Wei Wang. Isaac Caswell, and Ciprian Chelba. 2019. Dynamically composing domain-data selection with clean-data selection by cc--curiiclar learning" for neural machine translation. In Proceedings of the 57th Conference of the Association for Computa- tional Linguistics, ACI. 2019, Florence, haly, July 28- August 2, 2019, Volume 1: Long Popers, pages 1292-1292. Association for Computational Linguis- tics.
Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020. Improving multimodal named entity recognition via entity span detection with unified multimodal trans- former. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, ACL 2020. Online, July 5-10, 2020, pages 3342-3352. Association for Computational Linguistics.

---

Jianfei Yu, Jieming Wang, Rui Xia, and Junjie Li. 2022. Targeted multimodal sentiment classification based on coarse-to-fine grained image-target match- ing. In Proceedings of the Thirty-Firss International Joint Conference on Artificial Intelligence, IJCAI-22, pages 4482-4488. International Joint Conferences on Artificial Intelligence Organization, Main Track.
Methods
Backbone
Learning Rate
UMT-collapse
BERT+ResNet
5c-5
UMT
BERT+ResNet
5e-5
OSCGA-collapse
Glove+Mask RCNN
0.008
OSCGA
Glove+Mask RCNN
0.008
RpBERT
BEKT+ResNet
lc-6
TomBERT
BERT+ResNet
5e-5
CapTrBERT
BERT+ResNet
5e-5
Dong Zhang. Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021. Multi- modal graph fusion for named entity recognition with targeted visual guidance. In AAA/ 2021, Vir- tual Event, February 2-9, 2021, pages 14347-14355.
FITE
BERT=ResNet
5e-5
JML
BERT-ResNet
2c-5
VLP-MABSA
BART+Faster R-CNN
5e-5
Table 8: Hyperparameters of baseline methods.
Fei Zhao, Chunhui Li, Zhen Wu, Shangyu Xing, and Xinyu Dai. 2022a. Learning from different text- image pairs: A relation-enhanced graph convolu- tional network for multimodal ner. In Proceedings of the 30th ACM International Conference on Multime- dia, MM '22, page 3983-3992, New York, NY, USA. Association for Computing Machinery.
VLP-MABSA VLP-MABSA + M2DF
80
60
Minute
Fei Zhao, Zhen Wu, Siyu Long, Xinyu Dai, Shujian Huang, and Jiajun Chen. 2022b. Learning from adjective-noun pairs: A Knowledge-enhanced frame- work for target-oriented multimodal sentiment classi- fication. In Proceedings of the 29th International Conference Computational Linguistics, pages 6784-6794, Gyeongju. Republic of Korea. Interna- tional Committee on Computational Linguistics.
40
0
Twitter-15
Twitter-17
Dataset
Figure 4: Training times for VLP-MABSA and VLP- MABSA+M2DF on the JMASA task.
Changmeng Zheng, Zhiwei Wu, Tao Wang, Yi Cai, and Qing I.i. 2021. Object-aware multimodal named en- tity recognition social media posts with adversarial learning. IEEE Trans. Multim., 23:2520-255 25322.
foundation of BART, and the multimodal methods VLP-MABSA and VLP-MABSA M2DF both outperform BART, which indicates that image in- formation useful for the MABSA task. This aligns with the conclusions of previous methods.
A Implementation Details
As we mentioned in section 4.2, we chose some representative models as the foundations of our work for three sub-tasks. Therefore, when applying framework, we keep the parameters the same the baseline methods in Table 8. We implement all the models with the PyTorch framework, and run experiments on an RTX3090 GPU.
Effect of Different Multimodal Encoder and Im- age Encoder We utilize CLIP for multimodal feature extraction, due to its training vast col- lection of text-image pairs. This allows it to excel in capturing multimodal semantics better. Besides, we also try using ViLBERT (Lu al., 2019) extract multimodal features. For object detection, we try using Faster-RCNN to replace The results of these two methods on the JMASA task are shown in Table 10. We can observe that the Fl -core of VLP-MABSA + M2DF (ViLBERT) is lower than that of VLP-MABSA + M2DF (Ours). This is because CLIP benefits from larger-scale and more diverse training dataset, which enables the model to capture multimodal features more ac- curately. Thus, CLIP is better version to extract multimodal features. Moreover, the Fl-score VLP-MABSA + M2DF (Faster-RCNN) is slightly
B Additional Analysis
The Image Information is Useful Except for multimodal baselines, we also compare the M2DF with pure textual ABSA models, i.e., SPAN (Hu et al., 2019), D-GCN (Chen et al., 2020) and BART (Lewis et al., 2020), the results of the JMASA task are shown in Table 9. For bext- methods, it is clear that BART consistently out- performs all the baselines, we attribute this to the fact that the encoder-decoder structure of BART can grasp the overall meaning of the whole sen- tence in advance, which is important for predicting label sequences. VLP-MABSA is built upon the

---

TWITTER-15
TWITTER-17
Methods
Rec
Pre
Rec
SPAN (Hu al., 2019)
53.9
53.8
59.6
61.7
D-GCN (Chen al., 2020)
58.3
58.8
58.5
64.2
64.1
64.1
BART (Lewis et al., 2020)
62.9
65.0
63.9
63.2
65.6
65.4
VLP-MABSA (Ling et al., 2022)
64.1
68.6
66.3
65.8
67.9
66.9
VLP-MABSA M2DF
67.0410.20
08.3
67.6:10.18
67.9
68.8=0.37
68
08.11.118
Table 9: Test results on the TWITTER- 15 and TWITTER-2 17 datasets for the JMASA task (%). The results of text-based methods (i.e., SPAN, D-GCN, and BART) are retrieved from (Ling et al., 2022).
TWITTER-15
TWITTER-17
Methods
Pre
Rec
Pre
Rec
F1
VLP-MABSA M2DF (Ours)
67.0+0.20
68.3+0.26
67.6-0.18
67.9+0.10
68.8+0.37
68.3+0.18
VLP-MABSA M2DF (ViLBERT)
66.6.10.09
67.910.25
67.2=0.11
67.3:0.29
68.211013
67.710.17
VLP-MABSA M2DF (Faster-RCNN)
66.9+0040
68.1+0.18
67.5=0.11
67.7++0.18
68.6++0.18
68.1#0.09
Table 10: Effect of different image encoders (%).
lower than that of VLP-MABSA M2DF (Ours), which indicates that Mask-RCNN is better sion for object detection in the MABSA task. Cer- tainly, Faster-RCNN also a good choice.
Training Time
Figure lists the training times for VLP-MABSA and VLP-MABSA+M2DF on the JMASA task.
D Error Analysis
We randomly sample 100 error cases of VLP- MABSA + M2DF in the JMASA task, and then divide them into three error categories. Figure shows the proportions and some representative ex- amples for each category. The top category an- notation bias. As shown in Figure 5(a), the senti- ment of aspect term "Orange Carpet" annotated as "NEU" (Neutral), but it actually expresses the sentiment of "POS" (Positive). This type of error presents a significant challenge for our model to provide accurate predictions. The second category is lacking background knowledge. Figure 5(b), the aspect term "Orange the New Black" refers to a well-known comedy, the model typically can only extract part of the aspect term without the help of background knowledge. The third category is insufficient information. As shown in Figure 5(c), both the sentence and the image content are quite simple, which is unable to provide sufficient infor- mation for our model accurately identify aspect terms and their corresponding sentiment polarities.
There need for the development of more advanced natural language processing techniques to address the above problems.

---

(a)
(b)
(c)
Image
Who is at the Orange Carpet tonight ? Excited for the premier the Story Content
RT @ CollegeHumor This is how they up with the name Orange the Black
mnabih123 the Alexandria wall demolished
Text
Golden
(Orange Carpet, NEU)
(Orange is the New Black, NEU)
(Alexandria wall, NEG)
Error Type
Annotation Bias
Lacking Background Knowledge
Insufficient Information
Proportions
46%
15%
18%
VLP-MABSA M2DF
(Orange, NEU) (New Black, NEU)
(Orange Carpet, POS)
(Alexandria, NEU)
Figure 5: Three typical errors of VLP-MABSA + M2DE, POS: Positive, NEU: Neutral, NEG: Negative,


Reviews:
[{"review_id": "pRr89BE45H", "reviewer": "Reviewer_ANSt", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your appreciation of our well-organized paper, novel method, and better results. We answer your concerns as follows:\n\n__Q1: Comparison with other threshold-based data filtering methods.__\n\n__A1:__ Thanks for your valuable comment. Following your suggestion, we added two threshold-based data filtering methods (RDS[1] and ITM[2]) as baselines and extended our framework to them. Among them, RDS is designed for information extraction tasks, and ITM is proposed for classification tasks, so we conducted experiments on JMASA and MASC tasks respectively. To be specific, we replaced the threshold-based data filtering module in RDS and ITM with our M2DF framework and named them as RDS(M2DF) and ITM(M2DF).\n\nFor the JMASA task:\n\nMethods | TWITTER-15(Pre, Rec, F1) | TWITTER-17(Pre, Rec, F1) |\n| :--- | :----: | :----: |\nRDS | 60.8 61.7 61.2 | 61.8 62.9 62.3 |\nRDS(M2DF) | 61.2 63.0 62.1 | 62.4 63.6 63.0 |\n\nFor the MASC task:\n\nMethods | TWITTER-15(Acc, F1) | TWITTER-17(Acc, F1) |\n| :--- | :----: | :----: |\nITM | 78.3 74.2 | 72.6 72.0 |\nITM(M2DF) | 78.9 75.0 | 73.2 73.0 |\n\nCompared to the baselines RDS and ITM, RDS(M2DF) and ITM(M2DF) show consistent improvements. These results again demonstrate that the denoising effect of M2DF is better than that of the threshold-based data filtering module. We will incorporate the above results into the revision. Thanks for your suggestion again.\n\n[1] Different Data, Different Modalities! Reinforced Data Splitting for Effective Multimodal Information Extraction from Social Media Posts. COLING 2022.\n\n[2] Targeted Multimodal Sentiment Classification based on Image-Target Matching. IJCAI 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well organized and easy to follow.\n2. The proposed method is novel. They propose an effective approach to leverage all useful data.\n3. The extensive results show the proposed method can be easily combined with existing methods and obtain better results.\n", "reasons_to_reject": "Lack of comparison with other threhold-based data filtering methods ", "questions_to_authors": "", "paper_topic_and_main_contributions": "Problem: There are many noisy images unrelated to the text in the dataset, which will have a negative impact on model learning.\nSolution: The negative impact of noisy images can be reduced without filtering the data.\nContributions: They propose a Multi-grained Multi-curriculum Denoising Framework (M2DF), which can achieve denoising by adjusting the order of training data. They conduct extensive experiments on multiple datasets, and the results demonstrate the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UpXYZMklO6", "reviewer": "Reviewer_sBBK", "summary": "", "questions": "1. Did authors try other noise metrics to evaluate the degree of noisy images? How is the performance like?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your appreciation of our first attempt to use curriculum learning for denoising in the MABSA task. About the mentioned questions, our answers are as follows:\n\n__Q1: Did authors try other noise metrics to evaluate the degree of noisy images? How is the performance like?__\n\n__A1:__ Yes, we also tried a naïve method to evaluate the degree of noisy images in the pre-experiments. This method uses the absolute value of the difference between the number of aspects in a sentence and the number of visual objects in an image to measure image noises. For simplicity, we call it noise_metric_diff. The corresponding performance on the JMASA task is below:\n\nMethods | TWITTER-15(Pre, Rec, F1) | TWITTER-17(Pre, Rec, F1) |\n| :--- | :----: | :----: |\nVLP-MABSA | 64.1 68.6 66.3 | 65.8 67.9 66.9 |\nVLP-MABSA + noise_metric_diff | 67.2 66.4 66.8 | 67.6 67.0 67.3 |\nVLP-MABSA + M2DF | 67.0 68.3 67.6 | 67.9 68.8 68.3 |\n\nThe above results show noise_metric_diff brings some improvements, but it is still significantly inferior to our carefully designed framework M2DF.\n\n__Q2: Implementation details can be put into the appendix to leave some space for the analysis part. Case studies and error analysis are better to be put into the main content.__\n\n__A2:__ Thanks for your kind suggestion on structuring our paper. We will adjust the placement of implementation details, case studies, and error analysis to improve the overall presentation of our work in the revised version.\n\n\n__Q3: Authors should explain more on why we need curriculum learning in this work. Can we use other machine learning algorithms?__\n\n__A3:__ Thanks for your valuable feedback. As mentioned in the introduction, hard noise filtering using thresholds may lose some useful information. We use curriculum learning because it can achieve denoising without the need for data filtering. Specifically, it achieves denoising by adjusting the order of training data and reducing the exposure of noisy data during training. More theoretical analysis can be found in the paper \"Why curriculum learning work in big/noisy data: A theoretical perspective\"[3]. Up to now, we have not found other machine learning algorithms to achieve denoising without filtering the data.\n\n[3] Why curriculum learning work in big/noisy data: A theoretical perspective. Big Data & Information Analytics, 2016.\n\n__Q4: Readers might wonder whether other training schedules such as self-paced learning work.__\n\n__A4:__ Different from the two predefined task-specific noise metrics in M2DF, self-paced learning uses the training loss value as an automatic noise metric. To address your concern, we train the VLP-MABSA using self-paced learning on the JMASA task, and the results are as below:\n\nMethods | TWITTER-15(Pre, Rec, F1) | TWITTER-17(Pre, Rec, F1) |\n| :--- | :----: | :----: |\nVLP-MABSA | 64.1 68.6 66.3 | 65.8 67.9 66.9 |\nVLP-MABSA + self-paced learning | 65.8 67.1 66.5 | 65.9 68.5 67.2 |\nVLP-MABSA + M2DF | 67.0 68.3 67.6 | 67.9 68.8 68.3 |\n\nWe can observe that the results of VLP-MABSA + self-paced learning are slightly higher than VLP-MABSA, but far inferior to VLP-MABSA + M2DF. This is reasonable because self-paced learning adjusts the learning schedule only based on training loss, and training loss is difficult to reflect whether the image contains noise in the MABSA task. In contrast, our designed task-specific noise metrics in M2DF help reduce the exposure to noisy data and achieve better performance.\n\n__Q5: More training schedules should be involved as baselines in the experiments to ensure fair comparisons since this work is a new training schedule.__\n\n__A5:__ Thanks for your precious suggestion. Apart from the above self-paced learning mentioned in A4, we also tried cross-entropy based training schedules [4] (abbr. CL_CE). The results on the JMASA task are as follows:\n\nMethods | TWITTER-15(Pre, Rec, F1) | TWITTER-17(Pre, Rec, F1) |\n| :--- | :----: | :----: |\nVLP-MABSA | 64.1 68.6 66.3 | 65.8 67.9 66.9 |\nVLP-MABSA + CL_CE | 66.3 66.9 66.6 | 66.5 68.0 67.2 |\nVLP-MABSA + M2DF | 67.0 68.3 67.6 | 67.9 68.8 68.3 |\n\nWe can observe that the results of VLP-MABSA + CL_CE are inferior to VLP-MABSA + M2DF. The reasons are similar to self-paced learning, i.e., the value of the cross entropy is difficult to decide whether the image contains noise in the MABSA task. We will incorporate these results into our revision. Thanks for your comment again.\n\n[4] Uncertainty-aware curriculum learning for neural machine translation. ACL 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work is supposed to be the first attempt to use curriculum learning for denoising in the MABSA task.\n2. Several lines of recent work are included in the baselines and the results show the proposed method can improve the performance.", "reasons_to_reject": "1. This paper is not well-organized. Implementation details can be put into the appendix to leave some space for the analysis part. Case study and error analysis are better to be put into the main content.\n2. The motivation is not clear. Authors should explain more on why we need curriculum learning in this work. Can we use other machine learning algorithms? Also, this paper only show curriculum learning schedule and readers might wonder whether other training schedules such as self-paced learning work. \n3. More training schedules should be involved as baselines in the experiments to ensure fair comparisons, since this work is a new training schedule.", "questions_to_authors": "1. Did authors try other noise metrics to evaluate the degree of noisy images? How is the performance like?", "paper_topic_and_main_contributions": "This paper introduces a Multi-grained Multi-curriculum Denoising Framework (M2DF) to tackle the problem of Multimodal Aspect-based Sentiment Analysis (MABSA). They focus on whether the negative impact of noisy images can be reduced without filtering the data. Extensive experimental results show that the proposed framework consistently outperforms state-of-the-art work on three sub-tasks of MABSA.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CoilsW85aU", "reviewer": "Reviewer_Szeq", "summary": "", "questions": "Question A: The proposed framework is composed of several neural networks, such as CLIP for multimodal feature extraction and Mask-RCNN for object detection. Is the current framework the best version? Have you tried any other neural models for these purposes? \nQuestion B: As this paper argues for the overestimated importance of image information, you should also compare the framework with pure textual ABSA models to prove that the image information is still useful. \n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your appreciation of our new perspective, novel idea, and extensive experiments. We address your concerns as follows:\n\n__Q1: The proposed framework is composed of several neural networks, such as CLIP for multimodal feature extraction and Mask-RCNN for object detection. Is the current framework the best version? Have you tried any other neural models for these purposes?__\n\n__A1:__ We utilized CLIP for multimodal feature extraction, due to its training on a vast collection of text-image pairs. This allows it to excel in capturing multimodal semantics better. Besides, we also tried using ViLBERT[5] to extract multimodal features. For object detection, we tried using Faster-RCNN to replace Mask-RCNN. The results of these two methods on the JMASA task are as follows:\n\nMethods | TWITTER-15(Pre, Rec, F1) | TWITTER-17(Pre, Rec, F1) |\n| :--- | :----: | :----: |\nVLP-MABSA + M2DF(Ours) | 67.0 68.3 67.6 | 67.9 68.8 68.3 |\nVLP-MABSA + M2DF(ViLBERT) | 66.6 67.9 67.2 | 67.3 68.2 67.7 |\nVLP-MABSA + M2DF(Faster-RCNN) | 66.9 68.1 67.5 | 67.7 68.6 68.1 |\n\nWe can observe that the F1-score of VLP-MABSA + M2DF(ViLBERT) is lower than that of VLP-MABSA + M2DF(Ours). This is because CLIP benefits from a larger-scale and more diverse training dataset, which enables the model to capture multimodal features more accurately. Thus, CLIP is a better version to extract multimodal features. Moreover, the F1-score of VLP-MABSA + M2DF(Faster-RCNN) is slightly lower than that of VLP-MABSA + M2DF(Ours), which indicates that Mask-RCNN is a better version for object detection in the MABSA task. Certainly, Faster-RCNN is also a good choice.\n\n[5] ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. NeurIPS 2019.\n\n\n__Q2: As this paper argues for the overestimated importance of image information, you should also compare the framework with pure textual ABSA models to prove that the image information is still useful.__\n\n__A2:__ Thanks for your constructive suggestion. Due to space constraints, we didn't report the previous text-based ABSA methods in the main results. In the following, we retrieve the results of text-based methods (i.e., SPAN, D-GCN, and BART) from [6] for the JMASA task.\n\nMethods | TWITTER-15(Pre, Rec, F1) | TWITTER-17(Pre, Rec, F1) |\n| :--- | :----: | :----: |\nSPAN | 53.7 53.9 53.8 | 59.6 61.7 60.6 |\nD-GCN | 58.3 58.8 58.5 | 64.2 64.1 64.1 |\nBART | 62.9 65.0 63.9 | 65.2 65.6 65.4 |\nVLP-MABSA | 64.1 68.6 66.3 | 65.8 67.9 66.9 |\nVLP-MABSA + M2DF | 67.0 68.3 67.6 | 67.9 68.8 68.3 |\n\nIt is clear that multimodal methods VLP-MABSA and VLP-MABSA + M2DF both outperform all the text-based methods, which indicates that the image information is useful for MABSA. We will add these results to our revision. Thanks for your suggestion again.\n\n[6] Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis. ACL 2022.\n\n\n__Q3: For the JMASA task, VLP-MABSA + M2DF performs worse than the original VLP-MABSA model in terms of recall. What are the potential drawbacks of the proposed framework that lead to this result?__\n\n__A3:__ Thanks for your valuable comments. Compared to VLP-MABSA, VLP-MABSA + M2DF achieves a significant improvement in the precision, but at the same time, the recall has a certain degree of sacrifice, that is, the recall is slightly decreased or the improvement is not as obvious as the precision. This phenomenon might stem from M2DF's heightened stringency in prediction generation within the sequence-to-sequence framework of VLP-MABSA. M2DF tends to generate accurate predictions with high confidence, resulting in a substantial enhancement in precision by reducing the number of erroneous predictions. However, the heightened stringency in prediction generation by M2DF may inadvertently lead to the omission of some true instances, thereby resulting in a certain degree of sacrifice in recall. But overall, the F1-score achieves competitive results.\n\n\n__Q4: About typos.__\n\n__A4:__ Thanks for your careful review. We will fix these typos in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces a novel idea to denoise image data through reordering training samples. The noise of the image is measured comprehensively, from both the sentence-level and aspect-level. Experiments prove the effectiveness of the proposed framework. ", "reasons_to_reject": "1. For the JMASA task, the improvement of adding the M2DF framework is very little, and VLP-MABSA + M2DF performs even worse than the original VLP-MABSA model in terms of recall. What are the potential drawbacks of the proposed framework that lead to this result? ", "questions_to_authors": "Question A: The proposed framework is composed of several neural networks, such as CLIP for multimodal feature extraction and Mask-RCNN for object detection. Is the current framework the best version? Have you tried any other neural models for these purposes? \nQuestion B: As this paper argues for the overestimated importance of image information, you should also compare the framework with pure textual ABSA models to prove that the image information is still useful. \n", "paper_topic_and_main_contributions": "This paper targets the Multimodal Aspect-based Sentiment Analysis task. One of the problems with the current studies of the task is that the importance of image information is overrated and noisy images often bring a negative impact on the sentiment analysis result. Therefore, the authors propose a novel Multi-grained Multi-curriculum Denoising Framework inspired by Curriculum Learning to denoise training data by reordering data samples to train more times on clean data. \nThe main contributions of the paper are:\n1. Proposing a new perspective on MASBA noise reduction. \n2. Proposing a novel Multimodal Aspect-base Sentiment Analysis. \n3. Conducting extensive experiments to evaluate the model. \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In section 3.1 paragraph1, the degree of noisy images => The degree of image noise \n2. The tables should not contain any vertical lines. \n3. ...\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found the paper to be strong/good across all metrics. All reviewers highlighted the novelty of the work. Reviewers were mixed on the quality of the writing. Two reviewers noted the paper lacks comparisons with other methods.
Decision: Accept-Main