Visual Elements Mining as Prompts for Instruction Learning for Target-Oriented Multimodal Sentiment Classification
Authors: Bin Yang
Jinlong Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Target-oriented multimodal sentiment classification
Instruction learning

Abstract: Target-oriented Multimodal Sentiment Classification (TMSC) aims to incorporate visual modality with text modality to identify the sentiment polarity towards a specific target within a sentence. To address this task, we propose a Visual Elements Mining as Prompts (VEMP) method, which describes the semantic information of visual elements with Text Symbols Embedded in the Image (TSEI), Target-aware Adjective-Noun Pairs (TANPs) and image scene caption, and then transform them into prompts for instruction learning of the model Tk-Instruct. In our VEMP, the text symbols embedded in the image may contain the textual descriptions of fine-grained visual elements, and are extracted as input TSEI; we extract adjective-noun pairs from the image and align them with the target to obtain TANPs, in which the adjectives provide emotional embellishments for the relevant target; finally, to effectively fuse these visual elements with text modality for sentiment prediction, we integrate them to construct instruction prompts for instruction-tuning Tk-Instruct which possesses powerful learning capabilities under instructions. Extensive experimental results show that our method achieves state-of-the-art performance on two benchmark datasets. And further analysis demonstrates the effectiveness of each component of our method.

Visual Elements Mining as Prompts for Instruction Learning for Target-Oriented Multimodal Sentiment Classification
Bin Yang and Jinlong Li University of Science and Technology of China jcbxêmail www..stcc..... edu. cn, jlli@ustc. edu. cn
Abstract
MOST POWNT WI MITHOUT PANALS
Target-oriented Multimodal Sentiment Classi- fication (TMSC) aims to incorporate visual modality with text modality to identify the sen- timent polarity towards specific target within sentence. To address this task, we propose Visual Elements Mining as Prompts (VEMP) method, which describes the semantic infor- mation of visual elements with Text Sym- bols Embedded in the Image (TSEI), Target- aware Adjective-Noun Pairs (TANPs) and im- age scene caption, and then transform them into prompts for instruction learning of the model Tk-Instruct. In our VEMP, the text sym- bols embedded in the image may contain the textual descriptions of mne-grained visual ele- ments, and are extracted imput TSEI; tract adjective-noun pairs from the image and align them with the target to obtain TANPs, in which the adjectives provide emotional em- bellishments for the relevant target; finally, effectively fuse these visual elements with text modality for sentiment prediction, we in- tegrate them to construct instruction prompts for instruction-tuning Ik-Instruct which pos- sesses powerful learning capabilities under instructions. Extensive experimental results show that method achieves state-of-the-an performance on two benchmark datasets. And further analysis demonstrates the effectiveness of each component of our method!.
BRING YOUR BROOMS
BECAUSE IT'S MESS
[Cleveland Cavallers Cavallers
Marty Inggative
[Kevin Durant]posttive impressive list Bureau
Sentence: Warriors sweep
Sports
nbaaiinals.
Image
giving
group players standing next to each other.
Scene Caption: speech.
most points without turoover finals (since1970)
TSEI:
because
ANPs:
angry horizontal
guy, healthy teeth...
Two examples of the TMSC task. TSEI: text symbols embedded in the image. ANPs: adjective- noun pairs extracted from the image.
Figure 1:
With the boom of multimodal multimedia (c.g., tweets) on the Internet, TMSC has been valuable issue and has aroused wave of research enthusi- Early work (Yu and Jiang, 2019; Wang al., 2021: Zhang et al., 2021) focused on utilizing tention mechanism to fuse text features and visual features. Recently, Khan and Fu (2021) translated the image into textual image scene caption, which aligned different modalities with heightened inter- pretability since the image is represented in natu- ral language. We concentrate on this new research formula to project visual modality text modality for moosl fusion.
1 Introduction
Target-oriented Multimodal Sentiment Classifica- tion (TMSC) is fine-gra grained sentiment analysis task towards multimodal data, which aims clas- sify the sentiment polarity towards the given opin- ion target (a word or phrase) sentence-image pair. For example, we should infer that the senti- ment polarity towards the target term "Cleveland Cavaliers" " is negative with the provided sentence and image in Figure 1(a).
However, how to adequately mine visual ele- ments to get a comprehensive utilization of visual semantic information remains an unsolved chal- lenge. For instance, image scene caption has two deficiencies: (1) Generally, the image scene cap- tion lacks detailed depiction of the image, which means information loss during the image-to-text translation process. As shown in Figure/ 1, the image scene captions are just coarse overviews scenes in images. (2) It mostly involves neutral vo-
OOu code is publicly available at https: thub com/ long81 81 /VEMP.

---

cabulary and lacks explicit affective guidance for sentiment prediction. As depicted in Figure 1, the image scene captions are objective narratives with- out emotional characterization. To handle these problems, Yang et al. (2022) tried to introduce fa- cial emotion yet their method is constrained to im- ages with facial expressions and thereby lacks gen- cralizability. Therefore, we strive for more univer- sal and efficient solution and propose Visual Ele- ments Mining as Prompts (VEMP) method.
struction learning across variety of tasks (Zhang and Chai, 2021; Mishra et al., 2022: Ouyang al., 2022). Our instruction prompt consists of a task definition, three examples with "positive", "nega- tive" " and "neutral" sentiment labels respectively, and a to-be-classified sample.
Our contributions are as follows:
(1) We propose Visual Elements Mining as Prompts (VEMP) model, which mines TSEI, TANPs and image scene caption as textual descrip- tions the image, leading to a more comprehen- sive understanding of visual modality. To the best of our knowledge, we are the first to take TSEI into consideration for TMSC. Moreover, VEMP is applicable to all types of images and exhibits uni- versality.
Firstly, we extract Text Symbols Embedded in the Image (TSEI) to complement image scene cap- tion with fine-grained details. We notice that in the datasets Twitter-2015 and Twitter-2017 (Zhang et al., 2018; Lu et al., 2018) more than 50% of sam- ples' corresponding images contain TSEI, which can contribute ure 1, the TSEI "because it's a mess" are piv- otal to judge the "negative" sentiment polarity to- wards the target "Cleveland Cavaliers" while the TSEI "most points without a turnover in nba fi- nals (iinee977)) are crucial to infer the "positive" sentiment polarity towards to the target "Kevin Durant". Therefore, in VEMP, we attach importance TSEI and employ Optical Charac- ter Recognition (OCR) technique to extract them. order to denoise the extracted results for im- proved usability, we have executed elaborate post- processing refinements, such as spelling correc- tion, grammar filtering, etc.
TMSC. As delineated in Fig-
(2) Our VEMP adopt the instruction learn- ing paradigm and achieves state-of-the-art perfor- mance on benchmark datasets Twitter-2015 and 101tter-2017, which demonstrates the superiority of our method.
(3) We conduct experiments GPT-4 with zero-shot and few-shot settings to explore its po- tential for TMSC.
Related Work
2.1 Target-oriented Multimodal Sentiment Classification
Target-oriented Multimodal Sentiment Classifica- tion (TMSC), an extension task of Aspect-Based Sentiment Analysis (ABSA) tailored for multi- modal data, has attracted extensive attention lately due to the increase of multimodal information on the Web. Xu et al. (2019) and Yu et al. (2019) applied attention mechanism based on LSTM for cross-modal fusion. Yu and Jiang (2019), Wang et al. (2021) and Zhang et al. (2021) further chose BERT- architecture to model the cross- modal interactions. Khan and Fu (2021) ere- atively proposed a novel method to translate the image into textual image scene caption, which ef- fectively aligns multimodal features from differ- ent feature spaces. However, image scene cap- tion lacks detailed and emotional depiction of the image. To address that, Yang et al. (2022) cap- tured facial emotion from the image, which is lim- ited images containing facial expressions. Zhao et al. (2022) proposed a knowledge-enhanced framework (KEF) which employed adjective-nour pairs (ANPs) from the image to assist attention- based models. Ling et al. (2022) proposed a
Secondly, design a Target-aware Adjective- Noun Pairs (TANPs) extraction subtask capture fine-grained and emotional visual content. Ex- isting research has demonstrated the efficacy of ANPs in TMSC (Zhao et al., 2022), which directly employed ANPs to assist attention-base modal features fusion. From another perspective, depicted in Figure 1, we observe that the nouns are detailed objects in images and the adjectives imply emotional tendencies, which fill the above two gaps of image scene caption. Thus, we ex- tract ANPs from the image as supplements to im- age scene caption. Besides, we align ANPs with the target to obtain TANPs since ANPs relevant to the target are helpful while the others introduce noises.
cross-
Finally, utilize TSEI, TANPs, image scene caption and the sentence in the sentence- image pair to construct instruction prompts for instruction-tuming the model Tk-Instruct (Wang et al., 2022b), inspired by previous success of in-

---

Vision-Language Pre-training framework (VLP- MABSA) and trained on three types of task- specific pre-training tasks. Yu et al. (2022) intro- duced a multi-task learning architecture to capture both mmarse-grained and fine- tne-grained image-target matching relations.
3.3 TSEI Extraction and Refinement
In this module, we extract TSEI from the image and refine them to reduce noises.
Extraction with Maximum Area Strategy. We utilize well well-performing OCR model Easy- OCR² to recognize TSEI from the image V and select the primary TSEI from the region with the maximum area. The distribution of TSEI in V can be divided into multiple bounding boxes. TSEI from different bounding boxes are usually seman- tically incoherent if they are spliced together. Be- sides, TSEI from small bounding boxes are often too brief to provide valuable information, and in- stead add noises. Therefore, we concentrate on the primary content and select the recognized TSEI from the bounding box with the maximum area:
2.2 Instruction Learning
Recently, we have witnessed remarkable progress in enhancing the cross-task generalization of mod- els via instructions, such as FLAN (Wei et al., 2022), PromptSource (Sanh et al., 2022), Instruct- GPT (Ouyang et al., 2022) and Tk-Instruct (Wang et al., 2022b). Furthermore, myriad of ap- proaches utilizing instructions to handle domain- specific tasks have been proposed (Kuznia al., 2022; Luo al., 2022; Varia et al., 2022; Scaria et al., 2023; Wang et al., 2022a).
B, = (X.,, X,,, Yv,, Y2,, R,), 1,
(1)
3 Method
A X Xx,ol | - Yool
(2)
3.1 Task Formalization
O' Rg. where j arg max Ai E......!
(3)
Given set of multimodal samples r, ple Y E r is tuple V, where S {wI,..., wi..... 1/1 -........... w } is a wSI-word sentence, V an image, T = { (w,,..., -1 C S denotes word tar- get term, and E (positive, negative, neutral) indicates the sentiment polarity towards the target term. Our goal is to learn function to predict the sentiment polarity y.
where B, denotes the in. bounding box; X, X Yuo and Y, are the borderline coordinates of the bounding box; R, is the recognized TSEI; Ar de- notes the of the bounding box; and O' indi- cates the selected TSEI.
Spelling Correction. Due the limitation of the image quality and EasyOCR, there will be some errors O' such as misrecognition of char- acters and inappropriate spaces. Thus, we employ Symmetric Delete spelling correction algorithm' to correct and modify O':
3.2 Overview
As shown in Figure 2, our model consists of four modules: TSEI Extraction and Refinement module, Image Scene Caption Generation module, TANPs Extraction and Alignment module, and Instruction Learning with Prompts module. Given multi- modal input (S, V)T where S denotes the sen- tence and V indicates the image, in the first three modules, we extract TSEI, image scene caption and TANPs from V in natural language format respectively, which preserve both coarse-grained and fine-grained descriptions of V and thus har- vest sufficient comprehension of visual seman- tic information. In the fourth module, we in- tegrate the extracted TSEI, image scene caption and TANPs with S manually design instruction prompts for instruction-tuning Tk-Instruct, which generates the sentiment prediction the output. We will demonstrate the model details the fol- lowing subsections.
(4) Grammar Filtering. We feed O" into the RoBERTA fine-tuned on the CoLA dataset (Warstadt et al., 2018) judge the grammatical ac- ceptability of o" We filter out those with ungram- matical confidence above the threshold 01 0.9.
o" SymSpell((')
The reserved O" the resulted TSEI.
H ROBERTT(O"))
(5)
J - ( (CLS]] 0"
bb)
(6)
where W C 768x2 and b C R22
are parameters
learned in fine-tuning; and HI CLS)
E B is the
o" final hidden state of the [CL.S] token.
2 https://wwwwjaiieed ai/easyocr/ https://github.b com/ wwwp//aeeaaggaaeeSSsllllllll

---

TSEI Extraction and Refinement
Discard
Bounding Boxes
Recogeition
Strategy
No
Spelling Correction
RoBERTa
Image Scene Caption Generation
Instruction Learning with Prompts
Text Detection
Generation
What the photo of?
Tn-instruct
Image
Encoder
BLP-2
Intruction Prompt
Definition
Task
Eicample
Powttive
Regativv
Eicampe
Exomple
Nenarrll
Sample
group payeess..
Image Scote Caption
Burcsu larget: Kevin Durant, Soone desoription the image: group of baskedball players standing next each other, Adjective noun par extracted from image: guy contamea
TANPs Extraction and Alignment
Top-5
guy is
Selection
taeth heality
CLPP
guy nice
TANPs
Concut
Kevin Durand joins imprassiva llet, Via
Targat
Kevin Burani
Sentance
Eluo Sports Bureau )
Figure 2: The architecture of our VEMP.
3.4 Image Scene Caption Generation
3.5 TANPs Extraction and Alignment
We upply state-of state-of-the-ar vision- languaee pre- trained model BLIP-2 (Li et al., 2023) to generate textual image scene caption of the image V with the following procedures. First, we encode the image with frozen im- age encoder to get the embeddings:
We extract ANPs from the image V and conduct alignment with the target T to obtain TANPs.
ANPs Extraction. We follow KEF to use Sen- tiBnk toolkit' extract ANPs with the Top-5 confidence. And we rewrite these ANPs as "Noun is/are Adjective".
Hv
(7)
Alignment with Target. We utilize the model CLIP (Radford et al., 2021) evaluate the rele- vance between ANPs and the target for alignment. We concatenate ANPs with the target and feed them into the text encoder of CLIP, while input the image into the image encoder CLIP. Ben- efiting from pre-training on a large-scale dataset of text-image pairs, CLIP is able to work out the similarity score, which is subsequently passed to softmax layer to caleulate the matching proba- bility. We select ANPs with matching probability exceeding the threshold 6g = 0.5.
Then, we interact a set of learnable query em- beddings as soft prompts with Hy to extract language-informative visual representation Hy in a querying transformer:
Hy ( Hv))
(8)
where Q denotes the query embeddings; Aelf-ATT denotes self-attention layer; Cross-ATT denotes cross-attention layer; and FF denotes feed forward layer.
Next, a fully-connected (FC) layer projects Hyy into the same hidden dimension of large lan- guage model (LLM))
M = Softmax(CI.P((onnaa) (ottooncat(AA)))))) ()) (11)
AT = Ai, where M, > O2 i 1, 1,...,5
H² = FC
(9)
(12)
Finally, we pass the generation prompt P along with Hy to the LLM generate the image scene caption:
where A denotes ANPs; M E R denotes the matching probability; and Ay >4 indicates the re- sulted TANPs.
C - LLMM-Decdde(HHfi P)
(10)

---

Twitter-2015
Twitter-2017
Split
Pos
Neu
Neg
Total
Targets
Len
Pos
Neu
Neg
Total
Targets
Len
Train
928
1883
368
3179
1.34
16.72
1508
416
1638
3562
1.41
16.21
Dev.
303
679
149
1122
1.33
16.74
515
144
517
1176
1.43
16.37
Test
317
607
113
1037
1.35
17.05
493
168
573
1234
1.45
16.38
Table 1: The statistics of two benchmark datasets. Pos: Positive, Neu: Neutral, Neg: Negative, Len: Length.
Twitter-TSEI
cuses on fine-grained targets, we have at least two examples that share the same input but with differ- targets and sentiment labels. Lastly, we choose samples with acknowledged and uncontroversial sentiment labels. The more detailed instruction
Split
Pos
Neu
Neg
Total
Targets
Len
Train
1114
1568
347
3029
1.38
16.42
Dev.
393
552
131
1076
1.39
16.53
Test
358
527
133
1018
1.41
16.67
prompts example is shown in Appendix A. We use the following loss for instruction- tuning:
Table 2:
The statistics Twitter-TSEl dataset. Pos: Positive, Neu: Neutral, Neg: Negative, Len: Length.
3.6 Instruction Learning with Prompts
L
logro(y'lx")
(13)
this module, we aim to fuse the sentence S with the visual elements mined from the image V for modeling. To this end, we leverage the sentence S, the target T, TSEI, TANPs and image scene cap- tion to manually design instruction prompts, and then apply these instruction prompts to instruction- tune In-Instruct for generating the sentiment pre- diction.
where x² denotes the input instruction prompts se- quence; the model parameters; and y² indi- cates the output sentiment prediction token.
Experiment
To validate the advantuge of our method, com- pare it with state-of-the-art models and GPT-4, And we perform ablation study and case study to verify the effectiveness of each module of our method.
Appropriate prompts essential for stimu- lating the learning potential of language models (LMs). Empirically, based on the prior success (Scaria et al., 2023), we make an effort to en- sure the instruction prompts are concise, explicit, and easily comprehensible. Specifically, each of our instruction prompts comprises a task defini- tion, three examples of "positive", "negative" and "neutral" sentiment labels respectively, and to- be-classified sample. The format of every example sample as "Sentence: S. Target: T. Scene description of the image: C. Adjective-noun pair extracted from the image: AT-AA Text contained in the image: O"", where C, AT-A and O" de- note image scene caption, TANPs and TSEI re- spectively. We employ specific criteria for the se- lection of our few-shot examples. Firstly, "pos- itive". "negative", and "neutral" sentiment ples should all be included. Given the constraint imposed by Tk-Instruct's maximum input tokens limit and the lengthy nature of our examples, we opt for I-shot examples for each sentiment cate- gory. Secondly, since our classification task fo-
4.1 Experimental Setup
Datasets. We evaluate our method on two bench- mark datasets Twitter-2015 and Twitter-2017. Be- sides, we select samples with TSEI from Twitter- 2015 and Twitter-2017 to build a new dataset Twitter- TSEI to check the effect of our method more complex dataset. The statisties of Twitter- 2015, Twitter-2017 and Twitter-TSED are shown Table and Table 2.
Implementation Details. We employ the model Tk--Instruct-base-def-pss as backbone, and we set the batch size, the learning rate and the training epochs as 16, 5e-5 and 30 respectively. All instruction-tunm tuning experiments are based on PyTorch with NVIDIA GeForce RTX 3090 GPU.
Evaluation Metrics We use Accuracy (Acc) and Macro-FI as evaluation metrics, and report the average of 5 independent training runs as results.

---

Twitter-2015
Twitter-2017
trained language model which is equipped with strong learning capabilities to unseen tasks due rigorous training following instructions.
Method
Acc
Macro-F1
Acc
Macro-11
Visual Only
Res-Target
59.88
46.48
58.59
53.98
Text and Visual. (1) MIMN (Xu et al., 2019),
Text Only
Multi-Interactive Memory Network which learns both the cross-modality and the single-modality interactions. (2) ESAFN (Yu et al., 2019), an Entity- Sensitive Attention and Fusion Network. (3) TomBERT (Yu and Jiang, 2019), which applys BERT and attention mechanism to model intra- modality dynamics and inter-modality dynamics. (4) CF-CapTrBERT (Khan and Fu, 2021), which translates images into auxiliary sentences and then fuse with text-modality through a pre-trained lan- guage model. (5) FITE (Yang al., 2022), which translates facial expressions in images into emo- tional texts for fusion with text-modality. (6) KEF- TomBERT (Zhao et al., 2022), which combines a knowledge-enhancee framework with TomBERT to improve the capability of visual attention and sentiment prediction. (7) VLP-MABSA (Ling et al., 2022), which adopts task-specific vision- language pre-training. (8) ITM (Yu al., 2022). a multi-task learning architecture which captures both coarse- coarse-grained and fine-grained image-target matching relations.
AE-LSTM
70.30
63.43
61.67
$7.97
MGAN
71.17
64.21
64.75
61.46
BERT
74.15
68.86
68.15
65.23
Tk-Instruct
77.53
71.88
71.07
69.66
Text and Visual
MIMN
71.84
65.69
65.88
62.99
ESAFN
73.38
67.37
67.83
64.22
TOMBERT
77.15
71.75
70.50
68.04
EF-CapTrBERT
78.01
73.25
69.77
68.42
FITE
78.49
73.90
70.90
68.70
KEF-TOMBERY
78.68
73.75
72.
69.96
VLP-MABSA
78.60
73.80
73.80
71.80
ITM
78.27
74.19
72.61
71.97
Our Method
VEMP
78.88
75.09
73.01
72.42
Table 3: Experimental results for TMSC.
Twitter-TSEI
Method
Acc
Macro-F1
Text Only
Tk-Instruct
67.68
64.23
Text and Visual
4.3
Experimental Results and Analysis
ITM
68.86
65.59
In the subsection, we compare our method with other baselines on Twitter-2015, Twitter-2017 and Twitter- TSEI, and provide empirical analysis.
VEMP
71.72(42.86%) 68.61(+3.02555
Table 4: Experimental results on Twitter-TSE. dataset.
Based on the experimental results on Twitter- 2015 and Twitter-2017 datasets shown in Table 3, our observations are as follows: (1) Compared to Tr-CATTTBERTT and FITE, which also translate vi- sual modality into text modality for cross-modal fusion, our VEMP outweighs EF-CaTTTBBRTT by 1.84% and 4.00%, and surpasses FITE by 1.19% and 3.72%, on the Macro-Fl score on the Twitter- 2015 dataset and the Twitter-2017 dataset respec- tively. Besides, our VEMP also outperforms the state-of-the-art model ITM. These advance- ments corroborate the remarkable superiority of our method. (2) Insttuct outperforms all other text-only methods and even surpasses multimodal method TomBERT, which demonstrates its excel- lent learning capabilities under instructions. (3) Our VEMP improves text-only Tk-Instruct on the Macro-F1 score by 3.21% % on the Twitter-2015 datasct, and 2.76% on the Twitter-2017 dataset, which demonstrate the effectiveness our VEMP
4.2
Compared Baselines
We compare our method with three kinds of base- lines follows:
Visual-Only. Res-Target, which concatenates the target embeddings and the visual features ex- tracted by ResNet (He et al., 2016) for classifica- tion.
Text-Only. (1) AE-LSTM (Wang et al., 2016), an attention-based LSTM which utilizes attention mechanism to concentrate on the aspect in sen- tence. (2) MGAN (Fan et al., 2018), Multi- Grained Attention Network which aims to alle- viate the information loss of coarse-grained at- tention mechanism. (3) BERT (Devlin et al., 2019), bidirectional pre-trained language model which has context understanding abilitics. (4) Tk- Instruct (Wang et al., 2022b), a generative pre-

---

Method
Acc
Twitter-2015
Macro-F1
Acc
Twitter-2017
Macro-F1
Method
Twitter-2015 Macro-FI
Twitter-0017
Macto-F1
VEMP
78,88
75.09
73.01
72.42
Text Only
w/o ISEI
77.82
73.70
71.56
70.77
(PT-t(cro-sbbt)
64.61
59.35
59.24
$7.60
wha Image Scene Caption
78.40
74.38
72.04
70.88
GPT-t(ew-shot)
65.19
59.88
59.97
98.73
w/ TANPs
78.21
74.86
71.56
70.20
(inn-i(in--
76.66
71.02
68.23
66.22
Tk- Instructiinstcciionnn tuning)
77.53
71.88
71,07
69.66
Text and Visual
Table 5: Ablation study of the three visual elements mining modules in our VEMP. "w/o" indicates the re- moval of the corresponding module.
VEMP (asatrrsso))
65.00
59.95
58. 36.29
65.77
61.19
60.62
59.48
VEMPorT Pert Sernaaotntteneuuiii
77.43
71.86
66.83
VEMP V Do (istraaoonn))))))nnnnnnng)
78,88
75.09
73.01
72.42
Mathod
Asc
Twitter-2015
Maemo-F
Ace
Twitter-0177
Macro-Fi
Table 7: Experimental results of different learning paradigms. For example, VEMPPar--(eero--0 indi- cates that we utilize VEMP to mine visual elements for zero-shot learning on GPT-4.
VEMP
78.88
75.19
73.81
72.42
We Spelling Courectinn
77:92
73.81
71.56
70.79
Filering
74.61
We Spelling Couretion Granmar Filtering
78.11
74.34
71.22
Wh Alignment with larget
78.01
74.25
71.88
70.59
Table 6: Ablation study of detailed units in our VEMP. "w/o" indicates the removal of the unit.
age into textual scene outline for cross-modal fu- sion. (3) Eliminating the TANPs Extraction and Alignment module results in performance degrada- tion as well. In particular, the Accuracy declines 1.45% and the Macor-F1 score drops 2.22% on the 1witter-2017 dataset. This illustrates the TANPs Extraction and Alignment module's contribution to distill TANPs for emotional guidance. (4) Re- markably, we notice that removing the TSEI Ex- traction and Refinement module leads to the most intense performance decay, which validates the ra- tionality of our motivation to leverage TSEI for TMSC.
in leveraging visual modality for training models. We also report the experimental results on Twitter-TSEI dataset in Table 4. We can find that VEMP achieves more significant improvement over ITM than Twitter-2015 and Twitter-2017. It is reasonable because ITM ignores TSEI while we attach importance to them in our VEMP. More- over, VEMP still outperforms the text-only Tk- Instruct since VEMP can effectively exploit visual information.
Detailed Units for Visual Elements Mining. We implement further ablation study to check the influence of more dctailed units in our VEMP. We remove the Spelling Correction unit, the Grammar Filtering unit, both Spelling Correction and Gram- mar Filtering units, and the Aligument with Target unit respectively.
4.4 Ablation Study
Three Modules for Visual Elements Mining, To investigate the impact of TSEI Extraction and Re- finement module, Image Scene Caption Genera- tion module and TANPs Extraction and Alignment module in our model, we remove them separately and observe the resulting consequences.
Based on the experimental results Table 6, we can observe that the performance downgrades on the Twitter-2015 dataset and the Twitter-2017 dataset after removing the Spelling Correction unit, the Grammar Filtering unit or the both of them, which confirms their indispensable impacts denoising the extracted TSEI. Besides, directly applying the ANPs without alignment with the tar- get also results in performance decline since the ANPs irrelevant to the target add noises.
As depicted in Table 5, we can find that: (1) The performance declines significantly after re- moving the TSEI Extraction and Refinement mod- ule. Specifically, the Accuracy and Macro-F1 score drop 1.06% and 1.39% on the Twitter-0015 dataset respectively, and the Accuracy and Macro- FI score drop 1.45% and 1.65% on the Twitter- 2017 dataset respectively. This indicates that the TSEI Extraction and Refinement module can ef- fectively yield TSEI as beneficial information for TMSC. (2) Without the Image Scene Caption Gen- eration module, the Accuracy and Macro-F1 score decline about 0.5% on the Twitter20115 dataset and about 1% on the Twitter-2017 dataset, which demonstrates the importance of the Image Scene Caption Generation module in translating the im-
Instruction Learning with Prompts Module. To evaluate the effectiveness of Instruction Learn- ing with Prompts module. we replace it with other typical learning paradigms. such as fine-tuning on GPT-3 (Brown al., 2020), zero-shot learning on GPT-4 and sew-shot learning on GPT-4. For fnn-ttunin GPT-3, We adopt the base model ada

---

IN NBA FINALS
BRING YOUR BROOMS
Image
BECAUSE IT'S MESS
Sentence
[Cleveland Cavaliers legunive Marty Huggins said it best I Warriors Cavs sweep nhufinals.
(b) [Kevin Durant positive joins impressive list ( Via Elias Sports Bureau
Me. along with couple of friends -- [James Franco posétive Abrams- on
22/ 63. Everything looks cnol.. None.
TSEI
because it's mess.
most points without a turnover in aba finals (since1978). group of basketball players standing next to each other. guy nice.
Image Scene Caption
in suit and tie giving speech.
Three men posing picture room.
TANPs
man amgry.
family happy.
Instuntrcct
positive X
positive negative X
acutral X positive / pusitive /
ITM
positive X
VEMP
negutive
positive /
Table 8: Predictions of text-only Tk- Tk-Instruct, ITM and VEMP three samples.
and set the batch size, the learning rate and the
4.5 Case Study
training epochs as 8, 0.05 and 4 respectively. All
To understand the advantage of our method intu- itively, we present the predictions of text-only Tk- Instruct, ITM and VEMP on three test samples, as shown in Table 8. For sample (a), with the aid of the TSEI "because it's mess" and the TANPs "man is angry", our VEMP made the correct pre- diction whereas other models fell into errors. For sample (b), ITM made wrong prediction since just focused on the neutral visual content ex- cluding TSEL. In contrast, VEMP exploited the TSEI "most points without a turnover in nba fi- nals (since1978) which conveyed a positive emo- tional tendency and thus gave the correct predic- tion. For sample (c), despite without TSEI in the image, VEMP still predicted the sentiment polar- ity correctly, benefiting from the TANPs "family is happy".
fine-tuning experiments are implemented on Ope- nAI servers remotely via OpenAl API. For zero- shot and few-shot learning on GPT-4', we de- sign four kinds of prompts tailored for text-only zero-shot setting, text-only few-shot setting, multi- modal zero-shot setting and multimodal few-shot setting respectively, which are adapted from the report (Zhang et al., 2023) that utilized large lan- guage models (LLMs) for sentiment analysis. To ensure a fair comparison, we employ the same few-shot examples instruction learning Tk- Instruct. Specific prompt examples are shown in Appendix A.
Table depicts the experimental results of different learning paradigms. We can observe that: (1) GPT-4 with zero-shot or few-shot set- ting performs poorly on TMSC. We conjecture the reasons are that TMSC concerns the fine- grained target and sentences in tweets are infor- mal, which increase the complexity of the task so that zero-shot and few-shot learning are incompe- tent. (2) Instruction-tuning Tn-Instrcc surpasses other learning paradigms by a wide margin on both text-only and multimodal data, which demon- strates the effectiveness of Instruction Learning with Prompts module. Furthermore, we can con- clude that instruction-tuning on language models is competitive and promising in the era of LLMss.
Conclusion
In this paper, we propose Visual Elements Mining as Prompts (VEMP) method for Target- Oriented Multimodal Sentiment Classification (TMSC). In order to describe the image with suf- ficient details and emotional characterization, we extract Text Symbols Embedded in the Image (TSEI) and Target-aware Adjective-Noun Pairs (TANPs) as supplements image scene caption. And then leverage them construct instruc- tion prompts to instruction-tune Tk-Instruct for sentiment prediction. Experimental results show that our method outperforms state-of-the-an ap-
"The experiments are based on the August 23 version of GPT-4, It is important to note that future updates to the model may potentially impact the results presented in this paper.

---

proaches and GPT-4. Our approach is universally applicable to all types of images, thus we intend to extend to other multimodal tasks in the future.
deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers). pages 4171-4188, Minneapolis, Minnesota. Associ- ation for Computational Linguistics.
Limitations
Our method has notable limitation that it heav- ily relies the precision of visual elements min- ing results. Despite employing state-of-the-ar computer vision models and carefully designing post-pro posc-proes procedures, there still exists a cer- tain level of errors and noises in the extracted re- sults. Another limitation is that we depend on the text understanding capabilities of In-Instruct and instruction-tuning operation to align targets with sentences. In the future, we should explore more effective alignment strategies. In addition, the Tk-Instruct we have chosen may not be con- sidered cutting-edge. Moving forward, we plan to instruction-tune more advanced language models such as GPT-3.5 and GPT-4, provided that they of- fer fine-tuning accesses.
Feifan Fan, Yansong Feng, and Dongyan Zhao 2018. Multi Blult--grinedd attention network for aspect-level sen- timent classification. In Proceedings Conference on Empirical Methods in Natural Lan- guage Processing, pages 3433-3442, Brussels, Bel- gium. Association for Computational Linguistics.
the 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770- 778.
Zaid Khan and Yun Fu. 2021. Exploiting bert for multimodal target sentiment classification through input space translation. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3034 3042. ACM.
Kirby Kuznia, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2022. Less is more: Summary of long instructions is better for program synthesis. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 4532- 4552, Abu Dhabi, United Arah Emirates. Associa- tion for Computational Linguisties.
Ethics Statement
Our work employs uniform code for all images to extract TSEI, TANPs and image scene caption, en- suring objectivity and avoiding bias. Furthermore, it utilizes two publicly available Twitter datasets while adhering to Twitter's data policy.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-ing -mgge- pre- training with frozen image encoders and large lan- guage models. arXiv preprint arXiv:2301.125970
A Acknowledgements
We are greatly appreciative of the anonymous re- viewers for their valuable feedback and sugges- tions.
Yan Ling, Jianfei Yu, and Rui Xia. 2022. Vision- language pre-training for multimodal aspect- based sentiment analysis. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 2149- 2159, Dublin, Ireland. Association for Computa- tional Linguistics.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018 Visual attention model for name tagging in multimodal social media. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1990-1999.
Man Luo, Sharad Saxena, Swaroop Mishra, Mihir Par- mar, and Chitta Baral. 2022. Biotabqa: Instruction learning for biomedical table question answering. In CEUR Workshop Proceedings, volume 3180, pages 291-304. CEU
S Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of

---

Association for Computational Linguistics (Volume I: Long Papers), pages 3470-3487 Dublin, Ireland. Association for Computational Linguistics.
Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022h. Super-Natural Supper-aaur Instructoo::::::::::: ization via declarative instructions tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language P Processing. pages 5085-5109, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wiinwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Neural In- formation Processing Systems, 35:27730 27744.
General- 1600+ NLP
Alec Radford, Jong Wook Kim, Chris Hallacy, Ramesh Aditya, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, and Jack Clark. 2021. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pages 8748-8763. PMLR.
Alex Warstadt, Amanpreet Singh, and Samuel R Bow- man. 2018. Neural network acceptability judgments. arXiv preprint orXiv:1803. 12471.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Con- ference on Learning Representations.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, al. 2022. Multitask prompted training enables zero-shot task generalization. In International Con- ference Learning Representations.
Nan Xu, Wenji Mao, and Guandan Chen. 2019. Multi- interactive memory network for aspect based mul- timodal sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pages 371
Kevin Scaria, Himanshu Gupta, Saurabh Arjun Sawant, Swaroop Mishra, and Chitta Baral. 2023. In- structabsa: Instruction learning for aspect based sen- timent amalysis. arXiv preprint Xrvv:22302. 08624....
Hao Yang, Yanyan Zhao, and Bing Qin. 2022. Face- sensitive image-to-ene tor-tttotmmooo-eeetttt cross-modaa trans- lation for multimodal aspect-based sentiment anal- ysis. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 3324-3335, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Siddharth Varia, Shuai Wang, Kishaloy Halder, Robert Vacareanu, Miguel Ballesteros, Yassine Benajiba, Neha Anna John, Rishita Anubhai. Smaranda Mure- san, and Dan Roth. 2022. Instruction tuning for few- shot aspect-based sentiment analysis. arXiv preprint arXiv:2210.0662..
Jianfei Yu and Jing Jiang. 2019. Adapting bert for target-oriented multimodal sentiment classification. In Proceedings of the -wenty-Eighth International Joint Conference Artificial Intelligence, IJCAI- 19, pages 5408-5414, Macao, China. International Joint Conferences on Artificial Intelligence Organi- zation.
Jiawei Wang, Zhe Liu, Victor Sheng, Yuqing Song, and Chenjian Qiu. 2021. Saliencybert: Recurrent at- tention network for target-oriented multimodal sen- timent classification. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 3-15. Springer.
Liwen Wang, Rumei Li, Yang Yan, Yuanmeng Yan, Sirui Wang, Wei Wu, and Weiran Xu. 2022a. In- structionner: A multi-task instruction-based gener- ative framework for few-shot ner. arXiv preprint arXiv:2203.03903.
Jianfei Yu, Jing Jiang, and Rui Xia. 2019. Entity- sensitive attention and fusion network for entity-level multimodal sentiment classification. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:429-49 439.
Yequan Wang. Minlie Huang, Xiaoyan Zhu and Li Zhao. 2016. Attention-based LSTT for aspect- level sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 606-615, Austin, Texas. Association for Computational Linguisties.
Jianfei Yu, Jieming Wang, Rui Xia, and Junjie Li. 2022. Targeted multimodal sentiment classifcation based on coarse-to-fine grained image-target match- ing. In Proceedings of the Thirty-Firs International Joint Conference on Artificial Intelligence, IJCAI- 22, pages 4482-4488. International Joint Confer- ences on Artificial Intelligence Organization.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganch Kordi, Amireeza Mirzaci, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, An- jana Arunkumar, David Stap, Eshaan Pathak, Gi- annis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel,
Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named entity recognition in tweets. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 32.

---

Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023. Sentiment analysis the era of large language models: A reality check, arXiv preprint arXiv:230. 15005.
Yichi Zhang and Joyce Chai. 2021. Hicrarchical task learning from language instructions with unified transformers and self-monitoring. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 4202-4213.
Zhe Zhang, Zhu Wang, Xiaona Li, Nannan Liu, Bin
Guo, and Zhiwen Yu. 2021. Modalnet: aspect- level sentiment classification model by exploring multimodal data with fusion discriminant attentional network. World Wide Web, pages 1-18.
Chao Zhao, Wenlin Yao, Dian Yu, Kaigiang Song, Dong Yu, and Jianshu Chen. 2022. Learning-by- narrating: Narrative pre-training for zero-shot dia- logue comprehension. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), pages 212-218, Dublin, Ireland. Association for Computa- tional Linguistics.
Appendix

---

Definition
The output will be 'positive' the target identified in the sentence contains a positive sentiment. If the sentiment of the identified target in the input is negative the answer will be 'negative'. Otherwise, the output should be 'neutral'.
Positive example
Input: Sentence: RT @ BarackObama # ClimatCChangesRRaal. So are the actions $TØ is taking to tackle it Target: President Obama. Scene description of the image: A man in white shirt and tie standing in front of trees. Adjective-noun pair extracted from the image: hand is helping. Text contained in the image: the strongest action ever taken by an american president to tackle climate change. Output: positive
Negative example
Input: Sentence: Former Bridgecorp boss $Tş will be released from jail next month Target: Rod Petricevic. Scene description of the image: A man in suit and tie standing in a room. Adjective-noun pair extracted from the image: performance is poor. Text contained in the image: None. Output: negative
Neutral example
Input: Sentence:
Former $TŞ boss Rod Petricevic will be released from jail next month Target: Bridgecorp. Scene description the image: man in a suit and tie standing in room. Adjective-noun pair extracted from the image: performance is poor. Text contained in the image: None. Output: neutral
Input Sample
Now complete the following example- Input: Sentence: ST$ joins an im- pressive list via Elias Sports Bureau Target: Kevin Durant. Scene description of the image: A group of basketball players standing next to each other. Adjective-noun pair extracted from the image: guy is nice Text contained in the image: most points without turnover nba finals (since 1978). Output:
Label
positive
Table 9: An instruction prompt example for instruction learning in our VEMP
Definition
Please perform Target-Oriented Sentiment Classification task. Every sam- ple is tweet from twitter which contains sentence and target in the sen- tence. The input provides sample's sentence and target. Given the input, you should identify the sentiment towards the target and assign a sentiment label from l'positive', 'negative", 'neutral"], J. Return label only without any other text.
Input Sample
Input: Sentence: $T$ joins impressive list ( via Blias Sports Bureau ) Target: Kevin Durant.
Table 10: A prompt example for text-only zero-shot setting GPT-4.

---

Definition
Please perform ple is tweet from twitter which contains sentence and target in the sen- tence. The input provides sample's sentence and target. Given the input, you should identify the sentiment towards the target and assign a sentiment label from I'positive,, 'negative', 'neutral"]. Return label only without any other text.
Sentiment Classification task. Every sam-
Input: Sentence: RT @ BarackObama: # ChimateChangelsReal. So are the actions President Obama is taking to tackle it Target: President Obama. Label:positive
Input: Sentence: Former Bridgecorp boss Rod Petricevic will be released from jail next month. Target: Rod Petricevic. Label:negative
Examples
Input: Sentence: Former Bridgecorp boss Rod Petricevic will be released from jail next month. Target: Bridgecorp. Label:neutra
Input Sample
Sentence: $T$ joins an impressive list ( via Elias Sports Bureau ) Target: Kevin Durant.
Input:
Table 11: A prompt example for text-only few-shot setting on GPT-4.
Definition
Please perform Target-Oriented Multimodal Sentiment Classification task. Every sample is a tweet from twitter which contains sentence, target in the sentence and an image. The input provides a sample's sentence, target, scene description the image, adjective-noun pair extracted from the im- age and text contained in the image. Given the input, you should identify the sentiment towards the target and assign a sentiment label from l'positive', 'negative', 'neutral']. Return label only without any other text.
Input Sample
Input: Sentence: STS joins an impressive list ( (via Elias Sports Bureau ). Target: Kevin Durant. Scene description of the image: A group of basket- ball players standing next to each other. Adjective-noun pair extracted from the image: guy is nice. Text contained in the image: most points without turnover in nba finals (since 1978).
Table 12: A prompt example for multimodal zero-shot setting on GPT-4.

---

Definition
Please perform Burget-Orientie Multimodal Sentiment Classification task. Every sample is a tweet from twitter which contains sentence, a target in the sentence and image. The input provides sample's sentence, target, scene description of the image, adjective-noun pair extracted from the im- age and text contained in the image. Given the input, you should identify the sentiment towards the target and assign a sentiment label from I'positive', 'negative", 'neutral"]. J. Return label only without any other text.
Input: Sentence: RT @ BarackObama # ChimateChangelsReal So are the actions President Obama is taking to tackle it Target: President Obama. Scene description of the image: A man in a white shirt and standing in front of trees. Adjective-noun pair extracted from the image: hand is helping, Text contained in the image: the strongest action ever taken by an american president to tackle climate change. Label:positive
Input: Sentence: Former Bridgecorp boss Rod Petricevic will be released from jail next month. Target: Rod Petricevic. Scene description of the image A man in a suit and tie standing in a room. Adjective- pair extracted from the image: performance is poor. Text contained in the image: None. Lagel.negative
Examples
Input: Sentence: Former Bridgecorp boss Rod Petricevic will be released from jail next month. Target: Bridgecorp. Scene description of the image: A man in suit and tie standing in a room. Adjective-noun pair extracted from the image: performance is poor. Text contained in the image: None. Label:neutral
Input Sample
Input: Sentence: $TS joins an impressive list (via Elias Sports Bureau ). Target: Kevin Durant. Scene description of the image: A group of basket- ball players standing next to each other. Adjective-nour noun pair extracted from the image: guy is nice. Text contained in the image: most points without turnover in nba finals (since 1978).
Table 13: A prompt example for multimodal few-shot setting on GPT-4,


Reviews:
[{"review_id": "Wir6yCCB7R", "reviewer": "Reviewer_fbv8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We apologize for the omission. We did not provide a rebuttal because there was no content included in the \"Questions For The Authors\" section. We regret that you have lowered the soundness score and excitement score from 4. We hope that the following explanations can alleviate your concern.\n(1) In fact, your concern serves as a strength of our work. As new NLP models continue to emerge, the Target-oriented Multimodal Sentiment Classification (TMSC) field urgently requires innovative approaches that incorporate these new NLP models to drive its development, rather than remaining limited to traditional fine-tuning based on BERT. One of the contributions of our work is to demonstrate the obsolescence of BERT-base within this field and to propose Visual Elements Mining as Prompts (VEMP), that leverage new NLP models to address challenging problems in the field.\n(2) Many baseline models in Table 3 also used stronger encoder. VLP-MABSA used a BART encoder (140M), and it even utilized an additional dataset, MVSA-Multi. Meanwhile, ITM used RoBERTa-base (125M). Therefore, criticizing the inconsistency in encoder choices as a drawback of the paper may not be reasonable. If we aim for absolute fairness, it would imply that all future work should be conducted using BERT-base.\nWe would greatly appreciate it if you could restore the scores to 4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is intuitive and easy to understand.\n2. The paper is well written.\n3. The proposed approach is novel and the results demonstrate its effectiveness.\n4. They combine their model with ChatGPT and obtain better results.", "reasons_to_reject": "They adopt T5 as the text encoder which could result in an unfair comparison.\n\nUpdate: The authors did not address my concern.  The proposed model utilizes the T5-base(220M) as the encoder which is stronger than BERT-base(110M). However, the baseline models such as FITE adopt BERT-base as the backbone network. It's unfair. So, I will change my score.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper a Visual Elements Mining as Prompts (VEMP) method, which first maps the useful visual information into texts and leverage the strong instruction tuned language model to finish the task. This paper introduces a novel strategy to utilize the language models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cp1cUz0B0W", "reviewer": "Reviewer_BJVD", "summary": "", "questions": "A: My major concern with the paper is that many details that are critical to ensure the validty of comparisons with the baselines are not discussed in the paper. I have enumerated these below. However, as a consequence of these details missing from the paper, I am not completely convinced of the approach being superior to the considered baselines.\n- (i): how were the baseline models trained and was there any hyperparameter search while training them?\n- (ii): How were the hyperparameters for the proposed approach (descibed in Section 4.1) set?\n- (iii): What is the 'ada' model that the authors used for GPT-3 (there are several variants)? Similarly, which ChatGPT model did the authors use (gpt-3.5; gpt-4; which dates? as the models were updated frequently?)\n- (iv): How did the authors choose the few-shot examples provided to ChatGPT? Were there any evaluations done to quantify the sensitivity of results to the few-shot examples being used in-context?\n\nB. I am also unsure about the validity of the approach that the authors adopt to extract text from images. It is possible that in some cases the text in the image is split between two different blocks (for instance, in Figure 1, one half of the text is in the top of the image and the other half is in the bottom). Authors mention that they select the bounding box with the maximum area? Does this mean that the authors only consider only half of the text within the image? This would provide incomplete context to the later stages of the approach.\n\nC. The authors find that Tk-instruct perform better than ChatGPT (not sure which specific model, as it is not mentioned) but that contary to what has been observed in some recent work (Labrak et al., 2023: https://arxiv.org/pdf/2307.12114.pdf). This makes me wonder if the authors adopted approaches like Chain of Thought prompting (Wet el al., 2022: https://arxiv.org/pdf/2201.11903.pdf) to accurately assess the capability of ChatGPT as a baseline. The prompts provided in the appendix do not contain such strategies.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your diligent review and feedback. The following is our responses to the questions you raised.\n\nA.   \\\nQ: (i): how were the baseline models trained and was there any hyperparameter search while training them?\n\nA: For the baseline models presented in Table 3, **we did not train them and directly referenced the reported best results from the papers in which these models were introduced,** as listed in Section 4.2. \nOnly the Text-Only Tk-Instruct model was trained by us. We trained it to investigate the effectiveness of our approach in leveraging the visual modality to enhance performance. To ensure controlled variables, we adopted the same hyperparameters as outlined in Section 4.1.\n\nQ: (ii): How were the hyperparameters for the proposed approach (descibed in Section 4.1) set?\n\nA: We experimented with a wide range of hyperparameter combinations, and selected the configuration that exhibited the best performance on the validation set.\n\nQ: (iii): What is the 'ada' model that the authors used for GPT-3 (there are several variants)? Similarly, which ChatGPT model did the authors use (gpt-3.5; gpt-4; which dates? as the models were updated frequently?)\n\nA: GPT-3 has based versions like 'ada', 'babbage', 'curie' and 'davinci'. OpenAI recommends using the 'ada' model for classification tasks in their API reference documentation.\\\nWe used GPT-3.5 in our paper since the access to the GPT-4 API was restricted by OpenAI at the time of paper submission. To compare with the latest ChatGPT version, **we conducted supplementary experiments using GPT-4 from August 23rd to August 26th. And we will use the following results to update some of the results in Table 7**.  \n| Method     | Twitter-2015(Acc)     | Twitter-2015(Macro-F1) | Twitter-2017(Acc)   | Twitter-2017(Macro-F1) |\n| -------- | -------- | -------- | -------- | -------- |\n| GPT-4(zero-shot) | 64.61 | 59.35 | 59.24 | 57.60 |\n| GPT-4(few-shot) | 65.19 | 59.88 | 59.97 | 58.73 |\n| VEMP(GPT-4)(zero-shot) | 65.00 | 59.95 | 59.56 | 58.29  |\n| VEMP(GPT-4)(few-shot) | 65.77 | 61.19 | 60.62 | 59.48  |\n\n\nQ: (iv): How did the authors choose the few-shot examples provided to ChatGPT? Were there any evaluations done to quantify the sensitivity of results to the few-shot examples being used in-context?\n\nA: The **selection criteria** are as follows: (1) Positive, negative, and neutral sentiment samples should all be included. (2) As our classification focused on fine-grained targets, we aimed to have at least two examples using the same input but with different targets and sentiment labels. In Appendix Table 9, Table 11, and Table 13, the second and third examples share the same input but have distinct targets (Rod Petricevic and Bridgecorp) and sentiment labels (negative and neutral). (3) The sentiment labels for each example were uncontroversial. \\\n**We tested numerous combinations of different few-shot examples, and the three examples selected in this paper yielded relatively better results** (the result differences were not significant for different few-shot examples).   \\\n**The reason to select only 3-shot examples**: Due to the maximum input tokens constraint of Tk-Instruct and the long length of our examples, we were limited to using three examples. And to ensure a fair comparison between our approach and ChatGPT, ChatGPT shared the same 3-shot examples.\n\nQ: B. I am also unsure about the validity of the approach that the authors adopt to extract text from images. It is possible that in some cases the text in the image is split between two different blocks (for instance, in Figure 1, one half of the text is in the top of the image and the other half is in the bottom). Authors mention that they select the bounding box with the maximum area? Does this mean that the authors only consider only half of the text within the image? This would provide incomplete context to the later stages of the approach.\n\nA: As we have explained in lines 223-227: \"**Text Symbols Embedded in the Image (TSEI)** from different bounding boxes are usually semantically incoherent if they are spliced together. Besides, TSEI from small bounding boxes are often too brief to provide valuable information, and instead add noises\". For instance, in Figure 1(b), only the text \"Most points without...\" at the top contains useful information, while the text on the player's clothes introduced noises.    \\\nWe acknowledge that our approach might result in the loss of some useful text within the images. **However, considering the harm introduced by noisy text when incorporating all text from images, we consider our trade-off to be worthwhile. The following supplementary experimental results validate that.**\n\n| Method     | Twitter-2015(Acc)     | Twitter-2015(Macro-F1) | Twitter-2017(Acc)   | Twitter-2017(Macro-F1) |\n| -------- | -------- | -------- | -------- | -------- |\n| No TSEI | 77.82 | 73.70 | 71.56 | 70.77 |\n| All TSEI | 77.34 | 73.43 | 71.23 | 70.55 |\n| Our approach | 78.88 | 75.09 | 73.01 | 72.42  |\n\n**As shown in the above table, due to the negative impact of noises introduced by small bounding boxes, extracting all TSEI can even result in lower performance than not extracting any TSEI. In contrast, our approach (Extraction with Maximum Area Strategy) performs the best.**\n\nQ: C. The authors find that Tk-instruct perform better than ChatGPT (not sure which specific model, as it is not mentioned) but that contary to what has been observed in some recent work (Labrak et al., 2023). This makes me wonder if the authors adopted approaches like Chain of Thought prompting (Wet el al., 2022) to accurately assess the capability of ChatGPT as a baseline. The prompts provided in the appendix do not contain such strategies.\n\nA: Labrak et al.'s work used **Tk-Instruct in zero-shot or few-shot scenarios** for comparison with ChatGPT, whereas we **instruction-tuned Tk-Instruct on the training set**. Hence, our experimental results did not conflict with theirs.    \\\nWe did not employ the Chain of Thought prompting for ChatGPT. Instead, **we followed the prompt strategy outlined in the paper \"Sentiment Analysis in the Era of Large Language Models: A Reality Check\" (Zhang et al., 2023, as cited in line 514),** since it was a study of sentiment analysis and closer to our research direction.\n\nQ: Justification For Ethical Concerns:\nThe ethics statement of the paper seemed lacking and could have discussed issues around how their approach is influenced by the biased captioning of image and detection of adjective-noun pairs in images. Additionally, the authors do not discuss the data availability and related ethics. They use two existing Twitter datasets and it is unclear if the datasets contain just the Tweet IDs (if that is the case, the data may no longer be available for reproducibility studies or follow-up work).\n\nA: The two Twitter datasets were widely adopted in this field. Prior researchers have provided ethical statements in their papers, ensuring the ethical safety of the datasets. And we employed consistent code for all images to extract image captions and adjective-noun pairs, ensuring objectivity and minimizing bias.    \\\nThe two publicly available Twitter datasets **contain complete tweet texts and images**. Additionally, the papers cited in Section 4.2 provided the access to these datasets, **and we will also offer that in our released code at a later stage**. \n\nQ: Typos Grammar Style And Presentation Improvements:\nThe notations for the length of sentence S and target word T are not being used anywhere in the equations. However, they are still mentioned in line 191-192.\n\nA: We will remove them if you believe it's necessary.\n \nQ: There is an excessive usage of acronyms in the paper which makes it harder to follow.\n\nA: We primarily used four acronyms: Target-oriented Multimodal Sentiment Classification (TMSC),  Visual Elements Mining as Prompts (VEMP), Text Symbols Embedded in the Image (TSEI), and Target-aware Adjective-Noun Pairs (TANPs). The usage of acronyms aims for concise expression as they appeared many times in the paper. If you have any suggestions for improvement, we are open to making revisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-  The idea of extracting the additional information (OCR and TANPs) from the image to boost target-aware sentiment classification is an interesting one\n- The authors compare against several text-only and multimodal baselines, as well as one image-only baseline. Overall, the range of experiments in the paper is good", "reasons_to_reject": "- While there are several experiments in the paper, some important details are missing in many places\n- Several design choices in the paper are not explained or mentioned without any proper justification\n- The paper uses many acronyms, which gets a bit confusing and often gets in the way of comprehension \nI expand on these points in my questions to the authors.", "questions_to_authors": "A: My major concern with the paper is that many details that are critical to ensure the validty of comparisons with the baselines are not discussed in the paper. I have enumerated these below. However, as a consequence of these details missing from the paper, I am not completely convinced of the approach being superior to the considered baselines.\n- (i): how were the baseline models trained and was there any hyperparameter search while training them?\n- (ii): How were the hyperparameters for the proposed approach (descibed in Section 4.1) set?\n- (iii): What is the 'ada' model that the authors used for GPT-3 (there are several variants)? Similarly, which ChatGPT model did the authors use (gpt-3.5; gpt-4; which dates? as the models were updated frequently?)\n- (iv): How did the authors choose the few-shot examples provided to ChatGPT? Were there any evaluations done to quantify the sensitivity of results to the few-shot examples being used in-context?\n\nB. I am also unsure about the validity of the approach that the authors adopt to extract text from images. It is possible that in some cases the text in the image is split between two different blocks (for instance, in Figure 1, one half of the text is in the top of the image and the other half is in the bottom). Authors mention that they select the bounding box with the maximum area? Does this mean that the authors only consider only half of the text within the image? This would provide incomplete context to the later stages of the approach.\n\nC. The authors find that Tk-instruct perform better than ChatGPT (not sure which specific model, as it is not mentioned) but that contary to what has been observed in some recent work (Labrak et al., 2023: https://arxiv.org/pdf/2307.12114.pdf). This makes me wonder if the authors adopted approaches like Chain of Thought prompting (Wet el al., 2022: https://arxiv.org/pdf/2201.11903.pdf) to accurately assess the capability of ChatGPT as a baseline. The prompts provided in the appendix do not contain such strategies.", "paper_topic_and_main_contributions": "The paper proposes an approach for classifying the sentiment of multimodal data (visual + text) data towards a target. This is done by extracting text in the image (OCR), and target-aware adjective-noun pairs (TANPs), along with the image caption. The authors argue that including the first two attributes (i.e., OCR and TANPs) along witht the caption to construct an instruction prompt for instruct-tuning a language-only model, would lead to better performance than existing baseliens. To this end, their experiments demonstrate better performance on two-datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The notations for the length of sentence S and target word T are not being used anywhere in the equations. However, they are still mentioned in line 191-192\n- There is an excessive usage of acronyms in the paper which makes it harder to follow", "Justification_for_Ethical_Concerns": "The ethics statement of the paper seemed lacking and could have discussed issues around how their approach is influenced by the biased captioning of image and detection of adjective-noun pairs in images. Additionally, the authors do not discuss the data availability and related ethics. They use two existing Twitter datasets and it is unclear if the datasets contain just the Tweet IDs (if that is the case, the data may no longer be available for reproducibility studies or follow-up work)."}}, {"review_id": "lqolihcCgj", "reviewer": "Reviewer_HyRf", "summary": "", "questions": "See Reasons to reject.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your diligent review and feedback. The following is our responses to the questions you raised.\n\nQ: Less exciting model framework: basically every module adopts an existing model so the novelty is limited. The reasons of choosing these models are also not clear. Only the instruction prompt and instruction tuning look interesting, but the fine-tuning process is also mediocre.\n\nA: The reason to choose these models: They were the state-of-the-art and best-suited models to meet our requirements for extracting visual elements to the best of our knowledge. **For example, to generate more accurate image caption, we tried and compared multiple models: Caption Transformer (Khan and Fu, 2021, as cited in line 47), BLIP, and BLIP-2. And we selected BLIP-2 since it exhibited the best caption generation performance**.\n\nQ: Is providing one example in the prompt enough? Will increasing this number be helpful? Will the order of presenting information affect the instruction learning results?\n\nA: **We used 3-shot examples rather than one example (as shown in Appendix Table 9)**.    \\\n**The reason to select only 3-shot examples**: Due to the maximum input tokens constraint of Tk-Instruct and the long length of our examples, the number was limited.   \\\nIn the process of experiments, we did not observe that the order of presenting the 3-shot examples affected the results significantly.\n\nQ: How sensitive are the model performance to the choice of selected models in each module? for example, what if the model CLIP is replaced by another model, will the model performance be affected much?\n\nA: The selected models were the state-of-the-art and best-suited ones to meet our requirements for extracting visual elements to the best of our knowledge. If they are replaced with other inferior models, the extraction results may be worse, potentially leading to performance decline. For example, to generate more accurate image caption, we tried and compared multiple models: Caption Transformer (Khan and Fu, 2021, as cited in line 47), BLIP, and BLIP-2. And BLIP-2 exhibited relatively better caption generation performance and entire model performance than the others.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The considerations of text symbols in images and the adjective-noun pairs.\n2. Instruction tuning", "reasons_to_reject": "1. Less exciting model framework: basically every module adopts an existing model so the novelty is limited. The reasons of choosing these models are also not clear. Only the instruction prompt and instruction tuning look interesting, but the fine-tuning process is also mediocre. \n\n2. Is providing one example in the prompt enough? Will increasing this number be helpful? Will the order of presenting information affect the instruction learning results?\n\n3. How sensitive are the model performance to the choice of selected models in each module? for example, what if the model CLIP is replaced by another model, will the model performance be affected much? ", "questions_to_authors": "See Reasons to reject.", "paper_topic_and_main_contributions": "The manuscript focuses on the Target-oriented multimodal sentiment classification task and proposes a Visual Elements Mining as Prompts model to address the task. The main contributions are the consideration of text symbols in images and adjective-noun pairs in forming instruction prompts and state-of-the-art results on two datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most reviewers noted the paper was well written, intuitive and novel/interesting. All of the reviewers had concerns about various experimental design choices not being justified, and one reviewer cited concerns about unfairly comparing the results, but rated the overall soundness of the paper as "good". Nonetheless, after the rebuttal, all of the reviewers found the paper to be adequately sound ("Good").
Decision: Accept-Findings