Descriptive Prompt Paraphrasing for Target-Oriented Multimodal Sentiment Classification
Authors: Dan Liu
Lin Li
Xiaohui Tao
Jian Cui
Qing Xie
Venue: EMNLP 2023 Findings (2023)
Keywords: Target-Oriented Multimodal Sentiment Classification
Prompt Learning
Multimodal Sentiment Analysis

Abstract: Target-Oriented Multimodal Sentiment Classification (TMSC) aims to perform sentiment polarity on a target jointly considering its corresponding multiple modalities including text, image, and others. Current researches mainly work on either of two types of targets in a decentralized manner. One type is entity, such as a person name, a location name, etc. and the other is aspect, such as 'food', 'service', etc. We believe that this target type based division in task modelling is not necessary because the sentiment polarity of the specific target is not governed by its type but its context. For this reason, we propose a unified model for target-oriented multimodal sentiment classification, so called UnifiedTMSC. It is prompt-based language modelling and performs well on four datasets spanning the above two target types. Specifically, we design descriptive prompt paraphrasing to reformulate TMSC task via (1) task paraphrasing, which obtains paraphrased prompts based on the task description through a paraphrasing rule, and (2) image prefix tuning, which optimizes a small continuous image vector throughout the multimodal representation space of text and images. Conducted on two entity-level multimodal datasets: Twitter-2015 and Twitter-2017, and two aspect-level multimodal datasets: Multi-ZOL and MASAD, the experimental results show the effectiveness of our UnifiedTMSC.

Descriptive Prompt Paraphrasing for Target-Oriented Multimodal
Sentiment Classification
Dan Liu
Jian Cui Wuhan University of Technology. Wuhan, China
Lin Li
Xiaohui Tao 2
Qing Xie
The University of Southern Queensland, Toowoomba, Australia (liununew, cathylilin, 3 318971, felixxq} @ @whuttedu.cn Xiaohui. Tao @ unisq. edu.au
Abstract
Target-Oriented Multimodal Sentiment Clas- sification (TMSC) aims to perform sentiment polarity on a target jarget considering its cor- responding multiple modalities including text, image, and others. Current researches mainly work either of types of targets in de- centralized manner. One type is entity, such as a person name, a location name, etc. and other is aspect, such as 'food', 'service', etc. We believe that this target type based division task modelling is not necessary because sentiment polarity of the specific target is not governed by its type but its context. For this reason, propose unified model for target- oriented multimodal sentiment classification, so called UMSC. It prompt-base lan- guage modelling and performs well four datasets spanning the above two target types. Specifically, we design descriptive prompt para- phrasing reformulate TMSC task via (1) task paraphrasing, which obtains paraphrased prompts based the task description through paraphrasing rule, and (2) image prefix tuning, which optimizes a small continuous image vee- tor throughout the multimodal representation space of text and images. Conducted on two entity-level multimodal datasets: Twitter-2015 and -witter-2017 and aspect- level multi- modal datasets: Multi-ZOL. and MASAD, the experimental results show the effectiveness of our UNIfiedTMSC.
Input:
Inpus
RT BBC TeCTea Clasing
light and shadoows beautfful #'tran #archttcctrre
Bess is excrytitig #MCM
Chark Roes
MCM
Enneyy
Ently
Ousput Lobel
positive
iostral
Cuspus
Lobel
Entity-level MSC MSC
perforwaros, beautiful appeoranne, smonth game oparation, high camert pooess smplc interface, and the game can be high frume min mode The battary life is average, and the screen is not used for "Ik, butlt does not affect and the graiiness cannot be soon with the rakand
Aspect
Category
Terget
ballery life
appearance
coot performance
performance configuratin
photography effect
Anonct
Outpurt
10
(b) Aspect-level MSC
Figure Two forms of target-oriented multimodal sen- timent classification.
At present, fine- ane-ggrined Multimodal Sentiment Classification (MSC) includes two main tasks: entity-level MSC and aspect-level MSC, as shown in Figure 1. (1) For entity-level, the entity and its context encoded independent input text in some studies( Yu and Jiang. 2019; Yu al., 2020; Wang et al., 2021; Zhao al., 2022). Others jointly consider the encoding of an entity with its contex- tual learning, which is their main focus to achieve good MSC performance (Khan and Fu, 2021; Xiao et al., 2022; Yang et al., 2022). For example, Xiao et al. (2022) presented a dual stream adaptive multi- feature extraction graph convolutional network to convert an image into its caption.
1 Introduction
With the emergence of new media and advanced technology, the forms of information released by people are quietly changing, from moda-ity to multi-modality now, such as text, images, etc (Xu and Mao, 2017). This also pushes researchers conduct multimodal learning (Yu al., 2016; Xu et al., 2018; Long et al., 2022). For sentiment analysis, both and image are highly correlated with sentiment polarity. Moreover, they can com- plement and reinforce each other (Xu et al., 2019).
(2) For aspect- level, the aspect and its context are being encoded individually due to the semantics involved in an aspect itself (Xu et al., 2019; Zhang et al., 2021; Zhou et al., 2021). For example, Xu et al. (2019)

---

studied a multi-interactive memory network model. These TMSC works can promole human decisions by assisting users in knowing about certain targets.
model achieves state-of-the-art performance on two entity-level datasets: Twitter-2015 and Twitter- 2017, and two aspect-level datasets: Multi-ZOL and MASAD. (1) The results of the task descrip- tion based prompts are superior to those of arbi- trary prompt templates. datasets, our model improves Accuracy by 1.0%- 2.8% and 1.5%-3.7% macro-F1 by 1.5%-4.5% and 1.7%-7.5%% respectively. (3) On two aspect- level datasets, our model gains of 1.46%-8.879 and 1. 1.66%-3.02% on Accuracy, 3.77%-5.01 % and 2.19%-3.25% on macro-Fl are derived. The exper- imental results demonstrate the effectiveness our UnifiedTMSC.
As we know, an entity usually is person name, or a location name, etc (Li et al., 2022b). Such entity target only has a specific meaning when con- necting it with specific modality content, which means it is difficult to accurately understand the en- tity without its context. On the contrary, an aspect itself in some degree can represent what it means even without its context. Because of this obvious difference, previous studies show their interest in differently modelling the two tasks to capture tar- get related context (Xu et al., 2019; Yu et al., 2020; Zhang al., 2021; Song et al., 2023). For example, in Figure (a), "Chuck Bass", "MCM", and "Iran" do not have any exact meaning when they are dis- connected from the specific context. However, in Figure ((b), both aspect term and aspect category have their own meanings with a hidden sentiment tendency. For the "battery life" of mobile phones, the first reaction is that the longer the battery life, the better. Despite the above differences among the TMSC tasks, from the perspective of sentiment classification, the goal of TMSC is predict the sentiment polarity of target no matter whether is an entity or an aspect. Therefore, in our view, the boundary is unnecessary.
(2) On two entity-level
Related Work
2.1 Entity-level MSC
As pioneer, Yu and Jiang (2019) proposed a BERT-based multimodal architecture to determine the sentiment polarity an entity. Yu et al. (2020) introduced an entity-sensitive attention and fusion network. And Wang et al. (2021) put forward a recurrent attention network. Khan and Fu (2021) introduced an input space translation framework construct image context from the image. Zhao al. (2022) used the adjective nouun pairs extracted from images as the knowledge enhancement based on Yu and Jiang (2019) and Wang et al. (2021). Morcover, Yang et al. (2022) explored facial information in images to obtain visual and sentiment clues.
In this paper, we propose unified TMSC model via prompt based language modelling, called UnifiedTMSC, which is independent of the target type in TMSC. Our core is reconstruct the two TMSC tasks through descriptive prompt paraphras- ing. The prompts we design can place entity and aspect in their context, while also being close to the TMSC task description. To achieve this goal, we carry out our work from two aspects: (I) task para- phrasing. The task description is transformed into seed prompt, and different paraphrased prompts are obtained by using our paraphrasing rule. They serve discrete prompts for the text and fit into the Masked Language Modeling (MLM) format. And (2) image prefix tuning (Li and Liang, 2021). A segment vector is initialized for the image pre- trained embedding as the prefix continuous prompt. In the subsequent multimodal continuous represen- tation space, the image pre-trained embedding is fixed and only some segment of the initialized vec- tor is optimized. In this way, sentiment labels are generated through the dilzinfillngg method.
In the research stated above, the entity can be encoded as a distinct text input, or entity and con- text can be combined as the text input. Their goal to more effectively learn the semantics related to entity sentiment.
2.2 Aspect-level MSC
Aspect-level multimodal classification was first pro- posed by Xu et al. (2019), and they introduced a multi-interactive memory network to analyze mul- tiple correlations multimodal data. Zhang et al. (2021) presented a multimodal fusion discriminant attention network and designed discriminant ma- trix to supervise the modality fusion. Zhou et al. (2021) conducted multimodal interaction model that learns the relationships between text, image, and target aspect through interaction layers and ad- versarial training. One key difference between an aspect and an entity that the aspect has its own semantics inferred from the aspect words. There- fore, existing research usually regards the aspect
In our extensive experiments, our UnifiedTMSC

---

Task Paraphrasing
Task Paraphrasing (MLM)
Word postiton embed
Prefix inibalization embre
Seatiment Classiiiction
Task Description: A sentmmnt is basically openion penson expresses toavants aepect, entry, penson, event, feature, object, certain target
Special tolarn embed
Image pre trained embod
Outpot
Word token embed
(MA logit
CP
LYIM
Image position embed
Hiddes layer oulput
Seed Prompt sentiment that person expeesses lowarts cerialn lauge.
Feature path
Gradient path
Multimodal Transformers
paraphrasing rule
Trainable
Frozen
g(x) (yys,, no
fix) (Beginning. CaetuniiingMiMdiiie Middle Ending)
JD a ff ((arget)) Usenti
ment gisent ment
Paraphrased Prompt: (bargtt] express [MASK] sestmment
[CLS]
Paraphrased Prompt
Text
[SEP]
Figure 2: The model of our proposed UNI Seed prompt and keyword 'sentiment' are obtained based on the task description and are converted into diverse paraphrased prompts through the paraphrasing rule (Equation I), The paraphrased prompt and the original text input are concatenated to form the new text input which encoded by the encoder to gain text embedding. The image is divided into multiple regions and each of the regions is initialized as vector that is used as the prefix prompt for image pre- pre-trained embedding. The text embedding and image embedding are undergone cross attention to gain modality fusion embedding.
itself as an additional input.
the results are promising (Seoh et al., 2021; Li al., 2021, 2022a; Gao et al., 2022; Liu et al., 2023a). Inspired by the above studies, our unified model is through the task description based prompt para- phrasing with jointly soft and hard prompt tuning.
Our focus: In contrast to previous studies, our model can run across TMSC tasks. It combines the target (entity or aspect) and its context as a text input using task description based paraphrased prompts. We can get sentiment-related semantics about the target by providing its context in task description based prompts.
Method
3.1 Task Formulation
Given multimodal samples D, for each sample E D, it contains sentence S with words (Wi, W2, u2, .... and one more related images I, well as target T which contains m words (W1, w2, w2, (m)) and a sub-sequence of or pre- defined phrase. For the target T, is associated with a sentiment label Y. In general, Y E { pos- itive, neutral, negative }. and different tasks may have different sentiment labels. Our goal is to learn target-oriented sentiment classifier that can cor- rectly predict the sentiment label for each sample X (S, T,T).
2.3 Prompt paraphrasing
Prompt tuning has received increasing attention re- cently (Radford et al., 2021; Yao et al., 2021) and has been successfully applied in many domains (Han et al., 2022; Liu et al., 2023b). For example, in the field of Question Answering (QA), Khashabi al. (2020) reformulated many QA tasks as a text generation problem by fine-tuning seq2seq-base based pre-trained models and appropriate prompts from the context and questions. For Information Ex- traction (IE), Chen et al. (2022) first explored the application of fixed-pron prompt LM Tuning in relation extraction and Lu et al. (2022) applied prompt to control the information to be extracted. In other research fields, Cui et al. (2023) used prompt learn- ing text input to conduct the meme mining task. Recently, the prompt has been used for the task involving fine- grained text sentiment analysis, and
3.2 Overview
As shown in Figure our model consists of two modules: task paraphrasing (hard prompt) and im- age prefix tuning (soft prompt). For the given mul- timodal data X = (S, I,T), we obtain paraphrased prompts (P,, P2...) through the task description

---

Paraphrased Prompts
Y, Position >
T.Position
< K, Position
Synonym of K
(target) express [MASK] sentiment.
< [MASK].M>
[target].
< sentiment,E>>
n0
The emotion of (target) is [MASK].
[MASK],E> > < [target) J
< emootoon >
ye3
[target]'s 's emotion is [MASK].
[MASK],E> > (target), ,B
emotion.
yes
The [MASK] feeling is ftargetf's s sentiment.
< [MASK], > < [target] M >
< sentiment,
R0
The sentiment is [MASK] and conveyed by (targer).
< [MASKJM > < (turget ].E >> < sentiment,B >
R0
The emotion expressed by /target] is [MASK].
[MASK],E> > < (target) M >
< <emotion. >
yes
A [MASK] sentiment is expressed towards to (target). < [MASK]. > (target)
< sentiment,M >
R0
Table 1: Some examples of paraphrased prompts, as well as the relative positions Y (sentiment labels, replaced by [MASK] token, marked with bold). T (the targets that need to recognize sentiment polarity, annotated with italics), and K (keywordds extracted from seed prompt or replaced with synonyms, annotated with underline) in the prompts. "Synonym of K' indicates whether to replace the keyword 'sentiment" extracted from the seed prompt with synonyms. 'B", 'M', and 'E' respectively represent the positions in the paraphrased prompt as: 'Beginning". 'Middle', and 'Ending".
TMSC, which are used as prefixes for text input S in the task paraphrasing module (the left of Fig- ure 2). Paraphrased prompt P; contains [MASK] token and connected with text S. The text em- bedding obtained through an encoder in the bottom-right of Figure 2. In the image prefix tun- ing module, we apply an initialization vector V for pre-train image embedding E as the continuous prefix. After text embedding R and image embed- ding (V+ E) are added to their respective position encoding, the fusion vector is obtained through the multimodal Transformer in the middle right of Fig- ure 2. We take the hidden layer vector H and pass it through the MI.M head to get the prediction score of [MASK] position for each word in the vocabu- lary C. Finally, the cross entropy loss LMIM of the prediction result Output and the true sentiment label Y is calculated.
the task-related keyword K ('sentiment') from the task description. The seed prompt is transformed through the paraphrasing rule, and guides the gen- eration of paraphrased prompts that are close the original task description. Our paraphrasing rule can be formalized in the following form:
A f(K) A g(K)
(1)
where the function ʃ represents the relative posi- tion in the paraphrased prompts and the substitu- tion of synonyms for keyword ('sentiment') is illustrated by the function g. f(), {B, M, E} (meaning 'Beginning'. 'Middle', 'End- ing"), which respectively stands for the relative position of the sentiment label, the target entity, and the keyword derived from the task description. g() C [yes,no} means whether to replace syn- onyms of keywords K. Moreover, synonyms are synonymous explanations for the keyword 'senti- ment' in the dictionary. For example, in the Bing dictionary, the synonyms for 'sentiment' include 'emotion', 'feeling', 'opinion', etc.
e
3.3 Task Paraphrasing
In order to provide a specific context for a target, i.c. entity and aspect, we add the target-involved prompts to the text input. Since we change the original sentiment classification task in this way to the pre-trained MLM task, it is crucial to figure out how to develop prompts that are suitable for the original task, that is to say, the prompts should be consistent with the expression of the task descrip- tion. Inspired by prompt tuning in various domains, such as the visual grounding problem (Yao et al., 2021) and visual question answering task (Liu et al., 2022), we propose the task paraphrasing module to draw paraphrased prompts as a solution to the above issue.
The task paraphrasing module is shown in the left part of Figure 2. Generally speaking, based on the relative position combination of Y, K, and T, one seed prompt can be paraphrased to gain multi- ple candidate prompts. If we replace the keyword with different synonyms, we will receive more para- phrased prompts. Some examples of paraphrased prompts P: are listed in Table 1. When paraphrased prompt P: is in the training phase, the position of Y is replaced by [MASK] token which the prediction object. Finally, the P, and text S are concatenated to obtain a new text input:
We get the seed prompt according to the task de- scription composed of natural language, and take
Snew [CLS] P. S [SEP]
(2)

---

PREFIX
Image
Twitter-2015
Twitter-2017
Multi-ZOL
MASAD
Text-Image
Text-Image
Texi-Imaye
Text- Iaa-e
Train
3179
3562
22775
26292
Dev
1122
1176
2847
2925
Embed
V1
V/
6
8
3
Test
1037
1234
2847
8850
Total
5338
5972
28469
38532
Indexing
8
Table 2: Statistics of four TMSC datasets.
Pix. (1,2,1
1/s (4,5,6,7,8)
3.5 Multimodal Transformer With MLM
Figure 3: An example of image prefix tuning. tae is random initialization of the prefix sequence and Clas is the image pre-trained embeddingg
Training. Given a triplet (S, (S,,,T,Y) after the task paraphrasing module and image prefix tuning modulc, the updated text input Snew and image em- beddings Enew are obtained. Text encoder encodes Snew to gain text embedding R. R and the image embedding Frew undergo cross-attention through Transformer to obtain the fusion embedding H. In this multimodal Transformer, the image pre-trained embedding E fixed and not updated, and only is updated. Fusion embedding passes through an MLP to obtain prediction scores Logit:
3,4 Image Prefix Tuning
Discrete prompts, i.e., hard prompts, in the text are natural and easy to understand. For images, directly applying discrete prompts cannot ensure alignment between text based prompts and images since there are modality gaps between different modalities. Therefore, finding suitable prompts for images is crucial. This module mainly focuses on how to add prompts images to facilitate better modality fusion of images and text.
Logil = ML
(5)
The word with the highest prediction score is the prediction result:
Inspired by (Tsimpoukelli et al., 2021) and (Li and Liang, 2021), we introduce a continuous vector prefix, i.e., soft prompt, to image pre-trained embedding, shown in Figure 3. We first seg- ment the image into regions, and then initialize a vector V4 (i E (1.))...r) for each region to form the prefix embedding V {V1, #2,, ..,)) and E 20x2048 Here, Padr denotes the indices of the prefix sequence. Paarl denotes the length the prefix and Lidel indicates the length of the image pre-trained embedding. In F = (c1, C2 63, ca, es), E 2048 refers to the image embedding of the i- th region. The prefix embedding V = {2,, 22, 2/)3 and V 3x2048 If there are multiple images cor- responding the text, we initialize soft prompt for each image and their averaged vector is used as the final prefix prompt for the images. V and E are concatenated to obtain new image embedding Euew
Output = argmax Loggit)
(6)
Finally, the predicted result and the true senti- ment label Y are calculated in the cross-entropy loss to optimize model.
exp( exp(Loge Logits)
LMLM = - log(
(7)
2j=0 exp( Logit,))
|0
where is the true sentiment label number and C is the size of the language model vocabulary. Inference. In the inference stage, given triplet (S, 7, T), the sentiment polarity y' of T is deter- mined via the triplet.
(8) After obtaining the fusion embedding of text and images, take the argmax of the logit of the [MASK] position to obtain the final prediction result. Finally, for non-label words generated by the model, the answer engineering is used to map them to sentiment labels.
Y' arggar
Enew V + E
(3)
Enew (i)
0/ E(s (Paal)
i Pide otherwise
4 Experiment
4.1 Experimental Setup
Enew and gnew are image input and text input respectively, and the subsequent encoding process is carried out together.
Datasets. According to Zhou et al. (2021), there are four datasets for target-oriented multimodal

---

sentiment classification, including two entity-level
Method
Twitter-2015 2015
Twitter-001 2017
datasets: Twitter-2015 (Zhang et al., 2018; Yu and (Lu et al., 2018; Yu and Jiang. 2019) and two aspect-level datasets: Multi- 70L. (Xu et al., 2019), MASDA (Zhou et al., 2021) The statistics information of these four datasets are shown in Table 2, and their details about the data partitioning and label types are presented in A.2.
17.2
71.3
0101152
70.5
68.0
0,0156
SaliencyBERT BERT
72.4
67.2
Jiang,
CapTHBERT
78.0
73.3
0.0120
72.3
70.2
0.0134
JML-MASC
VLP-MABSA
78.6
73.8
0.0131
73.8
71.8
0.0111
FITE- Large
78.8
74.8
0,0048
73.9
73.0
0.0016
UnIRedTMSC
76.3
75.4
74.7
Table 3: The experimental results on the multimodal entity-level datasets: Twitter-2015 and Twitter-2017. The results presented in the table are the average of different prompt results.
Evaluation Metrics. To fairly compare with state-of-the-art approaches, our UMSS is evaluated across two TMSC tasks and adopts the Accuracy (Acc) and Macro-F1 score (F1), follow- ing Yu and Jiang (2019) and Ling al. (2022). Implementation Details. For the Twitter-2015, Twitter-2017, and MASAD, the batch size is set to 16 and the epochs are set to 6. The text encoder used is BERT-base-uncaser (Devlin et al., 2019). For the Multi-ZOL. dataset, in order to ensure fair- ness in the experiment, we employ BERT-base- chinese (Devlin et al., 2019) as the text encoder, the batch size is 8, and the epochs as 6. Moreover, multilingual pre-trained models can also be used as our text encoders.
Mulli-ZOL
MASAD
Method
Ape
FT
VVauue
Aue
IT
Valuue
MIMN
0.0316
93.55
0.0019
ModalNet
62.71
60.94
0,0266
MMAP
95.29
94.61
0.0049
CLIP
64.00
59.70
0.0092
95.88
94.21
0.0079
UnIeedTMSC
70.46
64.71
97.54
96.80
Table 4: The experimental results the multimodal aspect-level datasets: Multi-ZOL and MASAD.
(Neurocomputing, Zhou et al. (2021)) and CLIP (INT J COMPUT VISION, Zhou al. (2022)). The detailed introduction of all the baseline models mentioned above is in Section A.I.
For all datasets, we apply Resnet-50 (He al., 2016) as the image encoder to get the image prefix embedding, and the max length of the new text input (in Eq. (2)) is 96. The model learn- ing rate is set as le-5 and the dropout rate is le-2. Four layer Transformers (Vaswani et al., 2017) are aimed to perform cross attention between differ- ent modalities and the pre-training parameters not loaded. Four NVIDIA TITAN Xp GPUs, each with 12GB of memory, are employed in our experi- ments which are done on CentOS computer. The deep learning framework is Pytorch, and AdamW is used as the optimizer.
4.3 Experimental Results and Analysis
4.3.1 Overall Performance
The experimental results on multimodal entity- level and aspect-level datasets are presented in Ta- ble 3 and Table 4 respectively. The best results on each metric marked in bold and the second best results are highlighted with an underline. Multimodal Entity-level Datasets Results. As reported in Table 3, compared with the baselines, our UnificdTMSC makes significant improvements in entity-level MSC. On the Twitter-2015 dataset, our improvement is approximately 1.0% on Accu- racy and 1.5% on the macro-F1 compared to the FTTE-DE-Large. On the Twitter-2017 dataset, we achieve improvements over multimodal baseline VI.P-MABAA on the Accuracy by 1.5% and 1.7% on the macro-F1, which indicates that using prompt tuning to fuse entity and context can achieve a bet- ter sentiment-related semantic understanding of an entity, resulting in better classification results. Multimodal Aspect-level Datasets Results. Our UnifiedTMSC performs best in two datasets among all multimodal baselines, as listed in Ta- ble 4. This demonstrates the effectiveness of our proposed unified model based on prompt paraphras- ing. Especially on the Multi-ZOL dataset, our Uni- fiedTMMC outperforms the ModalNet on the Ac- curacy by 7.75% and 3.77% on the macro-F1. On
4.2 Compared Baselines
Because previous work has separated entity-level MSC and aspect-level MSC, the baseline mod- els for each task are different. For the Twitter- 2015 and Twitter-2017 datasets, we compare six baselines including TomBERT (ICCAI, Yu and Jiang (2019)), SaRR (PRCV, Wang et al. (2021)), CapTrBERT (ACM Multimedia, Khan and Fu (2021)), JML-MASC (EMNLP, Ju et al. (2021)), VLP-MABSA (ACL, Ling al. (2022)), FITE- DE-Large (EMNLP, Yang et al. (2022)). For the Multi-ZOL. and MASAD datasets, our model compared with MIMN (AAAL, Xu et al. (2019)), ModalNet (WWW, Zhang et al. (2021)), MMAP

---

Food
Buildings
Goods
Human
Scenery
Method
Acc
Acc
F1
Acc
Acc
Acc
MIMN MMAP
AAAI 2019
94.72
91.39
96.26
95.80
95.93
95.87
92.54
92.31
93.17
92.38
Neucomp 2021
95.75
92.89
96.86
96.85
96.55
96.44
92.74
92.74
94.57
94.15
CLIP UnifiedTMSC
ICV 2022
96.15
93.05
95.84
94.27
97.36
96.88
94.20
93.24
95.84
93.59
98.00
96.02
98.26
98.25
98.08
98.01
97.82
96.49
95.56
95.24
Table 5: The experimental results of each domain in the MASAD dataset. Because the accuracy and Macro FI score in the domain of animal and plant reach 99.07%%-99.225 due to the duplicated samples, they will not be displayed here.
the MASAD dataset, compared to the MMAP, ours improve performance by about 2.25% on Accuracy and 2.19% on the macro-F1.
1Witter-2015
Twitter-2017
Acc
F1
Acc
Fl
P1
80.2
77.6
76,9
76.7
paraphrased prompts
P3
81.1
77.1
76.6
75.9
Specifically, there are multiple domains in the MASAD dataset, conduct experiments each of them, and the experimental results are shown in Table It is clear from the results that our model has achieved large improvements in each domain. Finally, t-tests are conducted to demonstrate the effectiveness of UNS. From the P-value of other models in Table 3 and Table 4, it can be found that all P-values are less than 0.05. This shows significant difference in statistics between UnifiedTMSC and other models.
79.8
77.5
76.2
75.9
P1
79.1
75.5
73.8
73.7
arbitrary prompts
P.
79.0
75.1
74.3
73.6
P3
79.4
75.1
74.5
72.9
Table 6: The experimental results different prompts. The results of paraphrased prompts superior to the arbitrary prompts, and Pj. P2 and P33 are comparable. the experiments, one of the paraphrased prompts can be selected for experimentation.
three paraphrased prompts. The three arbitrary prompts follows:
After comparing and analyzing the experimental results, we can summarize the following two points from our prompt tuning:
P feel the /target/ is [MASK].
P.: The [target) made me feel [MASK].
Ps: [MASK] the /target).
Our prompt paraphrasing method delivers the target's context and fits the TMSC job effec- tively, and it produces good results, demon- strating the efficacy of our unified model.
where [target) is the entity that needs to deter- mine sentiment polarity, and [MASK] represents the masked word, i.e. sentiment label. The masked word in the P/ P2.PP is (good, ok, bad), (good, indifferent, bad] and (love, dislike, hate) respec- tively. After the masked words are generated, we perform answer engineering to map the predicted results to the sentiment polarity set, that is, the probabilities of these predicted words are made to be equal to the probabilities of being Positive, Neutral, and Negative.
II. Utilizing the target as separate input has worse results than taking the target and con- text together as text input. This shows that contextual information affects the target's se- mantics, and a contextual content that is ap- propriate for the task will result in a well un- derstanding of target with semantics.
The results of several different prompts are shown in Table 6. Through analysis and compari- son, we can obtain the following summaries:
4.3.2 The Effect of Prompt Designs
For the 2wi5ter-2015 and Twitter-2017 datasets. we select three paraphrased prompts from Table to conduct the experiments. The three selected paraphrased prompts are as follows:
I. Our paraphrased prompts created by using the task description are much superior to arbitrary prompt templates, demonstrating the value of our task paraphrasing module in producing paraphrased prompts that are appropriate for the original sentiment classification task. In addition, the performance of different para- phrased prompts is comparable, and in sub- sequent experiments, anyone can be selected for training and inference.
P1: [target) express a [MASK] sentiment.
Py: The emotion of (target/ is [MASK].
Ps: A [MASK] sentiment is expressed to- wards to (target).
In addition, verify the performance the task paraphrasing module. We design three arbitrary prompt templates and compare them with the above

---

Image
Uf 23
(e) Fan Tarows Water Bottle Justin Bieber After He Says He Doesn Know the
(b) Kim Kardasbian goes all out Kampe West burthdey in the Bahames
(c) II @ TrumpDoral Concratilatoons the the new # Missll Misss Colombia, Paulina Vega!
[d] Nbe - The Cars are Shocked Draymond Green Keeps Gerling Away With Kic-
Text
[Kim K Karlashhin, Positive) KKnyy West, Positive) abbamas Neutral)
(Nbs, Neutrai)
Colombia, Positive) (Paulina Vega, Positive)
Label)
(Justin Biebe, Negative)
(Cavs, Neutral)
(Dmymonn Green, Negative)
wio Pamphrased Prompe
Oustin Biebe, Neutral)
[Kim K andashian, Neutral) (Kanve West, Neutral) [Bahumas, Nentral) >
[Mis Colombia, Neutral) (Paulina Vegns Neutral)
(Cavs. Negati
(Nae, Neutral)
(Draymond Green, Pasilive) x
(Kim Konfashion, Positive)
(Nbs, Nestral)
w/o Image Prefix
(Tusti Blebe, Positive)
IKanye West, Neutral) [Babemas, Neutral)
(Miss Colombia, Neutral)
[Cass, Negalive)
(Paulina Vega, Positive) v
Draymond Green, Negative)
(Kim Kandashion, Pasitive)
(Nbo, Nestral)
(Miss Columbiaa, Positivel
UnIFiedTMSC
(Tustin Biebe. Negalive)
aKay West, Posilive) (Bahamas, Neutral) >
(Paulina Vega, Positiie)
[Cans, Neutral)
(Draymond Green. Negative)
Figure 4: Case study on four test samples. Red font indicates correctly predicted labels.
Twitter20115
Twitter-2017
bel of the target. The results are shown in Table
Acc
F1
Acc
FI
UnifiedTMSC
79.8
76.3
75.4
74.7
7. The absence of paraphrased prompts has sulted in considerable performance decrease. On the Twitter-2015 dataset, the Accuracy and macro-
w/o Paraphrased Prompts
75.7
72.0
68
65.9
Pi w/o Image Prefix
78.9
75.2
74.1
73.7
Pa w/o Image Prefix
78.6
75.7
74.4
73.8
Ps w/o Image Prefix
78.5
75.7
74.3
73.9
Fl are dropped by approximately 4.1 1% and 4.3%, while on the twitter-2017 dataset, the Accuracy declines by about 7.4% and the macro-F1 drops by about 8.8%. This demonstrates how using text paraphrased prompts can provide entity with its task-related semantics.
AVG w/o Image Prefix
78.7
75.5
74.3
73.8
Table 7: Ablation study of our UnifiedTMSC model.
II. The position of [MASK] in the paraphrased prompts can also have an impact on the ex- perimental results. In our case, the effect is best when the relative position of [MASK] is "Middle" rather than "Beginning" or "End- task,
Image Prefix. We only use the paraphrased prompt P. in Eq. (2) without applying the image prefix prompt V in Eq. (3) to study the importance of the image prefix. For text input, Pl, P2, and P3 are proceeded for the ablation study. From Table can be seen that the performance has dropped after taking out the image prefix V (in Eq. (3)). The Accuracy decreases by 1.1%, and the macro- Fi drops by 0.8%, according to the average results for the Twitter-200 2015 dataset. On the Twitter-2017 dataset, the Accuracy and macro-Fl decline by roughly 1.0%. This illustrates the effectiveness of the image prefix. Moreover, the ablation study also shows that different paraphrased prompts have var- ied outcomes, demonstrating the language models' sensitivity to prompts.
ing". Therefore, when meeting the position of [MASK] may be a factor to affect task performance.
4.4 Ablation Study
To further investigate the effects of paraphrased prompts and image prefix, because entity-level MSC is more challenging than aspect-level MSC, we conduct ablation analysis on the multimodal entity-level datasets: Twitter-2015 and Twitter- 2017. The results of the ablation experiments are shown in Table 7.
Paraphrased Prompts. P. is omitted from Eq. (2) and we just add the image prefix V in Eq. (3) for the experiment in order examine the effects of the paraphrased prompts. The linear classification layer uses the fusion vector derived by multimodal Transformers as input to estimate the sentiment la-
4.5 Case Study
In our case study, the compared methods are only image prefix (denoted by w/o Paraphrased Prompt), only text prompt (denoted by w/o Image Prefix),

---

and our UnifiedTMSC model with soft and hard prompts. We apply Pi for both w/o Image Prefix and UnifiedTMSC.
with grammar rules, and paraphrased prompts that comply with grammar rules may not necessarily be the best. The second limitation is that more at- tempts are needed to conduct experiments on multi- lingual pre-trained models. Furthermore, the last is that we have not explored whether our model can be extended to other multimodal research fields, which will be our future research direction.
As shown in Figure 4, for example (a), when there is no paraphrased prompt, the result obtained from text and image information is Neutral. When there is no image prefix, the image pre-trained em- bedding dominates the prediction results Posi- tive. Both of these two prediction results do not match the correct sentiment label Negative. For example (b), there are multiple targets that require sentiment classification. The sentiment labels pre- dicted for place name Bahamas are Neutral, and adding uppropriate prompts to both image and text can be predicted correctly. For examples (c) and (d), they are similar to example (b).
A Acknowledgments
This work is partially supported by NSFC, China (No.62276196).
References
Damian Borth, Rongrong Ji, Tao Chen, Thomas M. Breuel, and Shih-Fu Chang. 2013. Large-scale vi- sual sentiment ontology and detectors using adjective noun pairs. In ACM Multimedia Conference, MM '13 Barcelona, Spain, October 21-25, 2013, pages 223-232 ACM.
These four samples further confirm the useful- ness of our unified model. It can assign specific centiment-relater semanties to an entity via apply- ing a paraphrased prompt. And prefix tuning of images can obtain better task-specific image em- bedding than image pre-trained embedding.
Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng. Yunzhi Yao. Chuanqi Tan, Fei Huang. Luo Si, and Huajun Chen. 2022. Knowprompt: Knowledge- aware prompt-tuning with synergistic optimization for relation extraction. WWW '22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 29, 2022, pages 2778 2788-27788--1 ACM.
5 Conclusion and Future Work
There are currently two formats for target-oriented multimodal sentiment classification: entity-level and aspect-level. Our analysis shows that this bar- rier is superfluous. By incorporating paraphrased prompt and prefix vector into the multimodal input, the proposed model, i.e., UnifiedTMSC, unifies the two types of TMSC tasks. We conduct experiments on four datasets, and the results demonstrate the superiority and efficacy of our UMSS
Jian Cui. Lin Li, and Xiaohui Tao. 2023. Be-or- not prompt enhanced hard negatives generating for memes category detection. In IEEE International Conference Multimedia and Expo, ICME 2023, Brishane, Australia, July 10-14, 2023, pages 174- 179. IEEE.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. pages 4171-4186. Association for Compu- tational Linguistics.
Our ongoing effort will primarily concentrate two issues. One is to investigate how to de- sign paraphrasing nule to automatically generate paraphrased prompts without depending on human labor. The other is to investigate the generalizabil- ity of our model to see whether it can be used in other multimodal studies. In addition, we notice that the ruto-regressive model XLNet can alleviate the problem of generating non-label words, and our future work will consider this.
Tianhao Gao, Jun Fang, Hanyu Liu, Zhiyuan Liu, Chao Liu. Pengzhang Liu, Yongjun Bao, and Weipeng Yan. 2022. LEGO-ABSA: A prompt- based task assem- blable unified generative framework for multi-task aspect-basee based sentiment analysis. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 7002-7012. Inter- national Committee on Computational Linguistics.
Limitations
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2022. PTR: prompt tuning with rules for text classification. Al Open, 3:182-192.
Our model has three limitations. The first one is that the design of paraphrased prompts relies on human experience. Although our paraphrasing rule is designed based on the relative position and syn- onym substitution, manual experience is still re- quired to obtain paraphrased prompts that comply
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas,

---

NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Yan Ling, Jianfei Yu, and Rui Xia. 2022. Vision- language pre- vee-training for multimodal aspect-based sentiment analysis. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 2149-2159. Association for Computational Linguisties.
Brendan Jou, Tao Chen, Nikolaos Pappas, Miriam Redi, Mercan Topkara, and Shih-Fu Chang. 2015. Visual affect around the world: A large-scale multilingual visual sentiment ontology. In Proceedings of the 23rd Annual ACM Conference on Mullimedia Conference, MM 15, Brisbane, Australia, October 26- 30, 2015. pages 159-168. ACM.
Bin Liu, Tao Lin, and Ming Li. 2023a. Enhancing aspect-category sentiment analysis via syntactic data augmentation and knowledge enhancement. Knowl. Based Syst., 2644110333
Xincheng Ju, Dong Zhang, Rong Xiao, Junhui Li, Shoushan Li, Min Zhang, and Guodong Zhou. 2021. Joint multi- modal aspect-sentiment analysis with aux- iliary cross-modal relation detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Vir- tual Event / Punta Cana, Dominican Republic. 7-11 November, 2021, pages 4395-4405, Association for Computational Linguistics.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang.
Hiroaki Hayashi, and Graham Neubig, 2023b. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55(9): 1-::5-555:35.
Yuhang Liu, Wei Wei, Daowan Peng, and Feida Zhu. 2022. Declaration-bused prompt tuning for visual question answering. In Proceedings of the Thirty- First International Joint Conference on Artificial In- telligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 3264 3270. ijcai.org.
Zaid Khan and Yun Fu. 2021. Exploiting BERT for multimodal target sentiment classification through input space translation. In MM '2:: ACM Multimedia Conference, Virtual Event, China, October 20 24, 2021, pages 3034-3042. ACM.
Sijie Long, Lin Li, Jingling Yuan, and Jianquan Liu. 2022. Momnet: Gender prediction using mechanism of working memory. In Proceedings of the 31st ACM International Conference Information & Knowl- edge Management, Atlanta, GA, USA, October 17-21, 2022, pages 4259-4263. ACM.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sab- harwal, Oyvind Tafjord, Peter Clark, and Hannanch Hajishirzi. 2020. Unifiedqa: Crossing format bound- aries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1896-1907. Association for Computational I Linguistics.
Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018 Visual attention model for name tagging in multimodal social media. In Proceedings of the 56th Annuul Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume I: Long Papers, pages 1990-1999. Association for Computational Linguistics.
Chengxi Li, Feiyu Gao. Jiajun Bu, Lu Xu, Xiang Chen, Yu Gu, Zirui Shao, Qi Zheng, Ningyu Zhang, Yong- pan Wang. and Zhi Yu. 2021. Sentiprompt: Senti- ment knowledge enhanced prompt-tuning for aspect- based sentiment analysis. CoRR, 0tts/2109.08006
Yaojie Lu, Qing Liu, Dai Dai. Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Uni- fied structure generation for universal information extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 5755-5772. Association for Computational Linguistics.
Guowei Li, Fugiang Lin, WangQun Chen, DiWen Dong, and Bo Liu. 2022a. Prompt-based learning for aspect- level sentiment classification. In Neural Information Processing- 29th International Conference, ICONIP 2022, Virtual Event, November 22-26. 2022, Pro- ceedings, Part III, volume 13625 of Lecture Notes in Computer Science, pages 509-520. Springer.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, 1CML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2022b. A survey on deep learning for named en- tity recognition. IEEF Trans. Knowl. Data Eng., 34(1):50-70.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Longuage Processing, ACL/JJCNP 2021, (Volume I: Long Pupers), Virtual Event, August 1-6, 2021, pages 4582- 4597. Association for Computational Linguistics.
Ronald Seoh, Ian Birle, Mrinal Tak, Haw-Shiuan Chang, Brian Pinette, and Alfred Hough. 2021. Open aspect target sentiment classification with natural language

---

prompts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-11 November, 2021, pages 6311 1- 6322. Association for Computational Linguistics.
Symposium on Educational Advances in Artificial In- telligence, EAAI 2019, Honolulu. Hawaii, USA, Jan- wary 27 - February 1, 2019, pages 371-378. AAAI Press.
Hao Yang, Yanyan Zhao, and Bing Qin. 2022. Face- sensitive muage-to-ematiionl--ee cross-modal trans- lation for multimodal aspect-based sentiment anal- ysis. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, De- cember 7-11, 2022, pages 3324-3335. Association for Computational Linguistics.
Zhengxin Song, Yun Xue, Donghong Gu, Haolan Zhang, and Weiping Ding. 2023. Target-oriented multimodal sentiment classification by using topic model and gating mechanism. Inf. J. Mach. Learn. Cybern., 14(7):228--222 2229..
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. In Advances in Neural Information Pro- cessing Systems 34: Annual Conference on Neural Information Processing Systems 1021, NeurlPS 2021, December 6-14, 2021, virtual, pages 200-212.
Yuan Yao, Ao Zhang, Zhengyan Zhang. Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2021. CPT: col- orful prompt tuning for pre-trained vision-language models. CORR, abs/2109.11797.
Jianfei Yu and Jing Jiang. 2019. Adapting BERT for target-oriented multimodal sentiment classification. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019. Macao, China, August 10-16, 2019, pages 5408 5414. ijcai.org.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Jianfei Yu, Jing Jiang, and Rui Xia. 2020. Entity- sensitive attention and fusion network for entity-level multimodal sentiment classification. IEFE ACM Trans. Audio Speech anng Process., 28:429-439.
Jiawei Wang. Zhe Liu, Victor S. Sheng, Yuqing Song, and Chenjian Qiu. 2021. Saliencybert: Recurrent attention network for target-oriented multimodal sen- timent classification. In Puttern Recognition and Computer Vision 4th Chinese Conference, PRCV 2021, Beijing, China, October 29- November I, 2021, Proceedings, Part III. volume 13021 of Lecture Notes in Computer Science, pages 151 Springer.
Yuhai Yu, Hongfei Lin, Jiana Meng, and Zhehuan Zhao. 2016. Visual and textual sentiment analysis of a microblog using deep convolutional neural networks. Algorithms, 92):41.
Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named en- tity recognition in tweets. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelli- gence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial In- telligence (EAAI-18, (8), New Orleans, Louisiana, USA, February 2-7, 2018 pages 6674-5681, AAAI Press.
Luwei Xiao, Ejian Zhou, Xingjiao Wu, Shuwen Yang, Tianlong Ma, and Liang He. 2022. Adaptive multi- feature extraction graph convolutional networks for multimodal target sentiment analysis. In IEFE Inter- national Conference on Multimedia and Expo, ICME 2022, Taipei, Taiwan, July 18-22, 2022, pages 1-6. IBEE.
Zhe Zhang. Zhu Wang. Xiaona Li, Nannan Liu, Bin Guo, and Zhiwen Yu. 2021. Modalnet: an aspect- level sentiment classification model by exploring mul- timodal data with fusion discriminant attentional net- work. World Wide Web, 24(6):995-
Nan Xu and Wenji Mao. 2017, Multisentinet: A deep semantic network for multimodal sentiment analy- sis. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM 2017, Singapore, November 06 10, 2017. ACM.
Fei Zhao, Zhen Wu, Siyu Long, Xinyu Dai, Shujian Huang, and Jiajun Chen. 2022. Learning from adjective-noun pairs: A knowledge-enhanced frame- work for target-oriented multimodal sentiment clas- sification. Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 6784-6794, International Committee on Computational Linguistics.
Nan Xu, Wenji Mao, and Guandan Chen. 2018. A co- memory network for multimodal sentiment analysis. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018 Ann Arbor, MI. USA, July 08-12, 2018 pages 929-932. ACM.
Nan Xu, Wenji Mao, and Guandan Chen. 2019, Multi- interactive memory network for aspect based multi- modal sentiment analysis. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-Firss Innovative Applications of Artificial In- telligence Conference, IAAI 2019, The Ninth AAAI
Jie 7hou, Jiabao Zhao, Jimmy Xiangji Huang, Qin- min Vivian Hu, and Liang He. 2021. MASAD: A large-scale dataset for multimodal aspect-based sen- timent analysis. Newrocompuiing, 455:47-58.

---

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Learning to prompt for vision- language models. Int. J. Comput. Vis., 130(9):2337- 2348.
A.2 Dataset Details
(1) Twitter-2015 and Twitter-2017. The two Twitter datasets include user tweets released during 2014-2015 and 2016-2017, collected by Zhang et al. (2018) and Lu et al. (2018). Since the two publicly available multimodal datasets Twitter-2015 and Twitter-2017 only provide annotated targets in each tweet, Yu and Jiang (2019) ask three domain experts to annotate the sentiment towards each target, and take the ma- jority label among the three annotators the gold label. These two datasets contain multi- modal tweets and annotated mentioned entities in the text, well the sentiment polarity of each entity, including positive, neutral, and negative. Each multimodal tweet consists of a text and a corresponding image. We follow their official splitting for training, validation and testing. The statistics of Twitter-2015 and Twitter-0017 are in Table 8.
A Appendix
A.1 Baselines
(1) TomBERT (ICAAI Yu and Jiang (2019)), which employ BERT for the inter-modal in- teractions and Image matching layer to obtain a target-sensitive visual.
(2) SaliencyBERT (PRCV, Wang et al. (2021)), which design recurrent attention mechanism to capture the inter-modality dynamics.
(3) CapTrBERT (ACM Multimedia, Khan and Fu (2021)), a BERT-based model translating the image into caption and fusing the caption and input text-entity pair.
(2) Multi-ZOL. This dataset is collected by Xu al. (2019) from the Chinese website ZOL.com. The website consists of 40 large channels, including news, shopping malls, hardware, mobile phones, and more. They searched from pages 1 to 50 in the mobile phones channel. For each phone, only the re- views from the first 20 pages were crawled. The Multi-ZOL dataset contains 5288 multi- modal reviews and each multimodal review contains a textual content, image set, and at least one but no more than six aspects. The six aspects are cost performance, performance configuration, battery life, appearance, photog- raphy effect, and screen. For each aspect, the review has an integer sentiment score from to 10, which is regarded as the sentiment label in experiment. Actually, convert digital sentiment labels into Chinese characters, such as '10' 'ten'. Combining each aspect with the multimodal review, 28469 pairs of aspect review samples can be obtained. We randomly partition the dataset into training (80%), de- velopment (10%), and testing (10%) sets. The statistical information after dividing the dataset is shown in Table 9.
(4) JML-MASC (EMNLP, Ju et al. (2021)), which multi-task learning method with the cross- modal relation detection.
(5) VLP-MABSA (ACL, et al. (2022)), a task- specific pre-training vision-language model.
(6) FITE-DE--Lrge (EMNLP. Yang et al. (2022)). introducing FITE method focus captur- ing emotional cues through facial expressions.
(7) MIMN (AAAL, Xu et al. (2019)), using memory network to model multimodal data and learn the interactive influences in cross- modality and self- eeof-modality.
(8) MMAP (Neurocomputing, Zhou et al. (2021)), learning the interaction between text and im- age, text and aspect, and image and aspect through three interactive mechanisms.
(9) ModalNet (WWW, Zhang et al. (2021)), de- signing discriminant matrix to supervise the fusion of inter-modal information.
(10) CLIP (INT J COMPUT VISION, Zhou al. (2022)), multimodal pre-training model, con- verts image classification tasks into image-text matching tasks using comparative learning.
MASAD. This dataset was collected and pub- lished by Zhou al. (2021) based on the publicly available Visual Sentiment Ontology (VSO) dataset (Borth et al., 2013) and Multi-

---

Twitter-2015
Twitter-2017
#Positive
#Neutral
#Negative
Total
#Positive
#Neutral
#Negative
Total
Train
928
1883
368
3179
416
1638
3562
Dev
303
679
149
1122
144
517
1176
Test
317
607
113
1037
493
168
573
1234
Table 8: The statistics of Twitter-2015 and 1witter-2017 datascts.
#1
42
#3
#4
#5
#6
47
#8
#9
#10
Total
Max #lmage
Min #lmage
Train
6
1163
6
1190
3
3509
0
6952
0
9948
22775
111
Dev
139
0
150
0
433
0
872
0
1253
2847
111
Test
132
0
151
0
413
0
871
0
1280
2847
72
Table 9: The statistics of Multi-ZOL. dataset after partitioning. 'Max #Image' refers to the maximum number of image corresponding to the review, while 'Min #Image' is minimum quantity.
Train
Dev #Nogative
Test
#Positive
#Negative
Total
#Positive
Total
#Positive
#Nogative
Total
Food
2122
392
2514
239
41
280
592
109
701
Buildings
969
874
1843
109
96
205
273
245
518
Goods
2398
1512
3910
273
162
435
743
512
1255
Animal
2728
1979
4707
295
229
524
1126
670
1796
Human
1805
1648
3453
194
190
384
503
464
967
Plant
2542
2341
4883
277
266
543
1269
947
2216
Scenery
3222
1760
4982
378
176
554
907
490
1397
Total
15786
10506
26292
1765
1160
2925
5413
3437
8850
Table 10: The statistical information of MASAD datascts. The seven domains are Food, Buildings, Goods, Animal, Human, Plant, Scenery.
lingual Visual Sentiment Ontology (MVSO) dataset (Jou et al., 2015). Zhou et al. (2021) se- lected the samples from a partial VSO dataset (approximately 120k samples) that can express significant sentiments (about 38k samples) and categorized them into 7 domains, resulting in the MASAD dataset. The seven domains of the MASAD are food, buildings, goods, ani- mal, human, plant, and scenery. Each domain encompasses multiple aspects, such the an- imal domain, including cat, dog, horse, and so on. According to our statistics, there are a total of 57 predefined aspects. This dataset only includes training and testing sets, both containing positive and negative samples. We partition each domain in the training set into new training set and validation set in 9:1 ra- tio, keeping the original testing unchanged. The statistics of MASAD are in Table 10.
mal and plant.
In addition, there are instances of duplicate data in the testing and training sets in this dataset. In the domain of animal and plant, according our statistics, about 81.6% and 62.2% of the data in the testing set has ap- peared in the training set. order to compare more fairly with other baseline models, we con- duct experiments in other domains besides ani-


Reviews:
[{"review_id": "TSrQz1p7up", "reviewer": "Reviewer_xjQS", "summary": "", "questions": "1. The usage of synonyms is unclear. What does the synonym consist of? Are sentiment and emotion have their corresponding synonym dictionaries? Will these synonyms overlap? \n2. Which prompt will the author use for the final inference? \n3. Is all label only require one token to be generated? For example, does the vocabulary have a single token for the sentiment label \"10\"? \n4. How do you deal with ill-format decode labels? Do you apply post-process normalization? For example, the ground truth is \"positive\", but the model generates \"positively.\"\n5. Significant tests are needed to prove the effectiveness, such as single-tail t-tests.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **C1**: Novelty is somehow limited and more like a technical report; rule-based prompt data enhancement is the main innovation. \n- **R1**: Thanks for your serious comment. Our work has improved by 1.5%, 1.7%, 3.77%, and 2.19% on Macro-F1 on the four datasets of Twitter-2015, Twitter-2017, Multi-ZOL, and MASAD, respectively. In addition, our work has transformed the original classification task into the MLM pre-training task by adding prefix vectors and paraphrased prompts to the image-text data, which has changed the task format. Therefore, it is crucial to design paraphrased prompts that are closer to the multimodal sentiment prediction task, which is also **our main innovation: designing paraphrased prompts that are in line with the task description under the prompt paradigm**.\n\n- **C2**: It does not follow the pattern of unified training followed by direct inference across multiple tasks and datasets. (without dataset-specific fine-tuning) \n- **R2**: Thank you for reading our paper carefully and providing comments. For fair experimental comparison, each task is trained on their own dataset. If we conduct unified training and inference for multiple datasets, it increases the amount of training data. We add some new experimental results by training these tasks together. Taking the Twitter-2015 and Twitter-2017 as the example, the experimental results of unified training and inference are as follows. (The paraphrased prompt used is “P1: {target} express a [MASK] sentiment.”) \nTwitter2015: (Acc: 81.3, Macro-F1: 79.8)\nTwitter2017: (Acc: 78.1, Macro-F1: 77.8)\nThe results of separate training are: \nTwitter2015: (Acc: 80.2, Macro-F1: 77.6)\nTwitter2017: (Acc: 76.9, Macro-F1: 76.7)\n\n- **C3**: Important details are unclear and should be clarified. The description of the model's inner workings and implementation details could be more elaborate for the readers to replicate the study. \n- **R3**: Thank you for your suggestion. We will supplement the experimental details and open our code in GitHub. About the paraphrased prompt for each task, we design three paraphrased prompts P1, P2  and P3 . In addition, three traditional prompts P1’ , P2’ and P3’ are used to compare with paraphrased prompts. The results are reported in Table 6. The results indicate that the effect of paraphrased prompts is superior to the traditional prompts, and P1, P2  and P3 are comparable. In subsequent task, one of the paraphrased prompts can be selected for experimentation. P1, P2 , P3  P1’ , P2’ and P3’ are follows:\n“P1: {target} express a [MASK] sentiment.”,\n“P2: The emotion of {target} is [MASK].”,     \n“P3: A [MASK] sentiment is expresses towards to {target}”. \n“P1’: I feel the {target} is [MASK].”,\n“P2’: The {target} made me feel [MASK].” \n“P3’: I [MASK] the {target}.”\nIn addition, when the prediction label is a real value, we change the prompt P2 to “The score of the {target} is [MASK].” for the Multi-ZOL dataset. For the MASAD dataset, P2 is selected for experimentation. \n\n- **C4**: The usage of synonyms is unclear. What does the synonym consist of? Are sentiment and emotion have their corresponding synonym dictionaries? Will these synonyms overlap?\n- **R4**: Thank you for your comment. We‘re sorry that we don’t explain the synonyms clearly. In **Bing Dictionary**, when the meaning of 'sentiment' is 'a feeling or emotion', the similar words are 'feeling’ and 'emotion' respectively. When the meaning of 'sentient' is 'a view of or attitude towards a situation or event; an opinion', the similar words of ‘sentiment’ are such as ‘view, feeling, opinion’, etc. These similar words form a dictionary of synonyms, and we take the word from this dictionary to replace the ‘sentient’ of the seed prompt.\n\n- **C5**: Which prompt will the author use for the final inference?\n- **R5**: Thank you for pointing this out. Refer to R3, we design three different paraphrased prompts from traditional prompts, and the effects of these three paraphrased prompts are **comparable**. Therefore, we can choose any one of them for training and inference in the experiment. For example, for the MASAD dataset, we use P2 to conduct experiment.\n\n- **C6**: Is all label only require one token to be generated? For example, does the vocabulary have a single token for the sentiment label \"10\"?\n- **R6**: Thanks for the review. We ‘re sorry that we didn't explain it clearly. In our actual experiment, we convert digital sentiment labels into Chinese characters, such as \"10\" to \"ten\", and the converted Chinese characters correspond to a single token in the vocabulary. We will supplement it in A.2 of Appendix.  \n\n- **C7**: How do you deal with ill-format decode labels? Do you apply post-process normalization? For example, the ground truth is \"positive\", but the model generates \"positively.\" \n- **R7**: Thank you for your comment. In the inference stage, if non-label words are generated, we use the **answer engineering** in prompt to map the non-label words to label words. For example, when we use “P1’: I feel the {target} is [MASK].” for training and prediction, the words generated by the model are “good”, “ok”, and “bad”. Therefore, we need to use answer engineering to map these words to label words “positive”, “neutral” and “negative”. We notice that the **autoregressive** model XLNet can alleviate the problem of generating non-label words, and our future work will consider this.\n\n- **C8**: Significant tests are needed to prove the effectiveness, such as single-tail t-tests.\n- **R8**: Thank you for your suggestion. We will conduct the single-tail t-tests to verify the effectiveness of our model and supplement the results in the experimental results tables. Taking the Multi-ZOL dataset as an example, the P-Values between our model and the baselines are as follows: \nMIMN ：0.0316   \nModalNet:  0.0266  \nCLIP:    0.0092   \n\n- **C9**: Typos Grammar Style And Presentation Improvements:\n- **R9**: Thank you for carefully reading our paper and pointing out these issues. We will make revisions to the paper based on your suggestions and we will adopt **professional proof-reading service**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This proposes UnifiedTMSC uses a unique combination of prompt-based language modelling and descriptive prompt paraphrasing to achieve high performance on both entity-level and aspect-level tasks. \n2. The rigorous experiments conducted on four different datasets lend credibility to the approach and suggest it could be a promising avenue for future research in TMSC.", "reasons_to_reject": "1. Novelty is somehow limited and more like a technical report; rule-based prompt data enhancement is the main innovation.\n2. It does not follow the pattern of unified training followed by direct inference across multiple tasks and datasets. (without dataset-specific fine-tuning)\n3. Important details are unclear and should be clarified. The description of the model's inner workings and implementation details could be more elaborate for the readers to replicate the study.", "questions_to_authors": "1. The usage of synonyms is unclear. What does the synonym consist of? Are sentiment and emotion have their corresponding synonym dictionaries? Will these synonyms overlap? \n2. Which prompt will the author use for the final inference? \n3. Is all label only require one token to be generated? For example, does the vocabulary have a single token for the sentiment label \"10\"? \n4. How do you deal with ill-format decode labels? Do you apply post-process normalization? For example, the ground truth is \"positive\", but the model generates \"positively.\"\n5. Significant tests are needed to prove the effectiveness, such as single-tail t-tests.", "paper_topic_and_main_contributions": "This paper presents the UnifiedTMSC model for Target-Oriented Multimodal Sentiment Classification (TMSC) that leverages descriptive prompt paraphrasing to eliminate the division between entity-level and aspect-level tasks. \nThe model creates a unified task modelling framework using task paraphrasing and image prefix tuning, suggesting that sentiment polarity of the target is context-dependent rather than type-dependent. UnifiedTMSC demonstrates good performance on four diverse datasets, indicating its broad applicability and effectiveness.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Hu, Guimin, et al. \"UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* The overall writing could be improved in terms of clarity and formality.\n* Adding the decoding example in Figure 2 is recommended for better understanding.\n* Formula (4) is quite confusing; maybe it could be omitted.\n* Please provide definitions for variables at their first point of use. For instance, the text embedding 'R' is initially mentioned in line 212 but is not clearly defined until line 308. This sequence could confuse readers and disrupt the flow of the paper. Please consider revising for clarity.\n* Line 297: spliced —> concat?"}}, {"review_id": "EWlyk5xEYl", "reviewer": "Reviewer_KMtj", "summary": "", "questions": "1) Why do the authers use chinese-roberta-wwm-ext as textencoder for Multi-ZOL? How about the performance of BERT? The baselines should have the same base model for fair comparsion.\n2) Why can UnifiedTMSC improve the performance of TMSC. The authors use a unified model to model two subtasks. However, there is no module to improve the performance for TMSC and these two tasks are trained independently.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **C1**: The novelty of the proposed model is limited. The authors use discrete and continuous prompts to fine the language model, which is widely used in language model for aspect-based sentiment analysis [1-3] and TMSC [4]. The author should cite these references and compare with them.\n- **R1**: Thank you for carefully reviewing our paper and identifying the issues. We will supplement some missing references based on your suggestion. Our data modality is different from that of the papers [1-3]. We will consider conducting experiments without the image modality and comparing these papers in the future. For paper [4], they concatenated image encoding and text prompt, and we use cross-attention to fuse them. We have tried to directly concatenate image and text features before, but this method did not work well on our datasets. We will add our discussion to the paper.\n\n- **C2**: Why do the authors use chinese-roberta-wwm-ext as text encoder for Multi-ZOL? How about the performance of BERT? The baselines should have the same base model for fair comparison.\n- **R2**: Thank you for pointing this out. For fair comparison, we re-conducted the experiment using BERT-base-chinese as the text encoder for Multi-ZOL, while we will modify the experimental results and implementation details. The experimental results using BERT-base-chinese as the text encoder on the Multi-ZOL dataset are as follows. In our experiment, only this dataset is Chinese, and the following results indicate that using the Roberta pre-training model is more effective.\nIn our paper:  Multi-ZOL: (hf1/chinese-roberta-wwm-ext , Acc: 72.53, Macro-F1: 65.97)\nNew Result:   Multi-ZOL: (BERT-base-chinese , Acc: 70.46, Macro-F1: 64.71)\n\n- **C3**: Why can UnifiedTMSC improve the performance of TMSC. The authors use a unified model to model two subtasks. However, there is no module to improve the performance for TMSC and these two tasks are trained independently. \n- **R3**: Thank you for carefully reviewing our paper and providing comment. We can conduct unified training and reference, or we can conduct training and reference separately. The former increases the amount of data to improve the effect, while the latter aims to compare the effects of different datasets on our model more fairly. For fairness in comparison with the baseline, we report the results of independent training for two tasks in the paper. Taking the Twitter-2015 and Twitter-2017 as the example, the experimental results of unified training and inference are as follows. (The paraphrased prompt used is “P1: {target} express a [MASK] sentiment.”) \nTwitter2015: (Acc: 81.3, Macro-F1: 79.8)\nTwitter2017: (Acc: 78.1, Macro-F1: 77.8)\nThe results of separate training are: \nTwitter2015: (Acc: 80.2, Macro-F1: 77.6)\nTwitter2017: (Acc: 76.9, Macro-F1: 76.7)\n\n[1] Seoh R, Birle I, Tak M, et al. Open Aspect Target Sentiment Classification with Natural Language Prompts[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021: 6311-6322. \n[2] Li G, Lin F, Chen W, et al. Prompt-Based Learning for Aspect-Level Sentiment Classification[C]//International Conference on Neural Information Processing. Cham: Springer International Publishing, 2022: 509-520.\n[3] Li C, Gao F, Bu J, et al. Sentiprompt: Sentiment knowledge enhanced prompt-tuning for aspect-based sentiment analysis[J]. arXiv preprint arXiv:2109.08306, 2021.\n[4] Yu Y, Zhang D. Few-shot multi-modal sentiment analysis with prompt-based vision-aware language modeling[C]//2022 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2022: 1-6.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The author propose a UnifiedTMSC model to reformulate aspect-based and target-based sentiment analysis.\n2) This paper is well written and easy to follow.\n3) Series of experiments indicate the effectiveness of their model.", "reasons_to_reject": "1) The novelty of the proposed model is limited. The authors use discrete and continuous prompts to fine the language model, which is widely used in language model for aspect-based sentiment analysis [1-3] and TMSC [4]. The author should cite these references and compare with them.\n", "questions_to_authors": "1) Why do the authers use chinese-roberta-wwm-ext as textencoder for Multi-ZOL? How about the performance of BERT? The baselines should have the same base model for fair comparsion.\n2) Why can UnifiedTMSC improve the performance of TMSC. The authors use a unified model to model two subtasks. However, there is no module to improve the performance for TMSC and these two tasks are trained independently.", "paper_topic_and_main_contributions": "This paper proposes a unified TMSC model for aspect-based and target-based multimodal sentiment analysis tasks. They model these two tasks via descriptive prompt paraphrasing based on language model. Moreover, they conduct extensive experiments on four TMSC datasets. The results show their model outperform the existing baselines.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Seoh R, Birle I, Tak M, et al. Open Aspect Target Sentiment Classification with Natural Language Prompts[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021: 6311-6322.\n[2] Li G, Lin F, Chen W, et al. Prompt-Based Learning for Aspect-Level Sentiment Classification[C]//International Conference on Neural Information Processing. Cham: Springer International Publishing, 2022: 509-520.\n[3] Li C, Gao F, Bu J, et al. Sentiprompt: Sentiment knowledge enhanced prompt-tuning for aspect-based sentiment analysis[J]. arXiv preprint arXiv:2109.08306, 2021.\n[4] Yu Y, Zhang D. Few-shot multi-modal sentiment analysis with prompt-based vision-aware language modeling[C]//2022 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2022: 1-6."}}, {"review_id": "0b6MtoqqJJ", "reviewer": "Reviewer_5uYa", "summary": "", "questions": "A. Why does the structure for sentiment classification differ between the training and inference stages? Is this difference based on experimental comparisons?\n\nB. Has the choice of setting the number of layers for cross-attention to 4 been experimentally validated, or is it subjectively determined? Moreover, does directly using a 12-layer Transformer have any impact?\n\nC. It might be worth considering the utilization of LLMs for generating paraphrased prompts and conducting comprehensive tests to assess the robustness of UnifiedTMSC.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **C1**: The experiments do not encompass several recent methodologies involving large vision-language models (VLMs), such as Flamingo/OpenFlamingo and InstructBLIP (although relatively recent, it would be valuable to include them), which have demonstrated impressive zero/few-shot in-context learning performances on numerous vision-language tasks, including visual QA—tasks that the TMSC problem can be framed as.\n- **R1**: Thank you for your careful review. We also believe that zero/few shot in-context learning is a challenging direction. Thank you for your suggestions and guidance, and we will continue to try and research in this area.\n\n- **C2**: The primary innovation of the paper lies in the deliberate design of task-related paraphrased prompts, while other works primarily combine existing methods. In this respect, the innovation presented in this paper appears to be somewhat limited. \n- **R2**: Thanks for your serious review. We transform the original aspect/entity level multimodal sentiment classification into the MLM pre-training task by adding prefix vectors and paraphrased prompts to the image-text data, which changed the task format. Therefore, it is crucial to design paraphrased prompts that are closer to the original MSC task, which is also our main innovation: **designing paraphrased prompts that are in line with the task under the prompt paradigm**.\n\n- **C3**: The paper lacks a clear demonstration of the usefulness of the mentioned task description and seed prompt for subsequent paraphrased prompt design. As it stands, these two steps appear redundant. \n- **R3**: Thanks for your comment. We are sorry that we don’t explain this clearly. Since we converted the original sentiment classification task into the MLM pre-trained task through adding prompts, it is crucial to design prompts that fit the original task. **As shown in the left part of Figure 2**, we obtain seed prompt and keyword from the task description. Subsequently, the seed prompt is transformed through the paraphrasing rule, and guide the generation of paraphrased prompts that are close to the task description. We will add a more detailed explanation in section 3.3.\n\n- **C4**: The paper lacks essential experimental details, such as whether the Transformer used for cross-attention loads pre-training parameters, and the number of paraphrased prompts utilized for each data point during training. \n- **R4**: Thank you for carefully reading and pointing out these shortcomings. We will supplement these details in the implementation details section. The pre-training parameters of Transformer for cross-attention are not loaded. For the Twitter2015 and Twitter2017, we use three paraphrased prompts P1, P2  and P3 obtained based on the paraphrasing rule, and compare them with three arbitrary prompts P1’ , P2’ and P3’. The results are reported in Table 6. The results indicate that the effect of paraphrased prompts is superior to the traditional prompts, and P1, P2  and P3 are comparable. In subsequent task, one of the paraphrased prompts can be selected for experimentation. P1, P2 , P3  P1’ , P2’ and P3’ are follows:\n“P1: {target} express a [MASK] sentiment.”,\n“P2: The emotion of {target} is [MASK].”,     \n“P3: A [MASK] sentiment is expresses towards to {target}”. \n“P1’: I feel the {target} is [MASK].”,\n“P2’: The {target} made me feel [MASK].” \n“P3’: I [MASK] the {target}.”\nIn addition, when the prediction label is a real value, we change the prompt P2 to “The score of the {target} is [MASK].” for the Multi-ZOL dataset. For the MASAD dataset, P2 is selected for experimentation.\n\n\n- **C5**: Why does the structure for sentiment classification differ between the training and inference stages? Is this difference based on experimental comparisons?\n- **R5**: Thank you for your comment. We apologize for any errors in our expression here. Our training process and inference stage are consistent. \n\n\n- **C6**:  Has the choice of setting the number of layers for cross-attention to 4 been experimentally validated, or is it subjectively determined? Moreover, does directly using a 12-layer Transformer have any impact? \n- **R6**: Thanks for your comment. During our experiment, we set the cross-attention layers to 4, 6, and 12, respectively. We find that setting to 4 has the best effect, while setting the cross-attention layers to 6 and 12 resulted in poorer experimental results. Taking the Twitter-2015 dataset as an example, the following are the experimental results when the number of cross-attention layers is set to 4, 6, and 12 (the paraphrased prompt used is “P1: {target} express a [MASK] sentiment.”)\nTwitter2015: (4 layers, Acc: 80.2, Macro-F1: 77.6), (6 layers, Acc:79.6 Macro-F1:77.2), (12 layers, Acc: 80.1, Macro-F1:77.4)\n\n- **C7**: It might be worth considering the utilization of LLMs for generating paraphrased prompts and conducting comprehensive tests to assess the robustness of UnifiedTMSC.\n- **R7**: Thank you for your suggestion. Our paraphrasing rule is designed based on the relative position and synonym substitution, which essentially requires manual experience. One of our future works is to use LLMs to automatically generate paraphrased prompts and avoid relying on human labor.   \n\n- **C8**: Missing References\n- **R8**: Thank you for carefully reviewing our paper and identifying the issues. We will supplement some missing references based on your suggestion. We started **training from scratch** and only used datasets from related tasks for training and inference. SMP [1] is a multimodal pre training framework for multimodal sentiment analysis, which uses large image-text dataset for pre-training tasks and then performs fine-tuning in downstream tasks.\n\n[1] Junjie Ye, Jie Zhou, Junfeng Tian, Rui Wang, Jingyi Zhou, Tao Gui, Qi Zhang, Xuanjing Huang: Sentiment-aware multimodal pre-training for multimodal sentiment analysis. Knowl. Based Syst. 258: 110021 (2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. In comparison to several existing non-large language models, the method proposed in this paper attains superior results across four datasets.\n\n2. The paper not only compares the performance of its own model with other approaches but also examines various text paraphrased forms, enhancing the comprehensiveness of the experiment.\n\n3. In general, the paper's presentation flows smoothly, and the writing is clear.", "reasons_to_reject": "1. The experiments do not encompass several recent methodologies involving large vision-language models (VLMs), such as Flamingo/OpenFlamingo and InstructBLIP (although relatively recent, it would be valuable to include them), which have demonstrated impressive zero/few-shot in-context learning performances on numerous vision-language tasks, including visual QA—tasks that the TMSC problem can be framed as.\n\n2. The primary innovation of the paper lies in the deliberate design of task-related paraphrased prompts, while other works primarily combine existing methods. In this respect, the innovation presented in this paper appears to be somewhat limited.\n\n3. The paper lacks a clear demonstration of the usefulness of the mentioned task description and seed prompt for subsequent paraphrased prompt design. As it stands, these two steps appear redundant.\n\n4. The paper lacks essential experimental details, such as whether the Transformer used for cross-attention loads pre-training parameters, and the number of paraphrased prompts utilized for each data point during training.\n", "questions_to_authors": "A. Why does the structure for sentiment classification differ between the training and inference stages? Is this difference based on experimental comparisons?\n\nB. Has the choice of setting the number of layers for cross-attention to 4 been experimentally validated, or is it subjectively determined? Moreover, does directly using a 12-layer Transformer have any impact?\n\nC. It might be worth considering the utilization of LLMs for generating paraphrased prompts and conducting comprehensive tests to assess the robustness of UnifiedTMSC.", "paper_topic_and_main_contributions": "This paper introduces UnifiedTMSC, a model designed for TMSC tasks. The method involves constructing a hard prompt on the text modality through task paraphrasing, and a soft prompt on the image modality using image prefix tuning. These prompts are then fused through cross attention to generate the final representation and enable classification. Experimental results demonstrate that UnifiedTMSC outperforms other methods on four datasets, yielding the best results.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. As for baselines, apart from large VLMs, there is also a scarcity of models designed specifically for TMSC tasks, such as SMP [1].\n\n2. In section 4.1, references to well-known models like BERT [2], RoBERTa [3], ResNet [4], and Transformer [5] are missing.\n\n[1] Junjie Ye, Jie Zhou, Junfeng Tian, Rui Wang, Jingyi Zhou, Tao Gui, Qi Zhang, Xuanjing Huang: Sentiment-aware multimodal pre-training for multimodal sentiment analysis. Knowl. Based Syst. 258: 110021 (2022)\n\n[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT (1) 2019: 4171-4186\n\n[3] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov: RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs/1907.11692 (2019)\n\n[4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun: Deep Residual Learning for Image Recognition. CVPR 2016: 770-778\n\n[5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin: Attention is All you Need. NIPS 2017: 5998-6008"}}]

Author rebuttal:

Meta-review: Based on the idea that sentiment polarity of the specific target is not governed by its type but its context, authors propose a unified model for target-oriented multimodal sentiment classification, UnifiedTMSC. It is prompt-based language modelling and performers well on four datasets over two target types.

Reviewers give all good (3) in soundness; two ambivalent (3) and one strong (4) in excitement. Reviewers have some concerns about novelty (the use of prompts for language modeling is somehow not so exciting), clarity of details and reproducibility. However, this paper is well written and the comprehensive experiments support good experimental results.
Decision: Accept-Findings