Few-shot Joint Multimodal Aspect-Sentiment Analysis Based on Generative Multimodal Prompt
Year: 2023

Abstract:
We have witnessed the rapid proliferation of multimodal data on numerous social media platforms. Conventional studies typically require massive labeled data to train models for Multimodal Aspect-Based Sentiment Analysis (MABSA). However, collecting and annotating fine-grained multimodal data for MABSA is tough. To alleviate the above issue, we perform three MABSA-related tasks with quite a small number of labeled multimodal samples. We first build diverse and comprehensive multimodal few-shot datasets according to the data distribution. To capture the specific prompt for each aspect term in a few-shot scenario, we propose a novel Generative Multimodal Prompt (GMP)1 model for MABSA, which includes the Multimodal Encoder module and the NStream Decoders module. We further introduce a subtask to predict the number of aspect terms in each instance to construct the multimodal prompt. Extensive experiments on two datasets demonstrate that our approach outperforms strong baselines on two MABSA-related tasks in the few-shot setting.

heading: 1 Introduction
text: The Multimodal Aspect-Based Sentiment Analysis (MABSA) task has garnered significant attention in recent times, as evidenced by several recent studies (Chandrasekaran et al., 2021; Zhang et al., 2022a; Zhu et al., 2022; Gandhi et al., 2023). In the literature, MABSA is typically divided into three subtasks: Multimodal Aspect Term Extraction (MATE), Multimodal Aspect-oriented Sentiment Classification (MASC), and Joint Multimodal Aspect-Sentiment Analysis (JMASA) (Wu et al., 2020a; Zhang et al., 2021a; Yu and Jiang, 2019; Khan and Fu, 2021; Ju et al., 2021; Ling et al., 2022). Given a text-image pair, MATE aims to extract all the aspect terms mentioned in the text, MASC focuses on detecting the sentiment
1https://github.com/YangXiaocui1215/GMP.
corresponding to each extracted aspect term, and JMASA is designed to extract aspect terms and their corresponding sentiments jointly. Previous studies on Multimodal Aspect-Based Sentiment Analysis (MABSA) primarily focus on leveraging extensive training data (full training datasets), with some works resorting to additional data to improve performance (Ju et al., 2021; Ling et al., 2022). However, collecting and annotating such massive multimodal data for MABSA is time-intensive and laborious (Zhou et al., 2021). Moreover, in realworld applications, only a limited amount of labeled data is commonly available. To address this challenge, PVLM (Yu and Zhang, 2022) and UP-MPF (Yu et al., 2022) introduce prompt-based learning into Multimodal Aspect-oriented Sentiment Classification (MASC) in a few-shot scenario. Based on limited sentiment categories (three categories), PVLM and UP-MPF convert MASC to masked language modeling (MLM) tasks. However, the prerequisite of MASC is that the aspect terms are known, which requires aspect term extraction in advance, typically performed by Multimodal Aspect Term Extraction (MATE) or Joint Multimodal Aspect-Sentiment Analysis (JMASA). Both JMASA and MATE tasks are challenging due to the unknown and varying number of aspect items in each sample, as well as the distinct content of each aspect. Therefore, applying MLM in the fewshot setting is unsuitable for JMASA and MATE tasks, as depicted in Fig. 1. This paper addresses the challenges of JMASA, MASC, and MATE in a text-image few-shot setting, and to the best of our knowledge, there are no dedicated studies dealing with JMASA and MATE tasks in the multimodal few-shot scenario.
Prior few-shot text classification tasks with limited classification labels have manually designed general prompts for the entire dataset to mine knowledge from pre-trained language models (PLM) (Shin et al., 2020; Hosseini-Asl et al., 2022;
11575
Zhang et al., 2022b). However, in the case of Joint Multimodal Aspect-Sentiment Analysis (JMASA) and Multimodal Aspect Term Extraction (MATE), where the content of each aspect term is unknown and assorted, manual prompts are infeasible for aspect extraction. To address this challenge, we propose a novel Generative Multimodal Prompt (GMP) model for few-shot Multimodal AspectBased Sentiment Analysis (MABSA), which includes the Multimodal Encoder (ME) module and the N-Stream Decoders (NSD) module. It is crucial to sample diverse and comprehensive data to build practical few-shot datasets in the multimodal fewshot setting. We construct few-shot training and development datasets by sampling data with combinations of different sentiments in instances, according to the data distribution, as shown in Table 1. Since the number of aspect terms in JMASA and MATE is unknown and vital, we leverage the Multimodal Encoder (ME) and Aspect-Num Decoder (AND) to predict the number of aspect terms as a subtask. The clues required for each aspect of an instance may vary. We generate aspect-oriented prompts for each aspect (aspect-level) using the ME and Aspectoriented Prompt Decoder (APD). Similarly, we use the ME and Sentiment-oriented Prompt Decoder (SPD) to generate sentiment-oriented prompts. As the sentiment categories in all datasets are limited, we only reserve the instance-level sentiment prompts. The caption of the image modality is also captured as the image prompt. Lastly, specific multimodal prompts for different tasks are constructed based on the image caption, the predicted number of aspect terms, aspect prompts, and sentiment prompts. We feed the multimodal embedding with the multimodal prompt into the Multimodal Encoder-Decoder based BART model (Lewis et al., 2020) to generate triplet sequences. Our main contributions are summarized as follows:
• We propose a novel Generative Multimodal Prompt (GMP) model to handle Joint Multimodal Aspect-Sentiment Analysis (JMASA), Multimodal Aspect Sentiment Classification (MASC), and Multimodal Aspect Term Extraction (MATE) in the multimodal few-shot setting. To our knowledge, we are the first to focus on JMASA and MATE tasks in a multimodal few-shot scenario.
• To tackle the challenge of unknown number of multimodal aspect terms and construct effective multimodal prompts, we employ multitasking and build the few-shot dataset by taking into ac-
count the distribution of sentiment categories for each dataset.
• We conduct extensive experiments on the constructed few-shot datasets, and our results demonstrate that our proposed model outperforms strong baselines on JMASA and MASC in the few-shot setting.

heading: 2 Related Work

heading: 2.1 Multimodal Aspect Sentiment Analysis
text: In contrast to coarse-grained sentiment analysis (sentence-level) (Yang et al., 2021b; Li et al., 2022), MABSA requires not only extracting aspect terms, but also recognizing the corresponding sentiment associated with each aspect. Early research focuses on different subtasks, including Multimodal Aspect Term Extraction (MATE) (Sun et al., 2020; Yu et al., 2020; Wu et al., 2020b; Zhang et al., 2021b; Chen et al., 2022) and Multimodal Aspect Sentiment Classification (MASC) (Yang et al., 2021a; Yu and Jiang, 2019; Khan and Fu, 2021). More recently, Ju et al. (Ju et al., 2021) propose Joint Multimodal Aspect-Sentiment Analysis (JMASA), which jointly performs aspect term extraction and sentiment classification. Yang et al. (Yang et al., 2022b) introduce Cross-Modal Multitask Transformer (CMMT) for MABSA. VLP (Ling et al., 2022) further extends this by resorting to additional pre-training data and designing multiple pretraining tasks to enhance JMASA performance. However, few works specifically address MABSA in the few-shot scenario. Although VLP has conducted low-resource experiments, it includes over 17,000 pre-training data and utilizes the full development dataset, which violates our starting point of adopting few-shot data.

heading: 2.2 Few-shot Learning with Pre-trained Language Model
text: Prompt-based language modeling is applied to solve different few-shot tasks with PLM in Natural Language Process (NLP) due to its powerful representation (Liu et al., 2021), such as text classification (Shin et al., 2020; Hosseini-Asl et al., 2022), text regression (Gao et al., 2021), and text generation (Li and Liang, 2021). Existing works introduce Multimodal Prompt-based Fine-tuning (MPF) methods into multimodal settings by MLM, like Frozen (Tsimpoukelli et al., 2021), PVLM (Yu and Zhang, 2022), and UP-MPF (Yu et al., 2022).
Different from few-shot MASC (PLVM and UPMPF), we simultaneously extract aspect terms and perform sentiment detection for each aspect in the multimodal few-shot scenario.

heading: 3 Our Proposed Model
text: In Joint Multimodal Aspect-Sentiment Analysis (JMASA), our goal is to extract aspect terms and classify sentiment corresponding to each aspect. However, due to the varying number of aspect terms in each instance and each diverse aspect term, a different prompt is needed for each aspect in the few-shot setting. To address this, we propose a Generative Multimodal Prompt (GMP) for fewshot JMASA, as illustrated in Fig. 2. Leveraging BART, we generate aspect-oriented prompts for each aspect based on the multimodal context, as well as instance-level sentiment-oriented prompts.

heading: 3.1 Task Formulation
text: In this paper, we assume access to a pre-trained language model M, such as BART, that we wish to fine-tune for the aspect-sentiment sequence generation task using labeled data. For the few-shot multimodal training dataset Dtrain, we select K training examples based on sentiment categories for each dataset, resulting in Dtrain = (T j , Ij , Aj , Sj , Oj)j = 1K , where T = [t1, t2, ..., tlt ] is the text modality with lt as the text length; I is the image modality; A = [a1, ..., an] is the aspect list; S = [s1, ..., sn] is the sentiment list corresponding to A; and O = [(x1b , x 1 e, s 1), ..., (xnb , x n e , s
n)] is our output, which represents the index-sentiment list, e.g.,
O = [(5, 5, POS), (13, 14, NEU)] for the instance in Fig. 3. Here, n denotes the number of aspects, xkb and x k e represent the beginning and end indices of the kth aspect term, and sk ∈ {POS,NEG,NEU} denotes the sentiment label. For Ddev, we select the same size of data as the fewshot training dataset, i.e., |Ddev| = |Dtrain|. Our task is to generate O in the few-shot multimodal setting. Following the formulation in (Yan et al., 2021; Ling et al., 2022), we define the outputs of the three subtasks as follows2:
• JMASA: O = [(xb1, xe1, s1), ..., (xbn, xen, sn)].
• MASC: O = [(xb1, xe1, s1), ..., (xbn, xen, sn)].
• MATE: O = [(xb1, xe1), ..., (xbn, xen)].

heading: 3.2 Generative Multimodal Prompt
text: GMP consists of two main modules: the Multimodal Encoder module and the N-Stream Decoders module.

heading: 3.2.1 Multimodal Encoder
text: In this section, we design the multimodal encoder to capture multimodal representations. We start by extracting image representations using NF-ResNet (Brock et al., 2021), and then project them to the text modality space for the image modality, I .
V = Reshape(WiResNet(I) + bi)
= [v1, ..., vk, ..., vli ], vk ∈ Rdt , (1)
where V is reshaped image representation, Wi ∈ Rdv×dnt , bi ∈ Rdnt , and nt = li × dt. li, which is
2The underlined tokens are provided during inference.
a hyperparameter, is the number of image slots that reserve initial image representation, and dt represents the dimension of text embedding in BART.
Since BART is a pre-trained language model (PLM) that does not involve pre-training on image modality, we aim to alleviate the discrepancy issue of image representation in PLM. To achieve this, we further capture the image caption using ClipCap (Mokady et al., 2021), denoted as C, which can be regarded as the image prompt.
C = ClipCap(I). (2)
We utilize the BART model to obtain text embeddings for both the text input and the image caption.
ET = Embedding(T ), ET ∈ Rlt×dt , EC = Embedding(C), EC ∈ Rlcap×dt ,
(3)
where lcap is the length of image caption. The multimodal embedding EM can be obtained, EM = [Eimg, V , E/img, Eis, Ecap, EC , E/cap, Ebos, ET , Eeos].
Finally, we feed EM into the BART Encoder to obtain the multimodal representation. We argue that subsequent decoders require specific information, so we leverage different multimodal BART Encoders for this purpose.
HaM = MBART a E(EM ), H a M ∈ Rlm×d, HsM = MBART s E(EM ), H s M ∈ Rlm×d, (4)
where lm = li + lcap + lt + ls, ls is the length of special tokens, and d is the hidden dimension.

heading: 3.2.2 N-Stream Decoders
text: IIn this section, we utilize the encoded multimodal representation from Eq. 4 to predict the number of aspect terms and generate aspect-oriented and sentiment-oriented prompts using different decoders for each instance. The ‘N’ in ‘N-Stream’ varies depending on the task, with values of 3, 2, and 1 for JMASA, MATE, and MASC, respectively.
Aspect-Num Decoder (AND). In the JMASA task, the number of aspects in each instance is significant but unknown, so we predict the number of aspects based on the multimodal context using the Aspect-Num BART Decoder as a subtask. Specifically, we input the multimodal encoder output HaM and the special token bos into the Aspect-Num Decoder, which then predicts the number of aspects np ∈ R5 as follows3:
handn = AND(H a M ;Ebos),
np = Softmax(MLP (h and n )).
(5)
3Twitter 2017 dataset contains only 3 instances with more than 5 aspects. Therefore, we set "aspect-num" as 5 in the AND module to accommodate the maximum number of aspect terms in an instance.
We leverage the cross-entropy loss for the subtask,
Lc = − K∑
j=1
njglog(n j p), (6)
where njg represents the label for the number of aspect terms. It’s worth noting that in the MASC task, the gold number of aspect terms is provided to the model, and thus, this subtask is not required for MASC.
P ka = MLP k([hapd1 , h apd 2 ]), (7)
where k is the kth group of aspect of an instance, P ka ∈ R2×d. The generative aspect-oriented prompt AP = [P 1a , ..., P np a ] ∈ R2np×d.
Aspect-oriented Prompt Decoder (APD). Prompts for few-shot multimodal classification tasks can be manually designed for specific datasets due to limited categories, as demonstrated in PVLM (Yu and Zhang, 2022) and UP-MPF (Yu et al., 2022). However, each text-image pair carries different context information, and the aspects of the text are diverse. Therefore, in the few-shot setting, we need to capture various cues for each aspect. Inspired by this, we design our model to generate aspect-oriented prompts based on the multimodal context. Specifically, we first generate an instance-level prompt based on the encoded multimodal representation. The final output of the JMASA task is a triplet sequence, where the first two positions of each triplet represent the beginning and ending indices for each aspect term. We set two aspect slots for each generated aspect-oriented prompt, resulting in an instancelevel prompt length of 2np. The decoder takes the encoder outputs Ham and previous decoder outputs hapd< (lap− 1) as inputs to compute the current
hidden state.
hapdlap = APD(H a M ; (h apd <(lap−1))), (8)
where we feed the bos into APD as the beginning token and lap = 2. Sentiment-oriented Prompt Decoder (SPD). The sentiment corresponding to each aspect is related to each instance. Similar to APD, we generate the sentiment-oriented prompt based on multimodal context. For JMASA, the last position in each triplet of the output sequence predicts the sentiment. We set one sentiment slot for each generated sentiment-oriented prompt, i.e., the length of the instance-level prompt is np.
Ps = h spd 1 = SPD(H s M ;Ebos), (9)
where we feed the bos into SPD as the beginning token. As the sentiment categories are limited, they share a common label space. Therefore, we do not generate corresponding sentiment cues for each aspect. Instead, Ps is repeated np times to form the generative sentiment-oriented prompt SP = [P 1s , ..., P np s ] ∈ Rnp×d, where d represents the dimensionality of the prompt.

heading: 3.3 Multimodal Embedding with Prompt
text: We construct the multimodal prompt for different tasks, including JMASA, MASC, and MATE, based on the text-image pair, aspect-oriented prompts, sentiment-oriented prompts, and prediction of the number of aspect terms. For JMASA, we design multimodal embedding with a generative multimodal prompt, denoted as EPJ , as shown in Fig. 2. Similar to EPJ , we separately design multimodal embedding with prompt for MASC and MATE, e.g, EPS and E P A as Fig. 5 shows in Appendix A.

heading: 3.4 Triplet Sequence Generation
text: We next feed the multimodal embedding with prompt into the Encoder-Decoder model to generate the triplet sequence. We take the JMASA task as an example, as Fig. 3 shows.
HPJ = MBART J E(E P J ), H P J ∈ RlJ×d, (10)
where lJ is the length of EPJ . Then, we use the BART decoder to get the last hidden state,
hdJt = BART J D(H P J ; Ô<t), (11)
where t is the tth step and Ô<t is the output of the previous t steps. Following (Yan et al., 2021), we
predict the token probability distribution Pt with hdJt ∈ Rd, as follows:
Pt = Predict([ET ;ES ]h dJ t ), (12)
where Pt ∈ Rlt+lc ; ES is the embedding of the sentiment label set, and its length is lc = 3.
We employ cross-entropy loss for our sequence generation task.
Lg = − K∑
j=1
Ojlog(P j). (13)

heading: 3.5 Multitask Training
text: We optimize our main task and subtask.
L = Lg + λLc, (14)
where λ is the hyperparameter to control the contribution of each task.

heading: 4 Experiments
text: We conduct experiments on two groups of fewshot multimodal datasets built according to the distribution of sentiment categories from Twitter15 (15) and Twitter-17 (17) (Zhang et al., 2018; Lu et al., 2018). We compare our model with numerous approaches on three tasks, including Multimodal Aspect Term Extraction (MATE), Multimodal Aspect-oriented Sentiment Classification (MASC), and Joint Multimodal Aspect-Sentiment Analysis (JMASA).

heading: 4.1 Few-shot Datasets
text: To construct few-shot datasets for few-shot Multimodal Aspect-Based Sentiment Analysis (MABSA), it is important to select a few diverse samples that provide comprehensive coverage of the different sentiment categories. We sample data based on the distribution of sentiment categories in instances to create few-shot datasets. The statistics of the different datasets are presented in Table 1. For each dataset, we randomly sample three groups of few-shot training and development datasets based on three different seeds, such as [42, 87, 100], and each split is run 3 times. We report the average performance and standard deviation over 9 (3 × 3) times of training for a more robust evaluation.

heading: 4.2 Implementation Details
text: We utilize BART-Base with 140M parameters as our Pretrained Language Model (PLM), denoted as M, and NF-ResNet-50 as our visual encoder. The number of epochs is set to 70, and the batch size is set to 4 for all tasks. The learning rates (lr) are set to 6.5e-5 for JMASA and MATE tasks, and for the MASC task, we set lr to 8e-5 and 7.5e-5 for Twitter15 and Twitter-17, respectively. All models are implemented using PyTorch and the experiments are run on an A6000 GPU. Following (Ling et al., 2022), we evaluate our model on three subtasks of MABSA and use Micro-F1 score (F1), Precision (P), and Recall (R) as the evaluation metrics to measure the performance. For MASC, we also use Accuracy (Acc) to compare fairly with other approaches. GMP has 169.3M/155.6M/154.9M parameters for JMASA/MATE/MASC, respectively, and during training, all parameters are updated. The training time for GMP up to 70 epochs is 50/50/25 minutes for JMASA/MATE/MASC.

heading: 4.3 Baselines
text: To ensure a comprehensive comparison, we thoroughly evaluate our model against various approaches across different tasks.
Models for Joint Multimodal Aspect-Sentiment Analysis (JMASA). We first apply text-based approaches to perform Joint Aspect-Sentiment Analysis (JASA) with the following models: BART (Yan et al., 2021) adapts JASA to an EncoderDecoder model. D-GCN (Chen et al., 2020) proposes directional graph convolutional networks for JASA. SpanABSA (Hu et al., 2019) applies an extraction-then-classification framework using a span-based labeling scheme. Next, we accomplish JMASA and MATE using multimodal approaches with the following models: JML (Ju et al., 2021) performs JMASA by introducing auxiliary crossmodal relation detection. CMMT (Yang et al., 2022b) proposes a multi-task learning framework that leverages two unimodal auxiliary tasks. VLP (Ling et al., 2022), which designs multiple VisionLanguage pre-training tasks, is the state-of-the-art (SOTA) model for JMASA. However, since VLP introduces additional 17,000+ pre-training data, which violates our motivation to use few-shot data, we also present results for NVLP, which does not perform the pre-training task.
Models for Multimodal Aspect Sentiment Classification (MASC). We reproduce multimodal ap-
proaches that are trained in full MSA datasets from published paper for MASC. TomBERT (Yu and Jiang, 2019) models the intra-modality and intermodality dynamics to improve the performance of MASC. CapTrBERT (Khan and Fu, 2021) constructs an auxiliary sentence, which is the translation of the image, to provide multimodal information to a language model. KEF (Zhao et al., 2022) exploits adjective-noun pairs extracted from the image to improve the visual attention capability and sentiment prediction capability of the fine-grained MSA task FITE (Yang et al., 2022a), the stateof-the-art model for fine-grained MSA, leverages facial information from the image modality.
Additionally, we adapt and evaluate models originally designed for few-shot text classification tasks for multimodal aspect-based sentiment classification. LM-BFF (Gao et al., 2021) designs different text prompts based on each specific dataset and text demonstrations to solve few-shot text classification tasks. LM-SC (Jian et al., 2022) further introduces supervised contrastive learning based on LM-BFF to few-shot text tasks. GFSC (HosseiniAsl et al., 2022) converts the classification task into a generation task and solves text classification tasks in the few-shot setting through a pre-trained generation model, namely GPT2 (Radford et al., 2018). Recently, a few multimodal sentiment classification models in the few-shot setting have emerged. PVLM (Yu and Zhang, 2022) proposes a prompt-
based vision-aware language modeling approach to MASC in a few-shot scenario. UP-MPF (Yu et al., 2022) applies a unified pre-training for multimodal prompt-based fine-tuning model, which is the state-of-the-art model for few-shot MASC.

heading: 4.4 Experimental Results and Analysis

heading: 4.4.1 Results of JMASA
text: Table 2 presents the results of JMASA on fewshot multimodal datasets, and several key observations can be made. We can make the following observations. First, multimodal models generally outperform unimodal models. Among the multimodal models, JML and VLP, which leverage additional data for relation detection and pre-training, respectively, achieve better performance compared to NVLP, which does not involve pre-training tasks, indicating the effectiveness of pre-training tasks in improving model performance. When considering the amount of data used by the models, it is more reasonable to compare our model with NVLP. Our model consistently outperforms NVLP across both datasets, indicating its superior performance. Notably, our model also outperforms the secondbest model, VLP, by a significant margin, with 1.56 and 2.03 absolute percentage points in terms of F1 on Twitter-15 and Twitter-17, respectively. The superior performance of our model can be attributed to several factors. First, the generative multimodal prompt, which is based on the multi-
modal context, enables the model to capture practical knowledge for each sample from the pre-trained language model. Second, the subtask information provides valuable clues for constructing the multimodal prompt, leading to improved performance in few-shot multimodal sentiment classification.

heading: 4.4.2 Results of the MASC
text: The results of the MASC task on few-shot multimodal datasets, in terms of accuracy (Acc), are presented in Table 3, while the corresponding F1 results are shown in Table 6 from Appendix B.1. The models with “∗” are specifically introduced for fewshot scenarios. Several key observations can be made from the results. We can obtain the following observations. In the multimodal few-shot setting, 1) Our model demonstrates the best performance in the multimodal few-shot setting, indicating its superiority over other models in handling the challenges of limited labeled data. 2) Prompt-based methods outperform robust multimodal models, highlighting the effectiveness of prompt-based methods in lowresource scenarios. This suggests that leveraging prompt engineering techniques, such as our generative multimodal prompt, can lead to improved performance in few-shot MSA. 3) BART, which uses only the text modality, performs better than most multimodal models, indicating the strong performance of our base model. This suggests that the pre-trained language model, BART, provides a solid foundation for our multimodal model.

heading: 4.4.3 Results of MATE
text: Table 4 presents the results of the MATE task. Among the models, VLP achieves the best performance in MATE, although it deviates from our
initial goal of applying low-resource data due to its reliance on additional data and multiple pretraining tasks on the MVSA-Multiple Dataset (Niu et al., 2016). Similarly, JML also leverages additional data to enhance its performance. An interesting observation is that MASC performs poorly in VLP when compared to NVLP, despite VLP showing better performance on the MATE and JMASA tasks compared to NVLP. We hypothesize that the pre-training task of VLP may be more aligned with the MATE task, which in turn may have an impact on the performance of MASC.

heading: 4.5 Ablation Experiments
text: We performed ablation experiments on the GMP model to assess the effectiveness of different modules. The results, as shown in Table 5, indicate that the complete GMP model consistently the best performance across all tasks. First, we remove the image modality (w/o Image) and built generative prompts based only on the text modality. The model’s performance in all tasks is adversely affected, indicating that the image modality is crucial for achieving high performance in few-shot MSA tasks. Next, we only remove the image caption (w/o Caption) and retain the initial image features to evaluate the effectiveness of the image prompt. The results show that the image prompt contributes to the overall performance of the model, indicating its utility in capturing important information from the image modality. We also conduct experiments where we remove the multitask module (w/o Multitask) and set the number of aspect terms to 5 for each instance in the JMASA and MATE tasks. The performance of the models is affected, indicating that the subtask-specific modules are effective in capturing aspect-related information and improving performance. To verify the utility of the generative multimodal prompt, we remove the multimodal prompt (w/o Prompt) and use only the original textimage representation. The model’s performance degraded, indicating that our proposed multimodal
prompt is beneficial in providing valuable cues for the sentiment analysis task. We further remove the generative aspect prompt (w/o GAP) to assess the importance of GAP. Interestingly, we observe that using generated sentiment prompts (GSP) resulted in better performance in the MASC task (w/o GSP), whereas we obtain the opposite result in the JMASA task (w/ GSP). This suggests that the generated aspect prompt provides sufficient information to the model, and GSP may introduce redundant information in the JMASA task. However, in the MASC task, GSP provides effective cues for sentiment classification. We further experiment with different generated sentiment prompts (w DSPrompt) and find that the performance significantly decrease. There are two possible reasons for this observation. First, the sentiment categories in our dataset are limited. When using generated sentiment prompts for each aspect, it may introduce noise and irrelevant information to MASC. Second, the generated prompts for each aspect provide sufficient information to guide the model in capturing aspect-related sentiment information.

heading: 4.6 Hyperparameters Setting
text: The hyperparameter experiments of JMASA are shown in Fig. 4. The hyperparameter experiments on other tasks are in Appendix B.2.
Hyperparameters li and λ on JMASA. In order to effectively utilize image information through NF-ResNet, we conduct experiments with different
settings of the hyperparameter li in Eq. 1, and the results are shown in Fig. 4(a). We observe that our GMP model achieves the best performance on both datasets when the number of image slots, li, is set to 4. When li is smaller, the image information is not fully utilized, and the model’s performance is compromised. On the other hand, retaining more image features by setting a larger value for li results in redundant information being provided to the model, which also leads to decreased performance. When li was set to 0, GMP only utilized the image prompt, i.e., the image caption C, and discarded the initial image representation V . We also employ the hyperparameter λ to balance the contribution of the subtask, as shown in Fig. 4(b). We find that the best value of λ varied across different datasets, with 0.1 being the optimal value for Twitter-15 and 0.15 for Twitter-17. When λ is set to a larger value, the model’s performance dramatically drop. This is because a larger value of λ biases the model towards the subtasks, and we need to strike a balance among all tasks to achieve optimal performance.

heading: 5 Conclusion
text: We propose a novel Generative Multimodal Prompt (GMP) for Multimodal Aspect-Based Sentiment Analysis (MABSA) that includes JMASA, MASC, and MATE in the multimodal few-shot scenario. We further introduce a subtask to predict the number of aspect terms to form multitask training to improve the performance of GMP. Experimental results show that our proposed approach outperforms strong baselines on two subtasks of MABSA in the few-shot setting. We provide a new direction for related tasks of MABSA in the few-shot setting. In future work, we plan to exploit the fine-grained image features and achieve alignment between text and image modality to improve the performance of MABSA in the multimodal few-shot scenario.
Limitations
Although our model has shown superior performance, there are still a few limitations that could be improved in future work.
• We create few-shot datasets from the perspective of the combination of sentiment categories without considering the distribution of aspect items, such as the number of aspects in each sample. It may affect the performance of the model on the task of extracting aspects. We should create more efficient datasets for MABSA in the few-shot setting.
• As we put more emphasis on the performance of the main task, the performance of the subtask of predicting the number of aspect terms in each example may suffer. We will further improve the accuracy of the subtask in future work.
• We roughly exploit initial image features and do not perform alignment between text and image modalities. We plan to accomplish the alignment of multiple modalities further to improve the performance of MABSA in future work.

heading: Acknowledgements
text: Thanks to all co-authors for their hard work. The work is supported by National Natural Science Foundation of China (No. 62172086, No. 62272092), Doctoral Research Innovation of Northeastern University (No. N2216004), Chinese Scholarship Council, and Grants of Singapore (Project No. T2MOE2008, and Grantor reference No. MOE-T2EP20220-0017; Project No. RGAST2003).

heading: A Multimodal Embedding with Prompt
text: For the MASC task, we design multimodal embedding with generative multimodal prompt, EPS , as Fig. 5(a) shows.
For the MATE task, we design multimodal embedding with generative multimodal prompt, EPA , as Fig. 5(b) shows.

heading: B Experimental Results
text: B.1 F1 Results of MASC
The results of the MASC task in terms of F1 are shown in Table 6.
B.2 Hyperparameters Setting
Hyperparameters li on MASC: We use the gold number of aspect terms for the MASC task and don’t use the subtask. Thus we only conduct experiments on the hyperparameter li. Similar to the JMASA task, our model achieves the best performance on two datasets when li is 4, as Fig. 6 shows.
Hyperparameters li and λ on MATE: Fig. 7 shows the hyperparameters of the MATE, including li and λ. On both datasets, our model has the best results when λ is 4. For the hyperparameter, li, our model achieves the best performance when li is 4 on the Twitter-15 dataset, and li is 3 on the Twitter-17 dataset.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
3 A2. Did you discuss any potential risks of your work? Limitations
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract; 1; 5
7 A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Not applicable. Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4.1
C 3 Did you run computational experiments? 4.4; 4.5; 4.6
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4.2; 4.6
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4.1; 4.4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In future work, we plan to exploit the fine-grained image features and achieve alignment between text and image modality to improve the performance of MABSA in the multimodal few-shot scenario. Limitations
Although our model has shown superior performance, there are still a few limitations that could be improved in future work. • We create few-shot datasets from the perspective of the combination of sentiment categories without considering the distribution of aspect items, such as the number of aspects in each sample. It may affect the performance of the model on the task of extracting aspects. We should create more efficient datasets for MABSA in the few-shot setting. • As we put more emphasis on the performance of the main task, the performance of the subtask of predicting the number of aspect terms in each example may suffer. We will further improve the accuracy of the subtask in future work. • We roughly exploit initial image features and do not perform alignment between text and image modalities. We plan to accomplish the alignment of multiple modalities further to improve the performance of MABSA in future work