An Empirical Study on Configuring In-Context Learning Demonstrations for Unleashing MLLMs' Sentimental Perception Capability
Authors: Daiqing Wu
Dongbao Yang
Sicheng Zhao
Can Ma
Yu Zhou
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Sentiment Analysis
Multimodal Large Language Model
In-Context Learning
Primary area: applications->computer_vision
TL;DR: We conduct an in-depth investigation into three pivotal factors that influence the configuration of In-Context Learning demonstrations on Multimodal Sentiment Analysis.

Abstract: The advancements in Multimodal Large Language Models (MLLMs) have enabled various multimodal tasks to be addressed under a zero-shot paradigm. This paradigm sidesteps the cost of model fine-tuning, emerging as a dominant trend in practical application. Nevertheless, Multimodal Sentiment Analysis (MSA), a pivotal challenge in the quest for general artificial intelligence, fails to accommodate this convenience. The zero-shot paradigm exhibits undesirable performance on MSA, casting doubt on whether MLLMs can perceive sentiments as competent as supervised models. By extending the zero-shot paradigm to In-Context Learning (ICL) and conducting an in-depth study on configuring demonstrations, we validate that MLLMs indeed possess such capability. Specifically, three key factors that cover demonstrations' retrieval, presentation, and distribution are comprehensively investigated and optimized. A sentimental predictive bias inherent in MLLMs is also discovered and later effectively counteracted. By complementing each other, the devised strategies for three factors result in average accuracy improvements of 15.9% on six MSA datasets against the zero-shot paradigm and 11.2% against the random ICL baseline.

An Empirical Study on Configuring In-Context Learning Demonstrations for Unleashing MLLMs' Sentimental Perception Capability
Daiqing Wu
Dongbao Yang'
Sicheng Zhao 3
Can Ma
Yu Zhou 4
Abstract
Unacceptable Performance Gap
Fully Superupped Models
78.8
The advancements in Multimodal Large Lan- guage Models (MLLMs) have enabled various multimodal tasks to be addressed under zero- shot paradigm. This paradigm sidesteps the cost of model fine-tuning, emerging as a dominant trend practical application. Nevertheless, Mul- timodal Sentiment Analysis (MSA), pivotal chal- lenge the quest for general artificial intelligence, fails to accommodate this convenience. The zero- shot paradigm exhibits undesirable performance on MSA, casting doubt on whether MLLMs can perceive sentiments as competent as supervised models. By extending the zero-shot paradigm to In-Context Learning (ICL) and conducting an in- depth study on configuring demonstrations, we validate that MLLMs indeed possess such capa- bility. Specifically, three key factors that cover demonstrations' retrieval, presentation, and dis- tribution are comprehensively investigated and optimized. sentimental predictive bias inherent in MILMS is also discovered and later effectively counteracted. By complementing each other, the devised strategies for three factors result in av- erage accuracy improvements of 15.9% on six MSA datasets against the zero-shot paradigm and 11.2% against the random ICL. baseline.
-23.8
Boost Performance Miliiial
20
+15.9
MILMMS' with auticious Demonstrations (Durs)
70.9
68.3
Avempe
60
59.7
55.0
MILMM Zero- Shot Paradigm
Laborious Annetation
Sompiee
13% Tratning Set)
200 Samples
--300 Samples [Fol
Requirement for Annotated Data
Figure I. Comparison of fully supervise models, few-shot mod- els, and MLLMs based on average accuracy and annotated data requirement across six MSA datasets. The MLLMs' zero-shot paradigm, although avoiding the laborious annotation, exhibits a substantial performance gap compared to fully-supervised models. With proper demonstration configuration, this gap can be notably narrowed by In In-Context Learning (ICL).
mains have flourished. Among them, Multimodal Sentiment Analysis (MSA) aims to classify the sentiment polarity em- bedded in multimodal data. As corroborated by empirical (Zadeh et al., 2017) and theoretical (Huang al., 2021) evidence, the synergy between modalities facilitates more comprehensive modeling of sentiment clues compared to unimodal data. This superiority has led to growing interest in MSA from academia and industry (Zhang et al., 201 Sa; Yue et al., 2019).
In this paper, we focus on MSA with the input form of image-text posts, whose number grows exponentially thanks to the prosperity of social media. The current research land- scape can be broadly grouped into two streams: post-level branch (Xu & Mao, 2017), which identifies the overall sen- timent of posts, and aspect-level branch (Xu et al., 2019), which probes the sentiment associated with specific aspect terms within the context of posts. Leveraging elaborate mod- ules and large volume of annotated data, both streams have attained remarkable achievements. Entering the era of Mul- timodal Large Language Models (MLLMs), various multi- modal tasks can be reformulated as filling textual prompts and properly accomplished by MLLMs in a zero-shot man- ner (Yin et al., 2023). This emerging paradigm avoids time- intensive fine-tuning and task-specific annotation. However, recent surveys (Yang et al., 2023b; Lian et al., 2024) have
1. Introduction
Equipping models with emotional intelligence has been fascinating yet vital challenge over the past few decades (Zhao et al., 2022; Zhang et al., 2023). Studies var- ious facets of emotions and sentiments in numerous do-
'Institute of Information Engineering, Chineses Academy of Sciences 'School of Cyber Security. University of Chinese Academy of Sciences TTsinghua University *VCIP & TMCC & DISSec, College of Computer Science, Nankai University. Cor- respondence Dongbao Yang <yangdongbao ceaioonnnaa@iii..c..c.... ai.>,,.en>, Zhou <yzhou@ nankai.edu.cn..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Configuring ICL Demonstrations for Unleashing MLLMs' Sentimental Perception Capability
Prompt
P
Test Sample
MLLM Inference
Zero Shot Output
ICL Output
MLLM Inference
(A).
Zero-Shot Paradigm
pporr carrain aniinggs and inppen stectiyy the suttmmet exeect the jostthe, houttol Negater
Image:
Teet Hitoon Weveliire Merter and arttor Wethertiie tartera Pepworts 1995
SechaE
Pro Nest Cude
Such
0LUS 0.05
0.33
sepworth Sertiment:
Prodectore
Predettere
Prompt P
Demonstration (I, On)
Demonstration (Iz, O2)
Demonstration 3 (Ig, Oy!
Test Sample
i
Apot corsare an mnnngoo tixt sentimeet of
Inoge
Tust
Ieago
hat: reflet Eniing e NOLA
Waage:
Tel:
KAg
Test
(b).
Context Learming
the pont The spronal citeprtio are Negateel
3 pison in Wushingss
Houaten 10
Liccln Nerorld
Howe Draan ipeach
and thild Fan bork
Mithirdaay by Bartbra
Here arR www.scccossapppo coample..
Aspeet Wyrkeggooo
Aapect iton
Aeanct dingta
Alpect karsvo Heywortt
Figure 2. Comparison between MLLMs' zero-shot paradigm and ICL, In addition to the test sample, ICL sequences three demonstrations with inputs and corresponding outputs, facilitating more precise sentiment predictions for MLLMs.
revealed that, under the zero-shot paradigm, MLLMs lag behind supervised models by a significant margin on MSA, as also illustrated in Figure 1. Given the success of MLLMs on tasks such as image captioning (Li et al., 2023) and vi- sual question answering (Driess et al., 2023), it is natural to wonder whether the capabilities of MLLMs to perceive sentiment are yet to be fully explored. If not, how can we fully tap into their potential at minimal cost?
scenarios. Therefore, image captioning (Yang et al., 2023a) and text-to-image generation (Alimisis et al., 2024) are com- monly employed to convert between the two modalities to furnish supportive information in multimodal tasks. Inspired by this, we explore the effects of modality compositions on ICL, ultimately arriving at the most efficacious modality presentation. (3). The biases introduced by sentiment distribution. It has been observed on various tasks (Yang et al., 2023c; Li et al., 2024; Baldassini et al., 2024) that MILMS are prone to be affected by the biases in ICL se- quences, a phenomenon we also validate on MSA. Driven by this discovery, we devise various distribution protocols to intentionally incorporate sentiment biases and probe their influences. Comprehensive experiments reveal that appro- priate sentiment biases can counterbalance the inherent pre- dictive bias of MLLMs, thereby promoting fair prediction. In summary, our contributions are three-fold:
A feasible answer is In-Context Learning (ICL), which ex- tends MLLMs' zero-shot paradigm to few-shot scenario by sequencing a series of input-output pairs as demonstra- tions (Brown al., 2020). Figure 2 illustrates this process with an example. In this manner, ICL showcases the for- mulation of the task and the mapping between inputs and outputs (Pan et al., 2023), which has been proven beneficial for MIL.Ms in both unimodal and multimodal tasks (Chen et al., 2023a; Li et al., 2024). Despite its success, the ef- fieacy of ICI. heavily relies on the retrieval (Zhang et al., 2022), presentation (Li al., 2024), and distribution (Lyu et al., 2023) of demonstrations. Studying the impact of these factors is a prevalent subject in natural language process- ing, yet there's i a notable absence of necessary attention in multimodal settings, particularly for MSA.
Through configuring ICL demonstrations, we unleash the potential of MILMS MSA, validating that MLLMs are competent in perceiving sentiment.
We investigate and optimize three key factors covering the retrieval, presentation, and distribution of demon- strations in ICL on MSA. During the process, senti- mental predictive bias inherent in MLLMs is discov- ered and mitigated, facilitating faimess in sentiment prediction.
To fill this gap and unleash the potential of MIL.Ms, we systematically investigate the configuration of ICL demon- strations in MSA. Specifically, we delve deeper into three crucial factors underexplored in current studies. (1). The similarity measurement of multimodal data for demon- stration retrieval. The similarity between the test sample and demonstration is positively correlated with the effective- ness of ICL (Liu et al., 2022). The mainstream measurement method directly aggregates images' and texts' similarity scores (Yang et al., 2022). Nevertheless, our experiments re- veal that this approach overlooks fine-grained aspect terms and fails to weigh the significance of modalities. In re- sponse, we refine and customize similarity measurements specifically tailored for MSA. (2). The trade-off between multimodal information presented in the demonstra- tions. Owing to the disparity in information density, images and texts each have unique pros and cons under different
By complementing each other, the ICL strategies tai- lored for the three factors improve the accuracy of MLLMs by average of 15.9% on six MSA datasets compared to the zero-shot paradigm, and I 11.2% com- pared to the random ICI. baseline.
2. Related Works
2.1. Multimodal Sentiment Analysis
With the explosion of multimodal data from social media platforms in recent years, MSA has evolved as a pivotal topic in affective computing (Zhu et al., 2024; Yang et al.,
2

---

Configuring ICL Demonstrations for Unleashing MLLMs' Sentimental Perception Capability
Multinodal Sequence to Food into MILUM
Input Orrog,, Text, Aspect)
A OOutuu (Sentiment Poloriey)
Pesilvve
NNuttal
(Negaliec
Ground Touth Test Sumple
Nexcal
Prompt
Damo
Smillarity Score: 0.37
Damo
Similarity Score: 0.65
Demo
Similarity Score: 0.52
Test Sample
Sentiment Dutribution
(a). Similarity Measurement
(b). Modality Presentation
(c). Sentiment Distribution
Image
Captioned Text
Sentiment of Demonstcations
Sentiment Prediction of the Test Sample
Image
Image
Ocars Canten
Captioning
Same the Test Sample:
Goreected Image
AAAAAA
MLLM
Aspect
Aspect
Prodeater
Demomstration
Input of
Test Sample
Inpurt the
Text to Image Generation
Distinct the Sample:
Sorten Paryy
MLAM
Prodeter Puskiir
Various Semtiment Dittributiors to Diverss Predictions
Image
Tedt
Ga wwd ImAge Opttined lod
Practica::
AAAAA
Sapest:
OR
Sentiman:
Pig- MESION
Nor Aggertos
1gragel - (lee)
&J rnage)
CCCoptioned
Teef
Ax7
Figure 3. Illustration of the three factors to be investigated and optimized, during which we aim address the following questions. (a), How do we measure the similarity score between multimodal data? (b). How do we decide which modality should be presented in the imput? (c). What kind impact does the sentiment distribution of demonstrations have?
2024). Image-text posts (Xu & Mao, 2017) and videos (Zadeh et al., 2017) are two major forms of input in MSA. Our investigation centers on image-text posts, as they are more congruent with the input specifications of MILMS. Within this domain, the majority of research adheres to a similar methodology, typically initiating with the extraction of unimodal features followed by their fusion for sentiment classification. Over the past few years, notable advance- ments have been made in both post-level (Xu, 2017; Yang et al., 202 bb; Li et al., 2022; Wei et al., 2023; Wu et al., 2024a) and aspect-level (Xu et al., 2018; Ling et al., 2022: Zhou et al., 2023; Peng et al., 2024) branches, by refin- ing fusion strategies and learning from thousands of hand- annotated data. However, as highlighted by Yu al. (2022), the acquisition of such data is unrealistic in practical set- tings. Consequently, few-shot MSA has been proposed (Yu et al., 2022) and developed (Yang et al., 2023a). Although it reduces data requirements, the time-intensive fine-tuning persists as an impediment to the cost-effective application of MSA.
analogical learning (Dong al., 2024), a capability known as ICL (Wang et al., 2024). First identified in GPT-3 (Brown et al., 2020), ICL has sparked numerous studies on LLMs. Among these, a predominant emphasis has been placed on the impact of demonstration configurations. Through exhaustive explorations, various effective strategies have been proposed from distinct angles (Lu et al., 2022; Li & Qiu, 2023; Wu et al., 2023; Levy et al., 2023). Certain ICL. strategies of LLMs also demonstrate proficiency in tackling multimodal tasks (Yang et al., 2022; Cai et al., 2023) by converting them into textual format.
To transfer the ICL capability into MLLMs, Flamingo (Alayrac et al., 2022) interleaves cross-attention modules that receive visual imput into the pre-trained LLMs. Inspired by its success, series of open-sourced MLLMs armed with ICL capability are developed, such as Open-Flamingo (Awadalla et al., 2023), IDEFICS (Laurengon et al., 2023) and Kosmos (Huang et al., 2023). Parallel to the case in LLMs, the performance of MLLMS exhibits significant vari- ability in response to diverse demonstrations. A line of studies methodically probes into such effects in image cap- tioning (Yang et al., 2023c) and visual question answering (Li et al., 2024: Chen et al., 2024: Qin et al., 2024), uncov- ering valuable insights of ICL and important properties of MLLMs. Nonetheless, the unique challenges intrinsic to MSA remain inadequately overcome. We aim to fill this blank through a comprehensive empirical exploration.
The advent of MLLMS presents a viable alternative via the zero-shot paradigm (Lian al., 2024), which is sub- sequently augmented by ICL. By harnessing the general knowledge acquired during pre-training, MLLMS are en- dowed with the proficiency to address a wide array of down- stream tasks, requiring merely a minimal set of annotated samples and no gradient updates. Compared with fully- supervised and few-shot MSA, the ICL of MLLMs accords better with practical applications, holding the potential to evolve into prevailing trend for future development.
3. Configuring ICL Demonstrations
3.1. Task Definition
2.2. In-Context Learning
MSA is classification task where the target categories are sentiment polarities. Depending on the classification objective, MSA is grouped into post-level MSA and aspect-
The scaling up of model size has empowered Large Lan- guage Models (LLMs) to generalize to unseen tasks through
3

---

Configuring ICL Demonstrations for Unleashing MLLMs' Sentimental Perception Capability
level MSA. For post-level MSA, the input comprises an image-text post (i, f), and the target is identify the overall sentiment. For aspect-level MSA, the input includes additional aspect a: (?, t, a), and the target shifts to identify the sentiment of the aspect within the context of the post. We wrap the input as I and the output as O for uniformity.
Table I. Formulations of six devised strategies for measuring simi- larity hetween multimodal data.
Strategy
Formulation
A
KA = C(a, a) KIA- Ki + KA
IA
TA
KTA Kr+KA
Before feeding into the MILM, the test sample I and n demonstrations (I, O) are templatized into a multimodal sequence S- = (P: O1); (I2, O2): (In, O,)) where P is a textual prompt including the task description and the set of target categories. A 3-shot case is illustrated in Figure 2 (b). The output Ô is subsequently generated by MLLM (.) predicting the next single token:
ITA
KITA = +T
WIT
KWIT - a K1 + + K.
WITA
KWITA= = a K,, + B Kr + KA
O1):
Image-Text Based (IT) strategy. It extends the unimodal strategies to a multimodal form by aggregating the unimodal similarity scores: Kуr = K2 + Kr. This strategy is also referred to as RICES (Yang et al., 2022).
Ô = arg max ST..)
T
Despite their proven efficacy across various tasks, these strategies fall short in two critical areas when applied to MSA. In particular, they fail to consider the aspect-specific relevance and overlook the unequal significance of modal- ities. To probe the impact of these shortcomings and op- timize accordingly, we further devise six strategies in Ta- ble 1. Among these, the Aspect Based (A) strategy is an unimodal measurement indexed by aspects. The Image- Aspect Based (IA), Text-Aspect Based (TA), and Image- Text-Aspect Based (ITA) strategies are the aspect-inclusive versions of the I, T, and IT strategies, respectively. The Weighted Image-Text Based (WTT) and Weighted Image- Text-Aspect Based (WITA) strategies expand upon the IT and ITA strategies by incorporating weights ax B,77 y to modulate the impact of each modality.
Achieving desirable outcomes through ICL involves the in- terplay of various factors that each contribute to the process. Among these, the retrieval, presentation, and distribution of demonstrations have been empirically verified to wield significant impact (Zhang et al., 2022; Li et al., 2024; Lyu et al.. 2023). In light of MSA's multimodal and affective nature, we concretize these aspects into three key factors to optimize: similarity measurement, modality presentation, and sentiment distribution. The three factors are briefly il- lustrated in Figure 3 (a), (b), and (c), respectively, and are elaborated on in the subsequent sections.
3.2. Similarity Measurement
In demonstration retrieval, prior research has reached two primary agreements: demonstrations bearing greater simi- larity to the test sample are more beneficial for the reasoning process of MLLMs (Liu et al., 2022), and demonstrations should be ordered from lowest to highest similarity in the ICL sequence (Lu et al., 2022). Given adherence to these two fundamental principles, measuring the similarity be- tween multimodal data emerges as the crux of the matter.
3.3. Modality Presentation
The density of information embedded in images and texts naturally differs (Chen et al., 2023b; Wu et al., 2024b). Im- ages convey information in a broadly ranged yet abstract manner, whereas text is typically more precise and con- cise. By converting images to text through image caption- ing, salient objects are emphasized and the irrelevant fine- grained details are omitted (Xu et al., 2015; Yang et al., 2023c). Conversely, transforming texts into images through text-to-image generation can supplement the background and context beyond the description of texts. The extra in- formation is conducive to the elicitation of previously latent emotions (Kosti et al., 2017). As exemplified in Figure 3 (b), the captioned text summarizes the flyer's contents, and the generated image portrays the unmentioned details of the barbecue. Recognizing these benefits, various multi- modal studies leverage these techniques to derive auxiliary modalities to promote performance.
Considering the input of demonstration (i, t, a) and the test sample (6, i, a). there are currently three predominant strategies to measure similarity. Here we discuss the sce- nario of aspect-level MSA due to its generalizability.
Image Based (I) and Text Based (T) strategies. Under unimodal settings, the similarity score K can be straightfor- wardly obtained by computing the cosine similarity in the embedding space of pre-trained encoder E. Indexing by images, this process can be described as:
E(I) o 8() |E(I]| ||
K2
This drives us to investigate whether such success can be replicated in multimodal ICL, where the form in which in- formation is presented within the ICL sequence is of critical importance. Specifically, we obtain the captioned texts from
For simplicity, we abbreviate it as K2 = C(i, 8). Likewise, with text indexing, it would be Kr= = f(t,
4

---

Configuring ICL Demonstrations for Unleashing MLLMS' Sentimental Perception Capability
Table 2. Average accuracy across 4,8, 16-shot demonstrations re- trieved based on varying similarity measurements. represents the random retrieval.
MY3A-
NYSA-M
strategy
Losel
Aspect-Level
Strategy
MVSA-S
MVSA-M
Twitter-15
Twitter-17
STA
R
49.2
60.8
57.4
56.4
I
56.5
64.9
59.1
56.7
(a). Influences of weighting factors in the WIT strategy post-level MSA.
T
56.0
66.2
58.7
57.0
IT
55.7
66.2
61.4
57.6
TWRthee1S
Tulttee17
Jor 1e+39:-
8:2
2:8
61.3
57.4
IA
59.6
58.0
TA
60,9
57.3
ITA
61.0
58.1
the MILM itself, and the generated images from the dif- fusion model (Rombach et al., 2022). Subsequently, we reconstruct the inputs of demonstrations and the test sample with combinations of original and auxiliary modalities. An input form combining the captioned text and generated im- age is depicted the bottom of Figure (b), and broader range of combinations is investigated in the experiments.
5.5
(0)
5:3
(b). Influences of weighting factors In the WITA strategy on aspect-level MSA.
Figure 4. Average accuracy across 4,8, 16-shot demonstrations re- trieved based on the WIT and WITA strategies.
4. Experiments
3.4. Sentiment Distribution
4.1. Datasets and Implementation Details
Under ICL, MLLMs have been shown manifest short- effect (Lyu et al., 2023). Upon encountering test sample, MLLMs tend to duplicate the output from one of the demonstrations, and this tendency intensifies when mul- tiple demonstrations share identical outputs. This exposes MLLMs' susceptibility to biases presented within the ICL sequence, where divergent biases can lead to markedly dif- ferent predictions.
In exploratory experiments, IDEFICS-99 (Laurençon et al., 2023) is selected as a representative MILM (Shukor et al.,
2024). Datasets MVSA-S, MVSA-M (Niu al., 2016) utilized for post-level MSA, and Twitter-15 15 (Zhang et al., 201 8b), Twitter-17 (Lu et al., 201 8) are chosen for aspect- level MSA. For each dataset, demonstrations are retrieved from the support set sampled as Yang et al. (2023a), which accounts for 1% of the training set. MILMss performances are evaluated on the original test set, with accuracy serving as the primary metric. The three factors are investigated separately, and the optimal strategies subsequently com- bined for a comprehensive assessment. To validate the generalizability. additional experiments are carried out us- ing
Regarding MSA, its classification nature and limited cate- gories inevitably result in certain sentiment biases within the ICL sequence, reflected in the sentiment distribution of demonstrations. To analyze the potential impact of these biases and utilize them to optimize the ICL configuration, we formulate five distribution protocols that regulate the sentiment distribution of demonstrations, thereby injecting distinet sentiment biases. The first two protocols serve to highlight the spectrum of the impact of biases. (1), Ideal protocol: every demonstration aligns with the test sample's s sentiment. (2). Contrary to Ideal protocol: every demon- stration differs from the test sample's sentiment. Utilizing ground truth from test samples, they introduce extreme biases into the ICL sequence to establish the theoretical ceiling and floor. In contrast, the following protocols are devised for practical ICL configuration. (3). Unlimited protocol: demonstrations are retrieved without distribution restriction. (4). Category Balanced protocol: each senti- ment class carries an identical number demonstrations. (5). Identical to Support Set protocol: the sentiment distri- bution of demonstrations mirrors that of the support set.
(Awadalla et al., 2023) and on two other MSA datasets TumEmo (Yang et al., 202 la) and MASAD (Zhou et al., 2021).
Due practical considerations, scale, which also aligns with the primary research scope of Yang et al. (2023c); Li et al. (2024). By default, we adopt the IT strategy for similarity measurement, compose the input by the original image and text, and put no constraints on the distribution. Variations are introduced only to the pertinent settings when investigating specific factor.
choose the above model
4.2. Results and Analysis
4.2. I. SIMILARITY MEASUREMENT
Table 2 contrasts the impact of similarity measurement. Here we employ CLIP (Radford al., 2021) as the pre-
5

---

Configuring ICL Demonstrations for Unleashing MLLMs' Sentimental Perception Capability
Table Average accuracy over 4,8, 16-shot settings with the inputs composed of different modalities. "T" abbreviated for Image, T- Text, C- C-Captioned Text, G- Generated Image.
Positive
Hippo
Neutral
Giraffe
meckit Grafts
Negativo
Rhino
Post-Level
Aspect-Level
Modality
Mcan
(e). Mapping sentiments to animals.
MVSA-S MVSA-M Twitter- 15 Twitter-17 17
Averae Accusccy
51.7
56.7
57.4
53.4
54.8
43.5
442
C
43.9
49.5
56.2
52.1
50.4
I.C
56.9
59.3
54.3
50.1
55.3
Twitter-17 Twittar 2b
30.4
37.6
T
46.3
56.0
61.7
59.0
55.8
+1 T)
<C C,
4G 0. C, G,
G
46.8
55.5
54.8
55.2
53.1
(b). Influences of Inputs' constituent modalities on "Task Learning" effect.
T.G
47.7
58.1
54.0
54.8
53.6
Figure 5. Evaluation of ICL's "Task Learning" effect by progres-
I,T
55.7
66.2
61.4
57.6
60.2
sively incorporating modalities into the inputs.
IG
54.5
58.8
54.5
53.8
55.4
СTT
49.7
60.3
62.6
56.7
57.3
C,G
47.5
53.7
53.1
53.1
51.8
4.2.2. MODALITY PRESENTATION
I.C.T
55.3
64.4
60.3
55.6
58.9
I.T.G
50.8
62.0
59.7
54.8
56.8
Table 3 compares inputs composed of various modalities. When input is confined to single-modal information, pre- senting the text modality generally leads superior out- comes, reaffirming the significance of texts in MSA. As more modalities are incorporated, the MLLM can benefit from processing multimodal information, thereby enhanc- ing sentiment prediction in most cases. Noticeably, substi- tuting original modalities with auxiliary ones (e.g. from (I, T) to (C, G)) leads to universal performance degrada- tion. This implies that the potential benefits of modality conversions, which involve highlighting salient objects or providing richer context, are outweighed by the inherent drawbacks. These drawbacks encompass information loss and the generation of extraneous noise during the conver- sion process. Strangely, augmenting original modalities with auxiliary ones (c.g. from (I. T) to (I, C, T)) still re- sults in performance drop, despite the pure addition of information without any loss. Drawing from insights in Pan et al. (2023), associate this with the impairment of the "Task Learning" effect in ICL. Specifically, Pan et al. (2023) decompose ICL's role into "Task Recognition" and "Task Learning". The former prompts the task format for MLLMM to apply their prior knowledge, and the latter aids MLLMs in building mapping relationships between inputs and out- puts. Incorporating additional modalities complicates the inputs, making it more challenging for the MLLM to learn these relationships accurately.
C.T.G
47.4
58.2
56.5
54.1
54.1
I.C.G
50.3
57.0
52.9
52.1
53.1
I,C.T.G
49.2
61.4
58.7
53.1
55.6
trained encoder for both images and texts following previ- ous studies (Yang et al., 2022; Li et al., 2024). Observably, all measurements surpass random retrieval, revealing their validity reflecting the relationships among multimodal samples. Moreover, in aspect-level MSA, the consistent improvement achieved by factoring in aspect similarity un- derscores the aspect's significance. Counterintuitively, the IT or ITA strategies do not always yield optimal results, despite considering the most comprehensive information. Explanations for this necessitate an analysis in conjunction with the results of WIT and WITA strategies. As evidenced Figure 4, the performance of these strategies fluctuates in response alterations the weighting factors. This empha- sizes the criticality of maintaining a proper balance among similarity components, which the IT and ITA strategies fail to achieve, thereby leading to their deficiency.
Delving into the impact of weighting factors, we uncover that prioritizing textual similarity (a < (3 yields a pro- nounced benefit for the WIT strategy in lost-level MSA. This benefit peaks
B 2 8, where the WIT strat- egy achieves an accuracy of 57.6 on MVSA-S and 66.7 on MVSA-M, outperforming all competing strategies. In aspect-level MSA, as illustrated in Figure 4 (b), the WITA strategy should, instead, accord priority to the similarity of aspects. Interestingly, under this priority, the relationship between image and text weights is reversed. We attribute these to the fact that aspects are most directly correlated with sentiment prediction, and the similarity among aspects tends to eclipse the textual similarity, given that these aspects typ- ically originate from the texts. When a : 3 = 7 3 and (a + 3) 7 2 8, the WITA strategy achieves peak accu- racy of 61.8 on Twitter-15 and 59.0 on Twitter-17, making the optimal strategy for aspect-level MSA.
To validate this, we design the experiments in Figure 5 (a) to quantitatively evaluate the "Task Learning" effect. Before feeding into the MLLM, each sentiment is substituted with an animal unrelated the input semantics according to a pre- defined mapping. This approach ensures that the MLLM can longer rely on pre-trained prior knowledge but is instead compelled to learn the input-output mapping solely from ICL. As shown in Figure 5 (b), the "Task Learning" effect diminishes as additional modalities are incorporated into the input, supporting our explanation. To this end, employing original images and text to form inputs has empirically
6

---

Configuring ICL Demonstrations for Unleashing MLLMs' Sentimental Perception Capability
MVSA-5
(a-2) Positive
(a-3) Neutral
(a-4) Negative
(b) Twitter-17
(b-2) Positive
(b-3) Neutral
(b-4) Negative
Postticn Meunall: Negathie
Heee
Postiie:
Neutral
Megative
270 (53.0000::-6 et (Sans): 128 (2a pauss)
(90.09))-57 573 (NG/5S) 168 (13,00)
(a-1) Overall
b- Overall
US Ukinneder (1..0)
Segarier pean
Ideal
Contrary to
Unitmbted
Gotegory Balanced
Support Set
Figure 6. Impact of distribution protocols on MLLM's sentiment predictions. In addition to overall accuracy, precision and recall for each category are studied in 2-16 shot settings. The two selected datasets possess distinctive test set distributions, which are detailed above the accuracy charts. In (a-2,3,4) and (b-2,3,4), the results of the same protocol under various shots are grouped into clusters of three colors, and the results of different protocols under 16-shot are connected in a fixed order.
Unlimited
Category Balanced
Test Set
LSLL--Postte
142
2
/Nuural :Negatio
SLR-Oversl 1/4
2"SLR 2"SUR Noutral
-1/2
"SSLS-tiepetvv
Label
2.19
352
Labell
227
311
15
Test
S.--Poshivee 1/3
148
SUR Neumal= 7/12
SLR-Rugetive 1/2
New Prediction
Niru
Prediction
Figure 7. Illustration of SLR under 6-shot setting. Taking SLR- Neutral as an example, the ICL sequences of the two Neutral test samples contain 4 and Neutral demonstrations respectively, thus SLR-Neutral is computed as (3/6- + 4/6) 22 7/12.
Figure 8. Confusion matrices of IDEFICS under the Unlimited and Category Balanced protocols on Twitter-17 (16-shot setting).
MLLM performs notably well on the positive samples, while it shows considerably diminished precision on neutral sam- ples and recall on negative samples. This reveals a potential predictive bias in the MLLM, which favors positive and neutral over negative. Among three practical protocols, the Unlimited protocol stands out with superior performance, particularly in positive and negative samples. Compared to the Contrary to Ideal protocol, attains a higher SLR in these two categories. This indicates that similarity- based retrieval improves SLR upon the original distribution in the support set. Though resulting in the least favorable performance, the Category Balanced protocol exhibits an intriguing phenomenon. It achieves the highest SLR in the rarest category (Neutral) while obtaining the lowest SLR in a more prevalent category (Negative), leading the model to classify samples of the latter as the former.
proven optimal, as it attains the finest equilibrium between information enrichment and input complexity.
4.2.3. SENTIMENT DISTRIBUTION
When assessing the impact of sentiment biases within the ICL sequence, it is intuitive to assume that the more demon- strations sharing the same sentiment as the test sample, the more likely the MLLM will make the correct prediction. Indeed, this is one of the primary mechanisms by which distribution protocols affect the performance of MLLMs. To quantify this, we introduce a metric termed "Same Label Rate (SLR)". First, the proportion of demonstrations with identical sentiments to each test sample is calculated. Then, these proportions are averaged across a collection of test samples. Depending on whether the collection comprises the entire test set or all test samples of specific category, SLR can be calculated for all test samples (SLR-Overall) or the corresponding sentiment (e.g., SLR- Figure 7 provides an example illustrating SLR. Utilizing SLR, Fig- ure 6 evaluates the nuanced impact of distribution protocols.
On Twitter-17 17, most distribution protocols still adhere to the positive correlation between SL-R-Oeerlll and accuracy (Figure 6 (b-1)), except for the Category Balanced protocol. It substantiates SLR as feasible indicator to reflect the effi- cacy of ICL sequences and drives us to explore the reasons behind this exception. In parallel to MVSA-S, the MILM shows slightly lower precision than recall on positive and neutral samples (Figure 6 (b-2,3)). Whereas, on negative samples (Figure (b-4)), recall remains remarkably lower than precision, which is intuitively illustrated in Figure 8. This further validates that the predictive bias observed in MVSA-S is irrelevant to datasets or demonstrations, and might be intrinsic to the MLLM itself. Such bias could po- tentially originate from the curation of the pre-training data, where most negative instances are filtered out, impairing
On MVSA-S, when the number of shots fixed, overall accuracy exhibits a stable positive correlation with SLR- Overall (Figure (a-1)). Under the extreme sentiment bi- ases imposed by the Ideal and Contrary to Ideal protocols, MILM's sentiment prediction tends to adapt correspond- ingly, with this trend becoming more pronounced as the number of shots increases. validates the vulnerability of MLLM to sentiment biases within the ICL sequence. Inspecting each sentiment category (Figure 6 (a-2,3,4)),
7

---

Configuring ICL Demonstrations for Unleashing MLLMs' Sentimental Perception Capability
Table 4, Accuracy comparison of MLL.Ms, SOTA few-shot models and SOTA fully-supervised models. We report the ICL performance of MLLMs under a 16-shot setting, as it includes sufficient demonstrations while not exceeding MLLMs input limit.
Post-Level
Aspect-Level
Model & Strategy
Support Set
Mean
MVSA-S
MVSA-M
TumEmo
Twitie-15
Twitier-17
MASAD
Zero Shot Paradigm
38.6
56.5
46.6
60.7
54.7
73.0
55.0
ICL Random 16-shol
1% Training Set
49.9
60.4
52.1
61.1
57.5
77.2
59.7
ICL RICES 16-shot (Yang al., 2022)
1%
Training Set
57.9
64.2
61.4
61.6
61.5
82.7
64.9
ICL SQPA 16-shot (Li et al., 2024)
1% Training Set
53.6
62.2
64.6
55.8
50.4
74.2
60.1
MLLM IDFFICS
ICL MMICES 16-shot (Chen et al., 2024)
1% Training Set
59.2
65,3
60.5
61.6
56.6
80.5
63.9
ICL. Ours 16-shot
1% Training Set
66.5
67.7
63.4
67.0
62.0
88.2
69.1
ICL Random 16-shot
Full Training Set
57.5
77.3
ICL RICES 16 shot (Yang et al., 2022)
Full Training Set
58.3
65.5
61.7
62.7
61.8
83.4
65.6
ICL SQPA 16-sbot (Li et 2024)
Training Set
52.1
75.1
ICL. MMICES 16 shot (Chen et al., 2024)
Full Training Set
63.0
64.6
62.3
59.2
54.5
83.5
64.5
Ours 16-shot
Training Set
63.7
89.8
7ero-Shot Paradigm
52.5
59.2
27.8
34.4
47.1
69.6
48.4
ICL. Random 16-shot
12 Training Set
59.2
67.3
34.8
57.6
$1.9
75.0
$7.6
RICES 16-shot (Yang et al., 2022)
1% Training
61.5
67.9
55.8
84.7
ICL. SQPA 16 shot (Li et al., 2024)
1% Training Set
59.5
68.3
44.9
56.2
$3.9
75.5
59.7
MLLM
MMICES 16-shot (Chen et al., 2024)
Training
63.3
68.0
46.4
56.9
55,4
80.8
Open-Flamingo
ICL Ours 16-shot
1% Training Set
65.2
68.5
52.7
68.8
61.4
87.6
67.4
ICL Random 16-shot
Full Training Set
59.0
67.4
34.5
$7.6
51.7
75.1
57.5
ICI. RICES 1hothot (Yang al., 2022)
Full Training Set
62.2
68.0
44.3
61.5
56.4
85.5
63.0
ICL SQPA 16-shot (Li et al., 2024)
Full Training Set
61.4
68.3
43.5
60.3
54.7
76.3
60.8
ICL MMICES 16-shot (Chen et al., 2024)
Full Training Set
64.7
57.0
82.4
63.3
ICL Ours 16-shor
Full Training Set
66.3
68.7
54.1
70.4
62.6
89.3
68.6
SOTA Few-Shot Models SOTA Sully-Supervised Models
69.8
68.3
58.1
67.3
61.9
84.1
68.3
79,2
73.5
66.7
81.1
76.4
95.6
78.8
Table 5. Our final strategies for configuring ICL demonstrations on six MSA datasets. Due to the diversity of task focus and data distribution, the optimal strategies for the three factors of cach dataset are different.
strategy. "ICL SQPA" and "ICL MMICES" are originally designed for the Visual Question Answering task. The for- mer assigns pseudo-labels to the samples in the support set, while the latter first retrieves images and then retrieves texts. In our case, pseudo-labels are assigned as random sentiment categories, and "ICL MMICES" is identical to the I strategy.
Datzel
Retrieval
Presentation
Distribution
S
WIT a 8 2:8
Image, Test
Unlimited
MVSA-M TomFana
WIT ( 2.8
Image, Text
Unlimited
WIT a B 2:8
Irage, Text
Unlimited
Twitter 15
WITA la: 9- 7:3, (a (0+,8::3 88:07 2:
Imuge. Test
Category Balanced
From Table 4, our devised strategies exhibit consistent and pronounced superiorities against other ICL baselines, ex- tending to datasets and MLLMs unused the exploration. By expanding the support set to the full training set, our approach enables both MLLMs to outperform the SOTA few-shot models. Notably, configuring demonstrations oper- ates independently of annotations, ensuring that the anno- tated data required for each test sample remains constant. This renders MLLMs' ICL more efficient and accurate in MSA than few-shot models. However, compared to fully- supervised models, ICL still exhibits a performance gap, particularly noticeable in certain datasets. It suggests that the role of fine-tuning with specific data still cannot be eas- ily replaced. Overall, Table 4 demonstrates the effectiveness and generalizability of our devised ICL strategies, as well as the considerable potential of ICL for further development.
Twmns-17
WITA (a:9-1 T. 01:
Image, Text
Category Balanced
MASAD
WITA : 9- 7, (0:-------000 2:8)
Inaaee Teut
Unlimited
MLLM's discernment of negative samples. In contrast to MVSA-S, the rarest category on Twitter-17 is negative. Un- der this disparity, the Category Balanced protocol instead inclines the model to predict the negative category over the positive category, offsetting MLLM's predictive bias and attaining the peak performance across both categories.
To summarize, two conclusions have been deduced, Firstly, MLLM exhibits a predictive bias in sentiment analysis, with an inclination to avoid negative predictions. Secondly, in datasets with less prevalent negative samples, the Category Balanced protocol emerges as the optimal selection by mit- igating MLLM's bias effectively. Otherwise, the Unlimited protocol is the preferable alternative.
5. Conclusion and Discussion
4.2.4. OVERALL STRATEGY
The overall strategy is crafted by integrating three finely op- timized factors, as presented in Table 5. Table 4 compares our devised strategies and other models. Among the ICL baselines we compared, "ICL Random" randomly selects demonstrations from the support set. "ICL RICES" retrieves samples based solely on the similarity measured by the WIT
In this paper, seek unleash the sentimental percep- tion capability of MLLMs through ICL. As pioneering effort that applies ICL to MSA, we conduct an in-depth investigation into three pivotal factors that influence the con- figuration of ICL demonstrations: similarity measurement, modality presentation, and sentiment distribution. For these
8

---

Configuring ICL Demonstrations for Unleashing MLLMS' Sentimental Perception Capability
factors, we optimize strategies by balancing the similarity components, weighing the richness of information against the complexity of inputs, and offsetting the sentimental pre- dictive bias of MLLMs. Comprehensive experiments on six datasets and two MILMs demonstrate the pronounced superiority of our strategies against other ICL baselines, val- idating their effectiveness and generalizability. Our findings confirm that MLLMs can perceive sentiment as competent as supervised models, paving the way for further research and exploration.
Alimisis, P., Mademlis, I., Radoglou-Grammaiiiis P., Sari- giannidis, P., and Papadopoulos, G. T. Advances in dif- fusion models for image data augmentation: A review of methods, models, evaluation metrics and future research directions, 2024.
Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., and et. al. Openflamingo: An open-source framework for training large autoregressivv laiounlanguage models, 2023.
A primary limitation of this paper lies in the range of MLLMs investigated. The effectiveness of ICL heavily de- pends on the MLLMs themselves, however, some advanced MLLMS are beyond the research scope due to practical reasons. Despite our progress, multimodal ICL still in its infancy compared to ICL on text modality. Other pects of configuring ICI. demonstrations also merit further investigation.
Baldassini, F. B., Shukor, M., Cord, M., Soulier, L., and Piwowarski, B. What makes multimodal in-context learn- ing work? In CVPR 2024- Workshops, pp. 1539-1550. 2024.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., and et. al. Language models are few-shot learners. In NeurIPS 2020, 2020.
Cai, C., Wang, Q., Liang, B., Qin, B., Yang, M.. Wong, K., and Xu, R. In-context learning for few-shot multimodal named entity recognition. In Findings of EMNLP 2023, pp. 1969-2979, 2023.
Acknowledgements
This work is supported by the National Natural Science Foundation of China (Grant NO 62376266, 62406318 18 and 62441614), Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu Univer- sity of China, Beijing, China.
Chen, H., Yang, X., Huang, Y., Wu, Z., Wang, J., and Geng, X. Manipulating the label space for in-context classification, 2023a.
Impact Statement
Chen, S., Han, Z., He, B., Buckley, M., Torr, P., Tresp, V., and Gu, J. Understanding and improving in-context learn- ing on vision- laiiunla models. ICLR 2024 Workshop, 2024.
This manuscript focuses on configuring ICL demonstra- tions to unleash MILMs' sentimental perception capability. By investigating three factors, we aim to advance multi- modal ICL in MLLMs and shed light on MLLMs' sentiment- related properties. During the investigation, we discover a sentimental predictive bias, later mitigating it on the in- ference level to facilitate fairness MI.LMs' sentiment prediction. We have not delved deeper into the source lim- itations of this bias, as it lies beyond the primary scope of research.
Chen, Y., Yuan, J., Tian, Y., Geng, S., Li, X., Zhou, D., Metaxas, D. N., and Yang, H. Revisiting multimodal representation in contrastive learning: From patch and token embeddings to finite discrete tokens. In CVPR 2023, pp. 15095-15104, 2023b.
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., R., Xia, H., Xu, J., Wu, Z., Chang, B., Sun, X., Li, L., and Sui, Z. A survey on in-context learning, 2024.
However, systematic studies of these limitations can poten- tially contribute to both MLLMs and MSA. As further discussion, we attribute these limitations to pretraining data rather than model architecture, which is validated by Xie al. (2024). By constructing emotion-related data, en- hances MILMs' zero-shot performance visual emotion recognition. This success has the potential to be replicated in solving the sentimental predictive bias.
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., and et. al. Palm-e: An embodied multimodal language model. In ICML 2023, volume 202 of Proceedings of Machine Learning Research, pp. 8488, 2023.
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., and et. al. Language is not all you need: Aligning perception with language models. In NeurIPS 2023, 2023.
References
Alayrac, J.. Donahue, J., Luc, P., Miech, A., Barr, I., and et. al. Flamingo: a visual language model for few-shot learning. In NeurlPS 2022, 2022.
Huang, Y., Du, C., Xue, Z., Chen, X., Zhao, H., and Huang, L. What makes multi-modal learning better than single (provably). In NewrIPS 2021, pp. 10944 10956,, 2021.

---

Configuring ICL Demonstrations for Unleashing MLLMS' Sentimental Perception Capability
Kosti, R., Álvarez, J. M., Recasens, A., and Lapedriza, A. Emotion recognition in context. In CVPR 2017, 1960- 1968, 2017.
Niu, T., Zhu, S., Pang, L.,, and El-Saddik, A. Sentiment anal- ysis multi-view social data. In MMM 2016, volume 9517 of Lecture Notes in Computer Science, pp. 15-27, 2016.
Laurençon, H., Saulnier, L.,, Tronchon, I.., Bekman, S., Singh, A., and al. OBELICS: open web-scale filtered dataset of interleaved image-text documents. In NeurlPS 2023, 2023.
Pan, J., Gao, T., Chen, H., and Chen, D. What in-context learning "learns" in-context: Disentangling task recog- nition and task learning. In Findings ACL 2023, pp. 8298 8319, 2023.
Levy, I., Bogin, B., and Berant, J. Diverse demonstrations improve in- in-context compositional generalization. In ACI. 2023, pp. 1401-1422, 2023.
Peng, T., Li, Z.,, Wang, P., Zhang, L., and Zhao, H. novel energy based model mechanism for multi-modal aspect- based sentiment analysis. In AAAI 2024, pp. 18869- 18878, 2024.
Li J., Li. D., Savarese, S., and Hoi, S. C. H. BLIP-2: bootstrapping language-imane image pre-training with frozen image encoders and large language models. In ICML 2023, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742, 2023.
Qin, L.,, Chen, Q., Fei, H., Chen, Z., Li, M., and Che, W. What factors affect multi-modal in-context learning? an in-depth exploration. In NeurIPS 2024, 2024.
Li, L.,, Peng. J., Chen, H., Gao, C., and Yang, X. How to configure good in-context sequence for visual question answering. In CVPR 2024, pp. 26710-26720, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., and et. al. Learning transferable visual models from natural language supervision. In ICML 2021, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763, 2021.
Li, X. and Qiu, X. Finding support examples for in-context learning. In Findings of EMNLP 2023, pp. 6219-6235, 2023.
Li, Z., Xu, B.. Zhu, C., and Zhao, T. CLMLF: contrastive learning and multi-layer fusion method for multimodal sentiment detection. In Findings of NAACL 2022, pp. 2282-2294, 2022.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with la- tent diffusion models. CVPR 2022, pp. 10674-10685, 2022.
Lian, Z., Sun, L., Sun, H., Chen, K., Wen, Z., Gu, H., Liu, B., and Tao, GPT-4V with emotion: zero- shot benchmark for generalized emotion recognition. Inf. Fusion, 108: 102367, 2024.
Shukor, M., Rame, A., Dancette, C., and Cord, M. Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning. In ICLR 2024, 2024.
Ling, Y., Yu, and Xia, Vision-language pre-training for multimodal aspect- based sentiment analysis. In ACL 2022, pp. 2149-2159, 2022.
Wang, Y., Chen, W., Han, X., Lin, X., Zhao, H., Liu, Y., Zhai, B., Yuan, J., You, Q., and Yang, H. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning. 2024.
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for gpt- 3? In Agirre, E., Apidianaki, M., and Vulic, I. (eds.), DeeL @ ACL 2022, pp. 100-114, 2022.
Wei, Y., Yuan, S., Yang. R., Shen, L., Li, Z., Wang, L.,, and Chen, M. Tackling modality heterogeneity with multi- view calibration network for multimodal sentiment detec- tion. In ACL 2023, pp. 5240-5252, 2023.
Lu, D., Neves, L., Carvalho, V., Zhang, N., and Ji, H. Visual attention model for name tagging in multimodal social media. In ACL 2018, pp. 1990-1999, 2018.
Wu, D., Yang, D., Zhou, Y., and Ma, C. Robust multimodal sentiment analysis of image-text pairs by distribution- based feature recovery and fusion. ACM MM 2024, pp. 5780-5788 2024a.
Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACI. 2022, pp. 0086-8098, 2022.
Lyu, X., Min, S., Beltagy. I., Zettlemoyer, L., and Hajishirzi, H. Z-ICL: zero-shot in-context learning with pseudo- demonstrations, In Rogers, A., Boyd-Graber, J. L., and Okazaki, (eds.), ACI. 2023, pp. 2304-2317, 2023.
Wu, D., Yang, D., Zhou, Y., and Ma. C. Bridging visual affective gap: Borrowing textual knowledge by learning from noisy image-text pairs. In ACM MM 2024, pp. 602- 611, 2024b.
10

---

Configuring ICL Demonstrations for Unleashing MLLMs' Sentimental Perception Capability
Wu, Z., Wang, Y., Ye, J., and Kong, L. Self-adaptive in- context learning: An information compression perspec- tive for in-context example selection and ordering. In ACL 2023, pp. 1423-1436, 2023.
Yang, Z., Gan, Z.,, Wang. J., Hu, X., Lu, Y., Liu, Z., and Wang, L. An empirical study of GPT-3 for few-shot knowledge-based VQA. In AAAI 2022, pp. 3081-3089, 2022.
Xie, H., Peng, C., Tseng, Y., Chen, H., Hsu, C., Shuai, H., and Cheng, W. Emovit: Revolutionizing emotion insights with visual instruction tuning. In CVPR 2024, pp. 26586- 225966226555222 2024.
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey on multimodal large language models, 2023.
Yu, J. and Jiang, J. Adapting BERT for target-oriented multimodal sentiment classification. In Kraus, S. (ed.), IJCAI 2019, pp. 5408-5400000 5414,444,222 2019.
Xu, K. Ba, J., Kiros, R., Cho, K., Courville, A. C., Salakhut- dinov, R., Zemel, R. S., and Bengio, Y. Show, attend and tell: Neural image caption generation with visual atten- tion. In ICML 2015, volume 37, pp. 2048 2057, 2015.
Yu, Y. Zhang, D., and Li, S. Unified multi-modal pre- training for few-shot sentiment analysis with prompt- based learning. In ACM MM 2022, pp. 189-198, 2022.
Xu, N. Analyzing multimodal public sentiment based on hierarchical semantic attentional network. In ISI 2017, pp. 152- 154, 2017.
Yue, L., Chen, W., Li, X., Zuo, W., and Yin, M. A survey of sentiment analysis in social media. Knowl. Inf. Syst., 60(2):617-663, 2019.
Xu, N. and Mao, W. Multisentinet: A deep semantic net- work for multimodal sentiment analysis. In CIKM 2017, pp. 2392-2402, 2017.
Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L., Tensor fusion network for multimodal sentiment analysis.
In EMNLP 2017, pp. 1 1103-1114 2017.
Zhang, L., Wang. S., and Liu, B. Deep learning for sen- timent analysis: A survey, WIREs Data Mining Knowl. Discov, 8(4), 2018a.
Xu, N., Mao, W., and Chen, G. A co-memory network for multimodal sentiment analysis. In SIGIR 2018, pp. 929-932, 2018.
Zhang, Q., Fu, J., Liu, X., and Huang, X. Adaptive co- attention network for named entity recognition in tweets. In AAA/ 2018, pp. 5674-5681, 2018b.
Xu, N., Mao, W., and Chen, G. Multi-interactive memory network for aspect based multimodal sentiment analysis. In AAA/ 2019, pp. 371-378, 2019.
Zhang, S., Pan, Y., and Wang, J. Z. Learning emotion repre- sentations from verbal and nonverbal communication. In CVPR 2023, pp. 18993-19004 2023.
Yang, H., Zhao, Y., Wu, Y., Wang. S., Zheng, T., Zhang, H., Che, W., and Qin, B. Large language models meet text-centric multimodal sentiment analysis: A survey, 2024.
Zhang. Y., Feng, S.,. and Tan, C. Active example selection for in-context learning. In EMNLP 2022, pp. 9134-9148 2022.
Yang, X., Feng, S., Wang, D., and Zhang, Y. Image-text mul- timodal emotion classification via multi-view attentional network. IEEE Trans. Multim., 23:4014-4026, 2021a.
Zhao, S., Yao. X., Yang. J., Jia, G., Ding, G., Chua, T., Schuller, B. W., and Keutzer, K. Affective image con- tent analysis: Two decades review and new perspectives. Trans. Pattern Anal. Mach. Intell., (4(10):6729-6751 2022.
Yang, X., Feng. S., Zhang, Y., and Wang, D. Multimodal sentiment detection based on multi-channel graph neural networks. In ACI/UICNPP 202/, pp. 328-339, 2021b. b021b.
Zhou, J., Zhao, J., Huang, J. X., Hu, Q. V., and Hc, L. MASAD: A large-scale dataset for multimodal aspect- based sentiment analysis. Neurocomputing, 455:47-58, 2021.
Yang, X., Feng, S., Wang, D., Zhang, Y., and Poria, S. Few- shot multimodal sentiment analysis based on multimodal probabilistic fusion prompts. In ACM MM 2023, pp- 6045-6053, 2023a.
Zhou, R., Guo, W., Liu, X., Yu, S., Zhang, Y., and Yuan, X. Aom: Detecting aspect-oriented information for mul- timodal aspect-based sentiment analysis. In Findings of ACL 2023, pp. 8184-8196, 2023.
Yang, X., Wu, W., Feng, S., Wang, M., Wang, D., Li, Y., Sun, Q., Zhang, Y., Fu, X., and Poria, S. Mm bigbench:::: Evaluating multimodal models on multimodal content comprehension tasks, 2023b.
Zhu, X., Guo, C., Feng, H., Huang, Y., Feng, Y., Wang, X., and Wang, R. A review of key technologies for emotion analysis using multimodal information. Cogn. Comput., 16(4):1504-1533 1530, 2024.
Yang, X., Wu, Y., Yang, M., Chen, H., and Geng, X. Ex- ploring diverse in in-context configurations for image cap- tioning. In NeurIPS 2023, 2023c.

---

Configuring ICL Demonstrations for Unleashing MLLMS' Sentimental Perception Capability
A. Textual Prompts
The textual prompt P of multimodal ICL sequence S aims to provide the task description and the set target categories.
A.1. Aspect-Level MSA
P is "A post contains an image, a text and an aspect. Identify the sentiment of the aspect in the post. The optional categories are [Positive, Neutral, and Negative]. Here are some examples".
A.2. Post-Level MSA
P is "A post contains an image and a text. Classify the sentiment of the post into [Positive, Neutral, Negative]. Here are some examples".
A.3. Sensitivity of ICL to Prompt Variations
In the investigation, we experiment with various textual prompts and find that they significantly impact zero-shot performance. However, their impact on ICL is minimal. Since this manuscript primarily focuses on how ICL configurations influence MLLMS' sentiment perception capabilities, we select a set of appropriate textual prompts and keep them fixed throughout the investigation. The performance of IDEFICS under different prompts is reported in Table 6.
For post-level MSA:
#1 Prompt: A post contains an image and text. Classify the sentiment of the post into [Positive, Neutral, Negative].
#2 Prompt: Please classify the sentiment of the image-text post into [Positive, Neutral, Negative].
#3 Prompt: Here a post containing an image and text, The optional categories are [Positive, Neutral, Negative]. What the overall sentiment of the post?
For aspect-level MSA:
#1 Prompt: A post contains an image, text and an aspect. Identify the sentiment of the aspect in the post. The optional categories are [Positive, Neutral, Negative].
#2 Prompt: Please classify the sentiment of the aspect in image-texi post into [Positive, Neutral, Negative].
#3 Prompt: Here a post containing an image, a text and an aspect. The optional categories are [Positive, Neutral, Negative].
What is the sentiment of the aspect in the post?
Table Influence of prompt variations accuracy of IDEFICS.
MVSA-S
Twitter-15 #2 Prompt
Model & Strategy
Support Set
#1 Prompt
Prompt
#3 Prompt
#1 Prompt
#3 Prompt
Zero-Shot Paradigm
38.6
28.2 66.3
50.6 66.4
60.7
51.9
19.1
IEFFCC
ICL Ours 16-shot
1% Training Set
66.5
67.0
66.9
66.7
B. Dataset Details
The statistics of the adopted datasets are presented in Table
MVSA-S, MVSA-M (Niu et al.. 2016) are labeled on single modalities, where the sentiment categories include Positive, Neutral and Negative. The multimodal sentiment categories are obtained by majority voting following Xu & Mao (2017).
TumEmo (Yang et al., 20)1a) is weakly supervised dataset. Image-text posts are retrieved based seven emotion keywords: Love, Happy, Calm, Bored, Sad, Angry, Fear, and labeled accordingly.
Twitter-1e 15 (Zhang et al., 2018b), Twitter- 17 (Lu et al., 2018) are initially proposed for Multimodal Named Entity Recognition. Their named entities are later annotated by Yu & Jiang (2019) based on the sentiment polarities: Positive, Neutral, Negative, and utilized for aspect-level MSA.
12

---

Configuring ICL Demonstrations for Unleashing MILMS' Sentimental Perception Capability
MASAD (Zhou al., 2021) extends textual aspects visual aspects, and includes posts from diverse domains. The aspects labeled by sentiment polarities: Positive and Negative.
Table 7. Statisties of datasets.
Dataset
Train
Test
MVSA-S (Niu et al., 2016)
3608
452
Post-Levee
MVSA-M (Niu et al., 2016)
13618
1703
TumFmo (Yang et al., 2021a)
156217
19524
Twitter-15 (Zhang et al., 2018b)
3179
1037
Aspect-Level
Twitter- 15 (Lu et al., 2018)
3562
1234
MASAD (Zhou al., 2021)
14868
4935
C. Computational Overheads
In the optimized configuration, presenting and distributing demonstrations do not introduce additional computational overhead. The extra costs originate from demonstration retrieval and the expanded input sequence for MLLMs. The former scales with the size of the support set, each sample needs to be compared against all support set samples, while the latter is inherent ICL. We report the average time overhead (ms) of processing image-text sample under two support set scales on a single NVIDIA GeForce RTX 4090 GPU.
Table 8. Comparison of time costs.
Time Overhead (ms)
Model & Strategy
Support Set
Retrieval
Inference
Total
Zero-Shot Paradigm
0
78.1
78.1
ICI. Random 4-shot
136 / 1562 Samples
0
134.5
134.5
ICL Ours 4-shot
136 Samples
36.4
134.5
170.9
IDEFICS
ICL Ours 4-shot
1562 Samples
64.2
134.5
198.7
ICL Random 16-shot
136 / 1562 Samples
0
346.1
346.1
ICL Ours 16-shot
136 Samples
36.4
346.1
382.5
ICL Ours 16-shot
1562 Samples
64.2
346.1
410.3
D. Complete Results
In the main paper, we simplify the reported results to emphasize the key findings. Here we present the complete results in numerical form.
Table 2 reports the average accuracy across 4,8,16-shot demonstrations retrieved based on varying similarity measurements,
Table 9 is its complete version.
Figure reports the average accuracy across 4,8, 16-shot demonstrations retrieved based on the WIT and WITA strategies. Table 10 is its complete version.
Table 3 reports the average accuracy across 4,8,16-sho settings with the inputs composed of different modalities. Table 11 its complete version.
Figure 5 evaluates ICL's "Task Learning" effect by progressively incorporating modalities into the inputs. Table 12 its complete version.
13

---

Configuring ICL Demonstrations for Unleashing MILMS' Sentimental Perception Capability
Table Complete results of Table
4-shot
8-sbot
16-shot
Strategy
Levec
Peve-
Aspect-
Post-Level
Aspect-
MVSA-S
MVSA-M
Twitter155
Twitter-17
MVSA-8
MVSA-M
Twitter-15
Twitter-17
MVSA-S
MVSA-M
Twitter-15
Twitter-17
45.2
59.7
56.1
55.2
50.8
61.2
58.8
56.9
51.4
61.6
57.3
57.1
I
51.2
64.3
56.5
56.8
59.2
65.2
59.2
56.7
59.2
65.3
61.6
56.6
T
49.6
64.6
55.0
55.2
$8.1
67.2
58.6
56.8
60.0
66.8
62.5
58.9
49.5
64.8
60.5
58.7
57.7
66.3
62.2
56.2
60.1
67.3
61.6
57.9
A
60.2
57.0
60.5
$7.6
63.4
57.7
TA
56.6
61.6
57.3
62.4
58.0
ITA
59.3
58.3
61.6
57.5
62.2
58.5
Table I0. Complete results of Figure 4.
WIT Strategy
4-shot
8-shot
16-hbt
WITA Strategy
4-shot
8-shot
16-sbo
MVSA-S
MVSA-M
MVSA-S
MVSA-M
MVSA-S
MVSA-M
S:
Twitter-15 15
Twitte-117
Twitter-15
Twistr-17 17
Twitte-115
Twitter-
1:9
51.7
65.5
58.5
67.0
61.4
66.3
1:9: 5:5
58.6
56.4
62.5
57.5
03.2
58.5
67.1
66.5
$6.5
57.9
3:7
50.2
65.4
57.2
67.7
64.9
66.9
3/7: 8:2
56.6
55.8
60.5
55.9
62.9
59.1
466
49.0
64.9
57.7
67.2
62.7
67.2
3/7: 5:5
59.7
$8.2
61.7
$7.3
62.8
58.4
57.8
6:4
49.5
64.4
$8.1
67.2
59.4
66.7
5:5: 8:2
$7.8
57.1
62.5
$6.5
64.0
58.1
51.2
64.2
57.7
66.8
59.2
66.9
5:5;
58.8
$7.9
60.2
$6.8
62.0
58.2
51.2
64.6
58.8
67.0
57.7
66.7
5:5:
39.8
60.2
57.5
58.7
9:1
30.8
64.8
$7.9
65.6
572
66.2
7:3: 8:2
55.7
57.2
58.3
$8.2
62.4
58.6
7:3;
60.5
$8.9
00.5
59.3
64.4
58.7
9:1:
55.0
39.7
57.7
63.0
38.5
9:1:
60.5
58.8
60.6
$7.9
63.8
59.0
Table 11. Complete results of Table 3.
shot
shot
16 shot
Modality
Post- Post-Level
Aspect-Lev Level
Post-Leve
Aspect-Level
Post-
Aspect-Level
MVSA-S
MVSA-M
Twitter117
MVSA-M
Twitter-15
Twitter-17
MVSA-S
Twitter.
Twitter-17
33,5
45.7
55.2
51.7
57.8
58.5
53.8
66.3
66.5
5844
54.7
C
26.2
35.1
53.3
49.8
47.2
52.3
56.8
54.1
58.3
61.1
58.5
52.4
LCC
48.8
54.9
52.3
50.1
60.1
60.9
53.4
50.1
61.9
62.0
57.1
50.1
T
29.7
47.6
60.8
$8.9
48.6
56.3
61.5
59.1
60.8
64.2
62.9
59.0
31.7
44.0
60.3
55.2
T.G
34.2
47.5
49.0
61.0
55.4
55.0
58.3
65.7
57.6
55.4
LT
49.5
64.8
60.5
58.7
57.7
66.3
62.2
56.2
60.1
67.3
61.6
57.9
LG
47.3
54.2
53.1
$1.8
56.4
$8.9
$5.0
53.4
$9.8
63.3
55.4
56.2
C.T
32.2
54.5
61.5
56.4
55.7
61.8
63.9
57.3
61.2
64.6
62.4
56.3
C.G
33.7
41.0
54.3
59.0
55.2
63.3
56.5
$6.2
63.5
LT.G
45.0
58.6
58.0
53.9
53.9
62.4
60.3
55.2
$3.4
64.8
61.9
55.2
C.T.G
36.1
50.5
$1.0
$3.0
48.6
59.9
$7.8
55.4
$7.7
64.2
60.9
54.0
LC.G
41.0
52.3
52.8
51.1
54.1
57.5
51.1
52.4
55.9
61.2
54.7
52.9
L.CTTG
59.4
53.6
62.2
Table 12. Complete results Figure
4-shot
8-shot
16-shot
Modality
Twitter-1 15
Twitter
Twitter
Twitter 17
Twitter 15
Twitter
33.7
43.0
39.6
44.9
52.7
48.5
+1
30.4
36.7
44.7
48.2
46.7
C.
30.3
43.7
33.4
43.3
49.2
45.7
23.2
39.7
35.8
44.4
49.9
45.8
14


Reviews:
[{"review_id": "4G0Bfl82gQ", "reviewer": "Reviewer_nCbm", "summary": "The paper addresses Multimodal Sentiment Analysis using MLLMs by enhancing In-Context Learning through optimized demonstration retrieval, presentation, and distribution. It achieves significant accuracy gains over zero-shot and random ICL baselines and mitigates inherent sentiment bias.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback and helpful comments! Below, we present detailed responses to the questions (**Q**) and weaknesses (**W**).\n\n>**Q1. Generalization to other tasks or datasets**\n\nOur strategies cover three factors: similarity measurement, modality presentation, and sentiment distribution. These components can be harmoniously transferred to other multimodal tasks, such as multimodal sarcasm detection and multimodal crisis event detection. Both tasks accept image-text inputs, allowing direct adoption of the similarity measurement strategy (WIT) and modality presentation strategy (Image, Text). Regarding classification targets, multimodal sarcasm detection focuses on whether sarcasm expression is present, while multimodal crisis event detection aims to identify disaster types. By generalizing the concept of sentiment distribution to the distribution of different categories, we can replicate the Category Balanced protocol on these tasks. \n\nExperiments on HFM [1] (multimodal sarcasm detection) and CrisisMMD [2] (multimodal crisis event detection) demonstrate the generalizability of our strategies. We report accuracy and adopt 1% of the training set as the support set.\n\n|IDEFICS|HFM|CrisisMMD|\n|-|:-:|:-:|\n|Zero-Shot Paradigm|71.8|69.4|\n|ICL Random 16-shot|78.9|74.1|\n|ICL RICES 16-shot|79.2|81.6|\n|ICL Ours 16-shot|84.7|86.3| \n\nRegarding the scale and diversity of datasets, they do not affect the application of the strategies themselves. When applying to larger-scale datasets, the size of the support set can influence both the effect of ICL and computational overhead, necessitating a trade-off based on specific priorities.\n\n>**W1. Requirement of calibration in real-world scenarios**\n\nWhile the optimization process includes additional computational overhead, our optimized strategies exhibit strong generalizability across six MSA datasets (as in the manuscript), other multimodal tasks (according to the response to **Q1**), and diverse MLLM frameworks (according to the response to **C1** of Reviewer gx7F). Therefore, in practical applications, our strategies can achieve promising results without further calibration.\n\n>**Q2. Computational overhead compared to the zero-shot paradigm**\n\nIn the optimized configuration, presenting and distributing demonstrations do not introduce additional computational overhead. The extra costs originate from demonstration retrieval and the expanded input sequence for MLLMs. The former scales with the size of the support set, as each test sample needs to be compared against all support set samples, while the latter is inherent to ICL. We report the average time overhead (ms) of processing an image-text sample under two support set scales.\n\n\n|IDEFICS|# of Samples in Support Set|Retrieval|Inference|Total|\n|-|:-:|:-:|:-:|:-:|\n|Zero-Shot Paradigm|-|-|78.1|78.1|\n|ICL Random 4-shot|-|-|134.5|134.5|\n|ICL Ours 4-shot|136|36.4|134.5|170.9|\n|ICL Ours 4-shot|1562|64.2|134.5|198.7|\n|ICL Random 16-shot|-|-|346.1|346.1|\n|ICL Ours 16-shot|136|36.4|346.1|382.5|\n|ICL Ours 16-shot|1562|64.2|346.1|410.3|\n\nCompared to the zero-shot paradigm, most additional time overhead is introduced by ICL itself. The cost introduced by our strategies accounts for a minimal proportion, demonstrating their efficiency.\n\n>**Q3. Identification and mitigation of the sentimental predictive bias**\n\nIn the manuscript, the sentimental predictive bias is identified based on the precision and recall metrics of the MLLM across different sentiment samples. As shown in Figure 6 (a-4) (b-4), the recall of negative samples (0-30/0-30) is significantly lower than the precision (50-90/54-88) on both datasets, implying the MLLM’s tendency to favor positive and neutral predictions over negative predictions.\n\nOn Twitter-17, the Category Balanced protocol (indicated by diamond within the red clusters in Figure 6 (b-4)) configures more negative demonstrations (higher SLR-Negative) compared to the other protocols. Affected by the short-cut effect, this configuration amplifies the MLLM's tendency to conduct negative predictions during inference, thereby mitigating this bias.\n\nA more intuitive set of metrics is the proportion of negative samples classified by the MLLM as positive ($P_{pos}$), neutral ($P_{neu}$), and negative ($P_{neg}$). The table below demonstrates that nearly all negative samples under the Unlimited protocol are misclassified as positive or neutral, revealing significant predictive bias, whereas this bias is mitigated by the Category Balanced protocol. In the revised manuscript, we will integrate these analyses and include case studies to clarify our findings.\n|IDEFICS|$P_{pos}$|$P_{neu}$|$P_{neg}$|\n|-|:-:|:-:|:-:|\n|Unlimited|21.0%|73.0%|6.0%|\n|Category Balanced|11.3%|60.7%|28.0%|\n\n>**References**\n\n[1] Cai et al. Multimodal Sarcasm Detection in Twitter with Hierarchical Fusion Model. ACL, 2019. \\\n[2] Alam et al. CrisisMMD: Multimodal Twitter Datasets from Natural Disasters. AAAI, 2018.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There are no theoretical proofs provided in the paper.", "experimental_designs_or_analyses": "The experimental design is overall rigorous, validating the proposed improvements against zero-shot and random ICL baselines. However, concerns remain regarding dataset diversity, potential overfitting in demonstration selection, and computational overhead.", "relation_to_broader_scientific_literature": "The paper builds on recent advances in multimodal learning by extending ICL strategies for sentiment analysis. It contributes to the literature by refining demonstration selection and prompt engineering while addressing sentiment bias in MLLMs.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n+ The paper tackles Multimodal Sentiment Analysis (MSA), which is a pivotal yet challenging task in the realm of MLLMs. It demonstrates that with proper In-Context Learning (ICL) configuration, MLLMs can achieve significantly better performance in sentiment analysis.\n+ The discovery and mitigation of a sentimental predictive bias in MLLMs not only improves accuracy but also contributes to fairness in sentiment prediction, which is a valuable consideration in AI applications.\n\nWeaknesses\n- The requirement to fine-tune multiple aspects of demonstration configuration (retrieval, presentation, and distribution) may introduce additional complexity in implementation. This could make it challenging to deploy the approach in real-world scenarios without careful calibration.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Could you elaborate on how your ICL demonstration configuration strategies generalize to other multimodal tasks or larger, more diverse datasets beyond the six MSA datasets tested?\n2. What are the computational and time overheads associated with retrieving, presenting, and distributing demonstrations compared to the zero-shot paradigm?\n3. How exactly is the sentimental predictive bias identified and mitigated? Are there any specific metrics or case studies that demonstrate the effectiveness of this bias reduction?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UkeqZiNm5a", "reviewer": "Reviewer_gx7F", "summary": "This paper explores using In-Context Learning (ICL) to enhance MLLMs for Multimodal Sentiment Analysis (MSA). The authors identify that MLLMs under the zero-shot paradigm exhibit weak performance on MSA tasks. They propose a systematic study of three key factors in ICL demonstration configuration: similarity measurement, modality presentation, and sentiment distribution. By optimizing these factors, they achieve average accuracy improvements of 15.9% over zero-shot performance and 11.2% over random ICL on six MSA datasets. They also identify and mitigate a sentimental predictive bias in MLLMs, leading to fairer sentiment classification.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback and constructive suggestions! Below, we present detailed responses to the weaknesses (**W**), comments (**C**) and questions (**Q**).\n\n>**W1. Computational feasibility of optimal strategies**\n\nPlease refer to the response to **Q2** of Reviewer nCbm.\n\n>**W2. Fine-tuning MLLMs on support set**\n\nOn MVSA-S and Twitter-15, we sample three support sets (each comprising 1% of the training data) using three distinct random seeds. On these, we perform LoRA fine-tuning on the Q and V matrices within IDEFICS's gated xattn-dense layers, with a batch size of 1, a learning rate of 1e-4, and train for 3,000 steps. The training data is constructed in a zero-shot format, where the MLLM processes one image-text pair and its sentiment label at a time. The results (Accuracy) are reported below.\n\n|IDEFICS|Support Set|MVSA-S|Twitter-15|\n|-|:-:|:-:|:-:|\n|Zero-Shot Paradigm|-|38.6|60.7|\n|Zero-Shot Paradigm after Lora Fine-Tuning|#1|45.8|63.3|\n|ICL Ours 16-shot|#1|66.5|67.0|\n|Zero-Shot Paradigm after Lora Fine-Tuning|#2|48.9|62.3|\n|ICL Ours 16-shot|#2|67.2|66.6|\n|Zero-Shot Paradigm after Lora Fine-Tuning|#3|46.1|63.4|\n|ICL Ours 16-shot|#3|66.7|67.2|\n\nLoRA fine-tuning significantly improves the MLLM's sentiment perception capability under the zero-shot paradigm, yet it still lags behind the optimized ICL configuration. Across different support sets, ICL demonstrates more stable performance compared to LoRA fine-tuning. These results will be incorporated into the revised manuscript to enrich baselines and enhance the reliability of the findings.\n\n>**C1. Generalization to different MLLM architectures**\n\nWe evaluate our optimized strategies on two other MLLMs: MiniCPM-o-2.6-8B [1] and GPT4o [2], and report the results (Accuracy) below. Our ICL strategies still demonstrate consistent performance advantages over other strategies, confirming their generalizability.\n\n|MiniCPM-o-2.6|Support Set|MVSA-S|Twitter-15|GPT4o|Support Set|MVSA-S|Twitter-15|\n|-|:-:|:-:|:-:|-|:-:|:-:|:-:|\n|Zero-Shot Paradigm|-|56.0|52.8|Zero-Shot Paradigm|-|60.8|59.4|\n|ICL Random 16-shot|1% Training|60.6|59.9|ICL Random 16-shot|1% Training|63.6|61.0|\n|ICL RICES 16-shot|1% Training|62.5|61.5|ICL RICES 16-shot|1% Training|66.2|61.8|\n|ICL Ours 16-shot|1% Training|67.4|68.2|ICL Ours 16-shot|1% Training|72.5|68.7|\n|ICL Random 16-shot|100% Training|60.4|59.9|ICL Random 16-shot|100% Training|63.7|61.1|\n|ICL RICES 16-shot|100% Training|63.6|62.3|ICL RICES 16-shot|100% Training|67.4|62.3|\n|ICL Ours 16-shot|100% Training|67.9|70.3|ICL Ours 16-shot|100% Training|74.1|69.7|\n\n>**C2&Q2. Randomness in the selection of the support set**\n\nPlease refer to the response to **W2**.\n\n>**Q1. Sensitivity of ICL to prompt variations**\n\nIn the investigation, we experiment with various textual prompts and find that they significantly impact zero-shot performance. However, their impact on ICL is minimal. Since this manuscript primarily focuses on how ICL configurations influence MLLMs' sentiment perception capabilities, we select a set of appropriate textual prompts (#1 Prompt below) and keep them fixed throughout the investigation. The performance (Accuracy) of IDEFICS under different prompts is reported below. The support set contains 1% data from the training set.\n\nFor post-level MSA: \\\n#1 Prompt: A post contains an image and a text. Classify the sentiment of the post into [Positive, Neutral, Negative]. \\\n#2 Prompt: Please classify the sentiment of the image-text post into [Positive, Neutral, Negative]. \\\n#3 Prompt: Here is a post containing an image and a text. The optional categories are [Positive, Neutral, Negative]. What is the overall sentiment of the post? \n\nFor aspect-level MSA: \\\n#1 Prompt: A post contains an image, a text and an aspect. Identify the sentiment of the aspect in the post. The optional categories are [Positive, Neutral, Negative]. \\\n#2 Prompt: Please classify the sentiment of the aspect in image-text post into [Positive, Neutral, Negative]. \\\n#3 Prompt: Here is a post containing an image, a text and an aspect. The optional categories are [Positive, Neutral, Negative]. What is the sentiment of the aspect in the post? \n\n|IDEFICS|Prompt|MVSA-S|Twitter-15|\n|-|:-:|:-:|:-:|\n|Zero-Shot Paradigm|#1|38.6|60.7|\n|ICL Ours 16-shot|#1|66.5|67.0|\n|Zero-Shot Paradigm|#2|28.2|51.9|\n|ICL Ours 16-shot|#2|66.3|66.9|\n|Zero-Shot Paradigm|#3|50.6|19.1|\n|ICL Ours 16-shot|#3|66.4|66.7|\n\n>**References**\n\n[1] Yao et al. MiniCPM-v: A GPT-4V Level MLLM on Your Phone. Arxiv, 2024. \\\n[2] OpenAI. GPT-4 Technical Report. Arxiv, 2023.", "claims_and_evidence": "Yes.\nThe authors claim that MLLMs can achieve competitive sentiment perception through ICL. Three key factors (similarity measurement, modality presentation, and sentiment distribution) significantly impact ICL performance. Sentimental predictive bias exists in MLLMs but can be mitigated via distribution balancing. \nEmpirical results on six MSA datasets validate these claims. The proposed retrieval, presentation, and distribution strategies are tested against zero-shot, random ICL, and previous ICL strategies. Ablation studies strengthen the validity of the proposed method.", "methods_and_evaluation_criteria": "Yes. \nThis paper systematically studies three key factors in ICL demonstration configuration: similarity measurement, modality presentation, and sentiment distribution. For similarity measurement, the authors have evaluated various retrieval strategies, including aspect-based similarity and weighted multimodal similarity. For modality presentation, the authors explore how different combinations of texts, images, and generated modalities affect ICL performance. For sentiment distribution, the impact of sentiment biases in demonstrations is analyzed, leading to the discovery of predictive bias in MLLMs. The evaluation on six standard MSA datasets with accuracy as the primary metric also makes sense for this study.", "theoretical_claims": "This paper primarily focuses on empirical findings rather than theoretical derivations.", "experimental_designs_or_analyses": "Experiments on six datasets ensure the generalization of the proposed ICL strategy. Ablation studies validate individual contributions of retrieval, modality, and distribution factors. Baseline comparisons are thorough, including random ICL, prior ICL strategies, and supervised models.\nHowever, the experiments lack the analysis of the computational cost of configuring optimal ICL demonstrations.", "relation_to_broader_scientific_literature": "Although ICL for MLLMs has been studied in other multimodal tasks, it is underexplored for MSA, which is a pivotal challenge in the quest for general artificial intelligence. The paper builds on prior work in ICL (Brown et al., 2020), multimodal learning (Yin et al., 2023), and MSA (Zadeh et al., 2017). The discussion of MLLMs' zero-shot limitations aligns with recent findings in multimodal intelligence (Lian et al., 2024; Yang et al., 2023). The work extends ICL research to MSA, aiming to unleash the sentiment perception ability of MLLMs, which is a valuable contribution.", "essential_references_not_discussed": "No. The related works of this paper are sufficient, including fully-supervised, few-shot MSA methods and in-context learning methods.", "other_strengths_and_weaknesses": "Strengths:\n\n1.The paper presents a novel and systematic exploration of ICL for MSA, an area that has received limited attention. It combines existing ICL strategies with sentiment bias mitigation, offering new perspectives on optimizing MLLM performance.\n\n2.Addressing sentiment bias in MLLMs is an important contribution, as it improves both model fairness and reliability.\n\n3.The paper is well-written and structured. The proposed strategies are easy to follow.\n\n4.The experimental design is clearly explained, and sufficient ablation studies strengthen the empirical validation.\n\nWeaknesses:\n\n1.While ICL offers benefits, the computational feasibility of optimal retrieval strategies should be discussed.\n\n2.Fine-tuning on support set could serve as additional baselines to contextualize ICL’s strengths and trade-offs.", "other_comments_or_suggestions": "1.It would be better to provide an analysis of whether the ICL strategies can generalize to different MLLM architectures beyond IDEFICS and Open-Flamingo.\n\n2.The randomness of the 1% support set might influence the performance. A discussion is needed.", "questions_for_authors": "1.Did the authors experiment with different prompt engineering techniques for ICL in MLLMs? Understanding the sensitivity of results to prompt variations would be valuable.\n\n2.Have the authors tried to select the support set several times randomly?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E3yVfJKZ89", "reviewer": "Reviewer_pNwb", "summary": "This paper conducts an empirical study to unleash the power of MLLMs using in-context learning for multimodal sentiment analysis. The authors study three key factors influencing in-context learning performance: similarity measurement, modality presentation, and sentiment distribution. Experiments are performed on six datasets, and the proposed optimized strategies result in a significant improvement, outperforming both the zero-shot method and in-context learning with a random selection strategy. The paper also highlights and addresses an inherent sentiment prediction bias in MLLMs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback and valuable advice! Below, we present detailed responses to the weaknesses (**W**), comments (**C**), questions (**Q**) and other concerns (**O**).\n\n>**C1&W1. Unclear figures**\n\nIn the revised manuscript, we will replace redundant components in Figure 6 with more intuitive indicators and reformulate the calculation process in Figure 3(a) using variable representations.\n\n>**Q1. Explanation of Figure 5 experiment**\n\nTable 3 indicates that neither substituting nor augmenting original modalities with auxiliary ones improves ICL performance. Our explanation for the former is that modality conversion introduces information loss, outweighing potential benefits. The experiments in Figure 5 are designed to interpret why the latter degrades ICL performance.\n\nSpecifically, [1] decomposes ICL’s role into Task Recognition (TR) and Task Learning (TL). TR prompts the task format for MLLMs to apply their prior knowledge, and TL aids MLLMs in building a mapping between inputs and outputs. We hypothesize that the augmenting process weakens the TL effect. To validate this, we reformulate MSA tasks to map image-text pairs to specific animals, where MLLMs have no prior knowledge of the mapping. Figure 5(b) reveals a continuous performance decline with increased modalities, validating our hypotheses. Thereby, we interpret that the augmenting process complexifies input-output mappings and impairs the TL effect.\n\n>**C2. Influence of prompt design**\n\nPlease refer to the response to **Q1** of Reviewer gx7F.\n\n>**W2&Q2. Adoption of distribution protocols on Twitter-15 and Twitter-17**\n\nIn the manuscript, we determine the adopted protocol based on the proportion of negative samples in the dataset (lines 408-411). Given prior knowledge of datasets’ distribution, we adopt the Category Balanced protocol for Twitter-15 (12.1% negative samples) and Twitter-17 (13.6% negative samples), and the Unlimited protocol for the other datasets with higher proportions.\n\nHere is an intuitive explanation. Influenced by the short-cut effect, negative demonstrations in the ICL sequence can stimulate MLLMs to produce negative predictions, which intensifies as the ratio of negative demonstrations increases. By selecting protocols, we ensure an adequate ratio of negative demonstrations to mitigate MLLMs’ sentimental predictive bias. For datasets with few negative samples, the Category Balanced protocol guarantees one-third negative demonstrations, which other protocols fail to achieve. For datasets with more negative samples, reliable similarity measurement naturally retrieves sufficient negative demonstrations for negative test samples. The Unlimited protocol thereby outperforms the Category Balanced protocol by configuring more sentiment-aligned demonstrations for non-negative test samples.\n\n>**W3&Q3. Selection of final policy in practical applications**\n\nOur final policy is integrated by the optimal strategies of three factors. In practical applications, we decide the optimal similarity measurement strategy based on the type of task (WIT for post-level MSA and WITA for aspect-level MSA). The optimal modality presentation strategy is fixed, where we compose demonstrations with image and text. In cases where the proportion of negative samples is known, the optimal sentiment distribution strategies can be easily determined. Otherwise, protocol selection should depend on specific prioritizations. For applications prioritizing recall of negative samples (e.g., mental health monitoring), the Category Balanced protocol should be adopted. For applications prioritizing precision in non-negative sample identification or overall accuracy (e.g., public opinion monitoring), the Unlimited protocol is recommended.\n\n>**W4&Q4. Sentiment distribution strategies for post-level and aspect-level MSA**\n\nThe optimal distribution protocol for a dataset is determined based on its proportion of negative samples, which is independent of the task type.\n\n>**O1. Theoretical analysis of ICL's role in MSA**\n\nWe further analyze ICL's role in MSA through its two effects: Task Recognition (TR) and Task Learning (TL). The response to Q1 shows TL's non-negligible role in MSA. It is also evidenced by the positive correlation between ICL performance and the number of shots (Tables 7-9). Concurrently, TR also exerts a remarkable influence, as validated by ICL's robustness to textual prompts (response to **C2**). Textual prompts and TR share a similar function: both inform the MLLM about the task format. The contrast between the unstable zero-shot and stable ICL performance reveals the superiority of TR effects.\n\nTherefore, ICL's TR and TL roles are equally critical in MSA. This contrasts with prior findings in VQA, where TR dominates over TL. This highlights the unique characteristics of MSA and the necessity of task-specific investigations.\n\n>**O2. Broader validation on additional models**\n\nPlease refer to the response to **C1** of Reviewer gx7F.", "claims_and_evidence": "This paper makes several key claims:\n(1) The zero-shot performance of MLLMs for multimodal sentiment analysis is undesirable, but MLLMs can effectively perform multimodal sentiment analysis when provided with well-configured in-context learning demonstrations.\n(2) The choice of similarity measurement, modality presentation, and sentiment distribution significantly affects in-context learning performance.\n(3) In-context learning introduces biases that can be mitigated by strategic demonstration selection.\nThese claims are generally supported by experiments across six multimodal sentiment analysis datasets. The authors present comparative evaluations against zero-shot baselines, random in-context learning, and prior in-context learning techniques, demonstrating clear improvements.", "methods_and_evaluation_criteria": "The proposed various in-context learning strategies with the evaluation of six multimodal sentiment analysis benchmarks make sense for exploring in-context learning to enhance the sentiment perception ability of MLLMs. This paper evaluates weighted multimodal similarity for similarity measurement. The influence of different combinations of text, image, and generated modalities on in-context learning performance is also verified. Various sentiment distribution configurations are also verified, and the authors discover predictive bias in MLLMs. The accuracies of the experiments on six multimodal sentiment analysis datasets validate the effectiveness.", "theoretical_claims": "The paper does not present formal theoretical results but provides a strong empirical justification for its findings. \nThe discussion of in-context learning biases and mitigation strategies is insightful but could benefit from a deeper theoretical analysis of in-context learning's role in multimodal sentiment analysis.", "experimental_designs_or_analyses": "The study is thorough, covering multiple datasets and models. Ablation studies help isolate the contributions of each factor. Comparison with zero-shot, random in-context learning, and other multimodal sentiment analysis methods is well-executed.\nThe paper focuses on two MLLMs (IDEFICS-9B, Open-Flamingo2-9B) and fixed prompts; broader validation on additional models and prompt engineering would strengthen the results.", "relation_to_broader_scientific_literature": "The related works about the key contribution are sufficiently provided. Exploring in-context learning for multimodal sentiment analysis is a novel topic. The study builds on prior research in in-context learning (Brown et al., 2020). The efficacy of in-context learning relies on retrieval (Zhang et al., 2022), presentation (Li et al., 2024), and distribution (Lyu et al., 2023). The basic similarity calculation is based on (Liu et al., 2022 and Yang et al.2022). The biases in in-context learning are also observed by (Yang et al., 2023c; Li et al., 2024; Baldassini et al., 2024).", "essential_references_not_discussed": "The references that are essential to in-context learning for MLLMs and multimodal sentiment analysis are comprehensive.", "other_strengths_and_weaknesses": "(1) Exploring in-context learning for multimodal sentiment analysis is an underexplored yet impactful direction in the multimodal area. This paper provides valuable practical experience on how to effectively utilize MLLMs for multimodal sentiment analysis with low resource consumption. \n(2) This paper is empirical rigor. The experimental design is meticulously crafted, and the findings are articulated. The study addresses biases in in-context learning, adding depth to the contribution.\n(3) This paper is well-written and well-organized.\n\nThere are some unclear descriptions:\n(1) Some figures do not correspond to the text expression and are difficult to understand. For example, in Figure 3 (a), it is unclear how to calculate various similarities in Figure 3 (a). If these values are just schematic, using variable representation to show only the calculation process is better.\n(2) For the sentiment distribution, different data sets adopt different strategies (Table 4). The reason why only Twitter-15 and Twitter-15 adopt Category Balanced should be provided.\n(3) Different data sets adopt different final policies, and how to determine which policies to adopt in practical applications should be discussed.\n(4) The difference between strategies for post-level and aspect-level tasks in sentiment distribution should be clarified.", "other_comments_or_suggestions": "(1) It is recommended to provide more clear figures. For example, Figure 6 is too informative and should be simplified to show the experimental results more clearly.\n(2) A discussion about the influence of prompt design is suggested.", "questions_for_authors": "(1) Can the authors explain the meaning of the experiments conducted in Figure 5? Why can this experiment interpret the performance degradation brought by modality conversion?\n(2) For the sentiment distribution, different data sets adopt different strategies (Table 4). It is necessary to analyze why only Twitter-15 and Twitter-15 adopt Category Balanced. Are there any considerations or assumptions for this design?\n(3) Different data sets adopt different final policies, and how to determine which policies to adopt in practical applications should be discussed.\n(4) What is the difference between policies for post-level and aspect-level tasks in sentiment distribution?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oekWQ0MePy", "reviewer": "Reviewer_jCmi", "summary": "The paper explores how to enhance the performance of Multimodal Large Language Models (MLLMs) in Multimodal Sentiment Analysis (MSA) by optimizing the configuration of In-Context Learning (ICL) demonstrations. The main findings include:\n\n​1. Enhancing MSA Performance: The authors show that by carefully configuring ICL demonstrations, MLLMs can significantly improve their sentiment analysis capabilities. They identify three key factors: similarity measurement, modality presentation, and sentiment distribution.\n\n​2. Optimizing Similarity Measurement: The study introduces various strategies for measuring the similarity between multimodal data, focusing on image-text pairs. They refine traditional similarity measures to better capture the nuances of MSA.\n\n3. ​Modality Presentation: The authors investigate different combinations of modalities (image and text) and find that careful presentation of multimodal information can enhance sentiment prediction.\n\n​4. Sentiment Distribution: They explore different protocols for distributing sentiment labels in demonstrations to mitigate biases and improve fairness in predictions.\n\n5. ​Experimental Results: The proposed strategies lead to average accuracy improvements of 15.9% over the zero-shot paradigm and 11.2% over a random ICL baseline on six MSA datasets. The methods are shown to be effective and generalizable across different MLLMs and datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review and in-depth comments! Below, we present detailed responses to the weaknesses (**W**) and other concerns (**O**).\n\n>**W4(1). Metric definitions in Table 2**\n\nIn the caption of Table 2, we specify that \"R strategy\" represents random retrieval, and we report its average accuracy across 4,8,16-shot ICL.\n\n>**W4(2)&W1(2). CLIP’s role in similarity measurement**\n\nIn the manuscript, we compute unimodal scores via cosine similarity in CLIP’s embedding space, and multimodal scores are aggregated from unimodal ones. We employ CLIP’s visual encoder and the following projection layer to obtain visual embeddings, and CLIP’s textual encoder and the following projection layer to obtain textual embeddings. It is essentially an intra-modal retrieval. \n\nCLIP does not provide joint embeddings that embed the joint semantics of image and text. Therefore, an alternative cross-modal retrieval approach is computing similarity scores between images from one sample and texts from another. We adopt intra-modal retrieval for two reasons:\n\n1. CLIP aligns image and text embeddings. Both cross-modal and intra-modal retrieval operate within a unified embedding space. Their underlying mechanisms are similar.\n2. In [1,2], the similarity between image-text inputs is computed within each modality and later combined, which has been empirically proven effective.\n\nFollowing the setup in Table 2, we compare two retrieval approaches. T2I denotes image-to-text retrieval, and I2T denotes text-to-image retrieval. From the results, cross-modal retrieval offers no additional benefits. \n\n|IDEFICS|MVSA-S|Twitter-15|\n|-|:-:|:-:|\n|R|49.2|57.4|\n|I|56.5|59.1|\n|T|56.0|58.7|\n|T2I|55.7|58.9|\n|I2T|55.0|57.4|\n\n\n>**W2. Uncontrolled image generation- a flaw of the \"transfer emotional details\" claim**\n\nWhile your point is insightful, we are afraid you may misunderstand our motivation in investigating modality presentation, which is modality conversion can furnish supportive information (Line 68, 196-204). We have not made the \"transfer emotional details\" claim, as suggested in your comment. This misunderstanding may stem from the phrase \"which are conducive to evoking emotions\" in Line 204, where our original intent is to highlight a potential benefit. To clarify, we will rephrase this part in the revised manuscript. Following our motivation, we investigate both substituting and augmenting original modalities with auxiliary ones. Our analysis reveals that the modality conversion introduces extra noise (Lines 298-300), which aligns with your comment. This issue leads to our conclusion: modality conversion underperforms the use of original modalities.\n\nFacilitating emotional alignment in image generation is an interesting direction. However, this field remains in its infancy. Pioneering work [3] has not released pretrained checkpoints, and it is difficult to train a model and validate its effectiveness within the rebuttal period. Instead, we will explore it in future work.\n\n>**W3(1). Hyperbolic phrases**\n\nWe will carefully check potential hyperbolic phrases in the manuscript and revise them to appropriate expressions.\n\n>**W4(3). Ablations on design choices**\n\nIn our investigation, variations are introduced to the pertinent settings only when probing specific factors. Therefore, Figure 6 and Lines 326-411 cover ablations of the sentiment-balanced sampling, and Figure 4(b) and lines 307-323 cover ablations of the WITA weight.\n\n>**O1. Explanation for Figure 5 experiment**\n\nPlease refer to the response to **Q1** of Reviewer pNwb.\n\n>**W5&W3(2). Explorations for pretraining data skew or model inductive biases**\n\nThis manuscript focuses on configuring ICL demonstrations to unleash MLLMs' sentiment perception capabilities. Therefore, we mitigate the sentimental predictive bias with ICL instead of altering MLLMs themselves. We have not explored or tackled deeper limitations, as they lie beyond the primary scope of our research.\n\nHowever, systematic studies of these limitations can indeed substantially contribute to both MLLMs and MSA. As further discussion, we attribute these limitations to pretraining data rather than model architecture, which is validated in [4]. By constructing emotion-related data, it enhances MLLMs' zero-shot performance on visual emotion recognition. This success has the potential to be replicated in solving the sentimental predictive bias.\n\n>**W1(1). Scientific rigor**\n\nWe hope our responses can convince you of our methodology design and experimental rigor, a strength also recognized by Reviewers nCbm and pNwb.\n\n>**References**\n\n[1] Yang et al. Exploring Diverse In-Context Configurations for Image Captioning. NeurIPS, 2023.\\\n[2] Li et al. How to Configure Good In-Context Sequence for VQA. CVPR 2024.\\\n[3] Yang et al. EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models. CVPR 2024.\\\n[4] Xie et al. EmoVIT Revolutionizing Emotion Insights with Visual Instruction Tuning. CVPR 2024.", "claims_and_evidence": "​Claims Supported by Evidence:\n\nThe paper provides empirical evidence that optimizing In-Context Learning (ICL) demonstrations can significantly improve MLLMs' performance in Multimodal Sentiment Analysis (MSA). The improvements in accuracy over the zero-shot and random ICL baselines suggest that the strategies are effective.\n\n​Problematic Claims:\n\n​1. The claim that the method \"fully unleashes the potential\" of MLLMs in MSA may be overstated. While the improvements are significant, it is unclear if the method reaches the absolute maximum potential, especially without direct comparisons to state-of-the-art approaches.\n​Similarity Measurement: The reliance on traditional cosine similarity for multimodal data may be too simplistic, especially given the acknowledged semantic gap between text and images. The use of CLIP for measuring text-image associations could have provided a more nuanced approach.\n2. The method of generating images from text to enhance multimodal presentation raises concerns about the controllability of the generated images' emotional expressions. This could lead to misinterpretation of the original sentiment, particularly with neutral or ambiguous samples.\n​3. The experimental setup lacks clarity, particularly regarding the metrics used and how CLIP is applied. This makes it difficult to assess the effectiveness of the method comprehensively.", "methods_and_evaluation_criteria": "While the paper proposes interesting directions for optimizing ICL in MSA, several aspects of the methodology and evaluation raise questions about their alignment with the problem’s demands:\n\n​Multimodal Similarity Measurement:\nThe paper relies on cosine similarity for cross-modal retrieval (text+image), despite acknowledging the semantic gap between modalities. CLIP, which explicitly models text-image alignment, is mentioned but not leveraged for similarity scoring. This choice risks conflating modality-specific features (e.g., syntax in text vs. objects in images) rather than capturing joint semantics. Why not use CLIP’s joint embeddings for similarity?\n\n​Image Generation for Modality Presentation:\nGenerating images from text (e.g., via diffusion models) to “augment” demonstrations is creative but risks introducing uncontrolled biases. For example, a neutral text paired with a generated image that skews positive/negative (as seen in Figure 3b) could mislead the model. Without validating that generated images preserve the original sentiment, this approach risks confounding results. How do you ensure generated images align with the text’s intended sentiment?\n\n​Evaluation Metrics & Transparency:\nTable 2’s experimental setup lacks clarity on how CLIP is used (e.g., which layers, pooling strategies) and which metrics are computed (e.g., accuracy, F1). Similarly, the “Task Learning” experiment (Figure 5) abruptly switches to animal labels—this is clever but under-explained. How does this abstraction relate to sentiment prediction?\n\n​Negative Sentiment Bias:\nThe paper notes a bias toward positive/neutral predictions but stops short of diagnosing why (e.g., data imbalance in pretraining, model architecture). Without exploring this, claims about “mitigating bias” feel superficial. For example, does the “Category Balanced” protocol simply mask underlying issues in the model’s learned representations?", "theoretical_claims": "The paper focuses primarily on ​empirical strategies for configuring ICL demonstrations in MSA, with no explicit theoretical claims or proofs (e.g., convergence guarantees, generalization bounds). As such, there are no formal proofs to verify. However, the paper’s methodological assumptions and experimental design choices raise implicit theoretical questions:\n\n​Assumption of Modality Independence:\nThe use of cosine similarity (or CLIP embeddings) to combine text-image pairs assumes that modalities can be treated as independent features. This ignores the cross-modal alignment problem, where text and image embeddings may reside in disjoint semantic spaces. A more theoretically grounded approach might require proving that the proposed similarity metrics align with human perception of sentiment.\n\n​Bias Mitigation Without Guarantees:\nThe paper introduces heuristics (e.g., sentiment-balanced demonstration distributions) to counter MLLMs’ negative bias but provides no theoretical analysis of why these protocols work or whether they generalize beyond the tested datasets.", "experimental_designs_or_analyses": "See above", "relation_to_broader_scientific_literature": "The paper’s contributions are overshadowed by methodological gaps and lack of rigor:\n\n​Method Weaknesses:\nUses simplistic cosine similarity for text-image retrieval despite acknowledging the semantic gap.\nGenerates images from text without validating sentiment alignment (see Figure 3b’s misleading neutral-positive mismatch).\n\n​Experimental Opacity:\nFails to specify CLIP’s role in metrics (e.g., layers, pooling) or report statistical significance.\nTable 2’s \"ICL Random\" baseline lacks clarity on metrics (accuracy? F1?) and experimental controls.\n\n​Bias Analysis Superficial: \nAttributes negative sentiment bias to MLLMs without probing pretraining data or architectural flaws.\nBottom Line: The paper prioritizes engineering novelty over scientific rigor. Addressing these gaps is critical for meaningful contribution to MSA/ICL.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1. The focus on optimizing In-Context Learning (ICL) for Multimodal Sentiment Analysis (MSA) addresses a critical challenge in deploying MLLMs in real-world scenarios where labeled data is scarce.\n2. The experiments show consistent gains over zero-shot and random ICL baselines, demonstrating practical utility.\n\nWeaknesses:\n\n1. The paper prioritizes engineering hacks (e.g., cosine similarity, text-to-image generation) over scientific rigor. Ignoring CLIP’s joint embeddings for cross-modal retrieval is a missed opportunity to bridge the semantic gap.\n\n2. Figure 3b’s mismatched neutral text/positive image highlights uncontrolled sentiment leakage in generated visuals—a core flaw in the \"transfer emotional details\" claim.\n\n3. Phrases like \"fully unleashing MLLMs’ potential\" are hyperbolic. The method tweaks ICL prompts but avoids tackling deeper limitations (e.g., model architectures, dataset biases).\n​Experimental Opacity:\n\n4. Table 2’s \"ICL Random\" baseline lacks metric definitions (accuracy? F1?), and CLIP’s role in evaluation is unclear. How is CLIP used beyond vague \"similarity measurement\"?\nNo ablations on key design choices (e.g., sentiment-balanced sampling, WITA weights).\n​\n5. While negative sentiment bias is noted, the analysis stops at surface-level protocols (e.g., \"category balancing\"). No exploration of systemic issues like pretraining data skew or model inductive biases.", "other_comments_or_suggestions": "No", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper explores In-Context Learning (ICL) to enhance multimodal large language models (MLLMs) for sentiment analysis. It systematically investigates three critical ICL demonstration factors—similarity measurement, modality presentation, and sentiment distribution—achieving significant accuracy gains over zero-shot and random ICL baselines across multiple datasets. All reviewers recommend acceptance.
Decision: Accept (poster)