WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledge

Authors: wenbin  wang
liang  ding
li  shen
yong  luo
han  hu
dacheng  tao
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 03:59:11

Abstract: Multimodal Sentiment Analysis (MSA) focuses on leveraging multimodal signals for understanding human sentiment. Most of the existing works rely on superficial information, neglecting the incorporation of contextual world knowledge (e.g., background information derived from but beyond the given image and text pairs), thereby restricting their ability to achieve better multimodal sentiment analysis (MSA). In this paper, we propose a plug-in framework named WisdoM, to leverage the contextual world knowledge induced from the large vision-language models (LVLMs) for enhanced MSA. WisdoM utilizes LVLMs to comprehensively analyze both images and corresponding texts, simultaneously generating pertinent context. Besides, to reduce the noise in the context, we design a training-free contextual fusion mechanism. We evaluate our WisdoM in both the aspect-level and sentence-level MSA tasks on the Twitter2015, Twitter2017, and MSED datasets. Experiments on three MSA benchmarks upon several advanced LVLMs, show that our approach brings consistent and significant improvements (up to +6.3% F1 score).

Main paper: 
 introduction: 
 Sentiment analysis (SA) aims to identify the polarity of human sentiment [8, 33, 46] . With the prevalence of social networks, people express their sentiments using not only plain text, but also other modalities of data (e.g., images). Predicting the sentiments using only text data is challenging since the texts are often short and informal on social networks (such as Twitter), while the associated images can provide valuable complementary information. Therefore, it is beneficial to combine multiple modalities for accurate sentiment classification, and multimodal sentiment analysis (MSA) has attracted much attention in recent years [32, 44] . Recent studies improve MSA by carefully designing various strategies, which can be categorized into four types: 1) disentangled representation learning [16, 48] , 2) attention-based cross-modal interactions [20, 49, 60] , 3) fusion mechanisms [13, 15, 54] , and 4) well-designed auxiliary tasks [27, 28, 48] .Despite their empirical success, the above studies only consider the superficial information 1 between image and text (see the case "Aleppo" in Fig. 1 ), and sometimes it is difficult to predict the true polarity without their background world knowledge ("Aleppo has been severely affected by the ongoing Syrian Civil War..." induced from the large vision-language models). This raises the following question:Could world knowledge boost MSA?Take the test case in Fig. 1 as an example, given a comparative image at different periods alongside a sentence, it is required to answer the question: What's the sentiment polarity of "Aleppo"?We employ the current state-of-the-art (SOTA) MSA model [60] as the backbone. As expected, even the SOTA model gives the wrongThe sentiment is negative.Figure 1 : The simple schematic of our method. The sentiment polarity of Aleppo is negative, which is difficult to predict directly using existing methods while our WisdoM predicts correctly via incorporating context generated by the world knowledge-rich LVLMs.prediction (i.e., "neutral" rather than the groundtruth-"negative") due to the lack of deeper knowledge of "Aleppo" (which is a city in Syria, in conjunction with the image, we might infer that the difference between the before and after of this city is caused by the Syrian war). Therefore, employing world knowledge is essential. This motivates us to propose a plug-and-play framework termed WisdoM to utilize the contextual world knowledge (simply induced from the large vision-language models) to complement the existing text-image pair with only superficial information. In particular, our method consists of three main stages: 1 Prompt Templates Generation, 2 Context Generation, and 3 Contextual Fusion. In stage 1, we ask language models (e.g., Chat-GPT2 ) to generate prompt templates (See the Prompt Template " Give you a sentence and image, you can ... " in Fig. 1 ) which are used to construct instructions for stage 2. Then, we employ the advanced large vision-language model (LVLM, e.g., LLaVA [30] ) in stage 2 to generate the contextual information (See the Context " Aleppo, ... ongoing Syrian Civil War... " in Fig. 1 ) based on the provided image and sentence. Note that, we refer to this contextual informations as context. Since the derived context often contains some noise, we further introduce a novel, training-free Contextual Fusion mechanism in stage 3. This mechanism selectively integrates world knowledge by identifying hard samples and fusing them with context, thereby enhancing the model's effectiveness in MSA, particularly on the hard samples that current models may fail.We validate our WisdoM on several benchmarks, including Twit-ter2015, Twitter2017 [52] and MSED [18] over several models: LLaVA-v1.5 [29] , MMICL [57] , Qwen-VL [3] , AoM [60] , and ALMT [56] . The results across diverse granularities of MSA tasks consistently demonstrate that our approach has substantial improvements (on average +2.2% in terms of F1 score) over the state-of-theart approaches.To summarize, our main contributions are:• We propose a plug-in framework WisdoM , which leverages LVLM to generate explicit contextual world knowledge, to enhance multimodal sentiment analysis. • To achieve wise knowledge fusion, we introduce a novel contextual fusion mechanism to mitigate the impact of noise in the context. • We conduct extensive analyses and provide some insights on when and why our method works. 
related work 2.1 multimodal sentiment analysis: 
 Multimodal Sentiment Analysis (MSA), diverging from conventional text-based approaches [17] , incorporates diverse modalities (e.g., image) to enhance sentiment classification accuracy [38] . Numerous advanced models have been proposed, covering different levels of granularity, such as sentence and aspect: Sentence-Level MSA. Zhao et al. [58] explore image-text correlations in movie reviews. Li et al. [25] propose a ConvTransformer, blending Transformer [39] and CNN technologies for sentiment analysis. Das and Singh [7] propose a multi-stage multimodal method for the Assamese language, leveraging both text and images. Zhang et al. [56] present an advanced model, ALMT, enhancing multimodal analysis focusing on language-guided features to handle irrelevant or conflicting data across different modalities.Aspect-Level MSA. Yu and Jiang [52] introduce TomBERT, an aspect-oriented multimodal BERT model using annotated tweet datasets. Khan and Fu [21] propose a two-stream model combining an object-aware transformer with non-autoregressive generation [9, 47] for image translation. Zhou et al. [60] present AoM, an aspect-oriented network aimed at reducing distractions in complex image-text interactions.Although existing approaches relying on sophisticated techniques have achieved remarkable performance in MSA, their limitation lies in relying on superficial information, without incorporating contextual world knowledge.Large Vision-Language Models (LVLMs) are becoming a fundamental tool for solving general tasks [1, 4, 6, 11, 24, 30, 40, 51, 57, 61] . Liu et al. [29] introduce LLaVA, integrating the CLIP ViT-L/14 visual encoder [10] with Vicuna large language model via a projection matrix and two-stage instruction tuning. Zhao et al. [57] propose MMICL to improve LVLMs handling of multi-modal prompts from model and data perspectives. Bai et al. [3] introduce Qwen-VL which is designed to perceive and understand both texts and images.Here, we leverage LVLMs to enhance multimodal sentiment analysis by generating relevant world knowledge. Additionally, we introduce a new training-free module named Contextual Fusion, designed to minimize noise in the context.Retrieval-Augmented Generation (RAG) improves Large Language Models (LLMs) by adding retrieved text, enhancing performance in knowledge-based tasks [12, 14] . Traditional RAG [23] , or Naive RAG, helps in generation but struggles with inconsistent retrieval quality, inaccurate responses, and integrating retrieved context. Advanced methods like DSP [22] enhance context interaction between LLMs and retrieval models, and PKG [31] enables LLMs to access pertinent information for complex tasks without training.The working mechanism of our WisdoM is similar to RAG, but different in the following aspects: ① WisdoM utilizes LVLM to generate world knowledge to provide coherent and accurate context rather than retrieval, ② WisdoM incorporates a contextual fusion mechanism to diminish noise within the context. For additional experimental analysis and discussion, please refer to § 4.4.3. 
methodology 3.1 preliminary: 
 We first describe the notation of the MSA, then review two typical frameworks for modelling the MSA tasks, where we experiment with our schema upon them: task-specific framework [56, 60] and general-purpose framework [3, 29, 57] . Notation. Let M be a set of multimodal samples. Each sample 𝑚 𝑖 ∈ M consists of a sentence 𝑠 𝑖 and image 𝑣 𝑖 . For aspect-level MSA tasks, there are several aspects 𝑎 𝑖 which is a subsequence of 𝑠 𝑖 , i.e., 𝑎 𝑖 ∈ 𝑠 𝑖 . We denote 𝑓 (•) as the sentiment classifier. The output of 𝑓 (•) is the sentiment polarity 𝑦 𝑖 ∈ {negative, neutral, positive}, with corresponding predicted probability denoted as 𝑃 𝑖 = {𝑝 𝑛𝑒𝑔 𝑖 , 𝑝 𝑛𝑒𝑢 𝑖 , 𝑝 𝑝𝑜𝑠 𝑖 }. Task-Specific Framework. For aspect-level MSA tasks, the goal is to predict the sentiment polarity 𝑦 𝑖 and probability 𝑃 𝑖 for the specific aspect 𝑎 𝑖 conditioned on the (𝑣 𝑖 , 𝑠 𝑖 ), i.e., (𝑦 𝑖 , 𝑃 𝑖 ) = 𝑓 (𝑣 𝑖 , 𝑠 𝑖 , 𝑎 𝑖 ). For sentence-level MSA tasks, the 𝑦 𝑖 and 𝑃 𝑖 are predicted by sentence 𝑠 𝑖 alongside the image 𝑣 𝑖 , i.e., (𝑦 𝑖 , 𝑃 𝑖 ) = 𝑓 (𝑣 𝑖 , 𝑠 𝑖 ). General-Purpose Framework. To verify that our WisdoM works well on arbitrary architectures, we also apply WisdoM to generalpurpose LVLMs. We follow Wang et al. [45] to construct the task instructions 𝐼 𝑎𝑠𝑝𝑒𝑐𝑡 and 𝐼 𝑠𝑒𝑛𝑡𝑒𝑛𝑐𝑒 for each task to elicit its ability to the corresponding task. The task instructions are presented as singlechoice questions with well-formatted options (shown in Table 1 ). For each choice of a question, we compute the likelihood 𝑃 𝑖 that LVLM generates the content of this choice based on the given question. The choice with the highest probability is then selected as the prediction 𝑦 𝑖 . The LVLMs can be seen as a sentiment classifier 𝑓 (•). Thus, the aspect-level task and sentence-level task can be formulated as (𝑦 𝑖 , 𝑃 𝑖 ) = 𝑓 (𝐼 𝑎𝑠𝑝𝑒𝑐𝑡 , 𝑣 𝑖 , 𝑠 𝑖 , 𝑎 𝑖 ) and (𝑦 𝑖 , 𝑃 𝑖 ) = 𝑓 (𝐼 𝑠𝑒𝑛𝑡𝑒𝑛𝑐𝑒 , 𝑣 𝑖 , 𝑠 𝑖 ) respectively.In this part, we first provide a comprehensive overview of our method, and then introduce it in detail.Table 1 : Template of task instruction. NOTE: "[sentence]" and "[aspect]" are placeholders meant to be replaced with specific sentence and aspect from the dataset.Aspect-Level Taks Instruction 𝐼 𝑎𝑠𝑝𝑒𝑐𝑡 Sentence: [sentence] Use the image as a visual aids to help you answer the question. What is the sentiment polarity of the aspect [aspect] in this sentence? A). positive B). neutral C). negative Answer with the option's letter from the given choices directly.Sentence-Level Taks Instruction 𝐼 𝑠𝑒𝑛𝑡𝑒𝑛𝑐𝑒 Sentence: [sentence] Use the image as a visual aids to help you answer the question. Given the sentence and image, what is the sentiment conveyed? A). positive B). neutral C). negative Answer with the option's letter from the given choices directly.3.2.1 Overview. Fig. 2 illustrates the overview of our method following three stages. In the 1 Prompt Templates Generation, we use large language models, particularly ChatGPT, to provide prompt templates. These prompt templates are fed into the LVLM with sentence 𝑠 𝑖 and image 𝑣 𝑖 to generate context, also called the 2 Context Generation. During 3 Contextual Fusion, we first compute the confidence, determining if the sample is uncertain (referred to as a hard sample). For hard samples, we fuse the predicted probability 𝑃 𝑖 with P𝑖 which is obtained by incorporating context. Otherwise, we use 𝑃 𝑖 as the final prediction.Prompt Templates Generation. The main purpose of this stage is to design the prompt templates used to generate the context, so that the LVLM can better understand our intention and thus provide a more comprehensive contextual world knowledge. Inspired by [19, 59] , we ask ChatGPT to provide the appropriate prompt templates. The prompt templates provided by ChatGPT consider world knowledge from different perspectives, including historical, social, cultural, etc. We insert a "Sentence: [x]" at the end of the prompt template to place the input sentence 𝑠 𝑖 . The example of prompt templates are shown in Supp. A.4.In the context generation stage, prompt templates in 1 are used to generate context that explicitly incorporates world knowledge based on given the image 𝑣 𝑖 and sentence 𝑠 𝑖 by LVLMs [30, 51] . Specifically, we construct instruction by replacing the "[x]" in the prompt template with the sentence 𝑠 𝑖 . In addition, different LVLMs require a special token to indicate where the image 𝑣 𝑖 is inserted. Taking LLaVA [30] as an example, we insert a special token "<image>" at the beginning of the instruction. i.e., ( ŷ𝑖 , P𝑖 ) = 𝑓 (𝑣 𝑖 , 𝑠 𝑖 , 𝑐𝑜𝑛𝑡𝑒𝑥𝑡). However, the context may contain irrelevant information that could disturb performance. Therefore, we first determine hard samples and then fuse the 𝑃 𝑖 and P𝑖 in the hard sample.Determining the Hard Samples. Inspired by Zhang et al. [55] , we find that the ambiguous hard sample is commonly found around boundary areas of the sentiment polarity (e.g., the boundary areas of negative and neutral). Therefore, given a sample 𝑚 𝑖 , we only consider the difference 𝛿 𝑖 between the highest and the second highest probabilities to determine whether it is a hard sample:EQUATIONwhere 𝑃 𝑚𝑎𝑥 𝑖 and 𝑃 𝑠𝑒𝑐 𝑖 represent the highest and second highest probabilities respectively. Then, we denote uncertain threshold 𝛼 to select samples that do not exceed 𝛼 as hard, i.e., V ℎ𝑎𝑟𝑑 = {𝑚 𝑖 |𝛿 𝑖 ≤ 𝛼 }. Fusion with Context. Inspired by [26, 35] , we take the convex combinations of 𝑃 𝑖 and P𝑖 to obtain the final prediction P𝑖 for hard sample 𝑚 𝑖 ∈ V ℎ𝑎𝑟𝑑 :EQUATIONwhere 𝛽 is an interpolation coefficient. Intuitively, ( P𝑖 -𝑃 𝑖 ) represents the information incorporated by context. 𝛽 is used to control the proportion of information introduced in context. When 𝛽 → 0, the effect brought by context is completely ignored and vice versa. Note that, we use (𝑦 𝑖 , 𝑃 𝑖 ) as the final prediction when 𝑚 𝑖 ∉ V ℎ𝑎𝑟𝑑 . We study the impact of 𝛼 and 𝛽 in § 4.2. 
experiments: 
 In this section, we apply WisdoM to aspect-level and sentence-level MSA tasks to verify its effectiveness and conduct extensive analysis to better understand the proposed method.4.1.1 Datasets. For aspect-level tasks, our two benchmark datasets are Twitter2015 and Twitter2017 [52] . Twitter2015 and Twitter2017 are comprised of multimodal tweets, where each tweet incorporates textual content, an accompanying image, aspects contained within the tweet, and the sentiment associated with each aspect. Each aspect is assigned a label from the predefined set {negative, neutral, positive}. For sentence-level tasks, we evaluate our WisdoM on MSED [18] dataset, containing 9,190 text-image pairs. We show the statistics of Twitter2015, Twitter2017 and MSED in Supp. A.1.To demonstrate WisdoM generalizes across architectures and sizes, we conduct experiments on several models, including MMICL (14B) [57] , LLaVA-v1.5 (13B) [29] , Qwen-VL (9.6B) [3] , ALMT (112.5M) [56] , AoM (105M) [60] . The detailed model cards can be found in Supp. A.2.In stage 1, we utilize ChatGPT to generate the prompt templates corresponding to different types of world knowledge, executing this process only once. For Twitter2015 and Twitter2017 datasets, we employ the historical prompt template (see Supp. A.4) to generate context. For MSED dataset, the scientific prompt template is utilized for context generation.For aspect-level tasks, we use Accuracy (Acc.) and macro-F1 (Mac-F1) following the previous studies [20, 21, 28, 60] . For the sentence-level task, we adopt precision, recall, and macro-F1 (Mac-F1) as evaluation metrics. Following previous work [20] , we also perform the paired t-test using scipy 3 to test the significance of the difference between our WisdoM and baseline, with a default significant level of 0.05. To avoid large variances in the final results due to differences in context, we set the temperature to 0.2 for LVLM in stage 2. Additionally, we conduct each experiment five times and report the average of 5 runs for all our experimental results.In stage 3, Contextual fusion has two major hyper-parameters interpolation coefficient 𝛽 and uncertain threshold 𝛼. To investigate whether our approach is robust to different hyper-parameters, we employ grid search to study the effect of hyper-parameters on Twit-ter2015 dev set. As shown in Fig. 3 , we find that: 1) although the performance varies with hyper-parameters, the extreme values of the results are not significant. 2) 𝛼 and 𝛽 values within the range of [0.3, 0.6] demonstrate strong performance among various models. 3) Excessive values of 𝛽 result in performance degradation. We conjecture that this decline may be caused by introducing excessive noise within the context. To reduce the redundant adjustment with hyper-parameters, we set 𝛼 = 0.3 and select the optimal value of 𝛽 on the dev set for evaluation.3 https://www.scipy.org/ of Aspect-Level MSA Task. We compare against advanced aspect-level MSA methods on Twitter2015 and Twit-ter2017, and report the results on Table 2 . We show that our WisdoM achieves consistent and significant improvement on four models across two datasets. The WisdoM brings max 2.9% and 2.3% F1gains on Twitter2015 and Twitter2017 respectively, showing that our method has a clear advantage. Besides, we perform the paired t-test to assess the significance of the improvements made by our WisdoM over the baseline. The results demonstrate that our improvement is statistically significant with 𝑝 -𝑣𝑎𝑙𝑢𝑒 < 0.05.of Sentence-Level MSA Task. As shown in Table 3 , notably, our WisdoM upon Qwen-VL achieves the new SOTA F1 score: 91.8%, outperforming Qwen-VL (89.4%), LLaVA-v1.5 (88.8%), MMICL (86.2%) and ALMT (83.7%), consistently.The most significant improvement is achieved on ALMT (112.5M), where we bring an encouragingly 6.3% F1 gain, suggesting that contextual world knowledge is particularly crucial for small models in MSA. Besides, our WisdoM is statistically significant with 𝑝 -𝑣𝑎𝑙𝑢𝑒 < 0.05 when comparing with baseline.To better understand the role of each module in our WisdoM, Table 4 presents the ablation results of the gradual addition of different components. Compared with the baselines (MMICL and LLaVA-v1.5), only adding context results in a slight performance degradation (-0.3% average F1 score), while with the help of our proposed Context Fusion mechanism, we achieve a consistent and significant improvement (+1.8% average F1 score). Through (error) case studies in Supp. C, we find that 4.4.2 How does hand-crafted prompt template differ from LLM-generated in stage 1? To compare human-crafted prompt template and LLM-generated prompt template, we employ three human annotators and ChatGPT to provide prompt templates respectively. We provide the same meta-prompt ("Please provide prompt template that large vision-language model can generate the historical knowledge based on the image and sentence.") for human annotators and ChatGPT to generate prompt templates. The details of prompt templates can be seen in Supp. A.6. As shown in Table 5 , the results show that LLM-generated consistently outperformed hand-crafted across various metrics. Furthermore, the standard deviation for hand-crafted templates is 1.1, compared to only 0.3 for LLM-generated templates. We find that templates provided by human annotators directly seek historical insights from image and text, while templates generated by ChatGPT offer deeper analysis, reflecting important events, significance and background information.To further analyse the effect of context, we compare our context generated in stage 2 with the document retrieved by Naive RAG [23] and knowledge generated by PKG [31] , collectively termed as "context" for simplicity. The assessment focuses on the context's pertinence to a given image 𝑣 and sentence 𝑠, alongside its applicability in MSA tasks. We employ the LLM-based metric, i.e., LLM-as-a-Judge [5] to quantify the quality of context. Specifically, we craft a prompt for GPT-4V [36] to compare our context with that provided by Naive RAG and PKG. The detailed experimental settings can be found in Supp. A.7. As shown in Fig. 4 , our context significantly beats the Naive RAG and PKG counterparts, demonstrating its superiority. We provide examples of context provided by different methods in Supp. C.1.Besides analyzing the contexts' pertinence of different methods, we report their downstream performance on MSA tasks in Table 6 .Clearly, the MSA performance with our context is the best. The results above illustrate that our method can provide more precise context, thus bringing better MSA performance. 7 , we explore different fusion strategies, including 𝑚𝑒𝑎𝑛(𝑃 𝑖 , P𝑖 ) ("Average"), 𝑚𝑎𝑥 (𝑃 𝑖 , P𝑖 ) ("Max"), Jensen-Shannon divergence [34] ("JS"), conditional crossmutual information 𝑓 𝑐𝑥𝑚𝑖 [43] ("CXMI"), and our Context Fusion ("CF"). For JS, we calculate the JS divergence of 𝑃 𝑖 with the uniform distribution to serve as the fusion weight, i.e., 𝛽. As for CXMI, if 𝑓 𝑐𝑥𝑚𝑖 > 1.14 , we adopt (𝑦 𝑖 , 𝑃 𝑖 ) as our ultimate prediction, otherwise, we use ( ŷ𝑖 , P𝑖 ) as the final prediction. The results show that our Contextual Fusion module performs the best among all competitive alternatives, confirming its effectiveness.Our plug-in method is data-and model-agnostic, and therefore, it is expected to be highly scalable. Here we scale our WisdoM up to different model sizes and data volumes. To explore the relationship between context and LVLMs capability, we conduct experiments on AoM using contexts derived from mPLUG-Owl2 [51] (8.2B) and LLaVA-v1.5 (13B). As depicted in Table 8 , the results show that the stronger the capability of LVLMs, the more accurate and helpful the generated context is for MSA. To better understand when and why our method works, we conduct extensive analysis to provide the following insights:World Knowledge Enhances MSA, while Domain-related Knowledge is more Helpful. For better understanding the effects of different types of world knowledge, we conduct experiment with AoM on the Politician Twitter dataset [50] , covering 8 perspectives of world knowledge (the prompt templates can be found in Supp. A. Context is Dominant for Prediction. To draw a clearer picture of the information flow for MSA, we calculate 𝑆 𝑐→𝑝 and 𝑆 𝑖→𝑝 to represent the mean significance of information flow [37, 41] from context (c) and original input (i) to the prediction labels (p) respectively (detailed in Supp. A.8). Fig. 7 reveals that the significance of the information flow from context to the prediction label is remarkably higher than its input counterpart, suggesting that context outweighs image and sentence when making the final prediction.WisdoM Effectively Reduces the Uncertainty of Hard Samples. To further explore how our WisdoM affects the hard samples, we visualize the error rate within high entropy in Fig. 8 . After integrating WisdoM, the error rate is significantly decreased compared with the baseline (AoM and LLaVA-v1.5), demonstrating that our WisdoM effectively reduces the uncertainty of hard samples and improves performance. We hypothesize that part of such eliminated information may be multimodal hallucination [42] , while another part may be due to the model's inherent knowledge gaps [2] . Both aspects will be investigated in our future work. 
conclusion: 
 In this paper, we propose a simple yet effective plug-in framework WisdoM to enhance the ability of multimodal sentiment analysis. Our WisdoM contains three stages: Prompt Templates Generation, Context Generation, and Contextual Fusion. We empirically demonstrated the effectiveness and universality of the WisdoM on several widely-used benchmarks. From the results, we mainly conclude that:(1) World knowledge can improve MSA, and domain-related knowledge can be very beneficial; (2) A more precise context is more helpful for MSA; (3) World knowledge is particularly desirable for hard samples. In the future, we will develop adaptive mechanisms to dynamically integrate world knowledge into MSA models. 
 
 Appendix: Our work has potential limitations. We use LVLMs to generate context, but this can lead to hallucinations and incorrect MSA model results. Additionally, our WisdoM relies on LVLMs' internal world knowledge, which requires periodic updates over time. In future work, we aim to investigate adaptive mechanisms for dynamic integration of world knowledge into MSA models.https://chat.openai.comIn preliminary study, we grid-searched values ranging from 0.5 to 2.0, and 1.1 performs best on the dev set, thus leaving as our default setting. 


Decision: The manuscript introduces WisdoM, an innovative framework that effectively integrates Large Vision-Language Models (LVLMs) to advance Multimodal Sentiment Analysis (MSA) by embedding explicit contextual world knowledge and employing a training-free Contextual Fusion mechanism. This approach enhances the depth of sentiment analysis and demonstrates robust empirical validation and scalability across various benchmarks, underscoring its potential for real-world applications. However, to strengthen the paper further, it is crucial to address concerns such as the clarity on model usage in stage 3, the need for a discussion on LVLMs and prompt-based tasks, and an evaluation of efficiency vis-à-vis LVLMs' slower inference speed. Additionally, clarifying the motivation behind the study and distinguishing between improvements attributed to large models versus novel components would bolster the manuscript's impact. Lastly, considering the potential for LLM output hallucinations and the necessity for periodic updates of contextual world knowledge is critical for ensuring the framework's robustness. These aspects should be carefully addressed in the camera-ready version to enhance the paper's contribution and clarity.

Reviews: review: paper_summary: paper_summary: The paper presents WisdoM, a novel framework that integrates contextual world knowledge into Multimodal Sentiment Analysis (MSA) using Large Vision-Language Models (LVLMs). It features a three-stage process comprising prompt template generation, context generation, and a training-free contextual fusion mechanism to selectively enhance models with robust world knowledge and reduce noise. Experiments on Twitter2015, Twitter2017, and MSED datasets demonstrate significant improvements in sentiment analysis performance, with the potential to dynamically incorporate world knowledge into MSA models.
strength_weakness: strength_weakness: 1. **Framework Innovation**: The introduction of WisdoM, a novel plug-in framework that harnesses the generative capabilities of Large Vision-Language Models (LVLMs) to produce explicit contextual world knowledge, significantly enhancing the depth of Multimodal Sentiment Analysis (MSA).

2. **Knowledge Fusion Mechanism**: The development of a training-free Contextual Fusion mechanism that intelligently integrates world knowledge, mitigating the impact of noise in the context and improving the model's effectiveness, particularly for challenging samples that current models may find difficult to analyze.

3. **Empirical Validation and Scalability**: Extensive experimentation and analysis across multiple benchmarks, demonstrating not only the consistent and significant improvements in MSA performance but also the scalability of the WisdoM framework across various model sizes and data volumes, showcasing its potential for dynamic incorporation of world knowledge into sentiment analysis models.
limitations: limitations: 1.The most significant issue lies in the potential for hallucinations in the output of LLMs. How can you determine whether the Context produced by LLMs is always beneficial for multimodal sentiment analysis tasks?

2.Over time, the contextual world knowledge may require periodic updates.

3.The paper should include the limitations section.
paper_summary: paper_summary: The manuscript introduces an innovative framework, WisdoM, which integrates contextual world knowledge derived from large vision-language models (LVLMs) into multimodal sentiment analysis (MSA). This approach is commendable as it attempts to address the limitation of relying solely on superficial data by incorporating deeper, contextual insights, potentially resulting in more accurate sentiment analysis.
strength_weakness: strength_weakness: 1. The methodology employed in the study, involving stages like Prompt Templates Generation, Context Generation, and Contextual Fusion, is well-structured and detailed. The use of a plug-and-play framework that can be integrated with existing MSA systems is particularly noteworthy and could have significant implications for practical applications.
2. The evaluation of the WisdoM framework across multiple datasets (Twitter2015, Twitter2017, and MSED) and its comparison with several state-of-the-art models demonstrate a rigorous testing of the proposed solution. The reported improvements in F1 scores add credibility to the claims of enhanced performance.
limitations: limitations: 1. The motivation behind the study, as pointed out, appears somewhat unclear or inadequately justified. While the authors argue that existing methods predominantly rely on superficial information, there is a lot of work in sentiment analysis that already incorporates deeper understanding through various sophisticated models. A more thorough literature review and a clearer articulation of how WisdoM distinctly improves over these existing deep analysis methods would strengthen the manuscript.
2. It is crucial for the authors to dissect the sources of the observed performance improvements. The current manuscript does not clearly delineate whether the improvements are primarily due to the use of large models (like LVLMs) or from the novel components introduced in WisdoM. 
3. As mentioned above, although the paper mentions the use of advanced LVLMs, it does not provide a detailed comparison with other deep learning techniques that are specifically designed for deep semantic analysis in multimodal contexts. Including such comparisons could highlight the unique benefits or shortcomings of the proposed framework relative to existing deep learning solutions.
paper_summary: paper_summary: This paper proposes a plug-in framework WisdoM, which leverages LVLM to generate explicit contextual world knowledge, to
enhance multimodal sentiment analysis. To achieve wise knowledge fusion, this paper introduces a contextual fusion mechanism to mitigate the impact of noise in the context. They conduct extensive analyses and provide some insights on when and why their method works
strength_weakness: strength_weakness: The paper's writing is characterized by a clear and coherent style that effectively conveys complex ideas to readers. The proposed methodology is intricate yet well-conceived and logically sound. Moreover, the experimental section is comprehensive, allowing for a detailed evaluation of each model component's functionality and demonstrating superior performance compared to previous baselines.
limitations: limitations: Because of the slow inference speed of LVLMs, the efficiency of the algorithm is a concern. It is recommended to add a time vs. accuracy comparison chart to highlight the efficiency of this method.
paper_summary: paper_summary: This paper focuses on multimodal sentiment, and proposes a large vision-language model (LVLM) WisdoM for multimodal aspect-level and sentence-level sentiment analysis. Wisdom leverages the generation capability of ChatGPT to generate prompts and the inference capability of LVLM to retrieval related world knowledge, and introduces a contextual fusion mechanism for achieving final better sentiment analysis results. This paper proves the superiority of WisdoM comparing with related work by experiments.
strength_weakness: strength_weakness: 1. The WisdoM model proposed by this paper leverages the generation capability of ChatGPT and the inference capability of different LVLMs. In terms of the application of large (vision) language models, this is a deliberate exploration.
2. In this paper, sufficient experimental design, results presentation and analysis are given.
3. The structure and order of this paper are clear, and the content is readable.
limitations: limitations: 1. The issue of sentiment analysis (both sentence-level and aspect-level) itself and the idea of introducing external knowledge are not entirely new, and of course I do not deny the significance of the work in this paper.
2. In stage 3, it has not been given what models for MSA, although this does not affect the understanding of the work in this paper. In addition, how do plug and play reflect?
3. Would it have been better to use a large (vision) language model with prompt to complete the entire sentiment analysis task? If this issue is discussed in this paper, it will be more perfect.