Multimodal Multi-turn Conversation Stance Detection: A Challenge Dataset and Effective Model

Authors: fuqiang  niu
zebang  cheng
xianghua  fu
xiaojiang  peng
hu  huang
bowen  zhang
genan  dai
yin  chen
  multimodal multi
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 04:55:59

Abstract: Stance detection, which aims to identify public opinion towards specific targets using social media data, is an important yet challenging task. With the proliferation of diverse multimodal social media content including text, and images multimodal stance detection (MSD) has become a crucial research area. However, existing MSD studies have focused on modeling stance within individual text-image pairs, overlooking the multi-party conversational contexts that naturally occur on social media. This limitation stems from a lack of datasets that authentically capture such conversational scenarios, hindering progress in conversational MSD. To address this, we introduce a new multimodal multi-turn conversational stance detection dataset (called MmMtCSD). To derive stances from this challenging dataset, we propose a novel multimodal large language model stance detection framework (MLLM-SD), that learns joint stance representations from textual and visual modalities. Experiments on MmMtCSD show state-of-the-art performance of our proposed MLLM-SD approach for multimodal stance detection. We believe that MmMtCSD will contribute to advancing real-world applications of stance detection research.

Main paper: 
 introduction: 
 Social media enables users to frequently articulate their perspectives on controversial subjects regarding specific entities or topics. Aggregating and analyzing these expressed viewpoints reveals prevailing opinions on divisive topics spanning issues like abortion to epidemic prevention [16] . This wealth of opinionated data holds significant potential for applications in web mining and content analysis. The derived insights can inform various decision-making processes, such as advertising recommendations and presidential elections [26, 34] . Thus, automatically detecting stances in social media has emerged as a key approach to understanding user perspectives on diverse issues [21] .Stance detection aims to determine the expressed opinions or attitudes (favor, against, or none) in content toward specific targets [3, 35] . Conventional machine learning [14, 17] and deep learning [36, 41] approaches have shown significant progress in in processing and analyzing pure text. However, increasingly more social media platforms (e.g. Reddit) enable users to post multimodal content including text (posts and comments), images, etc. This facilitates expressing stances and opinions via multiple modalities. Thus, detecting stances from pure textual content may fail to accurately identify users' real perspectives toward targets. Consequently, multimodal stance detection (MSD) has garnered increased attention in recent research.From the looks of the fire damage, especially at the front of the vehicle, this couldn't have been spontaneous. Seems like the arson theory might be right.Tesla after charging I think it was arson. Given today's battery tech, it's highly unlikely for it to just catch fire while charging.Batteries are roughly centered under the car, majority of the damage is to the front..... Still, spontaneous combustion can't be ruled out completely. Solid analysis. It's probably not a technical malfunction.It's highly likely that the owner had flammable materials in the car, which caused the fire. Figure 1 : An example of multimodal multi-turn conversational stance detection, with symbols denoting "favor" (check), "against" (multiplication), and "none" (horizontal line) stances.To date, two MSD datasets have been developed and served as benchmarks for MSD tasks, namely MMVax-Stance (MMVax) [40] and MMSD [30] . However, a persistent challenge in real-world social media analysis is that users commonly articulate perspectives through conversational exchanges. Conventional context-free stance detection methods have difficulties accurately predicting stances in such conversational settings. For instance, Figure 1 illustrates a social media discussion. Within this conversational thread, it is difficult to detect the stances of 𝑈 4 towards Tesla without the conversational and image context.To foster advancements in MSD, we introduce a new multimodal multi-turn conversation stance detection dataset, denoted as MmMtCSD 1 , in which each example consists of a target, a text, and an image. MmMtCSD contain a total of 21,340 annotated data. Specifically, following [32] , we annotated tasks for two targets, "Tesla" and "Bitcoin". Additionally, following [1] , we construct tasks where sentences/posts serve as targets, denoted as "Post-T". In the Post-T setting, the targets are diverse, rendering the task more challenging. A salient characteristic of this dataset is statistics annotated by multiple experts showing 66% of conversations are highly related to the image content. This highlights the close interplay between textual and multimedia data. Additionally, the dataset presents two key challenges: first, stance-relevant content mentioned in text is inferable from the multimodal context; second, stance determination heavily relies on contextual cues.To deal with MmMtCSD, we proposed a novel multimodal large language model stance detection framework (MLLM-SD), which consists of a textual encoder, a visual encoder and a multimodal 1 https://github.com/nfq729/MmMtCSD fusion module. First, the textual encoder encodes the input conversational history information. Besides, to enhance the association between images and text, we also encode captions of the input images. Second, the visual encoder employs a Vision Transformer (ViT) [15] model to obtain representations of the input images. Third, in the multimodal fusion module, we fine-tuned the LLaMA model using the low-rank adaptation (LoRA) [18] method to integrate information across modalities. The process culminates in matching the output of the large language model (LLM) with the appropriate labels, ensuring a coherent and comprehensive approach to multimodal stance detection.The main contributions of this paper can be summarized as follows:• We introduce a challenging MmMtCSD dataset tailored for multimodal stance detection. To the best of our knowledge, this is the first multimodal multi-turn conversational dataset, and the release of MmMtCSD would push forward the research of MSD. • We propose the MLLM-SD framework, which effectively integrates information across multiple modalities. This framework leverages the comprehension capabilities of LLMs to facilitate a detailed understanding of conversational content coupled with image information. • We conduct a series of experiments on the MmMtCSD, which substantiated the efficacy of our proposed framework. Additionally, we incorporated various modules into the framework to validate its adaptability. 
related work 2.1 stance detection datasets: 
 Sentence-level text-only stance detection datasets. Over the years, many datasets have been proposed for sentence-level textonly stance detection datasets, serving as benchmarks in this field. The features of these datasets are shown in Table 1 . The SemEval-2016 Task 6 (SEM16) dataset [32] pioneered stance detection derived from Twitter and gained widespread adoption. The TSE2020 dataset [20] focused on the 2020 elections, enabling electoral stance analysis. The P-Stance dataset [26] features longer tweets in the political domain, while the expansive WT-WT corpus [9] offers a large labeled resource. Tailored datasets also emerged, including the COVID-19 Stance Detection dataset [16] for pandemic discourse. Most wide-ranging is the VAST dataset [1] for zero-shot stance detection across over a thousand topics. Together, these benchmarks spurred significant research in textual stance detection.Conversation-based text-only stance detection datasets. In real-world scenarios, users typically express perspectives in a conversational manner. Consequently, conversational stance detection, which identifies stances within conversation threads, has garnered increasing research attention recently. The SRQ dataset [38] pioneered stance detection for comment data, but was constrained to single-turn replies and shallow conversation depth, rendering it inapplicable for real commenting scenarios. The CANT-CSD dataset [25] aimed to address stance detection in multi-turn dialogues, providing a deeper commenting corpus. However, CANT-CSD is in Cantonese, with still limited conversation depth. The CTSDT [27] dataset relies mainly on automated annotations rather + 𝑛 English ✔ ✔ ✔ ✘ ✔ ✔ ✔ ✔ ✔ Conversation ✘ ✘ ✔ ✔ ✔ ✔ ✘ ✘ ✔ Multi-turn ✘ ✘ ✘ ✔ ✔ ✔ ✘ ✘ ✔than manual labeling. The MT-CSD dataset [34] In recent years, various approaches based on traditional machine learning and deep learning have been proposed to address stance detection for specific targets [2] . The task settings can generally be categorized into in-target [24] , cross-target [11, 29, 39] , and zeroshot settings [1, 28, 42] . Current methods can typically be classified into fine-tuning-based approaches and LLM-based approaches. Finetuning-based methods involve adding a fully connected layer to the [CLS] token of a pre-trained model (such as BERT) and fine-tuning the model for the stance detection task [5] . Recently, LLMs have demonstrated remarkable capabilities across diverse applications, owing to their inherent semantic capabilities [6, 7] . The semantic understanding of LLMs presents exciting opportunities for stance detection [23] . Most LLMs can easily perform stance prediction via zero-shot prompting by users, significantly enhancing usability [12] . 
dataset construction: 
 In this section, we provide a detailed overview of the creation process and unique attributes of our MmMtCSD dataset comprising 21,340 texts and images. To acquire authentic conversational data from social media, we leveraged Reddit2 , one of the largest and most extensive online forums, ensuring the richness and authenticity of the collected MmMtCSD data. We accessed the data through Reddit's official API 3 . To obtain topics with sufficient discussion and high relevance, during the data collection process, we gathered Reddit posts and associated popularity metrics, such as upvotes and comment counts.A manual review was conducted to assess the relevance of the posts (post texts and images) to the given targets, ensuring that the collected posts were highly pertinent and featured sufficiently in-depth comments to facilitate dataset annotation. Subsequently, we collected comments for each selected post. The resulting dataset encompassed relevant posts, associated discussions, and comments, providing a comprehensive overview of conversations centered around the specified targets. The selected targets for this dataset included "Tesla" and "Bitcoin", and we additionally constructed posts as targets.To ensure the high quality of this MmMtCSD dataset, we implemented several rigorous preprocessing steps:• High Relevance to Target: To ensure strong correlation between post content and the specified target, a rigorous evaluation process involving two reviewers was implemented. Only posts deemed highly relevant by both reviewers were included in the dataset. • Minimum 100 Comments per Post: A minimum threshold of 100 comments per post was established to ensure substantial attention and discourse. Posts with fewer comments were excluded to maintain conversational depth and complexity, essential for capturing nuanced stances in multi-turn exchanges. • Appropriate Text Length: Constraints were imposed on post length to maintain data quality. Posts were required to be between 15 and 150 words. Posts shorter than 15 words were considered too simplistic, while those longer than 150 words often contained redundant expressions. • Excluding Non-English Posts: To construct an all-English dataset, non-English posts were systematically removed to maintain language consistency. Multilingual stance detection remains a potential area for future exploration. Furthermore, to ensure the inclusion of multimodal content, we filtered the posts to remove those lacking multimodal elements. After completing these preprocessing steps, we obtained 107,249, 112,081, and 140,129 data instances for the Bitcoin, Tesla, and Post targets, respectively. The resulting data distribution is summarized in Table 2 .To enable meticulous annotation accounting for conversational context, we developed a custom system requiring reviewers to thoroughly examine all preceding multimodal content (text and images) prior to assigning stance labels. This framework was tailored to streamline high-quality annotation of multimodal conversation datasets. During the annotation phase, clear guidelines were provided to annotators, instructing them to label each comment as "against", "favor", or "none" to reflect their assessed stance. In addition to these stance labels, annotators were also tasked with assessing and marking the relevance of the data, determining whether it was related to the images in the post. This comprehensive annotation process ensured a thorough and contextually informed labeling of the dataset.We recruited seven researchers with expertise in natural language processing (NLP) to undertake the data annotation task. To validate the consistency and reliability of the annotations, two preliminary rounds of pilot annotation were conducted, followed by a review from three additional expert annotators to confirm the ability of each annotator to accurately execute the task. In the main annotation phase, we ensured that at least two annotators independently reviewed each data instance. Discrepancies between the initial annotators were resolved by involving a third annotator to evaluate the disputed instances, with the final stance determination achieved through a majority vote. This rigorous process not only secured the reliability of the dataset but also leveraged collective expertise to enhance the precision of the stance categorization.After completing the annotation process, we calculated the kappa statistic [8] to assess the consistency among annotators. Following Li et al. [26] , we specifically focused on the "Favor" and "Against" categories to determine the kappa statistic values. The kappa scores for Bitcoin, Tesla, and Post-T targets were found to be 0.72, 0.81, and 0.68, respectively, indicating a substantial level of agreement among the annotators and attesting to the reliability of the annotation process for our dataset.Table 3 presents the statistical data for our MmMtCSD dataset. The final annotated dataset encompasses 21,340 instances, with 14,083 of these instances, or 66%, being related to image content, underscoring the significance of multimodal data inclusion. Table 4 illustrates the distribution of instances across different conversational depths, including the number of words at each depth. Subsequently, we partitioned the dataset into training, validation, and test sets for all targets in a 70/15/15 ratio, ensuring a balanced representation for comprehensive evaluation and analysis. 
methodology: 
 This section provides a detailed description of our proposed MLLM-SD framework. Given a post and its accompanying comment text 𝑈 , along with images 𝑉 included in the post, the aim of MLLM-SD is to identify the stance label 𝑦 towards a specific target 𝑡. To effectively leverage both textual and visual information, we devised a targeted multimodal prompting approach, which is utilized to prompt the LLM for learning multimodal stance features.The architecture of our MLLM-SD is depicted in Figure 2 , comprises three primary components: (1) A textual encoder encoding textual prompts and content; (2) A visual encoder processing and encoding images with visual prompts; and (3) A multimodal fusion module where the LLM integrates information across modalities. The process culminates in matching the LLM's output with the appropriate labels, ensuring a cohesive and comprehensive approach to multimodal stance detection.The textual encoder aims to construct prompt templates for different types of text inputs, serving as the input for the subsequent LLM in the multimodal fusion stage. Specifically, within the textual encoder, in addition to the conversation history text data, we incorporate image caption designed to enhance the association between visual and textual information. Notably, we generate prompt templates for the conversational data, while for the image, we employ GPT4-Vision 4 to generate textual summaries.Specifically, to facilitate LLM understanding of multi-turn conversation stance detection, we first followed the LLaMA-2 conversation template design, specifying [𝑠𝑡𝑎𝑛𝑐𝑒 𝑑𝑒𝑡𝑒𝑐𝑡𝑖𝑜𝑛] as the task definition within the template. We then incorporated the conversation text U = ⟨𝑢 1 , 𝑢 2 , . . . , 𝑢 𝑛 ⟩, comprising a sequence of 𝑛 utterances 𝑢 𝑖 = ⟨𝑤 𝑖,1 , 𝑤 𝑖,2 . . . , 𝑤 𝑖,𝑗 ⟩ (∀𝑗 = 1, . . . , 𝑙 𝑖 ) representing posts or comments as the 𝑈 .Furthermore, inspired by the chain-of-thought (CoT) method [4] , to enhance the model's task performance, we define a one-shot prompt template 𝛿, which packages the P T , U, and provides an example, denoted as 𝐶𝑎𝑠𝑒, as a one-shot prompt, constructing a new input format. P T : The following is a conversation on social media based on a post. All comments are responses to the content of the post, and each comment replies to the previous one. There are three stances [favor, against, none]. Choose one of the three stances to express {name: 𝑢 𝑖 }'s stance towards "{target}".The 𝑃 𝑇 , conversation history 𝑈 , and one-shot CoT structure together form the one-shot prompt 𝛿:EQUATIONSecond, to allow better modeling of relationships between visual and textual information, we generated image descriptions using GPT4-Vision as 𝐶𝑎𝑝𝑡𝑖𝑜𝑛. The caption and one-shot prompt together 4 https://openai.com/research/gpt-4v-system-card constitute the textual modality input, denoted asEQUATIONFinally, we utilize a tokenizer from LLaMA to encode 𝛾 𝑇 into Γ 𝑇 .Following [13] , we first divide the image 𝑉 into a sequence of flattened 2D patches. These patches are then encoded into a linear sequence of embeddings which serve as input to the ViT model. We embed each patch into a embedding 𝑥 𝑖 𝑝 . Next, we embed each patch into a vector with positional encoding E 𝑝𝑜𝑠 through an embedding projection E:EQUATIONwhere 𝑣 0 is the embedding for the input image. Subsequently, we use the pre-trained Vision Transformer model ViT [15] to encode the input image for learning visual stance features. The resulting feature vector 𝑦 is processed by a linear transformation layer to adjust its feature dimensions to 𝑑 𝑣 = 4096, resulting in the image features Γ 𝑉 This transformation is critical for aligning the visual feature dimensions with the requirements of our model's architecture.In the multimodal fusion stage, we employed LoRA method to finetune the LLaMA2-chat (7B) [37] model, enabling it to effectively process text and images, manage multimodal inputs, and generate natural language responses to the proposed queries. To enhance the model's understanding of the input content, we utilized 𝑃 𝑉 (Figure 2 𝑃 𝑉 ) as a prefix for the input image, representing the structural form of the data. This prefix helps define the nature of the input and assists the model in grasping the structure of the multimodal data.The input architecture for the model is as follows:EQUATIONwhere [𝐼 𝑁 𝑆𝑇 ] denotes the user role, and [/𝐼 𝑁 𝑆𝑇 ] signifies the assistant role. The user input is organized into two segments: the first part comprises the image features (Γ 𝑉 ), and the second part consists of the textual instruction input (Γ 𝑇 ). Following the model's output generation, we engage in similarity matching to ascertain the stance conveyed by the output content. 
experimental setup: 
 In this section, we detail the baseline models utilized in our experiments, encompassing the experimental setup, which includes evaluation metrics, implementation details, and tests of the applicability of the MLLM-SD framework.We conduct extensive experiments utilizing state-of-the-art stance detection methods, broadly categorized into text-only and multimodal approaches. These experiments were designed to evaluate the performance of these methods in various stance detection scenarios, allowing for a comprehensive analysis of their effectiveness and the additional value provided by integrating multimodal data in the stance detection process.Text-only baselines. Fine-tuning based methods: (1) the pre-trained BERT [10] is fine-tuned on the training data; (2) the pre-trained RoBERTa [31] represents an enhancement over BERT, utilizing larger batch sizes and more data for training; (3) KEPrompt [19] uses an automatic verbalizer to automatically define the label words;(4) Branch-BERT [25] utilizes a CNN to extract important ngrams features incorporating contextual information in conversation threads. ( 5) GLAN [34] architecture adopts a three-branch structure to address the intricacies of conversational dynamics comprehensively. LLM-based methods: For the text stance detection task, we followed the prompting method proposed by Lei et al. [22] and utilized LLaMA-70b5 , Claude-3 6 , ChatGPT (gpt-3.57 and gpt-48 ) as the comparison method. Multimodal baselines. Three methods for multimodal modeling are employed as baselines: (1) BERT+ViT [30] utilizes BERT for textual encoding and ViT [13] for visual encoding. ( 2) TMPT [30] employs targeted prompts supplied to both the pre-trained language model and the pre-trained visual model. (3) GPT4-Vision 4 represents a vision-enhanced large model approach, integrating the capabilities of GPT-4 with visual processing to understand and analyze multimodal data for stance detection.Evaluation metrics. We adopt F1-avg as the evaluation metric to evaluate the performance of stance detection methods, consistent with the approaches in [26] and [33] . F1-avg represents the average F1 score computed for the "against" and "favor" stances, denoted as F1-against and F1-favor, respectively.Implementation details. During the training regimen of the MLLM-SD framework, the visual backbone is maintained in a static (frozen) state to preserve pre-trained visual features. The emphasis is placed on training the linear projection layer and fine-tuning the language model efficiently using LoRA. Through LoRA, the query and value weight matrices (𝑊 𝑞 and 𝑊 𝑣 , respectively) are fine-tuned, enabling model adaptation with minimal parameter updates. In our implementation, we specified the rank for LoRA as 64, striking a balance between adaptability and computational efficiency. Additionally, the model was consistently trained using an image resolution of 448x448 across all training phases, ensuring uniformity in visual data processing. 
experimental results: 
 In this section, we perform comprehensive experiments on our MmMtCSD dataset. Concretely, we present model comparisons in both in-target and cross-target setups. Notably, the reported results are averages obtained from three distinct initial runs.We first report the experimental results on the MmMtCSD dataset in the in-target setup, as shown in Table 5 , where the training and testing sets share identical targets. From the results, we make the following observations: First, the models built on multimodal inputs consistently outperform text-only models, highlighting advantages and necessity of multimodal inputs. Secondly, the simple concatenation of different modal LLMs does not lead to satisfactory performance on MMSD, with BERT+ViT achieving only 61.15%, while TMPT scored 64.47%, in evaluations across Tesla and Bitcoin targets. This phenomenon could be attributed to the difficulty in capturing high-level semantics within and across modalities through simple concatenation. Third, our MLLM-SD outperforms all baseline models on the MmMtCSD dataset. The significance tests comparing MLLM-SD to BERT+ViT and TMPT reveal that MLLM-SD exhibits a statistically significant improvement across most evaluation metrics (with a 𝑝-value of < 0.05). Fourth, even state-of-the-art stance detection methods, exemplified by MLLM-SD, exhibit an F1 score of only 71.85%, highlighting the persistent challenges in conversational stance detection.We undertook a series of cross-target experiments on the MmMtCSD dataset. The stance detection models are initially trained and validated on a source target and subsequently tested on a destination target. Our experimental design encompasses all available targets. The results of cross-target multimodal stance detection are reported in Table 6 . It can be seen that the LLMs achieve superior performance due to the need for detecting stances on unseen targets. This may be attributed to the powerful cross-target learning capability of LLMs. For our proposed MLLM-SD method, it achieves optimal results compared to large model-based methods on the Tesla and Bitcoin targets, while also outperforming all non-LLM baselines. This demonstrates the potential of our method in multimodal stance detection. When the stance targets become more diverse, MLLM-SD achieves sub-optimal results on the cross-target task (Post-T). This may be attributed to the reason that the training data in this setting To validate the impact of conversational context on stance detection, we constructed a comparison between single sentence and conversation history based models using LLaMA2-70b for both textonly and multimodal tasks. The results are illustrated in Figure 3 .The findings demonstrate that leveraging conversation history can significantly improve the accuracy of stance detection compared to relying solely on single-sentence inputs. Notably, in the multimodal stance detection scenario, the influence of contextual information on the results is more pronounced than in the text-only setting. This further accentuates the pivotal importance of incorporating comment conversation history in multimodal stance analysis.This analysis aims to examine the performance of different stance detection models across varying conversation depths. The results for different conversation depths are reported in Table 7 . The findings indicate that our model achieves enhanced performance across all measured depths. Notably, with the depth 5-6, our method demonstrates a more significant improvement in overall stance detection performance. This could be attributed to our innovative introduction of image captioning, which aids in capturing the contextual information between the text and images.To analyze the impact of different modules in our proposed MLLM-SD framework, we conduct an ablation study and report the results in Tables 5 6 7 . Note that the removal of caption (w/o 𝐶𝑎𝑝𝑡𝑖𝑜𝑛 ) sharply degrades the performance, which verifies the significance and efficacy of leveraging LLMs to summarize image content. This approach facilitates the model's comprehension of visual information and its association with the conversational text. In addition, the removal of the CoT process (w/o 𝐶𝑜𝑇 ) results in a substantial performance decline, suggesting that the CoT mechanism enhances the model's capability to interpret and analyze intricate instances. Furthermore, the concurrent removal of both CoT and Caption (w/o 𝐶𝑜𝑇 & 𝐶𝑎𝑝𝑡𝑖𝑜𝑛) components results in substantial degradation in performance, underscoring the critical role these elements play in enhancing the overall effectiveness of the framework. 
conclusion: 
 This paper introduces MmMtCSD, an extensive English multimodal conversational stance detection benchmark, specifically designed to emphasize multi-turn conversations. MmMtCSD addresses critical challenges in the multimodal stance detection task, aiming to bridge the gap between research and real-world applications. We propose a novel MLLM-SD framework that learns joint stance representations from textual and visual modalities. We conduct comprehensive experiments on our MmMtCSD dataset, and the experimental results demonstrate that MLLM-SD achieves superior performance on the MmMtCSD benchmark. Furthermore, extensive experimental findings underscore that MmMtCSD poses a more formidable challenge compared to existing benchmarks. This highlights substantial opportunities for advancements and innovations in stance detection. In future work, we plan to integrate linguistic knowledge and user information to further enhance the performance of multimodal conversational stance detection. 
 
 Appendix: https://www.reddit.comhttps://www.reddit.com/dev/apihttps://huggingface.co/meta-llama/Llama-2-70b-chat-hfhttps://www.anthropic.com/apihttps://platform.openai.com/docs/models/gpt-3-5https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo 


Decision: This paper introduces a multimodal multi-turn conversation stance detection task, contributing with a new dataset, MmMtCSD, and by a multimodal LLM-based framework to learn joint stance representations from text and images. 

Reviewers find the paper to be well-written (ey9i, A95K, Hd8L ) and the dataset contribution to be valuable for the community (ey9i, A95K, QNXi, Hd8L).

Regarding the experimental evaluation, while in general reviewers deemed it to be sufficient (A95K) and solid (Hd8L), several limitations were pointed out, with a common limitation being the lack of comparison with state-of-the-art methods (ey9i, A95K, QNXi, Hd8L).

The paper initially got mixed reviews, with two Weak Accepts, one Borderline Accept, and one Weak Reject. After the rebuttal, the authors managed to address some of the reviewers' concerns, and the paper ended with all reviewers giving a positive score, with two Borderline Accepts, one Weak Accept and one Accept.

Therefore, I agree with the reviewers' final assessment, and suggest this work to be accepted as Poster, provided that authors include the clarifications discussed in their rebuttal in the final manuscript.

Reviews: review: paper_summary: paper_summary: This paper proposes a new multimodal multi-turn conversational stance detection dataset to compensate for the shortcomings of existing research. And a new multimodal large language model stance detection framework was proposed to learn joint stance representations from text and visual modalities.
strength_weakness: strength_weakness: 1. This paper is well written and easy to follow.

2. This paper proposes a dataset for multimodal stance detection, which may promote the development of this field.
limitations: limitations: 1. This paper does not provide novel methods for this dataset. For example, the introduction of image captions and the utilization of conversation history information mentioned in this paper have already been extensively studied. 

2. The target of this dataset is not innovative, it has only been integrated based on existing work. 

3. This dataset consists of visual and textual modalities. So the baseline should compare more pre-trained models of image-text, such as InstructBLIP, MiniGPT-4, LLaVA etc.

4. There are many errors in the citation of references.
paper_summary: paper_summary: This paper introduces a new multimodal multi-turn conversational stance detection dataset to address the limitation of existing MSD studies ignoring the multi-party conversational contexts that naturally occur on social media. Besides, this paper proposes a novel multimodal large language model stance detection framework to derive stances from this new dataset.
strength_weakness: strength_weakness: 1 The overall writing of this paper is clear, the method level is clear and easy to understand.

2 This paper proposes a new multimodal stance detection dataset, which considers the multi-party conversational contexts in social media.

3 This paper leverages the comprehension capabilities of LLMs to detect stance across multiple modalities.

4 The experiment is sufficient and the related analysis is comprehensive.
limitations: limitations: 1 For dataset:

(1) Why Tesla and Bitcoin as targets? In addition, Post-T as a target is not very clear.

(2) Vision-Related in Table 3 reaches 65.99%, then how to deal with irrelevant images?

(3) Does each target have just one long conversation? Or a combination of multiple conversations? Is tagging an utterance for each utterance in a conversation?

(4) Different targets were evaluated in the experiment, how were the tasks evaluated? For example, in the Tesla target, input a comment about Tesla and output a stance, then the comment is a conversation? If it's a conversation, is it about the stance of the conversation as a whole, or about each utterance in the conversation?

(5) According to the statistical data in Table 3, the number of images and the number of text are not equal. Then how does the model handle the absence of image input?

(6) The lack of an ethical statement for the dataset and some copyright issues.

2 For approach, the MLLM-SD framework proposed in this paper is consistent with the existing multimodal large language model structure, and it does not seem to be very novel to impress me.

3 For experiments:

(1) Why not compare with existing multimodal large language models such as LLaVA, MiniGPT4, etc.? I was curious about the performance of existing open source multimodal large language models on MmMtCSD datasets.

(2) How was the result evaluated? I can see that F1 is calculated for both against and favor in Table 6 and 7, but the actual data does not contain three categories: against, favor and none, so none is not taken into account when calculating F1 of the two categories.

(3) GPT4-V is used in the results of multimodal experiments in Table 6, 7and 8. How is GPT4-V evaluated? What is the experimental setup like? Is it a zero-shot?
paper_summary: paper_summary: I would like to give a **weak accept** to this paper under review. The rest of my comments are organized into strengths, weaknesses, and suggestions for improvement.
strength_weakness: strength_weakness: - The authors successfully **identify a research gap** due to the absence of datasets that capture multi-party conversational contexts in multimodal stance detection. To address this, they introduce a new stance detection dataset, MmMtCSD.
- The paper proposes **a novel framework**, MLLM-SD, which integrates large language models with visual and textual data to effectively determine stances in social media conversational threads.
limitations: limitations: - **Missing Model Training Details**: The paper lacks a detailed section on model training and parameter settings, mentioning only the use of LoRA to fine-tune LLaMA at line 153 without further details.
- **Limited Cross-Domain Applicability**: The dataset focuses primarily on two specific domains (Tesla and Bitcoin), potentially limiting the generalizability of the model to other contexts or topics.
- **Incomplete Model Comparisons**: The comparisons are limited and may be biased; for instance, the paper lacks comparative analysis with other well-known large language models—such as Flan-T5, Vicuna, and others—after fine-tuning them on the same dataset, and disparities in model sizes could also lead to unfair comparisons.
title: title: Good paper, but has important weakness to address
paper_summary: paper_summary: The authors propose a new task and a new dataset named Multimodal Multi-turn Conversation Stance Detection. They also proposed an MLLM based method to address this task.
main_review: main_review: Advantages:

- The authors introduce an intriguing Multimodal Multi-turn Conversation Stance Detection dataset and task, which closely resembles real-world scenarios, setting it apart from previous stance detection research.
- Their novel approach utilizing multi-modal LLM for stance detection demonstrates effectiveness.
- The paper exhibits strong writing quality and clarity.

Despite these strengths, I didn't assign a higher rating due to the following weaknesses:

- Primary Weakness: The absence of comparison with the significant Multi-turn Conversation Stance Detection paper from ICDM 2023, "Contextual Target-Specific Stance Detection on Twitter: New Dataset and Method," in terms of both dataset creation and method effectiveness. The paper proposed one of the latest Multi-turn Conversation Stance Detection dataset, and also the SOTA method that should be considered a strong baseline. https://ieeexplore.ieee.org/abstract/document/10415673
- The authors need to emphasize the importance of "Multimodal Multi-turn Conversational Stance Detection." Specifically, they should clarify the necessity of both conversational context and visual information simultaneously. The current evidence, such as the relevance of images in 66% of conversations and the existence of cases requiring both types of information concurrently, appears insufficient.
- Unanticipated experimental results: Tables 6 & 7 show GPT-3.5 outperforming GPT-4 when targeting "Post-T," which seems unusual.
The authors should assess the robustness of their methods to different pre-given prompts, considering the significant impact prompts can have on model performance.

Additional Comments:

- The authors might consider extending their research to encompass multi-language support.
strength_weakness: strength_weakness: I think this is an interesting work of good quality. The strengths are listed as follows:.
- The authors introduce an intriguing Multimodal Multi-turn Conversation Stance Detection dataset and task, which closely resembles real-world scenarios, setting it apart from previous stance detection research.
- Their proposed approach, utilizing multi-modal LLM for stance detection, looks relatively novel and demonstrates effectiveness.
- Most of the evaluations are solid and comprehensive. 
- The paper exhibits strong writing quality and clarity.
limitations: limitations: Despite these strengths, I didn't assign a higher rating due to the following weaknesses:

- Primary Weakness: The absence of comparison with the significant Multi-turn Conversation Stance Detection paper from ICDM 2023, "Contextual Target-Specific Stance Detection on Twitter: New Dataset and Method," in terms of both dataset creation and method effectiveness. The paper proposed one of the latest Multi-turn Conversation Stance Detection dataset, and also the SOTA method that should be considered a strong baseline. https://ieeexplore.ieee.org/abstract/document/10415673
- The authors need to emphasize the importance of "Multimodal Multi-turn Conversational Stance Detection." Specifically, they should clarify the necessity of both conversational context and visual information simultaneously. The current evidence, such as the relevance of images in 66% of conversations and the existence of cases requiring both types of information concurrently, appears insufficient. 
- Unanticipated experimental results: Tables 6 & 7 show GPT-3.5 outperforming GPT-4 when targeting "Post-T," which seems unusual.
The authors should assess the robustness of their methods to different pre-given prompts, considering the significant impact prompts can have on model performance.

Additional Comments:

- The authors might consider extending their research to encompass multi-language support.