Multi-modal Stance Detection: New Datasets and Model (2024)

Abstract: Stance detection is a challenging task that aims to identify public opinion from social media platforms with respect to specific targets. Previous work on stance detection largely focused on pure texts. In this paper, we study multimodal stance detection for tweets consisting of texts and images, which are prevalent in today’s fast-growing social media platforms where people often post multi-modal messages. To this end, we create five new multi-modal stance detection datasets of different domains based on Twitter, in which each example consists of a text and an image. In addition, we propose a simple yet effective Targeted Multi-modal Prompt Tuning framework (TMPT), where target information is leveraged to learn multi-modal stance features from textual and visual modalities. Experimental results on our five benchmark datasets show that the proposed TMPT achieves state-of-the-art performance in multimodal stance detection.

heading: 1 Introduction
text: Stance detection is an important task for learning public opinion from social media platforms, which aims to determine people’s opinionated standpoint or attitude (e.g., Favor, Against, or Neutral, etc.) expressed in the content towards a specific target, topic, or proposition (Somasundaran and Wiebe, 2010; Augenstein et al., 2016). Existing conventional machine learning-based methods (Hasan and Ng, 2013; Mohammad et al., 2016; Ebrahimi et al., 2016) and deep learning-based methods (Augenstein et al., 2016; Sun et al., 2018; Zhang et al., 2020; Chen et al., 2021; Allaway et al., 2021; Liang et al., 2022a) have made promising progress in different types of stance detection tasks for pure texts.
* The first two authors contribute equally to this work. † Corresponding Author
However, more and more present-day social media platforms like Twitter allow people to post multi-modal messages, which encourages people to express their stances and opinions through multimodal content, posting texts with images for example. That is, detecting stance from the pure text modality may not accurately identify the user’s real view of a target. For example, Figure 1 shows a post composed of a text and an image. The stance expression towards “Donald Trump” and “Joe Biden” in this example can not be accurately identified based on text information unless combined with the information of the visual modality. Therefore, how to detect users’ stances on a topic from multi-modal posts might help better identify public opinion in social media.
For multi-modal stance identification, Weinzierl and Harabagiu (2023) discussed the multi-modal stance towards frames of communication. Different from conventional stance detection tasks that concentrate on the stance towards several predefined targets, their focus was centered on the frames of communication within multi-modal posts. Therefore, aiming to push forward the research of multi-
12373
modal stance detection, we create five new datasets, in which each example consists of a target, a text, and an image. These datasets contain a total of 17,544 examples across 5 domains and 12 targets, including hot topics, politicians, and debates.
To deal with multi-modal stance detection, we propose a simple yet effective Targeted Multimodal Prompt Tuning framework (TMPT), where the targeted prompt tuning is employed to adapt pretrained models for learning stance features from different modalities. Specifically, to leverage the target information in stance detection, we first devise targeted prompts for both textual and visual modalities. Then, the targeted prompts are fed to the pre-trained language model and pre-trained visual model to learn stance features for the target from different modalities. Further, a simple vector concatenation is used to fuse the features from different modalities for multi-modal stance detection.
The main contributions of our work are summarized as follows:
1) We manually annotate five new multi-modal stance detection datasets based on Twitter data from different domains. The release of the datasets would push forward the research in this field.
2) A simple yet effective targeted multi-modal prompting tuning framework is proposed to deal with multi-modal stance detection, where the target information is used to prompt the pre-trained models for learning multi-modal stance features.
3) A series of experiments on our datasets show that the proposed method significantly outperforms the baseline models1.

heading: 2 Related Work
text: Textual Stance Detection Various methods based on conventional machine learning (Hasan and Ng, 2014; Mohammad et al., 2016) and deep learning (Sun et al., 2018; Zheng et al., 2022; Li and Caragea, 2023; Li et al., 2023) have been proposed to deal with the stance detection regarding a specific target. For performing stance detection in real-world scenarios, many existing methods focus on the task of zero-shot stance detection2(Liu et al., 2021b; Liang et al., 2022b; Wen and Hauptmann, 2023; Zhao et al., 2023).
1To facilitate future research, our datasets and code are publicly available at https://github.com/Leon-Francis/ Multi-Modal-Stance-Detection
2Following (Allaway and McKeown, 2020), the term "zeroshot" here refers to the model’s ability to detect stance towards targets it has not encountered during training.
Prompt Tuning Prompting (Liu et al., 2021a) was originally aimed to design language instructions for pre-trained language models (PLMs) to transfer learning in downstream tasks (Shin et al., 2020; Jiang et al., 2020). Recent works begin to treat prompts as continuous vectors and optimize them during fine-tuning, called Prompt Tuning (Li and Liang, 2021; Lester et al., 2021; Liu et al., 2021c). Besides, Radford et al. (2021); Zhou et al. (2022); Ju et al. (2022); Jia et al. (2022) introduce prompt into vision-language models to leverage the ability of prompt tuning in multi-modal tasks.

heading: 3 Multi-modal Stance Detection Datasets
text: Based on three open-source textual stance detection datasets: Twitter Stance Election 2020 (Kawintiranon and Singh, 2021), COVID-CQ (Mutlu et al., 2020), and Will-They-Won’t-They (Conforti et al., 2020), and two hot topics in recent years: RussoUkrainian Conflict3 and Taiwan Question4. We create five multi-modal stance detection datasets of different domains to provide available data for this task: Multi-modal Twitter Stance Election 2020 (MTSE), Multi-modal COVID-CQ (MCCQ), Multi-modal Will-They-Won’t-They (MWTWT), Multi-modal Russo-Ukrainian Conflict (MRUC) and Multi-modal Taiwan Question (MTWQ).

heading: 3.1 Data Collection
text: We use Twitter Streaming API5 to collect tweets with corresponding keywords of five datasets (shown in Appendix A), keeping the posts containing text in English and at least one image or video/GIF. For videos/GIFs, we retain their first frame because the visual information contained in consecutive frames may be very similar. For posts with multiple images, we combine the text with each image to form multiple samples due to the fact that different images may contain completely different visual information. Finally, we obtain 130000, 90000, 60000, 100000, and 80000 candidate examples of five datasets, respectively.

heading: 3.2 Data Annotation
text: To ensure consistency with previous stance detection work, we follow the guidelines of Twitter
3https://en.wikipedia.org/wiki/ Russo-Ukrainian_War
4https://en.wikipedia.org/wiki/Cross-Strait_ relations
5https://developer.twitter.com/en/products/ twitter-api
Stance Election 2020 (Kawintiranon and Singh, 2021), COVID-CQ (Mutlu et al., 2020), and WillThey-Won’t-They (Conforti et al., 2020) to annotate the multi-modal stance of MTSE, MCCQ and MWTWT. For MRUC and MTWQ, the annotation guidelines are shown in Appendix B. The meaningless/noisy posts or those that do not comply with Twitter’s policies or annotation guidelines are discarded during the annotation. We invite eight experienced researchers6 to label the stance for each example. Each sample will be annotated by three different annotators, and the gold label is obtained by majority vote. For the disagreed results among the three annotators, we invited three additional annotators to annotate and then performed a majority vote to obtain the gold label7.

heading: 3.3 Quality Assessment
text: We use Cohen’s Kappa Statistic to evaluate the inter-annotator agreement (Cohen, 1960). The aver-
6We recruit experienced researchers who have worked on multi-modal learning over 3 years.
7During the annotation process, only 2.54% of the data need to be allocated to additional annotators.
age Cohen’s Kappa between our annotator pairs for MTSE is 0.703, for MCCQ is 0.689, for MWTWT is 0.729, for MRUC is 0.752, and for MTWQ is 0.691. This demonstrates that the Kappa scores of all datasets are substantial. In addition, the average Cohen’s Kappa reported in the related textual stance detection dataset Will-They-Won’t-They (Conforti et al., 2020) is 0.67, which also indicates the high quality of our new datasets from another angle.

heading: 3.4 Data Analysis
text: We finally got 17,544 well-annotated multi-modal samples across 5 domains and 12 targets. Each example consists of a text in English and an associated image. The statistics of datasets are reported in Table 1. Note that differences in label distribution between targets are common, which is also observed in other textual stance detection datasets (Mohammad et al., 2016; Mutlu et al., 2020; Conforti et al., 2020). Through analyzing the indication of whether the image conveys stance information introduced in Section 3.2, and analyzing the type of each image, we obtained statistics shown in Table 2. It can be seen that nearly half of the stance information in all five datasets comes from visual modalities. Further, the proportion of image types varies greatly among different datasets. This complicates image comprehension for stance detection, which is also the main challenge of multimodal stance detection.

heading: 4 Methodology
text: In this section, we introduce our proposed targeted multi-modal prompt tuning framework (TMPT) in
detail. Given a text S and an image I , the goal of multi-modal stance detection is to identify the stance label y for the specific target t based on S and I . Therefore, to leverage the target information for multi-modal stance detection, we design targeted multi-modal prompt tuning for both textual and visual modalities, which are utilized to prompt the pre-trained models for learning multimodal stance features. The architecture of our TMPT is illustrated in Figure 2, containing four main components: 1) Textual Prompt Tuning, which encodes the input of textual modality based on the textual targeted prompt; 2) Visual Prompt Tuning, which encodes the input of visual modality based on the visual targeted prompt; 3) Multi-modal Fusion, which fuses the representations from textual and visual modalities to capture the stance features. 4) Multi-modal Stance Detection, which derives the stance label for an input example according to the multi-modal stance features.

heading: 4.1 Textual Prompt Tuning
text: Textual Prompt Construction Inspired by the textual prompt tuning (Liu et al., 2023), considering the characteristics of stance detection, we devise a textual targeted prompt for each text to adapt the pre-trained language model to the stance detection task. As the example shown in Figure 2, take the target “Donald Trump” as an example, the textual targeted prompt is defined as:
PT = The stance on Donald Trump is : (1)
That is, the textual targeted prompts are designed according to the targets and the purpose of stance detection. The examples of textual targeted prompts are shown in Table 3. Other textual prompt settings are introduced in Section C.1.
Textual Encoder Based on the textual targeted prompts, we construct the input of each text on target t. Given a text S consists of a sequence of words S = {wi}ni=1, n is the length of S. The input of the textual modality is represented as:
IT = [CLS]PT [SEP]s[SEP] (2) Then, we adopt the pre-trained uncased BERTbase model (Devlin et al., 2019) to map each textual token into a dT -dimensional embedding:
[xT[CLS],x T 1 , · · · ,xTm+n+2] = BERT(IT ) (3)
Where xT[CLS] represents the embedding of the [CLS] token, m is the length of the textual targeted prompt. In this way, the pre-trained language
model can encode the input text according to the target, and obtain the feature representation that contains the target-specific stance information.

heading: 4.2 Visual Prompt Tuning
text: Visual Embedding Layer Following (Dosovitskiy et al., 2021), we first divide the image I into r fixed-sized patches I = {pi ∈ Rl2×c}ri=1, where (l, l) is the resolution of each patch, c is the number of channels. Then, following (Jia et al., 2022), we flatten the patches and map to dV -dimensional vector with a trainable linear embedding projection E. We refer to the output of this projection as the patch embeddings. Then we concatenate these patch embeddings in the sequence dimension and added the standard learnable 1D position embeddings Epos to the patch embeddings to retain positional information:
vj = pjE vj ∈ Rd, j ∈ {1, r} (4) V 0 = [{vj}rj=1] +Epos (5)
Where V 0 is the embedding for the input image.
Visual Prompt Construction Inspired by the visual prompt tuning proposed by (Jia et al., 2022), we devise visual targeted prompt, aiming to instruct the pre-trained vision model to learn the features according to the specific target. Specifically, we introduce continuous embedding (visual prompt tokens), as the visual prompt for target t. Each prompt consists of λ learnable embedding, which can be formulated as follows:
PV = {ei ∈ Rd V |1 ≤ i ≤ λ} (6)
Here, each different target corresponds to a different set of visual prompt embedding. In Section C.2, we introduced different initialization methods of visual prompt embedding.
Visual Encoder Based on the visual embedding V 0 and the visual targeted prompts PV , we use the pre-trained Vision Transformer model ViT (Dosovitskiy et al., 2021) with N layers to encode the input image for learning visual stance features on target t. Here, the targeted prompts are inserted into the first Transformer layer L1. Therefore, the first layer targeted prompted ViT is defined as:
[xV[CLS]1 ,Z1,V1] = L 1([xV[CLS]0 ,PV ,V0]) (7)
For layer k ∈ {2, N}, the targeted prompted ViT is defined as:
[xV[CLS]k ,Zk,Vk] = L k([xV[CLS]k−1 ,Zk−1,Vk−1])
(8)
Here, we use xV[CLS] to represent the embedding of [CLS] token learned by the final Transformer layer N , i.e., xV[CLS] = x V [CLS]N
. By using the visual prompts corresponding to each target, the pretrained visual model can encode the input image according to the target, and obtain the feature representation that contains the target-specific stance information.

heading: 4.3 Multi-modal Fusion
text: Based on the embeddings of [CLS] tokens, we first use two dense layers with Leaky ReLU (Maas et al., 2013) to derive the hidden representations of textual and visual modalities:
hT = f(W TxT[CLS] + b T ) (9) hV = f(W V xV[CLS] + b V ) (10)
Where hT ∈ Rdh and hV ∈ Rdh are the hidden representation of textual and visual modalities respectively. dh is the dimensionality of the hidden representation. f(·) represents the Leaky ReLU activation function. W T ∈ Rdh×dT and W V ∈ Rdh×dV are weight matrices. bT ∈ Rdh and bV ∈ Rdh are biases.
In our TMPT, vector concatenation is used to fuse the feature vectors from different modalities:
h = hT ⊕ hV (11)
Where h ∈ R2dh is the final multi-modal stance representation. ⊕ represents the vector concatenation operation.

heading: 4.4 Multi-modal Stance Detection
text: Then, the final multi-modal stance representation is fed into a fully-connected layer with a softmax function to capture a probability distribution ŷ ∈ Rdp in the stance decision space:
ŷ = softmax(W oh+ bo) (12)
Where dp is the dimensionality of stance labels. W o ∈ Rdp×2dh and bo ∈ Rdp are weight matrix and bias respectively.

heading: 5 Experimental Setup
text: To advance and facilitate research in the field of multi-modal stance detection, based on our new datasets, we conduct experiments on In-target Multi-modal Stance Detection, training and testing on the same target, and Zero-shot Multi-modal Stance Detection, performing stance detection on unseen targets based on the known targets.

heading: 5.1 Data Partition
text: For in-target multi-modal stance detection, each dataset is divided into training, development, and testing sets with a ratio of 7:1:28. For zero-shot multi-modal stance detection, the training and development set is built on known target(s), and the testing set is built on unknown target(s). Since there is only one target in the MCCQ dataset, we only use it for in-target multi-modal stance detection. In the MWTWT dataset, following (Conforti et al., 2020), we select four targets (CVS_AET, CI_ESRX, ANTM_CI, and AET_HUM) to perform zero-shot scenario since DIS_FOXA is not in the same domain as them. The statistics of each dataset are shown in Table 4.

heading: 5.2 Comparison Models
text: Pure textual modality baselines: 1) BERT (Devlin et al., 2019), the uncased BERT-base; 2) RoBERTa (Liu et al., 2019a), the RoBERTa-base; 3) KEBERT (Kawintiranon and Singh, 2022), a BERTweet-base model with specific knowledge of Twitter political posts. 4) LLaMA2 (Touvron et al., 2023), the LLaMA2-70b-chat; 5) GPT4 9. Pure visual modality baselines: 1) ResNet (He et al., 2016), the ResNet-50 v1.5; 2) ViT (Doso-
8To address the possibility of one text corresponding to multiple images, we use the Twitter ID to split datasets, thus avoiding the issue of data leakage.
9https://openai.com/research/gpt-4
vitskiy et al., 2021), the vit-base-patch16-224; 3) Swin Transformer (SwinT) (Liu et al., 2021d), the swinv2-base-patch4-window12-192-22k. Multimodal baselines: 1) ViLT (Kim et al., 2021), the vilt-b32-mlm; 2) CLIP (Radford et al., 2021), the clip-vit-base-patch32; 3) BERT+ViT, utilizing BERT as the textual encoder and ViT as the visual encoder, and concatenating the [CLS] vectors of textual and visual modalities for stance detection. 4) Qwen-VL (Bai et al., 2023), the Qwen-VL-Chat7b. 5) GPT4-Vision 10.

heading: 5.3 Experimental Settings
text: To leverage the powerful capabilities of large language models, following Gatto et al. (2023), we propose a variant of our TMPT model, named TMPT+CoT. By utilizing GPT4-Vision to generate a chain of thought from the text and image of the sample, we concatenate this chain of thought with the text to serve as the textual modality input for TMPT+CoT. The images are used as the visual modality input. This approach is employed for both the training and testing phases. We utilize the pretrained uncased BERT-base (Devlin et al., 2019) to embed each word as a 768-dimensional embedding and employ the pre-trained ViT-base (Dosovitskiy et al., 2021) to embed each image patch as a 768- dimensional embedding, i.e., dT = dV = 768. The resolution of the visual patch is set to (16, 16). The length of visual prompt tokens is set to c = 7. The dimensionality of hidden vectors is set to dh = 768. We use Macro F1-score to measure the model performance. The experimental results of our models are averaged over 5 runs to ensure the final reported results are statistically stable. For detailed settings of the experiments, please refer to Appendix C.

heading: 6 Experimental Results

heading: 6.1 In-target Multi-modal Stance Detection
text: The results of in-target multi-modal stance detection are shown in Table 5. It can be seen that our proposed TMPT outperforms fine-tuning based baselines on most of datasets, denoting that exploiting targeted prompt tuning can preferably leverage the target-specific multi-modal stance information, thus improving stance detection performance. Further, TMPT+CoT performs overall better than TMPT, which demonstrates that leveraging the knowledge
10https://openai.com/research/ gpt-4v-system-card
from large language models can improve the comprehension of textual modality, thereby achieving better performance. For uni-modal baselines, the performance of visual modality methods is unsatisfactory, while the performance of textual modality methods is much better. This indicates that the stance expression primarily resides in the textual modality. Further, CLIP, which considers both textual and visual modalities performs overall better than models which only consider textual modality, which proves the importance of visual modality in multi-modal stance detection. While, in datasets like MWTWT and MCCQ, GPT4-Vision underperforms GPT-4. Our analysis of images within these datasets revealed that a noteworthy proportion of images had comparatively high complexity levels. This also implies effective use of visual information is key to improving multi-modal stance detection performance. In addition, KEBERT, which integrates specific knowledge of Twitter political posts into BERTweet, achieves promising performance compared to other textual models, which indicates that exploring external target-related knowledge might improve the performance of multi-modal stance detection.

heading: METHOD MTSE MCCQ MWTWT MRUC MTWQ

heading: 6.2 Zero-shot Multi-modal Stance Detection
text: The results of zero-shot multi-modal stance detection are reported in Table 6. It can be seen that the large language models achieve superior performance due to the need for detecting stances on unseen targets. This may be attributed to the powerful zero-shot learning ability of large language models. For our TMPT, which does not use chain-ofthoughts from large language models, still achieves better performance than large language models in some targets, while also outperforming most of non-large language model baselines. This shows the promise of our TMPT in zero-shot stance detection. Further, TMPT+CoT performs overall better than TMPT, which indicates that obtaining powerful text and visual comprehension abilities from large models may be key to improving the performance of detecting stance for unseen targets.

heading: 6.3 Ablation Study
text: To analyze the impact of the targeted prompt tuning in our proposed TMPT, we conduct an ablation study and report the results in Table 7. Note that the removal of textual prompting (w/o PT ) sharply degrades the performance, which verifies the significance of textual prompting in learning textual targeted stance features for multi-modal stance detection. In addition, the removal of visual prompt tuning (w/o PV ) leads to considerable performance degradation, which indicates that utilizing visual prompt tuning can make better learning of visual targeted stance information and thus improves the performance of multi-modal stance detection.

heading: 6.4 Generalization of Targeted Multi-modal Prompt Tuning
text: Pre-trained Models Previous experiments have demonstrated that the stance expression primarily resides in the textual modality. Therefore, to investigate the generalization of our TMPT when used with different pre-trained language models, we conduct experiments with two variants of our TMPT by using two other promising PLMs: RoBERTa (Liu

heading: BERT+ViT -Cross-modal Attention -GAT TMPT (ours)
text: et al., 2019b) and KEBERT (Kawintiranon and Singh, 2021). The results are shown in Figure 3. Note that our Targeted Multi-modal Prompting can directly work with the two PLMs and achieve better performance, showing the compatibility of our method with various pre-trained models.
Multi-modal Fusion We choose cross-modal attention (Wei et al., 2020) and GAT (Velickovic et al., 2018) to analyze the performance of our TMPT with different multi-modal fusion methods. The results are shown in Figure 4. It can be seen
that no matter which fusion method we use, the experimental performance is better than BERT+ViT. This indicates that our TMPT can directly work with various multi-modal fusion methods and lead to improved multi-modal stance detection performance.

heading: 6.5 Error Analysis
text: From the results in Table 6, we can see that our TMPT performs well in the MWTWT dataset, but still insufficient in other datasets on zero-shot stance detection. One possible reason is that targets in MWTWT are all about expressing views on corporate mergers, so unknown targets can easily find commonalities in the dataset. However, for other datasets, the topics are diverse, which poses a challenge to mining targeted information. Therefore, how to better learn the correlation information between targets from data on diverse topics is a potential direction to improve the performance of the zero-shot scenario.
Further, after analyzing examples of misclassification, we found that among the incorrect samples, approximately 70% (based on calculations on randomly sampled 300 incorrect samples from five datasets) of the images contained stance-related content. This indicates that for our multi-modal stance detection task, it is important for further exploration in extracting and utilizing features from the visual modality.
Building on the previous step, we conduct error analysis on different image types. The results are illustrated in Table 8. A larger ERR-Img-prop indicates a weaker ability of the model to handle samples with images of that category, indicating that for images containing words, memes, and mixed features, more effective methods of feature extraction and understanding remain to be proposed.

heading: 6.6 Visualization
text: To qualitatively investigate how our TMPT improves the performance of multi-modal stance detection, we visualize the attention values calculated by the targeted prompts and the vectors of the final layer
of two encoders. The results are shown in Figure 5. It can be seen that the crucial textual tokens and the key visual patches regarding different targets are highly attended to and discriminated by our TMPT. This illustrates that the proposed TMPT can learn the important stance features for the specific target with the help of targeted prompt tuning, thus improving the learning ability of multi-modal stance detection.

heading: 7 Conclusion
text: In this paper, we present MTSE, MCCQ, MWTWT, MRUC and MTWQ, five new datasets for multimodal stance detection. Based on the created datasets, we present in-target multi-modal stance detection and zero-shot multi-modal stance detection, aiming to advance and facilitate research in the field of multi-modal stance detection. In addition, we propose a simple yet effective targeted multi-modal prompting framework (TMPT) to deal with multi-modal stance detection, where the target information is explored to prompt the pre-trained models in learning multi-modal stance features. Extensive experiments on the new datasets show that our proposed TMPT achieves overall better performance than state-of-the-art baseline methods.

heading: Acknowledgements
text: This work was partially supported by the National Natural Science Foundation of China (62176076), Natural Science Foundation of GuangDong 2023A1515012922, the Shenzhen Foundational Research Funding (JCYJ20220818102415032, JCYJ20210324115614039), the Major Key Project of PCL2021A06, Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies 2022B1212010005.

heading: Limitations
text: Our method needs to devise a specific prompt for the given target, which needs to take time and artificial effort to analyze the target information in the real-world scenario for designing and selecting appropriate prompts. Furthermore, our proposed method does not integrate external target-specific knowledge to improve the learning of multi-modal stance information, such as the background knowledge of targets. Integrating external knowledge related to the target can improve the performance of stance detection. In addition, the current version of the data does not consider audio modality and video information, which is also an issue we need to explore in the future.

heading: Ethics Statement
text: This work presents MTSE, MCCQ, MWTWT, MRUC and MTWQ, five new open-source datasets for the research community to study multi-modal stance detection. The MTSE, MCCQ, and MWTWT are the extension of Twitter Stance Election 2020 (Kawintiranon and Singh, 2021), COVID-CQ (Mutlu et al., 2020), and Will-They-Won’t-They (Conforti et al., 2020), which are three open-source textual stance detection datasets for academic research. We only collect the tweet text and image content needed for our research from Twitter following the privacy agreement of Twitter for academic usage, so there is no privacy issue. To annotate extended data, we recruited 8 experienced researchers who work on natural language processing or multi-modal learning. The detailed collect and annotate process has been illustrated in Section 3. Each researcher is paid $6.5 per hour (above the average local payment of similar jobs). The entire annotation process lasted 5 months, and the average annotation time of the eight researchers was 430 hours. During the annotation process, samples that contain personally identifiable information will be discarded, and only the tweet IDs and humanannotated stance labels will be shared. Thus, our data set complies with Twitter’s information privacy policy. The annotators have no affiliation with any of the companies that are used as targets in the dataset, so there is no potential bias due to conflict of interest. We used the ChatGPT service from OpenAI for our writing. We followed their term and policies. Some examples in our paper may include a stance or tendency. It should be clarified that they are randomly sampled from the dataset
for better studying the dataset and task, and do not represent any personal viewpoints.

heading: A Keywords
text: In this section, we introduce the keywords to retrieve tweets.

heading: A.1 Multi-modal Twitter Stance Election 2020
text: • Since Twitter Stance Election 2020 (Kaw-
intiranon and Singh, 2021) didn’t explicitly
give the Keywords for collecting tweets, We used the keywords related to the election while with no clear preference: one of #vote, #Debates2020, #USElection2020, #PresidentialDebate2020, #2020Election, #votersuppression, #GetOuttheVote, #2020elections + mention of Trump or Biden.
• Filter for posting time:
DT 01/01/2020 → 09/30/2020 JB 01/01/2020 → 09/30/2020

heading: A.2 Multi-modal COVID-CQ
text: • We followed the keywords for collecting tweets from COVID-CQ (Mutlu et al., 2020): one of hydroxychloroquine, chloroquine, HCQ.
• Filter for posting time:
CQ 04/01/2020 → 04/30/2020

heading: A.3 Multi-modal Will-They-Won’t-They
text: • We followed the keywords for collecting tweets from Will-They-Won’t-They (Conforti et al., 2020): one of merge, acquisition, agreement, acquire, takeover, buyout, integration + mention of a given company/acronym.
• Filter for posting time:
CVS_AET 02/15/2017 → 12/17/2018 CI_ESRX 05/27/2017 → 09/17/2018 ANTM_CI 04/01/2014 → 04/28/2017 AET_HUM 09/01/2014 → 01/23/2017 DIS_FOXA 07/09/2017 → 04/18/2018

heading: A.4 Multi-modal Russo-Ukrainian Conflict
text: • We used the keywords: Ukraine, Russia, Putin, Zelensky, Ukrainian, Russian
• Filter for posting time:
RUS 01/01/2022 → 06/30/2023 UKR 01/01/2022 → 06/30/2023

heading: A.5 Multi-modal Taiwan Question
text: • We used the keywords: Taiwan, Taiwan Crisis, Nancy Pelosi, Taiwan Strait
• Filter for posting time:
MOC 01/01/2022 → 06/30/2023 TOC 01/01/2022 → 06/30/2023

heading: B Annotation Guidelines
text: To ensure consistency with previous stance detection work, we follow the guidelines of Twitter Stance Election 2020 (Kawintiranon and Singh, 2021), COVID-CQ (Mutlu et al., 2020), and WillThey-Won’t-They (Conforti et al., 2020) to annotate the multi-modal stance of MTSE, MCCQ and MWTWT. For MRUC and MTWQ, the annotation guidelines are shown below:

heading: B.1 Annotation Guidelines of MRUC and MTWQ
text: The annotation process consists of choosing one of three possible labels, given a tweet and an image. The three labels to choose from are Support, Oppose and Neutral.
Label 1: Support - If the tweet and image use direct or indirect expressions to support the target, or support objects which can represent the target (such as leaders, events).
Label 1: Oppose - If the tweet and image use direct or indirect expressions to oppose the target, or oppose objects which can represent the target (such as leaders, events).
Label 1: Neutral - If the tweet and image only mention the target without expressing a stance.

heading: B.2 Details of Datasets Partition
text: As mentioned in Section 5.1, for each task/target, the dataset is divided into a training set, a development set, and a testing set with a ratio of 70%:10%:20%. In order to ensure that the partition is not affected by the data distribution bias. We performed 20 different random divisions for each task using the ratios above. We use the two baselines: BERT and BERT+ViT, to test every 20 groups of divisions. For each task, we take the division that can make the results of two baseline close to the median as the final partition.

heading: C Prompt Tuning Analysis

heading: C.1 Analysis of Textual Prompt Tuning
text: Frozen vs Tuned For textual prompt tuning, we utilize a frozen paradigm to make full use of the semantic information learned by the pre-trained language model. To analyze the effectiveness of the frozen paradigm, we also tried a tuned soft prompt, the results are shown in Table 9. The results of the tuned paradigm are extremely poorer than the frozen one and fluctuate greatly. One possible reason is that, unlike the continuous picture

heading: METHOD MTSE MCCQ MWTWT MRUC MTWQ
text: patch in the visual transformer, the token in the textual modality is discrete single words, so it is hard to use a gradient-based method to fine-tune the soft prompts like a visual modality. Thus, in this paper, we choose a manually designed fixed prompt to obtain better performance.
Hand-design textual prompts To analyze the impact of the type of the hand-design textual prompts, we design several types of textual prompts and report the experiments in Table 10. Take the MTSE dataset as an example, it can be seen that the experimental results of all types of prompts are superior to those without prompts (BERT+ViT). This demonstrates the significance of the targeted prompts in this task. Further, we can also see that there are considerable differences in performance between different types of prompts. Specifically, when using both target and stance to design prompts ( 3⃝, 4⃝, and 5⃝), the model performs significantly better. Therefore, in our method, we choose the type of 5⃝ to devise the textual prompts for multi-modal stance detection.

heading: C.2 Analysis of Visual Prompt Tuning
text: Depth of prompt tuning Following (Jia et al., 2022), we conduct experiments with different depths of visual prompt tuning to analyze the impact of the depth of prompt tuning in our model. Here, shallow prompt tuning refers to only finetuning the prompt tokens in the Embedding layer of ViT, while deep prompt tuning refers to finetuning the prompt tokens in the Embedding layer of ViT and each layer in the transformer. The ex-
perimental results are shown in Table 11. We can see that shallow prompt tuning has a better overall effect.
Number of visual prompt tokens To analyze the impact of the number of visual prompt tokens, we set the value range of tokens as {3, 5, 7, 9} for comparative experiments. The results are shown in Figure 6. Note that different values of tokens can have a certain impact on performance. When the value is 7, the overall performance of the model on all datasets is the best. Therefore, we set the number of tokens to 7 in our method.

heading: Future Work
text: In addition, the current version of the data does not consider audio modality and video information, which is also an issue we need to explore in the future