Tree-of-Counterfactual Prompting for Zero-Shot Stance Detection (2024)

Abstract: Stance detection enables the inference of attitudes from human communications. Automatic stance identification was mostly cast as a classification problem. However, stance decisions involve complex judgments, which can be nowadays generated by prompting Large Language Models (LLMs). In this paper we present a new method for stance identification which (1) relies on a new prompting framework, called Tree-of-Counterfactual prompting; (2) operates not only on textual communications, but also on images; (3) allows more than one stance object type; and (4) requires no examples of stance attribution, thus it is a "Tabula Rasa" Zero-Shot Stance Detection (TR-ZSSD) method. Our experiments indicate surprisingly promising results, outperforming fine-tuned stance detection systems.

heading: 1 Introduction
text: Stance was defined by Biber and Finegan (1988) as the expression of an author’s standpoint and judgment towards a given proposition. As such, stance is used for discovering attitudes from various sources, ranging from social media, to debates or news articles, cf. Hardalov et al. (2021). Regardless of the context in which it is used, stance always has a subject and an object. The subject of stance can be the speaker in a conversation or the author of a Social Media Posting (SMP). The stance object, as reported in Hardalov et al. (2021, 2022); Liu et al. (2023c), can be sometimes a topic, while other times it can be a frame of communication. Topics typically correspond to the name of a controversial political figure, e.g. Hillary Clinton, Donald Trump, or a noun phrase, e.g. “gun control” or “abortion”. Automatically inferring the stance towards a topic entails modeling complex judgments that sometimes use commonsense knowledge about the topic, as shown in the example illustrated in Figure 1(a). The SMP used in this
example implies that it is best to give birth - even at the cost of pain. This is a direct quote from Mother Teresa, in a famous speech concerning abortion at a National Prayer Breakfast in Washington, D.C., on February 5, 1994. Therefore, the SMP’s author has a stance of Reject towards the topic of “abortion”.
Frames of Communication (FoCs), the other possible objects of stance that we consider in this paper, highlight problems of interest by addressing what causes those problems, and determining moral judgments, as defined in Entman (1993). The FoC
861
illustrated in Figure 1(b) highlights the problem of collective responsibility during vaccination campaigns, explaining why people should make their own vaccination decisions. The SMP from this example contains both a text and an image. At first look, the FoC is contradicted by the text of the SMP, but interestingly, the image also contributes to the stance decision. The image represents the terrifying Nazi character of Hans Landa from the movie Inglourious Basterds, and the text superimposed on the image implies that he is looking for unvaccinated people to punish them. A closer inspection of the entire SMP indicates that the quoted text in fact is sarcastic, and combined with the image, it accounts for the inference of an Accept stance.
The examples illustrated in Figure 1 showcase some of the complex forms of reasoning required when inferring stance values. This motivated us to find whether existing Large Language Models (LLMs) and Large Multimedia Models (LMMs) can be used to access their knowledge, and, more importantly, to use their reasoning capabilities to infer the stance of communications involving both text and images. In this paper we focused on social media platforms where millions of users express their opinions and attitudes in conversations about issues of their interest, therefore we explored stance detection from SMPs.
Previous stance detection methods (Augenstein et al., 2016; Du et al., 2017; Sun et al., 2018; Siddiqua et al., 2019; Conforti et al., 2020; Miao et al., 2020) mostly used topics as stance objects, casting stance detection as a classification problem, trained on texts addressing a certain number of topics, and tested on texts addressing the same topics. However, as reported in Allaway and McKeown (2020); Allaway et al. (2021), stance detection methods designed in this way are limited in their ability to generalize to a wide variety of topics. To address this limitation, Zero-Shot Stance Detection (ZSSD) was considered in Allaway and McKeown (2020); Allaway et al. (2021); Liang et al. (2022a,b); Li and Yuan (2022), in which the stance topics from the test data are not seen in the training data. To take this a step further, in this paper we address the problem of “Tabula Rasa” Zero-Shot Stance Detection (TR-ZSSD), in which no example of stance inference towards any topic is needed, and moreover, the stance of SMPs towards topics as well as FoCs is identified.
Instead of classifying the stance value of an SMP towards a topic or an FoC, we considered a spe-
cial form of counterfactual reasoning, in which we prompt an LLM or an LMM to generate an explanation if the stance s of an SMP p towards the stance object o is considered to take a value v. Since the value v is not known, we consider counterfactuals, i.e. all alternative possibilities of stance values, given the observed p and o. This entails that for each pair (p, o) we obtain (1) an explanation eA, providing the rationale for assigning the value Accept to s; (2) an explanation eR, providing the rationale of the Reject value; and (3) an explanation eN , providing the rationale of the No Stance value. For each explanation, the LLM (or LMM) follows different reasoning paths, thus it generates a Tree-of-Thoughts, inspired by (Yao et al., 2023). Finally, the LLM (or LMM) verifies which of the three explanations is most likely, using Chain-of-Verification prompting (Dhuliawala et al., 2023), leading to the final inference of the stance value v. In all this reasoning process, based on multiple prompting of an LLM (or LMM), no need for prior examples of stance value assignments is required. We call this method Tree-of-Counterfactual (ToC) prompting for Tabula Rasa Zero-Shot Stance Detection (ToC-TR-ZSSD), making the following contributions: 1 We introduce the first Tabula Rasa Zero-Shot Stance Detection (TR-ZSSD) method, that operates without any prior example of stance value assignment. TR-ZSSD is a step forward from previous ZSSD, in which examples of stance value assignment are provided, but new topics are not available at training time. 2 Our TR-ZSSD method operates not only on texts, but also on SMPs that contain images. Moreover, the method allows the stance to consider different objects, e.g. topics or FoCs. 3 We introduce the first stance detection method that prompts LLMs or LMMs to generate explanations of possible stance values and to verify the explanations, selecting the most rational stance value. 4 Our ToC prompting for stance detection refocuses the problem of stance identification on reasoning from current approaches relying on classification, enabling an explainable approach to stance identification.
We make available all code, prompts, and experiments on GitHub1.
1https://github.com/Supermaxman/gpt-gleam

heading: 2 Related Work
text: Stance detection on social media attracted plenty of interest, spearheaded by Task 6 of SemEval2016 (Mohammad et al., 2016), where multiple neural systems (Zarrella and Marsh, 2016; Wei et al., 2016; Tutek et al., 2016) were evaluated. The dataset used in SemEval-2016 considered tweets discussing several controversial topics (e.g., “abortion”), annotated with the stance of each tweet’s text towards these topics. Continued interest was sparked by the release of TweetEval (Barbieri et al., 2020), a commonly utilized benchmark for stance detection (Nguyen et al., 2020; Loureiro et al., 2022; Du et al., 2023). Additional research interest formed around the Zero-Shot Stance Detection (ZSSD) setting, which focused on identifying the stance of novel topics, not present in the training data (Allaway et al., 2021; Liang et al., 2022a,b). Cross-topic ZSSD systems typically focus on stance datasets with hundreds or thousands of topics, available from VAST (Allaway and McKeown, 2020) and WT-WT (Conforti et al., 2020). This formulation of cross-topic ZSSD stands in contrast with the Tabula Rasa ZSSD employed in this work, in which no training examples are provided for the ToC-TR-ZSSD method. ToC-TR-ZSSD also contrasts with prior unsupervised stance detection research, which focused on identifying stance via the structural properties of the speaker graph (Murakami and Raymond, 2010; Pick et al., 2022). Furthermore, stance detection research has considered incorporating external knowledge, such as Li et al. (2023a), and In-Context Learning, such as Zhang et al. (2023a), which complement the introduction of ToC-TR-ZSSD.
Counterfactual reasoning considers alternatives to facts, events, or states, making inferences about what might have happened or what may have been possible. For example, if you trip and fall due to not tying your shoes, you can easily imagine a counterfactual reality where you tied your shoes and therefore did not trip. Counterfactual reasoning is known to play an important role in human learning and decision-making. In philosophical and AI literature (Lewis, 1973; Ortiz Jr., 1999; Pearl, 2009), counterfactuals were considered to give us reliable insights into the specifics of causality. However, recently, counterfactual reasoning has been integrated into NLP systems to enhance understanding and generation capabilities by considering altering narratives or decisions based on hypothetical scenar-
ios. For example, Qin et al. (2019) utilize counterfactual reasoning for story re-writing, while Zeng et al. (2020) incorporated counterfactual reasoning to improve Named Entity Recognition (NER) by generating counterfactual examples. Counterfactual reasoning also yields insights into the explainability of NLP systems, with Jacovi et al. (2021) focusing on identifying and altering features in input data with counterfactuals that change model predictions, thereby providing insights into model behavior, and Robeer et al. (2021) where classification predictions are explained with counterfactuals. Recently, counterfactuals have been utilized to improve prompt design for LLMs (He et al., 2022; Chen et al., 2023), yielding improved few-shot performance on vision and language tasks by discovering more generalizable prompts. However, none of these LLM prompting techniques directly utilized LLMs to generate, compare, and explain counterfactual rationales, as we do in the ToC-TR-ZSSD method.

heading: 3 The Method
text: Tree-of-Counterfactual (ToC) prompting of Large Language Models (LLMs)2 for stance detection is structured into three distinct phases. 2 Phase A, illustrated in Figure 2(A), involves constructing a tree of counterfactuals, considering each possible stance value. For instance, an Accept branch is constructed for the SMP pi and the stance object oj when we assume the stance value assigned to si,j is Accept. Similarly, counterfactual Reject and No Stance branches are constructed. Note that each branch is always constructed, regardless of the content of pi or oj . However, only one of the possible stance values will hold - but we need to proceed to the next phases to find out which stance value is the most rational for pi towards oj . Therefore Phase A sets the stage for the following phases to perform reasoning that allows si,j to be inferred. 2 Phase B illustrated in Figure 2(B), prompts the LLM to explain the reason why a certain stance value si,j is attributed to the pair (pi, oj), for each of the possible counterfactual stance values. The prompting approach seeks to generate post hoc rationalizations for why each counterfactual stance value is held by SMP pi towards stance object oj .
2We use Large Language Models (LLMs) and Large Multimodal Models (LMMs) interchangeably here, depending on the modality of the SMPs under consideration.
This Chain-of-Explanation (CoE) prompting instructs the LLM to provide step-by-step rationales:
You are an expert linguistic assistant. You will be tasked with explaining why a tweet may have a stance towards a provided [topic|frame of communication]. (Make sure to explain how the image and text together contribute to the stance.) You should discuss your reasoning in detail, thinking stepby-step.
Chain-of-Explanation Prompting
Sy st
em P
ro m
pt
Social Media Posting: {text} [Topic|Frame of Communication]: {object} Stance: {stance} ({image})Us er P ro m
pt
As can be seen, depending on the type of stance object, i.e. topic or FoC, the language of the CoE prompt is modified slightly, because of the square “[...|...]” brackets, with the pipe symbol “|” representing the possible exclusive options for the prompt. Furthermore, if multimodal stance detection is being performed with an LMM, additional sections are included in the prompt to address images, represented by round “(...)” brackets. Textual content from the SMPs is inserted into the “{text}” section, while the textual representation for the object of stance is included in the “{object}” section. Counterfactual stance values are inserted into the “{stance}” section. If multimodal stance detection
is being performed, then the image context is also included in the “{image}” section.
With CoE prompting of the LLM, we obtain counterfactual rationales eA, eR, and eN , one for each possible stance value. However, the final decision of the value of si,j is produced in phase C. 2 Phase C, illustrated in Figure 2(C), takes inspiration from Chain-of-Verification (CoVe) prompting, proposed by Dhuliawala et al. (2023). CoVe prompting provides LLMs the capability to deliberate on the responses they give, such that they can correct their mistakes through introspection. We employ the counterfactual rationales eA, eR, and eN , produced in phase B, as the baseline responses to consider for verification, and we alter the verification step to consider a form of contrastive verification. In this way we create Counterfactual Chain-of-Contrastive Verification (CCoCV) prompting to introspect upon the counterfactual rationales generated in Phase B, and compares their arguments with the contents of the post pi and stance object oj to contrast each stance value hypothesis:
You are an expert linguistic assistant. You will be tasked with judging which stance value a tweet has towards a provided [topic|frame of communication]. Thorough rationales will be provided for each stance value. You should discuss your reasoning in detail, thinking step-by-step. Discuss the strengths and weaknesses for each rationale, providing a final judgement for the stance value of the tweet towards the provided [topic|frame of communication]. Your final sentence should include only one possible stance value: Accept, Reject, or No Stance
Counterfactual Chain-of-Contrastive Verification Prompting
Sy st
em P
ro m
pt
Social Media Posting: {text} [Topic|Frame of Communication]: {object} Stance: Accept Rationale: {accept_rationale} Stance: Reject Rationale: {reject_rationale} Stance: No Stance Rationale: {no_stance_rationale} ({image}) U se r P ro m pt
As can be seen, the semantics of C-CoCV prompting uses the same semantics as CoE prompting, to allow for the various stance objects to be considered as well as for using the text and image content of the SMP or the text content of the stance objects. In addition, the counterfactual rationales generated from CoE prompting are inserted into the “{accept_rationale}”, “{reject_rationale}”, and “{no_stance_rationale}” sections.
The three phases of ToC prompting exploit the capacity of Large Language Models (LLMs) to generate and evaluate post hoc rationalizations for different stance hypotheses, facilitating a more explainable inference of stance, wherein each stance hypothesis is rigorously scrutinized for its validity.
This process echoes the natural human cognitive process of considering and comparing alternatives before arriving at a conclusion (Van Hoeck et al., 2015), allowing for a stance detection mechanism that is both robust and transparent. Because ToC prompting requires no use of annotated stance data or demonstrations, it can be considered a TR-ZSSD method.

heading: 4 Experimental Results

heading: 4.1 Experiments
text: We consider three distinct stance detection settings within which to compare and evaluate ToC-TRZSSD. These settings are determined by three different stance-annotated datasets, detailed in Table 1. Furthermore, as our method is entirely zero-shot, we discuss any possibilities of test collection contamination in the training data of the considered LLMs in Appendix B.
SemEval-2016 (Mohammad et al., 2016) introduced one of the first widely used stance datasets on Twitter. It addressed five controversial topics: “Abortion”, “Atheism”, “Climate Change”, “Feminism”, and “Hillary Clinton.” An ongoing leaderboard on the SemEval 2016 task was organized by TweetEval (Barbieri et al., 2020), which fostered continued competition on the dataset. This resource has been widely used in computational linguistics and natural language processing research to develop and benchmark methods capable of detecting stance from text, contributing significantly to advancements in automatic stance detection. Statistics for the distribution of stance values in the test collection of SemEval-2016 Task 6 A are provided in Table 1.
CoVaxFrames (Weinzierl and Harabagiu, 2022b) made available stance annotations on SMPs originating from Twitter, discussing COVID-19 vaccine hesitancy. COVAXFRAMES includes 113 Frames of Communication (FoCs), discovered through Question Answering as answers to questions from the Vaccine Confidence Repository (Rossen et al., 2019). Stance was annotated to-
wards these 113 FoCs for a collection of text-only SMPs. FoCs have recently been found to be superior stance object types (Weinzierl and Harabagiu, 2024) when compared to topics, requiring less implicit knowledge to infer the stance of a text towards an FoC than a topic. COVAXFRAMES includes FoCs such as “The COVID vaccine renders pregnancies risky, and it is unsafe for unborn babies” and “Preference for getting COVID-19 and fighting it off than getting vaccinated.” Statistics for the distribution of stance values in the test collection of COVAXFRAMES are provided in Table 1.
MMVax-Stance (Weinzierl and Harabagiu, 2023) is the only multimodal dataset that contains stance annotations, with the SMPs originating from Twitter. The objects of the stance annotations are the same 113 COVID-19 vaccine hesitancy FoCs as COVAXFRAMES. The complexity of multimodal stance annotation involving the interpretation of texts as well as memes, images, and videos, presents a significant challenge, making this dataset very valuable. Statistics for the distribution of stance values in the test collection of MMVAX-STANCE are provided in Table 1.

heading: 4.2 Results
text: To evaluate the quality of stance detection on the test collections from SemEval-2016, COVAXFRAMES, and MMVAX-STANCE we use the Precision (P), Recall (R), and F1 metrics for detecting the Accept and Reject values of stance. We also compute a Macro averaged Precision, Recall, and F1 score over the Accept and Reject stance values. The text-only stance detection evaluation results for SemEval-2016 are listed in Table 2, while the evaluation results for COVAXFRAMES are listed in Table 3. The multimodal stance detection results for MMVAX-STANCE are presented in Table 4. The bolded numbers represent the best results obtained for each setting: stance detection with fine-tuning and TR-ZSSD. All hyperparameters are included in the linked GitHub repository and discussed in Appendix A.
Two LLMs were considered in our text-only TR-
ZSSD experiments on SemEval-2016 and COVAXFRAMES: GPT-3.5 (Ouyang et al., 2022), and GPT-4 (OpenAI, 2023). Both these LLMs have been shown to have very high performance in complex zero-shot and few-shot reasoning tasks (Espejel et al., 2023; Ouyang et al., 2022; OpenAI, 2023). Furthermore, two baseline prompting methods were employed with GPT-3.5 and GPT4: Direct prompting and Chain-of-Thought (CoT) prompting (Wei et al., 2022).
Next, we discuss the results on each dataset.
SemEval-2016 Results: We compare against multiple fine-tuned systems from the original SemEval-2016 competition, including SVMngrams (Mohammad et al., 2016), MITRE (Zarrella and Marsh, 2016), and pkudblab’s system (Wei et al., 2016). Additional fine-tuned systems were considered which participated in the TweetEval benchmark, including RoB-RT (Barbieri et al., 2020), BERTweet (Nguyen et al., 2020), RoBERTa + Enriched (Liu et al., 2023c), MoLE - DANN (Hardalov et al., 2021), PNEM (Siddiqua et al., 2019), and TimeLMs (Loureiro et al., 2022). Among these systems, the TimeLMs system achieved the highest Macro F1 score, 72.9, demonstrating the importance of taking into account the diachronic nature of social media. Table 2 illustrates that GPT-3.5 and GPT-4 underperform nearly all the fine-tuned systems when performing TR-ZSSD with direct prompting, and only come close to fine-tuned systems when utilizing CoT prompting, demonstrating the importance of different prompting strategies. However, significant performance gains are achieved when utilizing ToC prompting for TR-ZSSD, yielding competitive zero-shot performance for GPT-3.5 and surpassing fine-tuned baselines with GPT-4. ToC prompting with GPT-4 was able to achieve a Macro F1 score
of 77.1, gaining over 4 points of F1 score over TimeLMs, the best fine-tuned system.
CoVaxFrames Results: We considered multiple baseline systems utilized by prior work on COVAXFRAMES, including SBERT-NLI (Hossain et al., 2020), DS-BERT (Weinzierl et al., 2021), LES-GAT (Weinzierl et al., 2021), LESGAT-MF (Weinzierl and Harabagiu, 2022b), and LACRScore (Weinzierl and Harabagiu, 2022a). The LACRScore system achieved the highest Macro F1 score, 76.2, of the fine-tuned systems on COVAXFRAMES due to the benefits of considering attitude consistency. Table 3 illustrates that GPT-3.5 and GPT-4 similarly underperform nearly all the fine-tuned systems when performing TR-ZSSD with direct prompting, and again, come close to the fine-tuned systems when utilizing CoT prompting. However, ToC prompting again yields massive F1 score increases, with GPT-4 yet again surpassing the fine-tuned baselines. GPT-4 with ToC prompting was able to achieve a Macro F1 score of 79.1, gaining nearly 3 points of F1 score over the LACRScore system with zero training examples from COVAXFRAMES.
MMVax-Stance Results: For our multimodal TR-ZSSD experiments on MMVAX-STANCE, two LMMs were considered: LLaVA-1.5 (Liu et al., 2023b,a), and GPT-4V (OpenAI, 2024). LLaVA connects a vision encoder and an LLM to form a large multimodal model designed for generalpurpose visual and language understanding, with additional instruction tuning from GPT-4. LLaVA1.5 expands upon the LLaVA architecture by incorporating a Multilayer Perceptron (MLP) as a vision-language connector and adding academictask-oriented Visual Question Answering (VQA) data to the pre-training tasks. GPT-4V has been shown to demonstrate a promising ability to align with human judgments across numerous multimodal tasks (Zhang et al., 2023b), with GPT-4V scoring top of the leaderboards for Cognition subtasks on the MME leaderboards (Fu et al., 2023). Two baseline prompting methods were employed with LLaVA-1.5 and GPT-4V: Direct prompting and CoT prompting.
DS-BERT, LES-GAT-MF, and LACRscore systems, employed previously on COVAXFRAMES, were utilized as baseline fine-tuned systems on MMVAX-STANCE, along with image captioning performed by BLIP-2 (Li et al., 2023b), as per Weinzierl and Harabagiu (2023). Additionally, CLIP-Joint utilized the text and the image en-
coder of CLIP (Radford et al., 2021) to perform stance classification, while VILT (Kim et al., 2021), FLAVA (Singh et al., 2022), and BridgeTower (BT) (Xu et al., 2023) were also fine-tuned for stance classification. Additionally, BT + SAll (Weinzierl and Harabagiu, 2023) was included, a BridgeTower system employing significant training data augmentation. The BT system achieved a Macro F1 score of 52.9 on MMVAX-STANCE, struggling to achieve performance parity with text-based stance detection systems due to the increased difficulty of multimodal stance detection. Table 4 illustrates that LLaVA-1.5 and GPT-4V underperform the finetuned systems in TR-ZSSD with direct prompting, and become very competitive with the fine-tuned systems with CoT prompting. Furthermore, ToC prompting again yields massive F1 score gains, with ToC prompting with GPT-4V achieving a Macro F1 score of 60.6, an increase of nearly 8 points of F1 score over BT with no multimodal training examples from MMVAX-STANCE. However, GPT-4V with ToC prompting failed to outperform the best fine-tuned system, which employed massive additional synthetic data. BT + SAll achieved a Macro F1 score of 71.3 with 46,606 additional synthetic multimodal stance examples upon which the system was trained - representing neatly a 600% increase in training data (Weinzierl and Harabagiu, 2023).

heading: 4.3 Discussion
text: Achieving a Macro F1 score of 77.1 on SemEval2016, 79.1 on COVAXFRAMES, and 60.6 on MMVAX-STANCE in TR-ZSSD demonstrate that ToC-TR-ZSSD performs surprisingly well, as it outperforms fine-tuned supervised systems on textonly stance detection and achieves competitive performance for stance detection from multimedia SMPs. We believe that this is in large part due to ToC prompting. Similar F1 scores were unattainable with zero training data only a few years ago, with systems typically requiring at least hundreds, if not thousands, of training examples. Furthermore, accessing the reasoning capabilities of LLMs with improved prompting methodologies continued to yield improvements in stance detection, with ToC prompting generating the most impressive results. We hypothesize that the superiority of ToC prompting, compared to CoT prompting, is due to the fact that CoT prompting requires LLMs to immediately commit to a rationale that supports the final answer, which provides little space for
LLMs to consider all possible alternative hypotheses. Improvements to CoT prompting have been proposed to alleviate this issue, such as Self Consistency with CoT (CoT-SC) prompting (Wang et al., 2023), which samples multiple Chains-of-Thought from an LLM and considers the majority voted final answer from all chains, and Tree-of-Thought (ToT) prompting (Yao et al., 2023), which maintains a tree of thoughts by sampling many thought continuations from an LLM and evaluating early whether each of those should continue or end. However, fundamentally these CoT-based approaches provide little space for the LLM to explore alternative hypotheses, as each chain or tree path must still commit early to an accurate final answer. Furthermore, sampling from the next-word prediction probabilities produced by LLMs is unlikely to yield varied rationales and conclusions, which greatly limits the considered hypothesis space. This highlights a general problem with sampling from LLMs, which can result in bias in the generated text that tends to lack diversity when generating text from underrepresented groups (Lahoti et al., 2023). ToC prompting alleviates these concerns of hypotheses exploration by explicitly considering each of the counterfactual stance values.
Exploring examples of counterfactual rationales generated by GPT-4 provides insights into why ToC prompting improves upon baseline prompting methods. ToC prompting consists of first performing CoE prompting with each counterfactual stance value, and then finally performing C-CoCV prompting to determine a final stance value. Figure 3 illustrates an example from the SemEval-2016 Task 6 A test set, employing CoE prompting with GPT-4
with a counterfactual stance value of Accept. We find that GPT-4 generates an interesting analysis of how the SMP could hold an Accept stance towards the topic of “Climate Change”, possibly due to the ambiguity and context in which the SMP was made. Figure 4 illustrates the same SMP, but a different counterfactual stance value of Reject towards “Climate Change”. GPT-4 generates a convincing rationale, with how the language of the SMP implies a “...rejection of the mainstream understanding and urgency of climate change.” Finally, Figure 5 illustrates the same SMP, but with the final counterfactual stance value of No Stance towards “Climate Change”. Herein, GPT-4 rejects the No Stance value entirely, only being able to find evidence to the contrary. This ToC prompting example ends with C-CoCV prompting on this same SMP, with the response by GPT-4 provided in Figure 6. GPT-4 compares and contrasts the counterfactual hypotheses and evidence gathered, and comes to the ultimate conclusion that the stance of the SMP towards “Climate Change” is Reject. This conclusion aligns with the annotated stance value of Reject in the SemEval-2016 test set. An additional exploration of counterfactual rationales that considers diverse hypotheses is provided in Appendix C.
The ToC results when prompting GPT-4V on MMVAX-STANCE demonstrate significant advancement in multimodal reasoning. Multimodal stance detection is extremely challenging, demonstrated by the performance of the fine-tuned baseline systems, as it requires the integration of textual and visual cues to accurately interpret nuanced and often sarcastic or ironic expressions - an extremely
challenging linguistic phenomenon, as reported in Joshi et al. (2017). This complexity demands advanced computational techniques capable of understanding not only the literal content but also the contextual and cultural nuances that influence interpretation. We believe that the results of using ToC prompting with GPT-4V clearly demonstrate that LMMs are capable of counterfactual reasoning, probably capturing cultural knowledge and handling well sarcastic multimodal content.
A quantitative and qualitative analysis of the mistakes made with ToC prompting was performed by researchers at the University of Texas at Dallas. On SemEval-2016, 412 incorrect stance values were predicted for the test set by GPT-4. A majority of mistakes (76%) were made on SMPs labeled with the Reject stance, with a majority of these examples (74%) being incorrectly identified as No Stance by GPT-4. Similarly, of the mistakes made on SMPs labeled with an Accept stance, 81% were misidentified as having No Stance by GPT-4. Furthermore, GPT-4 outright refused to process 1% of the SMPs from SemEval-2016 due to the controversial nature of the SMPs. These SMPs were assigned a default stance value of No Stance for evaluation, but this assignment does not account for the significant number of false negatives produced by GPT-4.
An analysis of the 723 mistakes by GPT-4 made on the test set from COVAXFRAMES reveals a similar pattern: A majority of mistakes (68%) were made on SMPs labeled with the Accept stance, with a majority of these examples (61%) being incorrectly identified as No Stance by GPT-4. Of the mistakes made on SMPs labeled with a Reject stance, 81% were again misidentified as having No Stance by GPT-4. Additionally, GPT-4 outright refused to process 1% of the SMPs from COVAXFRAMES due to the controversial nature of the SMPs. Analyzing the 599 mistakes made by GPT-4V on the test set from MMVAX-STANCE emphasized the challenge of utilizing OpenAI models for sensitive tasks: GPT-4V outright refused to process 2% of the SMPs from MMVAX-STANCE, primarily due to the controversial nature of the images and memes utilized by these SMPs. A majority of mistakes (50%) were made on SMPs labeled with the Accept stance, with a majority of these examples (69%) being incorrectly identified as No Stance by GPT-4V. Inspecting the SMPs labeled with the Reject stance, 84% of the mistakes were again misidentified as having No Stance by GPT-4V.
Manually inspecting the counterfactual ratio-
nales and verification rationales elucidated why many examples were misidentified as having No Stance: GPT-4 & GPT-4V were extremely conservative in assigning Accept or Reject stance values to SMPs discussing sensitive topics such as “Abortion”, or sensitive FoCs such as “The COVID-19 Vaccine is a satanic plan to microchip people.” Mitigating this conservative bias in favor of improving stance detection performance could yield significant improvements in TR-ZSSD. However, this bias is likely present due to the safety tuning performed by OpenAI, and therefore caution should be taken when attempting to bypass these safeguards.
The ToC-TR-ZSSD results support the recommendation that additional research should focus on more advanced prompting methods to get the most out of knowledge available in LLMs and to unleash their reasoning capabilities. Significant attention is currently being paid to in-context learning (Dong et al., 2023) and selecting good demonstrations for few-shot learning (Brown et al., 2020; Zhao et al., 2021). However, we believe the results from this paper demonstrate that prompting methods that creatively exploit the problem domain can yield similar performance improvements with LLMs, without requiring demonstrations.

heading: 5 Conclusion
text: In this paper we introduce the task of Tabula Rasa Zero-Shot Stance Detection (TR-ZSSD). We demonstrate that Large Language Models (LLMs) and Large Multimodal Models (LMMs) are capable of performing high-quality TR-ZSSD with Tree-ofCounterfactual (ToC) prompting, a novel prompting methodology relying on counterfactual reasoning. Stance detection experiments were performed over several varied stance datasets, and numerous strong baseline systems were considered for comparison. ToC prompting with LLMs, such as GPT4, outperformed fine-tuned baselines and alternative prompting methodologies for text-based TRZSSD. Furthermore, ToC prompting with LMMs, such as GPT-4V, demonstrated competitive performance with fine-tuned baselines for multimodal TR-ZSSD. Our ToC-TR-ZSSD methodology required zero labeled examples, successfully generalized across different objects of stance (i.e. topics or frames of communication), improved performance across both text-only and multimodal stance detection, and generated explanatory rationales, enabling LLM and LMM interpretability.

heading: 6 Limitations
text: The method introduced in this work was tested only on SMPs from Twitter / X. Our approach may not work as well on SMPs originating from other social media platforms, particularly platforms such as Reddit, where longer textual content is typical. Furthermore, our method relies on the textual content of SMPs along with included images. Many SMPs use multiple images, GIFs, videos, and other multimedia content that our current method would not utilize. In future work, we plan to extend our methods by enabling them to operate across more modalities and longer-form content on additional social media platforms.
An important limitation of our approach derives from the need to have Large Language Models (LLMs) or Large Multimodal Models (LMMs) infused with significant cultural and moral knowledge. While current LLMs demonstrate strong cultural and moral understanding in majority Englishspeaking countries, LLMs have been found to be less accurate in predicting empirical moral norms and cultural knowledge across non-majority English-speaking different countries (Ramezani and Xu, 2023). Furthermore, while a limited form of reasoning appears as an emergent ability in LLMs at scale, it remains unclear to what extent LLMs are truly capable of reasoning (Huang and Chang, 2023). In future work, we plan to measure and address these biases by discovering and employing more diverse Frames of Communication on a wide swath of social media platforms.

heading: 7 Ethical Statement
text: We respected the privacy and honored the confidentiality of the users that have produced the posts in SemEval-2016, COVAXFRAMES, and MMVAXSTANCE. We received approval from the Institutional Review Board at the University of Texas at Dallas for working with these Twitter social media datasets. IRB-21-515 stipulated that our research met the criteria for exemption #8(iii) of the Chapter 45 of Federal Regulations Part 46.101.(b). Experiments adhered to rigorous professional standards, with test collection evaluations deferred until a final method was chosen based on training and validation performance. All experimental settings, configurations, and procedures were clearly laid out in this work, the supplemental material, and the linked GitHub repository. We believe our research carries minimal risks, given its purpose to
enhance comprehension of the adoption or rejection of COVID-19 vaccine hesitancy frames on social media. The central focus of the conducted research was the public good, aiming to benefit both natural language processing and public health research.

heading: A Prompting Hyperparameters
text: All prompting experiments employed through the OpenAI API utilized the same hyperparameters: 1024 max generated tokens, a temperature of 1.0, and 0.7 top-p sampling. GPT-4 utilized the gpt-41106-preview model, while GPT-3.5 employed the gpt-3.5-turbo-1106 model. GPT-4V made use of the gpt-4-vision-preview model, while LLaVA-1.5 employed the official LLaVA GitHub code3, with minor modifications. LLaVA hyperparameters also mimic those utilized with the OpenAI models, with max generated tokens of 1024, a temperature of 1.0, and 0.7 top-p sampling. Complete details are provided in the linked GitHub repository, including configuration files with all hyperparameters utilized and prompts employed.

heading: B Data Contamination Discussion
text: As ToC-TR-ZSSD is an entirely zero-shot method, it is paramount that examples from the test collections of SemEval-2016 Task 6 A, COVAXFRAMES, and MMVAX-STANCE were not included in the training of LLMs, such as GPT-3.5, GPT-4, and LLaVA-1.5. GPT-3.5 and GPT-4 may have seen some parts of SemEval-2016 Task 6 A, as we are not privy to the secret datasets infused into these methods by OpenAI. However, we believe this is unlikely, due to the low performance of direct prompting for both models on this dataset. Regardless, we believe the experiments on SemEval-2016 Task 6 A still demonstrate that ToC prompting improves upon other zero-shot prompting methods for this dataset.
COVAXFRAMES was published in late May, 2022, while MMVAX-STANCE was published in early October, 2023. Both datasets do not have publicly available raw tweets at the time of publication to Twitter / X Developer TOS / IRB concerns. Access to the raw data requires utilizing the Twitter / X API to “hydrate" these tweet IDs. GPT-3.5 is stated to have a knowledge cutoff of September 20214, and therefore cannot have access to COVAXFRAMES or MMVAX-STANCE. GPT-4 is stated to have a knowledge cutoff of April 20235, and therefore cannot have access to MMVAX-STANCE, and is unlikely to have access to the raw Twitter
3https://github.com/haotian-liu/LLaVA 4https://platform.openai.com/docs/models/ gpt-3-5-turbo 5https://platform.openai.com/docs/models/ gpt-4-and-gpt-4-turbo

heading: 1. "Alarmists" - This term is often used to describe individuals or groups who are concerned about the dangers of climate change and are vocal about the urgent need for action. The use of the term "alarmists" can sometimes carry a negative connotation, implying that the concern is exaggerated or unwarranted.

heading: 2. "Deniers" - In the context of climate change, "deniers" refers to individuals or groups who reject the scientific consensus that climate change is happening and is primarily caused by human activities. The term "denier" is used to suggest a refusal to accept established facts or evidence.

heading: In summary, the tweet seems to have a dismissive stance towards those who are vocal about the dangers of climate change, referring to them as "alarmists" and ironically labeling them as "deniers." This suggests that the tweeter may not accept the mainstream view of climate change as a serious and urgent issue, though the tweet is somewhat ambiguous and could

heading: 1. The term "alarmists" is a key indicator of the tweet's stance. In the context of climate change, "alarmists" typically refers to individuals or groups who are concerned about the effects of climate change and advocate for immediate action to mitigate its impact. By using this term, the tweeter is implying a negative view of those who are concerned about climate change, suggesting that their alarm is unwarranted or exaggerated.

heading: 2. The use of the word "actual" serves to emphasize the tweeter's belief that the so-called "alarmists" are, in reality, the ones who are in denial. This implies that the tweeter believes that the concerns of climate change advocates are based on denial of some other truth or fact.

heading: 3. The term "deniers" is typically used to describe individuals who reject the scientific consensus that climate change is real, human-caused, and a serious threat. By placing the word "deniers" in quotation marks, the tweeter is using irony to suggest that while climate change advocates label others as deniers, they themselves are the true deniers.

heading: 4. The absence of any direct statement supporting the reality of climate change or the need for action suggests that the tweeter does not align with the position of those advocating for recognition and action on climate change.

heading: 5. The tweet is a response to another user, indicated by the "@user" at the beginning. Without additional context, it is not possible to know what the original statement was, but the response suggests disagreement or criticism of a viewpoint that may have expressed concern about climate change.
text: / X data for COVAXFRAMES. Finally, LLaVA1.5 is stated to have been trained on: 558K filtered image-text pairs from LAION/CC/SBU, captioned by BLIP, 158K GPT-generated multimodal instruction-following data, 450K academic-taskoriented VQA data mixture, and 40K ShareGPT data6. Therefore, we believe it is highly unlikely that LLaVA-1.5 contains any of the data from these three datasets.
6https://huggingface.co/liuhaotian/llava-v1. 5-7b

heading: C Examples of Counterfactual Rationales
text: Figure 7 illustrates an example from the COVAXFRAMES test set, employing CoE prompting with GPT-4 with a counterfactual stance value of Accept. GPT-4 thoroughly outlines the concerns of the SMP relating to the FoC under question: “Homeopathic / Oriental medicine / alternative medicine / naturopath may Be Alternative to COVID-19 Vaccine.” A strong argument is made that the SMP is in support of alternative medicine options when compared to the COVID-19 vaccines, by “... acknowledging and defending alternative medicine as a legitimate option...” Upon considera-

heading: 1. **Use of the term "alarmists":** The word "alarmists" is typically used to describe people who are considered to be exaggerating a danger and thus causing unnecessary worry or panic. In the context of climate change, "alarmists" often refers to individuals who are very concerned about climate change and advocate for immediate action to address it. The use of this term usually indicates a critical view of those individuals and suggests that their concerns are overblown.

heading: 2. **The concept of "deniers":** Within the context of climate change, "deniers" are individuals who deny the scientific consensus that climate change is occurring and/or is primarily caused by human activities. Labeling someone a "denier" in this context is to suggest that they are ignoring scientific evidence.

heading: 3. **The word "ironic":** The tweet claims it is ironic that the "alarmists" are the actual "deniers." Irony involves a contrast between expectations and reality. This suggests that the expected situation would be for those who are alarmed by climate change to be the ones accepting the reality of it, but the tweet implies the opposite—that they are in denial.

heading: 4. **Quotation marks around "deniers":** The use of quotation marks can imply skepticism or suggest that the word is not being used in its standard sense. In this case, it seems to imply that the term "deniers" is being misapplied to those concerned about climate change.

heading: 5. **Contextual interpretation:** The tweet is a response to another user, which could imply a dialogue or debate about climate change. The use of sarcasm or irony in this response indicates a critical view of the position held by the "alarmists."

heading: In summary, the tweet exhibits a stance through the use of language that is critical of individuals who are concerned about climate change, by labeling them as "alarmists" and
text: tion of the Reject counterfactual, Figure 8 demonstrates that GPT-4 possesses the capability to reason counterfactually, recognizing that “...the SMP’s author is advocating for a more inclusive and less biased consideration of different medical traditions rather than outright stating that these traditions should replace the vaccine.” This hypothesis would be extremely unlikely to arise through direct or CoT prompting approaches, as it requires one to assume that the post has a Reject stance towards the “homeopathic alternatives to the COVID-19 vaccine” FoC to make such nuanced distinctions between “advocating” for homeopathic alternatives versus seeking to “replace the vaccine.” Figure 9 further explores the No Stance counterfactual, demonstrating again a nuanced consideration of the SMP concerning the FoC. Finally, Figure 10 demonstrates how GPT-4 is able to take each of these nuanced hypotheses and weigh the evidence. GPT-4 assesses that the Accept rationale is “...strong because it highlights the SMP’s alignment with the frame’s suggestion...”, while the Reject rationale “...seems weaker” and the No Stance rationale “...fails to recognize the implicit stance the SMP takes by defending alter-

heading: 1. **Identification of Alternatives to Vaccines**: The tweet acknowledges the existence of traditional oriental medicine, alternative medicine, and naturopathy as potential alternatives to vaccines. This already aligns with the frame that presents these methods as possible substitutes for the COVID-19 vaccine.

heading: 2. **Defense of Alternative Medicine**: The tweet defends these alternative methods against the critique of being "unscientific" or "occult/New Age." This defense implies a belief in the legitimacy or efficacy of these methods, which is congruent with the frame that they could be viable alternatives to vaccination.

heading: 3. **Concern Over Prejudice**: The author of the tweet is concerned about the prejudice against these alternative methods. This suggests that they believe these methods are not being given fair consideration, which again aligns with the frame that proposes these methods as alternatives.

heading: 4. **Trust in Institutions**: The tweet mentions that the author will "completely lose trust" if schools continue to discourage alternative methods and promote vaccines in a way that the author perceives as prejudiced. This indicates that the author's trust is contingent on the acceptance and fair treatment of alternative medicine, further supporting the frame that these could be alternatives to vaccines.

heading: 5. **Implicit Critique of Mainstream Medicine**: By opposing the characterization of alternative medicine as "unscientific," the tweet implicitly critiques the mainstream medical establishment's approach to health, which often prioritizes vaccines and other pharmaceutical interventions. This critique is a common theme in communications that are supportive of the frame suggesting alternative treatments to vaccines.

heading: 1. **Identifying the Frame Elements**: The frame suggests that there are alternative methods, such as homeopathic, oriental medicine, alternative medicine, and naturopathy, that could be considered in place of the COVID-19 vaccine. The framing here is one that presents these alternatives as potentially valid options for dealing with COVID-19.

heading: 2. **Tweet's Position on Traditional and Alternative Medicine**: The author of the tweet is advocating for a respectful consideration of traditional and alternative medicine. They express a concern about the dismissal of these practices by educational institutions, which are pushing for vaccination and marginalizing other forms of medicine by categorizing them as unscientific or occult.

heading: 3. **Conflict with the Frame**: The tweet's stance conflicts with the frame because the tweet does not suggest that alternative medicines should be seen as replacements for the COVID-19 vaccine. Instead, the tweet is critical of the bias against these traditional and alternative medical practices. The tweet does not explicitly endorse these alternatives as substitutes for the vaccine; it is more focused on the issue of respect and open-mindedness towards different medical traditions.

heading: 4. **Implication of the Stance**: By stating that the tweet's stance is "Reject," it implies that the author of the tweet does not agree with the frame that presents alternative medicines as possible substitutes for the COVID-19 vaccine. However, the tweet itself does not provide a clear position on whether these alternatives should be considered as replacements. It is more concerned with the prejudiced attitude towards them.

heading: 5. **Understanding the Tweet in Context**: The tweet could be seen as rejecting the frame not because it disagrees with the potential efficacy of alternative medicines but because it disagrees with the dichotomy being presented—that one must choose between the vaccine and alternative medicine. The author seems to be calling for a more inclusive approach that does not automatically dismiss non-Western medical practices.

heading: In conclusion, the tweet rejects the frame, but not necessarily because it opposes the use of alternative medicine as a potential alternative to the COVID-19 vaccine. Instead, the rejection is based on the tweet's defense of traditional and alternative medicine's legitimacy and its criticism of the prejudice shown by schools in favor of the vaccine and against these other practices. The tweet's author is advocating for a more inclusive and less biased consideration of different medical traditions rather than outright stating that these traditions
text: native medicines.” Ultimately, GPT-4 judges correctly that the stance of this SMP is Accept towards the FoC under consideration, aligning again with the human judgments present in COVAXFRAMES.
Figure 11 illustrates a multimodal SMP from the test set of MMVAX-STANCE. The SMP is short, stating in text only “The covid19 vaccine story”
and providing a meme of Jim Halpert, a character from The Office. In this meme, Jim points to a flip chart, and in the first section, the text on the flip chart says, “The COVID vaccine should be tested on politicians and bureaucrats.” In the second section, where Jim is smiling, the flip chart says, “If they survive, the vaccine is safe. If they don’t

heading: 1. The tweet mentions that schools are encouraging students to get a vaccine.

heading: 2. It then states that schools discourage going deeper into traditional oriental medicine/alternative medicine/naturopathy.

heading: 3. The tweet criticizes the schools for labeling these alternative practices as "unscientific folk medicine or occult/New Age" and suggests that this is done with prejudice.

heading: 4. The author of the tweet indicates that if this situation continues, they will lose trust.

heading: 1. The tweet is critical of the dismissal of alternative medicines by schools.

heading: 2. It shows concern that these alternative practices are being marginalized and labeled negatively.

heading: 3. The implication is that the author values these alternative medicines and is against the idea that they are unscientific or prejudiced.

heading: 4. The tweet does not explicitly state that alternative medicines should be considered as substitutes for the COVID-19 vaccine, but the defensive tone towards these practices suggests a belief that they should not be dismissed outright and may have some merit.

heading: In summary, while the tweet does not directly state a position on whether alternative medicines should replace the COVID-19 vaccine, its defensive tone towards alternative medicines and criticism of their dismissal suggests an implicit stance that is more aligned with the frame of vaccine hesitancy, which includes considering alternative medicines as potential
text: survive, our country is safe.” The FoC under consideration as the object of stance is “The Federal government lied about the COVID-19 vaccines to reduce the information about effective treatments.” Figure 11 also includes the response generated by GPT-4V with CoE prompting with a counterfactual stance value of Accept. GPT-4V describes the content of the meme, and continues to provide a 3-point analysis of how the meme and the text contribute to the counterfactual stance value of Accept, recognizing that “...the SMP indirectly supports the frame...” However, Figure 12 illustrates how a counterfactual stance value of Reject can shift the perspective on this SMP. GPT-4V identifies that the SMP is satirical and humorous in nature, concluding that the SMP “...does not directly support the idea that the government lied about the vaccines...” A consideration of the No Stance counterfactual is provided in Figure 13, yielding a similar rationale to that of the Reject counterfactual: that “...the

heading: 1. Suggesting that politicians and bureaucrats should be the test subjects for the COVID-19 vaccine implies a lack of trust in the safety or efficacy of the vaccine. It insinuates that the vaccine might not be safe and thus should be tested on those who are deemed expendable or less trustworthy by the author of the tweet.

heading: 2. The conditional statements about survival indicate a cynical view of the political and bureaucratic class, implying that their well-being is not the primary concern and that there might be a positive outcome for the country regardless of the vaccine's safety.

heading: 3. The overall tone of the image is sarcastic and critical, which aligns with the provided vaccine hesitancy frame that suggests the Federal government might have lied about the vaccines. The image seems to question the honesty of the information disseminated by the government and those in power.

heading: By using humor and sarcasm, the tweet indirectly supports the frame that there is a reason to doubt the information provided by the government about COVID-19 vaccines, which could include lying about the vaccines to downplay other effective treatments. The acceptance stance is therefore reinforced by the image's insinuation that there is a lack of transparency
text: meme does not engage with the specific claim that the government lied about the vaccines...” and con-
cludes that “...it does not affirm or deny the frame, but rather exists independently as a piece of satirical commentary.” Figure 14 provides the response produced by GPT-4V with C-CoCV prompting on the same SMP, with all counterfactual rationales included. GPT-4V compares and contrasts each of the counterfactual hypotheses and evidence present, and concludes that “...the meme’s satirical take on vaccine testing on politicians does not directly address the claim of government dishonesty about the efficacy of vaccines...” and that “...the stance of the SMP towards the provided vaccine hesitancy frame is best described as: No Stance.” Again, ToC prompting yields an aligned response to the human judgments present in MMVAX-STANCE, as this post is annotated as having No Stance towards the FoC under consideration.

heading: Future Work
text: In future work, we plan to extend our methods by enabling them to operate across more modalities and longer-form content on additional social media platforms. An important limitation of our approach derives from the need to have Large Language Models (LLMs) or Large Multimodal Models (LMMs) infused with significant cultural and moral knowledge. While current LLMs demonstrate strong cultural and moral understanding in majority Englishspeaking countries, LLMs have been found to be less accurate in predicting empirical moral norms and cultural knowledge across non-majority English-speaking different countries (Ramezani and Xu, 2023). Furthermore, while a limited form of reasoning appears as an emergent ability in LLMs at scale, it remains unclear to what extent LLMs are truly capable of reasoning (Huang and Chang, 2023). In future work, we plan to measure and address these biases by discovering and employing more diverse Frames of Communication on a wide swath of social media platforms