Chain-of-Thought Embeddings for Stance Detection on Social Media
Authors: Joseph Gatto
Omar Sharif
Sarah Masud Preum
Venue: EMNLP 2023 Findings (2023)
Keywords: chain of thought
prompting
chatgpt
stance detection
TL;DR: Motivating Chain-of-Thought outputs from ChatGPT as an additional input to stance detection classifiers.

Abstract: Stance detection on social media is challenging for Large Language Models (LLMs), as emerging slang and colloquial language in online conversations often contain deeply implicit stance labels. Chain-of-Thought (COT) prompting has recently been shown to improve performance on stance detection tasks --- alleviating some of these issues. However, COT prompting still struggles with implicit stance identification. This challenge arises because many samples are initially challenging to comprehend before a model becomes familiar with the slang and evolving knowledge related to different topics, all of which need to be acquired through the training data. In this study, we address this problem by introducing COT Embeddings which improve COT performance on stance detection tasks by embedding COT reasonings and integrating them into a traditional RoBERTa-based stance detection pipeline. Our analysis demonstrates that 1) text encoders can leverage COT reasonings with minor errors or hallucinations that would otherwise distort the COT output label. 2) Text encoders can overlook misleading COT reasoning when a sample's prediction heavily depends on domain-specific patterns. Our model achieves SOTA performance on multiple stance detection datasets collected from social media.

Chain-of-Thought Embeddings for Stance Detection on Social Media
Joseph Gatto, Omar Sharif, Sarah M. Preum Department of Computer Science, Dartmouth College
{joseph. m. www.aatt...a.. gr, omar omar..harif.. swarif.gr, sarah. sarah.masud. preum) @ dartmouth. edu
Abstract
Tweet want proof voter fraud really exists? plenty of evidence.
Chain-af-Thought
Without further context, it difficult to determine the stance of the towards Donald Trump. Therefore, the
Stance detection on social media is challeng- ing for Large Language Models (LLMs), as emerging slang and colloquial language in on- line conversations often contain deeply im- (COT)
CheiGPT
Sinnee towards Trump?
stance is [NEUTRAL] implicii
Tweet
plicit stance labels. Chain-of-Thouu prompting has recently been shown to im- prove performance on stance detection tasks alleviating some of these issues. However, COT prompting still struggles with implicit stance identification. This challenge arises be- cause many samples are initially challenging to comprehend before a model becomes famil- iar with the slang and evolving knowledge re- lated to different topics, all of which need to acquired through the training data. this study, address this problem by introduc- ing COT Embeddings which improve COT performance on stance detection tasks by em- bedding COT reasonings and integrating them into a traditional RoBERTa- based stance detec- tion pipeline. Our analysis demonstrates that 1) text encoders can leverage COT reasonings with minor errors or hallucinations that would otherwise distort the COT output label. 2) Text encoders can overlook misleading COT rea- soning when a sample's prediction heavily de- pends Nomain-speciffe patterns. Our model achieves SOTA performance on multiple stance detection datasets collected from social media.
love seeing barbarn jordan, the first black waman (and first give keynate speech at the dnc. #demsonvention
Chain-af-Theergee Thoughh the stance the tweet [IN
ChanaPT
FAVOR] of Barbara Jordan and the Democratic National Convention
Stance towards
Stance Hallocination
made by Chain-of-Thouggh reasoning models. Implicit Stance Confusion refers to LLMs inability to understand the implicit reference to the stance topic. In the example above, ChatGPT should have predicted that the tweet is [IN FAVOR] of Trump. In this context, Stance Label Hallucination refers the scenario where LLMs wrong point. In this example, the reasoning is correct, but ChatGPT used the [IN FAVOR] label towards the wrong topic.
Figure I: Common
the label space argue the
is difficult for LLMs with knowledge cut-offs to un- derstand and may only be discernible by observing similarly labeled samples in the training set.
One way to solve this problem by employing models with extensive world knowledge. For exam- ple, recent works have shown that using ChatGPT on Stance Detection can provide significant perfor- mance increases (Zhang et al., 2023a,b). Unfor- tunately, LLMs (c.g., ChatGPT, Llama) still have many issues understanding complex stance rela- tionships from Twitter data. this study, we high- light two issues with the ofate-of-the-an Chain- of- ohoThh (COT) prompting approach to stance detection. 1) Implicir Stance Confusion: As shown in Figure 1, LLMs continue to struggle with un- derstanding implicit tweet stance, even when em- ploying advanced prompting strategies like COT reasoning (Wei et al., 2023). 2) Stance Label Hallu- cination: L.LMs are prone to hallucinations, which cause them to output sound reasonings, but for the wrong stance topic (see Figure example). Even
1 Introduction
Detecting the stance of a text with respect cer- tain topic vital to many NLP tasks (Hardalov et al., 2022). Detecting stances on social media platforms like Twitter poses unique challenges, as emerging knowledge and colloquial language pat- terns can make difficult to detect stances without additional context. For example, consider the top tweet shown in Figure 1, This tweet contains no direct mention of Donald Trump, and is thus diffi- cult to classify without further context - such as how Trump supporters on Twitter widely supported voter fraud propaganda. Such emerging knowledge

---

when LLMs analyze the correct topic, they are also prone using the provided label space incorrectly, producing accurate but ill-structured outputs. In this study, we mitigate these two problems by introducing Thoouhtf (COT) Em- beddings. Our approach feeds the COT reasoning text to transformer encoder to be used as an additional feature in traditional stance detection pipeline. The intuition behind this approach is fhree-fold: (i) Text encoders are robust to stance label hallucinations if the COT reasoning is correct. This can make incorrect COT predictions useful in a text classification pipeline. (ii) Text encoders can choose to ignore certain signals as needed. Thus, when sample is too implicit to be understood by LLMs, the model may choose to focus on how similar tweets were classified. (iti) COT reasonings can inject world knowledge into text encoder. That is, COT texts often contain reasonings and justifications grounded in world knowledge not available in the tweet. We find that, by using this approach, we can achieve state-of- the-art results on multiple stance detection datasets.
Presidential- Stance
Tweet-Stance
IIC
FM
LA
AT
CC
BD
TR
Train
620
597
587
461
355
875
875
Dev Test
69
67
66
52
40
295
285
280
220
169
375
375
Class-wise distribution of topics
Neutral Against
256
170
167
145
203
487
565
511
544
464
26
385
499
Favor
163
268
222
124
335
378
341
Total
984 949
933
733 564
1250
1250
Table 1:
Topic-wise (e.g.. HC, FM. TR) distribu- tion of train, development, test, and classes of the Sweet-Stance and Fresidential-Stanee datasets. The Presidential-Stance dataset does not have a develop- ment
Liu et al., 2021). This work is motivated by knowl- edge infusion but substantially differs from existing works. To the best of our knowledge, while some prior work has used prompting for stance detec- tion (Zhang al., 2023b), no work has attempted to use LLMs as a knowledge base for improved stance detection. While we also do not explicitly explore LLMs as a knowledge extraction tool, we do find that our method has the capacity to inject world knowledge into inference pipeline due to the nature of COT text generation.
A summary of our contributions is as follows:
1. the best of our knowledge, this the first investigation into the embedding of COT rea- sonings. Our approach achieves state-of-the- art results on two stance detection datasets: Tweet-Stance (Mohammad et al., 2016; Barbi- eri et al., 2020) and Presidential-Stanee (Kaw- intiranon and Singh, 2021).
LLMs for Stance Detection Recently. few works have used ChatGPT for stance detection di- rectly Zhang et al. (2023a,b). In (Zhang al., 2023b), the authors achieve superior performance on several stance detection datasets by prompting ChatGPT to do Chain-of-Thouge inference. this study, we use a similar prompting strategy to perform stance detection, but show the benefits of embedding these COT reasoning texts and using them feature in a stance detection pipeline.
2. Our error analysis COT reasoning high- lights two key flaws on stance detection tasks: Implicit Stance Confusion and Stance La- bel Hallucinations. Our approach, Chain-of- Thought Embeddings, makes COT outputs more robust to these two issues.
Methods
2 Related Work
We employ 1-shot COT prompt for each tweet in each dataset, aiming to determine the stance of the tweet relation to specific topic'. We specif- ically ask the models to provide a COT reasoning and to include its predicted label in brackets (e.g. [NEUTRAL] for neutral tweet), so the output may be parsed and converted to numeric repre- sentation. An example tweet and corresponding
Stance Detection: This task is a well-explored research problem, where early studies employed various machine learning and deep learning tech- niques (Hardalov et al., 2022). The emergence of large language models has further pushed the state- of-the-art performance on many stance detection datasets (Li and Caragea, 2021). Many stance de- tection problems require domain-specific solutions with models which explicitly inject world knowl- edge into stance detection systems (He et al., 2022;
Pleasee refer to Appendix B for details on the prompts used for cach task

---

Favor
Against
Neutral
(LA), Atheism (AT), and Climate Change (CC). The Presidential-Stanc dataset contains two sub- tasks focusing on the 2020 election cycle, with annotation for stance towards presidential candi- dates Joe Biden (BD) and Donald Trump (TR). The topic-wise and class-wise distribution and statistics for the training, development, and test sets of both datasets are presented in Table 1 and Table 2, re- spectively. The class-wise distribution indicates that both datasets are skewed towards the against class.
Tweet-Stance
Train
678
1254
688
Dev
75
141
78
Test
304
715
230
Presidential-Stane Sectonniii--tttee---eee SttanS-tt---aee BBiden
Train
266
279
330
Test
112
106
157
Presidential Stance- Trump
Stance-Trump
Train
243
347
285
Test
98
152
125
Table 2: Class-wise (i.e., neutral, against, favor) train. development, and test set statistics of the Tweet-Stance and Presidential Stannce datasets. Note that we aggregate the topies in Tweet-Stance in our experiments.
3.2 Evaluation
Tweet-Stance: We report the macro average of the Favor and Against F1 scores as defined in (Bar- bieri et al., 2020). We report baseline performance of 3 encoder- based stance detection models: BERT- Spc (Devlin et al., 2018), BERT-GCN (Lin et al., 2021) and PT-HCL (Liang et al., 2022) well two ChatGPT prompting based methods: DQA and SISQA (Zhang et al., 2023b). All baseline scores are extracted from (Zhang et al., 2023b), where we note that evaluation was conducted on only a subset of the label space.
COT excerpt can be found in Figure 1.
After producing COT reasoning for given text, we embed it with transformer encoder and use it as part of a stance detection pipeline. We specif- ically use RoBERTa model (Liu et al., 2019) trained on Twitter data encoder since it has been shown to perform better on Tweet-Stance when compared RoBERTa-base?, We denote this model as Toirter-RoBERTa (TR) in this paper.
Presidential-Stance: We report both the per- class Fl score and the macro average Fl score as re- ported in (Kawintiranon and Singh, 2021). Due to the lack of development set in Presidential-Stance, report the average results three experimen- tal trials with different random seeds. We report the results of three baseline models BERT (Devlin et al., 2018), SKEP (Tian et al., 2020), and KE- MLM (Kawintiranon and Singh, 2021).
We consider three different Sriiter-RoBERTI variants in our experiments. TR-Tweet: We fine- tune with only tweet information. TR-COT: Fine- tune using only COT reasoning as the input and TR- Tweet+COT: Fine-tune Woitter-RoBERT where tweet and COT reasoning are treated as pair wasc input the model (i.e. Tweet and COT reason- ing texts are concatenated and jointly encoded by the pre-trained language model). All fine-tuning follows the standard text classification pipeline in- troduced in (Devlin et al., 2018). Please refer to Appendix A for model hyperparameters and train- ing details for each stance detection task.
Results
4.1 Tweet-Stance
Results on Tweet-Stance are exhibited in Table 3. Results show that TR-Twect+COT produces the best-performing model on Tweet-Stance, with an Fl score of 76.3. Notably, we can retain most of the performance by only embedding the COT rea- soning, as TR-COT has only a 0.6 difference Fl from CR- Our best model provides 6.1-pt improvement over our ChatGPT COT reasoning model, and simply embedding COT pro- vides 5.5 boost in F1 vs extracting results from COT directly.
3.1 Dataset
We assess our method on two well-known Twitter- based stance detection datasets: Tweet-Stance (Mohammad et al., 2016; Barbieri et al., 2020) and Presidential-Stame (Kawintiranon and Singh, 2021). These datasets involve 3-way classifica- tion task to determine whether tweets are in fa- vor, against, or neutral towards a specific topic. The Tweet-Stance dataset comprises five topics: Hillary Clinton (HC), Feminism (FM), Abortion
After investigating the subset of samples where TR- Tweet+COT correct, but disagrees with the prediction from ChatGPT COT, we find that 74%
'uuggingface Model: cardifhlpttviice--roetttaaaaa
scp2022

---

Model
FM
LA
AT
CC
Flaey
lower-performing LLMs such as Falcon-7b (Al- mazrouci et al., 2023) do not provide useful COT, highlighting the importance of LLM performance on this task.
Baselines
BERT-Spe!
49.6
41.9
44.8
BERT-GCNT
50.0
44.3
44.2
PT-HICL
54.5
54.6
50.9
DQA!
78.0
69.0
59.3
4.2 Presidential-Stance
SISQAL
78,9
68.7
61.5
Table 4 presents the results of the Presidential- Stance dataset. Results indicate that our approach outperforms all baseline models. When we ana- lyze the Biden data, TR-Tweet+COT outperforms previous works by 1.4 F1-pts. A very interest- ing result is the extreme difference in performance between ChatGPT-COT and TR-COT, which pro- vides a 20.7-pt boost in Fl score. This is driven by a large number of Implicit Stance Confusion exam- ples where it's challenging to understand the label without seeing other training samples. Specifically, our model is correcting Neutral class predictions 56% of the time tions of democratic figures ideals are taking a stance on Joe Biden which not always the case, causing under-predicteo on Neutral samples. Our error analysis also found stance label hallucina- tions as ChatGPT was found to go off-topic when the focus of the tweet is on another political figure: "wow bernie sander is the only one who supports democracy #demdebate" provoked ChatGPT re- sponse of ".. this tweet is [IN FAVOR] of Bernie Sanders." " which of course not the question being asked.
ChatGPT Only
0-Shot
71.5
61.6
49.1
21.6
37.1
51.6
COT
75.3 71.3
62.6
58.3
67.3
70.2
COT Embbeddids Twitter RoBERTa (TR)
TR-Tweet
59.0
56.6
64.0
67.0
52.6
69.0
TR-COT
81.3
72.6
61.4
70.7
69.3
75.7
TR- Tweet
78.7 70.6 63.8
72.9
54.1
76.3
+COT
Table 3: Results on the Tweet-Stance dataset. The Fl avg column represents the Fl score on the full test set. Per- topic Fl score is additionally reported above by sub- setting TweetStance by topic and computinng the F1 score. Results marked with are taken from prior work.
ChatGPT can assume men-
(131/175) of the samples are on tweets incorrectly labeled as neutral by ChatGPT COT. This confirms our intuition that passing COT information to text encoders may help solve the Implicit Stance Con- fusion problem. Of the remaining 44 samples TR- Tweet+COT was able to predict correctly, we man- ually inspected the 20/44 where ChatGPT predicts "Against" when the true label was "In Favor". We find that 9/9 samples from the HC, FM, LA, AT topics are examples of stance label hallacination. For example, consider the COT reasoning: "... is clear that [NO] this text is against Jeb Bush and in favor of Hillary". This text was marked "[NO] Against Hillary" by our COT parser but was able to be handled by our encoder model the reasoning accurate. The remaining 11 samples in this analysis are from the climate change topic, where most COT errors largely pertain to questions of what it means to be "in favor" or "against" climate change, which we view as more of a natural misun- derstanding than instances of stance label halluci- nation. Future works may explore better prompts to elicit better predictions on climate change tweets.
Similarly, on the Trump data, we find that our best pest-performing model outperforms the closest baseline by 2.4 F1-pts. Interestingly, we note that our best model does not use the tweet information at all, TR-COT obtains the highest average F1 score (81.5). This outcome suggests that the COT reasoning is often logically sound, but our TR-COT model makes the predictions more robust to errors in the ChatGPT COT output structure.
....i
In Table 5, we again evaluate the performance of COT produced by different LLMs Presidential Stance. We find that on both the Biden and Trump datasets, ChatGPT provides the highest performing COT. On both the Biden and Trump datasets, we also find that Llama-2 performs much better than Falcon, again highlighting the importance of LLM quality in our pipeline. Notably, Llama-2 only pro- vides helpful COT for the Biden dataset, not Trump. This result, however, is expected ChatGPT, a seghhr-perfomming language model than Llama-2-
In Table 5, we evaluate the performance of COT produced by different LLMs. We find that while ChatGPT produces the highest performing COT, we achieve a meaningful performance increase when employing the smaller open-source LLM Llama-2-7b (Touvron et al., 2023). Unfortunately,
"htpss//huggingffac.commmmm wlwae/lllaaaaaaaaaaaa/aaaa Laammaa//aaaam222. waw.-aamm--a-aaaa 2-7-chattht

---

Biden
Trump
5
Conclusion
Model
F
N
Avg
F
N
Avg
In this study, we have shown that embedding Chain- of of- Thought reasoning extracted from LLMs (e.g., ChatGPT, Lllama) can boost the performance of stance detection models. Specifically, highlight how we can outperform vanilla COT by augment- ing text encoders with COT embedding. Our analy- sis highlights how text encoders are robust to LLM hallucinations and aid the prediction of deeply implicit stance labels. We encourage future works to consider embedding COT reasoning for stance detection and similar tasks using social media data.
Baselines
BERT"
73.2
68.7
71.5
71.1
75.7
81.0
69.5
75.4
SKEP
79.2
71.5
73.4
74.7
78.5
81.6
71.5
77.2
KE-MLM
79.2
73.2
74.7
75.7
80.9
81.8
73.5
78.7
ChatGPT Only
O-shot
62.1 57.9
65.9
62.0
66.3 69.4
65.8
67.1
COT
55.0 52.7 44.2 50.6
77.0 79.9
71.6
76.2
COT-Embedlee Embeddinsss +
TR-Tweet TR-COT
77.4 72.6 71.8
73.9
81.6
82.6
73.0
79.1
74.2 71.5 68.2
71.3
81.6
85.5
77.6 81.5
TR-Tweet +
80.6 75.9 74.8
77.1
82.3 84.8
75.1 80.7
COT
Limitations
Table 4: Fl scores for both the Biden and Trump subsets dataset. We show class-wise
A limitation of this work is that stance detection us- ing COT reasoning is very sensitive to the prompt provided to ChatGPT (Zhang et al., 2023b). In this study, we do not thoroughly investigate which COT prompt produces the best results, but rather try a few standard approaches inspired by related works. Future works aiming to optimize COT prompt struc- ture for stance detection may find ways to reduce the effects of еггоr hallucinations. In general, our work reduces the need for prompt optimization by mitigating issues pertaining to common COT er- rors.
of the Presidential-Stane performance on Favor, Against, and Neutral classes. The average score is the Macro-Fi score across each class. All Embeddinns + RoBBRTa ex- periments are the average trials. Baseline results marked with are taken from prior work. Standard deviation of each experiment shown in Appendix D.
of three experimental
Model
Tweet Stance
P-Biden
P-Trump
Baselines
Another limitation of this work is that one of its core takeaways - that COT Embeddings reduce effects of implicit stance confusion may only be applicable to popular social media platforms where colloquial language is constantly changing. Application of COT Embeddings to other domains where all necessary information for inference is present in single sample (e.g., in certain NLI tasks), COT Embeddings may not be helpful, Finally, we note that the addition of COT embed- dings may impact the computational efficiency of the model. Specific measures of computational effi- ciency are currently outside the scope of this paper. However, we highlight that if one is in a setting where the COT reasoning can be pre-computed, the impact of COT on computational efficiency is lim- ited. While if COT reasonings had to be computed at inference time, there may be noticeable inference speed degradation depending on the efficiency of the LLM used for COT reasoning.
TR-Tweet
69.0
73.9 (0.87) 79.1 (0.59)
Falcon- ib-instruct
TR-COT
66.5
53.0 (1.60) 61.9 (1.11)
TR- Tweet+COT
65.3
76.0 (0.99)
76.8 (0.19)
Llama-2-bb-che
TR-COT
69.3
63.9 (0.49) 69.1 (1.08)
TR- Tweet+COT
72.5
77.0 (0.47)
78.4 (0.73)
CharGPT (8pt-3.5)
TR-CO CO-TOT 75.7
71.3 (1.28) 81.5 (0.33)
TR- Tweet+COT
76.3
77.1 (1.46)
80.7 (0.61)
Table 5: Comparing the F1 score of different LLMs on Tweet Stance, Presidential Stance Biden (P-Biden) and Presidential Stance Trump (P-Trump). Recall that Presidential Stance has no development set, thus we report the mean result (and standard deviation) over three experimental trials. Our results find that, in gen- eral ChatGPT is the highest performing LLM. We also validate our approach works using Llama-2, a smaller open-source model.
References
7b, only provides a minor improvement over the baseline TR-Tweet.
Ebtesam Almazrouei, Hamza Alobeidii, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru,

---

Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-4OB: an open large language model with state-of-the-art performance.
Findings of the Association for Computational Lin- guistics: ACL-UCCLPP 2021, pages 3152-3157, On- line. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Francesco Barbieri. Jose Camaach-Coollaos, Luis Es- pinosa Anke, and Leonardo Neves. 2020. TweetEvall Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644-1650, Online, Association for Computational Linguistics.
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sob- hami, Xiaodan Zhu, and Colin Cherry. 2016. Semeval- 2016 task 6: Detecting stance in tweets. In Proceed- ings of the 10th International Workshop on Semantic Evaluation (SemEval-20066) pages 31 31-41.
Jacob Devlin, Ming-Wei Chang, Kenton Lec, and
Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR, abs/1810.04805.
Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang. and Feng Wu. 2020. SKEP: Sentiment knowledge enhanced pre-training for sen- timent analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guisties, pages 4067-4077 Online. Association for Computational Linguisties.
Momehil Hardalov, Arna Arora, Preslav Nakov, and Isabelle Augenstein. 2022. A survey on stance de- tection for mis- and disinformation identification. In Findings of the Association for Computational Lin- guistics: NAACL 2022, pages 1259-1277, Seattle, United States. Association for Computational Lin- guistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu. Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller. Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov. Pushkar Mishra, Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten. Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tuned chat models.
Zihao He, Negar Mokhberian, and Kristina Lerman. 2022. Infusing knowledge from wikipedia to enhance stance detection.
Kornraphop Kawintiranon and Lisa Singh. 2021. Knowledge enhanced masked language model for stance detection. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 4725 4735,, Online. As- sociation for Computational Linguistics.
Yingjie Li and Cornelia Caragea. 2021. Target-aware data augmentation for stance detection. In Proceed- ings of the 2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1850-1860. Online. Association for Computational Linguistics.
Bin Liang. Zixiao Chen, Lin Gui, Yulan He, Min Yang. and Ruifeng Xu. 2022. Zero-shot stance detection via contrastive learning. In Proceedings of the ACM Web Conference 2022, WWW '22, page 2738-2747, New York, NY, USA. Association for Computing Machinery.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Fdd Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning in large language models.
Bowen Zhang, Daijun Ding, and Liwen Jing. 2023a. How would stance detection techniques evolve after the launch of chatgpt?
Yuxiao Lin, Yuxian Meng, Xiaofei Sun, Qinghong Han, Kun Kuang. Jiwei Li, and Fei Wu. 2021. BertGCN: Transductive text classification by combining GNN and BERT. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 1456-1462, Online, Association for Computational Linguistics.
Bowen Zhang, Xianghua Fu, Daijun Ding, Hu Huang. Yangyang Li, and Liwen Jing. 2023b. Investigating thain-of-thought with chatgpt for stance detection on social media.
Rui Liu, Zheng Lin, Yutong Tan, and Weiping Wang. 2021. Enhancing zero-shot and sew-shot stance de- tection with commonsense knowledge graph. In

---

Appendix
structures in the samples below. The <Tweet> and <Topi>> markers are parameters of the prompt and dynamically change given the sample. Consider the following two examples which COT prompt for stances on Atheism and Donald Trump:
A Training Details
Tweet-Stance: For training, we aggregate the training set of all stance topics. Since there mix of topies in the training set, we include the topic as part of the input (i.c. pretend tweet with topic information) during training. We train each model with a batch size of 16 for a maximum of 10 epochs with early stopping on the development set with patience = 2 keeping other related parameters default 5 The learning rate for each of our models was chosen via modest grid search of [le-3, le-4, 2e-5, 5e-5]. All other parameters are the default provided by the Huggingface Trainer
Example 1: Read the following tweet and decide if the stance of the tweet is in favor [YES], against [NO], or neutral [NONE] with regards to the topic Atheism:
Tweet: You cant think by yourself about life and believe in god. It just doesn't add up #SemST
Presidential-Stance:: For training, since there is no available development set, we simply fine-tune each model with batch size of 32 for 5 epochs using the default parameters provided by the Hug- gingface trainer. We report the average macro Fl score and report the standard deviation over three experimental runs with different random seeds.
Stance: Lets think step by step. Since this text finds belief in god to be contradicting with the notion of thinking by oneself, it must be the case that [YES] this text is in favor of atheism.
Tweet: <Tweet>
Prompting Details
Stance: Lets think about this step by step.
In our experiments, we use the following COT prompting template.
Example Read the following tweet and decide if the stance of the tweet is in favor [IN FAVOR], against [AGAINST], or neutral [NEUTRAL] with regards to the topic Joe Biden:
Read the following tweet and decide if the stance of the tweet is in favor [IN FAVOR / YES], against [AGAINST] NO], neutral INEUTRAL / NONE] with regards to the topic <topic>:
Tweet: america's ceos say trump failed on coronavirus and they're backing biden HTTP
Tweet: <Tweet>
Stance: <COT Example>
Stance: Lets think step by step. Since the tweet mentions Trump's failures and how important CEOs in america are backing Joe Biden, then this tweet is [IN FAVOR] of Joe Biden.
Tweet:
<Tweet>
Stance: Lets think about this step by step.
The label to be output by the model is in brackets I1 and is used to parse the output string to convert the label into numeric representation. Notice that we explore two label structures, the support- ing label is either [IN FAVORJ [YES], the non- supporting label is either [AGAINST] or INO], and the stance-free label is either [NEUTRAL] or [NONEJ. We provide examples of both label
Tweet: <TWeet>>
Stance: Lets think about this step by step.
1-shot COT examples were chosen randomly from each training set with the COT reasoning writ- ten by the author. We note that our O-Shot ChatGPT bascline uses the same prompt without the 1-shot COT example.
https://tuggingiiee..oo/s/s//// coldes///tttssssrrees/s/////////
main wtwpee/sccallaaccckssssoooooo Early Stonrtyttopiinaallbaa wtg.ginnffcccooooocc///aatnmmmeo classes/rainer

---

C Variability of Presidential-Stance Results
Due to space constraints we are unavailable to add the standard deviations of our Presidential-Stance experiments to Table 4 in the main paper. Please refer Table 6 for the standard deviation of each experiment.
Model
Biden
Trump
TR-Tweet
73.9 + 0.87
79.1 0.59
TR-COT
71.3 + 1.28
81.5 + 0.33
CR-Tweet+COT
77.1 + 1.46
80.7 + 0.61
Table Macro scores with standard deviations over three experimental trails with different random seeds.
D Variability of Sweet-Stance Results
While Tweet-Stance results are the product of dev set optimization, we also re-run our model five times with five different random seeds to highlight model variability, this is fairly low-resource problem. Please refer Table 7 for the resulting standard deviations of each experiment.
Model
F1
TR-Tweet
70.58 1.20
TR-COT
76.53 + 0.54
TR- COT
76.55 + 1.72
Table 7: F1 scores on Sweet-Stance with standard devia- tions five experimental trails with different random seeds.


Reviews:
[{"review_id": "WfMfVJQOWL", "reviewer": "Reviewer_n7UD", "summary": "", "questions": "For the 1-shot COT proposed by the authors, how sensitive the approach is to the COT reasoning written by the authors?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Overview\nWe thank the reviewers for their thoughtful feedback. We are encouraged that the reviewer finds our method of embedding Chain-of-Thought (COT) reasonings from Large Language Models (LLMs) to be novel. We specifically appreciate the reviewer’s feedback on our analysis, number of evaluation datasets, and for highlighting the potential of data leakage via the use of ChatGPT. We answer specific questions and feedback from the reviewer below. \n\n\n###  Questions\n___\n> **Question (Q1.1):**  “More datasets needed.“\n\n**Answer (A1.1):** We highlight that our choice of evaluating on two benchmark datasets is in-line with prior works on stance detection which evaluate on 1-3 benchmarks, including EMNLP publications (e.g. [1,2,3]). This experimental design choice is common as social media-based stance detection data is difficult to collect at scale, leading to limited available datasets. \n\n\n> **Question (Q1.2):** [Need more] “in-depth analysis … I suspect the improvement of the performance may be due to the existence of certain stance-related cue words generated by chatgpt, rather than injecting world knowledge. The paper could be more informative if the authors could provide some analysis on the generated COT reasonings.” \n\n**Answer (A1.2):** We agree that the paper would be improved with a more in-depth analysis of why the COT reasonings are improving performance. However, we note that in the limited space allotted for a short paper, it was challenging to include additional analysis. We believe this work is well-suited for a short paper as our methods require only a brief section to explain in complete detail. \n\nRegarding your concern about world knowledge, we have manually checked a few samples. While some samples have no overt need for additional world knowledge, we do find numerous samples where ChatGPT does inject world knowledge not found in the original text. For further investigation, we will release all the LLM outputs as a part of our paper. Here is an example of a ChatGPT COT injecting world knowledge for your reference.  \n\n> **Tweet:** Take hold of your Authority in Christ! Take back what the enemy has stolen! #authority #Life #Calling #SemST\t\n\n> **Label** *Against* Atheism \n\n> **ChatGPT COT:** Since this text mentions Christ and the enemy, it is likely coming from a Christian perspective. Therefore, the stance of the tweet is [NO], as it is promoting belief in a higher power.\n\nWe additionally highlight the non-trivial nature of identifying the importance of stance-related cue words. Macro-level analysis of word-level contributions to performance would require significant human annotation. Given the difficulty of running such an experiment, we instead opt for extrinsic evaluation of COT reasonings via their ability to increase performance on two stance detection tasks. We, however, plan to release the outputs from our models for further analysis — making additional analysis feasible for future works.\n\n> **Question (Q1.3):** “I'm also not convinced that COT embedding could help implicit stance detection, since the authors only provide related analysis on one dataset (Twitter Stance).”\n\n**Answer (A1.3):** We appreciate your perspective on this matter. However, we might have different interpretations. We want to share additional details that might clarify our point. Implicit stance detection is required for both datasets evaluated in our study — namely Tweet-Stance and Presidential-Stance. We provide analysis of errors related to implicit stance detection on both datasets in sections 4.1 and 4.2. However, we appreciate that the example from the Presidential-Stance dataset in Appendix B does not highlight the need for implicit stance detection in this dataset. Thus, please consider an additional sample, which is In Favor of Joe Biden. We will add this example to the paper for clarity. \n\n>“@USER @USER and @USER have to sow doubt and cheat and will say after the debate that biden cheated - as they already are prior to. his cult will believe whatever he says. it’s so disturbing”\n\nImplicit in this sample is that Biden supporters view Trump supporters as 1) being members of a cult, 2) cheaters, and 3) willing to believe whatever Trump says. Such opinions are commonly held on Twitter amongst Biden supporters but difficult to ascertain from this sample alone.\n\n> **Question (Q2):** “[Results may suffer from] data leakage. It is possible that chatgpt has already seen these stance datasets during its training. It may be interesting to compare the results on other LLMs with chatgpt.”\n\n**Answer (A2):** Yes, you are right! Our results will be more sound if we can validate that the embeddings of COT-reasonings from other LLMs can improve baseline performance. Thus, we added additional experiments using two popular LLMs, Llama 2 [4] and Falcon [5], as they are high-performing open-source LLMs. The results of the additional experiments can be found in Table 1 and Table 2.  Recall that we use the Twitter-RoBERTa (TR) backbone to encode the tweet-only (TR-Tweet), the COT only (TR-COT), and both the Tweet and COT texts together (TR-Tweet+COT). Note that gpt-3.5-turbo results are from the original paper. LLama-2 and Falcon results are additional results run as part of the rebuttal. \n\nOur experiments confirm that with the recently released [llama-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat) our results hold for most experiments. Specifically, embedding Llama 2 COT reasonings gives us a **3.1 pt boost** on the Biden subset of Presidential-Stance and a **3.5 pt boost** on the Tweet-Stance dataset, respectively. However, we find that results do not improve with Llama2 COT embeddings on the Trump subset of the Presidential-Stance dataset. This however was expected as gpt-3.5-turbo, a model much more powerful than Llama2-7b-chat (which is evident from the results in [4]) only provides a minor improvement over the baseline TR-Tweet (Trump subset). \n\nWe find that COT-reasonings from falcon-7b-instruct are less impactful, however this is expected as Falcon is significantly less powerful than Llama 2 (as indicated in [4]). *Thus, our additional experiments confirm that LLM performance is critical to the effectiveness of the proposed CoT based approach.* Please see the tables below for the full details of these additional experiments. \n\n|                | LLMs used to generate COT reasoning |  Biden      | Trump       |\n|----------------|--------------------------------------|-------------|-------------|\n| TR-Tweet       | -                                    | 73.9 (0.87) | 79.1 (0.59) |\n|   TR-COT       |             gpt-3.5-turbo            | 71.3 (1.28) | **81.5 (0.33)** |\n|                |            Llama-2-7b-chat           | 63.9 (0.49) | 69.1 (1.08) |\n|                |          Falcon-7b-instruct          | 53.0 (1.60) | 61.9 (1.11) |\n|   TR-Tweet+COT |             gpt-3.5-turbo            | **77.1 (1.46)** | 80.7 (0.61) |\n|                |            Llama-2-7b-chat           | 77.0 (0.47) | 78.4 (0.73) |\n|                |          Falcon-7b-instruct          | 76.0 (0.99) | 76.8 (0.19) |\n\n**Table 1:** F1 scores on Presidential-Stance using COT embeddings from 3 different LLMs. Due to the lack of development set in Presidential Stance,  results are reported as the mean of three experimental trials. The standard deviation is reported in parentheses.  \n\n\n|                | LLMs used to generate COT reasoning | F1-score |\n|----------------|--------------------------------------|----------|\n| TR-Tweet       |                                      |   69.0   |\n|   TR-COT       |             gpt-3.5-turbo            | **75.7** |\n|                |            Llama-2-7b-chat           |   69.3   |\n|                |          Falcon-7b-instruct          |   66.5   |\n|   TR-Tweet+COT |             gpt-3.5-turbo            | **76.3** |\n|                |            Llama-2-7b-chat           |   72.5   |\n|                |          Falcon-7b-instruct          |   65.3   |\n\n**Table 2.**  F1 scores on Tweet-Stance using COT embeddings on 3 different LLMs. Unlike Presidential-Stance, Tweet-Stance has a development set, so we report results after early stopping as mentioned in Appendix A.\n\nWe will share these new results in the main paper (as space permits) and/or in the appendix.\n\n>**Question (Q3):** “For the 1-shot COT proposed by the authors, how sensitive the approach is to the COT reasoning written by the authors?”\n\n**Answer (A3):** We did not do any prompt optimization for the 1-shot COT in this study. We simply picked a random training example and wrote an accurate COT to help the model understand the task. We hypothesize that analysis of prompt optimization can potentially boost performance even further.\n\n---\n\nWe appreciate your time and efforts to review our paper! We hope this response satisfies your concerns and resolves any potential ambiguity. Please let us know if our replies address your concerns.\n\n----\n\n## References \n[1] Yang Li and Jiawei Yuan. “Generative Data Augmentation with Contrastive Learning for Zero-Shot Stance Detection”. (EMNLP 2022)\n\n[2] Yingjie Li and Cornelia Caragea., Target-Aware Data Augmentation for Stance Detection. (NAACL 2021)\n\n[3] Matero, M., Soni, N., Balasubramanian, N., & Schwartz, H. A., MeLT: Message-level transformer with masked document representations as pre-training for stance detection. (EMNLP Findings 2021)\n\n[4] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., ... & Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.\n\n[5] https://huggingface.co/tiiuae/falcon-7b-instruct", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic and approaches are relatively novel for stance detection.", "reasons_to_reject": "I have following concerns:\n1) In-depth analysis and more datasets needed. I suspect the improvement of the performance may be due to the existence of certain stance-related cue words generated by chatgpt, rather than injecting world knowledge. The paper could be more informative if the authors could provide some analysis on the generated COT reasonings. I'm also not convinced that COT embedding could help implicit stance detection, since the authors only provide related analysis on one dataset (Twitter Stance).\n2) Data leakage. it is possible that chatgpt has already seen these stance datasets during its training. It may be interesting to compare the results on other LLMs with chatgpt.", "questions_to_authors": "For the 1-shot COT proposed by the authors, how sensitive the approach is to the COT reasoning written by the authors?", "paper_topic_and_main_contributions": "This paper explores to incorporate the embedding of chain-of-thought reasoning generated by chatgpt into traditional pre-trained Twitter-roberta fine-tuning process.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2XaDDkjylr", "reviewer": "Reviewer_bcgD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Overview\n\nWe thank the reviewers for their comments on our paper. We are encouraged by the reviewer's view that our work on embedding Chain-of-Thought (COT) outputs from LLMs is novel, useful to the task of stance detection, and interesting to the research community. We also appreciate the feedback regarding the quality of our experimental comparison. We hope to address all of the reviewers' questions and concerns below. \n\n## Questions\n---\n> **Question (Q1):** “The experimental comparison is limited, despite the availability of more recent models published between 2021 and 2023.”\n\n**Answer (1):** Assuming that this comment is in reference to the use of only one LLM, i.e., ChatGPT, which has a knowledge cutoff of 2021, we address this concern via additional experiments on recently released LLMs. Specifically, we evaluate our approach using both Falcon and Llama 2 [1], which has knowledge through September 2022. The results of the additional experiments can be found in Table 1 and Table 2. Recall that we use the Twitter-RoBERTa (TR) backbone to encode the tweet-only (TR-Tweet), the COT only (TR-COT), and both the Tweet and COT texts together (TR-Tweet+COT). Note that gpt-3.5-turbo results are from the original paper. LLama-2 and Falcon results are additional results run as part of the rebuttal. \n\nOur experiments confirm that with the recently released [llama-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat) our results hold for most experiments. Specifically, embedding Llama 2 COT reasonings gives us a 3.1 pt boost on Presidential-Stance (Biden Subset) and a 3.5 pt boost on Tweet-Stance . However, we find that results do not improve with Llama2 COT embeddings on the Presidential-Stance dataset (Trump Subset). This however was expected as gpt-3.5-turbo, a model much more powerful than Llama2-7b [1], only provides a minor improvement over the baseline TR-Tweet (Trump). We find that COT-reasonings from [falcon-7b-instruct](https://huggingface.co/tiiuae/falcon-7b-instruct) are less impactful, however this is expected as Falcon is significantly less powerful than Llama-2 (as indicated in [1]). Thus, *our additional experiments confirm that LLM performance is critical to the effectiveness of the proposed COT based approach.* Please see the table below for the full details of these additional experiments. \n\n|                | LLMs used to generate COT reasoning |  Biden      | Trump       |\n|----------------|--------------------------------------|-------------|-------------|\n| TR-Tweet       | -                                    | 73.9 (0.87) | 79.1 (0.59) |\n|   TR-COT       |             gpt-3.5-turbo            | 71.3 (1.28) | **81.5 (0.33)** |\n|                |            Llama-2-7b-chat           | 63.9 (0.49) | 69.1 (1.08) |\n|                |          Falcon-7b-instruct          | 53.0 (1.60) | 61.9 (1.11) |\n|   TR-Tweet+COT |             gpt-3.5-turbo            | **77.1 (1.46)** | 80.7 (0.61) |\n|                |            Llama-2-7b-chat           | 77.0 (0.47) | 78.4 (0.73) |\n|                |          Falcon-7b-instruct          | 76.0 (0.99) | 76.8 (0.19) |\n\n**Table 1:** F1 scores on Presidential-Stance using COT embeddings from 3 different LLMs. Due to the lack of development set in Presidential Stance,  results are reported as the mean of three experimental trials. The standard deviation is reported in parentheses.  \n\n\n|                | LLMs used to generate COT reasoning | F1-score |\n|----------------|--------------------------------------|----------|\n| TR-Tweet       |                                      |   69.0   |\n|   TR-COT       |             gpt-3.5-turbo            | **75.7** |\n|                |            Llama-2-7b-chat           |   69.3   |\n|                |          Falcon-7b-instruct          |   66.5   |\n|   TR-Tweet+COT |             gpt-3.5-turbo            | **76.3** |\n|                |            Llama-2-7b-chat           |   72.5   |\n|                |          Falcon-7b-instruct          |   65.3   |\n\n**Table 2.**  F1 scores on Tweet-Stance using COT embeddings on 3 different LLMs. Unlike Presidential-Stance, Tweet-Stance has a development set, so we report results after early stopping as mentioned in Appendix A.\n\nWe will share these new results in the main paper (as space permits) and in the appendix.\n\n----\n\n> **Question (Q2):** “The contribution of the paper seems incremental - applying some variation of known methods to a known problem...I think its impact is limited.”\n\n**Answer (A2):** We appreciate your perspective on this matter. However, we might have different interpretations. We want to share additional details that might clarify our point. Our work does not necessarily apply a *variation* of known methods but rather a *novel* combination of known methods. We believe this is a relevant and timely contribution for the following reasons. \nOur technical implementation is simple and intuitive for use in future downstream NLP applications. We provide novelty as we are the first to embed COT reasonings as a means for providing additional context and world knowledge to the stance detection pipeline, unlike prior works that simply use COT for text classification. Additionally, we believe advancements in stance detection are of high impact as it is a task of great importance for large-scale aggregation of public opinion. Our work provides an increase in performance over prior works at a rate consistent with prior publications on Stance Detection @ EMNLP such as [2,3].\n\n----\nWe appreciate your time and efforts to review our paper! We hope this response satisfies your concerns and resolves any potential ambiguity. Please let us know if our replies address your concerns.\n\n----\n### References\n\n---\n[1] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., ... & Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.\\\n[2] Yingjie Li, Chenye Zhao, and Cornelia Caragea., Improving Stance Detection with Multi-Dataset Learning and Knowledge Distillation. (EMNLP 2021)\\\n[3] Matero, M., Soni, N., Balasubramanian, N., & Schwartz, H. A., MeLT: Message-level transformer with masked document representations as pre-training for stance detection. (EMNLP Findings 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think that the paper would be of interest for people working on stance classification. This work tackles a useful task and the approach is reasonable.", "reasons_to_reject": "As stated above, the paper makes incremental contributions to existing problems, therefore, I think its impact is limited.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This manuscript is focused on the issue of Chain-of-Thought for stance detection. This paper proposes the CoT embedding method and the model is evaluated on several stance detection dataset, and is shown to outperform a variety of baseline models.\n\nStrengths: This work tackles a useful task, and the general approach seems novel and interesting to many researchers.\n\nWeaknesses:\nThe experimental comparison is limited, despite the availability of more recent models published between 2021 and 2023.\nThe contribution of the paper seems incremental - applying some variation of known methods to a known problem.\nThe writing could be improved.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "x7ESzmC8w1", "reviewer": "Reviewer_pvkN", "summary": "", "questions": "A) line 79: \"COT reasoning is an inexpensive way to inject world knowledge into\na text encoder\"\n=> I agree with this statement but I would not say that using ChatGPT (here to generate COT) is \"inexpensive\"\n\nB) The datasets used for evaluation are rather small (for example, 1800 intances for the class FAVOR). Table 1 should also give the distribution for each class in the train/test sets. \n\nC) Why not testing all baseline models on both datasets?\n\nD) line 155: \"tweet and COT reasoning are treated as a pair-wise input to the model\"\n=> What do you mean by \"pair-wise input\"? Are the embeddings concatenated? or do you mean a pair-wise learning method which attempts to find a loss function for feature pairs?\n\nE) line 202: \"Results show that TR-Tweet+COT produces the best-performing model on Tweet-Stance\"\n=> However when looking per topic, TR-COT seems to be better than TR-Tweet+COT.\n=> In Table 2 caption, I don't understand \"Per-class F1 score is additionally reported\": F1-score are given per topic, but not per class. \n\nF) line 471: \"1-shot COT examples were chosen randomly from each training set with the COT reasoning written by the author. We note that our 0-Shot ChatGPT baseline uses the same prompt without the 1-shot COT example.\"\n=> How many examples were chosen for the 1-shot COT? \n=> In example 2 (line 453): the topic is Trump but the 1-shot COT example is about Biden (which is called label hallucination in the paper). Has the author written an incorrect example on purpose? \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Overview\nWe thank the reviewers for their very thoughtful and detailed comments on our paper. We believe this feedback will make our work stronger. We are encouraged by the reviewers’ appreciation of our Chain-of-Thought (COT) embedding approach and how it improves results over previous works. We appreciate the questions and feedback the reviewer provided regarding dataset size and evaluation strategy. We address all reviewer comments and questions below. \n\n## Questions\n\n----\n> **Question (A):** “line 79: \"COT reasoning is an inexpensive way to inject world knowledge into a text encoder\" => I agree with this statement but I would not say that using ChatGPT (here to generate COT) is \"inexpensive\"” \n\n**Response (A):**  This is a great point and perhaps a poor choice of words, as the use of OpenAI’s ChatGPT API is not free to use. In this line, we are trying to communicate that it is low-cost with respect to the lack of need for additional model training parameters + lower implementation difficulty. We will be sure our final draft reflects this change in phrasing. \n\n---\n> **Question (B.1):** “The datasets used for evaluation are rather small (for example, 1800 instances for the class FAVOR). \n\n**Response (B.1):** We agree with the reviewer but note that we are unfortunately restricted to evaluation on available public datasets for which there are limited available resources. Stance-detection on social media is often inherently resource constrained due to significant expense and effort of large scale annotation. In the future we hope to evaluate our work on larger datasets. Note that related works from top conferences such as [1,2,3] also utilize Tweet-Stance for evaluation.\n\n---\n> **Question (B.2):** “Table 1 should also give the distribution for each class in the train/test sets.” \n\n**Response (B.2):** Thank you for this great suggestion. Please find this information below for reference. \n\n| Split | Favor | Against | Neutral |\n|:-----:|:-----:|:-------:|:-------:|\n| Train | 678   | 1254    | 688     |\n| Dev   | 75    | 141     | 78      |\n| Test  | 304   | 715     | 230     |\n\n**Table 1:** Class distribution for each data split on Tweet-Stance \n\n|    Split    | Favor | Against | Neutral |\n|:-----------:|:-----:|:-------:|:-------:|\n| Biden-Train | 266   | 279     | 330     |\n| Biden-Test  | 112   | 106     | 157     |\n| Trump-Train | 243   | 347     | 285     |\n| Trump-Test  | 98    | 152     | 125     |\n\n**Table 2:** Class distribution for each data split on Presidential-Stance\n\nIf space permits, we will add this information to the main paper, otherwise we will be sure to include it in the appendix. \n\n____\n> **Question (C):**  “Why not testing all baseline models on both datasets?”\n\n**Response (C):** We did not run all baselines ourselves. Rather, certain baseline results were listed as they had been reported in prior works, where evaluation on certain stance topics was missing. We are unable to fill-in missing baselines from prior works by the rebuttal deadline but will consider doing so for our final submission. This issue arises as some results reported from prior works unfortunately only cite the model used for the baseline without providing training details. We would thus be unable to run the experiment ourselves in a consistent manner. \n\n___\n> **Question (D):**  line 155: \"tweet and COT reasoning are treated as a pair-wise input to the model\" => What do you mean by \"pair-wise input\"? Are the embeddings concatenated? or do you mean a pair-wise learning method which attempts to find a loss function for feature pairs?\n\n**Response (D):**  We refer to pairwise input in the sense of how one might pass a pairwise task to a BERT-based model for text classification. (See Figure 2 in the [BERT](https://arxiv.org/abs/1810.04805) paper for a visualization). Specifically, we feed 
 Tweet 
 COT-Reasoning 
 to the RoBERTa model. Thus, the Tweets and COT reasoning are cross-encoded together in this manner. We will clarify this point in the final draft of our manuscript. \n\n___\n> **Question (E.1):** “line 202: \"Results show that TR-Tweet+COT produces the best-performing model on Tweet-Stance\" => However when looking per topic, TR-COT seems to be better than TR-Tweet+COT.\n\n**Response (E.1):** This is correct. However, our conclusion that TR-Tweet+COT is the best-performing model is based on the overall F1 score, which is performance on the full test set. We include per-topic scores for reference, but given that different topics have different distributions / different numbers of samples, we think that it’s unfair to classify a model as higher performing because it got a higher score on certain topics.\n\nFor example, Tweet-Stance  F1 is defined as mean F1 on **Favor** and **Against**, with no consideration of **Neutral** predictions. The Climate Change topic only has 26 **Against** samples vs the Hillary Clinton topic which has 565 **Against** samples.  \n\n---\n> **Question (E.2):** “In Table 2 caption, I don't understand \"Per-class F1 score is additionally reported\": F1-score are given per topic, but not per class.”\n\n**Response (E.2):**  We apologize for the confusion on this point. You are correct in stating that the terminology we should have used is “Topic” not “class”, as class would refer to (Favor, Against, Neutral) for this task. Here we are showing the overall F1 score as well as per-topic F1 score. This typo will be fixed in our final manuscript. \n\n___\n> **Question (F.1):**  “line 471: \"1-shot COT examples were chosen randomly from each training set with the COT reasoning written by the author. We note that our 0-Shot ChatGPT baseline uses the same prompt without the 1-shot COT example.\" => How many examples were chosen for the 1-shot COT?” \n\n**Response(F.1):** We randomly select one sample per-topic in each training set. For example, since there are 5 topics in Tweet-Stance , there are 5 total samples used for 1-shot COT prompting. When we pass a Hillary Clinton sample to an LLM for 1-shot COT, it would have a different demonstration than when we pass a Climate Change sample. However, only one demonstration is shown in both cases. We will clarify this point in our final manuscript! \n\nNote that we did no prompt optimization in this study as it was considered out of the scope of this work. We simply pick a random sample and write one correct COT reasoning to help guide the model on the task. \n\n____\n> **Question (F.2):** “In example 2 (line 453): the topic is Trump but the 1-shot COT example is about Biden (which is called label hallucination in the paper). Has the author written an incorrect example on purpose?”\n\n**Response (F.2):** We apologize for this manuscript typo as the underlined topic in this example should read “Joe Biden”. This will be fixed in the final version of the manuscript.\n\n---\n\nWe appreciate your time and efforts to review our paper! We hope this response satisfies your concerns and resolves any potential ambiguity. Please let us know if our replies address your concerns.\n\n---\n\n### References\n___\n[1] Yang Li and Jiawei Yuan. 2022. Generative Data Augmentation with Contrastive Learning for Zero-Shot Stance Detection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6985–6995, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\\\n[2] Yingjie Li and Cornelia Caragea. 2021. Target-Aware Data Augmentation for Stance Detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1850–1860, Online. Association for Computational Linguistics.\\\n[3] Matero, M., Soni, N., Balasubramanian, N., & Schwartz, H. A. (2021). MeLT: Message-level transformer with masked document representations as pre-training for stance detection. ACL Findings", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Embedding COT reasoning texts in a stance classification model improves the results.\n", "reasons_to_reject": "- The datasets used for evaluation are rather small.\n- Some details are unclear (prompting, distribution, F1-score...)", "questions_to_authors": "A) line 79: \"COT reasoning is an inexpensive way to inject world knowledge into\na text encoder\"\n=> I agree with this statement but I would not say that using ChatGPT (here to generate COT) is \"inexpensive\"\n\nB) The datasets used for evaluation are rather small (for example, 1800 intances for the class FAVOR). Table 1 should also give the distribution for each class in the train/test sets. \n\nC) Why not testing all baseline models on both datasets?\n\nD) line 155: \"tweet and COT reasoning are treated as a pair-wise input to the model\"\n=> What do you mean by \"pair-wise input\"? Are the embeddings concatenated? or do you mean a pair-wise learning method which attempts to find a loss function for feature pairs?\n\nE) line 202: \"Results show that TR-Tweet+COT produces the best-performing model on Tweet-Stance\"\n=> However when looking per topic, TR-COT seems to be better than TR-Tweet+COT.\n=> In Table 2 caption, I don't understand \"Per-class F1 score is additionally reported\": F1-score are given per topic, but not per class. \n\nF) line 471: \"1-shot COT examples were chosen randomly from each training set with the COT reasoning written by the author. We note that our 0-Shot ChatGPT baseline uses the same prompt without the 1-shot COT example.\"\n=> How many examples were chosen for the 1-shot COT? \n=> In example 2 (line 453): the topic is Trump but the 1-shot COT example is about Biden (which is called label hallucination in the paper). Has the author written an incorrect example on purpose? \n", "paper_topic_and_main_contributions": "The paper presents an approach for stance detection in social media, using Chain-of-Thought (COT) embeddings along with texts to be classified.\nThis approach follows (Zhang et al, 2023) and asks ChatGPT to generate reasoning texts (COT) for a given tweet and to predict its stance towards a given topic. Then, COT are used as a feature in a stance classification model (RoBERTa) in order to inject world knowledge instead of using a real knowledge base. \nThis model is evaluated on 2 datasets and outperforms other baselines, including ChatGPT.  \n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NNfsWbxKA2", "reviewer": "Reviewer_XojB", "summary": "", "questions": "A. How do COT Embeddings affect the time needed for inference, compared to direct classification?\n\nB. Appendix C: Consider reporting these numbers for the Tweet-Stance dataset as well.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Overview \n\nWe kindly thank the reviewer for their feedback on our paper. We are encouraged by the reviewers’ high excitement score, as well as the reviewers appreciation of our intuitive and  novel approach to applying Chain-of-Thought (COT) embeddings to stance detection. We specifically appreciate the reviewer taking note of our methods' potential applicability to other downstream text classification tasks. Please find our specific response to the reviewer’s comments and questions below. \n\n## Questions/Comments\n\n___\n> **Comment (1):** “Limited reproducibility: ChatGPT, a commercial product, was used as the language model and it might not be possible \nfor future work to reproduce the results.”\n\n**Response (1):** We agree with the reviewer that our results will be more reproducible if we can validate that the embeddings of COT-reasonings from other open-source LLMs can improve baseline performance. We have thus added two additional experiments using the popular Llama 2 [1] and [Falcon](https://huggingface.co/tiiuae/falcon-7b-instruct) LLMs. The results of the additional experiments can be found in Table 1 and Table 2. Recall that we use the Twitter-RoBERTa (TR) backbone to encode the tweet-only (TR-Tweet), the COT only (TR-COT), and both the Tweet and COT texts together (TR-Tweet+COT). Note that gpt-3.5-turbo results are from the original paper. LLama-2 and Falcon results are additional results run as part of the rebuttal. \n\nOur experiments confirm that with the recently released [llama-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat) our results hold for most experiments. Specifically, embedding Llama 2 COT reasonings gives us a 3.1 pt boost on Presidential-Stance (Biden Subset) and a 3.5 pt boost on Tweet-Stance . However, we find that results do not improve with Llama 2 COT embeddings on the Trump subset of the Presidential-Stance  dataset. This however was expected as gpt-3.5-turbo, a model much more powerful than Llama2-7b-chat (as per results in [1]) only provides a minor improvement over the baseline TR-Tweet (Trump). \n\nWe find that COT-reasonings from [falcon-7b-instruct](https://huggingface.co/tiiuae/falcon-7b-instruct) less impactful, however this is expected as Falcon is significantly less powerful than Llama-2 (as per [1]). Thus, our additional experiments confirm that *LLM performance is critical to the effectiveness of the proposed CoT based approach.* Please see the tables below for the full details of these additional experiments. \n\n|                | LLMs used to generate COT reasoning |  Biden      | Trump       |\n|----------------|--------------------------------------|-------------|-------------|\n| TR-Tweet       | -                                    | 73.9 (0.87) | 79.1 (0.59) |\n|   TR-COT       |             gpt-3.5-turbo            | 71.3 (1.28) | **81.5 (0.33)** |\n|                |            Llama-2-7b-chat           | 63.9 (0.49) | 69.1 (1.08) |\n|                |          Falcon-7b-instruct          | 53.0 (1.60) | 61.9 (1.11) |\n|   TR-Tweet+COT |             gpt-3.5-turbo            | **77.1 (1.46)** | 80.7 (0.61) |\n|                |            Llama-2-7b-chat           | 77.0 (0.47) | 78.4 (0.73) |\n|                |          Falcon-7b-instruct          | 76.0 (0.99) | 76.8 (0.19) |\n\n**Table 1:** F1 scores on Presidential-Stance using COT embeddings from 3 different LLMs. Due to the lack of development set in Presidential Stance,  results are reported as the mean of three experimental trials. The standard deviation is reported in parentheses.  \n\n\n|                | LLMs used to generate COT reasoning | F1-score |\n|----------------|--------------------------------------|----------|\n| TR-Tweet       |                                      |   69.0   |\n|   TR-COT       |             gpt-3.5-turbo            | **75.7** |\n|                |            Llama-2-7b-chat           |   69.3   |\n|                |          Falcon-7b-instruct          |   66.5   |\n|   TR-Tweet+COT |             gpt-3.5-turbo            | **76.3** |\n|                |            Llama-2-7b-chat           |   72.5   |\n|                |          Falcon-7b-instruct          |   65.3   |\n\n**Table 2.**  F1 scores on Tweet-Stance using COT embeddings on 3 different LLMs. Unlike Presidential-Stance, Tweet-Stance has a development set, so we report results after early stopping as mentioned in Appendix A.\n\nWe will share these new results in the main paper (as space permits) and/or in the appendix.\n\n___\n> **Comment (2):** “The authors do not indicate that they will share the LM outputs they collected.”\n\n**Response (2):** Thank you for this great idea! We will be sure to release the LLM outputs along with our code upon acceptance of the paper! \n\n___\n> **Question (A):** How do COT Embeddings affect the time needed for inference, compared to direct classification? \n\n**Response (A):** This is an important question that should be addressed in detail in future works — however measures of computational efficiency are currently outside the scope of this paper. Intuitively if one is in a setting where the COT-reasoning can be pre-computed, the impact is limited. However, if COT-reasonings had to be computed at inference time, there would be noticeable inference speed degradation. Hopefully this answers your question! \n\n___\n> **Question (B):** “Appendix C [Referring to standard deviation of Presidential-Stance results]: Consider reporting these numbers for the Tweet-Stance dataset as well.”\n\n**Response (B):** We can certainly report the mean result over numerous trials on Tweet-Stance in the final version of our paper. However, we remind the reviewer that our Tweet-Stance results were optimized using the development set and that Presidential-Stance does not have a development set, which provoked the need for reporting mean performance. However, given that there may be variability in results with different seeds, we have run this experiment on Tweet-Stance and report the new results below. \n\n|  Experiment  |   F1  | Standard Deviation |\n|:------------:|:-----:|:------------------:|\n|   TR-Tweet   | 69.8  |        1.27        |\n|    TR-COT    |  76.0 |        1.10        |\n| TR-Tweet+COT |  77.6 |      1.11e-16      |\n\n**Table 3:**  Mean and standard deviation of Tweet-Stance with ChatGPT COT Embeddings after 3 experimental trials with different random seeds. Results using COT leverage outputs from ChatGPT (gpt-3.5-turbo). \n\nWe will share these new results in the appendix or our final manuscript. \n\n---\n\nWe appreciate your time and efforts to review our paper! We hope this response satisfies your concerns and resolves any potential ambiguity. Please let us know if our replies address your concerns.\n\n---\n### References\n___\n[1] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., ... & Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The two-step approach presented in this paper is a simple, and likely novel, idea that is relevant for a range of text classification tasks.", "reasons_to_reject": "- Limited reproducibility: ChatGPT, a commercial product, was used as the language model and it might not be possible for future work to reproduce the results. The authors do not indicate that they will share the LM outputs they collected.", "questions_to_authors": "A. How do COT Embeddings affect the time needed for inference, compared to direct classification?\n\nB. Appendix C: Consider reporting these numbers for the Tweet-Stance dataset as well.", "paper_topic_and_main_contributions": "This short paper is about stance detection on social media posts, where stance is often difficult to detect. The paper proposes to use a large language model, while not extracting the label directly from the LM output. Instead, a BERT classifier is trained to use the LM output as an additional (or alternative) feature.\nExperiments show an improvement over more simple baselines on political stance detection of English tweets.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an approach for stance detection in social media, using Chain-of-Thought (COT) embeddings. The paper does not extract the label directly from the LM output. Instead, a BERT classifier is trained to use the LM output as an additional (or alternative) feature. Reviewers commented that the approach is simple and likely novel, and the idea could be relevant to a range of text classification tasks.

The main concerns were limited reproducibility because of the use of ChatGPT. Experimental comparison was also limited. Also, certain details such as data distribution were missing. Authors have provided additional experiments in the rebuttal for some of these concerns, and these should be added to the main paper.
Decision: Accept-Findings