Multi-label and Multi-target Sampling of Machine Annotation for Computational Stance Detection
Authors: Zhengyuan Liu
Hai Leong Chieu
Nancy F. Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: stance detection
machine annotation
computational social science
TL;DR: In this paper, we investigate the efficacy of leveraging large language models on automated labeling for computational stance detection, and introduce a multi-label and multi-target sampling strategy to improve the machine annotation quality.

Abstract: Data collection from manual labeling provides domain-specific and task-aligned supervision for data-driven approaches, and a critical mass of well-annotated resources is required to achieve reasonable performance in natural language processing tasks. However, manual annotations are often challenging to scale up in terms of time and budget, especially when domain knowledge, capturing subtle semantic features, and reasoning steps are needed. In this paper, we investigate the efficacy of leveraging large language models on automated labeling for computational stance detection. We empirically observe that while large language models show strong potential as an alternative to human annotators, their sensitivity to task-specific instructions and their intrinsic biases pose intriguing yet unique challenges in machine annotation. We introduce a multi-label and multi-target sampling strategy to optimize the annotation quality. Experimental results on the benchmark stance detection corpora show that our method can significantly improve performance and learning efficacy.

Multi-label and Multi-target Sampling of Machine Annotation for Computational Stance Detection
Zhe Liu', Hai Leong Chieu', Nancy F. Chen Institute for Infocomm Research (TR), A **STAR, Singapore
*SS National Laboratories, Singapore liu siu_zhengyuanii2r.cm www..a.sstaeeeeeee edd. chaileon@dso org. sg nfychen@i2r 2r. a-star. edu.
Abstract
Inpul:
Target:
Data collection from manual labeling pro- vides domain-specific and task-aligned supervi- sion for data-driven approaches, and a critical mass of well-annotated resourees is required to achieve reasonable performance in natural language processing tasks. However, manual annotations often challenging to scale up in terms of time and budget, especially when domain knowledge, capturing subtle semantic features, and reasoning steps needed. In this paper, we investigate the efficacy of lever- aging large language models automated la- beling for computational stance detection. We empirically observe that while large language models show strong potential as alternative to human annotators, their sensitivity to task- specific instructions and their intrinsic biases pose intriguing yet unique challenges in ma- chine annotation. We introduce a multi-lahel and multi-target sampling strategy optimize the annotation quality. Experimental results on the benchmark stance detection corpora show that our method can significantly improve per- formance and learning efficacy.
Task- specific Instruction:
"Clasrtly the hawet' stance on 'Athelom" into ravor, Againe, Bons.
Annolded Sampli
LLMs
Lightwiisht language
Supervized Training
Model Generation: "The shence on the target is "svve""
Task Speoific Modads
Label
Figure 1: Overview of machine annotation via zero-shot inference with eeneral-purpose large language models.
Data-driven approaches largely benefit from the available human-annotated corpora, which provide domain-specifie and task-aligned supervision. De- veloping state-of thete-arr models via in-domain fine- tuning usually requires certain well-annotated train- ing data, such as machine translation and text sum- marization. However, manual annotations are chal- lenging scale up in terms of time and budget, pecially when domain knowledge, capturing subtle semantic features, and reasoning steps are needed. For instance, for computational stance detection, there are some corpora with target-awarc annota- tion (Mohammad et al., 2016; Allaway and Mcke- own, 2020; Li et al., 2021: Glandt et al., 2021), but they are limited in sample size, target diversity, and domain coverage. Learning from insufficient data leads low generalization out-of-domaii sam- ples and unseen targets (Allaway and Mckeown, 2020), and models are prone over-fit on superfi- cial and biased features (Kaushal et al., 2021).
1 Introduction
Stance detection is one of the fundamental social computing tasks in natural language processing, which aims to predict the attitude toward specified targets from piece of text. It is commonly for- mulated as a target- based classification problem (Küçükk and Can, 2020): given the input text (e.g.. online user-generated content) and a specified tar- get, stance detection models are used to predict a categorical label (e.g., Favor, Against, None) (Mo- hammad et al., 2016; Li et al., 2021). Upon so- cial networking platforms' growing impact lives, stance detection is crucial for various down- stream tasks such as fact verification and rumor detection, with wide applications including analyz- ing user feedback and political opinions (Chiluwa and Ifukor, 2015; Ghosh et al., 2019).
Recently, how the versatile large language mod- els (LLMs) would contribute to data annotation raised emerging research interest (He et al., 2023). Machine annotation also one approach to knowl- edge distillation from large and general models (Hinton et al., 2015), and student models can be lightweight and cost-effective to deploy. this pa- per, we leverage and evaluate LLMs on automated

---

labeling for computational stance detection. We empirically find that the general-purpose models can provide reasonable stance labeling, and show the potential as an alternative to human annotators. However, there are substantial variances from mul- tiple aspects, and models exhibit biases on target description and label arrangement (93). Such issues pose challenges regarding the quality and reliability of machine annotation. To optimize the annotation framework, we introduce the multi-label inference and multi targe- sampling strategy (54). Evaluation results on five benchmark stance detection corpora show that our method can significantly improve the performance and learning efficacy (85).
Instruction Prompt A:
Classify tweet stance (target-paaccholder] into "Favor", "Against", or "None".
Instruction Prompt B:
What is the following tweet's stance on placehorloo
Select one label from "Favor", "Aguinst", or "None". Instruction Prompt C:
Given the following Iweet, give its stance label [target- placeholder from "Favor", "Against". ", or "None".
Specified Target "utheism" Specified Target "no helief in gods"
Specified Target C: "religion" (w/ reversed label)
Specified Target D: "believe in god" (w/ reversed label)
Label Arrangement Favor, Against, None Label Arrangement Against, Favor, None Label Arrangement C: None, Favor, Against
Table I: Different settings for the machine anno- tation (in the Vicuna style). For the variation of target description, we select "arheism" from the SemEval-16 16 Task-6 A (Mohammad et al., 2016) one example.
2 Related Work
Computational stance detection aims to automati- cally predict polarities given the text and stance target. The model design for this task has been shifted from the traditional statistical methods such support vector machines (Hasan and Ng. 2013), to neural approaches such as recurrent neural net- works (Zarrella and Marsh, 2016), convolutional neural networks ( (Vijayaraghavan et al., 2016; Au- genstein et al., 2016; Du et al., 2017; Zhou et al., 2017), and Transformer (Vaswani et al., 2017). To facilitate the development of data-driven ap- proaches, human-annotated datasets for stance de- tection are constructed for model training and evaluation (Mohammad et al., 2016; Allaway and Mckeown, 2020; Li et al., 2021; Glandt et al., 2021). While the Transformer- based models, es- pecially pre-trained language backbones (Devlin et al., 2019; Liu et al., 2019), boost the perfor- mance on stance detection benchmarks (Ghosh et al., 2019; Li and Caragea, 2021), due to the limited training data and low labeling diversity, these supervised models showed degraded perfor- mance on unseen-target and out-of-domain evalua- tion (Kaushal& et al., 2021). To address this problem, recent work introduced de-biasing methods (Clark et al., 2019; Karimi Mahabadi et al., 2020), multi- task learning (Yuan et al., 2022), and the linguistics- inspired augmentation (Liu et al., 2023).
they outperform supervised state-of-the-art models, and even achieve comparable performance to hu- man annotators (Bang et al., 2023; Qin et al., 2023). In this section, we investigate the fcasibility and reliability of machine annotation for computational stance detection. To compare automated outputs with human annotation, our experiments con- ducted on well-annotated tweet stance detection corpus: SemEval- Task-6 (Mohammad et al., 2016), which serves as a benchmark many pre- vious works. For extensive comparison, we select three general-purpose LLMs: Alpaca-138 (Taori et al., 2023), Vicuna- (Chiang et al., 2023), and
3.1 Vanilla Machine Annotation Setup
The overview of the machine annotation process is shown in Figure 1. Since stance detection is commonly formulated as target-based classifica- tion task, the text sample and specified target are two elementary components of model input (All- away and Mckeown, 2020; Liu al., 2023). To leverage LLMs for automated stance labeling, we formulate task-specific instructions as the prompt (e.g., "Classify the tweet's 's stance on the target into Favor, Against, None."), and adjust their format accordingly to different prompt styles (see details in Appendix Table 4). Since generative system out- puts are not in a pre-defined label space, we adopt keyword matching and label re-mapping, and ob- tain the final model prediction. Moreover, to assess model performance under different settings, we do multiple inference rounds by changing prompts from three aspects, as shown in Table 1.
3 Machine Annotation for Computational Stance Detection
Based on large-scale pre-training and instruction tuning, language models are capable of solving various downstream tasks in a zero-shot manner (Ouyang et al., 2022). In many NLP benchmarks,

---

Alpaca-138
Vicune138
GPT-3.5-urbo
04
05
06
03
03
03
Intraction
Intrudts homptt
arrckl PomptC
19TH Pooke
Inrrecii hompt Intrdde
Figure 2: scores of 3-class stance label prediction under different task specific instructions. The average of human-annotation agreement and supervised training performance is highlighted in the red dotted line.
27
37
64
22
21
Specked TagicA Specliee Tugeca Spacled Tugact Spc Tageco
Sprdes Speitee Toger Sandinntog Eage!: "pstatlege
wolled laget spoctd lugit sped beget qpc lagotu
Figure Fl scores of 3-class stance label prediction on different variants of one stance object.
Libe A
Label 3
Labs AnigrnertC
Label Aniiyment
Label lndppmmnn
Label Aarignment
Lake Aeigeeettt
Figure 4: F1 scores of 3-class stance label prediction with 3 different label arrangements.
3.2 Vanilla Annotation Result Analysis
a certain variance of predicted labels under three different instructions.
Following previous work (Mohammad al., 2016; Allaway and Mckeown, 2020), we use the classi- fication metric Fll score to quantitatively measure the model performance. As shown in Figure 2, the general-purpose LLMs can provide reasonable zero-shot inference results, demonstrating the fea- sibility of machine annotation. In particular, the 5-turbo is comparable human annota- tors. Not surprisingly, it significantly outperforms Vicuna-138 and Alpaca-133 13B, since it has much larger parameter size and is further optimized via human-reinforved instruction tuming. On the other hand, we observe that models do not achieve consis- tent performance across different prompt settings (see details in Table 1), and the annotation accuracy is prone fluctuate from multiple aspects:
(2) Model predictions are sensitive to the target description. As stance objects can be in the form of entity, concept, claim, then assess the mod- els' robustness target variants. Here select "atheism" " as one example, and use three relevant terms "no belief gods", "religion", and "believe in god" for comparison. particular, labels of "atheism" " and "religion "/"believe in god" are reversed. For instance, if the original label of one text on "atheism" is Favor, then its label of "religion"/" belbevevve in god" is Against. As shown in Figure 3, models perform much better the target "religion" than the original one, and their results are different across the four variants.
stance
(3) Model predictions are sensitive the label arrangement. Current LLMS still suffer from ex- posure biases, such as positional bias in summa- rization, and majority label bias in classification (Fei et al., 2023; Wang et al., 2023). To evaluate the intrinsic bias label arrangement, keep the Instruction Prompt A fixed, but re-order the stance labels (i..., Favor, Against, None), As shown Figure the selected LLMs perform differently given the re-ordered labels, and we observed that prediction of the label None is more affected.
(1) Model predictions are sensitive to task- specific instructions. The nature of human in- struction results in diverse prompts for one specific task (Chiang et al., 2023), and there is gold standard. To simulate this scenario, paraphrase the instruction for stance detection and keep the prompts semantically identical. We then compare model generations by feeding rephrased instruction prompts. As shown in Figure 2, all models show

---

Spectke Tapgnt
Spatleed Spagie Dagge
Goop wentt
Labe Deggeeee
Figure S: scores of 3-class stance label prediction without (in blue) and with (in green) the two-hop instruction. We evaluate the model Vicuna-13B all three aspects that affect machine annotation performance.
Restrustion Stop
DDscrriba the tweet's explict or implott mantion "Athelsm"
Input Sample Text:
"Remooe the Nittomass from heart, Lard. Amon."
The Ewaet implicitty ralated to Athnism. The use of "Amon" raiglous exprassion.
Specified Target:
Specified Target:
Instruction Step
Clessily the
"Athetsm
None.
Implict Mention
Mention
Stance Label: Against
Stance Label: Favor
Multi Multi-iabel Inference
"The stance on the target Agolinze
Mutti-lbed and Multi-targal Sampling
Figure Overview the multi-label and multi-target machine annotation framework.
4 Improving Machine Annotation via Multi-label and Multi-target Sampling
(2) Diverse Sampling with Multi-target Predic- tion The SemEval-16 Task-6 A corpus only covers five stance targets (e.g.. penis cli- mate change, atheism), and each sample is associ- ated with one target with a human-annotee annotated stance label. To reduce superficial target-related patterns and biases from single target labeling, following previous work (Liu et al., 2023), we apply an adver- sarial multi-targe target sampling for machine annotation. First, off-the aff-the-shelf constituency parser is used to extract all noun phrases from the text. We then use these phrases as targets for stance labeling: if one phrase
The aforementioned labeling variances pose ques- tion mark on the reliability of machine annotation, and the noisy data cannot be readily used to train dedicated parameter-efficient models. Previous work proved that capturing target referral infor- mation and simulating human reasoning steps are beneficial for stance detection (Yuan et al., 2022; Liu et al., 2023). Therefore, we introduce two methods improve the machine annotation qual- ity. In this section, we evaluate the Vicuna-138 3B, as it is an open model and reaches a balance between performance and computational complexity.
contrary stance label to the original target, we include it as an additional sample. This adversarial multi-target sampling helps models to condition their prediction more on the given target, as well as learn some correlation between explicit and implicit objects. In our experiment, ob- tain 1.2k multi-target samples, where the original corpus size is 2914.
(1) Enhanced Multi-label Instruction One chal- lenge of computational stance detection comes from the ubiquitous indirect referral of stance ob- jects (Haddington et al., 2004). Social network express their subjective attitude with brevity and variety: they often do not directly mention the final target, but mention its related entities, events, or concepts. Inspired by chain-of-thought prompt- ing (Wei et al., 2022) and human behavior in stanc- etaking (Du Bois, 2007; Liu et al., 2023), to en- courage models to explicitly utilize the referral in- formation, we decompase the inference process to two steps. More specifically, as shown in Figure 6, before the stance label prediction, add one step describe the context-object relation (i.e., implicit explicit mention). With the enhanced two-hop instruction, Vicuna-13B 3B becomes comparable to the GPT-3. S-turbo (see Figure 5).
Leveraging Machine-annotatee Data for Model Training
Based on our improvements in Section 4, to com- pare the learning efficacy from machine-annotale and human-annotated data, we conduct experi- ments by training supervised model.
Experimental Setting As light- language backbones are more applicable to practical use cases, we select RoBERTa (Liu et al., 2019) which provides state-of ofitte--f-thh-eee the-art performance in many pre-

---

Model: RoBERTa-bar
Original Corpus Training
Vanilla Instruction Precision
Test
F1
Precision
Recall
Recall
SemEval-16 16 Task-6
0.6849
0,6755
0.7169
0.4686
0.5496
0.4838
Seméval-16 Task-6
0,4134
0.5132
0,4389
0.3969
0.5123
0.4442
P-Stance
0.3454
0.4840
0.3980
0.4171
0.5155
0.4369
VAST
0,4079
0.4215
0,4140
0.3232
0,4286
0.4119
Tweet-COVID
0.3579
0.4334
0.4032
0.3637
0.5150
0.4088
Model: RoBERTa-base
Enhanced Instruction
Malti-target Sampling Precision
Test Set
F1
Precision
Recall
F1
Recall
SemEval- 16 Task-6
0,5514
0.5445
0,5779
0,5553
0.5458
0,5824
1emEval-16 6
0.4968
0,5049
0.4975
0,5954
0.6118
0.585
P-Stance VAST
0.4769
0.5125
0.4992
0,4811
0,5116
0.4858
0.3909
0,4362
0.4317
0.6325
0.6443
0.6406
Tweet-COVID
0,5152
0.5271
0.5178
0.5554
0.5815
0,5638
Table 2: Supervised fine-tuning results of the 3-class stance classification. The single-domain training corpus is highlighted in blue, and the model is evaluated on multiple test sets. Macro-averaged F1, Precision, and Recall scores are reported. Training with human-annotated and machine-annotated data is highlighted in green and pink, respectively. Results of 2-class macro macro-average averaged scores are shown in Appendix Table 5.
vious works on stance detection (Liu et al., 2023; Zhao et al., 2023). We train the model on the SemEval-16 Task-6 A corpus, and collect five rep- resentative benchmarks to build the in-domain, out-of-domain, and cross-target evaluation settings (Küçük and Can, 2020), including SemEval- 16 Task-6 A and Task-6 (Mohammad et al., 2016), )), P- Stance (Li al., 2021), VAST (Allaway and Meke- own, 2020), and Tweel-COVID (Glandt et al., 2021). Model input is formulated as "
 (target) 
 <S> [context] 
", and the final hidden represen- tation of the first token "
" is fed to a linear layer and softmax function to compute the output probabilities. Experimental details are shown in Appendix Table 3 and Table 4. We the 3-class prediction scheme, as the None label necessary for practical use cases. Following previous work (Mohammad et al., 2016; Allaway and Mckeown, 2020), we adopt the macro- averaged FI, Precision, and Recall as evaluation metrics.
ative gain); this consistent with the improved annotation accuracy shown in Figure 5.
(2) The multi-target sampling can improve out- of of-domain and cross-target performance. The single-target data under vanilla and enhanced two- hop instruction only enable the model achieve reasonable results on in-domain samples and ex- isting targets. In contrast, the multi-target sam- pling brings substantial and consistent improve- ment on out-of-domain and unseen targets, where relative gains on SemEval-16 16 Task-6B and VAST are 19.8% and 61.2%, and it approaches 83% of supervised training on human-annotate datasets. This demonstrates that the model learns more gen- eral and domain-invariant features. We also port 2-class macro-averaged scores (i.e., Favor, Against), where the None label is only discarded during inference (see Appendix Table 5), and our methods bring performance elevation at all fronts.
Conclusions
Results and Analysis Since train the model on single-domain corpus, testing on out-of- domain samples and various unseen targets poses a challenging task. As shown Table 2, compared with training on numan-annotater data, applying machine annotation upon the vanilla instruction in Section results in lower scores, especially on the SemEval- 16 Task-6 A; F1, precision, and recall scores are all affected. When applying our pro- posed methods, observe that:
In this work, we investigated the potential and chal- lenges of leveraging large language models on au- tomated labeling for computational stance detec- tion. Quantitative analyses demonstrated the fea- sibility, but their predictions are sensitive to task- specific instructions, and LLMs inevitably present exposure biases on target description and label ar- rangement. We improved machine annotation by adopting multi-label and multi-target sampling strategy. Experimental results on several stance detection benchmarks showed the effectiveness of our proposed methods. This work can shed light on further extensions of machine annotation.
(1) The multi-label instruction can improve in- domain performance. Our proposed two-hop in- struction provides training data with higher quality, particularly in the in-domain evaluation (17.6% rel-

---

Limitations
Ji. Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt reasoning. hallucination, and interactivity. arXiv preprint arXiv:2302.04000
All samples used in this work are in English, thus to apply the model to other languages, the result might be limited by the multilingual capability of language backbones. Moreover, we are aware that it remains an open problem to mitigate biases in human stancetaking. Of course, current models and laboratory experiments are always limited in this or similar ways. We do not foresee any unethical uses of our proposed methods or their underlying tools, but hope that it will contribute to reducing incorrect system outputs.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Innocent Chiluwa and Presley Ifukor. 2015. 'war against our children': Stance and evaluation in# bringbackourgirls campaign discourse twitter and facebook. Discourse & Society, 26(3):267-299
Ethics and Impact Statement
Christopher Clark. Mark Yatskar, and Luke Zettlemoyer. 2019. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLPP pages 4069- 4082. Association for Computational Linguistics.
We acknowledge that all of the co-authors of this work are aware of the provided ACL Code of Ethics and honor the code of conduct. All data used in this work are collected from existing published NLP studies. Following previous work, the anno- tated corpora are only for academic research pur- poses and should not be used outside of academic research contexts. Our proposed framework and methodology in general do not create direct soci- etal consequence and are intended to be used to pre- vent data-driven models from over-fitting domain- dependent and potentially-biaeed features.
Jacob Devlin, Ming-W Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 4171-4186.
Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural atten- tion networks. International Joint Conferences on Artificial Intelligence.
Acknowledgments
This research was supported by funding from the Institute for Infocomm Research (I²R), A*STAR, Singapore, and DSO National Laboratories, Singa- pore. We thank Yong Kcong Yap and Wen Haw Chong for the research discussions. We also thank the anonymous reviewers for their precious feed- back to help improve and extend this piece of work.
John W Du Bois. 2007. The stance triangle. Stancetak- ing discourse: Subjectivity, evaluation, interaction, 164(3):139-182
Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut, 2023. Mitigating label biases for in-context learning. arXiv preprint arxiv:2305.199 19148.
Shalmoli Ghosh, Prajwal Singhania, Siddharth Singh, Koustav Rudra, and Saptarshi Ghosh. 2019. Stance detection weh and social media: a comparative study. In International Conference of the Cross- Language Evaluation Forum for European Lan- guages, pages 75-87. Springer.
References
Emily Allaway and Kathleen Mckeown. 2020. Zero- shot stance detection: A dataset and model using generalized topic representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8913-8931.
Kyle Glandt, Sarthak Khanal, Yingjie Li, Doina Caragea, and Cornelia Caragea. 2021. Stance detec- tion COVID- 19 tweets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1596-1611, Online. Association for Computational Linguistics.
Isabelle Augenstein, Tim Rocktäschel, Andreas Vla- chos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 876-885. As- sociation for Computational Linguistics.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei
Pentti Haddington et al. 2004. Stance taking news interviews. SKY Journal of Linguistics, 17:101-142.

---

Kazi Saidul Hasan and Vincent Ng. 2013. Stance clas- sification of ideological debates: Data, models, fea- tures, and constraints. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1348-1356, Nagoya, Japan. Asian Federation of Natural Language Processing.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27777
Xingwei He, Zhenghao Lin, Yeyun Gong, A Jin, Hang Zhang. Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. 2023. Annollm: Making large language models to be better crowdsourced annota- tors. arXiv preprint arXiv:2303. 16854.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver? arXiv preprint arXiv:2302.06477
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-folloingg llama model. https:/ gi thub. wwt.:/cccaaaaaaaeeee wattss-aabssttaorrd.aa
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arX.i:1503.0255:5
Rabech Karimi Mahabadi, Yonatan Belinkov, and James Henderson, 2020. End-to-end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Association for Compu- lational Linguistics. pages 8706-8716 8716, Online. Asso- ciation for Computational Linguisties.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, In Advances in Neural Information Processing Systems, pages 6000-6010.
Ayush Kaushal, Avirup Saha, and Niloy Ganguly. 2021. twt-wt: A dataset to assert the role of target entities for detecting stance of tweets. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3879-3889.
Prashanth Vijayaraghavan, Ivan Sysoev, Soroush Vosoughi, and Deb Roy. 2016. Deepstance at semeval-2016 task 6: Detecting stance in tweets us- ing character and word-level cnns. arXiv preprint 17X6v:6006.0560
Dilek Küçük and Fazli Can. 2020. Stance detection: survey. ACM Computing Surveys (CSUR), 53(1).
Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao. Yang Ding, Ai Ti Aw, and Nancy F Chen. 2023. Scaeval for multilingual foundation models: From cross-lingual alignment cultural reasoning. arXiv preprint arXiv:2309.04: 04766.
Yingjie Li and Cornelia Caragea. 2021. Target-aware data augmentation for stance detection. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le. Denny Zhou, et al. 2022. Chain-of-thougee prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-2455 24837.
Yingjie Li, Tiheriu Sosea, Aditya Sawant, Ajith Jayara- man Nair, Diana Inkpen, and Cornelia Caragea. 2021. P-stance: A large dataset for stance detection in polit- ical domain. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 2365.
Jianhua Yuan, Yanyan Zhao, Yanyue Lu, and Bing Qin. 2022. Ssr: Utilizing simplified stance reasoning pro- cess for robust stance detection. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6846-6858.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 10::1190 1907. 1162..
Guido Zarrella and Amy Marsh. 2016. Mitre at semeval- 2016 task 6: Transfer learning for stance detection. arXiv preprint arXiv: 1606.03784
Zhengyuan Liu, Yong Keong Yap, Hai Leong Chieu, and Nancy Chen. 2023. Guiding computational stance detection with expanded stance triangle frame- work. Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 3987-4001, Toronto, Canada. Association for Computational Linguistics.
Chenye Zhao, Yingjie Li, and Cornelia Caragea. 2023. C-stance: A large dataset for chinese zero-shot stance detection. In Proceedings of the 61 Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13369-133333
Yiwei Zhou, Alexandra Cristea, and Lei Shi. 2017. Connecting targets to tweets: Semantic attention- based model for target-specific stance detection. In International Conference on Web Information Sys- tems Engineering, pages 18-32. Springer.
Saif Mohammad, Svetlana Kiritchenko. Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-200 2016),, pages 31 41. Association for Computational Linguistics.

---

Corpus
Original Targets for Stance Labeling
Train
Valid
Test
1emHval-16 Task-6 A
Atheism, Climate Change, Feminist Movement, Hillary
2914
1249
Clinton, Legalization of Abortion
1e Task-6
Donald Trump (for zero-shot cvaluation)
707
P-Stance
Donald Trump, Joe Biden, Bemie Sanders
19228
2462
2374
VAST
Various Targets by Human Annotation
13477
2062
3006
Tweer-COVin
Keeping Schools Closed, Dr. Fauci, Stay at Home Orders,
4533
800
800
Wearing a Face Mask
Table 3: Statistics of the collected stance detection datasets for model training and evaluation.
Machine Annotation
Experimental Configuration
Alpaca-13B
Below is instruction that describes task, paired with input that provides further context. Write response that appropriatcly completes the request. ### Instruction: Select stance label from "Favor", "Against", "None", with short sentence.
### Input:
Given the tweet: (tweet-placcoodde)) what
the target (target-
placeholder)?
#HHi Response: (model-output)
Vicuna-13B
Classify a tweet stance on placcholaacchhldrrr into "Favor", "Against", or "None". Tweet: [tweet-placeholdor)
Stance: /modcl-output)
Vicuna-13B
Target Referral Instruction:
Describe tweet's explicit or implicit relation to target (larget-placchoolder)
Tweet: [tweet-placeholder]
Answer: (mudel-oulput)
GPT-3.5-turbo
Given tweet: plwcct-lacchooldr), what is its stance to the target harget-placeholders Select one label from "Favor", "Against", and "None", with one short sentence.
Environment Details
GPU Model
Single Tesla A100 with GB memory; CUDA version Pytorch==1.. 1.13.1: Transformers=42266
Library Version
Computational Cost
LLMs inference (Alpaca-13B Vicuma-13B), average evaluation hour one round. For GP1-3.5-turbo, we utilize the OpenAl official API. For supervised training, the average training
hours for one round. Average rounds each reported result (calculating the mean of scores).
Dataset Processing
Details
Corpus
The datascts we used for training and evaluation are from published works (Mohammad ct al., 2016; Allaway and Mckeown, 2020; Li et al., 2021; Glandt et al., 2021) with the Creative Commons Attribution 4.0 Interaatoonal license.
Pre-Processing
All samples are in English, and only for rescarch use. Upper-case, special tokens, and hashtags are retained.
Supervised Training RoBERTa-base
Experimental Configuration
RoBERTa-bra (Liu el al., 2019)
Base Model: Transformer (12-layer. 768-hidden, 12-heads, 125M parameters).
Learning Rate: 2e-5, AdamW Optimizer, Linear Scheduler: 0.9.
Table Details of experimental environment and hyper-paramest setting.

---

Model: RoBERTa-base
Original Corpus Training
Vanilla Instruction Precision
Set
F1
Precision
Recall
Recall
SemEval-16 Task-6 SemEval-16 Task-6 B
0.7023
0.7047
0.7318
0.5826
0.5376
0.6496
0.3143
0.5126
0,2814
0.5224
0,4526
0.6432
P-Stance
0.4436
0.6597
0.5118
0.6017
0.6436
0.6442
VAST
0.3905
0.4192
0.3906
0.4419
0.4185
0.5711
Tweet-COVID COVID
0.2575
0.4377
0.1966
0.3409
0.3432
0.4788
Model: RoBERTa-base Test Set
Enhanced Instruction Precision
Multi-target Sampling Precision
Recall
Recall
SemEval-16 16 Task-6 A SemEval-16 Task-6 B
0,5856
0,5935
0.6037
0.6034
0.6071
0.5911
0.5318
0.5111
0.5578
0.5905
0.6085
0.5813
P-Stance VAST
0.5992
0.6747
0.5968
0.6431
0,6765
0.6596
0.4867
0.4312
0.5834
0.5659
0.6168
0.5382
TWeeI-COVID
0.4711
0.4625
0.4978
0.4791
0.5346
0.4460
Table 5: Results of the 2-class stance classification multiple corpora. Macro-averaged Fl, Precision, and Recall scores are reported. Results 3-class macro-averaged scores are shown Table 2.


Reviews:
[{"review_id": "ULvbLYKvhs", "reviewer": "Reviewer_jYAZ", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **1.1) Our contributions**\nIn this work, we focus on evaluating and improving the quality of machine annotation for computational stance detection.  \nWe first **discussed the inconsistency of machine annotation from three aspects  (Section 2)**: the ways of instruction, the expression of targets, and the sequence of labels. Alterations in any of these aspects result in prediction variation. This motivates us to reduce the prediction inconsistency observed in machine annotation.  \nTherefore, we then adopted the multi-label inference and the multi-target sampling strategy **to optimize the annotation framework (Section 3)**.  \nFinally, to demonstrate the effectiveness of our strategies and the reliability of the machine-annotated data generated by our methods, we **fine-tuned a task-specific model on machine-annotated data, and evaluated its performance on five stance detection test sets (Section 4)**. The results clearly show that the performance of the model trained on our improved machine-annotated data is on par with that trained upon human annotation, strongly demonstrating the effectiveness of our proposed strategies to improve the quality of machine annotation.  \nWe’re not doing/comparing two approaches as the reviewer commented, i.e., zero-shot stance detection and fine-tuning.  \n\n#### **1.2) For the figures shown in Section 2**\nBasically, Figure 2, Figure 3, and Figure 4 are to show the inconsistency in machine annotation. The substantial variation showed up just by changing the ways of instruction (Figure 2), expression of targets (Figure 3), and the sequence of labels (Figure 4), which is surprising and not reported before. We apologize for the confusion, but we believe **the patterns shown in the figures clearly demonstrate our motivation to improve the quality of machine annotation.**  \n\nMore specifically, for your quick reference, here are the 3 human-written instruction prompts (Figure 2) following the Vicuna format (one is shown in Appendix Table 3):  \n> Instruction Prompt A:  \nClassify a tweet stance on {target-placeholder} into “Favor”, “Against”, or “None”.  \n> Instruction Prompt B:  \nWhat is the following tweet’s stance on {target-placeholder}? Select one label from “Favor”, “Against”, or “None”.  \n> Instruction Prompt C:  \nGiven the following tweet, give me its stance label on {target-placeholder} from “Favor”, “Against”, or “None”.  \n\nThe target examples in Figure 3 (Section 2) are:  \n> “atheism” (Specified target A)  \n> “no belief in gods” (Specified target B)  \n> “religion” (Specified target C)  \n> “believe in god” (Specified target D)  \n\nThe label sequences in our experimental setting (Figure 4) are (the label order is one instance of label assignment):   \n> “Favor, Against, None” (Label Assignment A)  \n> “Against, Favor, None” (Label Assignment B)  \n> “None, Favor, Against” (Label Assignment C)  \n\nNoteworthy, the model prediction variance is **not only related to the above specified instructions**, even small changes to the prompt also make a difference (e.g., change “a tweet” to “the tweet”). We found that model predictions are not consistent when different prompts are provided (even ChatGPT), which poses challenges when machine annotation is applied.  \n\nWe’ll revise these figures to help the audience better understand.  \n\n#### **1.3) For the two proposed methods in Section 3**\nIn Section 3, we do both “improving zero-shot approach” as well as “data augmentation” to improve the machine annotation for computational stance detection.  \n+ To improve the accuracy and reduce the prediction variation in machine annotation, we conducted a “two hop” inference process by adding one instruction for describing the context-object relation.  \n+ To improve the learning efficiency of the machine annotation, we conducted a diverse sampling with adversarial multi-target labeling (in the last paragraph of Section 3).  \n\n#### **1.4) For the follow-up fine-tuning experiment in Section 4**\n**Section 4 is to train a stance detection model using the machine-annotated data generated in Section 3, and evaluate the performance of learning from machine annotation.**  \nThe reviewer understood the results shown in Table 1 correctly, but we want to emphasize that these experiments were conducted NOT to repeat the machine annotation process, BUT to prove that training on machine-annotated data can produce reasonable results on several benchmark datasets, which highlighted the effectiveness of our proposed methods on machine annotation.  \n\n#### **2) For the “learning efficiency” of training on machine-annotated data**\n**The “learning efficiency” is evaluated on the relatively small number of machine-annotated training data.**  \nIn Section 3, the machine-annotated corpus size of multi-target samples using our proposed methods is 1.2k (see Line 186), whereas the original human-annotated corpus size is 2.9k (2914). In the follow-up stance detection experiments, we fine-tuned the Roberta model that is trained on the 1.2k machine-annotated corpus, and the results on multiple test sets showed its comparable performance to that trained on the 2.9k human-annotated data (see Line 235), clearly demonstrating the promoted “learning efficiency” of our methods.  \n  \nWe will expand on this point in the revised manuscript.  \n\n#### **3) For the concern of test data leakage from pre-training models**\nWe don’t think the test data leakage will affect the results presented in our work, though it is a general concern.  \n+ For our machine annotation experiments, Alpaca and Vicuna are Llama-based open-source models [1] trained via general-purpose instruction-tuning, and no stance detection corpora are directly used in their training process. Compared with LLMs that are instruction-tuned on various existing NLP datasets (e.g., FLAN-T5 [2]), our model selection has less concern about the test data leakage issue.   \n+ **If the models presumably saw the data before, it makes the observed inconsistency issue more pronounced.** This indeed highlights our contribution to the field by bringing this issue and providing one solution to improve the sample quality of machine annotation.  \n\n#### **4) For combining data from human and machine annotation**\nIn the practical cases, there would be two situations: large-scale human annotation is available or unavailable. As human annotation is time-consuming to scale and cover various domains (most existing stance detection corpora are only labeled on a few stance targets), machine annotation would serve as a bootstrapping step or an alternative. In case people would like to combine human-annotated and machine-annotated data, the quality of machine-annotated data must be on par with the human-annotated ones, otherwise noise will be introduced. In either case, **the quality of machine annotation needs to be taken into account, which is exactly the aim of our work.**  \n\n\n#### **5) For the reproducibility of our work**\nWe don’t fully understand why the reviewer claims _“I'm quite sure the idea being described here could not be reproduced.”_  \nIn fact, we will release our code and data for both the machine annotation and stance detection experiments. As we used the open-source models Alpaca and Vicuna for machine annotation, and Roberta for stance detection, we believe that **interested readers will be able to reproduce our experiments following our paper and code.**  \nMoreover, the observation of the prediction variation and our methods are not limited to the models and corpora we evaluated in this paper. This work can be readily extended to other models and various data for computational stance detection.  \n\n#### **6) For typos, table format, and missing references**\nThanks for your suggestions on the typos, table format, and missing references, we will revise them accordingly in the manuscript.  \n\n####\n####\n**We hope the above points address the comments well and the reviewer can appreciate the contributions of our work, and we will greatly appreciate it if the reviewer can re-rate the scores.**  \n\n####\n#### **Reference:**\n[1] Touvron, Hugo, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière et al. \"Llama: Open and efficient foundation language models.\" arXiv preprint arXiv:2302.13971 (2023).  \n[2] Chung, Hyung Won, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li et al. \"Scaling instruction-finetuned language models.\" arXiv preprint arXiv:2210.11416 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "While I appreciate the paper's focus on one worthwhile problem, there was unfortunately very little about the proposed approach that was clearly motivated by, or obviously unique to, the stance detection task.", "reasons_to_reject": "The experiments in section 2 are extremely unclear.  In figures 2-4 the independent variables are never fully defined, nor are their values (the figure unhelpfully labels them \"instruction prompt A,\" \"specified target B,\" and so on -- the labels are never aligned to an explanation of the design choice).  The text claims, for example, that \"mdoels perform much better on the target religion than the original one\" but it's not clear which target among A-D is \"religion\" vs \"the original one,\" or what the original one was.  All the reader is told about the different instructions is \"we adjust task-specific instructions according to their format (see appendix table 3).\"  I really couldn't make any sense out of what the experiments show, because the explanation was so vague. \n\nBenefits to \"learning efficiency\" are claimed in the abstract and intro but never discussed again.\n\nA major limitation of this work is its partial dependence on closed models.  The authors/reviewers have no way to verify that the evaluation data was excluded from the pretraining data.\n\nI couldn't follow what was being presented in the last paragraph of section 3.  At first I thought this was going to be another way to improve the zero-shot approach.  But it sounds more like a set of transformations on the original data (i.e., some kind of data augmentation technique).  I couldn't understand what the authors were trying to do, and the claim that it \"helps models to condition their prediction more on the given target as well as learn some correlation between explicit and implicit objects\" isn't connected, as far as I can tell, to any experiments.\n\nSection 4 is entirely disconnected from the rest of the paper's exposition; it turns to finetuning Roberta (a completely different type of model) to carry out the same task, but now evaluating on a wider range of datasets.  What I take away from the experiments here (table 1) is that training on silver data (I think, the labels produced by the models in section 3?  this is not made perfectly clear) is worse than training on human annotations, but the gap narrows if the proposed techniques are used.\n\nA major gap in the experimental story:  what about combining the gold data with the silver data?  That's what's usually done when models are used to automatically produce training data for supervised learners, and it's more relevant to real world practice.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is about stance detection.  There are two different approaches discussed.  The first explores zero-shot stance detection with a range of different models, showing first that diffferent instruction wording, different target entities (i.e., toward who/what is the author's stance to be determined?), and the ordering of the possible labels (in the prompt) can all have a notable effect on the accuracy.  A \"two hop\" approach to prompting is introduced and shown to improve performance of one of the language models (vicuna) across these settings.  The second approach turns to finetuning the Roberta model for the task, varying the instructions and substituting automatically generated data for the original training data.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The \"enhanced\" instruction approach looks a lot like chain of thought prompting, exemplified by this paper:  https://arxiv.org/abs/2201.11903  ... it would be appropriate to discuss the connection to those kinds of approaches or to prompt engineering.", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 133:  \"sensitive to the label assignment\" -- this is confusing, it would be better to call this \"the label order in the prompt\"\n\nThe results in table 1 make it very hard to compare apples to apples and draw conclusions.  Perhaps a collection of scatterplots (one per test set), with recall and precision as the axes, and different colors showing alternate training set construction methods, would make this easier to understand?"}}, {"review_id": "2LQN5841bu", "reviewer": "Reviewer_5SF3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **1) Our contributions**\n  In this work, **we focus on assessing and improving the quality of machine annotation for computational stance detection**.  \n  We first brought up **the inconsistency of machine annotation** based on our experiments (Section 2), which would significantly affect the robustness yet has not drawn enough attention. By changing the ways of instruction, the expression of targets, and the sequence of labels, we found any alterations resulted in prediction variation. This thus motivates us to reduce the prediction variance introduced by machine annotations.  \nTherefore, we then adopted multi-label inference and multi-target sampling strategies **to optimize the machine annotation framework** (Section 3).  \nFinally, to demonstrate the effectiveness of our strategies and the reliability of the machine-annotated data generated by our methods, we fine-tuned a task-specific model on our annotated data, and evaluated its performance on five stance detection test sets (Section 4). The **results clearly show that the performance of the model trained on our improved machine-annotated data is on par with that trained upon human annotation**, strongly demonstrating the effectiveness of our proposed strategies to improve the quality of machine annotation.  \n\n#### **2) For the novelty of our work**\n**The robustness of machine annotation is important yet has not drawn enough attention.** In this work, to strategically improve machine annotation for computational stance detection, our proposed methods are based on our thorough analyses of prediction variations, and we extend theory-inspired practice from human annotation [1][2] to machine annotation.  \n\n\n#### **3) For the presentation improvements (figures/tables readability)**\nThanks for your suggestions!  \nWe will (1) specify the “instruction prompt”, “specified target”, and “label assignment” in the corresponding figures, and (2) highlight the best results in Tables 1 and 4.  \n\nFor your quick reference, here are the 3 human-written instruction prompts (Figure 2) following the Vicuna format:  \n> Instruction Prompt A:  \nClassify a tweet stance on {target-placeholder} into “Favor”, “Against”, or “None”.  \n> Instruction Prompt B:  \nWhat is the following tweet’s stance on {target-placeholder}? Select one label from “Favor”, “Against”, or “None”.  \n> Instruction Prompt C:  \nGiven the following tweet, give me its stance label on {target-placeholder} from “Favor”, “Against”, or “None”.  \n\nThe target examples in Figure 3 (Section 2) are:  \n> “atheism” (Specified target A)  \n> “no belief in gods” (Specified target B)  \n> “religion” (Specified target C)  \n> “believe in god” (Specified target D)  \n\nThe label sequences in our experimental setting (Figure 4) are (the label order is one instance of label assignment):   \n> “Favor, Against, None” (Label Assignment A)  \n> “Against, Favor, None” (Label Assignment B)  \n> “None, Favor, Against” (Label Assignment C)  \n\nNoteworthy, the model prediction variance is **not only related to the above specified instructions**, even small changes to the prompt also make a difference (e.g., change “a tweet” to “the tweet”).\n\n#### **4) For the code and data availability**\nWe will release our code and data for both the machine annotation and stance detection experiments. As we used the open-source models Alpaca and Vicuna for machine annotation, and Roberta for stance detection, we suppose that interested readers will be able to reproduce our experiments following our paper and code.  \n\n####\n####\n**We thank the Reviewer 5SF3 for supporting the soundness and reproducibility of our work.  \nHopefully we have addressed all the comments/questions, and the focus/contribution of our work is better delivered/explained.**  \n\n####\n#### **Reference:**\n[1] Ayush Kaushal, Avirup Saha, and Niloy Ganguly. 2021. twt–wt: A dataset to assert the role of target entities for detecting stance of tweets. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3879–3889.  \n[2] Zhengyuan Liu, Yong Keong Yap, Hai Leong Chieu, and Nancy Chen. 2023. Guiding Computational Stance Detection with Expanded Stance Triangle Framework. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 3987–4001, Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Sound experimental setup\n- Good reproducibility thanks to the use of an openly available language model. The authors indicate that they are planning to share their code and/or data, but I could not verify this as no supplementary material was shared.\n", "reasons_to_reject": "- Lack of focus: I did not have the impression that this short paper makes a focused contribution, but rather it seems to explore several techniques in a relatively superficial manner. Unfortunately, there is little analysis of the different approaches.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This short paper explores the use of language models for labeling stance detection training data. The paper compares different zero-shot labeling approaches (multi-hop, different instructions), as well as automatically augmenting the training data with additional targets (Liu et al., ACL 2023). The authors then evaluate a classifier trained on those weakly labeled data, showing that the techniques improve the cross-target generalization of the classifier.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- I find Figures 2–4 hard to understand. Consider making the figures more self-contained.\n- I find it difficult to compare the four approaches in Tables 1 and 4. Consider changing the table layout to make comparison easier, for example by providing averages.\n"}}, {"review_id": "tYLYm7MdW7", "reviewer": "Reviewer_KnpT", "summary": "", "questions": "1. In line 180 - could you please elaborate on how you create these samples, maybe provide an example.\n2.  Regarding Figure 5 and all other measurements, how many runs were done per measurement? do you have confidence intervals for your measurements?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **1) For the adversarial multi-target labeling process**\nThe adversarial multi-target labeling basically means to **select the explicit object that shows a stance dis-alignment to the specified target**. Previous work (Liu et al., 2023) has demonstrated its effectiveness **to reduce superficial target-related patterns and biases from single target labeling** (the issues resulting from low data diversity and spurious correlation [1][2]), and emphasize object-object relationship.  \n\nFor the detailed adversarial multi-target labeling process, here is one example:  \n\n```\nGiven the tweet “So can unborn children have rights now?” with the original specified target “abortion” for stance analysis, 4 steps are used to generate multi-target labeling:  \n\n>>> Step 1: The machine annotator predicts a label “AGAINST” of the implicit target “abortion”.  \n\n>>> Step 2: We run an off-the-shelf constituency parser to extract the noun phrases (explicit targets) from the tweet. In this example they are “unborn children” and “rights”.  \n\n>>> Step 3: For both of the explicit targets “unborn children” and “rights”, the machine annotator predicts the stance label “FAVOR”, which is a contrary label to the original specified target “abortion”. If no such explicit target is found, Step 4 will be skipped.   \n\n>>> Step 4: We randomly select one of the explicit targets with the contrary label (e.g., “unborn children”), and then obtain an adversarial multi-target labeling of the tweet (“abortion” -> “AGAINST”, “unborn children” -> “FAVOR”).  \n\n```  \n####\nAside from the framework overview shown in Figure 6, we will illustrate the above example in appendix.\n\n####\n#### **2) For the experimental setting of running times**\nThe experiments were run three times, and the mean value is reported in this manuscript. These setups are included in the “Environment Details” section in Appendix Table 3 on Page 7. We could include the means ± SDs in the result tables.  \nGiven the reproducibility, the code and data described in this manuscript will be released in our final version upon acceptance.  \n\n#### **3) For the presentation improvements (figures/tables readability)**\nThanks for your suggestions!  \nWe will (1) specify the “instruction prompt”, “specified target” and “label assignment” in the corresponding figures; (2) include GPT3.5 performance data in Figure 5 for better comparison; and (3) highlight the best results in Table 1.  \n\n####\n####\n**We thank the Reviewer KnpT for the positive comments of the novelty and soundness of our work.  \nHopefully we have addressed all the questions. We believe our work can shed light on the future studies of machine annotation.**  \n\n####\n#### **Reference:**\n[1] Shalmoli Ghosh, Prajwal Singhania, Siddharth Singh, Koustav Rudra, and Saptarshi Ghosh. 2019. Stance detection in web and social media: a comparative study. In International Conference of the CrossLanguage Evaluation Forum for European Languages, pages 75–87. Springer.  \n[2] Ayush Kaushal, Avirup Saha, and Niloy Ganguly. 2021. twt–wt: A dataset to assert the role of target entities for detecting stance of tweets. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3879–3889.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Show how LLMs work for automated labeling, and show what aspects affect their performance. In addition, they show a model's performance when finetuning with machine annotations: they show an improvement in the performance for out-of-domain corpora compared to the model trained on human annotations.", "reasons_to_reject": "I would hope to get some clarifications for my questions and some of my suggested edits to be implemented. Given that clarifications make sense, I don't have strong reasons for the rejection. ", "questions_to_authors": "1. In line 180 - could you please elaborate on how you create these samples, maybe provide an example.\n2.  Regarding Figure 5 and all other measurements, how many runs were done per measurement? do you have confidence intervals for your measurements?\n", "paper_topic_and_main_contributions": "This work proposes the usage of LLM as an alternative to human annotations and justifies this by conducting experiments on 3 LLMs. Their task setup is target based classification problem. The authors demonstrate what affects models' predictions. Also, the authors finetune Roberta and evaluate it on 5 benchmark corpora.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In descriptions in Fig 2-4, I would specify what you mean by A, b, and C for each figure so that it is readable without looking at the specific part of the text. Scrolling back and forth is annoying.\n2. Line 170 - if the model becomes comparable to gpt 3.5 and this is an important claim for the paper, please, add gpt bar in Figure 5. (Again this would improve the readability)\n3. For the readability of Table 1 please for each measure highlight the best result (for example, use the bold font) \n"}}]

Author rebuttal:

Meta-review: The paper explores the feasibility of using LLM for data annotation, using the task of computational stance detection as a case study.  The authors find that while LLMs like Alpaca, Vicuna, and GPT-3.5 turbo achieve reasonable performance on a benchmark dataset for stance labeling, models are sensitive to variation along a number of dimensions (such as prompt and label) and show evidence of intrinsic bias.  They then test a multi-target multi-label setup to improve robustness in their annotation task, finding that training on additional machine-labeled examples improves out-of-domain and cross-target performance.

Overall, the paper presents a solid experiment on the use of machine-labeled data.  An initial critique of the paper across reviewers is that the main focus of the paper is unclear in its current state.  Reviewer 5SF3 stated that they did not find that “this paper makes a focused contribution”, citing the fact that it seemed to run a number of experiments on using models to label stance without very much analysis on any given variation in approach.  While it isn’t clear what exact experiments the reviewer found unclearly motivated or analyzed, reviewer jYAZ similarly felt that section 4, which ran the finetuning experiments on RoBERTa, felt disconnected from the rest of the paper, as it suddenly seemed to be evaluating a new model on a wider range of datasets.  While I understood the purpose of section 4 to be testing the actual performance of models trained on data extracted from the multi-target multi-label annotation setup, the presentation of this section is unclear in its current form.  While the paper is a short paper, and thus, has stricter page constraints, ideally the paper would provide more motivation for a. why they are training the RoBERTa models and b. why they are examining performance across a variety of stance datasets, some of which are out-of-domain from the original training set of SemEval-16 Task-6 A.

Another issue with clarity that both reviewers KnpT and jYAZ identified was in the presentation of the results in Figures 2-4, which show the F1 of 3-class stance prediction using 3 different LLMs.  Reviewer KnpT, for example, found it frustrating to have to refer back to the text to understand what the figures were describing.  Reviewer jYAZ found it unclear what was meant by Specified Target A, Instruction Prompt A, … etc.   While the authors clarify in their rebuttal what each of the dimensions on the X-axis of figures 2-4 mean, these should ideally be included in some form in the paper (either in the main body or appendix).

Finally, one issue that may affect the soundness of the paper is the issue of potential test data leakage in the pre-trained models used for annotation.  While the authors argue in their rebuttal that none of the open-source models explicitly include stance-detection corpora in their instruction-tuning process, there is still the potential for data leakage outside of instruction-tuning.  The potential presence of test data in the pretraining data for the LLMs performing annotation does raise concerns over the true robustness and extensibility of using LLM annotation.  The performance improvement of the finetuned-RoBERTa models on out-of-domain test sets, for example, may be attributed to the LLM annotation scheme reproducing the label distribution of data it should not have already seen.  While the authors argue that the instability between the vanilla instruction and the multi-target/multi-label settings provides some evidence against the potential data leakage issue, it’s hard to attribute the instability across different settings directly to the presence of data leakage, as LLMs are notoriously sensitive to how instructions are presented.  While there’s not a feasible approach to directly address the data leakage issue, a more convincing argument that the machine-annotation method is robust to out-of-domain data would be to repeat the section 4 analysis, taking each of the original “out-of-domain” datasets as the basis of their training corpora once and testing every other dataset against the model trained on the “leave-one-out” dataset.
Decision: Accept-Findings