Definitions Matter: Guiding GPT for Multi-label Classification
Authors: Youri Peskine
Damir Korenčić
Ivan Grubisic
Paolo Papotti
Raphael Troncy
Paolo Rosso
Venue: EMNLP 2023 Findings (2023)
Keywords: GPT-3
zero-shot classification
LLM
TL;DR: We analyze the importance of label definition in GPT zero-shot labeling and introduce a new method to generate definition from examples

Abstract: Large language models have recently risen in popularity due to their ability to perform many natural language tasks without requiring any fine-tuning. In this work, we focus on two novel ideas: (1) generating definitions from examples and using them for zero-shot classification, and (2) investigating how an LLM makes use of the definitions. We thoroughly analyze the performance of GPT-3 model for fine-grained multi-label conspiracy theory classification of tweets using zero-shot labeling. In doing so, we asses how to improve the labeling by providing minimal but meaningful context in the form of the definitions of the labels. We compare descriptive noun phrases, human-crafted definitions, introduce a new method to help the model generate definitions from examples, and propose a method to evaluate GPT-3's understanding of the definitions. We demonstrate that improving definitions of class labels has a direct consequence on the downstream classification results.

Definitions Matter: Guiding GPT for Multi-label Classification
Youri Peskine*
Damir Korenčić*
Ivan Grubišié
EURECOM
Univ. Politécnica de València (UPV)
UPV
vouri.peskineeurecco..
fr shorenc@uppneet.uy.c
Ruder Bošković Institute
grubisic@irb.hr
Paolo Papotti EURECOM paolo. pooloo.pppotiieureccom fr raphael. troncy@eurecom.c..m fr
Raphael Troncy EURECOM
Paolo Rosso
UPV prosso@dsic.upv.co
Abstract
ples, but both approaches are inferior "defining" classes via train set of annotated examples.
Large language models have recently risen in popularity due their ability to perform many natural language tasks without requiring any fine-tuning. In this work, we focus on two novel ideas: (1) generating definitions from examples and using them for zero-shot clas- sification, and (2) investigating how an LLM makes use of the definitions. We thoroughly analyze the performance of GPT-3 model for ene-ggrined multi-label conspiracy theory clas- sification of tweets using zero-shot labeling. In doing so, we asses how to improve the label- ing by providing minimal but meaningful con- text in the form of the definitions of the labels. We compare descriptive noun phrases, human- crafted definitions, introduce help the model generate definitions from exam- ples, and propose a method to evaluate GPT-3's understanding of the definitions. We demon- strate that improving definitions of class labels has a direct consequence on the downstream classification results.
Misinformation has been major research topic in the last few years, due to its significant impact society. Social media have been a large of fake-news, especially during the COVID-19 pandemic. We revisit the problem of LLM zero- shot classification on the use-case fine-grained multi-label conspiracy theory detection in tweets (Langguth et al., 2023), using GPT-3 model (Brown et al., 2020). We show that the challenging task of distinguishing conspiracy theories related to COVID-15 19 provides valuable insights into the capa- bilities and limitations of GPT-3. In particular, we aim at exploring the role played by the definition of the labels zero-shot classification and explore the ability of GPT-3 in generating such definitions from a small set of labelled examples. Additionally, we propose a method to test the ability of GPT-3 to understand the definitions and apply them to classification
method to
1 Introduction
Related Work
Recent success of Large Language Models (LLMs) is due to their superior performance various tasks (Qin et al., 2023; Liu et al., 2023; Yang et al..
Zero-shot learning (Larochelle et al., 2008) is a machine learning problem where a model predicts classes without access to training data. It is based the knowledge transfer from classes (Ye al., 2020), using auxiliary informa- tion about the classes that can be provided the model, such as text descriptions (Larochelle et al., 2008; Ba et al., 2015), semantic attributes (Xian et al., 2018; Lampert et al., 2013) or concepts de- fined in ontologies (Wang et al., 2018; Zhang et al., 2019a). Within the domain of natural language processing (NLP), especially text classification, the zero-shot learning approach has been used in variety of tasks, such argument (Zhang et al., 2021), keywords (Nam al., 2016), intent (Liu
2023), such text generation (Dai al., 2023;
to unseen
Khalil and Er, 2023), summarization (Wang al., 2023), question answering (Peinl and Wirth, 2023;
Tan et al., 2023), information retrieval (Omar et al., 2023), machine translation (Hendy et al., 2023;
Jiao et al., 2023), and inductive reasoning (Wei et al., 2022; Kojima et al., 2022; Zhong et al., 2023). However, LLMs had much less success with solv- ing specific tasks (Qin et al., 2023), such as text classification, where they still lag behind fine-tuned transformer models (Yang al., 2023). Class defi- nitions are commonly communicated to LLMs in the form of zero-shot prompts and few-shot exam-
'The code is available at: zerrostt
Equal contribution

---

al., 2019) and utterance (Dauphin al., 2013)
3.1 Conspiracy Theory Classification Dataset The COCO dataset contains Twitter posts anno- tated w.r.t. 12 named COVID-19 related conspir- acy theories (Langguth et al., 2023). Part of this dataset was used in MediaEval FakeNews chal- lenge(Pogorelov et al., 2022), where participants had to detect mentions of 9 conspiracy theories? given the tweet texts. In this work, we use the Me- diaEval dataset, on which the best approaches used fine-tuned CT-BERT models (Müller al., 2020; Peskine et al., 2021; Korenčić et al., 2022). The test set has not been shared publicly before early 2023, we can safely use it in our experiments as it is not part of the training data of GPT-3 that contains data up to September 2021
classification.
Unlike the traditional approach for zero-shot text classification, LLMs are capable of solving zero-shot classification with no auxiliary informa- tion (Zhu et al., 2023; Törnberg, 2023; Kuzman et al., 2023). However, there is evidence that LLMs can outperform crowd workers in zero-shot text annotation tasks (Gilardi et al., 2023). A different approach is to fine-tune the LLM with examples made of manually crafted lists of dictionary/en-- cyclopedia entries related to the target label (Gao al., 2023) or fine-tune smaller pre-trained lan- guage models with examples retrieved (Yu al., 2023) or generated by LLMs (Zhang et al., 2023; Ye et al., 2022; Meng et al., 2022).
3.2 Zero-shot Classification & Definitions
Approaches that use class definitions for con- structing zero-shot classifiers and prompts have been proposed, both for CNNs and transformers (Zhang et al., 2019b; Yin et al., 2019), and for LLMs (Brown et al., 2020; Gilardi et al., 2023). However, they do not experiment with different variants of class definitions, but use definition type for classification and report the results. Addi- tionally, to the best of our knowledge, no previous research has used LLMs to generate a class-specific definition directly from examples and used that in- formation for zero-shot text classification as we investigate in this work.
We leverage GPT-3 to perform multi-labes zero- shot conspiracy theory detection the test of the MediaEval data. In particular, we perform bi- nary classification for each conspiracy category. labeling cach tweet as either mentioning the con- spiracy or not.
Our baseline method relies on zero-shot (ZS) conspiracy theory classification from the textual la- bel of the classes only (c.g. antiivaacinationn' 'Harmful Radiation', 'Satanism', etc). This sesses if the knowledge encoded in GPT-3 able differentiate between similar conspiracy theories. Our next two approaches aim at improving the model's understanding of the label by providing more context in the prompt, specifically with a short defimition of the label. We compare two types of definitions: Human-Writted (HW) and Example- Generated (EG).
Semantic abilities of LLMs are commonly eval- uated on NL understanding and reasoning tasks (Yang et al., 2023), but work on targeted evalua- tion of fine-grained semantic properties is scarce. Sahu et al. (2022) propose to evaluate the LLM's comprehension of query-related concepts by using knowledge graph. To the best of our knowledge. there is no previous work focused on the ability LLMs to understand and apply definitions.
The HW definitions given in the dataset overview paper (Langguth et al., 2023), and are part of the guidelines that were given to the human annotators of the data. Despite the definitions being well-written, annotators had to regularly discuss their understanding of the categories, suggesting the difficulty of the task at hand*.
3 Methodology
The EG definitions are generated with GPT-3 from the training set, by providing GPT-3 with 25 examples of tweets mentioning a given conspiracy
In this section, we first describe the dataset used for conspiracy theory classification. Then, we present the approaches to definition- zero-shot clas- sification using GPT-3, and describe the method for evaluation of GPT-3's understanding of defini- tions. In this work, we use the term 'definition' to denote any additional explanation of the textual labels used to describe the class.
SSuppressed Cures (SUP), Behaviour Control (BHC), Anti- Vaccination (AVX), Fake Virus (FAK), Intentional Pandemic (INT), Harmful Radiation (HAR), Depopulation (DEP), New World Order (NWO), Satanism (SAT) 'According to https://platfrmm. wawaattforoo..ooai.......m aooenai.com/deee com/docs/ nodels/gpt-3--
"The authors report a 92% inter-annotator agreement and more than half tweets had at least one disagreement.

---

theory and 25 examples of tweets not related the conspiracy theory. We use different random seeds to randomly select the example tweets", resulting in 45 definitions generated total. We then ask the model to come up with a short textual description that could separate the sets of tweets. In this setting, we do not provide the textual label of the conspiracy theory, but we only give example tweets to the model. This prevents the model to rely some of its pre-trained knowledge from reading the textual label. Examples of definitions which have been generated are in Appendix A.
similar HW ones guide the model produce better classification results. This is achieved by cor- relating the similarity between the EG definitions and the corresponding HW ones, and the perfor- mance of the classifiers based on the generated definitions.
The second test measures whether mutually sim- ilar EG definitions guide the model produce sim- ilar predictions. This is achieved by correlating the similarity between two EG definitions on one side, and the similarity of the corresponding classifiers' predictions on the other side. Similarity between two sets of predicted binary labels is calculated using Cohen's 's a chance-corrected measure of annotator agreement.
For prompting the model we rely on simple prompts, using both OpenAT's 'system' and 'user' roles in our request. The 'system' message con- tains description of the task, while the 'user' mes- sage contains the tweet's content to be classified. For the classification of the tweets, the definition is appended at the end of the 'system' message. Example prompts used to generate EG definitions and to annotate conspiracy theories are provided in Appendix B.
Results
4.1 Conspiracy Theory Classification
In this section, we discuss the results of the dif- ferent approaches on the classification of the full test set, totalling 823 tweets. Average results are in Table I, and per-category results are in Figure 1. We use Matthews correlation coefficient, Precision, Recall and Fl score compute the classification performance.
3.3 Definition Understanding
Approach of definition-based zero-shot classifica- tion leads to the question whether GPT-3 is able to correctly "interpret" definitions and "apply" them to text classification, which, in our case amounts to detection of conspiracy categories in texts. We pro- pose two tests aimed at assessing if GPT-3 indeed "understands" the definitions given in the prompts. The general approach is to use semantic similar- ity to measure how similarity between definitions correlates with the output of the definition- sefiiition-based classifiers, which we view as result of GPT-3's "interpretation" and "upplication" of definition. For example, one expectation is that similar defini- tions should lead similar outputs. We perform the tests using the 15 example-generated defini- tions, which represent a challenging test case of mutually close definitions - randomly varied and derived from related categorics. We define the se- mantic similarity of two definitions as cosine simi- larity of their embeddings, using state-of-art sen- tence embedding model (Reimers and Gurevych, 2019).
Results show that both EG and HW definitions outperform the ZS baseline. It supports the claim that GPT-3 is capable of leveraging the knowledge provided via the definitions to perform classifica- tion and, therefore, that definitions of the labels can be used to guide the model better perform NLP tasks. While EG definitions do not reach the same performance as HW ones, they can still be used to significantly improve classification accuracy, es- pecially in cases where the HW definition is not available. Our method shows that we can infer a textual description from examples and that GPT-3 can use it to better annotate future samples. Indeed, the usage of EG definitions leads to average rel- ative gain of around + +10% in MCC, Precision and FI scores compared to the ZS baseline. HW defi- nitions see an even greater improvement of around +30% in average, showing the importance a well- defined definition. However, these results are still far from the state-of-the-art CT-BERT fine-tuning methods.
The first test of GPT-3's "understanding" of the definitions measures whether EG definitions more
Figure/ 1 reports the performances for all ap- proaches per conspiracy theory. We observe a gen- eral trend with definitions having positive impact on the performance for most conspiracy theories. However, some concepts are seemingly harder for
'Tweets in both sets can also support other conspiracy theories (multi-label classification problem) "Wo use top-ranked all-mpnet base-v2 model: https: // /ww. sbert. net ceee///cc///ae wttd///pppppp/eeeeeeeeeee pretrained, andeeddddeddddessseeeeeee html

---

Approach
MCC
Precision
Recall
F1
Zero-shot
0.398
0.331
0.852
0.440
w/ Example-generater definitions
0.442
0,371
0.831
0,485
w/ Human Human-written definitions
0.516
0.464
0.823
0.555
CT-BERT ensembling
0.780
0.779
0.849
0.810
Table Performance of the LLM and transformer models using macro-averaging
1.0
increase is result of the quality the quantity of information in the definitions. To address this question correlated the length of the 45 EG def- initions measured by the number of tokens with their classification performance measured by MCC. We found a lack of correlation - a very small p of 0.062. We take this as evidence supporting the claim that the performance depends on the qual- ity, and not on the quantity, of information in a definition.
zs
EG def HW def
0.8
0.6
MCC
0.4
0.0
Conspiracy
MCC F1
Figure MCC score on the test set. Error bars show the minimum and maximum values (5 random seeds)
Similarity (EG, HW)
0.375
0.390
Cohen's x
Similarity (EG, EG)
0.107
GPT-3 to produce useful definitions, such as Sa- tanism, where the EG definitions lead to worse results than the ZS baseline. Also, some conspira- cies are more robust to the EG definitions, as the variance is low and changing the samples lead to similar results, such as Intentional Pandemic, or Fake Virus. Lastly, some EG definitions lead to bet- ter results than the HW ones, suggesting that with better sampling the examples, this method could generate nigher--erforning definitions.
Table 2: Results of the two definition understanding tests
based on semantic similarity and classification results. Top row contains Spearman's correlations of similarity between EG and HW definitions, and performance of EG zero-shot classifiers. Bottom row contains correla- tions of similarity between pairs of EG definitions, and cohen's kappa of their classification.
Discussion & Future Work
4.2 Definition Understanding Tests
While classification of conspiracy theories using EG definitions is done in a zero-shot fashion, the generation the definitions still relies on anno- tated examples. This is different than standard in-context few-shot classification as these exam- ples do not need to be part of the classification prompt. Indeed, it can be seen way to com- press the information from few-shot examples into shorter descriptive context that can be appended in the zero-shot prompt. Further experiments could explore this approach and compare it to standard in-context few-shot classification.
The Spearman's rank correlation coefficients be- tween semantic similarity of the definitions and the results of the definition-based zero-shot clas- sifiers shared in Table 2. The strength of the correlations is fair, which supports the claim that GPT-3 is able correctly interpret the definitions and apply them to conspiracy detection. Namely, higher similarity between EG and HW definitions leads to more accurate classifications, which sug- gest that the model can translate better definitions into better predictions. Additionally, higher simi- larity between two EG definitions correlates with higher agreement between their corresponding pre- dictions, which suggest that the model translates similar definitions into similar predictions.
The correlation tests of definition understanding in Section 4.2 support the claim that GPT-3 indeed interpret and apply the definitions correctly. This complemented by the results in Section which show that better definitions lead to better results. However, further testing should be done on
An interesting question that stems from the vari- ation of the definitions whether the performance

---

more LLMs and with other corpora. Such experi- ments are an interesting direction for future work with the potential to shed light on the semantic capabilities of LLMs.
provements might not be directly applicable other multi-label classification tasks.
We mainly focus on providing better definitions for labels, but other factors, such as the quality of the input data, the prompt, and the model architec- ture can also play a role in improving classification performance.
The results of the definition- zero-shot clas- sifiers imply several practical recommendations and potential applications, all of which represent topics for future work. They include the use of the (high recall) classifiers to create more balanced samples for labeling, application of the classifiers to detect annotation errors, recommendations for mitigation of the low precision, and use of EG defi- nitions for few-shot lcarning. More details on these topics can be found in Appendix C.
The definition understanding analysis is based on several complex artifacts: LLMs, the model of semantic similarity, and the human-crafted defini- tions and annotations. Therefore, there might be other viable explanations of the obtained positive correlations, including unexpected interactions.
Ethics Statement
6 Conclusion
Our study aims improve the classification of con- spiracy theories in social media posts, which might assist in detecting and mitigating misinformation, thus contributing to a more reliable and trustworthy online environment.
In this work, we analyze the impact of label defini- tions on the performance of GPT-3 zero-shot classi- fication, on challenging task of fine-grained con- spiracy theory detection. We show that the of better definitions leads significant gain in most metries (MCC, Precision, F1). We also demon- strate an approach of generating definitions from examples. Sluman-Written definitions still provide definitions
Biases We are aware of the biases of L.Mss classification tasks (Bender et al., 2021). Using language models to generate definitions might lead to unintended biases in the generated definitions, which could impact the performance of the classi- fier. However, our goal is to study the impact of definitions the performance of GPT-3, which can bring insights on how reduce bias with the right prompts and more diverse training data.
better results, while show promising performance. Additionally, we successfully tested GPT-3's s ability to understand and apply these definitions for classification.
Acknowledgements
Environmental Impact The of large-scale Transformers requires lot of computations and GPU>/TUs for training, which contributes global warming (Strubell et al., 2020). This is a smaller issue in our case, as we do not train such models from scratch; rather, we fine-tune them on relatively small datasets or use models for inference in shoo settings.
This work has been partially supported by CHIST- ERA within the CIMPLE project (CHIST-ERA-19- XAI-003) and by ANR within the ECLADATTA project (ANR-22-CE23-0022) Universitat Politècnica de València is carried out in the framework of the XAI-DisInfodemics re- search project on eXplainable Al for disinforma- tion and conspiracy detection during infodemics (Grant PLEC2021-0076811 funded by MCIN/AEI/ 10. 13039/50 10001 1033 and by European Union NextGeneration EU/PRTR.
The work at the
References
J. Ba, K. Swersky, S. Fidler, and Salakhutdinov. 2015. Predicting Deep Zero-Shot Convolutional Neural Net- works Using Textual Descriptions. In 2015 IEEE In- ternational Conference on Computer Vision (ICCV), pages 4247-4255, Los Alamitos, CA, USA. IEEE Computer Society.
Limitations
We conduct our experiments using only the GPT- 3.5 model, which is not open sourced, and is ac- cessible only cloud service which might incur high usage costs, and forces the users rely third-party service.
Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Mod- Be Too Big? FAceT '11 2021 ACM Confer- ence on Fairness, Accountability and Transparency.
The experiments are conducted on conspiracy theory classification, and thus the performance im-

---

Virtual Event / Toronto, Canada, March 3-10, 2021, pages 610-623, ACM.
Damir Korenčić, Ivan Grubišié, Gretel Liz De La Peña Sarracén, Alejandro Hector Toselli, Berta Chulvi, and Paolo Rosso. 2022. Tackling Covid-19 Conspiracies on Twitter using BERT Ensembles, GPT-3 Augmen- tation, and Graph NNs. In MediaEval Benchmarking Initiative for Multimedia Evaluation Workshop.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford. Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Taja Kuzman, Nikola Ljuhešić, and Igor Mozetič. 2023. Chatgpt: Beginning of an end of manual annotation? use case of automatic genre identification. ArXiv preprint arXiv:2303.0399
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. 2013, Attribute-Bascd Classification for Zero-Shot Visual Object Categorization. IEEE trans- actions on pattern analysis and machine intelligence. 36(3):453-465
Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang. Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu. Wei Liu, Ninghao Liu, et al. 2023. AugGPT: Leveraging ChatGPT for Text Data Augmentation. arXiv preprint arXiv:2302. 13007.
Johannes Langguth, Daniel Thilo Schroeder, Petra Filkuková, Stefan Brenner, Jesper Phillips, and Kon- stantin Pogorelov. 2023, COCO: an Annotated Twit- Dataset of COVID-19 Conspiracy Theories. Jour- nal of Computational Social Science.
Yann N Dauphin, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2013. Zero-Shot Learning for Semantic Utterance Classification. ArXiv preprint arXiv:1401.0540
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. 2008. Zero-Data Learning of New Tasks. In AAAI. volume 1, page 3.
Lingyu Gao, Debanjan Ghosh, and Kevin Gimpel. 2023. The Benefits of Label-Descriptio Training for Zero-Shot Text Classification. ArXiv preprint arXiv:2305.02339.
Han Liu, Xiaotong Zhang, Lu Fan, Xuandi Fu, Qimai Li, Xiao- Maio-Ming Wu, and Albert Y.S. Lam. 2019, Re- constructing Capsule Networks for Zero-shot Intent Classification. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4799-4809, Hong Kong, China, Association for Computational Linguistics.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT Outperforms Crowd-Woo Workers for Text-Annotation Tasks. In Proceedings of the Na- tional Academy of Sciences.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. ArXiv preprint axXi::2302.92210.
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Lin Zhao, Dajiang Zhu, Xiang Li, Ning Qiang, Dingang Shen, Tianming Liu, and Bao Ge. 2023. Summary of ChaiGPT-Relatte wards the Future of Large Language Models. Meta- Radiology, 1(2):000117.
Research and Perspective To-
Wenxiang Jiao, Wenxuan Wang. Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine. ArXiv preprint Mxii:2301.0874.
Yu Meng. Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating Training Data with Language Mod- els: Towards Zero-Shot Language Understanding. In Advances in Neural Information Processing Systems, volume 35, pages 462 477. Curran Associates, Inc.
Mohammad Khalil and Erkan Er. 2023, Will ChatGPT Get You Caught? Rethinking of Plagiarism Detec- tion. In Learning and Collaboration Technologies: 10th International Conference, LCT 2023, Held as Part of the 25th HCI International Conference, HCII 2023, Copenhagen, Denmark. July 23-28, 2023, Pro- ceedings, Part I. page 475 Berlin, Heidelberg. Springer-Verlag.
Martin Müller, Marcel Salathé, and Per E Kummervold. 2020. COVID-Twitee Twitter-BERTT A Natural Language Processing Model to Analyse COVID-19 Content on Twitter. ArXiv:2005.07500
Takeshi Kojima, Shixiang (Shane) Gu. Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In Ad- vances in Neural Information Processing Systems, volume 35, pages 22199-222111
Jinseok Nam, Eneldo Loza Mencía, and Johannes Fürnkranz. 2016. All-in text: Learning document, label, and word representations jointly. In Proceed- ings of the AAAI Conference on Artificial Intelligence. volume 30.

---

Reham Omar, Omij Mangukiya, Panos Kalnis, and Es- sam Mansour. 2023. ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots. ArXiv preprint arXiv:2302.064660
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. 2018. Zero-shot recognition via semantic embeddings and knowledge graphs. In Proceedings of the IEFE con- ference on computer vision and pattern recognition, pages 6857-6866.
René Peinl and Johannes Wirth. 2023. Evaluation of medium large language models at sero-shot& closed book generative question answering. ArXiv preprint arXiv:2305.11 1991.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le. and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reason- ing in Large Language Models. ArXiv preprint arXiv:2201.11900
Youri Peskine, Giulio Alfarano, Ismail Harrando, Paolo Papotti, and Raphael Troncy. 2021. Detecting COVID- conspiracy theories in tweets. In MediaEval Benchmarking Initiative for Multimedia Evaluation Workshop.
Yonggin Xian, H. Christoph Lampert, Bert Schiele, and Zeynep Akata. 2018. Zero-Shot Learning- A Comprehensive Evaluation of the Good, the Bad and the Ugly. TPAMI.
Konstamtin Pogorelov, Daniel Thilo Schroeder, Stefan Brenner, Asep Maulana, and Johannes Langguth. 2022. Combining Tweets and Connections Graph for FakeNews Detection at MediaEval 2022. In MediaE- val Benchmarking Initiative for Multimedia Evalua- tion Workshop.
Jingfeng Yang. Hongye Jin, Ruixiang Tang. Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. 2023. Harnessing the power of Ilms in prac- tice: A survey on chatgpt and beyond. ArXiv preprint arXiv:2304.37113
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiang- tao Feng. Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022. ZeroGen: Efficient Zero-shot Learning via Dataset Generation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 11653-1166) Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Ji- aao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a General purpose Natural Lan- guage Processing Task Solver? ArXiv preprint arXiv:2302.06476.
Nils Reimers and Iryna Gurevych. 2019. Sentence- Sentence Embeddings using Siamese BERT- Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
BERT:
Zhiquan Ye, Yuxia Geng, Jiaoyan Chen, Jingmin Chen, Xiaoxiao Xu, SuHang Zheng, Feng Wang. Jun Zhang, and Huajun Chen. 2020. Zero-shot Text Classifica- tion via Reinforced Self-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3014-3024, On- line. Association for Computational Linguistics.
Pritish Sahu, Michael Cogswell, Yunye Gong, and Ajay Divakaran. 2022. Unpacking Large Language Mod- els with Conceptual Consistency, ArXiv preprint arXiv:2209.1r 15003.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Bench- marking zero-shot text classification: Datasets, eval- uation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-ICCNLP), pages 391 3914-3923, Hong Kong, China. Association for Computational Linguistics.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2020. Energy and policy considerations for modern deep learning research. In The Fhurth
AAAI Conference on Artificial Intelligence, New York. NY, USA, February 7-12, 2020, pages 13693-13696.
AAAI Press.
Yue Yu, Yuchen Zhuang, Rongzhi Zhang, Yu Meng. Jiaming Shen, and Chao Zhang. 2023. Re- Gen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval. ArXiv:2305.10703 [cs].
Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Can chatgpt replace traditional khga models? an in-depth analysis of the question answering performance of the gpt 11m family.
Hongming Zhang, Haoyu Wang, and Dan Roth. 2021. Zero-shot label-aware event trigger and argument classification. In Findings of the Association for Computational Linguistics: ACL-IICNLP 2021, pages 1331-1340.
Petter Törnberg. 2023. ChatGPT-4 Outperforms Ex- perts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning. ArXiv preprint arXiv:2304.06555
Jingqing Zhang, Piyawat Lerivittayakumjonn and Yike Guo. 2019a. Integrating semantic know lewwledge to tackle zero-shot text classification. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics:
Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023. Zero-Shot Cross- Lingual Summarization via Large Language Models. ArXiv preprint arXiv:2302.1422 14229.

---

Human Language Technologies, Volume (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics.
A.2
Definitions of Harmful Radiation Conspiracy Category
A.2.1 Example Generated (Seed 4)
Jingqing Zhang, Piyawat Lertvittayakunjonn and Yike Guo. 2019h. Integrating semantic knowledge to tackle zero-shot text classification. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1031-1040, Minneapolis, Min- nesola. Association for Computational Linguistics.
The definition of the concept is a conspiracy the- ory that claims that the COVID-19 pandemic is not caused by a virus, but rather by the implementation of 5G technology and that the pandemic is being used distraction implement digital cur- rency and implantable microchips for population control. This theory has been widely discredited by scientific evidence and health organizations.
Ruohong Zhang, Yau-Shian Wang, and Yiming Yang. 2023. Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction- tuned GPT. ArXiv:2304.11877
A.2.2 Example Generated (Seed 1)
The definition of the concept is conspiracy theory that claims that the COVID-19 pandemic is caused by the deployment of 5G wireless networks. This theory suggests that the radiation from 5G towers weakens the immune system, making people more susceptible to the virus, or that the virus is some- how being transmitted through the 5G network. Despite being debunked by scientists and health ex- perts, this theory has gained traction among some individuals and has led to the destruction of 5G towers in some areas.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can ChatGPT Understand Too? Comparative Study ChatGPT and Fine-tuned BERT. ArXiv preprint arXiv:2302.10198
Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. 2023. Can ChatGPT Reproduce Human-GGenerated Labels? A Study of Social Com- puting Tasks, ArXiv preprint wrXiv:2304.01444
A Examples of Definitions
A.2.3 Human Written
A.1 Definitions of Suppressed Cures Conspiracy Category
Harmful radiation: Narratives that connect COVID- 19 to wireless transmissions, especially from equipment, claiming for example that deadly and that COVID-19 is a coverup, or that lows mind control via microchips injected the bloodstream.
A.1.1 Example Generated (Seed 0)
The definition the concept is a conspiracy theory that suggests the existence of a deep state that is orchestrating the COVID-19 pandemic and block- ing the release of Hydroxychloroquine, cure for the virus. This theory also involves the belief that the pandemic being used push liberal agendas, create economic recession, help China's economy, and stop Trump rallies. is often associated with the QAnon movement and involves the idea that Dr. Fauci is a Deep State Killer.
A.3 Definitions of Depopulation Conspiracy Category
A.3.1 Example Generated (Seed 2)
The definition of the concept is depopulation, which refers to the deliberate reduction of the world's population through various means such disease, vaccines, and other methods. The tweets in the first set suggest that there is group of peo- ple, including Bill Gates, who are actively working towards depopulation as part of their agenda. The tweets in the second set, on the other hand, do not mention depopulation and focus on other topics related to COVID-19 19 such as conspiracy theories. mental health, and political issues.
A.1.2 Example Generated (Seed 1)
The definition of the concept is the use of hydrox- ychloroquine (HCQ) as possible treatment for COVID-19.
A.1.3 Human Written
Suppressed cures: Narratives which propose that effective medications for COVID-19 were avail- able, but whose existence or effectiveness has been denied by authorities, either for financial gain by the vaccine producers or some other harmful intent.
A.3.2 Example Generated (Seed 3)
The definition of the concept is depopulation, which refers to the deliberate reduction of the hu- man population through various means such as dis-

---

ease, sterilization, other forms of population con- trol. The tweets in the first suggest that some individuals believe that the COVID-19 pandemic and the development of vaccines are part of larger depopulation agenda orchestrated by individuals such as Bill Gates. These beliefs are not present in the second set of tweets, which focus on other topics related to COVID-19 such as its impact mental health, the effectiveness of vaccines, and political responses to the pandemic.
A.4.3 Human Written
Satanism: Narratives in which the perpetrators are alleged to some kind satanists, perform ob- jectionable rituals, or make use of occult ideas or symbols. May involve harm or sexual abuse of children, such as the idea that global elites harvest adrenochrome from children.
B Prompt Description
B.1 Example Prompt for EG Definitions
A.3.3 Human Written
SYSTEM "You will be given two sets of tweets. The first set of tweets contains examples of texts that mention the same concept. The second set of tweets contains examples of texts that mention other concepts, but not the same concept that tweets from the first set. Your task is provide the definition the concept present in the first set"
Depopulation: Conspiracy theories on population reduction or population growth control suggest that either COVID-19 or the vaccines are being used to reduce population size, either by killing people or by rendering them infertile. In some cases, this is directed against specific ethnic groups.
A.4
Definitions of Satanism Conspiracy Category
A.4.1 Example Generated (Seed 0)
The definition of the concept is the belief in con- spiracy theory that the COVID-19 19 pandemic is part of larger plan orchestrated by group of powerful individuals organizations, such as the Illuminati, the New World Order, the Deep State, exert more control over the population and ultimately lead to the implementation
USER "First set of tweets: [25x Tweets containing the conspiracy]
the "mark of the beast," which could be a microchip or a vaccine that would be mandatory and used to monitor and control peoplc's actions and movements. This be- lief often involves the idea that the pandemic is hoax exaggerated, and that the real agenda behind it is to push for global government and depopulation.
Second set tweets: 125x Tweets not containing the conspiracy]
Given those two sets of tweets. what is the definition of the concept present in the first set that is not present in the second set of tweets? Start
A.4.2 Example Generated (Seed
your answer with: 'The definition of the concept is
The definition of the concept is the belief that the COVID-19 vaccine or any other form mandatory vaccination is the "Mark of the Beast" as described in the Book of Revelation. This belief is often associated with conspiracy theories involving the government, deep state, and Luciferian Freemasons who are seen trying to control and enslave the population through the of tracking chips and microchipped vaccines. The concept is rooted in religious and apocalyptic beliefs and is often used to justify opposition to vaccination and other public health measures.
B.2 Example Prompt for annotating Tweet with regard conspiracy theory
SYSTEM "Your task is to label tweets regarding the '[ CONSPIRACY] COVID-19 conspiracy theory. The available labels are: 1) mentions the conspiracy. 2)

---

does not mention the conspiracy.
Our preliminary analysis indicates that this is indeed the case. We randomly selected 5 false neg- ative texts per category and checked the annotations using the category definitions from Langguth et al. (2023). We found, on average, 3.8 labeling errors per category (76% of inspected texts).
The definition of the 'I CONSPIRACY]' conspiracy theory is the following: [CONSPIRACY definition]
Mitigating the low precision The classification results in Table show that the definition-basee zero-shot classifiers suffer from low precision. This means that there is a high occurrence of false posi- tives - texts belonging to other related categories being recognized as adhering to the definition of the category being classified. A possible remedy for this could be upgrade the category definitions with text explicitly excluding similar categories.
USER = "[TWEET]
Does the tweet: 1) mention the '[ CONSPRACCY conspiracy, 2) do not mention the '[CONSPIRACY I' conspiracy? Please include the corresponding number in your answer."
Example-generated definitions use cases for few- shot learning An interesting use-case of EG def- initions is the fact that they serve as a way to en- code lot of information into shorter paragraph. Indeed, the LLMs can provide a descriptive defini- tion of the task from set of examples. This way, rather than providing all the examples each time want to annotate a sample, we can provide a much shorter context, allowing to reduce the prompt size, and thus the cost, significantly.
Recommendations For Practical Use
this section, we elaborate on some recommenda- tions for applications of definition- zero-shot classifiers. These recommendations are mainly mo- tivated by the classification results from Section 4.1.
Fixing the class imbalance for labeling Recall the definition-based zero-shot classifiers high and comparable to the recall of the fine-tuned model. Therefore, a possible application of such classifiers is the selection of text data for label- ing, with the goal of fixing the class imbalance, i.c., increasing the expected proportion of positive examples. This approach could help mitigate the rarity of positive examples in many text classifi- cation use-cases, such as various misinformation detection scenarios.
Also, the quality of the definition matters, mean- ing we can actually use a more powerful model (such as GPT-4) to generate the definition, but still use a cheaper model to run the annotation (such as GPT-3.5-turbb) This allows annotate large amount of data with higher-quality definition without increasing the cost by much.
Correcting annotation errors Another potential application of the definition-based zero-shot classi- fiers is detecting and correcting annotation errors.
The approach we propose is perform error anal- ysis of the classifiers based on human definitions, which are commonly used for text annotation. As suggested by low precision scores (see Table 1), the number of false positives is high on average 145.89 texts per category for the test set of 830 texts. However, the number of false negatives is lower and more tractable (on average 27.11 texts per category). Additionally, high recall implies that the texts tend to be correctly detected non- conspiracies, so the false negatives also seem more likely to identify examples wrongly annotated as conspiracies.


Reviews:
[{"review_id": "sFM3iHMWfv", "reviewer": "Reviewer_u3c7", "summary": "", "questions": "\nI haven't conducted a thorough literature review, but I find it a bit surprising the claim that this is the first study to examine the handling of definitions, especially when in the following section, it defines \"definition\" as the description of classes. Are there truly no existing works on zero-shot learning where the language model is introduced to class definitions?\n\nThe analysis of similarity between EG and HW definitions and their correlation with effectiveness is interesting. Even though this is more of an engineering-focused article than a scientific one, at least the analysis regarding the impact of definitions provides very reusable knowledge. However, I'm not sure if the second experiment holds much interest, as it seems obvious that the more the model's input resembles the output. The overall comparison between approaches with and without definitions is also intriguing. Nevertheless, it's not clear to me what the alternative to definitions would be. That is, it's understood that generally, providing more information in the prompt leads to more effective responses. In this article, the authors focus on definitions. But is there any information that can be added to the prompt unrelated to definitions? To what extent are we simply confirming the phenomenon of the benefit of adding more problem-related information to the prompt in general?\n\nI have no objections regarding the formal aspects of the article. I believe it is well-written, and the examples are illustrative. Furthermore, the state of the art is quite comprehensive. My only concern pertains to the significance of the contributions. \n\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "In this work, we focus on examining how GPT-3 uses label definitions for zero-shot classification. We also note other possibilities of study, such as few-shot prompting, in the paper. To keep the paper short and focused, we do not perform an exhaustive examination of prompting strategies, but we rather perform an investigation of the handling of definition-based prompts by a LLM. \n\nWe do not claim that “this is the first study to examine the handling of definitions”. Within the general context of “definition handling”, approaches that use class definitions for constructing zero-shot classifiers and prompts have been proposed. Such approaches can be based on traditional language models and transformers [1,2] or LLMs [3,4]. However, they do not experiment with variants of class definitions, but use one approach for classification and report the results. We thank the reviewer for this comment, since this line of related work will add context and clarity to the paper, and emphasize the novelty of our approach.\n\n[1] *Integrating Semantic Knowledge to Tackle Zero-shot Text Classification* (Zhang et al., NAACL 2019)\n\n[2] *Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach* (Yin et al., EMNLP-IJCNLP 2019)\n\n[3] *Language Models are Few-Shot Learners* (Brown et al., NEURIPS 2020)\n\n[4] *ChatGPT Outperforms Crowd-workers for Text Annotation Tasks* (Gilardi et al., Proceedings of the National Academy of Sciences, 2023)\n\n\nAs we stated in the paper, we focus specifically on (1) generating definitions from examples and using them for zero-shot classification, and on (2) investigating how a LLM makes use of the definitions. The latter goes beyond definition-based classification and uses the classification results as a proxy for gauging definition handling. We have claimed that (1) and (2) are novel ideas. We agree that this should be formulated more clearly and we thank the reviewer for pointing out this confusion. \n\nResults may not be surprising - more similar definitions lead to more similar classifications, statistically. However, considering the complexity of GPT-3 and current level of its understanding, we argue that systematically testing its abilities to correctly apply definitions is useful. Specifically, we interpret the correlation as an indicator of the ability of GPT3.5 to interpret definitions correctly. Moreover, we construct an experiment that can be used to measure the correctness of the definition application, confirm a model property in a systematic manner, and provide a benchmark for other LLMs.\n\nIt should be noted that the two correlation experiments (EG ~ HW, EG ~ EG) are similar and serve the same purpose: to gauge the quality of the definition handling. The difference is that in the EG~HW experiment, one of the definitions (HW) and the associated (gold) labels are fixed. But the idea of the test is the same - a semantically similar definition should, if applied correctly, lead to similar classification results.\n\nWhile our experiments confirm that adding more problem-related information to the prompt is beneficial for the classification, we are primarily concerned by the quality of information, as opposed to the quantity of information. Analogously to what the reviewer correctly pointed out about the quantity of information, we also believe that it is generally understood that adding higher quality information leads to better prompts. The experiments in our view confirm this expectation quantitatively in a systematic manner. We mention the alternative ways to add information: examples, and problem descriptions, but focus solely on the label definitions in the context of zero-shot classification.\n\nInspired by the previous discussion we correlated the definitions’ lengths (num. tokens) with the classification performance in the EG~HW experiment, using the length as the approximation of the quantity of information in the definition. We found a lack of correlation: very small Rho of 0.062 and a large p-value of 68%. More specifically, the probability of this level of correlation is 68% under the null hypothesis of there being no correlation. This can be interpreted as evidence that the performance quality, ie, content, of the definition matters, and not its length. This interesting datapoint will be added to the paper, but we provide it here primarily for the purpose of discussion, and for clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The methodology is appropriate.\n- The literature review is comprehensive.\n- The conclusions about the effect of adding label definitions is useful.\n- The paper is easy to follow.", "reasons_to_reject": "- Some doubts about the paper contribution. No alternative overspecification of the prompt is studied apart from the label definitions.\n\n- The second experiment (similarity between EG vs. output) is obvious.\n\n", "questions_to_authors": "\nI haven't conducted a thorough literature review, but I find it a bit surprising the claim that this is the first study to examine the handling of definitions, especially when in the following section, it defines \"definition\" as the description of classes. Are there truly no existing works on zero-shot learning where the language model is introduced to class definitions?\n\nThe analysis of similarity between EG and HW definitions and their correlation with effectiveness is interesting. Even though this is more of an engineering-focused article than a scientific one, at least the analysis regarding the impact of definitions provides very reusable knowledge. However, I'm not sure if the second experiment holds much interest, as it seems obvious that the more the model's input resembles the output. The overall comparison between approaches with and without definitions is also intriguing. Nevertheless, it's not clear to me what the alternative to definitions would be. That is, it's understood that generally, providing more information in the prompt leads to more effective responses. In this article, the authors focus on definitions. But is there any information that can be added to the prompt unrelated to definitions? To what extent are we simply confirming the phenomenon of the benefit of adding more problem-related information to the prompt in general?\n\nI have no objections regarding the formal aspects of the article. I believe it is well-written, and the examples are illustrative. Furthermore, the state of the art is quite comprehensive. My only concern pertains to the significance of the contributions. \n\n", "paper_topic_and_main_contributions": "The article asses how to improve the labeling by providing minimal but meaningful context in the form of the definitions of the labels. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LHlK90z2xs", "reviewer": "Reviewer_rBbv", "summary": "", "questions": "- Q1) When was the gpt-3.5-turbo-0301 model trained? This information would help our trust in the \"Because the test set has not been shared publicly before early 2023, we can safely use it as it is not part of the training data of GPT-3. \" (lines 126-128) statement.\n- Q2) MCC, precision, recall, F1 are all metrics for binary classifiers. Do you use micro or macro averaging to aggregate the 12 mis-information types?\n- Q3) Is the original COCO/MediaEval problem a 12-way classification task? Or do they do binary classification?\n- Q4) I am confused at the \"45 definitions generated in total\" statement. Were the 5 random seeds each used to sample 25 tweets from each conspiracy label? Does one random seed yield 25 tweets, which in turn yields one definition?\n- Q5) Do the tweets sometimes contain multiple conspiracy types? I wonder if this would effect the example-based generation of definitions.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* W1 mentions that this work would benefit from using additional datasets and/or tasks. Indeed, this can be done in future follow-up works as the idea is not only tied to conspiracy theory. However, fine-grained multi-label conspiracy detection is a fitting task to experiment with generated definitions, as the definition of a conspiracy theory is not obvious and the difference between some conspiracies can be subtle (i.e. semantically similar). This paper is short one appropriate to describe findings per the ACL community tradition.\n\n* W2 states that the work could also benefit from using other LLMs. We experimented with GPT-3 as it was the best performing model to use out-of-the box. A similar trend was observed with GPT-4 on a smaller subset, but we did not scale the experiments because of the higher costs. Preliminary experiments with some smaller models (e.g. Bloom-7B, GPT-J) were not successful, as they were not able to differentiate between the 9 different conspiracies. We did not experiment further, but follow-up work could assess how other LLMs (e.g. Llama2, Falcon, Claude, etc.) handle definitions.\n\n* W3 highlights that the terms “co-prediction” and “in-context” are not defined in our work. By “in-context”, we mean the approach based on the LLM text interface that provides all the task- and example- related data as text prompt. Additionally, we define “co-prediction” as the agreement between two labeling results. In our case, we used Cohen’s kappa to measure the agreement between GPT-3 annotations given two different definitions. Our results suggest that the more similar the definitions are, the higher the agreement is, which means the more the co-predictions are aligned. The term “co-prediction” might be confusing, so we will edit the paper and refer to Cohen’s Kappa instead.\n\n* Q1 asks when was the gpt-3.5-turbo-0301 model trained. According to OpenAI API documentation (https://platform.openai.com/docs/models/gpt-3-5), this model was trained on data up to September 2021. Since the test set of MediaEval has only been released publicly in early 2023, the model should not have seen the data during training.\n\n* Q2 refers to how we averaged the metrics to aggregate the misinfo-types. We use macro-averaging to aggregate the 9 classes. This correction will be edited in the paper. \n\n* Q3 requests more information about the original datasets. Both COCO and MediaEval are multi-label multi-class datasets. They annotate each tweet with a 3-way label for 12 and 9 classes respectively. The original datasets are not binary and have two different labels for “discussing” a conspiracy and “promoting” it; we merged both labels to just consider tweets “mentioning” the conspiracy. COCO is an upgraded version of MediaEval dataset that contains 9 original conspiracy categories and augments the taxonomy with additional categories: a rare label \"Esoteric misinformation\", \"other conspiracies\" and \"other misinformation\". In our work, we opted for the MediaEval dataset as it contains all the relevant conspiracy-related labels. These clarifications will be edited for the final version of the paper.\n\n* Q4 points out confusion about the number of generated definitions and the seeds to pick examples. To clarify, one seed is used to pick the 50 random examples, 25 of which refer to the conspiracy and 25 of which do not. These examples are used to generate a definition. Because there are 9 classes in the MediaEval dataset and we generate definitions with 5 random seeds, this equals to 45 generated definitions.\n\n* W4 and Q5 refer to the presence of multi-label tweets in the data and how it can impact the generation of the definitions. Indeed, most tweets contain multiple conspiracy types. This definitely has an impact on the quality of the generated definition. However, it is not clear what strategy would yield the best results: to only include tweets that mention a single conspiracy theory, or to select tweets with multiple conspiracies and sample tweets mentioning the same other conspiracies in the negative examples. We decided to randomly select the examples on 5 different seeds to assess the variance of the generated definition. \n\nWe thank the reviewer for pointing out the grammar mistakes and the inconsistencies in reference styles. These will be fixed in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- I appreciate the \"Definition Understanding\" analysis. This adds an extra layer to the paper beyond just model classification performance.\n- Examples are provided in the Appendix, which were interesting to read.\n- The scope of this paper is rather small, but I could see this type of paper being accepted to Findings or to a workshop.", "reasons_to_reject": "- W1) The paper evaluates only on one dataset and on one task. Results and conclusions would be stronger if the analysis were applied to more datasets and more tasks. \n- W2) Similarly, only one LLM model (GPT-3) is examined.\n- W3) Some terms like \"co-prediction\" (line 278) and \"in-context\" (line 285) are not defined or explained.\n- W4) A potential weakness is that the paper overlooks the presence of multi-label tweets. For instance, do multi-label tweets impact (i.e., harm) the effectiveness of the EG approach? I imagine that multi-label tweets would confuse the model insofar as the model would have a harder time generating a clear definition of a single label concept. See Q5 below.", "questions_to_authors": "- Q1) When was the gpt-3.5-turbo-0301 model trained? This information would help our trust in the \"Because the test set has not been shared publicly before early 2023, we can safely use it as it is not part of the training data of GPT-3. \" (lines 126-128) statement.\n- Q2) MCC, precision, recall, F1 are all metrics for binary classifiers. Do you use micro or macro averaging to aggregate the 12 mis-information types?\n- Q3) Is the original COCO/MediaEval problem a 12-way classification task? Or do they do binary classification?\n- Q4) I am confused at the \"45 definitions generated in total\" statement. Were the 5 random seeds each used to sample 25 tweets from each conspiracy label? Does one random seed yield 25 tweets, which in turn yields one definition?\n- Q5) Do the tweets sometimes contain multiple conspiracy types? I wonder if this would effect the example-based generation of definitions.", "paper_topic_and_main_contributions": "This paper explores the zero-shot capabilities of GPT-3 at the task of conspiracy theory detection in tweets about covid-19. (For instance, one conspiracy theory label used in the dataset used in the paper's evaluation is related to alternative cures being suppressed/hidden by the government.) The paper analyzes standard zero-shot performance --- where the model is only provided with the text input as well as a label (to predict yes/no) --- as well as scenarios where the model is also provided with descriptions/definitions of what the conspiracy category is about. For this latter setting, the paper explores handwritten definitions (provided by the creators of the dataset) as well as model-generated labels (generated after sampling several instances and asking the GPT model to describe definitions of each label category given these samples). The model finds that zero-shot performance is worse than fine-tuned model performance, and that the original handwritten definitions yield better zero-shot performance, which are not totally surprising results.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- \"Recent success of Large Language Models (LLMs) is due to their superior performance\" (line 021) sounds like circular reasoning.\n- There are some various minor grammar mistakes. For instance I believe \"Approach of definition-based...\" should be \"The approach of definition-based...\" (on line 182). Similarly \"as cosine similarity\" should be \"as the cosine similarity\" on line 200.\n- Inconsistent reference styles. For instance the references on lines 593-596, 601-604, 575-579, and 560-564 are all arXiv papers but have different reference formats. Similarly, only some references have URLs available.\n- Section 5 mentions \"in-context few-shot learning\" but it is not clear what \"in-context\" refers to, nor is \"few-shot\" mentioned other than in the introduction."}}, {"review_id": "nkDcxr8Id2", "reviewer": "Reviewer_hx9P", "summary": "", "questions": "The correlations part interchangeably uses terms : Cohen’s kappa, Spearman’s correlation and Matthews correlation. Can you uniform that?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We agree that closed models incurring usage costs is a limitation, albeit not a crucial one, as closed models should also be evaluated scientifically to the best of the ability of the academic community, in particular, when such models are proven to be effective in multiple benchmarks.\n\nGPT3.5 is a widely used state-of-art model that is commonly included as a benchmark, and many experiments are done solely with this model. To mitigate this issue and improve reproducibility, we provide all the outputs of GPT-3 including the generated definitions. The fact that the model is not open source is a limitation, but we point out that we are treating the model as a black box and that our evaluation methods do not require probing the model’s internals. \n\nWe thank the reviewer for pointing out the potential unclarities about using Cohen’s kappa, Spearman’s correlation and Matthews correlation interchangeably. We note that Spearman’s Rho is used to measure correlation between (1) definition’s similarity and (2) either the zero-shot performance in EG\\~HW (using Matthews correlation), or the similarity of classification outputs in EG\\~EG (using Cohen’s kappa). These three metrics are distinct and used as described above.\n\nAll the metrics in the table serve to compare two sets of classification labels. Top row contains standard classification performance metrics, since we compare classifiers’ labeling with the human gold labels. Bottom row compares labels produced by two classifiers and in this case without a reference label we think it is more correct to use a symmetric measure of inter-annotator agreement since there is no reference labeling.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written, well-structured and clearly-formulated. That makes understanding easy also for people that do not work deeply in the LLMs domain. The offered approaches are complementing each other. They are derived reasonably from the lack of performance. However, the techniques also focus on low laborious work, hence, they are efficient.\n\nIt shows some good alternatives to time-consuming fine tuning and zero-shot problems.", "reasons_to_reject": "I feel that the fact that the GPT version used is not open sourced and can incur usage costs a bit irritating. ", "questions_to_authors": "The correlations part interchangeably uses terms : Cohen’s kappa, Spearman’s correlation and Matthews correlation. Can you uniform that?", "paper_topic_and_main_contributions": "The authors describe a way to improve zero-shot multi-label classification of large language models in the domain of conspiracy theory. Especially, the authors suggest two low-laborious techniques that prompt more context information to overcome low performance in specialized classification tasks: i) Human-Written (HW) label definitions, and ii) example-generated definitions (EG). The latter is designed by providing the system with 25 examples each, containing a certain conspiracy theory or not. These approaches are compared to the zero-shot classification from class labels. Semantic correlation is documented between the HW definitions and EG, which in turn also are compared/correlated to the system output’s performance. All experiments are performed using GPT3.5\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "See questions, also Table 2 is not entirely clear. Why don't you provide an F1 for Cohen's kappa?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a simple prompting strategy for zero-shot classifying tweets according to COVID-19 conspiracy theories. Examples are augmented with a human-written or generated definition of the conspiracy theory being classified. The added definitions (especially human-written and generated ones similar to human-written ones) improve accuracy at the task. Reviewers generally thought the method is well-motivated and the paper is clear and well structured. One concern is the narrow set of experiments (one task and only using ChatGPT), but I think this lack of breadth is acceptable, although not ideal, for a short paper. The authors address reproducibility concerns over ChatGPT in the rebuttal. Reviewers did not give high excitement scores because the method is simple and the results are mostly unsurprising. However, that does not prevent the method from being useful or worth studying, and the paper does provide a focused contribution suitable for a short paper.
Decision: Accept-Findings