Primacy Effect of ChatGPT
Authors: Yiwei Wang
Yujun Cai
Muhao Chen
Yuxuan Liang
Bryan Hooi
Venue: EMNLP 2023 Main (2023)
Keywords: Primacy Effect
ChatGPT
Large Language Models
Natural Language Understanding

Abstract: Instruction-tuned large language models (LLMs), such as ChatGPT, have led to promising zero-shot performance in discriminative natural language understanding (NLU) tasks. This involves querying the LLM using a prompt containing the question, and the candidate labels to choose from. The question-answering capabilities of ChatGPT arise from its pre-training on large amounts of human-written text, as well as its subsequent fine-tuning on human preferences, which motivates us to ask: Does ChatGPT also inherit humans' cognitive biases? In this paper, we study the primacy effect of ChatGPT: the tendency of selecting the labels at earlier positions as the answer. We have two main findings: i) ChatGPT's decision is sensitive to the order of labels in the prompt; ii) ChatGPT has a clearly higher chance to select the labels at earlier positions as the answer. We hope that our experiments and analyses provide additional insights into building more reliable ChatGPT-based solutions. We release the source code at https://github.com/wangywUST/PrimacyEffectGPT.

Primacy Effect of ChatGPT
Yiwei Wang Yujun Cai
Muhao Chen
Yuxuan Liang1
Bryan Hooi!
University of California, Los Angeles $ Metta
8 University of California, Davis National University of Singapore Hong Kong University of Science and Technology (Guangzhou)
wangyw. evanegmail.
Abstract
0.5
CharGPT
Density Distribution
Instruction-tuned large language models (LLMs), such as ChatGPT, have led to promising zero-shot performance criminative natural language understanding (NLU) tasks. This involves querying the LLM using a prompt containing the question, and the candidate labels to choose from. The guestion- capabilities of ChatGP arise from its pre-training on large amounts of human-wriin written text, as well as its sub- sequent fine-tuning on human preferences, which motivates
0.4
0.3
dis-
0.2
0.1
0.gg
10
20
30
40
Prodicted Label Index
Figure Primacy Effect of ChaiGPT: ChatGPT tends to return labels in earlier positions the answer. This plot shows the distribution of ChatGPT's predicted label indices in TACRED (42 classes), where randomly shuffle labels before every prediction (see Sec. 2.2).
ask: Does CharGT also inherit humans' cognitive biases? In this paper, we study primacy effect of ChatGPT: the tendency of selecting the labels at earlier positions as the answer. We have main findings: i) ChatGPT's decision is sensitive to the order of labels in the prompt; ii) ChatGPT has a clearly higher chance select the labels earlier positions the answer. We hope that our experiments and analyses provide additional insights into building more reliable ChatGPT-based solutions. We release the source code at hpsps: //github httphhu...o/wwwggy//g.e wwtp.///wwwwwss///reeeeeeeeeeeee imacyEffectGPT.
Recently, instruction-tunel large language mod- els (LLMs), represented by ChatGPT (OpenAI, 2022), have received wide attention on their capa- bilities of imitating humans in question-answering and problem-solving However, this underlying be- havioral similarity between ChatGPT and humans naturally leads an intriguing question: Is Chat- GPT also affected by the primacy effect?
ChatGPT provides a convenient way to achieve the discriminative natural language understanding (NLU) (Li et al., 2023; Wei et al., 2023; Yuan et al.. 2023). People only need to list the labels in the prompt and asking ChatGPT select the label(s) that match the input text. In this work, to ana- lyze the primacy effect of ChatGPT, we start by testing with random label shuffling, i.e., shuffling labels listed in the prompt before every prediction. We compare the predictions on the same instance with two different label orders. Then, we count the predicted label indices many instances with label shuffling. The motivation that: a fair NLU model should give the same prediction on an in- put instance regardless of how the labels are or- dered; consequently, it should produce uniformly distributed label indices under label shuffling for
1 Introduction
Humans tend to recall information presented at the start of a list better than information at the middle or end. This phenomenon is known the primacy effect (Asch, 1946), which is cognitive bias that relates to humans" attention spans (Crano, 1977), rehearsal (Tan and Ward, 2000), and memory sys- tems (Li, 2010). Similarly, in advertisement sys- tems and search engines, humaans tend to interact with items in higher positions regardless of the items' actual relevance (Chen et al., 2023). Pri- macy effect influences humans' behaviors to make unfair decisions. Similarly, if it exists in machine learning models, may lead to worse performance.
"Equal contribution.

---

any instance.
Prompts for Zero-shot Intent Detection
Through extensive experiments with series of NLU datasets, we find that
Label Definitions
Label 1: change pin
ChatGPT's prediction is sensitive to the order of labels in the prompt. Specifically, Chat- GPT's prediction changes after label shuf- fling on 87.9% of the instances in TACRED.
Label 2: card arrival
Label 3: activate my card
Target Text: need new PIN.
ChatGPT is affected by the primacy effect: ChatGPT tends to select labels in earlier posi- tions in the prompt (see Fig. 1), which present clear bias with respect the label order.
Which Label matches the intent expressed the Target Text?
Figure 2: A prompt example for ChatGPT.
On the whole, our work contributes to a better understanding of ChatGPT's behavions and build- ing more faithful ChatGPT-based NLU solutions.
Ideally, fair model should return the pre- diction when labels are shuffled, and consequently, the predicted label index should follow uniform distribution under random shuffling. Next, we introduce our two ways of using ran- dom label shuffling analyze ChatGPT.
2 Primacy Effect of ChatGPT
this section, we first introduce the general prompt design of ChatGPT in discriminative nat- ural language understanding (NLU). Then, we an- alyze the primacy effect of ChatGPT using label shuffling in prompts.
Instance A reli- able and consistent classifier is expected to consis- tently choose the same label for the same instance irrespective of the label order. To evaluate such consistency of ChatGPT, we perform the random shuffling for the same instance twice to produce two prompts. We feed these two prompts to Chat- GPT and compare the corresponding two predic- tions with each other. We apply the above process to all the test instances and compute the fraction of the instances where the prediction changed after label shuffling. The higher the fraction is, the more sensitive ChatGPT is the label order.
Prediction Comparison
2.1 Prompts for ChatGPT
Prompts are a key component to the effective use of ChatGPT on discriminative NLU tasks (Wei et al., 2023; Yuan et al., 2023). Generally, prompts for such tasks involve two key components: (i) label definitions, and (ii) a task description and input text (see an example in Fig. 2).
ChatGPT's capability of understanding instruc- tions in the prompt benefits from its training with human feedback (OpenAI, 2022), but this also cre- ates the risk of inheriting humans' cognitive biases. In this paper, discuss cognitive bias in Chat- GPT: the primacy effect, which indicates the ten- dency of selecting labels in earlier positions in the prompt.
Statistics of Predicted Indices Taking further step, we perform statistical analysis on the pre- dicted indices for instances where the prediction changed after label shuffling. ChatGPT does not have any preference on the label orders, its pre- dicted label indices should be uniformly distributed. By comparing the predicted label index distribu- tion of ChatGPT to the uniform distribution, we can assess its fairness and preferences regarding label orders.
2.2 Analysis with Label Shufiling
Analyzing the primacy effect requires us to distill the effects of label orders in the prompts. However, this is non-trivial because there are many factors influencing ChatGPT's decisions, such as the input text and label definitions. In our work, to distin- guish the primacy effeet of ChatGPT from other factors, conduct random shuffling for labels listed in the prompts. Specifically, before every prediction, we shuffle the labels as visualized in Fig. Label shuffling erases the discriminative se- mantics of the specific label orders in the prompts.
3 Experiments
We analyze the primacy effect based on the afore- mentioned strategics using three relation extraction datasets and an intent detection dataset.

---

Prompt
Label I: change pin Lobel 2: card arrival Label activate my card
Label index: 3
Label: activate my card
label shuffling
count
compare
Prompt 2
Label I: card arrival
Label 2: activate my card
Label index:
Label: card arrival
Label 3: change pin
Figure 3: We analyze the primacy effects of ChatGPT by randomly shuffling the labels in the prompts.
3.1
Experiment Setup
3.2 Consistency under Label Shuffling
First, we observe the low consistency ChatGPT confronted under label shuffling. As shown in Ta- ble 1, ChatGPT changes its label prediction after label shuffling in over 85% of the test instances on the TACRED, TACREV, and Re- Re-TACRED datasets, and in 35.7% of instances ChatGPT changes its label prediction after label shuffling in over 69% of the test instances on the datasets of GoEmotions and more than 30% of instances on MASSIVE and 20 Newsgroups. contrast, the fine-tuned BERT classifier maintains consistent predictions after label shuffling. This discrepancy challenges the widely-held belief that ChatGPT can comprehend human instructions and provide consistent responses. One possible planation is that ChatGPT's understanding of the prompt is obtained by training on numan-labelee data, which inherits humans' cognitive bias of treat- ing labels at different positions unfairly.
We mainly chose relation extraction and intent de- tection tasks in our experiments since these tasks naturally come with adequately sized decision spaces to illustrate the underlying primacy effect of labels. For relation extraction, we experiment three benchmark datasets including TACRED (Zhang et al., 2017), TACREV (Alt al., 2020), and TACRED (Stoica al., 2021). For intent detection, we conducted experiments on Banking77 (Casanueva et al., 2020a) and MASSIVE (FitzGer- ald et al., 2022). MASSIVE (FitzGerald et al., 2022) is a parallel dataset of massive utterances with annotations for the Natural Language Under- standing tasks of intent prediction. Utterances span 60 intents.
Banking77. Also,
We additionally conducted experiments on the NLP datasets: GoEmotions (Demszky et al., 2020) and 20 Newsgroups (Albishre et al., 2015) for a more comprehensive evaluation. GoEmotions (Demszky et al., 2020) is dataset for fine-graing grained emotion classification. It is a corpus of 58k care- fully curated comments extracted from Reddit, with human annotations for 27 emotion categories and neutral one. The 20 Newsgroups (Albishre al., 2015) dataset is collection of approximately 20,000 newsgroup documents, partitioned across 20 different newsgroups.
It is worth noting that the ratio of instances with changed predictions is consistently high across the relation extraction datasets but lower on intent de- tection. This discrepancy can be attributed to the fact that information extraction tasks are shown to be challenging for ChatGPT and other LLMs (Wang al., 2023; Li et al., 2023). In more diffi- cult tasks, ChatGPT lacks sufficient discriminative semantic understanding from the input text and may be more affected by the label order.
3.3 Primacy Effect ChatGPT
We follow the existing work (Wei et al., 2023; Li et al., 2023) to apply ChatGPT to these tasks via the OpenAl API gpt-3. 5-turbo. Specifically, we set the temperature 0.0 to minimize the randomness of ChatGPT's outputs. For comparison, we adopt the existing work (Casanueva et al., 2020b; Zhou and Chen, 2022) to tinettuneee the BERT model with an MLP classification head.
The empirical results in Section 3.2 indicate that ChatGPT's predictions are affected by label order. To deeper delve into the effects of label orders on ChatGPT, analyze the distribution of predicted label indices (e.g., if the prediction the first label, the label index is 1), as introduced in Section 2.2. We visualize the distributions in Fig. 4. Notably,

---

Method
TACRED
TACREV Re-TACRED
Banking77
GoEmotions
MASSIVE
Newsgroups
ChatGPT w/ Prompt
87.9
85.9
88.6
35.7
69.3
32.8
34.1
BERT w/ MLP
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Table I: Fraction of the instances that have their predicted label changed after label shuffling.
T kJ SERT
QueeT
CRIRETT Trace
ClaCd Pruge BERT
Podittd Lase bats
Prouadd Lald
Phedated
(a) TACRED
(b) TACREV
Re-TACRED
(d) Banking77
Figure 4: The distribution of predicted indices of the test instances with label shuffling before every prediction.
Method
TACRED
TACREV
Re-TACRED
Banking?7
Topsoe, 2004) the metric evaluate how im- bulancod/skewed/umfaar prediction P is. The measurement symmetric (i.e., JS(PIU) = P)) and strictly scoped.
CharGPT w/ Prompts
$7.9
57.8
58.1
18.8
CharGP CoT
57.6
57.9
58.3
18.6
BERT w/ MLP
1.8
1,9
2.3
2.1
Table 2: Experimental results (unfairness; %) on the test sets TACRED, TACRED-Revisit, Re-TACRED, and Banking77 (lower is better). The best results in each column are highlighted in bold font.
To evaluate the label order bias of ChatGPT, we compute the average relative label order imbal- ance (LOI): LOT is defined the IS divergence between the predicted label index distribution P and the uniform distribution U:
the distribution of ChatGPT's predictions consis- tently deviates from the uniform distribution, dis- playing a consistent bias towards smaller indices across different datasets. In contrast, BERT ex- hibits no preference for label orders and consis- tently demonstrates uniform distribution in its predicted label indices.
LOI JS(P(x U),
(1)
where represents an input instance, D is the test set, P(x) is the predicted label index, and U is the uniform distribution. LOI captures the dispar- ity between the predicted indices and uniform distribution.
We term this tendency of ChatGPT as the pri- macy effect, where the model tends to favor the labels presented earlier in the prompt. The magni- tude of these primacy effects varies across tasks, as illustrated in Fig. 4. Notably, the influence of primacy effects is higher in more challenging tasks. This observation aligns with the results discussed in Sec. 3.3, wherein the impact of primacy effects is greater when ChatGPT tackles more difficult tasks. the next section, we will quantitatively analyze the primacy effects of ChatGPT.
We conduct the fairness evaluation following the experimental settings described in Section 3.3, and the results are presented in Table 2. The findings demonstrate that ChatGPT exhibits unfair treat- ment of label indices when making relation label predictions for input texts. Furthermore, the degree of unfairness increases with the task's difficulty, which aligns with the empirical results discussed in Sections 3.2 and 3.3. In contrast, BERT demon- strates significantly better fairness, as its predic- tions are not influenced by label orders.
We additionally test the performance of Chat- GPT with CoT -ffain-of-thoughts)) (Wei et al., 2022). With CoT, ChatGPT still exhibits the pri- macy effect. The above results show that with without CoT, ChatGPT consistently exhibits the pri- macy effect. A reason for this phenomenon could be that the CoT encourages the LLMs for "slow thinking" about the question but does not neces-
3.4 Evaluation on Fairness
The fairesss of trained model can be assessed by examining the imbalance skewness in its predictions (Sweeney and Najafian, 2019). Fol- lowing prior studies (Xiang et al., 2020; Sweeney and Najafian, 2019; Qian et al., 2021; Wang et al., 2022), we employ the JS divergence (Fuglede and

---

sarily mitigate the cognitive bias in the reasoning steps of COT.
the answer. The primary effect seriously influences ChatGPT's fairness. Collectively, our findings pro- vide a new understanding of how ChatGPT works given the instructional prompts.
4 Related Work
Large Language Models (LLMs) (Brown et al.. 2020; Rae et al., 2021; Thoppilan et al., 2022; Smith et al., 2022), such as GPT-3 (Brown et al., 2020), LaMDA (Thoppilan et al., 2022) and PaLM (Chowdhery ct al., 2022), refer to large scale pre- trained models that contain more than a hundred bil- lion parameters. Based the highly parallelizable Transformer architecture (Vaswani et al., 2017), these Large Language models have shown power- ful capability to produce reasonable results with very few samples or task descriptions as input.
Conclusion
While previous work often takes ChatGPT as uni- versal method applicable to all text-related tasks, we argue that its flexibility comes with the risk of inheriting human's 's cognitive biases. These bi- ases lead to unfair judgments which can affect the performance of the machine learning model. This work studies a cognitive bias of ChatGPT: primacy effects. We propose simple yet effective label shuffling method to analyze the influence of label orders ChatGPT. We discover the primacy ef- feet of ChatGPT and finds that highly influences the faimess of ChatGPT in NLU. Our work con- tributes to better understanding of the behaviors of ChatGPT and building more faithful solutions with ChatGPT in NLU applications.
A key milestone in the development process of LLMs is ChatGPT, which is developed by Ope- nAT based on InstructGPT(Ouyang et al., 2022). ChatGPT is able interact with humans through multiple turns of dialogue, understand user intent, accomplish instructions, and return human-like re- sponses. This attracts huge attention from research field, motivating numerous recent work (Zhang et al., 2022; Ma et al., 2023; Wan et al., 2023; Zhong et al., 2023; Susnjak, 2023) to utilize Chat- GPT to different tasks.
Limitation
Our work has a few potential limitations. Firstly. we primarily evaluate the primacy effect of Chat- GPT, which is one of the most widely-used instruction-legacy models for each task. It would be beneficial to assess this effect on other LLMs models (such as Google Bard, vicuna (Chiang et al., 2023)) and explore additional tasks to examine this primacy effect. Secondly, this work focused alyzing the primacy effect of ChatGPT through ex- periments. We encourage further studies to propose effective solutions that can mitigate the negative impacts associated with the primacy effect.
As ChatGPT is a proprietary model, and OpenAl does not disclose its training specifies, researchers are actively investigating its associated implica- tions and capabilities. There has been some work analyzing the performance, robustness, faithful- ness, and explain-ability of ChatGPT (Gao et al., 2023; Han et al., 2023; Li et al., 2023). For ex- ample, (Malinka et al., 2023) investigates the ed- ucational integrity of ChatGPT and evaluates the ChatGPT's abilities solve assignments of various levels in computer security specialization. (Haque et al., 2022) and (Krügel et al., 2023) investigate the ethical risks of ChatGPT.
A Acknowledgement
The authors would like to thank the anonymous reviewers for their discussion and feedback. Yi- wei Wang and Bryan Hooi are supported by NUS ODPRT Grant A-0008067-00-00, NUS ODPRT Grant R252-000-A81-133. and Singapore Min- istry of Education Academic Research Fund Tier 3 under MOEs official grant number MOE2017- T3- 1007. Muhao Chen is supported by the NSF Grant IIS 2105329, the NSF Grant ITE 2333736, the DARPA MCS program under Contract No. N66001 1924033 with the United States Office Of Naval Research, Cisco Research Award, two Amazon Research Awards, and a Keston Research Award.
Before ChatGPT, LLMs' inference has been ac- companied by in-context learning (ICL) which adds few demonstrations in the prompt (Dong et al., 2022; Fei et al.. 2023). Accordingly, some work investigates the effects of demonstration or- ders for the LLMs before ChatGPT (Lu et al., 2021). (Zhao et al., 2021) finds the majority la- bel, recency, and common token biases of LLMs" ICL.
Different from the above work, we focus on a new phenomenon of ChatGPT: the primacy effect,
which is the tendency of selecting the first labels as

---

References
2020. Goemotions: dataset of fine- one-grained emo- tions. arXiv preprint 2005
Khaled Albishre, Mubarak Albathan, and Yuefeng Li.
2015. Effective 20 newsgroups dataset cleaning. In 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), volume 3, pages 98-101. IEEE
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv prepriat arXiv:2301.0022
Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. TACRED revisited: thorough eval- uation of the TACRED relation extraction task. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1558- 1569, Online. Association for Computational Linguis- tics.
Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. Mitigating label biases for in-context learning. arXiv preprint arXiv:2305. 19148.
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach. Vishesh Kakarala, Richa Singh, et al. 2022. Massive: A Im-example multiin- gual natural language understanding dataset with 51 typologically-diverss languages. arXiv preprint arXiv:2204.088: 0.8522...
Solomon E Asch. 1946. Forming impressions of per- sonality. The Journal of Abnormal and Social Psy- chology, 413:22258
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Bent Fuglede and Flemming Topsoe. 2004. Jensen- shannon divergence and hilbert space embedding. In International Symposium oninformation Theory, 2004. ISIT 2004. Proceedings. page 31. IEEE
Jinglong Gao, Xiao Ding, Bing Qin, and Ting Liu. 2023. Is chatgpt a good causal reasoner? a comprehensive evaluation. arXiv preprint arXiv:2305.07377
lñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020a. Ef- ficient intent detection with dual sentence en- coders. In Proceedings of the 2nd Workshop on NIP for ConvAl ACI. 2020. Data avail- able at http:/ghuuboooooollllll-ll/nnsa/ datasets.
Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang,
Lu Liu, and Xiang Wan. 2023. Is information extrac- tion solved by chatgpt? an analysis of performance, evaluation criteria, robustness and errors. arXiv preprint arX5v:2305.1 14450.
Mubin UI Haque, Isuru Dharmadasa, Zarrin Tasnim Sworna, Roshan Namal Rajapakse, and Hussain Ahmad. 2022. tive technology": Exploring sentiments of chatgpt early adopters using twitter data. arXiv preprint urXiv:2212.05885
lñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulié. 2020b. Effi- cient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807.
think this is the most disrup-
Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. 2023. Bias and de- bias recommender system: survey and future directions. ACM Transactions on Information Sys- tems, 1133).
Sehastian Krügel, Andreas Ostermaier, and Matthias Uhl. 2023. The moral authority of chatgpt. arXiv preprint 201.022301.07098.
Bo Li, Gexiang Fang. Yang Yang. Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023. Evaluating chatgpt's 's information extraction capabilities: An as- sessment of performance, explainability, calibration, and faithfulness. arXiv preprint arX4v:2304. 1633.
Wei-Lin Chiang. Zhuohan Li, 7i Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Cong Li. 2010. Primacy effect or recency effect? long- term memory test of super bowl commercials. Journal of Consumer Behaviour: An International Research Review, 31):32-44.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint D.Xiv:204.023111.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where find them: Overcoming few-shot prompt order sensitivity. arXiv preprint 1:Xiv:2004.88888
William D Crano. 1977. Primacy versus recency in retention of information and opinion change. The Journal of Social Psychology, 101(1))87996.
Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun, 2023. Large language model is not a good sew-shot information extractor, but good reranker for hard samples! arXiv preprint 2303.:2300088555
Dorottya Demszky, Dana Movshovitz-Atte Jeongwoo Ko, Alan Cowen, Gauray Nemade, and Sujith Ravi.

---

Kamil Malinka, Martin Peresíni, Anton Firc, Ondrej Hujnak, and Filip Janus. 2023, On the educational impact of chatgpt: Is artificial intelligence ready to obtain a university degrec? In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. I, pages 47-53.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances neural information pro- cessing systems, pages 3998-6008.
Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. 2023. Gpt-re: In-context learning for relation traction using large language models, arXiv preprint arXiv:2305.02100,
OpenAI. 2022.
Introducing
chatgpt
httpss//openi..co//hb//haaagg
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances Neural Information Processing Systems, 35:27730-2771 277444.
Fei Wang, Wenjie Mo, Yiwei Wang, Wenxuan Zhou, and Muhao Chen. 2023. A causal view of entity bias (large) language models. arXiv preprint OrXiv:2305.14693
Chen Qian, Fuli Feng, Lijie Wen, Chunping Ma, and Pengjun Xie. 2021. Counterfactual inference for text classification debiasing. In Proceedings of the 59th CorrratiooooCCCmmaaatttioon Computational Linguistics and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers). pages 5434-5445.
Yiwei Wang, Muhao Chen, Wenxuan Zhou, Yujun Cai. Yuxuan Liang, Dayiheng Liu, Baosong Yang. Juncheng Liu, and Bryan Hooi. 2022. Should we rely on entity mentions for relation extraction? debi- asing relation extraction with counterfactual analysis. arXiv preprint arXiv:2005.037%.
Annual Meeting of the Association for
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring. Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 1rXiv:2112.1144:
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Lc. Denny Zhou, et al. 2022. Chain-of-thouge soning large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
prompting elicits rea-
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023. Zero- shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302. 10205.
Shaden Smith, Mostofa Patwary, Brandon Norick. Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. 2022. Using deep- speed and megatron to train megatron-turing nlg 530b, large-scale generative language model. arXiv preprint arXiv:2201.1 19900.
I Liuyu Xiang, Guiguang Ding, and Jungong Han. 2020. Learning from multiple experts: Self-paced knowl- edge distillation for long-tailes classification. In Eu- ropean Conference on Computer Vision, pages 247- 263. Springer.
George Stoica. Emmanouil Antonios Platanios, and Barnaáás Póczos. 2021. Re-tacred: Addressing short- comings of the tacred dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pages 13843-13855
Chenhan Yuan, Qianqian Xie, and Sophia Ananiadou. 2023. Zero-shot temporal relation extraction with chatgpt. arXiv preprint arXiv:2304.05454
Teo Susnjak. 2023. Applying bert and chatgpt for senti- ment analysis of lyme disease in scientific literature. arXiv preprint 231.:230020044744
Bowen Zhang, Daijun Ding, and Liwen Jing. 2022. How would stance detection techniques evolve after the launch of chatgpt? arXiv preprint 12.iv:2212.14548.
Chris Sweeney and Maryam Najafian. 2019. A trans- parent framework for evaluating unintended demo- graphic bias in word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1662-1667.
Yuhao Zhang, Victor Zhong, Dangi Chen, Gabor Angeli, and Christopher D Manning. 2017. Position- aware attention and supervised data improve slot filling. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35- 45.
Lydia Tan and Geoff Ward. 2000. A recency-based account of the primacy effect in free recall. Journal of Experimental Psychology: Learning, Memory, and Cognition, 26(6):1589)
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint urXiv:2201.082
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706. PMLR

---

Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? a comparative study chatgpt and fine-tuned bert. arXiv preprint 230..
Wenxuan Zhou and Muhao Chen. 2022. An improved baseline for sentence-level relation extraction, pages 161-168.


Reviews:
[{"review_id": "ECM8Hkmxdl", "reviewer": "Reviewer_SvBZ", "summary": "", "questions": "Does Chain-of-thoughts prompting which increase the number of words in the answer improve such cognitive bias?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Comments from Reviewer SvBZ**\n \nWe greatly thank the reviewer SvBZ for his/her helpful and insightful comments. We provide our responses to the comments as follows.\n\n***Q1: Their evaluation of the primacy effect in ChatGPT is interesting. Performing experiments on more datasets might make the claim stronger.***\n \nResponse: Thanks for your helpful suggestion. We agree with the reviewer that the experiments on more datasets can further improve our paper. Therefore, we additionally conducted experiments on more NLP datasets: GoEmotions [1], MASSIVE [2], 20 Newsgroups [3] for a more comprehensive evaluation.  GoEmotions [1] is a dataset for fine-grained emotion classification. It is a corpus of 58k carefully curated comments extracted from Reddit, with human annotations for 27 emotion categories and a neutral one. MASSIVE [2] is a parallel dataset of > 1M utterances with annotations for the Natural Language Understanding tasks of intent prediction. Utterances span 60 intents. The 20 Newsgroups [3] dataset is a collection of approximately 20,000 newsgroup documents, partitioned across 20 different newsgroups.\n\nWe conduct the experiments following the experimental settings in Sec. 3 to evaluate ChatGPT on the above datasets. First, we observe the low consistency of ChatGPT confronted under label shuffling. As shown in the table below, ChatGPT changes its label prediction after label shuffling in over 69% of the test instances on the datasets of GoEmotions and in more than 30% of instances on MASSIVE and 20 Newsgroups.\n\n| Method|GoEmotions |MASSIVE |20 Newsgroups |\n|-|-|-|-|\n|ChatGPT w/ Prompt|69.3|32.8|34.1|\n|BERT w/ MLP|0.0|0.0|0.0|\n\nIn contrast, the fine-tuned BERT classifier maintains consistent predictions after label shuffling. This discrepancy challenges the widely-held belief that ChatGPT can better comprehend human instructions and provide consistent responses. One possible explanation is that ChatGPT’s understanding of the prompt is obtained by training on human-labeled data, which inherits humans’ cognitive bias of treating labels at different positions unfairly.\n\nTo deeper delve into the effects of label orders on ChatGPT, we analyze the distribution of predicted label indices (e.g., if the prediction is the first label, the label index is 1), as introduced in Section 2.2. Notably, we observe that the distribution of ChatGPT’s predictions consistently deviates from the uniform distribution, displaying a consistent bias towards smaller indices on the above datasets. In contrast, BERT exhibits no preference for label orders and consistently demonstrates a uniform distribution in its predicted label indices. The above experiments further reflect the primacy effect of ChatGPT on a wider range of NLP datasets. We follow your suggestions to add the above experiments and discussion to improve our paper.\n \n***Q2: It's better to shuffle the labels more times to prove the primacy effect.*** \n\nResponse: Thanks for your helpful suggestion. We agree with the reviewer that more times of label shuffling can better demonstrate the primacy effect of ChatGPT. Therefore, we perform the random label shuffling for every instance fifty times to do the experiments in Sec. 3. We observed a similar distribution of predicted indices of the test instances with more label shuffling to Fig. 4. In other words, given considerable times of label shuffling, ChatGPT’s predictions consistently deviate from the uniform distribution, displaying a consistent bias towards smaller indices across different datasets. The above results validate that the primacy effect is still significant for ChatGPT given more times of label shuffling. Quantitatively, we follow up with Sec. 3.4 to evaluate the fairness of ChatGPT by examining the imbalance or skewness in its predictions. To evaluate the label order bias of ChatGPT, we compute the average relative label order imbalance (LOI) as defined in Eq. (1). We conduct the fairness evaluation following the experimental settings described in Section 3.3, and present the results in the table below.\n\n| Method|TACRED|TACREV|Re-TACRED|Banking77|\n|-|-|-|-|-|\n|ChatGPT w/ Prompt|57.9|57.4|58.3|18.9|\n|BERT w/ MLP|1.9|2.1|2.2|1.9 |\n\nThe above results are close to those presented in Table 2, which further validates that the primacy effect of ChatGPT consistently exists given more times of label shuffling. We follow your suggestion to add this analysis to improve our paper.\n \n***Q3: Does Chain-of-thoughts prompting which increases the number of words in the answer improve such cognitive bias?*** \n \nResponse: Thanks for your question. We agree that testing whether CoT (Chain-of-thoughts) may influence the primacy effects can further improve our paper. Therefore, we accordingly add experiments following the experimental settings in Sec. 3. We observed the similar distribution of predicted indices of the test instances with CoT to Fig. 4. In other words, with CoT, ChatGPT still exhibits the primacy effect. Quantitively, we follow Sec. 3.4 to evaluate the fairness of ChatGPT with CoT by examining the imbalance or skewness in its predictions. To evaluate the label order bias of ChatGPT, we compute the average relative label order imbalance (LOI) as defined in Eq. (1). We conduct the fairness evaluation following the experimental settings described in Section 3.3, and present the results are in the below Table.\n\n| Method|TACRED|TACREV|Re-TACRED|Banking77|\n|-|-|-|-|-|\n|ChatGPT w/ Prompt|57.9|57.8|58.1|18.8|\n|ChatGPT w/ CoT|57.6|57.9|58.3|18.6|\n|BERT w/ MLP|1.8|1.9|2.3|2.1|\n\nThe above results show that with or without CoT, ChatGPT consistently exhibits the primacy effect. A reason for this phenomenon could be that the CoT encourages the LLMs for “slow thinking” about the question but does not necessarily mitigate the cognitive bias in the reasoning steps of CoT. We follow your suggestion to discuss this in detail to improve our paper.\n \n***Q4: Missing References: Mitigating Label Biases for In-context Learning Symbol tuning improves in-context learning in language models***\n \nResponse: We appreciate your suggestion and add this work to the discussion.\n \n\n[1] Demszky, Dorottya, et al. \"GoEmotions: A dataset of fine-grained emotions.\" arXiv preprint arXiv:2005.00547 (2020).\n \n[2]  FitzGerald, Jack, et al. \"Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages.\" arXiv preprint arXiv:2204.08582 (2022).\n \n[3] Albishre, Khaled, Mubarak Albathan, and Yuefeng Li. \"Effective 20 newsgroups dataset cleaning.\" 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT). Vol. 3. IEEE, 2015.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Their evaluation of primacy effect in ChatGPT is interesting.", "reasons_to_reject": "1. Performing experiments on more tasks might make the claim stronger, for example, other classification tasks or even generation tasks like QA/summarization.\n2. It's better to shuffle the labels more times to prove the primacy effect.", "questions_to_authors": "Does Chain-of-thoughts prompting which increase the number of words in the answer improve such cognitive bias?", "paper_topic_and_main_contributions": "This paper studies one cognitive bias, primacy effect, in ChatGPT, which tends to select labels that appear earlier in the context. And they find that ChatGPT's performance is sensitive to the order of labels in the prompt and they tend to select labels in earlier positions.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Mitigating Label Biases for In-context Learning\nSymbol tuning improves in-context learning in language models"}, "extra_text": {}}, {"review_id": "d32eaI8jSF", "reviewer": "Reviewer_Zv7g", "summary": "", "questions": "l28: citation for such a well-established concept as the primacy effect should not be some post --> remove and/or insert proper citation\n\nSection 3.1. - how do these datasets range on the complexity scale? \n\nHow were the actual prompts formatted? I would exchange Figure 2 with an actual example.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Comments from Reviewer Zv7g**\n \nWe greatly thank the reviewer Zv7g for his/her helpful and insightful comments. We provide our responses to the comments as follows.\n \n***Q1: The paper is interesting and self-contained - it contributes to the current effort of understanding the latest technological developments. While I see why the temperature was set to zero, it would still be good to compare the results with other temperatures (medium, high), to confirm the findings.***\n \nResponse: Thanks for your helpful suggestion on comparing results with other temperatures. Accordingly, we add more settings with temperatures of 0.5 and 0.8, and conduct the experiments following the experimental setup in Sec. 3. Consequently, we observe the lower consistency of ChatGPT confronted under label shuffling given higher temperatures. As shown in the below Table, ChatGPT changes its label prediction after label shuffling in over 88% of the test instances on the TACRED, TACREV, and Re-TACRED datasets when the temperature is 0.5, and in more than 90% of instances when the temperature is 0.8.\n\n| Method| TACRED|TACREV|Re-TACRED|Banking77|\n|-|-|-|-|-|\n|ChatGPT w/ Prompt (temperature is 0.0)|87.9|85.9|88.6|35.7|\n|ChatGPT w/ Prompt (temperature is 0.5)|89.0|88.2|89.3|43.2|\n|ChatGPT w/ Prompt (temperature is 0.8)|90.5|90.2|91.1|48.1|\n|BERT w/ MLP|0.0|0.0|0.0|0.0|\n\nThe reason for the above results is that the ChatGPT’s outputs are more diverse given higher temperatures, which leads to lower prediction consistency. In addition, we observed the distribution of predicted indices of the test instances with higher temperatures similar to Fig. 4. In other words, with higher temperatures, ChatGPT still exhibits the primacy effect. We follow your suggestions to add the above analysis to improve our paper.\n \n***Q2: Insert a more proper citation for the primacy effect.***\n \nResponse: Thanks for your helpful suggestion. We have accordingly cited the paper\n“Asch, Solomon E. \"Forming impressions of personality.\" The Journal of Abnormal and Social Psychology 41.3 (1946): 258.”\nas the reference for the primacy effect. This paper first examined the primacy effect in a study using sentences with reversed order of adjectives. We follow your suggestions to add this citation to improve our paper.\n \n***Q3: How do the used datasets range on the complexity scale?***\n \nResponse: Thanks for your question! We chose relation extraction and intent detection tasks in our experiments since these tasks come with adequately sized decision spaces to illustrate the underlying primacy effect of labels. We agree that more comprehensive experiments on more datasets can further improve our paper. Therefore, we additionally conducted experiments on more NLP datasets: GoEmotions [1], MASSIVE [2], 20 Newsgroups [3] for a more comprehensive evaluation.  GoEmotions [1] is a dataset for fine-grained emotion classification. It is a corpus of 58k carefully curated comments extracted from Reddit, with human annotations for 27 emotion categories and a neutral one. MASSIVE [2] is a parallel dataset of > 1M utterances with annotations for the Natural Language Understanding tasks of intent prediction. Utterances span 60 intents. The 20 Newsgroups [3] dataset is a collection of approximately 20,000 newsgroup documents, partitioned across 20 different newsgroups.\n\nWe conduct the experiments following the experimental settings in Sec. 3 to evaluate ChatGPT on the above datasets. First, we observe the low consistency of ChatGPT confronted under label shuffling. As shown in the table below, ChatGPT changes its label prediction after label shuffling in over 69% of the test instances on the datasets of GoEmotions and in more than 30% of instances on MASSIVE and 20 Newsgroups.\n\n| Method|GoEmotions |MASSIVE |20 Newsgroups|\n|-|-|-|-|\n|ChatGPT w/ Prompt|69.3|32.8|34.1|\n|BERT w/ MLP|0.0|0.0|0.0|\n\nIn contrast, the fine-tuned BERT classifier remains with consistent predictions after label shuffling. This discrepancy challenges the widely-held belief that ChatGPT can comprehend human instructions and provide consistent responses. One possible explanation is that ChatGPT’s understanding of the prompt is obtained by training on human-labeled data, which inherits humans’ cognitive bias of treating labels at different positions unfairly.\n\nTo deeper delve into the effects of label orders on ChatGPT, we analyze the distribution of predicted label indices (e.g., if the prediction is the first label, the label index is 1), as introduced in Section 2.2. Notably, we observe that the distribution of ChatGPT’s predictions consistently deviates from the uniform distribution, displaying a consistent bias towards smaller indices on the above datasets. In contrast, BERT exhibits no preference for label orders and consistently demonstrates a uniform distribution in its predicted label indices. The above experiments further reflect the primacy effect of ChatGPT on a wider range of NLP datasets. We follow your suggestions to add the above experiments and discussion to improve our paper.\n \n***Q4: How were the actual prompts formatted?***\n \nResponse: Thanks for your suggestion. As we have written in Sec. 2, we follow the existing work [4,5] to provide the prompts including two key components: (i) label definitions, and (ii) a task description and input text. We follow your suggestion to elaborate on this to improve our paper. Fig. 2 is an actual example that we use for ChatGPT on intent detection on the dataset Banking77. We do not list all the label definitions here since there are 77 labels in the dataset and listing all of them will take up excessive space. We agree with you that more prompt examples that we use for different datasets can help the readers understand our work. We follow your suggestion to list them in the Appendix for the readers’ easy reference. \n \n***Q5: There are minor typos in the paper, for instance in the abstract: `Does ChatGPT also inheritS'***\n \nResponse: Thanks for your helpful suggestion. We follow your suggestion to revise the typo ‘inheritS’ to ‘inherits’. Also, we carefully recheck the whole draft to correct the typos to improve our paper.\n \n[1] Demszky, Dorottya, et al. \"GoEmotions: A dataset of fine-grained emotions.\" arXiv preprint arXiv:2005.00547 (2020).\n \n[2]  FitzGerald, Jack, et al. \"Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages.\" arXiv preprint arXiv:2204.08582 (2022).\n \n[3] Albishre, Khaled, Mubarak Albathan, and Yuefeng Li. \"Effective 20 newsgroups dataset cleaning.\" 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT). Vol. 3. IEEE, 2015.\n \n[4] Wei, Xiang, et al. \"Zero-shot information extraction via chatting with chatgpt.\" arXiv preprint arXiv:2302.10205 (2023).\n \n[5] Yuan, Chenhan, Qianqian Xie, and Sophia Ananiadou. \"Zero-shot temporal relation extraction with chatgpt.\" arXiv preprint arXiv:2304.05454 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is interesting and self-contained - it contributes to the current effort of understanding latest technological developments.", "reasons_to_reject": "The paper does not contribute any new technological development in NLP, it instead reflects on the current SOTA.\n\nl. 156: While I see why the temperature was set to zero, it would still be good to compare the results with other temperatures (medium, high), to confirm the findings.", "questions_to_authors": "l28: citation for such a well-established concept as the primacy effect should not be some post --> remove and/or insert proper citation\n\nSection 3.1. - how do these datasets range on the complexity scale? \n\nHow were the actual prompts formatted? I would exchange Figure 2 with an actual example.", "paper_topic_and_main_contributions": "This paper reports on a study of the primacy effect in zero-shot ChatGPT which finds that the order to labels in the prediction task have an influence on the prediction result.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are some minor typos across the paper, for instance in the abstract: `Does ChatGPT also inheritS'"}, "extra_text": {}}, {"review_id": "zmRMVWqfKt", "reviewer": "Reviewer_4BGH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Comments from Reviewer 4BGH**\n \nWe greatly thank the reviewer 4BGH for his/her helpful and insightful comments. We provide our responses to the comments as follows.\n \n***Q1: This work presents an interesting perspective by investigating whether the order of candidate labels affects model performance and finds that ChatGPT tends to select labels in earlier positions in the prompt. The finding helps to understand how prompt works and facilitates the study of more powerful prompt techniques. In the experiments, do the authors consider the influence of temperature setting, since a positive temperature can make ChatGPT provide different predictions even with the same input?***\n \nResponse: Thanks for your question. We agree with the reviewer that for ChatGPT, a positive temperature value could lead to high randomness of the model’s output and thus influence the experiments and analysis. We have considered this issue in our initial experiments. As we have written in Sec. 3.1, we set the temperature as 0.0 to remove the randomness of ChatGPT’s outputs. In this way, our experiments and analysis on the primacy effect of ChatGPT are not influenced by the temperature-related randomness of ChatGPT.\n\nBesides, We believe that it is beneficial to compare the results with other temperatures. Therefore, we additionally set the temperature of ChatGPT as 0.5 and 0.8, and we conduct the experiments following the experimental settings in Sec. 3 to evaluate ChatGPT given different temperatures. First, we observe the lower consistency of ChatGPT confronted under label shuffling given higher temperatures. As shown in the below Table, ChatGPT changes its label prediction after label shuffling in over 88% of the test instances on the TACRED, TACREV, and Re-TACRED datasets when the temperature is 0.5, and in more than 90% of instances when the temperature is 0.8.\n\n| Method| TACRED|TACREV|Re-TACRED|Banking77|\n|-|-|-|-|-|\n|ChatGPT w/ Prompt (temperature is 0.0)|87.9|85.9|88.6|35.7|\n|ChatGPT w/ Prompt (temperature is 0.5)|89.0|88.2|89.3|43.2|\n|ChatGPT w/ Prompt (temperature is 0.8)|90.5|90.2|91.1|48.1|\n|BERT w/ MLP|0.0|0.0|0.0|0.0|\n\nThe above observation is attributed to that the ChatGPT’s outputs are more diverse given higher temperatures, leading to less consistent predictions. In addition, we observe that the distribution of predicted indices of the test instances with higher temperatures is similar to Fig. 4. In other words, with higher temperatures, ChatGPT still exhibits the primacy effect. We follow your suggestions to emphasize the above analysis to improve our paper.\n \n***Q2: It is unclear whether the phenomenon happens just because ChatGPT has low confidence when generating the labels for the input questions.***\n\nResponse: Thanks for your helpful suggestion. We agree with you that ChatGPT may not be confident on some instances. However, no matter whether confident or not, a natural language understanding (NLU) model is desired to give consistent predictions no matter how the labels are ordered. An example is a finetuned BERT, which is shown to always give consistent predictions that are invariant to the order of labels. This property is not satisfied by ChatGPT, as found in our work.\n\nWhile previous work often takes ChatGPT as a universal method applicable to all text-related tasks, we argue that its flexibility comes with the risk of inheriting human's cognitive biases. These biases lead to unfair judgments which can affect the performance of the machine learning model. This work studies a cognitive bias of ChatGPT: primacy effects. We propose a simple yet effective label shuffling method to analyze the influence of label orders on ChatGPT. We discover the primacy effects of ChatGPT and finds that it highly influence the fairness of ChatGPT in NLU. Our work contributes to better understanding the behaviors of ChatGPT and building more faithful solutions with ChatGPT in NLU applications.\n \nWe also agree that the evaluation of ChatGPT on more NLP datasets can further demonstrate the primacy effect of ChatGPT. Therefore, we additionally conducted experiments on more NLP datasets: GoEmotions [1], MASSIVE [2], 20 Newsgroups [3] for a more comprehensive evaluation.  GoEmotions [1] is a dataset for fine-grained emotion classification. It is a corpus of 58k carefully curated comments extracted from Reddit, with human annotations for 27 emotion categories and a neutral one. MASSIVE [2] is a parallel dataset of > 1M utterances with annotations for the Natural Language Understanding tasks of intent prediction. Utterances span 60 intents. The 20 Newsgroups [3] dataset is a collection of approximately 20,000 newsgroup documents, partitioned across 20 different newsgroups.\n\nWe conduct the experiments following the experimental settings in Sec. 3 to evaluate ChatGPT on the above datasets. First, we observe the low consistency of ChatGPT confronted under label shuffling. As shown in the below Table, ChatGPT changes its label prediction after label shuffling in over 69% of the test instances on the datasets of GoEmotions and in more than 30% of instances on MASSIVE and 20 Newsgroups.\n\n| Method|GoEmotions |MASSIVE |20 Newsgroups |\n|-|-|-|-|\n|ChatGPT w/ Prompt|69.3|32.8|34.1|\n|BERT w/ MLP|0.0|0.0|0.0|\n\nIn contrast, the fine-tuned BERT classifier maintains consistent predictions after label shuffling. This discrepancy challenges the widely-held belief that ChatGPT can comprehend human instructions and provide consistent responses. One possible explanation is that ChatGPT’s understanding of the prompt is obtained by training on human-labeled data, which inherits humans’ cognitive bias of treating labels at different positions unfairly.\n\nTo delve deeper into the effects of label orders on ChatGPT, we analyze the distribution of predicted label indices (e.g., if the prediction is the first label, the label index is 1), as introduced in Section 2.2. Notably, we observe that the distribution of ChatGPT’s predictions consistently deviates from the uniform distribution, displaying a consistent bias towards smaller indices on the above datasets. In contrast, BERT exhibits no preference for label orders and consistently demonstrates a uniform distribution in its predicted label indices. The above experiments further reflect the primacy effect of ChatGPT on a wider range of NLP datasets. We follow your suggestions to add the above experiments and discussion to improve our paper.\n \n[1] Demszky, Dorottya, et al. \"GoEmotions: A dataset of fine-grained emotions.\" arXiv preprint arXiv:2005.00547 (2020).\n \n[2]  FitzGerald, Jack, et al. \"Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages.\" arXiv preprint arXiv:2204.08582 (2022).\n \n[3] Albishre, Khaled, Mubarak Albathan, and Yuefeng Li. \"Effective 20 newsgroups dataset cleaning.\" 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT). Vol. 3. IEEE, 2015.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work presents an interesting perspective by investigating whether the order of candidate labels affects model performance and finds that ChatGPT tends to select labels in earlier positions in the prompt. The finding may help understand how prompt works and facilitate studied of more powerful prompt techniques.", "reasons_to_reject": "The experiment is preliminary as there are still unresolved concerns that may result in different conclusions. \n\nFirstly, it is reasonable that ChatGPT may provide different predictions even with the same input (with different temperature settings during decoding) because of the nature of a generation model. The comparison with BERT might be unfair to justify if the phenomenon happens in ChatGPT only or other generation models. \n\nSecondly, it is unclear whether the phenomenon happens just because ChatGPT has low confidence when generating the labels for the input questions, in which case ChatGPT just gives random predictions.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work studies the problem of whether ChatGPT inherits humans' cognitive bias by selecting the labels at earlier positions as the answer. To begin with, this work introduces the definition of the primacy effect as the tendency to recall information presented at the start of a list better than the information at the middle or end. Then, this work takes the natural language understanding tasks as the testbed and analyzes the phenomenon by shuffling labels listed in a prompted input before every prediction. Finally, this work compares the predictions on the same instance with two different label orders and counts the predicted label indices on many instances with label shuffling. \n\nThis paper finds that ChatGPT’s prediction is sensitive to the order of labels in the prompt and ChatGPT tends to select labels in earlier positions in the prompt. This work would contribute to the research line of investigating how prompts affect the model performance. As recent studies have revealed the sensitivity of the order of in-context learning examples and the influence of label correctness, this work may provide another interesting perspective with regard to the order of candidate labels for an individual test instance. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the ‘primacy effect’ in ChatGPT (The primacy effect refers to a tendency to better remember the first piece of information compared to information received later on).
Authors observe that ChatGPT's decision is sensitive to the order of candidate outputs/labels in the prompt.

All reviewers pointed out the interestingness of the key research question -- does ChatGPT exhibit a primacy effect or not? The study has the potential to advance our understanding of the working of LLMs. Reviewers had raised a few concerns and authors seem to have addressed most of them through the rebuttals. For example, authors provided results on additional datasets, with additional temperature settings, and also in a setup where CoT (chain-of-thought) strategy is used with ChatGPT. 

One of the reviewers raised a concern if the observed effect can be explained by the low confidence of ChatGPT for a subset of data. Authors should consider addressing the reviewer's point in future revisions through additional experiments.
Decision: Accept-Main