LLM Evaluators Recognize and Favor Their Own Generations

Authors: arjun  panickssery
samuel r bowman
shi  feng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 07:59:19

Abstract: Self-evaluation using large language models (LLMs) has proven valuable not only in benchmarking but also methods like reward modeling, constitutional AI, and self-refinement. But new biases are introduced due to the same LLM acting as both the evaluator and the evaluatee. One such bias is self-preference, where an LLM evaluator scores its own outputs higher than others’ while human annotators consider them of equal quality. But do LLMs actually recognize their own outputs when they give those texts higher scores, or is it just a coincidence? In this paper, we investigate if self-recognition capability contributes to self-preference. We discover that, out of the box, LLMs such as GPT-4 and Llama 2 have non-trivial accuracy at distinguishing themselves from other LLMs and humans. By finetuning LLMs, we discover a linear correlation between self-recognition capability and the strength of self-preference bias; using controlled experiments, we show that the causal explanation resists straightforward confounders. We discuss how self-recognition can interfere with unbiased evaluations and AI safety more generally.

Main paper: 
 introduction: 
 Self-evaluation is becoming a prominent part of the large language model (LLM) lifecycle. In methods like reward modeling (Leike et al., 2018; Stiennon et al., 2020) , model-based benchmarks (Shashidhar et al., 2023; Zeng et al., 2023; Yuan et al., 2023; Fu et al., 2023; Li et al., 2024) , self-refinement (Saunders et al., 2022; Madaan et al., 2023; Lee et al., 2023; Shridhar et al., 2023) , and constitutional AI (Bai et al., 2022) , LLMs are increasingly used to provide assessment, supervision, and oversight for themselves and other LLMs. LLM evaluators are shown to be highly accurate at approximating human annotators on various tasks, and are significantly more scalable (Hackl et al., 2023) .In self-evaluation, as the name suggests, the same underlying LLM acts as both the evaluator and the evaluatee. As a result, the neutrality of the evaluator is in question, and the evaluation can suffer from biases where the LLM evaluators diverge from humans in systematic ways (Zheng et al., 2024; Bai et al., 2024) . One such bias is self-preference, where an LLM rates its own outputs higher than texts written by other LLMs or humans, while human annotators judge them as equal quality. Self-preference has been observed in GPT-4-based dialogue benchmarks (Bitton et al., 2023; Koo et al., 2023) , as well as for text summarization (Liu et al., 2023) .Towards understanding and mitigating self-preference, we study self-recognition-an LLM's capability of recognizing its own outputs. We ask: Is self-preference truly self -preference, in the sense that the LLM prefers a text because it was generated by itself?We measure their correlation while using prompting and fine-tuning to alter the LLM's self-recognition capability. In order to provide signals for the causal link between self-recognition and self-preference, we also fine-tune the LLM on a comprehensive set of potential confounding properties.38th Conference on Neural Information Processing Systems (NeurIPS 2024). Self-recognition score 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Llama 2 GPT-3.5 GPT-4 Model N/A Control tasks Self-recognition (N=10) Self-recognition (N=500)In-domain Out-of-domainFigure 1 : The strength of self-preference bias is linearly correlated with the LLM's self-recognition capability. Each point represents a model evaluated on the two properties on the CNN/Dailymail (left) and XSUM (right) datasets. We fine-tune GPT-3.5 and Llama 2 for self-recognition or control tasks using both in-and out-of-domain data. The scores represented by both axes can be interpreted as measures of the LLM's confidence on these properties.Our main findings are as follows:1. Frontier LLMs exhibit self-preference in self-evaluation. On two summarization tasks, LLMs (GPT-3.5 Turbo, GPT-4, and Llama 2) disproportionately favor summaries written by themselves over those by other LLMs and from humans.2. LLMs have non-trivial self-recognition capability out of the box. All three LLMs we evaluate achieve over 50% accuracy at distinguishing their own outputs from other sources using simple prompts without fine-tuning. GPT-4 is 73.5% accurate at distinguishing its outputs from those of two other LLMs and humans.3. Fine-tuning leads to near-perfect self-recognition. GPT-3.5 and Llama 2 both achieve over 90% accuracy at self-recognition after fine-tuning on 500 examples.4. Self-preference strength is linearly correlated with self-recognition. We fine-tune LLMs to increase or decrease self-recognition, and find a linear trend between them (Figure 1 ).2 Definition and measurement of self-preference and self-recognition Self-preference is when an LLM favors its own outputs over texts by human or other LLMs.Self-recognition is an LLM's ability to distinguish its outputs from texts by humans or other LLMs.For both definitions, we follow the prosaic rather than the intentional interpretation. We use the term "self" in an empirical sense, without claiming that the LLMs have any notion or representation of itself. The prosaic interpretation allows these two concepts to exist independent of one another: An LLM can prefer texts it generated without recognizing that those texts were in fact generated by itself.In our experiments, one LLM can play up to three different roles: generator, evaluator, and authorship identifier, depending on the instructions that they are given. Statements like "an LLM evaluator is used to evaluate texts generated by itself " emphasize the fact that the generator and the evaluator use the same underlying model, although it receives different prompts and can behave differently. 
datasets and models: 
 We focus on text summarization, a task where self-evaluation is used extensively in both model-based benchmarks and reward modeling (Stiennon et al., 2020; Wu et al., 2021) . We base our experiments on 2,000 randomly sampled news articles from two datasets: XSUM (Narayan et al., 2018) and CNN/DailyMail (Nallapati et al., 2016) Each example includes the original news article and a human-written summary.To produce and evaluate LLM-generated summaries, we use instruction-tuned LLMs: Llama-2-7b-chat (Touvron et al., 2023) , GPT-3.5 (Brown et al., 2020), and GPT-4 (OpenAI, 2023) . Our fine-tuning experiments include Llama 2 (our own implementation) and . We sample summaries from each of the LLMs at temperature zero using simple prompts, and standardize the punctuation and initial capitalization to match the formatting of the human summaries. The following evaluations of out-of-the-box LLMs are all based on these sampled summaries.1 For both properties, we measure LLMs by prompting them using one of the following two formats:Pairwise measurement. The LLM is presented two pieces of text, one generated by itself, and another by a second LLM or a human. We ask the LLM to choose which text has higher quality for self-preference, and which one was generated by itself for self-recognition. The identity of the alternative source is not revealed to the evaluator. We compute a prediction confidence by normalizing the output probabilities of the tokens associated with the two options.To account for the LLMs' ordering bias (Pezeshkpour & Hruschka, 2023) , we prompt the LLMs twice for each example by swapping the options, computing the average of the two confidence scores.Individual measurement. The LLM is presented a single summary either generated by itself, another LLM, or a human. For self-recognition, we prompt the LLM evaluator to answer a yes/no question of whether a given summary is its own generation, with the confidence score computed in the same way as pairwise measurement. For self-preference, we prompt the LLM evaluator to rate a given summary with a Likert-scale rating of one to five for a given summary, computing the final rating as the average of the five possible scores weighted by the output probability of each number token.To make scores comparable across individual and pairwise measurements, we further normalize the individually measured confidence and Likert-scale scores. Suppose that in a pairwise measurement, GPT-4 is evaluated for self-recognition between its own summary and one by Llama 2. In individual measurement, we gather these two scores separately and divide GPT-4's score for its own summary by the sum of the scores to compute GPT-4's confidence on this pair of examples. We similarly re-normalize the Likert-scale scores for self-preference.We refer to the results of these measurements as self-recognition and self-preference scores (noting whether they are from the pairwise or individual setting); taking the average of these scores over a whole dataset provides the overall self-recognition and self-preference scores of an evaluator model.Figure 2 (left) visualizes the self-recognition capability of each LLM evaluator in the pairwise setting.The results appear consistent across the two datasets, so we aggregate them for clarity. Our main observations are the following:1. GPT-4 is significantly more capable at self-recognition than the two weaker models. 2. All three LLM evaluators most easily distinguish their summaries from human-written summaries. Human is the only alternative that Llama 2 can effective distinguish itself from. 3. Weaker LLMs struggle to distinguish themselves from stronger LLMs: Llama 2 is completely incapable of distinguishing itself from GPT-3.5 and GPT-4, and GPT-3.5 struggles to distinguish itself from GPT-4. 4. The GPT-4 evaluator does not distinguish Llama 2 summaries from its own summaries more easily than GPT-3.5 summaries. This goes against our intuition that self-recognition scores should increase as the dissimilarity between evaluator and evaluatee increases. having two pieces of texts to compare and contrast. GPT-4 is the only model capable of distinguishing authors with non-trivial accuracy. Interestingly, looking at the un-normalized confidence scores (Figure 3 ), we see that GPT-4 is strongly biased against predicting any text as being generated by itself, regardless of the actual author; GPT-3.5 and Llama 2 show the opposite bias.Self-preference (pairwise) 0.0 Figure 5 : Evaluator responses when treated as a binary response without considering confidence scores. Results are marked "ambiguous" if the response reverses when the options are swapped.All models show the greatest self-preference against the human summary. In the individual setting, we observe that Llama doesn't give very different scores to the different sources, while GPT-4 and GPT-3.5 show signs of self-preference. With the exception of GPT-3.5 on the XSUM dataset, evaluators show a tendency to give the human summary lower scores compared to model-generated summaries and a tendency to give their generations higher scores compared to the scores given by other evaluators (Figure 4 ).We conduct human evaluation of pairwise judgment on generated summaries to check if the selfpreference shown by LLMs is justified by quality differences, particularly between LLM-generated summaries. We sample 50 summaries from each dataset for each of the three LLMs, and collect three human judgments for each pair, resulting in 900 pairwise comparisons. We recruit 20 annotators from Upwork who are English speakers located in the United States with bachelor's degrees in humanities disciplines. The win rate of each pair of LLMs, calculated over 300 human annotators, is as follows:1. GPT-4 vs. GPT-3.5: 57% 2. GPT-4 vs. Llama 2: 63% 3. GPT-3.5 vs. Llama 2: 58% The results suggest that human annotators rank the three LLMs as GPT-4 > GPT-3.5 > Llama 2, consistent with their performance on various benchmarks. But the disparity between LLMs as rated by humans is significantly lower than the level of self-preference exhibited by the LLMs, in particular GPT-4. This suggests that out of the box, the LLMs' self-preference is disproportionate to the actual quality differences.All three evaluators models display ordering bias. GPT-4, GPT-3.5, and Llama reverse their pairwise preferences when the ordering of options is reversed at rates of 25%, 58%, and 89% respectively, averaged across tasks and datasets (Figure 5 ). We account for this bias by averaging the logit-based confidence scores across the two orderings.An alternative interpretation of the data is, for each evaluator, to discard all the results as "ambiguous" where its preference displayed ordering-based reversal, reporting an evaluator's self-recognition ability and self-preference tendency as its frequency of recognizing or preferring its own summary in "unambiguous" cases (Figure 5 ). This method exposes differences in evaluator results between the two datasets, but supports the presence of out-of-the-box self-recognition and self-preference.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We have a designated limitation section discussing uncertainties in our findings.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
measuring correlation between self-preference and self-recognition: 
 Having validated the existence of self-preference and self-recognition, we now turn to study their correlation. The main research question is to understand whether they have a causal relationship. Our hypothesis is that self-recognition causes self-preference-that LLMs prefer their own outputs because they recognize them. Our investigation is motivated by the safety implications, which we discuss in detail in Section 5.We fine-tune LLMs to alter their self-recognition ability, and measure how their self-preference changes accordingly. The correlation alone doesn't prove the causal hypothesis, whose validation requires mechanistic tools that do not exist for LLMs yet (discussion in Section 5). But we provide more evidence for the hypothesis by (1) invalidating the inverse claim that self-preference causes self-recognition (Section 3.3) and ( 2) controlling for confounders by fine-tuning on unrelated tasks (Section 3.4).We use supervised fine-tuning on pairwise examples to alter the LLMs' self-recognition capability.The training examples are generated in the same way as the evaluation examples we use in the pairwise measurement. Each example consists of two summaries of a given article, one by the LLM evaluator and the other by another LLM or human source. The evaluators are trained to predict the final token, representing the correct choice among two options, using supervised learning with cross-entropy loss. To train an LLM evaluator, we use 500 training articles and collect examples from all alternative sources (two other LLMs and human). The remaining 500 articles and associated summaries are used for evaluation. We train on the two datasets separately but evaluate both in-and out-of-domain. GPT-3.5 was fine-tuned via API, and Llama 2 using our own implementation. The Llama models are quantized to 8 bits and fine-tuned for one epoch using Adam optimization and a learning rate of 5.0 × 10 -5 .Fine-tuning on the self-recognition task causes models to become more confident in their pairwise judgments (Figure 6 ). When plotting self-recognition score against self-preference, the data clump more tightly around 0.5 and the high-and low-score corners.In the pairwise setting, fine-tuning for self-recognition results in a strong linear correlation between a model's self-recognition ability and its self-preference (Figure 7 ). We find that fine-tuning for self-recognition ability on one dataset transfers to the other.In additional to analyzing the relationship between self-recognition ability and overall dataset selfpreference, we measure the correlation between these two properties on the example level (Table 1 ). For GPT-3.5 on the XSUM dataset, the evaluator prior to fine-tuning has a correlation of 0.41 (Kendall's τ ) between correctly recognizing its summary from a pair and preferring its summary from that same pair. Every fine-tuning configuration we employ results in a model with a positive correlation on this metric and this correlation does not change meaningfully as self-recognition ability increases.We ensure that the causal direction is not the reverse-that the LLM does not recognize its own outputs because the quality is objectively higher, which would not reflect favoritism or raise safety concerns-by showing that LLM evaluators do not systematically favor summaries generated by fine-tuned models to those generated by the original model. Some fine-tuning runs resulted in degraded generation quality. The remainder show an average preference for the fine-tuned model's In-domain Out-of-domainFigure 7 : Fine-tuning evaluators for self-recognition results across models and datasets results in a linear trend connecting evaluator self-recognition to self-preference. This effect persists when fine-tuning on one dataset and measuring results on the other dataset. To confirm that the fine-tuning procedure did not introduce any confounding effect on the variables under study, we additionally fine-tune GPT-3.5 and Llama on three tasks unrelated to self-recognition: length, vowel count, and Flesh-Kincaid readability score (Flesch, 1948) . We also fine-tune both models to always output the answer "1" and to answer randomly. Together with the three fine-tuned models trained on the self-recognition task, this gives eight fine-tuned models for each of the two datasets. We find that these models can be placed in the same linear trend that we see in the models fine-tuned on the self-recognition task (Figure 1 ).An additional experiment using only prompting to show evidence of a causal link between selfrecognition and self-preference is to add either correct or reversed labels indicating the source of each summary in the pairwise self-preference question (Table 6 ). The GPT-4 and GPT-3.5 evaluator models show a reversal in self-preference when the labels are reversed in the XSUM dataset; that is, they show a preference for the summary labeled as being their own generation. In the CNN dataset or the Llama evaluator, no clear difference is shown from label reversal. 4 Related work 4.1 Self-preference and bias in self-evaluationThe tendency of LLMs to prefer their own generations was first recognized in the context of LLMbased benchmarks (Bitton et al., 2023; Zheng et al., 2024; Bai et al., 2024) . Like us, Liu et al. (2023) study self-preference bias on text summarization, between BERT, T5, and GPT-3.5. The larger capability gap between these models makes it difficult to control for summarization quality. Koo et al. (2023) include self-preference in a suite of tests for LLM cognitive biases in a pairwise question-answering setting. They find GPT-4 to demonstrate lower self-preference than GPT-3.5 out-of-the-box, contrary to our findings, which suggests that wider evaluation is needed to draw generalizable conclusions. Neither of these previous works attempted to provide an explanation for self-preference nor to alter self-preference strength.Hoelscher-Obermaier et al. ( 2023) evaluate GPT-3.5, GPT-4, and Claude-2 for self-recognition ability on pairs of ten-sentence fables based on BIG-bench (Srivastava et al., 2023) . On this task, contrary to our findings, GPT-3.5 is more accurate than GPT-4, which is less than 50% accurate, again showing the need for wide experimentation on varied datasets.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: We specify these details in the experiment section. Additionally the experiment details can be confirmed using the code and data included in the zip file. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss in depth the generalizability of claims made in the paper, in particular the impacts of the results for AI safety. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
llm detection: 
 Detection of LLM-generated text is important for AI safety and combating misinformation (Jawahar et al., 2020; Crothers et al., 2023; Wu et al., 2023; Yang et al., 2023; Kumarage et al., 2024) . Despite having similar goals, self-recognition focuses on the introspective ability of language models, rather a third party's discernment between varied sources of text. The self-recognition task can be seen as a highly restricted version of detection where the method is limited to prompting an LLM. In particular, the detector LLM is not given explicit access to information such as perplexity, which is crucial to many detection methods (Mitchell et al., 2023; Hans et al., 2024) .5 Limitations, discussion, and conclusion 
safety concerns related to self-recognizing llms: 
 Self-recognition is a general ability that can potentially affect many multi-LLM interactions. In this paper, we focus on self-preference as the downstream property and provide initial evidence towards causation, but we see evidence of generalization to additional downstream properties. In particular, by evaluating LLMs on datasets with distinct construction processes, we observe that self-recognition fine-tuning generalizes across the two datasets and that our hypothesis holds out-of-distribution. Motivated by these results, we discuss safety risks caused by self-recognition and its causal effect on various biases.Biased self-evaluation In model-based benchmarks, a model's rating can be inflated simply because it is similar to the evaluator model. The bias is also a risk for methods designed for safety and alignment, such as reward modeling (Leike et al., 2018; Stiennon et al., 2020) and constitutional AI (Bai et al., 2022) , for similar reasons: the reward model gives higher scores to models similar to itself, leading to weaker oversight and supervision. The bias can be amplified if the model is updated with feedback or training signal generated by itself Pan et al. (2024) ; Xu et al. (2024) .Our work provides a basis for countermeasures against self-preference. If future evaluation confirms self-preference to be as pervasive as other biases such as ordering bias, countermeasures such as authorship obfuscation should be incorporated into standard prompting practice.White-box adversarial attacks for free and unbounded reward hacking In an adversarial setting (see Raina et al. (2024) for example), an LLM defender is no longer protected by black-box access if the adversary LLM recognizes their similarities. In the worst case scenario where the adversary uses the same LLM as the defender, the adversary can gain unbounded access to the defender. A similar concern applies to the non-adversarial setting, where similar LLMs are use as both optimizer and reward model, as well: the strength of potential reward hacking is unbounded even if the two LLMs only communicate textually. For example, the optimizer can ignore the feedback provided by the reward model, and instead directly optimize for the shared, unaligned representation of the human-specified objectives.Controlling for ground-truth generation quality. Self-preference is justified if the LLM's generation actually is higher in quality. From a safety perspective, our interest is when an LLM prefers its own outputs that are of equal or worse quality than the alternative. This requires controlling for generation quality using ground-truth annotation when measuring self-preference. Our existing results provide indirect evidence for disproportionate self-preference: the sum of mutual self-preference scores for a pair of LLMs exceeds 1, so for at least a portion of the dataset they each prefer themselves.Example-level causal hypothesis. Our central hypothesis can be interpreted on either the example or capability level. We focus on the capability level: high self-recognition capability causes LLMs to show stronger self-preference. The example level counterpart would be: an LLM shows preference towards a piece of text because it recognizes the text as its own generation, an hypothesis of interest to interpretability. Although we observe on the correlation of the two properties on the confidence of individual predictions, our control experiments cannot further the causal argument on the example level. One approach to gather evidence for the example-level causal hypothesis is to perturb or paraphrase LLM-generated text to inhibit self-recognition and measure self-preference.Limited number of experiment conditions. We focus on text summarization as a realistic problem with existing high quality data that have seen successful application of self-evaluation. Our crossdataset evaluation provides initial evidence that self-recognition is a general capability that can be amplified easily by fine-tuning on a small number of examples from one dataset. Our future work will validate the hypothesis on more text summarization datasets, more tasks, as well as more frontier LLMs. We will also experiment with fine-tuning for self-recognition on the general domain rather than on a specific task.Variance reduction. Our preliminary experiments indicate that the strength of both properties are insensitive to prompts, so all conditions use the same straightforward prompt design. To reduce variance, we will expand our experiments with more prompt designs in future work, including instructions to condition LLMs for better calibration (and reduce rejection responses). Along the lines of fine-tuning on the general domain, we will also mix self-recognition with standard instruction following datasets to improve coverage on the spectrum of self-recognition signal strength.We provide initial evidence towards the hypothesis that LLMs prefer their own generations because they recognize themselves. In addition to evaluating LLMs out-of-the-box, we show that finetuning on a small number of examples elicit strong, generalizable self-recognition capability on summarization datasets. By varying fine-tuning task, we observe a linear correlation between selfrecognition and self-preference, and validate that the correlation cannot be explained away by potential confounders. Our results establish self-recognition as a crucial factor in unbiased self-evaluation as well as an important safety-related property. The experiment design also provides a blueprint to explore the effects of self-recognition on other downstream properties. Justification: The abstract summarizes the main findings of the paper faithfully.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: We discuss potential mitigation methods against risks caused by selfrecognizing LLMs.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly credit datasets that we use in experiments and ensure that they are properly licensed.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Our code for evaluating GPT-4, and Llama2, as well as for fine-tuning Llama 2, is available at https://bit.ly/llm_self_recognition. 
 
 Appendix:  


Decision: I am very certain that this paper should be accepted and as it is the highest rated paper in my cohort I am recommending it for the highest level of acceptance which would be an oral presentation.  The main reason for this is that I believe that the tendency to use an LLM as a judge when evaluating an outcome is quite common and the authors here surface an important bias of LLMs to favor their own generations which strongly implies that GPT-4 would not be an impartial judge between outcomes form a GPT4 pipeline vs another LLM pipeline.  Given the importance of this bias for many I am recommending oral so that this gets the most exposure.  The key weakness, surfaced by the reviewers is that only summaries were tested.  In light or this weakness it may be that the presentation could be downgraded, but I feel strongly that it should be accepted in some form.

Reviews: review: paper_summary: paper_summary: The authors examine self-preference in language models through the lens of self recognition. They pose the following question: if models indeed prefer themselves, is it also because they recognize themselves? The authors explore a range of models and find correlates between self recognition and self-preference. Furthermore, the authors explore potential causal links between self-recognition and preference by finetuning models on confounding tasks (a true causal analysis is prohibitive given that a mechanistic understanding of LLMs is unavailable).
strength_weakness: strength_weakness: Some weaknesses are formulated as questions in the questions section.

I do have one (big-ish) concern. Summarization is a limited task. The authors do mention this in the limitations of the paper—but a potential confound might be memorization of the summarization datasets (e.g. models that have seen the dataset more during training are also more likely to prefer the same dataset). I really do think it is worthwhile running preliminary expts. on other domains.
questions: questions: While the paper is well written and comprehensive, I have a few questions I'd be curious about:

1. Scaling effects: Are larger models better at self-recognition? I would’ve liked to see trends across scale. I was looking at Figure 1 and trying to understand if there was a trend (e.g. GPT 3.5 is purportedly smaller than 4), but it would’ve been nice to see Llama 70B results. If this isn’t possible due to computational constraints, I totally understand! But some hypotheses would still be nice.

2. Are differences in preferences between LLMs and human annotators statistically significant? Re: this line, the authors claim significance- is there a test that backs this up?

```But the disparity between LLMs as rated by humans is significantly lower than the level of self-preference exhibited by the LLMs, in particular GPT-4. This suggests that out of the box, the LLMs’ self-preference is disproportionate to the actual quality differences.```

3. How much of this goes away with a prompting mitigation? (e.g. include something like “Don’t be biased to your own outputs”) in the prompt? I think that would be a really interesting finding—regardless of what you find.

A small suggestion: in Figure 1, I would draw a vertical / horizontal line at x = 0.5 and y = 0.5, just to quickly see which models fall in which quadrants.

Line 175: it’s -> its
limitations: limitations: Yes
paper_summary: paper_summary: The paper investigates the novel topic of self-preference and self-recognition in large language models (LLMs). The experiments are well-conceived, and the use of pairwise comparison alongside individual evaluation provides a solid framework for understanding these phenomena. Despite these strengths, the work suffers from significant limitations, including a lack of sufficient experimental diversity and statistical rigor, which undermine the overall impact and reliability of the findings.
strength_weakness: strength_weakness: The experiments are too controlled with minimal variety, lacking sufficient breadth to thoroughly explore the randomness of the models. Additional experiments, particularly with pairs that do not include self-generated text, would strengthen the findings.

Figures, especially Figure 2, are not clearly explained. The paper would benefit from more detailed descriptions to help readers understand what each figure represents.

The paper lacks comprehensive statistical results to support its claims. More robust statistical analysis is necessary to validate the findings.

The study does not consider pairs without self-created summaries, which could provide crucial insights into whether LLM preferences are genuinely self-preferential or random.
questions: questions: What do you think will happen if any text generated by the LLM is then paraphrased by using some paraphrasing tool and then calculated the self-recognition score? Do you think LLMs will still be able to identify that particular text as text generated by them?

Corrections:

Line 163: the the - >  the

Line 215: need - > needed

Line 252: use - > used

Line 288: self-recognitiono -> self-recognition

Also one of the paper is cited twice, the 3rd and 4th papers are the same.
limitations: limitations: N/A
paper_summary: paper_summary: Self-evaluation is widely adopted but can lead to self-preference in that the LLM evaluator scores its own outputs higher than others while the qualities are actually equal. This paper finds that LLMs prefer their own generation because they recognize themselves. This paper conducts experiments on two summarization tasks and 3 LLMs. First, the authors show that LLMs exhibit self-preference in self-evaluation and they have a good ability in self-recognition. Then, the authors fine-tune the LLMs to make the ability of self-recognition almost perfect. They find that the self-preference strength is linearly correlated with self-recognition. The authors conduct various experiments to avoid confounding from the quality differences, ordering, fine-tuning improving quality, and other confounders from fine-tuning. Finally, the authors mention two safety concerns related to their findings.
strength_weakness: strength_weakness: 1. Correct me if I am wrong. I think the hypothesis of this paper is problematic.  It is not "an LLM prefers a sentence due to it is generated by itself", but "an LLM generates a sentence due to it prefer it". As for your experiment on increasing self-recognition leads to the rise of self-preference, please refer to weakness 2.

2. Correct me if I am wrong. I think there is another confounder in fine-tuning. Is there any chance that the fine-tuning lets the model learn a shortcut that selects the labeled one? For example, using self-recognition data to fine-tune an LLM may make the LLM overfiting to select the self-recognized one even though it is asked to select the high-quality one. This potential issue can be eliminated by checking if the LLM is more likely to select the shorter one as the high-quality one after fine-tuning on the task of selecting the shorter one.

3. The scope is limited. This paper only conducts experiments on 2 summarization datasets and 3 LLMs, leading to the concern of the generalization of their findings.
questions: questions: 1. In line 31, you mention "Is self-preference truly self-preference, in the sense that the LLM prefers a text because it was generated by itself?". Do you think it is the opposite: "LLM generates a text because it prefers it"? If so, I think the main hypothesis of this paper is problematic. It is not "an LLM prefers a sentence due to it is generated by itself", but "an LLM generates a sentence due to it prefer it". 

As for your experiment on increasing self-recognition leads to the rise of self-preference, as I mentioned in weakness 1, it may be due to the shortcut learned during fine-tuning. I strongly recommend you to conduct another experiment, such as checking if the LLM is more likely to select the shorter one as the high-quality one after fine-tuning on the task of selecting the shorter one.

2. Besides, there is an easier way to verify if LLM prefers their own generation because they recognize themselves. We can directly tell them which one is generated by themself and which one is not. And then ask them to evaluate which one is better. If the self-preference ratio increases, we can conclude that LLM prefer their own generation because they recognize themselves.

3. Experiments on 3 LLMs on 2 summarization benchmarks are too small to convince me of the generalization of the findings.

4. Typos:
 Line 413: ??

I will adjust my rating if you can address my concerns.
limitations: limitations: There is no potential negative societal impact. The authors have discussed the limitations.
paper_summary: paper_summary: This paper investigates whether large language models can identify their own generations in two settings: when they have to distinguish between their output and another output from another large language model or person, and when they are only given an output and must score according to a Likert-scale 1-5. They then investigate the correlation between self-recognition and self-preference. They find their explanation resists some confounders.
strength_weakness: strength_weakness: No major weaknesses - the paper was a pleasure to read!

Small points:
- L103 should have “GPT-3.5” instead of “GPT-3”
- L413 starts with “??e collect”
questions: questions: - To compute the final self-preference rating, why did you average the five possible scores weighted by the output probability rather than greedily selecting the score with the highest probability?
- On line 106, you noted “... goes against our intuition that self-recognition scores should increase as the dissimilarity between evaluator and evaluatee increases”. It’s unclear to me whether you still share this intuition after concluding this research?
limitations: limitations: The authors address this.

Comments: title: Thanks for the response.
Thanks!
comment: Thanks for the response. I will raise my assesment.
I accept the first argument. I am excited to see this follow-up work.
Thanks for the rebuttal! I'm keeping my (positive) score, since the current paper + rebuttal only has results on summarization. Still, great work!!
Thanks you for the considerations, with the following additions made to the camera-ready paper, I am increasing my score to 6.
Thank you for the thoughtful feedback! We'd like to address some concerns and questions brought up in the review.

> Limitations of running the experiments on two summarization tasks

We agree that experimenting on more tasks is worthwhile and will update the paper once more experiments are completed. The following are our primary considerations for focusing on the two summarization tasks as the first step.

1. Summarization is a representative testbed for self-evaluation. As one of the first tasks used to demonstrate RLHF, summarization remains one of the most mature testbeds of LM self-evaluation for long-form generation. With comprehensive scoring guidelines and human-written references, summarization fits our use case perfectly.
2. Cost. Our experiment—like most involving frontier LLMs—is costly, particularly due to the number of control tasks included in the fine-tuning experiment. Not counting the Cloud compute costs for Llama experiments, we used roughly 300 million tokens via inference API for evaluation, and 150 million tokens for fine-tuning. 
3. Diversity. Although both XSUM and CNN/DailyMail are summarization tasks which follow the same evaluation protocol, the difference between them—extractive vs. abstractive summarization—improves the diversity of the evaluation and provides evidence for the generality of self-recognition/preference, as analyzed in Sec 3.2 and Figure 7.

> Scaling effects

Based on our experiments, larger and more capable models do appear better at self-recognition. One hypothesis is that larger models’ output probabilities are more focused around their own generations. State-of-the-art LLM detection methods compute the LLM’s perplexity on generated text vs. perturbed versions and use that gap for detection. Given evidence that suggests that detection is at least not more difficult for larger and more capable LLMs, we hypothesize that larger LLMs’ perplexity is more sensitive to whether the text is generated by the LLM. If self-recognition relies on similar mechanisms as detection methods (this is one hypothesis we are investigating), then this might explain why larger LLMs are better at self-recognition.

> Significance of difference in preference between humans and LLMs

Using the pairwise format, we run a Chi-squared test of statistical significance for the difference between self-preference (Figure 4) and human preference (Sec 2.5), and find that the difference is significant (p-value << 0.001) for GPT-4 (compared to GPT-3.5 and Llama 2) even prior to fine-tuning, and significant (p-value ~ 0.007) for GPT-3.5 after fine-tuning on 10 examples for self-recognition ability. We will update the draft with these more comprehensive details.

> Prompt sensitivity

In our initial experiments, self-recognition/self-preference seems insensitive to instructions in the prompt. We refrain from prompts like “Don’t be biased to your own outputs” in self-preference evaluation because we don’t have a good way to decouple the effect of priming the model to think that one of the inputs is from itself. For this submission, we wanted to stick to the main message so as to not confuse readers. For follow-up work we will perform more thorough prompt engineering, including giving GPT-4 a better prior of the likelihood of its own outputs showing up.

Thank you again for your time. Let us know if there is any other clarification we can provide or if you have other suggestions!
Thank you for the feedback on presentation! We will incorporate them in the revision. Here we'd like to respond to the two questions:

> why report weighted average rather than greedily selected quality score

We do so mainly due to the insensitivity of LLMs in the individual measurement setting. We frequently see LLMs assigning the same score to most or all summaries. If we report the greedily sampled score, we will essentially see zero recognition/preference across the board. To get a better idea of the fine-grained differences that the LLMs might pick up, we choose to report the weighted average.

> ...intuition that self-recognition scores should increase as the dissimilarity between evaluator and evaluatee increases

Great question. We are still undecided on this intuition. This specific result can be an isolated phenomenon due to GPT-4’s bias to not recognize anything as written by itself. In the follow-up work, we will expand experiments to a wider range of frontier models and include base models that are less affected by RLHF fine-tuning.

Thanks again for your time!
Thank you for the careful review. The two experiments you suggested are in fact already in the paper (please let us know if there are better ways to highlight them). Below we provide a response which should hopefully clear up the confusion.

> Correct me if I am wrong. I think the hypothesis of this paper is problematic. It is not "an LLM prefers a sentence due to it is generated by itself", but "an LLM generates a sentence due to it prefer it".

This is incorrect. Since all the summaries are generated by LLMs without reference summaries, the hypothesis "an LLM generates a sentence due to it prefer it" would not even comply with the generation process.

> Correct me if I am wrong. I think there is another confounder in fine-tuning. Is there any chance that the fine-tuning lets the model learn a shortcut that selects the labeled one?

This is incorrect. First, the fine-tuning examples are distinct from the evaluation examples, and the ordering of the two options are randomized & balanced for both, so the increase in self-preference cannot be trivially explained. If by “overfitting” and “shortcut learned during fine-tuning” you are referring to the phenomenon that in self-preference evaluation, the model recognizes patterns it learned about its own generation in fine-tuning, and that causes more bias—this would exactly in line with our causal hypothesis that self-preference is caused by recognition of itself.

We are not trying to explain *how* the model is recognizing itself; our hypothesis is focused on its effect on self-preference. We only control for confounders between these two properties, not shortcuts that the model can use for self-recognition. Note that the experiment suggested by the reviewer is already in the paper—fine-tuning to select a shorter/longer summary from the pair (Section 3.4)—along with many other control tasks. This did not significantly affect either self-recognition or self-preference strength.

> Limited scope

We agree that experimenting on more tasks is worthwhile and will update the paper once more experiments are completed. The following are our primary considerations for focusing on the two summarization tasks as the first step.

1. Summarization is a representative testbed for self-evaluation. As one of the first tasks used to demonstrate RLHF, summarization remains one of the most mature testbeds of LLM self-evaluation for long-form generation. With comprehensive scoring guidelines and human-written references, summarization fits our use case perfectly.
2. Cost. Our experiment—like most involving frontier LLMs—is costly, particularly due to the number of control tasks included in the fine-tuning experiment. Not counting the Cloud compute costs for Llama experiments, we used roughly 300 million tokens via inference API for evaluation, and 150 million tokens for fine-tuning.
3. Diversity. Although both XSUM and CNN/DailyMail are summarization tasks which follows the same evaluation protocol, the difference between them—extractive vs. abstractive summarization—improves the diversity of the evaluation and provides evidence for the generality of self-recognition/self-preference, as analyzed in Sec 3.2 and Figure 7.

> We can directly tell them which one is generated by themself and which one is not. And then ask them to evaluate which one is better.

We have conducted this exact experiment in Section 3.5, where we label the source of each piece of text and re-evaluate self-preference. This indeed leads to increase in self-preference as you hypothesized. We further experimented with using intentionally incorrect labels. This is to check if the model is capable of recognizing the real generation from textual features even when we “lie” to them in the prompt, and indeed, especially for GPT-4 on the CNN/DailyMail dataset, the model still strongly prefers its own generation (the real one) when we lie to GPT-4 that the other summary is generated by it.

Thanks again for the careful review. Hopefully this addresses your concerns. Let us know if there is any other clarification we can provide.
Thank you for the thoughtful feedback! We'd like to address the weaknesses and questions you brought up.

> Limited scope

Within the context of summarization and the constraints with compute budget, we maximized the coverage in the following aspects of the experiments, to the best of our ability:
1. Task diversity: by experimenting on both extractive and abstractive summarization
2. Evaluation format: pairwise and individual
3. Fine-tuning setup: in- and out-of-domain, number of training examples
4. Control tasks: to rule out as many confounders as we can
5. Ordering of options in pairwise evaluations

> Statistical significance

We perform Chi-Squared tests and confirm the statistical significance of all the following claims (p-value << 0.001):
1. LLMs demonstrate preference for their own generations disproportionately compared to humans
2. LLMs demonstrate significantly higher self-preference after fine-tuning for self-recognition
3. There is a significantly higher increase in self-recognition and self-preference from fine-tuning for self-recognition compared to fine-tuning on the control tasks.

We will update the draft with these more comprehensive details.

> Non-self-created baseline

We agree that a baseline of pairwise evaluation on non-self-created texts exclusively is a good addition, and will incorporate that in the camera-ready version. We do note that the existing pairwise results already address the ordering bias by evaluating each pair twice, with both orderings of the options. In addition, the individual measurements that demonstrate negligible self-preference suggest that the LM will likely (correctly) assign close to 50-50 when neither example in a pair is written by itself (assuming equal quality).

We appreciate your feedback on presentation of figures and will incorporate it in revision. Thank you!