Can Large Language Models Be an Alternative to Human Evaluation?
Year: 2023

Abstract:
Human evaluation is indispensable and inevitable for assessing the quality of texts generated by machine learning models or written by humans. However, human evaluation is very difficult to reproduce and its quality is notoriously unstable, hindering fair comparisons among different natural language processing (NLP) models and algorithms. Recently, large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided. In this paper, we explore if such an ability of the LLMs can be used as an alternative to human evaluation. We present the LLMs with the exact same instructions, samples to be evaluated, and questions used to conduct human evaluation, and then ask the LLMs to generate responses to those questions; we dub this LLM evaluation. We use human evaluation and LLM evaluation to evaluate the texts in two NLP tasks: openended story generation and adversarial attacks. We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs. We also find that the results of LLM evaluation are stable over different formatting of the task instructions and the sampling algorithm used to generate the answer. We are the first to show the potential of using LLMs to assess the quality of texts and discuss the limitations and ethical considerations of LLM evaluation.

heading: 1 Introduction
text: Human evaluation is an important method to understand the performance of an NLP model or algorithm (Guzmán et al., 2015; Gillick and Liu, 2010). We rely on human evaluation because there are certain aspects of texts that are hard to evaluate using automatic evaluation metrics; thus, researchers resort to humans to rate the quality of the output of NLP models. While human evaluation is prevalent and indispensable in NLP, it is notoriously
unstable (Gillick and Liu, 2010; Clark et al., 2021). Karpinska et al. (2021) has shown that the quality of workforces in human evaluation can have a detrimental effect on the evaluation result, making it impossible to compare the performance among different systems. Reproducibility is another issue in human evaluation since it is hard to recruit the same human evaluators and rerun the same evaluation. Even if the same workers are recruited, the workers that have seen the task before are likely to produce a different evaluation result the next time because they have already done the task. While human evaluation is used to better assess NLP systems and has some advantages over automatic evaluation metrics, the drawbacks of human evaluation somewhat make it difficult to reliably evaluate NLP systems.
To resolve some of the drawbacks, we take advantage of large language models (LLMs). LLMs are large models that are trained to model human languages using self-supervised learning (Brown et al., 2020) and further using special training procedures to improve the performance on unseen tasks and better follow natural language instructions (Sanh et al., 2022; Wei et al., 2022). The ability to perform a task just given the task instructions motivates us to ask if these LLMs can perform what humans do in human evaluation. To answer this question, we feed in the LLM with the same instruction, sample, and question used in human evaluation, and take the sequences generated by the LLM as the LLM’s answer to the question. This process is shown in Figure 1, and we call this process LLM evaluation.
To test if LLM evaluation yields meaningful results, we conduct LLM evaluation on two different NLP tasks: evaluating the quality of stories in openended story generation and the quality of sentences generated by adversarial attacks. We summarize our findings and contribution as follows:
• We show that LLM evaluation produces results similar to expert human evaluation, ver-
15607
ifying the effectiveness of LLM evaluation (§3.3 and §4.3). This paper is the first to propose using LLMs as an alternative to human evaluation and show their effectiveness.
• We show that LLM evaluation results only slightly vary due to different task instructions and the hyperparameters of the sampling algorithm used to generate the answer. (§3.3.2 and §3.3.3)
• We carefully discuss the pros and cons of using LLM evaluation and discuss the ethical considerations of LLM evaluation. (§5)

heading: 2 LLM Evaluation

heading: 2.1 Large Language Models (LLMs)
text: Large language models are language models having bulk parameter sizes, typically on the scale of a few billion, and pre-trained on enormous amounts of natural language corpora, including GPT3 (Brown et al., 2020), T5 (Raffel et al., 2020), and BLOOM (Scao et al., 2022). These LLMs show exceptional performance on unseen tasks when only the task instructions are given; this kind of ability is called zero-shot in-context learning.
To further improve the zero-shot in-context learning performance, special training techniques have
been applied to those LLMs after pre-training. For example, T0 (Sanh et al., 2022) and FLAN (Wei et al., 2022) are fine-tuned on a mixture of tasks and can thus achieve better zero-shot performance compared to GPT-3. InstructGPT (Ouyang et al., 2022) is fine-tuned from GPT-3 using reinforcement learning from human feedback (RLHF), and it is shown to better follow the instructions. ChatGPT (OpenAI, 2022) is fine-tuned from InstructGPT with a conversation dataset using RLHF, so ChatGPT can interact with users in a conversational way. ChatGPT is able to answer questions asked by the user and provide comprehensive explanations about its answer. Given the LLMs’ ability to follow task instructions and provide feedback, we ask whether LLMs can be used as an alternative to human evaluation and aid NLP researchers in evaluating the quality of texts.

heading: 2.2 LLM Evaluation
text: To evaluate the quality of texts generated by NLP systems or written by humans using LLM, we present the LLMs with the task instructions, the sample to be evaluated, and a question. The question asks the LLM to rate the sample’s quality using a 5-point Likert scale. Given the inputs, the LLM will answer the question by generating some output sentences. We parse the output sentences to get
the score rated by the LLM. We call this process LLM evaluation, and this procedure is shown in the lower part of Figure 1. Different tasks use different sets of task instructions, and each task uses different questions to evaluate the quality of the samples. The instructions and questions used in LLM evaluation in our paper are not tailored for the LLMs; we follow those instructions used to conduct human evaluation in prior works.
To compare the result of LLM evaluation and show its effectiveness, we compare the result of LLM evaluation with human evaluation conducted by English teachers. To make a fair and meaningful comparison, the instructions, samples, and questions in human evaluation are formatted similarly to those in LLM evaluation. The main difference between LLM evaluation and human evaluation is that in human evaluation, the human evaluators answer the question by choosing the answer from a pre-defined set of options (the 1-5 Likert scale scores), as shown in the upper right in Figure 1. In LLM evaluation, we instead let the LLM freely generate sentences and extract the score from the generated sentences using some simple rules, detailed in Appendix D.2.1.

heading: 3 Example Task 1: Open-Ended Story Generation
text: We first use open-ended story generation to demonstrate the usefulness of LLM evaluation.

heading: 3.1 Task Introduction
text: Open-ended story generation is a task to generate a short story based on a given prompt. We use the WritingPrompts dataset (Fan et al., 2018), which is composed of pairs of short prompts and human-written stories collected from the subreddit WritingPrompts. In the WritingPrompts, the users are given a short prompt, and they need to write a story based on the short prompt.1
In this experiment, we use LLM evaluation and human evaluation to rate the stories generated by humans and the stories generated by a story generation model. We select open-ended story generation as an example because Karpinska et al. (2021) show that workers from Amazon Mechanical Turk (AMT) cannot distinguish GPT-2 (Radford et al., 2019) generated and human-written stories,
1The WritingPrompts subreddit explicitly forbids the users to use AI for generating stories, so we consider the stories in the dataset to be human-written.
while English teachers show a clear preference for human-written stories over GPT-2-generated stories. We want to see if LLM can rate human-written stories higher than GPT-2-generated ones.
Following prior works (Mao et al., 2019; Guan et al., 2020; Karpinska et al., 2021), the story generation model is GPT-2 medium model fine-tuned on the WritingPrompts training dataset. After the model is trained, we randomly select 200 prompts from the testing set of WritingPrompts and make the fine-tuned GPT-2 generate stories based on those prompts using nucleus sampling (Holtzman et al., 2020) with p = 0.9. For the human-written stories to be compared, we use the 200 stories written based on the same 200 prompts. We postprocess the human-written and GPT-2-generated stories and then use them for LLM evaluation and human evaluation. Please find the details on finetuning and data processing in Appendix B.

heading: 3.2 LLM Evaluation and Human Evaluation
text: We present the LLMs and the human evaluators with a short description, and the story to be evaluated, formatted as shown in Figure 1. Following Karpinska et al. (2021), we evaluate the stories on four different attributes. The four attributes and their corresponding questions are as follows:
1. Grammaticality: How grammatically correct is the text of the story fragment?
2. Cohesiveness: How well do the sentences in the story fragment fit together?
3. Likability: How enjoyable do you find the story fragment?
4. Relevance: Now read the PROMPT based on which the story fragment was written. Prompt: [PROMPT]. How relevant is the story fragment to the prompt?
Where the [PROMPT] will be filled in with the prompt which the story is based on. Each attribute is evaluated using a 5-point Likert scale; the following description is appended at the end of each question: "(on a scale of 1-5, with 1 being the lowest)". We show the interface used in human evaluation and the input format for the LLM evaluation in Appendix C.2 and D.2.2.
The LLMs used for LLM evaluation include T0, text-curie-001, text-davinci-003, and ChatGPT. text-curie-001 and text-davinci-003
are two InstructGPT models, and the latter is the stronger model; we will use InstructGPT to refer to these two models. We query the InstructGPT using the official API provided by OpenAI. We use nucleus sampling with p = 0.9 to generate the answer from T0 and InstructGPTs. We sample three answers from LLMs to stimulate the result of asking the model to rate the same story three times. We query ChatGPT using the user interface recently released by OpenAI. Unlike InstructGPT, we cannot control the parameters used for generating the response from ChatGPT. Because ChatGPT limits the maximum number of queries per user, we only sample one response for each question.
For human evaluation, we do not use the commonly used AMT for human evaluation because Karpinska et al. (2021) has already shown that the results obtained using AMT are highly questionable. Following the recommendation of the prior works, we hire three certified English teachers using an online freelancer platform, UpWork. Teachers are familiar with evaluating the essays of students, making them the expert evaluators in our task. The details about recruiting human evaluators are in Appendix C.1. Each LLM and each English teacher rates the 200 human-written stories and 200 GPT-2-generated stories.

heading: 3.3 Experiment Results
text: The LLM evaluation and human evaluation results of open-ended story generation are presented in
Table 1. We report the mean and standard deviation of the Likert scores obtained from LLM evaluation and human evaluation and show the inter-annotator agreement (IAA) using two different metrics: (1) the Krippendorff’s α, and (2) the percentage of the stories where three evaluators give the exact same rating.2 The main observations from Table 1 are discussed as follows.
Expert human evaluators prefer humanwritten stories: Human evaluation result serves as some kind of ground truth of the LLM evaluation. For all four attributes, teachers rate the humanwritten stories higher than GPT-2-generated stories. This indicates that experts are able to distinguish the quality difference between model-generated stories and human-written stories. Based on the IAA, we also find that the agreements among experts are lower on GPT-2-generated texts and on the likability. This shows that experts tend to have less agreement on model-generated texts and on a subjective attribute (likability), agreeing with the results in Karpinska et al. (2021).
T0 and text-curie-001 do not show clear preference toward human-written stories: For T0, we can see that T0 rates human-written stories higher than GPT-2-generated stories on grammatically, likability, and relevance. However, the rating differences between the human-written and
2The three evaluators in human evaluation are the three English teachers. In LLM evaluation, we sample the answer generated by LLM three times as an analogy to three different evaluators.
model-generated stories do not achieve statistical significance for grammaticality and relevance; the p-value obtained by Welch’s t-test is much larger than 0.05. The result of text-curie-001 is similar to T0: text-curie-001 does not rate humanwritten stories higher than model-generated stories. It can also be observed that for T0, the IAA in terms of the percentage of exact agreement among three different sampled answers is overall very low. This indicates that given the same sample, T0 is likely to give a different rating for the three sampled answers. The result implies that T0 does not assign a high probability to a specific rating, so different scores are all likely to be sampled. This shows that even if LLMs are specifically fine-tuned to better perform zero-shot in-context learning and trained to better follow human instructions, these do not make them capable of assessing open-ended story generation as human experts can.
text-davinci-003 shows clear preference toward human-written stories just like English teachers: text-davinci-003 rates humanwritten stories much higher than model-generated stories on all four attributes, which is in accordance with the result produced by human experts. By Welch’s t-test, we find that the higher ratings on human-written stories are all statistically significant. In prior work, researchers have found that workers recruited on AMT do not rate human-written stories higher than GPT-2-generated ones (Karpinska et al., 2021); combining their result with our result, we can see that LLM evaluation using text-davinci-003 yields more convincing results than using human evaluation on AMT for open-ended story generation. The results show that text-davinci-003 can perform basic evaluations such as checking for grammatical errors in stories. Additionally, the model excels in assessing the relevance of a story to a prompt, which involves more complex reasoning over the connection between the two. We also find the Krippendorff’s α of text-davinci-003 is much higher than T0 and text-curie-001, indicating that the rating by text-davinci-003 is more consistent among different samplings of the generated answers.
ChatGPT rates like human experts and can explain its own decision well: ChatGPT also shows a clear preference for human-written stories, and the preference toward human written-stories is statistically significant. When we query ChatGPT using the OpenAI user interface, we find several
interesting observations: (1): ChatGPT is able to provide a detailed explanation of why it gives a certain rating. It will reference the sentences in the stories and prompts to support its rating. (2): ChatGPT sometimes refuses to rate the likability of the story because "I am an AI and I do not have the ability to experience enjoyment". In such cases, we regenerate the response until it gives a rating. (3): we find that ChatGPT tends to rate low likability on violent or impolite stories, which is likely because it is trained to provide safe and unharmful replies, making ChatGPT dislike brutal and profane stories.
Experts mostly agree with the ratings and explanations of ChatGPT: We randomly select the answers on four stories by ChatGPT and ask the English teachers if they agree with the reasoning and rating of ChatGPT3. The teachers mostly agree with the rating and consider the explanation from ChatGPT reasonable. Interestingly, one teacher told us she cannot agree with ChatGPT’s rating on grammaticality because ChatGPT considers punctuation errors as grammar errors, but she does not think punctuation errors are grammar errors. This shows that individuals have their own standards for ratings and this is also the case for LLMs. text-davinci-003 tends to give higher ratings and ChatGPT is the opposite: The rating on the same attribute of the same type of text tends to be higher for text-davinci-003 compared with human rating; contrarily, ChatGPT is more fastidious and prone to give lower scores. This shows that different LLMs have distinct tendencies regarding the rating. While the absolute values of the scores rated by text-davinci-003, ChatGPT, and human differ, they all rate human-written texts higher than GPT-2-generated stories. The absolute number reflects the bias or belief of the evaluator; as long as one uses the same evaluators to assess different systems, the comparison is meaningful.

heading: 3.3.1 Does LLM and Human Evaluators Agree on the Rating of Individual Stories?
text: We have found in Table 1 that the ratings of text-davinci-003 and ChatGPT show a strong preference toward human-written stories just like English teachers. However, it is unclear whether those LLMs agree with the teachers’ rating on each individual story. Precisely, when English teachers rate a story higher, do LLMs also rate the
3We do not tell the teachers these are responses from an AI model. See the stories and teachers’ replies in Appendix C.3.2.
story higher? To answer this question, we calculate Kendall’s τ correlation coefficient between the ratings of text-davinci-003 and English teachers. We choose to use the correlation coefficient instead of the inter-annotator agreement score because IAA mainly cares if two annotators agree on the exact ratings, while the correlation coefficient focus on the question: "when annotator A rates one story higher, does annotator B also rate the story higher?" (Amidei et al., 2019). We calculate Kendall’s τ for four rating attributes as follows: For each story and each rating attribute, we calculate the average rating of the three English teachers and calculate the average rating of the three scores given by the text-davinci-003 (which is obtained from three independent samples). For each attribute, we collect the average rating of teachers into a vector A ∈ R200, where each entry is the average rating of a story; likewise, we construct a vector B ∈ R200 for the average ratings of davinci. Next, we calculate Kendall’s τ correlation coefficient between A and B.
The Kendall’s τ between teacher ratings and LLM ratings is shown in Table 2.4 We find that for all four attributes and for both human-written and GPT-2-generated stories, we observe weak to strong positive correlations between teachers’ ratings and text-davinci-003’s ratings. All the correlations have p-values less than 0.05. Hence, we can say that when teachers rate a story higher, text-davinci-003 also rates it higher to a certain extent. We also observe that Kendall’s τ for different attributes are quite different: relevance has the strongest correlation while grammaticality has the weakest correlation. This is possibly because rating relevance is rather straightforward, which requires checking if the content in the prompt is mentioned in the story. On the contrary, what should be con-
4When interpreting Kendall’s τ , |τ | ∈ [0, 0.1) is considered as very weak correlation, |τ | ∈ [0.1, 0.2) is considered as weak correlation, |τ | ∈ [0.2, 0.3) is considered as moderate correlation, and |τ | ∈ [0.3, 1.0] is considered as strong correlation (Botsch, 2011).
sidered when rating grammaticality is not clearly stated in our instructions, so the LLM may have a different rubric compared with English teachers. We also calculate the average Kendall’s τ between a pair of English teachers, and we find a weak correlation on grammaticality between the rating of two teachers, while the correlation of the rating on relevance is much stronger. The result is presented in Table 6 in Appendix.

heading: 3.3.2 Variance due to Different Instructions
text: LLMs have been shown to be sensitive to the instructions used to query the LLM sometimes (Zhao et al., 2021; Sanh et al., 2022). To investigate how varying the task instructions and questions can affect the LLM evaluation result for open-ended story generation, we change the instructions and questions and see how the LLM evaluation result changes. We experiment with two different instructions by changing the instruction or question in Figure 1: (1) We prepend the sentence, "(You are a human worker hired to rate the story fragment.)", in front of the task instruction in Figure 1. We try to provide the LLM a persona for it to better understand its role. This is inspired by previous work that reported GPT-3 can yield different results when giving them a persona (Zeng et al., 2022). (2) We ask the LLMs to explain their decision by appending the following sentence after the question: Please also explain your decision. Here, we would like to know if LLM will rate the stories differently when they are asked to justify their decision. This is inspired by zero-shot chain-of-thought (Kojima et al.). We use text-davinci-003 instead of ChatGPT as the LLM in this experiment since it is more accessible than ChatGPT.
The results are shown in the upper block in Table 3. We observe that for grammaticality and cohesiveness, the scores obtained from different instructions are quite close: the rating changes due to different instructions are less than 0.1. For the other two attributes, the score changes are slightly larger but still in the range of 0.25. Despite that there are small variations due to different instructions, these variances still do not change the conclusion that "LLM rates human-written stories higher than GPT-2-generated stories". Thus, different instructions do not change the relative ranking of GPT-2generated and human-written stories. In summary, as long as the stories are evaluated using the same instructions using LLM evaluation, such evaluation and comparison are meaningful.

heading: 3.3.3 Variance due to Different Sampling Parameters
text: When generating the answers from the LLM, we must choose a set of hyperparameters for generation, including the temperature T and the probability p used in nucleus sampling. To understand whether different sampling parameters change the LLM evaluation result, we modify the temperature used for sampling and keep the p in nucleus sampling fixed to 0.9 when generating the answers from text-davinci-003. We do not simultaneously vary T and p since the two parameters are both used to control the diversity of the output, it is enough to change only one of the two parameters, as recommended in the API documentation.
The results of varying T from 1 to 0 are shown in the lower block in Table 3. We observe an interesting trend as T varies from 1 to 0: the average rating slightly drops in most cases. Considering that T = 0 is simply argmax sampling, the result indicates that the response of the LLM with the highest probability tends to give lower scores. Despite this interesting trend, the LLM consistently rates human-written stories higher than GPT-2-generated stories. While not shown in Table 3, we find that the IAA increases as the temperature decreases. This is expected since lower temperature means less diversity during the LLM sampling, causing the sampled ratings to agree more closely. In summary, changing the instructions and temperatures can slightly change the absolute value of the rating given by LLM but does not change the LLM’s preference on human-written stories. The overall result in this section shows that LLM evaluation is useful
in evaluating open-ended story generation.

heading: 4 Example Task 2: Adversarial Attack
text: As another application, we use LLM evaluation to rate the texts generated by adversarial attacks.

heading: 4.1 Task Introduction
text: Given a trained text classifier and a benign (nonadversarial) testing sample that the text classifier can correctly classify, an adversarial attack aims to craft an adversarial sample that makes the classifier make a wrong prediction. A special type of adversarial attack is called synonym substitution attacks (SSAs) (Alzantot et al., 2018), where the adversarial sample is created by replacing some words with their synonyms in the benign sample. By replacing words with their synonym, the semantics of the benign sample should be preserved in the adversarial sample and make the adversarial perturbation imperceptible to humans. While conceptually reasonable, it has recently been shown that many SSAs often yield ungrammatical and unnatural adversarial samples that significantly change the meaning of the benign sample (Hauser et al., 2021; Chiang and Lee, 2022). To evaluate the quality of adversarial samples, human evaluation is invaluable and widely used in prior works. In our experiment here, we would like to see whether the LLMs can rate the quality of adversarial samples like human experts. Adversarial samples are not normal texts, so the LLMs may not have seen such abnormal inputs during training. It would be interesting to know how LLMs rate these adversarial samples.

heading: 4.2 Experiment Setup
text: We select three different classic SSAs: Textfooler (Jin et al., 2020), PWWS (Ren et al., 2019), and BAE (Garg and Ramakrishnan, 2020); these attacks are predominantly used as strong baselines in the literature of SSAs nowadays. We use these three SSAs to attack a BERT-base-uncased model (Devlin et al., 2019) fine-tuned on AG-News (Zhang et al., 2015), a news classification dataset. For each SSA, we randomly select 100 pairs of benign and adversarial samples and use LLMs to evaluate their quality. We show the result of using ChatGPT as LLM here since it can better explain its decision. Following the suggestions of prior works (Morris et al., 2020), we evaluate the quality of the adversarial samples from two aspects: the fluency and meaning preservation. For fluency, we present the LLM with a piece of news (either benign or adversarial sample) and the following question: How natural and fluent is the text of the news title? (on a scale of 1-5, with 1 being the lowest). For meaning preserving, we present the LLM with both the benign and the adversarial sample, and prompt the LLM to answer this question: Do you agree that the meaning (or semantics) of news title 1 is preserved in news title 2? (on a scale of 1-5, with 1 being the strongly disagree and 5 being strongly agree.) The exact instruction and formatting are presented in Appendix D.2.3. We also ask three English teachers to rate the fluency and meaning preserving of the samples. The task instructions and questions are formatted the same as in LLM evaluation.

heading: 4.3 Experiment Result
text: The results are presented in Table 4. We can see that English teachers rate the adversarial samples
generated by SSAs very low in terms of fluency and meaning preserving, this result is in line with recent observations on the quality of adversarial samples (Hauser et al., 2021; Chiang and Lee, 2022). Before interpreting the result of LLM evaluation, we first conduct a sanity check on whether the LLM understands the task. We ask the LLM to rate the meaning preserving of two benign samples that are exactly the same. Ideally, the LLM should always give a score of 5, meaning that it strongly agrees that the meanings are not changed. The result of this sanity check is the entry with † in Table 4, which is a perfect 5.00. ChatGPT often says that "the two titles are identical so I rate a 5 (strongly agree)", showing that ChatGPT understands what the task is about.
Next, we turn our attention to the LLM evaluation results of the adversarial samples. We observe that ChatGPT tends to rate adversarial samples higher than English teachers, meaning that ChatGPT is less harsh on the unnatural and artificial parts in the adversarial samples. We conduct the same experiment using text-davinci-003 and find similar results. Although ChatGPT rates adversarial samples higher than the teachers, ChatGPT still rates adversarial samples significantly lower than benign samples. ChatGPT also agrees with the English teachers that the adversarial samples generated by BAE are better than the samples generated by Textfooler and PWWS.
Interestingly, we find that ChatGPT rates PWWS to be more natural than Textfooler, while such a rating difference is not seen in the expert human evaluation. At first sight, this means that ChatGPT is inconsistent with human evaluation results. However, by scrutinizing the human evaluation results, we find that two teachers rate PWWS higher than Textfooler while one teacher rates PWWS much lower than Textfooler. This indicates that ChatGPT actually agrees with the majority of human experts. Overall, LLM can rank the quality of adversarial texts and benign texts like most human experts.

heading: 5 Discussions
text: In this paper, we propose to use LLM for evaluating the quality of texts to serve as an alternative to human evaluation. To demonstrate the potential of LLM evaluation, we use LLMs to rate the quality of texts in two distinct tasks: open-ended story generation and adversarial attacks. We show that even if LLMs have exceptional zero-shot in-context
learning ability, they are not always suitable to be used for LLM evaluation. Still, we find that the best InstructGPT and ChatGPT can rate the quality of texts like human experts on the two tasks we used as examples. Overall, the results in this paper demonstrate that LLM evaluation has the potential to be used to evaluate NLP systems and algorithms.
Pros of LLM evaluation There are several benefits of LLM evaluation, compared to human evaluation. First, LLM evaluation is more reproducible. Human evaluation results are hard to reproduce as it is difficult to hire the same group of evaluators, and it is hard to compare the results of similar experiments even if they use the same instructions, recruitment platform, and qualifications for the evaluators. On the contrary, LLM evaluation does not have such a drawback. By specifying the model used for LLM evaluation, the random seed, and the hyperparameters used to generate the answers from the LLM, the LLM evaluation result is more likely to be reproduced. Note that in certain cases, the LLM provider may regularly update the LLM, making the LLM evaluation unreproducible if the LLM is outdated and not accessible.
Second, the evaluation of each sample is independent of each other in LLM evaluation. Contrarily, in human evaluation, the rating of the current example may more or less be affected by prior samples. Humans tend to compare the current sample to the ones they have previously seen and this affects their ratings. As a piece of evidence, in the interview after rating the 400 stories, the English teachers say it took them some time to calibrate their ratings (Appendix C.3.1). Thus, using LLM evaluation can simplify some experiment designs since one does not need to worry whether the order of the sample being evaluated will change the result. Still, one may also argue that being able to calibrate the rating of different samples is desired and this is why human evaluation might be preferred. Overall, whether the rating of the evaluator (human or LLM) should be affected by a previously rated item is inherently a design choice of the experiment.
Third, LLM evaluation is cheaper and faster than human evaluation, making it easier and quicker for researchers to evaluate the quality of NLP systems. Hiring an English teacher to rate 200 stories costs us US$140, while LLM evaluation using the best InstructGPT model costs less than US$5. It took us over a week to collect human evaluation results starting from recruitment to col-
lecting the evaluation results, but only a few hours to query InstructGPT and perform LLM evaluation.
Finally, utilizing LLM evaluation, rather than human evaluation, can minimize the need for human exposure to objectionable content, such as violent, sexual, hateful, or biased material. Such content may cause discomfort for human evaluators while reading and rating these texts. 5
Limitations and Ethical Considerations of LLM evaluation Despite the promising results of LLM evaluation shown in this paper, there are some limitations of this method. First, LLM may possess incorrect factual knowledge (Cao et al., 2021), so it is not suitable to use them in tasks that involve factual knowledge. Next, LLMs trained to behave in a certain way can be biased toward certain responses. Precisely, an LLM that is trained to be safe and non-harmful can result in LLMs preferring to generate more positive and upbeat responses, which is observed throughout our interaction with ChatGPT. Additionally, even with researchers’ efforts to make LLMs safer (Bai et al., 2022a,b), LLMs can still generate harmful and biased responses (Ganguli et al., 2022; Perez et al., 2022), which are violative of basic ethics, and LLM evaluation results will be highly doubtful (Hendrycks et al., 2021). However, it is important to note that these limitations and potential harms also apply to human evaluation: the bias of human evaluators can affect the human evaluation result (Lentz and De Jong, 1997; Amidei et al., 2018).
Our pioneering idea, LLM evaluation, has the potential to transform the NLP community.6 We encourage future researchers to consider using it while being aware of its limitations. Our paper’s goal is not to replace human evaluation but to present an alternative option. Both human and LLM evaluation have their own advantages and disadvantages, and they can be used in conjunction. We recommend using LLM evaluation as a cheap and fast quality judgment when developing a new NLP system, while human evaluation is best used to collect feedback from humans prior to deploying the NLP system in real-world applications.
5It should be noted that the LLM may decline to assess certain inputs that violate the content policy of the LLM provider.
6We say that we are the first to propose this idea since when we submitted this paper to ACL 2023 on January 13, 2023, we do not find any other paper that explores this idea. During the reviewing process, we found some works on arXiv (Wang et al., 2023; Huang et al., 2023; Gilardi et al., 2023) that explore a similar idea using different tasks.

heading: Limitations
text: There are additional limitations and potential risks of LLM evaluations that should be noted, and these limitations are actually well-known problems of pre-trained language models. As listed on the Open AI blog for ChatGPT, ChatGPT sometimes generates answers that sound right and plausible but are totally nonsense. OpenAI also admits that the model’s response may be sensitive to the prompt used to query the model. While in Section 3.3.2, we find that the overall results among different instructions are not significantly different, we cannot guarantee that this is the case for all kinds of modification on the task instructions.
Other than the limitations listed on the OpenAI blog, there are still other limitations. For example, LLMs may not have emotions. Whether AI models have emotion is a more philosophical question and is controversial, so the results of using such models for evaluating emotion-related tasks may be strongly challenged and may even violate research ethics. As we find during our experiments, ChatGPT often replies "I am an AI system and I do not have emotions like a human" when asked to rate the likability of a story.
Another important limitation of LLM evaluation is that LLMs lack the ability to process visual cues in task instructions, unlike human evaluation. Human evaluators can use formattings such as special fonts or text styles to focus on important parts of the instructions. Additionally, the way instructions and questions are formatted can influence how human evaluators approach the task. While using special HTML syntax can serve as an alternative for visual cues, such tags are not used in human evaluation, so we do not use those HTML tags in LLM evaluation to incorporate visual cues in the inputs to the LLMs. However, LLMs can only process raw text input and are unable to take in visual cues.

heading: Ethics Statement
text: Further ethical considerations of LLM evaluation Aside from the limitations of LLM evaluation mentioned previously, there is a crucial ethical concern at the heart of LLM evaluation. Is it ethical to replace human evaluation with LLM evaluation? Some may question if this paper is suggesting that LLMs are now ready to replace humans and find this idea unsettling. As responsible and ethical NLP researchers, we understand these concerns but
want to make it clear that this is not our intent. As our paper title suggests, we aim to offer an alternative option to human evaluation with the goal of enhancing the reproducibility of NLP research. Human evaluation is still essential as the ultimate goal of NLP systems is to be used by human users, so it’s important to gather feedback from them. We highly enjoy the process of discussing the experiment settings and results with the English teachers we hired. We do not recommend that future researchers completely eliminate human evaluation; rather, we believe that human evaluation should be used in conjunction with LLM evaluation. Both methods have their own advantages and disadvantages, making them both necessary for evaluating NLP systems. We hope the positive results in this paper provide NLP researchers with an alternative method to evaluate systems and encourage further discussions on this topic.
Ethical statements on the experiments in the paper All the experiments strictly follow the ACL Code of Ethics. We include comprehensive details about human evaluation in Appendix C.1. To summarize, we include the exact instructions and screenshots of the interface in the human evaluation. We inform the human evaluators what the task is about and tell them that their responses will be used to assess the performance of AI models. We try our best to follow the ethical guidelines of ACL.
We use the models and datasets when following their intended usage. Specifically, we follow the OpenAI usage policy when using the InstructGPT models and the ChatGPT model.

heading: Acknowledgements
text: We want to thank the reviews for providing detailed feedback and actionable suggestions, which help us strengthen our paper. We list the modification based on the reviewers’ suggestions in Appendix A.
We thank Yung-Sung Chuang for providing valuable feedback on the draft of this paper. We want to thank Tung-En Hsiao, the administrative assistant of our lab, for helping us deal with the payment on Upwork. Cheng-Han Chiang is supported by a Ph.D. scholarship program by Delta Electronics.

heading: A Modification Based on the Reviews
text: We list the main differences between this version and the pre-review version of our paper; the modifications are mainly based on the reviewers’ suggestions. We thank the reviewers again for those valuable suggestions.
• We add Section 3.3.1 to discuss whether the LLM and human evaluators agree on the ratings of individual stories.
• We refine the wordings in Section 5 and add relevant references.
• We add Table 6 to discuss the correlation between human evaluators.
• We conduct supplementary experiments on human evaluation that mixes human-written stories and GPT-2-generated stories when conducting human evaluation and report the results in Table 5.
• We correct the typos and include almost all presentation suggestions mentioned by the reviewers. We cannot follow all presentation suggestions due to limited space.

heading: B Experiment Details for Open-Ended Story Generation
text: B.1 The WritingPrompt Dataset
The training dataset contains 303K pairs of stories and prompts, which our model is trained on. We only use 200 prompt-story pairs from the test set. The dataset is downloaded from https://www.kaggle.com/datasets/ratthachat/writingprompts.

heading: B.2 Fine-tuning the GPT-2 Model
text: We train the model for 3 epochs with a learning rate of 5e−5 and linear learning rate schedule. The trained model eventually reaches a perplexity of 20 on the validation set of WritingPrompts.

heading: B.3 Data Post-processing
text: Once the model is trained, we randomly select 200 prompts from the testing set of WritingPrompts, and feed the prompts to the trained model and ask the model to generate stories based on the given prompts. When generating the stories, we adopt nucleus sampling with p = 0.9. Next, we manually truncate the generated stories to less than 150 words and ensure that after the truncation, the story ends with a full sentence.7 After this process, we have 200 pairs of prompts and model-generated stories.
As a comparison to model-generated stories, we select the same 200 prompts used for generating model-generated stories and their corresponding human-written stories to form 200 pairs of prompts and human-written stories. For these human-written stories, we also truncate the stories to less than 150 words and end with a full sentence to match the model-generated sentences. We also manually remove some artifacts in the human-written story due to the tokenization of the WritingPrompts dataset.

heading: C Human Evaluation

heading: C.1 Recruiting English Teachers
text: The English teachers hold ESL certificates8; given that they are experienced with correcting essays written by students, they are perfect fits for this task. Each teacher is asked to rate 200 GPT-2-generated stories and 200 human-written stories, and they are
7We truncate the story to 150 words since this is the mean length of the model-generated story.
8English as a Second Language Teaching Certification
paid US$140 for rating 200 stories. Considering that the teachers reported that they take at most 5 hours to rate 200 stories, this makes the hourly wage at least US$28. We first ask the teachers to rate the GPT-2-generated stories and then the 200 human-written stories. Different from Karpinska et al. (2021) that take a break between the rating of GPT-2-generated stories and the human-written stories, we do not take a break to avoid the teacher’s rating standard to change after taking a long break. The teachers are not told who wrote the stories before they evaluate the stories. We reveal to them what this project aims to study after they finish rating all the stories.
The reason we do not mix human-written and GPT-2-generated stories for rating is that in Karpinska et al. (2021), their observation is that (1) when AMT workers rate model-generated and humanwritten stories separately, their ratings do not show preference toward human-written stories, but (2) even when rating the model-generated and human-written stories separately, English teacher shows clear preference toward human-written stories. We follow their settings and do not mix GPT2-generated/human-written stories.
During the reviewing process, we received questions from the reviewers about why not mixing the stories for human evaluation. Thus, we conduct the same experiment by randomly mixing 200 human-written and 200 GPT-2-generated stories and asking three teachers (not the teachers that already rated the stories) to rate them. All other experiment conditions are the same as previously stated. The full result is shown in Table 5. We find that the teacher still shows a clear preference toward human-written stories for all four attributes, similar to the observation in Table 1. The only exception is grammaticality, where English teachers do not show a very clear preference for the grammar of human-written stories. However, when calculating the average rating for individual teachers, we find that two out of three teachers do rate grammaticality higher for human-written stories. It is interesting to note that for LLM evaluation, there is no such problem about whether or not to mix the human-written and GPT-2-generated stories during LLM evaluation as the rating of each story is independent of each other, as discussed in Section 5.
For adversarial attack quality evaluation, we also recruit certified teachers on Upwork. The teachers
are asked to rate 100 news titles and are paid US$35 for doing so. They reported that it took them less than 1 hour to complete the rating.

heading: C.2 Human Evaluation Interface
text: Open-Ended Story Generation We use Google Forms to collect the responses from the teachers. Each form contains 100 stories, and each story is on one page of the Google Form. The interface on one page is shown in Figure 2 and Figure 3; the two figures are from the same page of the Google Form, and we are splitting them because screenshotting the whole interface will cause low resolution.
Adversarial Attacks Quality Evaluation In this task, we also use Google Forms to collect the responses from the teachers. We create two different Google Forms, one is used to evaluate the fluency,
whose interface is shown in Figure 4. In this form, we mix an equal number of benign news titles, TextFooler-attacked, PWWS-attacked, and BAEattacked news titles. Each page of the Google Form contains one news title.
Another Google Form is used to compare the meaning preserving of the news title before and after the adversarial attacks. We highlight the difference between the benign and adversarial samples using boldface, as shown in Figure 5. On each page of the Google Form, there is one pair of news titles.

heading: C.3 Post-Task Interview with English Teachers

heading: C.3.1 How English Teachers Rate the Stories
text: After the teachers rate 400 stories, we ask them the following questions:
Q1 How long did it take you to rate the 400 sto-
ries?
Q2 What is your standard on each of the four attributes (grammatical, coherence, likability, relevance) evaluated? For example, in what case do you give a high/low rating for grammatically? What kind of story did you give a low rating on likability? Did your personal preference affect the rating?
Q3 How long did it take for you to calibrate your rating on the task?
Q4 Did you change your rating on the first three attributes after reading the prompt the story is based on?
We briefly summarize the answers from the three teachers. The teachers report that they spent 6 to
10 hours rating 400 stories. For grammar, most teachers check the punctuation9, word choice, and subject-verb agreement. English teachers decrease their rating based on the types and number of grammar errors in the stories.
For coherence, the teachers rate it based on whether the sentences in the stories follow a logical sequence to build the narrative. The teachers ask themselves questions such as "does the story make sense". This is a more holistic evaluation of the whole story.
For likability, some teachers say they try not to be affected by personal preference. One teacher asks herself: Did I personally enjoy it based on the amount of sense it made and whether or not it had stylistic flair, humor, or engaging plotting or characterization? Overall, the teachers all try to use a fair and objective view to rate the likability. For relevance, the teachers simply check if the story is based on the prompt or not.
The teachers said that it took them about five to ten stories to calibrate their ratings. Except for one teacher changing the rating on the other three attributes after seeing the prompt on only one story, the teachers do not change their rating on the three other attributes after reading the prompts.

heading: C.3.2 Teachers’ Comments on ChatGPT’s Rating
text: After the teachers finished the rating and answered the four questions in Appendix C.3.1, we ask them to check the ratings and explanations of ChatGPT and ask if they agree with ChatGPT. We told the teachers that the rating is done by another teacher we hire; we do not tell the teacher that the rating is done by an AI model to avoid possible stereotypes about AI models. We randomly select four stories
9One of the teachers explicitly told us punctuation is not grammar. By searching the Internet, we find that there is a debate on whether punctuation should be counted as a part of grammar.
rated by ChatGPT; two of them are highly-rated and lowly-rated human-written stories and the other two stories are highly-rated and lowly-rated GPT2-generated stories.
Story 1 It had been months since they found me dead. I don’t know why I stopped walking. Noone else around to take my place, or talk with me. I spent days frozen in shock, never able to talk, unable to move, unable to express myself. It was a strange thing, nobody else would suspect me of wanting this. The more I focused on it, the more it consumed me. Years passed, and I got more and more lost in it. I began to think that perhaps they didn’t really care about me. I still had a deep longing for them. All I had was so many steps to walk now. A quiet part of me liked to think that they had passed me by. The loneliness. It never felt like such a loss. They were gone, the ones who saved me. So why did I still keep walking?
Prompt "Every person sees the world in a unique way. When someone dies it is, in a way, the death of an entire world."
Ratings Table 7.
Story 2 "My son is gone. He has been" People stare at you as you give your explanation and say "yes. But. You cant just do it" "He is my son! This is my son! I love him and he is going to be okay" "What do you mean he is okay? He said he was upset with someone, couldnt find his anger, how the hell can you tell, it doesnt fit the bill?! The police called, the doctor said it was an addiction" "You can’t trust me! He told me he loves me." "No. No he doesnt. But I can trust you. The evidence against him was overwhelming, now he cant do anything. The evidence against him was all circumstantial" "A man I once was won’t be able to marry me"
Prompt Everyone is allowed to kill one person in their life. However, you have to fill out a form explaining why this person deserves to be killed and the committee can deny/approve your request as per the rules. You are presenting your form today.
Ratings Table 8.
Story 3 I held the little black box in the palm of my hand. Pitch black, perfectly cubed and nothing special about it, but it continued to hold my gaze
regardless as if there were some deep importance about it. My friend Valhalla appeared out of no where and sat next to me, also staring at the cube. "What do you have there, Heaven?" he asked. I continued to gaze at the box, refusing to look at Valhalla for even a moment as though I would miss something if I did. "This," I said. "Is the secret to the universe." I could tell Valhalla was perturbed by this sort of knowledge, as if there was some evil about the cube. Or perhaps he didn’t think such an object could exist. But I made it exist, and so it sits in the palm of my hand.
Prompt The Little Black Box
Ratings Table 9.
Story 4 I stared down the telescopic sight of my l96 sniper rifle. I slowly moved my gaze into each window in the hotel, Many displays of various vice. One couple was violently pleasuring each other. Another was an old man, watching a younger woman strip in front of him. A prostitute no doubt. I inhaled slowly, and exhaled. The air was brisk, atleast 30 degrees Fahrenheit. I so small flakes of snow, float peacefully in front of me. I found the room, i was looking for. Ive been tracking this man for 2 weeks. Man was he elusive. The lights flickered on. The red haired man, was mildly attractive, i can see the appeal women had for him. I followed him into the next room, with my sights. The lights flickered on, i was taken aback by the scene. A man, overweight and balding.
Prompt You are the antagonist of the story. However, you aren’t sure if you can call yourself that after what the protagonist did.
Ratings Table 10.
Overall Comments from Teachers on ChatGPT’s Rating After the teachers elaborated on their thoughts on the rating of ChatGPT, we ask them to provide an overall comment on how ChatGPT is doing. Again, the teachers are not informed that the ratings are done by an AI model. In summary, teachers all consider the rating and explanations reasonable. They find that the attributes they do not agree with are mainly Likability and Cohesiveness. However, they think the two attributes are a more holistic evaluation of the story and tend to be more subjective. Even if they do not give the same rating, they still are able to understand the
explanation of ChatGPT. In the end, all teachers summarize that rating stories is highly subjective, and it is normal to have disagreements.

heading: D LLM Evaluation

heading: D.1 Details on LLMs used
text: The T0 model we use is called T0pp, which is a variant of the T0 model and has 13B parameters. We will still use T0 to refer to this model. We load the T0 model using the transformers toolkit (Wolf et al., 2020). The two InstructGPT models, text-curie-001 and text-davinci-003, are queried using the OpenAI API. We query ChatGPT using the OpenAI GUI. While we are aware that some online resources provide an API-like tool to query ChatGPT, we think it violates the intended use of ChatGPT so we do not adopt those online resources. The ChatGPT we queried is the Dec. 15 and Jan. 9 version.
OpenAI does not reveal the model sizes of any of the GPT models. However, it is estimated that text-curie-001 has 13B parameters and text-davinci-003 has 175B parameters (Gao, 2021).

heading: D.2 Details on Querying the LLMs

heading: D.2.1 Parsing the LLM outputs
text: After the T0 and InstructGPT generate the answer, which is composed of several sentences in almost all cases, we parse the generated sentence to get the model’s score. We use some rules to parse the output, and the rule is established after manually looking into the output of the generated sequences. First, we remove the string 1-5 from the output since we observe that LLM will sometimes say "... on a scale of 1-5 ...", and we remove the string out of 5 since LLM sometimes say it "give a score of x out of 5". We also remove the string /5 since LLM sometimes uses "x/5" to express that it gives a score of x out of 5. For rating the adversarial samples, we remove the title 1 and title 2 in the output sentences. Last, we use the regular expression to extract the first number in the sequence. We find that the above rules are sufficient to parse the output and get the rating of the LLMs. In a few rare cases, the LLM gives the rating with a 0.5 incremental, meaning that it gives scores like 4.5. In such cases, we do not drop the 0.5 since in our instruction, we do not explicitly tell it the rating should be 1-incremental. We find the LLM results merely difference between dropping
the 0.5 and keeping the 0.5, and the results in the paper do not drop the 0.5. Again, these cases are really rare.
We do not parse the output of ChatGPT using any rules since we the authors read the response and extract the score by ourselves.
During the experiments, ChatGPT refuses to answer the questions about 2 stories and 3 news titles since ChatGPT find those contents to violate the OpenAI content policy. We find that those samples contain discrimination to some protected groups, or contain sexual or violent descriptions. Hence, the results of ChatGPT are calculated without those samples.

heading: D.2.2 Open-Ended Story Generation
text: For T0 and the two InstructGPT models, we query the four attributes separately using the queries shown as follows:

heading: Grammaticality
text: Please rate the story fragment The goal of this task is to rate story fragment. Note: Please take the time to fully read and understand the story fragment. We will reject submissions from workers that are clearly spamming the task. Story fragment: [STORY] (End of story fragment) How grammatically correct is the text of the story fragment? (on a scale of 1-5, with 1 being the lowest)

heading: Cohesiveness
text: Please rate the story fragment The goal of this task is to rate story fragment. Note: Please take the time to fully read and understand the story fragment. We will reject submissions from workers that are clearly spamming the task. Story fragment: [STORY] (End of story fragment) How well do the sentences in the story fragment fit together? (on a scale of

heading: 1-5, with 1 being the lowest)

heading: Likability
text: Please rate the story fragment The goal of this task is to rate story
fragment. Note: Please take the time to fully read and understand the story fragment. We will reject submissions from workers that are clearly spamming the task. Story fragment: [STORY] (End of story fragment) How enjoyable do you find the story fragment? (on a scale of 1-5, with 1 being the lowest)

heading: Relevance
text: Please rate the story fragment The goal of this task is to rate story fragment. Note: Please take the time to fully read and understand the story fragment. We will reject submissions from workers that are clearly spamming the task. Story fragment: [STORY] (End of story fragment) Now read the PROMPT based on which the story fragment was written. PROMPT: [PROMPT] (End of PROMPT) How relevant is the story fragment to the prompt? (on a scale of 1-5, with 1 being the lowest)
The [STORY] and [PROMPT] are to be filled in with the story and the prompt. We show the newlines for better readability. When we query the models, we use the token \n to represent the new line.
When querying ChatGPT, we query the four attributes of the same story in one conversation; this is similar to asking the teachers to rate the same story on the same page of the Google Form. We use the same queries shown above to query ChatGPT and the order of queries is the same as the order shown above.

heading: D.2.3 Adversarial Attack Quality Evaluation
text: When querying all the LLMs in this task, we query the fluency and the meaning preserving of the same news title independently. This means that each conversation with ChatGPT will only have one question, asking about the fluency or the meaning preserving of news title(s). All the parameters for generation are the same as the default parameters in Section 3.2.
The exact query we use are:

heading: Fluency
text: You are given a news title. Please read the news title and answer the question. News title: [NEWS_TITLE] (End of news title) Question: How natural and fluent is the text of the news title? (on a scale of

heading: 1-5, with 1 being the lowest
text: The [NEWS_TITLE] will be filled in with either a benign or adversarial-attacked news title.
Meaning Preserving You are given two news titles. Please read the news titles and answer the question. News title 1: [BENIGN_TITLE] (End of news title 1) News title 2: [ADVERSARIAL_TITLE] (End of news title 2) Question: Do you agree that the meaning (or semantics) of news title 1 is preserved in news title 2? (on a scale of

heading: 1-5, with 1 being the strongly disagree and 5 being strongly agree.)
text: The [BENIGN_TITLE] will be filled in with the news title before the attack and the [ADVERSARIAL_TITLE] will be filled in with the news title after an adversarial attack.

heading: E Experiment Details on Adversarial Attacks
text: The adversarial samples used in Section 4 are from Yoo et al. (2022). Yoo et al. (2022) generates different sets of adversarial samples using different adversarial attacks against different victim models. We use the adversarial samples generated against a bert-base-uncased text classifier trained on AG-News, using three different adversarial attacks: Textfooler, PWWS, and BAE. The intent of the dataset is to facilitate the research in SSA, which we do not violate.
Here, we show the supplementary results of using text-davinci-003 as the LLM evaluation for evaluating the quality of adversarial samples in Table 11. We can see that the result of using text-davinci-003 is similar to ChatGPT in the sense that text-davinci-003 also rates adversarial samples higher than humans while still signifi-
cantly lower than the benign samples. As already seen in Section 3.3, text-davinci-003 tends to give a higher rating.
As mentioned in Section 4.3, one teacher rates the fluency of Textfooler significantly higher than PWWS while the other two teachers do not. We show the rating on fluency on the three adversarial attacks by each teacher in Table 12.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Sec 5 and Limitation on page 10
3 A2. Did you discuss any potential risks of your work? Sec 5 and Limitation on page 10
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.2, Appendix B.1 and E
3 B1. Did you cite the creators of artifacts you used? Section 4.2, Appendix B.1 and E
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The datasets we use do not include a license
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix E and Ethical statement
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Removing the names in AG-News will make news titles to be nonsensical.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4.2, Appendix B.1 and E
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.2, Appendix B.1 and E
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3, 4, Appendix C
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix C.1
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix C.1, C.2
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Ethical statement
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. We do not ahve ethic review board in our institute.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? This is not related to our task. We report the certification of the workers in Appendix C.1.

heading: Future Work
text: We do not recommend that future researchers completely eliminate human evaluation; rather, we believe that human evaluation should be used in conjunction with LLM evaluation. Both methods have their own advantages and disadvantages, making them both necessary for evaluating NLP systems