# Towards Automatic Evaluation of NLG Tasks Using Conversational Large Language Models

Md Riyadh, M. Omair Shafiq

► To cite this version:

Md Riyadh, M. Omair Shafiq. Towards Automatic Evaluation of NLG Tasks Using Conversational Large Language Models. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.425-437, 10.1007/978-3-031-34107-6_34. hal-05498085

HAL Id: hal-05498085

https://inria.hal.science/hal-05498085v1

Submitted on 6 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_3.png" /></figure>

# Towards Automatic Evaluation of NLG Tasks
Using Conversational Large Language Models

Md Riyadh(✉) and M. Omair Shafiq(✉)

Carleton University, Ottawa, ON, Canada

mdriyadh@cmail.carleton.ca1, omair.shafiq@carleton.ca2

**Abstract.** Evaluating the quality of machine generated open-ended texts is a long-standing challenge in Natural Language Processing (NLP). Even though there have been dramatic advancements in the machine learning technologies that propelled the research work concerning Natural Language Generation (NLG), a subdivision of NLP that focuses on text generation, a promising and widely adopted automatic evaluation technique for NLG tasks is yet to be developed. In this paper, we propose leveraging conversational Large Language Models (LLMs) as automatic evaluators for several open-ended NLG tasks. Our experiments with a recently released conversational LLM named ChatGPT demonstrate the viability of our proposal.

**Keywords:** NLG · Automatic Evaluator · LLM · ChatGPT

## 1 Introduction

With the advent of Transformers [1] and large pretrained language models (LLM) (e.g., [2]) in recent years, there have been a plethora of research work in Natural Language Processing (NLP) that contributed significantly to the advancement of this field. NLP tasks are often broadly categorized into two: Natural Language Understand (NLU) and Natural Language Generation (NLG) [3]. LLMs have dramatically advanced area of NLG, catalyzing development of sophisticated machine learning architectures to execute tasks like paraphrasing, question-answering, machine translation etc. with impressive performance [4]. While there have been rapid advancements in the NLG field, the difficulty in evaluating them has become more prominent. Unlike many NLP tasks such as classification tasks like sentiment analysis, NLG tasks cannot be easily evaluated with automatic evaluation techniques. This is due to the nature of the NLG tasks, many of which can be open ended. Some researchers have attempted to develop automatic evaluation techniques for NLG tasks [5, 6]. However, none of them have received widespread adoption primarily due to their poor generalizability to evaluate open ended and diverse NLG tasks. As a result, a vast majority of the NLG tasks still require performing tedious and lengthy evaluations by human participants.

A recent addition to the LLM technology is ChatGPT [7], which is closely related to the InstructGPT [8] model and fine-tuned from a GPT-3.5 variant [9]. As the name
suggests, ChatGPT is a conversational LLM that can provide detailed answer to a variety of questions. It is trained to understand many simple and complex instructions and execute those instructions effectively. Guo et al. [10] experimented with ChatGPT's capability to perform a variety of NLP tasks. In this study, we tap into the similar capabilities of ChatGPT but with the objective of understanding its ability to be an automatic evaluator for some NLG tasks. In addition to some common NLG tasks, we focus on the domain of Emotion-Cause Analysis (ECS) [11], which consists of several tasks related to the cause of an expressed emotion in a given text. Our experiments cover all the NLG tasks within the ECA domain that are currently available: Emotion-Cause Generation (ECG) [12] and Emotion-Cause mitigating Suggestion Generation (ECSGen) [13]. Through a series of experiments, we attempt to understand if ChatGPT can be an effective automatic evaluator for various NLG tasks, particularly the ones that do not require specific domain knowledge and that typically need to be evaluated by human participants.

The objective of this study is to understand the capability of conversational LLMs as automatic evaluators for open-ended NLG tasks that do not require domain specific information. The main contributions of this study include: (a) We propose leveraging a recently released conversational LLM named ChatGPT as an automatic evaluator for the NLG tasks, especially the open-ended tasks that do not need domain specific knowledge. (b) We perform a series of experiments to understand ChatGPT's ability as an automatic evaluator for three common NLG tasks: Paraphrasing, Question-Answering, Story-cloze test (i.e., evaluates models' commonsense reasoning capability). (c) We also experiment with two ECA related NLG tasks: ECG, ECSGen. These are currently the only two NLG tasks available within the domain of ECA. Consequently, our experiments on the usage of ChatGPT as an automatic evaluator covers all ECA related NLG tasks that exist today. To the best of our knowledge, this is the first attempt to suggest an automatic evaluator for ECA related NLG tasks.

Despite rapid advances in NLG, automatically evaluating the open-ended generated text by machines remains a challenge. We propose using conversational LLMs as automatic evaluators for some common NLG tasks, including for recently introduced ECA related NLG tasks. Our experiments demonstrate the effectiveness of this approach.

## 2 Related Studies

The evaluation of NLG systems has conventionally been conducted by human evaluators [14]. In such procedure, human evaluators are presented with generated texts from the NLG systems and written texts from human. The performance of the NLG system is determined by comparing the ratings assigned to each. This evaluation approach, which was first introduced in the mid-90s [15], continues to be utilized in the present day in various forms for the majority of the NLG tasks. Researchers later suggested that evaluating NLG systems by comparing the generated texts to a corpus of human-written texts offers the potential for faster, more cost-effective and scalable evaluation [16–18]. Despite its advantages, this method has faced criticism, with objections such as that the generated texts may differ from corpus texts while still meeting the criteria of successfully completing a NLG task [19].
Some other researchers developed a group of alternative automatic metrics called word-based metrics (WBM). WBMs typically evaluate the similarity between the output text produced by the systems and human-generated reference texts based on various criteria [20]. The closer the similarity between the output and reference texts, the higher the score obtained from the metric. BLEU [21], ROUGE [22] etc. are some common examples of such metrics. However, none of these metrics can perform the tasks typically done by human evaluators due to their limited scope.

In addition to WBM, automatic metrics related to NLG also include readability metrics such as Flesch Reading Ease score [23] that quantifies the “reading ease” of a text based on the number of characters per sentence, words per sentence, and syllables per word. Grammatical correctness is also another automatic metric that some NLG studies tend to use to understand if an output by a machine learning model is grammatically correct [24]. More recently, there has been a growing interest in the use of Transformer-based metrics for evaluating NLG tasks. They can be broadly divided into two categories: reference-based models that require human references (e.g., [27, 28]) and reference-free models that do not (e.g., [25, 26]). The latter category is more cost-effective to implement. However, some researchers pointed out that these metrics are still inadequate for evaluating specific NLG tasks without infusing task specific knowledge in them – which can be burdensome and increase the scope and complexity of any given study [29]. This could be one of the reasons why older and less accurate metrics such as BLEU continue to dominate the field of automatic evaluation of NLG tasks despite their limitations [30, 31].

Due to all these limitations of the existing automatic evaluation techniques for NLG tasks, particularly for tasks that more open ended in nature such as paraphrasing, most researchers still opt for human evaluation in order to conduct more robust and trustworthy analysis.

## 3 Conversational LLMs as Automatic Evaluators for NLG Tasks

We propose leveraging conversational LLM as an alternative automatic evaluator for many NLG tasks, especially the ones that do not require specific domain knowledge, and that typically require human evaluators. To evaluate our proposal, we use ChatGPT [7], a recently released conversational LLM. Evaluating NLG tasks with human evaluators requires the evaluators to understand the tasks with simple instructions and execute those tasks accordingly. For example, to evaluate the meaningfulness of a summary of a given news article, human evaluators are typically given instructions on how to evaluate the relevance of such summaries in relation to the original news article. Based on that, they may be tasked with ranking the relevancy of those summaries. As we demonstrate in the upcoming sections, ChatGPT is similarly capable of understanding such instructions and complete the given task. This capability, in our opinion, makes ChatGPT a suitable candidate to become an automatic evaluator for many NLG tasks.

To use ChatGPT for such evaluations, the instruction set of each task needs to be designed carefully, ideally through experimenting with ChatGPT and by observing its response. These instruction sets are often called “prompt” in the context of LLMs [32]. After an effective prompt template is determined for set a given NLG task, prompts
for each datapoint can be generated automatically by developing simple custom logic
in programming languages such as Python [33]. These prompts then can be provided
to ChatGPT using the available interface and then its response can be collected sub-
sequently. In the following section, we conduct a series of experiments to understand
ChatGPT's capability as an automatic evaluator for some NLG tasks.

4 Evaluation

In this study, we evaluate ChatGPT as an automatic evaluator for the following five NLG
tasks: (a) Paraphrasing, (b) Question-Answering, (c) Story-cloze Test [34], (d) ECSGen
[13], (e) ECG [12].

4.1 Procedure

We perform five small scale experiments to evaluate ChatGPT for the five selected
NLG tasks. Each experiment has some specificities, but they all share some common
procedures, which we discuss below:

* We take a small sample of data (i.e., about 100 datapoints) from an existing dataset for each task. This is since we leveraged ChatGPT's Research Preview [7] for this study which (at the time of this study) is accessible through a browser interface requiring tedious manual input. For sampling, we leverage different random sampling techniques including balanced and stratified sampling for different tasks to present diverse data compositions to ChatGPT.

* To use the dataset for our intended evaluation, we preprocess each of them differently depending on their original format and evaluation process. We auto-generate ChatGPT prompts for each datapoints for all five tasks using Python [33]. We use different prompt formats for each task, which we designed by iteratively experimenting with ChatGPT for the best outcome in terms of aiding ChatGPT to understand the task as well as receiving answer from it in a format that is easy to process for analysis.

* We enter these prompts into ChatGPT's interface and manually collect its response for each of the datapoint. This interaction with ChatGPT Research Preview occurred between Jan 26, 2023, and Feb 16, 2023.

* After that we compare ChatGPT's responses with the ones included in the datasets. To observe how ChatGPT performs as an automatic evaluator for different NLG tasks that can be assessed differently, we leverage diverse techniques such as classification accuracy, multiple choice type questions (MCQ), correlating Likert responses etc. We use Python [33], IBM SPSS [35], and Google Sheet [36] for our analysis. Experiments for each task included some distinct procedures, particularly as it relates to the dataset used for each, how the task is conducted as well as the analysis methods we used for them. We discuss them below.

For the paraphrasing task, we take sample from the GLUE dataset's Quora Question Pairs [37]. The dataset contains two one-sentence long questions. The task is to determine whether the questions are paraphrases of each other. To understand if ChatGPT can automatically evaluate if a paraphrase is correct, we frame this as a binary classification
problem for analysis where ChatGPT attempts to determine the correct and incorrect paraphrases. ChatGPT's performance on this would indicate how it can perform as an evaluator for the NLG task when machine learning models generate a paraphrase for a given text (in this case, short text).

For the question-answering task, we utilize the WikiQA dataset [38] where each question has several correct and incorrect answers. The goal is to determine which answers are correct and which ones are not. For the ease of analysis, we randomly sample some questions with one incorrect answer and some with one correct answer. We then frame this task as a binary classification problem for our analysis where ChatGPT's task is to determine which answers are correct for a given question and which are not. We expect this to indicate how ChatGPT can perform as an evaluator to understand a model's performance on generating answer for a given question.

Similarly, we leverage an existing dataset for the Story-cloze test [34] which also evaluate the common-sense reasoning of machine learning models. Each datapoint in this dataset contains a short story consisting of five sentences. For each, the first four sentences are provided in correct order. For the fifth sentence, there are two options, only one of them being correct. The task is to determine which option is the appropriate choice as the fifth or last sentence of the story. ChatGPT's performance in determining the correct fifth sentence in this task would indicate how well it can evaluate the models that are tasked with completing unfinished stories using common-sense reasoning.

The ECSGen dataset [13] contains 5-point Likert responses [39] from human evaluators for each “emotion-cause mitigating suggestion”, representing the degree of relevance of the suggestion in relation to the cause of the emotion. The five levels of Likert scale are: 1: Not at all relevant, 2: Slightly relevant, 3: Somewhat relevant, 4: Moderately relevant, and 5: Very relevant. The task for ChatGPT is to rank the sampled suggestions similarly. The dataset contains 3 suggestions for each statement. We sampled 120 datapoints; consequently, our sample contains 360 suggestions. For analysis, we frame this as a multi-category classification problem. Since the human response for each suggestion can be extensively subjective for this type of questions, particularly with 5 intensity levels to choose from, we compare ChatGPT's response with the ones from human evaluators using various techniques such as, correlation testing, classification accuracy at 5-levels, and classifications accuracy at 3 and 2-levels by recoding the Likert labels accordingly. We believe ChatGPT's performance in this experiment can indicate how well it can rank the relevancy of a suggestion that aims to mitigate the cause of an emotion expressed in a short text.

For the ECG task [12], we preprocess a sample from an existing Emotion-stimuli dataset [40] for our study. We replace the “cause-span” for each datapoint with an underscore (“_”). We convert the dataset to a comma-separated values format where the separated cause-span, and the original texts (without the cause-span) are placed into separate columns. Then we create a duplicate column from the “cause-span” column and randomize the order of this new column. Thus, for each datapoint, we have a correct cause and an incorrect cause for the expressed emotion. We remove the datapoints where we found duplicates between these two cause columns and where both causes seemed equally appropriate. We frame it as an MCQ-style binary classification task for ChatGPT where it attempts to pick the correct cause for each statement with an emotional
expression. We expect this to indicate ChatGPT's ability to evaluate the meaningfulness or relevance of a generated cause in relation to the emotion expressed in a text. For sample prompts for each task that we explored in this study, please check this repository<fnref n="1" />.

In this study, we only evaluate ChatGPT, a recently released conversational LLM, as an automatic evaluator for a set of NLG tasks and report our findings. We do not compare ChatGPT's performance for this with any other LLMs. This is due to the fact our evaluation setting involves providing instructions to ChatGPT in a conversational manner, similar to the ones typically given to human evaluators. To the best of our knowledge, other than ChatGPT, there is no model that is publicly available for inference that can aptly understand these instructions to describe complex tasks and execute the tasks as instructed. Consequently, we only experiment with ChatGPT and report our findings accordingly.

## 4.2 Results

**Paraphrasing:** ChatGPT classified the correct and incorrect paraphrases with an accuracy score of 0.81. For the correct ones, precision and recall scores are 0.84 and 0.77 with an F1 score of 0.80 for 60 datapoints, and for the incorrect ones, precision and recall scores are 0.78 and 0.85 with an F1 score of 0.82 for the same amount of datapoints.

**Question-Answering:** ChatGPT classified the correct and incorrect answers with an accuracy score of 0.89. For the correct answers, precision and recall scores are 0.81 and 0.88 with an F1 score of 0.85 for 34 datapoints, and for the incorrect answers, precision and recall score are 0.94 and 0.89 with an F1 score of 0.91 for 65 datapoints.

**Story-cloze Test:** For this task, ChatGPT was able to choose the correct fifth sentence with an accuracy score of 0.99 for a total datapoints of 157 short stories. Since it is an MCQ style question that requires identifying only the correct answer between two options, the individual scores (e.g., precision, recall) for each option is not relevant, hence we do not report them.

**ECSGen:** We conduct a series of analyses to understand ChatGPT's performance on the ECSGen task as the only dataset available for this task is relatively more complicated compared to the other evaluated tasks, which uses 5-point Likert scale representing the degree of relevance of a suggestion to an emotion-cause. As explained in the previous section, 5-point Likert responses can be extremely subjective, particularly between the adjacent levels. Consequently, we analyze this using several techniques to understand ChatGPT's suitability as an automatic evaluator for the ECSGen task. The mean score of 5-point Likert responses from human evaluators and ChatGPT are 2.83 and 2.47 respectively, with a median of 3 and 2, and a standard deviation of 1.451 and 1.117 respectively. For the correlation test, we primarily use Spearman coefficient (0.334) as it fits the data type and distribution used in the study. However, we also report its parametric counterpart: Pearson coefficient (0.33), as an additional correlation measure. Both correlation

<footnote n="1">¹ https://github.com/riyadhctg/ChatGPT-prompt-for-NLG-tasks</footnote>
tests indicate positive correlation between ChatGPT and human responses and both tests
were significant with a p-value < 0.001.

We also report the distance between the Likert responses for each question to under-
stand the percentages of the answers that varied differently. About 26.11% suggestions
had the exact same ranking by human evaluators and ChatGPT. ChatGPT and human
evaluators' responses varied by only 1 Likert point for 40% of the suggestions. This
indicates that more than 65% of the responses are exact match or only varied by 1
Likert-point. For the remainder of the suggestions, 22.50% varied by 2 Likert points,
10.28% varied by 3, and only 1.11% varied by 4 Likert-point.

Additionally, we measure ChatGPT’s effectiveness as an automatic evaluator for the ECSGen task by framing this analysis as a classification problem. However, since the responses are in 5-point Likert scale that uses typical labels for intensity measurement, the accuracy score is expectedly low (0.26) for classification with five categories (i.e., 5-point Likert scale). Consequently, we perform further analysis by recoding the Likert scores.

First, we recode this as a 3-category classification problem, where Likert score 1 is
recoded as 1, 5 is recoded as 3, and the more ambiguous scores in between are recoded to
2. This expectedly increases the classification accuracy (0.55) compared to the 5-category
approach. Next, we recode the original data again in 3-categories. However, this time, we
recode 1 and 2 as 1, 4 and 5 as 3, and 3 as 2. This yields a similar classification accuracy
(0.48) as the previous 3-category approach although slightly lower. We also analyze
this from a binary classification perspective, for which we use two different approaches.
First, we remove the datapoints with Likert score 3 from the original 5-point Likert data
(i.e., 3 representing the mathematical mid-point in 5-point Likert scale). We recode 1
and 2 as 1, and 4 and 5 as 2. This increases the classification accuracy further to 0.67.
Lastly, we look at the agreement of human evaluators and ChatGPT for only the lowest
and the highest scores: 1 and 5. This dramatically reduces the datapoints for analysis, but
it provides a closer representation of a binary classification problem. The classification
accuracy with this approach is significantly higher with a score of 0.91. The confusion
matrices in Fig. 1 provides a more detailed view into these various classification results.
<figure><img src="image_4.png" /><figcaption>**Fig. 1.** ECSGen confusion matrices: a) 5-class. b) binary with only label 1 and 5. c) binary with label 3 removed and 1, 2 recoded to 1 and 4, 5 recoded to 2. d) 3-class with label 1 and 2 recoded as 1; 4 and 5 as 3; and 3 as 2. e) 3-class recoded 1 and 2 as 1; 4 and 5 as 3; and 3 as 2.</figcaption></figure>

**ECG:** Similar to the Story-cloze test, for the ECG task, ChatGPT is tasked with MCQ-style questions where the requirement is to choose an emotion-cause from the two given options that more appropriately explains the emotion expressed in a given statement. ChatGPT achieved an accuracy score of 0.98 for a total datapoints of 120.

Table 1 represents the summary of ChatGPT's performance in evaluating all the NLG tasks considered in this study.
<table><caption>**Table 1.** Summary of ChatGPT's performance as an evaluator for various NLG tasks. Asterisk (*) indicates alternative recoding technique used (more details above).</caption><thead><tr><th>Task</th><th>Analysis Method(s)</th><th>Score(s)</th></tr></thead><tbody><tr><td>Paraphrasing</td><td>Classification Accuracy</td><td>0.81</td></tr><tr><td>Question Answering</td><td>Classification Accuracy</td><td>0.89</td></tr><tr><td>Story-cloze</td><td>Classification Accuracy Score (MCQ)</td><td>0.99</td></tr><tr><td>ECSGen</td><td>Spearman Coefficient, Pearson Coefficient</td><td>0.334, 0.33</td></tr><tr><td>ECSGen</td><td>Classification Accuracy - Number of classes: 5, 3, 3*, 2, 2*</td><td>0.26, 0.55, 0.48*, 0.67, 0.91*</td></tr><tr><td>ECG</td><td>Classification Accuracy (MCQ)</td><td>0.98</td></tr></tbody></table>

## 5 Discussion

We experimented with five NLG tasks to understand if ChatGPT, a recently released conversational LLM, can be an automatic evaluator for these tasks. Three of them are among the common NLG tasks and the remaining two have been recently introduced within the ECA domain. We observe that ChatGPT can understand all these tasks and conduct the evaluation for them with minimal instructions akin to human evaluators. In addition, we leverage various techniques to evaluate ChatGPT's strength as an automatic evaluator for the given NLG tasks. These include framing the analysis as classification problem with two or more categories, MCQ style questions, and Likert-scale ranking. We observe that ChatGPT is capable of understanding these diverse framings of problems with minimal instructions, which, we believe, reinforce its potential as an automatic evaluator for many NLG tasks.

We observe that ChatGPT achieves almost 100% accuracy for two tasks: ECG and Story-cloze test. For question-answering and para-phrasing it scored 0.89 and 0.81 respectively. We opine that these consistent high scores represent ChatGPT's ability to be an effective evaluator for these NLG tasks. In our analysis, ECSGen task's evaluation is different than the others experimented with in this study given the fact that its dataset comes with a 5-point Likert response from human evaluators which is prone to subjectivity. As explained earlier, we perform more analysis for this task to understand ChatGPT's suitability to be an effective evaluator for this task. Based on the classification accuracy that ChatGPT achieved in our analysis of the ECSGen task, particularly when framed as a binary problem, we suggest that ChatGPT can be considered as a useful evaluator for this task as well. This is also reinforced by the positive score in the correlation tests between ChatGPT and human evaluators' ranking. In addition, we observe that the distance between the Likert responses from ChatGPT and human evaluators indicate high correlation between them. Particularly, more than 65% of the responses having a Likert distance of 1 or 0, while only about 12% of the responses varied widely with Likert distance of 3 or more. However, we believe that for ECSGen, a further study will be beneficial to confirm the findings of our study, especially with binary ranking
from human evaluators, which can be then compared against ChatGPT's response with more accuracy.

Based on our overall results, we suggest that ChatGPT can be an effective automatic evaluator for the NLG tasks used in this study. We believe that this statement can potentially be generalized to many other similar NLG tasks. This is because in our observation, ChatGPT is able to understand instructions for evaluating variety of NLG tasks, some of which have been introduced very recently. In our opinion, this ability to understand how to evaluate a new NLG task from simple instructions, and then executing those instructions with decent performance are the two most important features of ChatGPT that can make it an effective automatic evaluator for many NLG tasks in future. However, to use ChatGPT as an automatic evaluator for NLG tasks other than the ones used in this study, we recommend performing a small-scale evaluation with ChatGPT for that task. We expect that the evaluations we performed in this study can be an inspiration for such future experiments.

We observe that designing an appropriate prompt is crucial to help ChatGPT understand the task properly and respond accordingly. We recommend careful considerations while designing these prompts to use ChatGPT as an automatic evaluator for a given NLG task.

Lastly, since this is a newly suggested automatic evaluation technique, we expect this to mature over time with more successive contributions from the researcher community, especially with larger-scale studies. We expect this to initially complement existing evaluation techniques such as human evaluations. This may pave the way for further adoption of this technique in future for many NLG tasks, especially the ones that do not require specific domain knowledge. There is ongoing research to develop more conversational LLMs like ChatGPT [41]. When more such LLMs are available, preferably with enhanced performance, we believe researchers can leverage multiple conversational LLMs as automatic evaluators and report their findings accordingly for increasing the robustness of their analysis.

## 5.1 Guidelines for Researchers

We recommend the following general guidelines while using conversational LLMs such as ChatGPT as an automatic evaluator for NLG tasks:

* After the NLG task is complete, researchers can pick a small sample from the output (e.g., 10-20 datapoints, depending on the type of task) and they can use the conversational LLM of choice to evaluate these sampled outputs.

* Optionally, researchers can then provide their own evaluation of these sampled outputs (or they can choose to ask human evaluators to do this).

* Then, these evaluations of the sampled output can be presented to readers in detail including the input text used, and the generated text.

We believe this will provide the readers an understanding of the performance of the automatic evaluator which they can consider when interpreting the findings in a given study that includes automatic evaluation of NLG tasks using conversational LLMs.
# 6 Limitations and Future Studies

There are several limitations of this study. The chief among them is the fact that we used a small number of samples in our experiments. As described above, this is because ChatGPT Research Preview currently only allows access through a web browser and manually inputting a large amount of datapoint can be tedious and erroneous. It is also because we believe that such sample is sufficient to get an indication on the general efficacy of ChatGPT as an evaluator of the NLG tasks discussed in this study. It is however a noteworthy limitation, and we acknowledge the need of investigating this further with larger samples.

We investigated ChatGPT's effectiveness as an automatic evaluator for several NLG tasks. However, there are many other NLG tasks that we have not experimented with. We constrained this study to a small subset of NLG tasks to keep the experiments within the scope of this study. We acknowledge that our results cannot be generalized all NLG tasks. We encourage researchers to investigate the efficacy of ChatGPT and possible future variants as an automatic evaluator for more NLG tasks.

The current public access of ChatGPT only allows interacting with the model, but not to gain full access to it. In addition, the model is regularly updated with more training data. As a result, its response for one question may change over time. While we expect the response of ChatGPT to become more enhanced over time, we however, suggest that the apparent varying nature of ChatGPT needs to be considered while interpreting the results we publish in this study.

# 7 Conclusions

We experimented with five NLG tasks to understand the performance of a recently released conversational LLM named ChatGPT as an automatic evaluator for these tasks. We observe that ChatGPT is able to understand how to execute these tasks with simple instructions akin to the ones typically given to the human evaluators. We believe that this capability to understand NLG tasks from simple instructions along with the results from our experiments, suggest that ChatGPT and possible future and more advanced variants of such conversational LLM can be effective automatic evaluators for the NLG tasks explored in this study. We opine that with more experiments, this finding can be potentially generalized to many other similar NLG tasks, especially the tasks that do not require domain specific knowledge.

# References

1. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Pro-cessing Systems, Jun. 2017, vol. 2017-December, pp. 5999–6009. Accessed 02 May 2021. https://arxiv.org/abs/1706.03762v5

2. Brown, T.B., et al.: Language Models are Few-Shot Learners (2020). Accessed 12 June 2022. https://commoncrawl.org/the-data/

3. “NLP vs. NLU vs. NLG: the differences between three natural language processing concepts - Watson Blog. https://www.ibm.com/blogs/watson/2020/11/nlp-vs-nlu-vs-nlg-the-dif-ferences-between-three-natural-language-processing-concepts/. Accessed 20 Jan 2023
4. Subramanyam Kalyan, K., Rajasekharan, A., Sangeetha, S.: AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021). https://mr-nlp.github.io. Accessed 29 Aug 2021

5. Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C.M., Eger, S.: MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2023). http://tiny.cc/vsqtbz. Accessed 18 Feb 2023

6. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BertScore: evaluating text generation with BERT (2022). https://github.com/Tiiiger/bert_score. Accessed 12 June 2022

7. ChatGPT: Optimizing Language Models for Dialogue (2023). https://openai.com/blog/cha-tgpt/. Accessed 11 Jan 2023

8. Ouyang, L., et al.: Training language models to follow instructions with human feed-back, March 2022. https://doi.org/10.48550/arxiv.2203.02155

9. Model index for researchers - OpenAI API. https://platform.openai.com/docs/model-index-for-researchers. Accessed 18 Feb 2023

10. Guo, B., et al.: How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. https://chat.openai.com/chat. Accessed 11 Feb 2023

11. Gao, Q., et al.: Overview of NTCIR-13 ECA Task

12. Riyadh, M., Shafiq, O.: towards emotion cause generation in natural language processing using deep learning. In: ICMLA 2022 (2022)

13. Riyadh, M., Shafiq, M.O.: ECSGen and iZen: a new NLP Task and A Zero-shot Framework to Perform It,” Knowledge Based Systems, vol. In Review (2023)

14. Mellish, C., Dale, R.: Evaluation in the context of natural language generation. Comput. Speech Lang. 12(4), 349–373 (1998). https://doi.org/10.1006/CSLA.1998.0106

15. Coch, J.: Evaluating and comparing three text-production techniques. In: COLING 1996 (1996)

16. Bangalore, S., Rambow, O., Whittaker, S.: Evaluation Metrics for Generation

17. Marciniak, T., Strube, M.: Classification-based generation using TAG. In: Belz, A., Evans, R., Piwek, P. (eds.) Natural Language Generation. LNCS (LNAI), vol. 3123, pp. 100–109. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-27823-8_11

18. Langkilde-Geary, I.: An Empirical Verification of Coverage and Correctness for a General-Purpose Sentence Generator

19. Reiter, E., Sripada, S.: Should Corpora Texts Be Gold Standards for NLG?, pp. 97–104 (2002). https://aclanthology.org/W02-2113. Accessed 18 Feb 2023

20. Belz, A., Reiter, E.: Comparing Automatic and Human Evaluation of NLG Systems

21. Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: BLEU: a Method for Automatic Evaluation of Machine Translation

22. Lin, C.-Y.: ROUGE: A Package for Automatic Evaluation of Summaries

23. Flesch, R.: A new readability yardstick. J. Appl. Psychol. 32(3), 221–233 (1948). https://doi.org/10.1037/H0057532

24. Novikova, J., Dušek, O., Curry, A.C., Rieser, V.: Why We Need New Evaluation Metrics for NLG. https://github.com/glampouras/JLOLS_. Accessed 12 Jun 2022

25. Zhao, W., Gao, Y., Eger, S.: Evaluating Machine Translation without Human Refer-ences Using Cross-lingual Encoders”

26. Song, Y., Zhao, J., Specia, L.: SentSim: crosslingual semantic evaluation of machine translation. In: NAACL-HLT 2021 - 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, pp. 3143–3156 (2021). https://doi.org/10.18653/V1/2021.NAACL-MAIN.252

27. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTSCORE: EVALUATING TEXT GENERATION WITH BERT. https://github.com/Tiiiger/bert. Accessed 16 May 2022
28. Colombo, P., Staerman, G., Clavel, C., Piantanida, P.: Automatic text evaluation through the lens of Wasserstein Barycenters. In: EMNLP 2021 - 2021 Conference on Em-pirical Methods in Natural Language Processing, Proceedings, pp. 10450–10466, August 2021. https://doi.org/10.48550/arxiv.2108.12463

29. Leiter, C., Lertvittayakumjorn, P., Fomicheva, M., Zhao, W., Gao, Y., Eger, S.: Towards Explainable Evaluation Metrics for Natural Language Generation. https://github.com/Gri ngham/explainable-metrics-machine-translation. Accessed 18 Feb 2023

30. Marie, B., Fujita, A., Rubino, R.: Scientific credibility of machine translation research: a meta-evaluation of 769 papers. In: ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pp. 7297–7306, June 2021. https://doi.org/10.48550/arxiv.2106.15195

31. Grünwald, J., Leiter, C., Eger, S.: Can we do that simpler? Simple, Efficient, High-Quality Evaluation Metrics for NLG

32. Yang, K., et al.: Tailor: A Prompt-Based Approach to Attribute-Based Controlled Text Generation

33. About PythonTM | Python.org. https://www.python.org/about/. Accessed 08 Jan 2023

34. Sharma, R., Allen, J.F., Bakhshandeh, O., Mostafazadeh, N.: Tackling the story ending biases in the story cloze test ACL 2018–56th annual meeting of the association for computational linguistics. Proc. Conf. (Long Papers) 2, 752–757 (2018). https://doi.org/10.18653/V1/P18-2119

35. “SPSS Software | IBM. https://www.ibm.com/spss. Accessed 08 Jan 2023

36. Google Sheets: Online Spreadsheet Editor | Google Workspace. https://www.google.com/she ets/about/. Accessed 18 Feb 2023

37. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.: GLUE: a multi-task benchmark and analysis platform for natural language understanding

38. Yang, Y., Yih, W.-T., Meek, C.: WIKIQA: A Challenge Dataset for Open-Domain Question Answering, pp. 17–21 (2015). http://aka.ms/WikiQA. Accessed 18 Feb 2023

39. Dobson, K.S., Mothersill, K.J.: Equidistant Categorical Labels For Construc-tion of Likert-type Scales (1979)

40. Ghazi, D., Inkpen, D., Szpakowicz, S.: Detecting Emotion Stimuli in Emotion-Bearing Sentences. In Computational Linguistics and Intelligent Text Processing, pp. 152–165 (2015)

41. Google AI updates: Bard and new AI features in Search. https://blog.google/technology/ai/ bard.google-ai-search-updates/. Accessed 18 Feb 2023