TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks
Authors: Dongfu Jiang
Yishan Li
Ge Zhang
Wenhao Huang
Bill Yuchen Lin
Wenhu Chen
Venue: Accepted by TMLR ()

Abstract: We present TIGERScore, a \textbf{T}rained metric that follows \textbf{I}nstruction \textbf{G}uidance to perform \textbf{E}xplainable, and \textbf{R}eference-free evaluation over a wide spectrum of text generation tasks. 
Different from other automatic evaluation methods that only provide arcane scores, TIGERScore is guided by natural language instruction to provide error analysis to pinpoint the mistakes in the generated text. Our metric is based on LLaMA-2, trained on our meticulously curated instruction-tuning dataset MetricInstruct which covers 6 text generation tasks and 23 text generation datasets. The dataset consists of 42K quadruple in the form of (instruction, input, system output $\rightarrow$ error analysis). We collected the `system outputs' through from a large variety of models to cover different types of errors.
To quantitatively assess our metric, we evaluate its correlation with human ratings on 5 held-in datasets, 2 held-out datasets and show that \metricname{} can achieve the open-source SoTA correlation with human ratings across these datasets and almost approaches GPT-4 evaluator. As a reference-free metric, its correlation can even surpass the best existing reference-based metrics. To further qualitatively assess the rationale generated by our metric, we conduct human evaluation on the generated explanations and found that the explanations are 70.8\% accurate.
Through these experimental results, we believe \metricname{} demonstrates the possibility of building universal explainable metrics to evaluate any text generation task.

Published in Transactions on Machine Learning Research (05/2024)
TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks
'Donffu Jiang", Yishan Li", Ge Zhang, Wenhao Huang, Bill Yuchen Lin, Wenhu Chen
University of Waterloo, Tsinghua University, 01.AI, Allen Institute for AI
jiang, wentuuchen)ewaterrloo.ooo liyisha1 10masaal tsinghua.
Reviewed on OpenReview: http://doeeerrvveee.rrr/oooooots
Abstract
We present TIGERSCORE] Trained metric that follows Instruction Guidance to per- form Explainable, and Reference-fres evaluation over a wide spectrum of text generation tasks. Different from other automatic evaluation methods that only provide arcane scores, TIGERScore is guided by natural language instruction to provide error analysis to pinpoint the mistakes in the generated text. Our metric is based on LLaMA-2, trained on our metic- ulously curated instruction-tuning dataset Metriclnstruct which covers text generation tasks and 23 text generation datasets. The dataset consists of 42K quadruple in the form of (instruction, input, system output + error analysis). We collected the 'system outputs' through from a large variety of models to cover different types of errors. To quantitatively assess our metric, evaluate its correlation with human ratings on 5 held-in datasets, held-out datasets and show that TIGERSCORE can achieve the open-source SoTA corre- lation with human ratings across these datasets and almost approaches GPT-4 evaluator. As a reference-free metric, its correlation can even surpass the best existing reference-based metrics. To further qualitatively assess the rationale generated by our metric, we conduct human ovaluation on the generated explanations and found that the explanations are 70.8% accurate. Through these experimental results, we believe TIGERScORE demonstrates the possibility of building universal explainable metrics to evaluate any text generation task.
1 Introduction
Evaluation for natural language generation tasks is a long-standing challenging problem. With the recent advancement of large pre-trained language models (Brown et al.) 2020 OpenAl 2023 Touvron et al., 2023
text generation models have become more widely adopted in real- world applications than ever. Newly developed text-generative models are being deployed across a wide range of downstream applications. As more and more people use these generative models, there is an increasing need to develop trustworthy evaluation metrics. Though GPT-4 has shown to achieve strong evaluation correlation with humans (Zheng et al. 2023 the open-sourced metrics are lagging significantly behind and mainly suffer from specific issues:
Dependency on references: Some evaluation metrics like ROUGE (Ln 2004), BLEU (Papineni et al. 2002b), COMET (Rei et al.) 2020), InstructScore (Xu et al., 2023c would require gold references to measure the quality. These metrics compare the generated output against one or more reference texts to assign the evaluation score. However, such an assumption can be highly unrealistic in many downstream applications where the gold reference is hard to collect.
Limited to specific domains: Some evaluation metrics are limited to specific domains, lacking the ability to generalize to broader text generation tasks. For example, COMET (Rei et al. 2020 BLEURT (Sellam)
DDonfuu Jiang and Yishan Li have led the project and contributed equally.
Project website: https://tiger-aa
al-labb al thub. io/TIGERScoee/

---

Published in Transactions on Machine Learning Research (05/2024)
Breakdown Error Anolysis
Instruction (T): Please summarizz the text
Errorl (Ez):
Input Context (x): 2023 May, there accident happening in Mississippi
location (l.): "In 2023, accident in Mississip"
Aspect (a,): Relevance
Explonation (et): Output misses the information
Output (y'): In 2023, accident in Mississip
TIGERScore
Penelty (s,):
Error2 (Bg):
location (lz): "a accident"
Metric-Instruct
Aspect (az): Fluency
Sumnnn
D2T
Trons
finetune
Explanation (ez): grammar error of "o"
Penalty (s2): -J
LF
inst
QA
ruct
Finel Score: (-)++C3)=--4
Kendall Correlation of different Metrics across 7 tasks
BARTScore-para
COMET-KI
Ulama2-13b-0-ih
GPTScore
BARTScore-con
TIGERScore [ours]
15.4
40
+3.1
+0.1
+12.74
+4.2
+12.0
+14.8
30
+9.9
20
10
Summ
Trans (Translation)
D2T [Data2Text]
LF-QA (Long-Form QA)
MathoA
Instruct
Story-Gen
Average
(Summarization)
[Instruction
(Story Generation)
Following)
The upper part shows the input and output format of TIGERSCORE, which takes an instruction and an input context, along with the model-generated output that is to be evaluated as the inputs, and output detailed breakdown error analysis with explanations. It is fineluned on our curated dataset Metric MetricInstruce The lower part shows the Kendall's correlation of different metrics w.r.t human ratings, from which we can see TIGERSCORF achieves the SoTA correlation on most generation tasks.
Figure 1:
et al. 2020), SESCORE2 (Xu et al., 2023b and InstructScore (Xu et al. 2023b) are specifically designed for tasks like machine translation or WebNLG tasks.
Lack of attribution: Some evaluation metrics tend to directly output score without any attributions, c.g., where the errors
and why. For instance, BARTScore (Yuan et al. 2021). BERTScore (Zhang eel al. 2019) and GPTScore (Fu et al., 2023) adopt the pre-trained language models' log likelihood as the evaluation metric. Such metrics do not provide the location and reason for the assigned score, which limits their trustworthiness and reliability.
To address these issues, we propose a novel metrie, TIGERSCORE, Trained metric that follows ruction Guidance perform Explainable and Reference-free evaluation. As shown in Figure I] the input to TIGER- SCORE consists of an instruction describing the task definition, the task input, and the system output. TIGERSCORE is capable of generating breakdown error analysis that can (1) locate each mistake, (2) ex- plain the mistake and suggest revisions, and (3) provide penalty score (between (-5, -0.5)) for each mistake. The final score for the system output can be calculated by summing up all the penalty scores.
TIGERSCORE is built by fine-tuning LLMs on our curated MetricInstruct dataset, which contains a total of 42K examples of (instruction, imput, system output, error analysis), obtained from diverse text gen- eration datasets. The dataset includes system outputs from more than 50 systems, covering a wide variety of errors. The error analysis is curated by prompting GPT-4 (OpenAl 2023 and filtered through various strategies. The tuned model TIGERSCORE has shown the highest overall Kendall's correlation with human ratings on seven major text generation tasks as depicted in Figure TIGERSCORE is highly convenient to use because does not. require any additional reference. We evaluate TIGERSCORE on 5 held-in datasets and 2 held-out datasets, As a reference- free metric, TIGERSCORE surpasses the best reference- free metrics (GPTScore (Fu al.] 2023)) by 15% and surpasses the best referenco-based metrics (COMET-22 (Rei al., 2022a)) by margin of 8% terms of Kendall's score. On the two held-out datasets, TIGERSCORE demon- strates unprecedented generalization capabilities. We further employ humans to evaluate the explanations
2

---

Published in Transactions on Machine Learning Research (05/2024)
generated by TIGERSCORE and found that over 70% of the generated explanations are highly accurate and trustworthy. Our analysis shows that the success of TIGERSCORF is attributed to three key aspects in our curated MetricInstruct: (1) dataset diversity, (2) error coverage, and (3) high quality, which enable TIGERScORE to generalize better on unseen tasks than any other metric.
2 TIGERScore
TIGERSCORE is built upon three design criteria: (1) It is driven by instructions, making easily adaptable to any text generation task. (2) The evaluation process eliminates the need for "gold standard" or perfect example for comparison. (3) It is highly explainable, the model can generate an error analysis that helps the user understand each identified mistake and its associated penalty.
2.1 Background
The pursuit of improved metrics for text evaluation has been significant focus since the inception of language models. Automatic n-gram- based metrics (Elliott & Keller 2014 Kallison-Buree et al., 2006 Isozaki et al,, 2010) have always served as the default metric, computing the n-gram match F-1 score with the reference text until rescarch highlighted their significant weaknesses in aligning with human preferences. Later, neural metrics were proposed to belter capture the semantic similarity in addition to mere morphological similarily only by either computing based on neural representation (Zhang et al.) 2019 Yuan et al., 2021) or directly fino-tuning with human preferences :Rei al. 2020j. It has also been demonstrated that multi-aspect scores, using the logits of large language models with desl-designed instructions for various aspects as prompts (Fu et al. 2023 could achieve an even higher correlation.
There have been some attempts to build explainable metrics leveraging the great capacity of large language models. For instance, UniEval (Zhong et al. 2022a) constructs a multi-aspect evaluation system by individ- ually training on aspect-specific data. PandaLM (Wang et al. 2023a) compares two responses for a given instruction and input to judge which is better, providing a short reason for its decision. InstructScore (Xul et al. 2023c) evaluates the quality of translation by training Llama-2 to compare the reference and transla- tion, listing errors with structured information. However, none of these metrics has been able to address all the issues mentioned in section concurrently.
2.2 Problem Formulation
Suppose y' is the system output from given source context with specific natural language instruction I to describe the task definition. And y is a corresponding reference output. If a metric uses y, it's reference- based, otherwise, it's reference-free. For example, when T refers to "translation", an instruction I for that task could be "Translate the following text from Chinese to English". For each task type T, we ask the evaluation metric to focus on few pre-defined evaluation aspects Ay like relevance, factuality, fluency, etc.
TIGERSCORE reference-free metric, defined by a function f take the triple (I,2,3') input, produce a list of structured errors {Ei,..., Em) as the otuput, called the error analysis content:
{..., E,. = {..., (4, e,, e,, s.), = = x(,,
(1)
where (4, (4 G4, st) denotes specific information of the error Eq. Specifically, 4 points to the location of the error, and a; Ar is a pre-defined aspect to which this error belongs. e; comprises both the explanation of this error and its revision suggestion. is the penalty score reduced for this error, which lies in [-5, 00.5]. The final score of y is computed as the sum of all the penalty scores: Sy = Σi 8/- The range of the final score lies within (-o0,0], 0], where 0 means perfect generation and a lower score indicates worse quality and more severe errors. However, in practice, the number detected errors affected by both the training data and the generation length, thus leading to less than 10 errors most of the time during the evaluation.
2.3 Multi-Aspect Evaluation
As stated in the problem formulation, each error considered to be attributed to a certain aspect that the system output might make mistakes For each task, 3 4 aspects designed with the help of both
3

---

Published in Transactions on Machine Learning Research (05/2024)
Metric-Instruct
Prompting Error Analysis
Inst
Summ
D2T
Trans
QA
Instruction (f):
Query
Instruction
lepot Contert (8)
Oueput (y
Input Context (x))
Referance (y))
Aspects (431
System Outputs
Generation
Based on the given contenes, identity eners that asseciated with the provided espects
Reference (y):
Real-World channel
Instruction (T): Please summarize the text
Output (y')
Imput Context (x); In 2023 May, there accident happening in Mississippi
Breakdown Error Analysis
prompt
Erroit (Ra):
location (4)): ): In 2023, accident in Mississip
Summarization: BART, BRIO, Long-form QA: BART, Vicuna, MathQA: WizardMath (50+ Systems)
OpenAl models
Aspect (a,): Relevonce
generating
Explanation (e,): Output misses the information
Penalty (s:):
prompt
Output in 2023, accident Mississip
Output 2: 2023, accident Missiasip
Intraction (
Referince (7)
Instruction (I)
Query
Bosed the
Select and filtering
Input Context (x))
incorrect cutput from reforence that
cantain asseciated with the
Good Output: 2023, on accident Mississip Error Output: in 2023, occident in Mississip
Reference (y): ...
Synthetic channel
proiied aspects.
Also identlly the your generated
Figure Overall pipeline of constructing MetricInstruct through the two-channel collection. The main difference is that the real-world channel collects outputs y from real-world models, while the synthetic channel asks GPT-4 to synthesize an error output based on the provided reference.
GPT-4 prompting and human supervision, detailed in subsection A.6 These aspects designed to be both non-overlapping (mutually exclusive) and collectively exhaustive. For instance, 4 aspects to evaluate output for the instruction-following tasks is shown in Table
2.4 Training Setup
TIGERSCORE is finetuned Tlama-2-7B and Llama-2-13B Touvron al. 2023 respectively with both's batch size being 128. The prompt templates used for both the fine-tuning and inference given in [Table The model's maximum context length is set to 1024. We use cosine learning scheduler with the warmpup ratio being 0.1 We finetune the 7B version on 4 A100 GPUs (80GB) for 3 epochs with a learning rate of 20-5. And 13B version is run on A100 GPUs (80GB) for 2 epochs with a learning rate of 1e-5. Inference of TIGERSCORE is conducted on a single A100 GPU with the assistance of the VLLM toolkit to increase the speed. (Kwon al. 2023)
Instruct
Aspect
Definition
Comprebension
Evaluates how well the output understands the given instruction.
Accuracy Mvasures the currectntss the oulput in relation to the instruction and the paired input context.
Informativeness
Assesses the relevancy and usefulness of the information provided by the output.
Coherence
Evaluates how logically the output flows and connects.
Table 1: Definitions of evaluation aspects of TIGERSCORE for cnarruction-oolloiinn (Instruct) as an exam- ple. See full table in Table 10 for aspect definitions of all 6 text generation tasks
4

---

Published in Transactions on Machine Learning Research (05/2024)
Real-Worid (training set)
Syuthetic (training set)
Task
Dataset
Output Source
# Sample
Dataset
Output Source
# Sample
SummEEal,, XSum,
CNN DSMm
Summarization (Sumi)
Systems
4339
GPT-4
Nowsroom,SAMuur
Cigaword
Transiation (Trans)
WMT-224
18 Systems
5507
WMT-22
GPT-4
672
WebNLGG-0200
WikiTableText
Data2Text (D2T)
17 Systems
4701
GPT-4
160
WiliTabbleText.oTTT
Dart,ToTT
ASQA,FeTaQA
Long-Form QA (L.F-QA)
CAs1
Systems
3370
Cosios QA,ELI5
GPT-4
2146
MathQA
GSM8K
Systems
4529
None
Instruction-oll Following
Alpaca Dolly Guanaco, OASST
MixInstruetf
11 Systemss
2248
GPT-4
3014
(Iustruct)
Table 2: The composition of our dataset. For synthetic data, the output is generated by asking GPT-4 to synthesize incorrect outputs that contain a few specific types of crrors. For the datasets with f. we take their pre-released system outputs. For the others, we collect the system outpuls on our own.
3 Metriclnstruct
We present the MetricInstruct dataset, which is employed to fine-tune TIGERSCORE. The three underlying criteria for dataset. construction are: (1) dataset diversity: we choose 23 distinctive datasets as the source context to cover enough generation tasks. (2) error coverage: we take system outputs generated
from 50+ text generation systems to cover all types of errors and guarantee balanced distribution. (3) quality ensurance: to ensure MetricInstruct is tailored to gather in-depth error analysis as detailed in subsection 2.2 we sourced it by prompting GPT-4 (OpenAT 2023) and then filtered through different heuristics to eliminate low-quality error analysis.
3.1 Diverse Dataset Source
MetricInstruct incorporates samples from 23 distinctive text generation datascts, which are categorized into 6 major categories of text generation tasks. While the collection encompasses well-researched tasks such as summarization (Summ), translation (Trans), and data2text (D2T), it also introduces popular new tasks like Long-Fort Forg QA (LF-QA), MathQA, and instruction-foolloiing (Instruct). These latter tasks have wit- nessed limited evaluation research. Although the assessment of traditional tasks has dominated the research landscape, posit that effectively evaluating these new tasks is crucial for constructing comprehensive evaluator for all text generation domains.
In addition, we meticulously selected datasets for each task to ensure diverse coverage across the knowledge domain, as shown in Table 2 For instance, in the case of the LF-QA task, we utilize ASQA (Stelmakh ct al. 2022b) to include knowledge about ambiguous factoid questions, while FeTaQA Naa et al. 2022b) is employed handle challenges related tabular source question answering. We then selected samples from each dataset's 's training set, following the predefined constraints like maximum input and reference output lengths, topics balancing of instances, and
3.2 Broad Coverage of Errors
As shown in Figure 2, our "system outputs" come from two channels, namely real-world system outputs and synthetic incorrect outputs, representing 2 main components of MetricInstrrut..
We consider a wide range of systems and use their outputs as our evaluation input, as shown in Table 2 These outputs are either collected from pre-released outputs, like WMT-2021 official evaluation systems, or generated by us by prompting existing domain-speccillize models, like BRIO [Liu et. al. 2022 Wizard- Math (Luo et al. 2023), etc.
As illustrated in Figure 2 for those self-generated outputs, we sample 5 different outputs using top-p sampling for each instance using various output systems. After that, we BARTScore to sort the 5
5

---

Published in Transactions on Machine Learning Research (05/2024)
outputs. Outputs with lower BARTScore are heuristically considered to contain more errors and will be preferred to be chosen the final hypothesis output y to be evaluated.
Subsequently, we utilized meticulously designed prompting templates (see in A.7 to elicit standardized error analysis from GPT-4. The main idea is to provide them with the instruction I, input 2, the reference output y, and system output y along with the definitions of pre-defined aspects Ar, to query the OpenAT models generate the list of errors, as described in subsection 2.2 We also report their correlation performance in Table 4] showcasing the reference- based ChatGPT results.
Synthetic: While real-world system outputs ensure the error distribution is aligned with practical sce- narios, they might be overly tailored to the bias of these limited systems and omit certain error cases their outputs fail to cover. Therefore, we decided to synthesize more incorrect outputs that can balance under-represented errors by prompting GPT-4 with our designed templates.
Firstly, to complement the real-world system outputs to cover broader error cases, we prompt GPT-4 to deliberately generate designated erroncous outputs, modified from the existing reference output y, using the prompt template in (Table 18 By supplying GPT-4 with combination of randomly selected aspects and their definitions Ar, we control the aspect of errors it produces so that error aspects can be more balanced. Datasets used in this step are reported in the right column of Tabll 2
Secondly, to improve TIGERSCORE'S generalization capability in evaluation for tasks or errors it has not previously encountered, we employ a strategy that involves prompting GPT-4 to generate data with cus- tomized error aspects lailored individual instructions. Our rationale is that while the manually designed aspects are logical, they may not be specific enough for tasks that involve following instructions where various types of instructions will be included. Bosides, this also prevent the trained model from being overfitted to the pre-defined aspects in subsection A.6 To do this, supplement the first part of synthetic data by sampling 10k data points from the Alpaca-524 dataset and using the template in Table 19 to prompt GPT-4 to get this second part of synthetic data.
3.3 Heurstic-based Filtering
We refine our preliminary raw datasot drawn from both real-world and synthetic channels using heuristic filtering techniques. Initially, we remove any anomalous data in JSON format. Instances marked by hal- lucinated mismatched error locations, illogical severity labels, excessively long outputs are considered flawed and excluded. Furthermore, given the reference-free nature of TIGERSCORF, relies on reference outputs for justification is excluded. These steps climinate approximately 35% of the initial data. Subsequently, we employ GPT-4 to assess the reasonableness of our error analysis, utilizing the template outlined in [aalle 20 This process further filtered out about 15% percentage data. This phase further excludes about 15% of the data. Ultimately, we compile a high-quality dataset of 42,484 instances, designated as MetricInstruct.
explanation that
4 Experiments
4.1
Evaluation Datasets
We have gathered both the held-in and held-out datascts to compare the performance of TIGERSCORE with the existing baseline metrics. The basic statisties of some main datasets for each task are shown in Table 3 System outputs to be evaluated of each test dataset are either from official releases, such
SummEval FFbbbri et al. 2021). WebNL.G-2020 (Zhou & Lampouras) 2020), WMT-22 (Freitag et al. 2022
and OpenMEVA (Guan et al.] 2021 or by generating by ourselves, such as A-F-E-C (Stelmakh et al. 2022a
Nan et al. 2022a Fan el al. 2019; Huang et al. 2019). GSM8K (Cobbe et al. 2021 LIMA (Zhou et al.
2023) and AlpacaEval et al. 2023).
Human preference scores are necessary to conduct the correlation analysis. To get the gold preference scores we used to conduct the evaluation experiments, we collected either existing human ratings like WMT-22
6

---

Published in Transactions on Machine Learning Research (05/2024)
Task
Eval Dataset
Output Source
# Inputs
# Samples
Held-in Evaluation Dataset (test set)
Summ
SummEval
16 Systems
100
1600
Trans
WMTT-22 (zh-en)
Systems
1875
33750
D2T
WebNI.G-2020
18 Systems
179
28418
ASQA, FeTaQA,
LF-QA
Systems
400
1600
CosmosQA, EL15
Math QA
GSM8K
2 Systems
1319
2638
Held-out Evaluation Dalaset (lest set)
Instruct Story-Gen
LIpea,lpa
9 Systems
500
4500
OpenMEVA (ROC)
Systems
200
1000
Table 3: Overview of our main evaluation datasets. "# Samples" is computed by multipling "# Inputs"
with the number of outputs systems, representing the total instances that metric need to compute.
MQM scores for translation, or GPT-4 rated scores for Long-form QA. Details of the curation are shown in subsection A.5
4.2 Baselines
We categorize the basclines into referenco-based and reference-free metrics for fair comparison.
Reference-based: We choose popular metrics, including BLEU Paapineni al. 2002a), ROUGE (Lin
2004), BERTScore (Zhang et al. 2019), BLEURT (Sellam et al. 2020 and BARTScore (Yuan et al. 2021
Recent emerging metrics are also included, like COMET-22 (Rei et al. 2022a). UniEval (Zhong et al. 2022b
GPTScore (Fu et al.) 2023), and InstructScore (Xu et al.) 2023c). Specifically, we use BARTScore-re to denote that we adopt the ref-hypo scoring type. For GPTScore, we use FLAN-T5-base (Ching et al. 2022 as base models and use GPTScore-rre to denote that we adopt the f-1 average of the ref-hypo and hypo-ref scores. We also report the zero-shot results of EPT--3.-turbl with the same prompting templates as those used for generating real-world channel data. Reporting this baseline will help us understand whether TIGERSCORE has surpassed its easy substitute using the cheap OpenAl model, thus proving its effectiveness.
feference-free: We choose BARTScore, GPTScore and COMETKiwi (Rei et al. 2022b as reference- baselines compare. Specifically, use BARTScore-sre to denote the src-hypo scoring type, thus making it referenco-free metric. For GPTScore, still use the FLAN-T5-base model and use GPTScore-sre to denote that we use the src-hypo scoring lype. We also include the frequently used (-shot results of Llama- 2-13b TTouvron et al. 2023) and GPT-4 (OpenAT 2023/for better comparison and to know more about the performance gaps with GPT-4.
4.3
Main Results
We present a comprehensive analysis of TIGERSCORE across all 5 held-in tasks and 2 held-out task in Table 4 reporting Kendall correlation results. Additionally, we provide supplementary results on Pearson and Spearman correlations in the Appendix (see Table 25 and Table 26 We average the performance across these tasks gauge the general ability of the model.
Our results highlight the significant advantages of TIGERSCORE over other reference-free metrics. Notably, it has surpassed all other reference-free metrics in Kendall correlation. In Pearson correlation, it is the highest for 6 out of 7 tasks. This underscores the robustness and consistency of TIGERSCORE in evaluating text generation tasks.
Compared with reference- based basclines, TIGERSCORE generally outperforms most reference-based met- rics. However, it does score lower than some task-specific metrics like UniEval (summ) in summarization.
7

---

Published in Transactions on Machine Learning Research (05/2024)
Tasks-+
Summ
Trans
D2T
LF-QA
MathQA
Instruct
Story-Gen
Average
GPT-basen
Metrics
GPT-3.5-turbo (zero-shot)
30.45
32.30
30.38
20.91
58.57 46.63
17.73
3.26
27.65
GPT-4 (zero-shot)
29.32
35.38
32.26
35.85
49.50
25.69
36.38
Reference-basse Metrics
BLEU
8.71
14.50
23.13
7.73
17.25
35.92
-0.89
15.
ROUGE-2F
10.67
13.19
24.74
11.73
18.07
34.59
1.78
16.40
InstructScore
20.86
40.44
30.21
15.64
-3.87
13.87
13.50
18.66
CPTScore-ref
10.80
18.74
27.47
22.13
14.86
25.40
12.78
18.88
BARTTScrre-cn (hypo-ref)
10.00
21.06
27.04
20.67
19.07
24.70
20.16
BARTScore-para (hypo-ref)
10.41
24.90
28.42
20.24
14.10
26.13
12.11
19.47
BERTScore
17.39
31.57
30.74
17.70
9.41
35.61
2.00
20.63
BLEURT
12.69
36.12
34.48
23.11
2.88
27.94
19.18
22.34
UniEval (summ)
35.89
16.08
28.56
29.32
16.15
11.93
31.22
24.17
COMET-22
25.01
42.79
23.43
21.66
-4.52
36.17
27.52
25.01
Reference free Metrics
BARTScore-pans (src-hypo)
29.12
7.01
22.32
18.80
-2.21
4.26
14.15
13.35
BARTScore-cnn (sre-hypo)
26.63
9.40
23.69
28.93
1.23
19.09
23.29
18.89
25.22
11.79
23.45
15.96
1.08
19.50
21.52
16.93
COMETKiwi
11.87
36.37
19.08
12.23
-9.38
26.46
12.78
15.63
GPTScore-sre
28.20
6.50
19.81
27.64
11.64
20.04
16.36
18.60
TIGERScore-75
28.79
33.65
32.44
33.93
19.98
38.13
29.72
30.95
TIGERScore-133
31.29
36.50
36.43
33.17
21.58
41.84
35.33
33.73
A (ours best reference-free)
+2
+0
+13
+4
+10
+15
+14
+155
A (ours best reference- baase))ce-based)
+4
+2
+5
+8
Table 4: The Kendall correlation results of all the baseline metrics and TIGERSCORE on the evaluation datasets shown in TTble 3. For each task, the metric with the highest correlation to average performance is highlighted in bold. The results of Pearson and Spearman are reported in Table 25 and Table 26 respectively,
showcasing the same great performance of TIGERSCORE
COMET-22 in translation. We consider these discrepancies acceptable, because the compared metrics are beference-based and all specifically fine-tuned for single task. Furthermore, TIGERSCORE achieves signif- icantly higher overall correlation, compared with the cheap API substitute GPT-335-Turbo (zero-shot) and the Llama-2-11 2-13b-chat (zero-shot), proving its effectiveness. What's exciting to note that TIGERSCORE- 13b can achieve comparable correlation performance with GPT-4 (zero-shot), even higher on summarization, translation, data2text, and story generation.
4.4 Human Evaluation
To better understand the quality the generated error analysis provided by TIGERSCORE, a random selection of 50 error analyses from each evaluation dataset was assessed by human experts who rated them from the following perspectives: 1) Reasonableness, 2) Comprehensiveness, 3) Effectiveness, 4) Overall Estimation, whose definitions given in the following:
Reasonableness: The human experts directly pointed out which errors are problematic in error analyses, examining whether the analysis contained hallucination or illogical reasoning.
Comprehensiveness: The human experts carefully review the source, output, and error analyses to de- termine if there are any additional errors unnoticed by TIGERSCORF. Based on human exports' analysis, they give score on scale of 4, specifically focused on identifying potential errors that may have been overlooked in the original analysis conducted by TIGERSCORE.
8

---

Published in Transactions on Machine Learning Research (05/2024)
Aspects
Explanation Error?
Overlooked Errors
Revision Suggestions
Overall Rating
Rate-
No
Yes
1
2
4
1
3
4
2
3
4
5
Summ
70
35
2
17
15
16
6
19
7
14
3
10
17
7
13
Trans
54
25
3
8
17
22
2
17
6
18
6
15
9
17
D2T
19
21
8
11
9
10
7
18
LF-QA
42
19
4
10
11
25
5
8
14
7
16
6
8
10
6
20
MathQA
39
26
5
12
12
21
5
19
9
10
13
14
Instruct
9
5
5
32
21
5
2
19
9
4
7
27
Story-Gen
66
29
7
16
13
14
7
16
10
11
7
12
11
9
11
Total
295
164
27
76
86
161
57
43
99
40
111
43
59
70
58
120
Table 5: Human evaluation results, the first question is asked per error in error analyses, and the others are per sample. Superior performance is indicated by higher numerical values. The most-voted rate of each task for each human evaluation aspect is bolded.
Effectiveness:
The revision suggestions in error analyses are evaluated by human experts, on a scale of 1 to 5, to determine their appropriateness and effectiveness in enhancing the output quantity.
The Human experts further assign an overall score on a scale of based on the reasonableness, comprehensiveness, and effectiveness of the error analysis.
Overall:
We report the detailed human evaluation results in Table 5| it is found that 64.3% of TIGERSCORE'S error analyses are deemed reasonable, that is, the answer to the first question is 'no errors in interpretation' This suggests that most error analyses accurately identified and explained errors. In 70.6% of cases, evaluators gave a positive score (3 or 4) for question 2, implying no missing errors were found. This demonstrates TIGERSCORE's effectiveness positive ratings to 5), indicating good quality and usefulness in identifying and explaining errors according to human experts.
comprehensive error analysis. Overall, 70.8% of error analyses received
4.5 Hallucination Analysis
Alignments on no-error outputs In order to understand the hallucinations generated by TIGERScore, we used TIGERScore to evaluate the gold reference, which used in reference- based metrics, and expect TIGERScore not to hallucinate errors on these no-crror instances. Results are shown in Table
In tasks related to instruction-foollvingg and long-form QA, TIGERScore demonstrrtes high level of accu- racy, avoiding hallucinations in over 85% of cases. This highlights its proficiency in producing factual and error-free analysis. However, TIGERScore is less consistent in tasks like summarization, translation, and data-to-text conversion. In these areas, it often fails to achieve perfect scores (0), but still frequently identi- fies gold references as either flawless or only minimally flawed (with score reductions less than 2). This could due to the subjective nature of these tasks, where minor errors, such as the substitution of similar words, maybe more open to interpretation. Additionally, TIGERScore faces challenges in tasks like MathQA and story generation. These difficulties may stem from the inherent complexity of MathQA problems and the subjective nature of story creation, well as specific limitations of TIGERScore in these areas. Improving TIGERScore's performance in these challenging tasks remains lopic for future research.
Hallucinations analysis of TIGERScore outputs To better understand how TIGERScore handles hallucinations, conducted experiments across six different tasks. For each task, ran TIGERScore- 13B on 20 samples with errors in the system output. We then used GPT-4 to determine if these samples contained hallucinations or factual inaccuracies, as outlined in the prompting templates found in Tabll 22 According to the results in Table 7 approximately 89.28% TIGERSCORE's error analyses are freee from hallucinations or factual errors. We acknowledge the limitations of our study, including the small sample size and the reliance on GPT-4 rather than human evaluators. Nonetheless, our findings are significant, demonstrating that TIGERSCORR effective at avoiding hallucinations in generated content.

---

Published in Transactions on Machine Learning Research (05/2024)
Tasks-
Summ
Trans
D2T
LF-QA
MathQA
Instruct
Story-Gen
Average
Gold reference's score =
TIGERScore-7B
16.00
3.57
45.51
84.75 94.50
34.36
73.98
34.00
41.74
TIGERScoro-13B
48.00
21.01
23.03
25.28
86.38
46.00
49.17
Gold reference's score >-
TIGERScore-7B
68.00
72.48
78.65
84.75
34.36
92.48
34.00
66.39
TIGERScore-13B
97.00
83.63
94.94
94.50
25.28
96.14
49.00
77.21
Table 6: TIGERScore's score on the gold reference of the test set. For each task, the column refers the percentage that TIGERScore reduces 0 scores for the gold references. The -2 column refers to the percentage that TIGERScore reduces less or equal to score the gold references.
Tasks
Summ
Trans
D2T
LF-QA
MathQA
Instruct
Story-Gen
Total
Accuracy
95.00
95.00
90.00
85.00
90.00
75.00
95.00
89.28
Table 7: The accuracy of the analysis from TIGERScore-131 assessed by GPT-4 that do not contain hallucinations or factual errors. if includes hallucinations or factual errors, assessed by GPT4.
Spearman Correlation
Syn
ReW
40
Mix
30
20
10
Sum
Trans
D2T
LF-QA
MathQA
Instruct
Story-Gen
Figure 3: Investigation of the influence of Real-World & Synthesic mix training on the 7B model.
4.6 Abaltion Study on Data Source
Ablation of Two Channel Data Collection To assess the effectiveness of our two-channel data in en- hancing TIGERSCORE'S performance, we fine-tune 3 models using different dataset setups for experiments: using only Real-World data (RcW), using only Synthetic data (Syn) and using the mixed of both (Mix). The results are detailed in the Spearman correlation Figure 3 the Mix model outperformed both Re W and Syn 5 of the tasks. It also achieves an average correlation approximately 20% higher than ReW and 31% higher than Syn. Slight decreases in correlation for D2T and Story-Gen tasks are deemed acceptable compromises for better overall performance.
Ablation of Each Single Generation Task In order to investigate the contribution of cach task in the MetricInstruct, we conducted experiments to see whether multi-lask training will benefit the single task. We trained a model for each task and compared its performance that TIGERSCORE. As depicted in the Table 8 the multi-task learning paradigm does benefit almost all tasks' performance, except for MathQA. We contend that Math QA poses a significant challenge for LLMs, and separately trained model is more adept at handling this task.
10

---

Published in Transactions on Machine Learning Research (05/2024)
Metries] Tasks->
Summ
Trans
D2T
LF-QA
MathQA
Instruct
Average
Single Task
35.55
41,65
39.75
41.60
26.75
41.19
37.75
Multi Task
36.81
44.99
45.88
46.22
23.32
47.03
40.71
A (%)
3.56
8.02
15.43
11.09
-12.83
14.18
7.84
Table 8: Ablation of the influence of multiple tasks mix or single task on the 13B model.
5 Related Work
5.1
Instruction-driven Large language models
Instruction tuning has recently become the standard to "align" language models with more useful objectives and human preferences. The instruction tuning step is normally done to enhance certain skillset of large language models. Previously, instruction tuning has been focused activating models' general capabilities to follow instructions to solve general tasks. Some work has been published like NaturalInstruction (Wang et al. 2022). FLAN (Wei et al., 2021 and TO (Sanh et al. 2021) are the earliest work in the field. Later on, FLAN-v2 Chunng et al. 2022) have been proposed to understand the effect of scaling up the instruction datasets to understand its impact on model performance. These approaches mainly adopt
datasets to build the instruction following dataset. More recently, multiple works (Wang al. 2023b Xu
al. 2023a propose to utilize synthetic instruction following data distilled from GPT-4 to align open-source LLMs. Our work differs from them in the sense that our method aims to activate specialized capability to generale error analysis according to instruction, which is the first of its kind.
5.2
Explainable Metrics
The increasing focus on model interpretability has led to a surge in research dedicated to explainable metrics. Research these fields aims to build a metric system for a certain task that readable to humans and is ex- pectod to help the development of better text generation systems (Leiter et al. 2022). Early endeavors in this area delved into explainability via multi-faceted evaluations, as exemplified by works such as Unieval (Zhong) et al. 2022b) and GPTSScoee (Fu et al. 2023). As LLM blooms, researchers began to directly prompt LLMs to create interpretable metrics. One instance is PandaLM, trained on Llama to compare two responses pair- wisely and provide textual rationale for its decisions (Wang et al., 2023a). Another noteworthy approach is Snsre, leveraging large language models as knowledge repositories to obtaín premium error analysis examples (uu et al. 2023ep. Despite these commendable advancements, most existing explainable metrics still require gold references and are often limited concerning the task domain. Our contribution distinguishes itself by offering reference-free nature and the cross-task ability brought by instruction-tuning over large language models, aiming to serve as universal explainable metric.
6 Conclusion
In this paper, we propose the novel metric TIGERSCORF, which is able to evaluate any text generation task guided by natural language instruction. We demonstrate the exceptional performance of TIGERScORE by its high correlation with human preference. We also demonstrate the high accuracy of its generated rationale. However, TIGERSCORE does hallucinate sometimes generate false explanations. On the other hand, we found that TIGERSCORE is not good in evaluating reasoning tasks like mathQA. In the future, we plan to devote more effort to enable more faithful explanations and unleash its potential to evaluate errors for harder tasks, like reasoning errors.
Limitation
Hallucinated Errors Despite substantial efforts to minimize hallucinations in TIGERScore's output, we still observe hallucinated errors, particularly in challenging tasks such as mathQA. This issue attributed to both the quality of our training data and the limitations of our base model. A potential solution involves
11

---

Published in Transactions on Machine Learning Research (05/2024)
initial fine-tuning specific tasks for generation purposes, followed by further fine-tuning for evaluation. However, additional strategies are necessary to effectively reduce these hallucinations.
Evaluation Efficiency TIGERScore, fine-tuned on the 7B and 13B versions of Llama-2, faces challenges with inference speed when used as evaluation metric. Our testing reveals that TIGERScore achieves an evaluation speed of approximately 0.2 seconds per instance on single A800 GPU with the assistance of VLLM. While this is manageable in interactive environments, further improvements are needed for efficiency in large-scale batch evaluations, compared to faster traditional metrics like BLEU and BERTScore.
Discrepancy Between the Local Errors and Global Evaluation Dividing output evaluation into multiple local errors is logical, but using a simple summation of these errors as a global evaluation metric can lead to discrepancies. Longer outputs often have multiple errors, while shorter ones might be simply judged as entirely erroneous in a single error. Compared global evaluation methods, like rating a score out of 10, developing a structured and reasonable method accumulate and represent these errors remains area for further exploration.
Ethics Statements
Our work collected data from publicly available datasets that were ethically curated with informed consent from all participants. We ensure that all privacy data is excluded. We acknowledge the potential of our machine-learning models to generate hallucinated, biased, or unfair content. Methods have been adopted prevent the generation of these kinds of content with our best efforts. Our research involves human evaluation experiments and ensure that each participant's privacy is excluded and protected We make sure each participant is paid fairly according to their work amount. Our hourly rate is 12 dollars, which is above the US lowest payment rate.
side.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Lan- guage models are few-shot learners. In H. Tarochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Н. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 1877 1901. Cur- ran Associates, Inc., 2020. URL https: bettps://procee http:///oooceedns.eeeeee petprrpppp.c//pppr....e////ppppp 445Tco66666444444448888884444444
.dd
Chris Callison-Burch, Miles Osborne, and Philipp Koehn. Re-evaluating the role of Bleu in machine translation rescarch. In 1 11th Conference of the European Chapter of the Association for Computational Linguislics, pp. 249 256, Trento, Italy, April 2006. Association for Computational Linguistics. URL https://acllantho.oog www.oo/g///////. 1032
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Joff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction finetuned ruttuii---inn language models, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/2110.14411 2021. URL https: //api. semanticscholar. org//orppsSD::29998551
12

---

Published in Transactions on Machine Learning Research (05/2024)
Desmond Elliott and Frank Keller. Comparing automatic evaluation measures for image description. In Proceedings of the 52nd Annual Meeting of the Associalion for Compulational Linguistics (Volume 2: Short Papers), pp. 452 457, Baltimore, Maryland, June 2014. Association for Computational Linguisties. URL httpss//aclaatthloogg.
doi: 10.3115/v1/P14-200
2074
Alexander R Fabbri, Wojeiech Kryšcińíski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Compu- tational Linguistics, 9:391-409 2021.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguislics, pp. 3558 3567, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 20.18653/v1/P19-4 1346. URL https://aclanthooogg.orrgg 1ttp/////oooo..r//1/1919443
Markus Freitag, George F. Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. Ex- perts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460-1474, 2021. URL https::/appl semanticscholar. org/CorppuSD:2233444755
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. Results of WMT22 metries shared task: Stop using BLEU neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 46-68, Abu Dhabi, United Arab Emirates (Hybrid). December 2022. Association for Computational Linguisties. URL https:///allanthologg// www.ayy..gg/2..../..// 1.2
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire, 2023.
Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wenbiao Ding, Xiaoxi Mao, Changjie Fan, and Minlie Huang. Openmeva: A benchmark for evaluating open-ended story generation metrics. arXiv preprint arXiv:2105.0800 2021.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehen- sion with contextual commonsense reasoning. Conference on Empirical Methods in Natural Language Processing, 2019. CRL https://api. http://p...ssmaanccsschcooocooos htt.::/aa..rrrr/ropus0:22:2:2:00 202540590
Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. Automatic evaluation of translation quality for distant language pairs. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 944-952, Cambridge, MA, October 2010. Association for Computational Linguistics. URL https://aclaantoology..oo org///e--9//1/
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-bbender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pp. 14165 14178, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: v..18653/v12202..ar atl.:/222......loooooo lcl- long.792. 792. URL https: //aclanthology.orgg orw.y/23.../2033.0 wrw.://0o0/222...1oooooo ac1-1ong..77 792.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Christoph Leiter, Piyawat Lertvillayakumjonn, M. Fomicheva, Wei Zhao, Yang Gao, and Steffen Eger. Towards explainable evaluation metrics for natural language generation. ArXiv, Abs//2203.11133 2022.
URL http:t:://pp...eeetttccccoooaaa wwt.aaattcccccooooooooooooorraaa otg//oorpusD::245544.522
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An autonatic evaluator of Instruction-followin models. https:
//github. thub. com/ wtt.ttccomtattttlaabb///// tatss-
2023.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Chin-Yer Lin. Rouge: A package for automatic evaluation of summaries. In Teat summarization branches oul, pp. 74-81, 2004.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. Brio: Bringing order to abstractive summa- rization. Xiv preprint or Xrx::22003. 16804, 2022.
Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. Error analysis prompting enables human- like translation evaluation in large language models: A case study on chatgpt. ArXiv, abs/2303.1380 13809, 2023. URL
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308. 09583, 2023.
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojeiech Kryécińsski, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Tsabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. FeTaQA: Free- form table question answering. Transactions of the Association for Computational Linguistics, 10:35-49, 2022a. doi: 10.1162/tacl a 00446. URL https://acllatthoopgs
wtrg/2022.22 tacl-1.3
Linyong Nan, Chiachun Hsich, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojeiech Krys- ciński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, et al. Fetaqa: Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35 49, 2022b.
OpenAL. GPT-4 technical report. CoRR, abs/ /2303.08774, 2023. doi: 10.48550/ar http://aXx....3.077777 2303.0877. URL
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation
machine translation. In Annual Meeting of the Association for Computational Linguistics, 2002a. URL
https://api..emantiicchhoe .oree.orrrrooeeee orwgCCorpusDD... 1 11080756)
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. Tn Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311 318, 2002b.
Ricardo Rei, Craig Stewart, Ana Fariuha, and Alon Lavie. COMET: neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pp. 2685 2702, Online, November 2020. Association for Computational Linguisties. doi: 0..18653/v1/2000 eemeepmminmmnnn
URL httpp::/aaclannhooooyy..//////// wttp:/rr////00...mmmmmmm........ emnlp-main.21
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 578 585, Abu Dhabi, United Arab Emirates (Hybrid), December 2022a. Association for Computational Linguistics. URL https://aclanthont
org/ 2022. wmt
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. Tn Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 634 645, Abu Dhabi, United Arab Emirates (Hybrid), December 2022b. Association for Computational Linguistics. URL htttt::://claattiiitt org/ 2022. wm6-1.60
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2021.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Thibault Sellam, Dipanjan Das, and Ankur Parikh. Bleurt: Learning robust metrics for text generation. In Proceedings of the 58th Arouua Meeting of the Association for Compulational Länguistics, pp. 7881-7892, 2020.
Tvan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-We: Wei Chang. ASQA: Factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8273 8288, Abu Dhabi, United Arab Emirates, December 2022a. Association for Computational Linguis- emn.p-main.5666 m66..566. URL
tics. doi: 10.18653/ 10653/vv/22/ / http//22eemlnppp/me.ee
Tvan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wao Wei Chang. Asqa: Factoid questions meet. long-form answers. Xixiv preprint ar Xxiv:2204.06092, 2022b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothéc Lacroix, Bapliste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint WwXiv2230.13977 2023.
Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang. Hao Chen, Chaoya Jiang, Rui Xic, Jindong Wang, Xingxu Xie, Wei Ye, Shi-Bo Zhang, and Yue Zhang. Pandalm: An automalic evaluation benchmark for Ilm instruction tuning optimization. ArXiv, semanticscholar... hwwpcccccchcaaroooo/rrooore argg/oorpuII::25910 259108266
2023a. URL https://api.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganch Kordi, Amirreza Mirzaci, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Aujana Arunkunar, David Stap, et al. Super- naturalinstruucions:: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5085-5109, 2022.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Han- naneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. The 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023), 2023b. URL https:
//aclanthology.org///
ortpy..rrg///0 w023. acl. long. 754.754... pdff
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference Learning Representations, 2021.
Can Xu, Qingleng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint 2KN.:2344.12244 2023a.
Wenda Xu, Xian Qian, Mingxuan Wang, Lei Li, and William Yang Wang. SESCORE2: Learning text generation evaluation via synthesizing realistic mistakes. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Popers), pp. 5166-5183, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: v.11653/v1/2023.ad long.288...3. URL https: //aclanthology.org//// 2ow..ooor//22......1oogg 283,
Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. Instructscore: Towards explainable text generation evaluation with automatic feedback, 2023c.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation.
Advances in Neural Information Processing Systems, 34:27263 27277, 2021.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2019.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, al. Judging llm-as-a-judge with mt-bench and chatbot arena. Xiviv preprint ar Xic:2306. 05685, 2023.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Peng Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards unified multi-dimensional evaluator for text generation. In Conference on Empirical Methods in Natural Language Processing, 2022a. URL https://api htt.://w....a.i......ooooooooroa 2527733311
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards a unified multi-nimeniionall evaluator for lext generation, 2022b.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyor, and Omer Levy. Lima: Less is more for alignment, 2023.
Giulio Zhou and Gerasimos Lampouras. Webnlg challenge 2020: Language agnostic delexicalisation for multilingual rdf-to-text generation. In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pp. 186-191, 2020.
A Appendix
A.1 Prompting Strategies
In our study extract high-quality error-analysis insights from GPT-4, we employed various intuitive prompting strategies. These strategies are detailed in the prompting templates found in subsection A.7 Key stralegies are outlined below:
Two-Step Generation and Formatting Process One of the challenges in eliciting structured knowledge from LLMs is directing them to generate content in specific format. Although GPT-4 often adheres to given formats, there are instances of deviations. We observed that enforcing a strict format can compromise content quality, as indicated by reduced correlation in our analysis. Our approach involves initially allowing GPT-4 to generate responses freely in the first. conversation turn. In the subsequent turn, we request GPT-4 to reformat its response according to a predefined format. The initial generation templates for each task, along with singular template for the formatting step, are listed subsection A.7
Incorporation of Task-Specific Words in Initial Queries To leverage GPT-4's task-specific knowl- edge, we designed varied prompting templates for different lasks with slight modifications. Keywords like 'Source,' "Instruction," 'Reference," 'Output,' 'Solution," 2 "Translation," and 'Summary' are dynamically uti- lized in various task contexts. In tasks like instruction-folooingg where the context is self-cxplanatory, we omitted specific keywords.
Integration of Predefined Aspect Definitions for Focused Evaluation Directly requesting GPT-4 to evaluate task outputs often led to low-quality error identification. Tt either points out simple discrepancies with the reference misses crucial evaluation aspects, thus overlooking some errors. To address this, we incorporated predefined evaluation aspects defined in Table 10 into the templates, guiding GPT-4 to produce more focused responses. Exceptionally, for data2text, we found that. directly evaluating errors was good enough, and thus, we did not include our predefined aspects.
Classification of Errors Using Major/Minor Error Guidelines Drawing inspiration from the MQM translation human rating system and InstructScore prompting template (Frcitag et al. [2021] Xu et al. 2023c
we classified translation errors as either major or minor. This classification helped GPT-4 in assigning more
consistent scores each error, countering its instability with numerical judgments (Lu al., 2023
Adopting a 0.5 to 5 Scale for Scoring Error Severity Initially, we used an integer scale from 1 5 for error penalty scores. While effective translation tasks, this scale is less effective in tasks like summarization, data2text, and custruction-oolooiing Our experiments demonstrated that more nuanced scoring scale ranging from 0.5 to 5 yielded better correlation across all tasks
16

---

Published in Transactions on Machine Learning Research (05/2024)
A.2 Creation of evaluation aspects
With the assistance of GPT-4, we have carefully curated the evaluation aspects of each task that are mulually exclusive and collectively exhaustive. The steps include:
Step We prompt GPT-4 to output 20 candidate aspects for cach task.
Step 2: We ask GPT-4 to summarize these aspects into 3 to 5 general aspects for this task.
Step We ask GPT-4 to generate detailed definition and specific error types under each aspect.
In each step, we check the reasonability of GPT-4's responsc and make necessary modifications the responses, including summarized error aspects, definition, and error types, to make them clear, concise, and typical. Example prompts used in each step are show in Table 21
A.3 TIGERScore prompt template
You are evaluating errors in a model-generated oulput for a given instruction. Instruction: 8(generation ingeneratio__istrreeeeee instruction
((input_contee context)
Model-generater Output:
S ttyyothesis_outpphe output]
For each ertor you give in the response, please also elaborate the following information: error location (the words that are wrong in the output)
error aspect it belongs to.
explanation why it's an error, and the correction suggestions.
severity of the error ('Major' or 'Minor').
reduction of score (between 0.5 and 5 given the severity of the error)
Your evaluation output:
Table 9: The prompt template used for the fnn-tuning and inference of TIGERSCORE
17

---

Published in Transactions on Machine Learning Research (05/2024)
Context Length (Token)
Number
Tasks
4000
TIGERScore Context Length
3500
# Examples
3000
D2T Tharts
3000
2000
Matho4
Examples
Instruct
1000
2000
1500
Errors
Severity
1000
Minor Major
12170
10595
7434
10026
10112
9530
10000
500
-5000
5724
11655
5885
3855
846
3667
250
500
750
1000
1250
1500
1750
2000
Context Length
Trans
D2T
LF-QA
MathQA Instruct
Figure Distribution analysis of Metriclnstruct training data for the context length, number errors, and error severities. The left. figure shows the context length distribution. The right-lop plot illustrates the distribution of the per-instance number of errors. The right-bottom figure illustrates the counts of 'Major" and "Minor" crrors for all six tasks
A.4 Metriclnstruct data statistics analysis
In order to find the secret of the success MetricInstruct, we conducted a deep analysis of its inherent statistics, as shown in Figure 4 we first count the distribution of data length through the Llama tokenizer. The results show that more than 90% of data has length lower than 1024. Due to the lack of GPU resources for fine-tuning in the longer context length scenario, this distribution further demonstrates the reasonability of using 1024 as the context length of TIGERScore.
Furthermore, we examined the incidence of errors per data instance across various tasks. The figure illustrates that each task contains most instances with the number of errors being 1 or 2 and fewer instances are considcred perfoct and very erroncous, reflecting a naturally occurring distribution. We contend that such a balanced distribution crucial for the model's performance it helps reducing fabricated errors in correct outputs and aids in the precise identification of minor and major errors in outputs that are partially correct entirely incorrect.
Additionally, categorized errors 'Major' or 'Minor' and quantified their occurrences. Our analysis reveals that tasks of more subjective nature, such as translation and summarization, tend to have higher frequency of minor errors. Contrastingly, in tasks like MathQA, the predominance of major errors is in alignment with the expectation that mathematical inaccuracies are generally more critical."
A.5 Gold scores for correlation computation
Human preference scores are necessary to conduct the correlation analysis. Those datasets with official system outputs released are usually accompanied by systematic human preference scores, like the WMT- MQM score for the translation task. However, these scores are not available for tasks like long-form QA, MathQA, and interuction-folloiing where we need create on our own.
Therefore, here introduce what human preference ments. For summarizztion, data2text, and story generation tasks, use their official human ratings from multiple aspects of their released outputs. For translation, we use the official WMT-22 MQM scores as the gold scores. For MathQA, we simply use the accuracy (1 or 0) as the gold preferences. For instruct-following, we use human ratings from the hugging face community of datasct where 500 instances are sampled from LIMA and AlpacaEvla. For Long-form QA, we use the powerful GPT-4 to perform pairwise comparisons for them and count the winning times the way rank them, which is similar to how Jiang al. (2023) constructs MixInstruct.
have used to conduct the evaluation experi-
18

---

Published in Transactions on Machine Learning Research (05/2024)
A.6
Evaluation aspects for all tasks
Task
Aspecl
Definition
Relevance
degree which summarized output accurstely reflects the key points input text.
Fact Consistency
facts summary cousistent with facts original
Summ
Coherence
Pertains the logical and mesningful arrangement of ideas the sumunry..
Fluency
Reviews the model-generated output's use of language, including grammar, punctuation, and vocabulary that aflect the quality the senteuces.
Accuracy
degree which the translated text adheres to original text, maintaining same meaning, context and cultural nuances.
Fluency
How naturally the translation reads in the target langnage.
Trans
Terminloogy
The appropriate use of specifie terms and jargon related to particular field or industry. Translator's s ability to maintain the same style, tone, voice as the original text. Example eroor types include:
Style Matching
Accuracy
Deals with the correctiess of the inforuation presented the output.
Logical Coherence
How well the output transforns structured data into a comprehcusible, logical, and engaging text.
D2T
Fluency
Reviews the model- model-geeerteed outpot's 's use of language, inclurting grammar, ponctuation, and vocabnlary that
quality
sentences.
Accuracy
Evaluates the factual correctness the answer.
Completeness
Evaluates if the answer leaves out critical parts details that were asked in the question.
LF-QA
Informativruess
Asseses the quality of the respotse in lerms of how helpful at is for the user to understand the auswer.
Clarity
Asseses the readahility and understandabiiity of the resporse.
Prohlem Understanding
output accurately comprehend the text Involves translating the problem from textual form into mathematical equation of equations that can be solved.
Assesrs
deseription the math problem.
Problem Formulation
MathQQ
Computing Accuracy
Asseses the output's 's ability to perform the mathematical operations accurately to arrive at the correct solution.
Solution Interpretation
Involvee the how well the output correctly interpret the solution of the problem context of the original problem statement.
Comprehension
Evaluates how well the outpot understands the given instruction.
Accuracy
Measures the correctnes the outpui relation to the instruction and the puired input rontext.
Instruct
Informativeness
Assesrs relevancy usefulness
information provided by output.
Coherence
Evaluates how logically the output flows and connects.
Table 10: Definitions of evaluation aspects of TIGERSCORE for the 6 text generation task.
19

---

Published in Transactions on Machine Learning Research (05/2024)
A.7 Prompting templates
Source: [[input_context]
Reference: {{reference_our _utteet (hapuut:(hyyohhsssse _oupput]
Based on the given Source and Reference, please evaluate the quality of summary(Qutpat) written for the input text. Please score the summarization with 0.5 to for aspects below. Then, identify the major and minor errors in this output for the $task task. There may be multiple errors or no error in the output. Here are the aspects you need to focus on: (aspects descriptions)
Table 11: Prompting templates for summarization task
Translation Instruction: #ggeneration_instrreeee
Source Text: S(input 3oopput__cotext]
{(reference (ectrrencc__oupput>
shodel-generated Translation: S(hypothesis >
Please identify and categorize the errors in the model-generated translation either Major Minor. Major errors significantly impact the task, while Minor errors are subjective and represent minor imperfections. When identifying errors, do not solely rely the reference translation for comparison. Provide explanations expert in the task domain, without explicitly mentioning the reference output.
Table 12: Prompting templates for translation task
Task instruction: (generationn instruction] Source: S (input__cnneed context)
(reference _atere___ttttt_
Output: $ (hypothesis _ottpttt__ttt
Based on the given source and reference, identify the major and minor errors this Output for the data text task, which is to $generation instruction. Note that Major errors refer to actual errors that affects the task severely, may change the meaning the output, and Minor errors refer smaller imperfections, and purely subjective opinions about the output. There may be multiple errors or error in the output.
Table 13: Prompting templates for data2text task
8 (gennratioon instruction]
context)
The correct solution
(reeference_or _
model-generated solution is: $(hypothesis_oog )
Please identify all the errors in this output considering the following aspects: 8(aspects list.
Table 14: Prompting templates for mathQA task
20

---

Published in Transactions on Machine Learning Research (05/2024)
Source: 8(input _
(rreference j
Output: (hyppothesis /
Based on the given Source and reference, identify the major and minor errors which is to dggeneration_instreeeee change the meaning the output, and Minor errors refer smaller imperfections, and purely subjective opinions about the output. You should check about 8(aspects descriptions). There may be multiple errors error in the output.
Output $(task) task, Note that Major errors refer to actual errors that affects the task severely, may
Table 15: Prompting templates for long- QA task
8 ((generttion instruction and _ource] (rrferenne_ontppnn
Output: (hypothssis ) output}
Based on the given Source and reference, identify the major and minor errors in this Output for the $(task) task. Note that Major errors refer actual errors that affects task severely, may change meaning of the output, and Minor errors refer to smaller imperfections, and purely subjective opinions about the output. You should check about 8 (aspects _escrrpptissss There may be multiple errors or no error output.
Table 16: Prompting templates for instruction-follovin task
For each identified in your response, please provide the following information in specific JSON format: Error Location: The substring in the Output that contains the error.
Error Aspect: Choose only one from $(aspects. list).
Explanation: Describe why the identified issue is an error, and offer suggestions for correction. Explain as an expert in the lask domain, without explicitly mentioning the reference oulput.
Severity: Classify the error as "Major" or 'Minor'.
Score Reduction: Assign a reduction score between 0.5 5, considering the severity of the error. JSON Format for Output:
If there are no errors:
{" deeroos": {})
If there are errors:
(' ('errors": {
"error_1"
"error location':
"error "error_aspect aspect":
'explanation';
severity"; "
score "sccor_rreducteee reduction":
}.
}}
Table 17: Prompt template to format the response into json format
21

---

Published in Transactions on Machine Learning Research (05/2024)
You are generating an output given the following instruction and context: (ggnneration instruction] (input >
The correct output is: S(reference _
Please generate an incorrect output for the given instruction and context by modifying the correct output. The incorrect output should contain the following errors:
For each error, give me the error location (the substring that is wrong in the generated incorrect output)
error aspect
explanation (the generic error lype description, why it's an error, and the correction suggestions) severity ("major" or "minor")
score reduction (an integer between 1 to 5 given the severity of the error)
Output format:
Generated incorrect output:
Error location 1:
Error aspect 1:
Explanation 1:
Severity
Score reduction 1:
Table 18: Prompt template to generate data in the synthetic channel for all 6 tasks.
22

---

Published in Transactions on Machine Learning Research (05/2024)
Instruction: (instruction)
{(npput}
A ground-truth response:
{oulput)
model will asked to respond to this instruction. However, that response might contain errors various aspects.
Please first output possible error aspects if a model is asked to generate a response for the above instruction. The error aspects don't have be of the above aspects and can be any aspect that you think is reasonable for this instruction.
Then generate an incorrect response contains up to {(num_errorr) errors of these aspects. Each error corresponds of the aspect. The incorrect response should mimic style the real-generaticn of a model.
Then give an analysis of these errors. For each error, give me the error location (the substring that is wrong generated incorrect output)
error aspect
explanation (the generic error type description, why it's an error, and the correction suggestions)
severity ("major" or "minor")
score reduction (an integer between 1 to 5 given the severity of the error)
Output format:
Generated incorrect output:
Error location
Error aspect 1:
Explanation
Severity 1:
Score reduction 1:
Table 19: Prompt template to generate data with free-form aspects in the synthetic channel for instruction- following task. Data generated from this template aims to improve TIGERScore's generalization ability on the unseen error cases and enhance the robustness.
(instruction) (input
Model-generated output: S(output)
An error analysis provided: 8(error_ analysis)
Is the error analysis reasonable? Answer "yes" or 'no" only.
Table 20: Prompt template used to check whether the error analysis synthesized is reasonable or not.
23

---

Published in Transactions on Machine Learning Research (05/2024)
You are evaluating a model-generated output for the mathematical word problem. There might be multiple errors in the output and they can focus different evaluation aspects. Please list 20 evaluation aspects for mathematical word problems and their definitions.
Please summarize the above-provided aspects into 3 to 5 general aspects that are mutually exclusive and collectively exhaustive.
Mathematical word problems are a type of math task that requires model to read and understand text-based description of math situation and then formulate and solve mathematical equation to solve the problem. Now you are evaluating a model-generated output for mathematical word problem. you might want to design few aspects where errors are identified in the outputs could be attributed to these aspects.
These aspects could be "Problem Understanding', 'Problem Formulation',
"Computing Accuracy", 'Solution Interpretation", provide the definition of each aspect above and 5 specific error types for each aspect. Note that these aspects are used to evaluate single output instead of system.
Table 21: An example prompt when creating the template of mathQA
24

---

Published in Transactions on Machine Learning Research (05/2024)
Task instruction: Kinstructioo) Source: {(iputt}
Reference output: creference_ouupult
Model-generated output:
{(uutut))
An error analysis provided:
((error analysis}
Does error analysis include factual errors or hallucinations of the model-generated output?
Table 22: The prompt template used to identify whether there are hallucinated contents factor errors in TIGERSCORE'S response
25

---

Published in Transactions on Machine Learning Research (05/2024)
A.8 Human evaluation case study
Task Instruction Answer the following ambignous factoid question by introducing additional knowledge, clarifying the relationship between multiple possible answers (if any) and resolving the ambiguity
Input Context When did the song stayin alive come out?
A modeel-gneerate response - The song 'Stayin' Alive" is a disco hit produced by the Bee Gees and was included in the soundtrack of the film 'Saturday Night Fever" in 1977. The song released single in December 16, 1577 and quickly became a popular dance anthem, topping the charts in several countries. However, it is important to note that there another song called "Streetlife" by The Crusaders that also became popular around the same time and has similar tempo to 'Stavin' Alive' which could be confusing.
Identified errors the model- generated response from TigerScore-
The model-generated output contains errors, with a total score reduction of 6.
Error location 1: "The song 'Stayin' Alive' is disco hit produced by the Bee Gees and was included in the soundtrack of the film 'Saturday Night Fever' 1977. The song was released as a single in December 16, 1577'
Error aspect 1: Accuracy
Explanation 1: The error is factual inaceuracy. The song 'Stayin' Alive" was released in 1577, but in 1977. The correct information should be provided to ensure the accuracy of the answer. Severity Major
Score reduction 1: 4
Error location 'However, important to note that there is another song called 'Streetlife" by The Crusaders that also became popular around the same time and has similar tempo to 'Stavin' Alive' which could be confusing." Error aspect 2: Informativeness
Explanation 2: This is an example of irrelevance. While it is true that there is a song called 'Streetlife" by The Crusaders that became popular around the same time as 'Stayin' Alive', it is not relevant to the question asked. The user asked for the release date of "Stayin' Alive', not for similar songs that may have been popular the same time. The answer should stick to the information requested. Severity 2: Minor Score reduction 2: 2
Table 23: A case study our human evaluation results for long-form QA task, the main body. Human ovaluation questions and example answers are shown Table 24
26

---

Published in Transactions on Machine Learning Research (05/2024)
1. Please first read through all the contents above. Which identified errors are not reasonable?
All the errors reasonable and the identified errors are true errors
Error Error
2. there any significant errors that not identified?
No.
There
missing identified
Yes. But the missed error is minor and does not affects the overall evaluation
major error, but most other major Yes. And the evaluation completely misses all the major errors and does not make sense.
Yes. And the missed error
also identified.
3. Please rate the quality of TigerScore evaluation results for the model-generated output from to
1. The identified errors by TigerScore does not make sense and help at all
2. TigerScore manages to identify some errors, though few explanations of them are reasonable, you can still find something useful in the outputs.
3. TigerScore able to idenfitify part major errors and some of explanations are helpful, despite some errors still being missed.
4. TigerScore is able to identify most of the errors some most of the explanations are helpful. The missing errors are minor and are dispensable.
5. TigerScore identified all the errors and all the explanations make sense and are very helpful.
4. Does the explantion provide helpful revision suggestions?
No. Not at all
Yes. But they are not helpful at all.
Yes. And some of them are helpful.
Yes. And most them are helpful despite some small mistakes
Yes. And they all helpful enough for human revise this model generatee output.
Table 24: Human evaluations questions and corresponding answers from human rater for the instance in
Table 23
27

---

Published in Transactions on Machine Learning Research (05/2024)
A.9
Correlation results Pearson and Spearman
Tasks
Summ
Trans
D2T
LF-QA
MathQA
Instrct
Story-Gen
Average
Metries
GPT-3.5-turbo (zero-shot)
45.53
43.77
47.76
61.26
15.36
7.80
35.90
GPT-4 (zero-shot)
40.75
33.92
46.83
49.30
54.98
60.45
37.74
46.28
Reference-based Metries
BLEU
11.66
17.47
34.29
18.21
18.12
29.47
-0.64
18.37
ROUGE-2f
16.03
16.26
35.85
19.66
20.69
33.49
2.88
20.69
InstructScore
27.40
51.55
47.28
20.59
0.36
20.98
12.81
25.85
GPTScore-ref
13.47
21.05
48.70
33.40
18.22
29.66
18.94
26.20
BARTšcore-cnn (hypo-ref)
16.67
23.56
45.08
32.78
23.09
26.57
27.61
27.91
BARTScore- SaRTacore-para (hypo-ref)
19.73
29.04
47.89
32.70
17.33
30.20
17.76
27.81
BERTScore
26.26
37.65
48.22
26.39
11.19
45.58
4.08
28.48
BLEURT
17.27
43.00
54.32
34.26
3.98
39.15
27.89
31.41
UniEval (summ)
53.22
23.11
51.14
36.95
17.69
30.87
44.88
36.84
COMET-22
35.32
58.46
43.82
36.79
-5.58
49.68
40.12
36.94
Reference- free Metrics
BA BARTSScor-paara (sre-hypo)
43.11
6.96
37.82
29.86
-0.41
19.37
19.99
22.38
BARTScore-CH (yro-hypo)
39.72
9.53
45.43
41.48
3.28
34.97
33.51
29.70
Llama- 2- 13b-chat-O-shot
29.59
9.09
41.32
21.67
2.80
22.71
21.13
21.19
COMETKiwi
14.22
50.91
23.63
22.59
-13.35
34.46
19.12
21.65
GPTScore-ste
41.71
6.82
41.19
39.79
13.99
27.59
23.22
27.76
TigerScore-7B
43.95
37.70
49.13
46.10
21.77
38.26
39.90
39.54
TigerScore-
44.21
41.54
52.87
44.76
24.41
47.52
47.66
43.28
A (ours best reference-free)
+1
-9
+7
+5
+10
+20
+14
+13
A (ours best referencce-ased)
-9
-17
-2
+9
+1
-2
+-3
+-6
Table 25: The Pearson correlation results of all the baseline metrics and TIGERSCORE on the evaluation datasets shown in "Tbble 3 For each task, the metric with the highest correlation to average performance is highlighted in bold.
28

---

Published in Transactions on Machine Learning Research (05/2024)
Tasks
Summ
Trans
D2T
LF-QA
MathQA
Instret
Story-Gen
Average
GPT-based Metrics
GPT-3.5-turbo (zero-shot)
38.50
10.53
40.20
29.33
66.46
23.20
4.77
34.71
GPT-4 (zero-shot)
36.46
43.87
44.04
48.95
51.71
58.53
32.48
45.15
Reference-basel Metrics
BLEU
11.98
19.73
33.29
11.38
21.12
46.61
-1.17
20.42
ROUGE-2f
14.53
17.83
35.49
16.83
22.12
44.56
2.34
21.96
Instruct.Sore
26.33
50.43
38.54
21.62
-4.15
16.19
16.13
23.58
CPTScore-ref
14.73
24.95
39.42
31.60
18.20
33.14
18.24
25.75
BARTSSor--cu((upp--e)]
13.64
28.53
36.12
29.57
23.35
32.49
26.64
27.19
BARTScore-para (hypo-ref)
17.18
33.72
40.79
28.94
17.27
34.47
17.43
27.11
BERTScore
23.67
42.41
43.75
25.60
45.77
27.95
BLEURT
17.30
48.41
48.76
33.26
3.53
36.46
27.52
30.75
Uniuxa)(summ)
47.52
21.90
38.38
41.83
19.78
16.02
44.46
32.84
COMET-22
33.75
56.35
33.92
35.28
-5.53
46.13
39.20
34.16
Reference-free Metrics
BARTScore-para (sre-hypo)
38.68
9.60
32.26
26.86
-2.70
5.92
20.55
18.74
BARTScore-cnn (sre-hypo)
35.50
12.83
34.33
40.96
1.50
25.43
26,29
Llama-2- 1amma-2-1ue 2-13b-chat-0-shoo
28.53
14.38
29.24
19.91
1.08
21.37
26.78
20.18
COMETKiwi
16.27
48.48
27.90
18.05
-11.48
34.86
18.47
21.79
GPTScore-src
37.41
8.90
28.82
39.48
14.25
26.46
23.91
25.61
TIGERScore-7B (ours)
35.11
41.50
42.39
47.11
21.23
43.57
39.26
38.60
TIGERScore-13B (ours)
36.81
44.99
45.88
46.22
23.32
47.03
46.36
41.52
A (ours - best reference-free)
-2
-3
+12
+5
-9
+14
+13
+16
A (ours best reference-bused)
-9
-11
-3
+00
-2
+7
Table 26: The Spearman correlation results of all the baseline metrics and TIGERSCORE on the evaluation datasets shown in Table 3 For each task, the metric with the highest correlation to average performance is highlighted in bold.
29


Reviews:
[{"review_id": "rXT9eAFymi", "reviewer": "Reviewer_upZt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=== Strengths ===\n- Overall the paper reads good and is easy to follow.\n- The idea of introducing a reference-free metric that also provides explanations (i.e., error analysis) for NLG tasks is wise.\n- The constructed dataset *MetricInstruct* could be a good contribution to the research community if open sourced.\n\n=== Weaknesses ===\n\nThe main concerns are two folds:\n- The experimental setup is unclear and not justified, which could be biased and lead to unfair comparison. \n- Unconvincing human evaluation, failing to reflect the actual performance of *TIGERScore* in generating error analysis \n\nBelow are the itemized questions:\n1. In section 2.2, do we need to assume the availability of natural language instruction $I$ for each\n task? How is it determined in this work?\n2. As presented in Table 3, only a small number of input texts $x$ are sampled from each dataset. How is this selection determined? One potential concern is that human selection could be biased and lead to unfair comparison. Are the comparison results in Table 4 sensitive to the sample selection strategy and/or number of testing samples? Do we have any concerns about increasing the test data size to make it more representative?\n3. In section 4.4, how many human experts are employed to evaluate the generated error analysis? And what is their agreement rate? The statistics in Table 5 suggests the entire human evaluation is based only on one human evaluator, is that correct? If so, then the human evaluation may not be convincing enough.\n4. In section 4.5, using GPT-4 to determine if the generated error analysis of *TIGERScore* contained hallucinations or factual inaccuracies sounds problematic given TIGERScore is trained on GPT-4 generated data -- it's known that GPT-4 as judge can have biased evaluation for texts generated by itself. Therefore, the hallucinations analysis of TIGERScore outputs may not be accurate. Do we have any idea to mitigate this bias?\n\n=== Typos ===\n- Caption of Figure 1: \"... **moel-generated** output ...\" $\\rightarrow$ \"... **model-generated** output ...\"\n- First line of Section 4.3: \"**6 held-in** tasks and **1 held-out** task\" $\\rightarrow$ \"**5 held-in** datasets and **2 held-out** tasks\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work investigates the automatic evaluation of natural language generation (NLG) tasks. It introduces a trainable reference-free evaluator/metric *TIGERScore* that provides not only the final evaluation score but also the error analysis of the generated texts when assessing the performance of a NLG model. Specifically, *TIGERScore* is a LLaMA-2 model tuned on a manually crafted dataset, namely *MetricInstruct*, where the samples are selected from 23 existing datasets across 6 NLG tasks, with additional annotated error analysis generated by prompting GPT-4. Experimental results show the effectiveness of the proposed *TIGERScore* on 5 held-in datasets and 2 held-out datasets. A human evaluation is also conducted to validate the accuracy of error analysis generated by *TIGERScore*.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the concerns in weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ojVawoqe63", "reviewer": "Reviewer_2PF2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThe proposed method is evaluated on 6 tasks across 23 datasets, providing a solid experimental validation.\n\n\n**Weaknesses:**\n\n1. The instruction following mechanism and providing (instruction, input, system output → error analysis) input-output framework as well as the format of error analysis report are highly similar to InstructScore [1], leading to the novelty of this paper limited.\n\n[1] Instructscore: Towards explainable text generation evaluation with automatic feedback. EMNLP 2023.\n\n2. The performance of InstructScore is pretty bad, while in their paper, their performance is good. Did the author train their datasets on all baselines which are trained metrics?  If not, I don't think this is a fair comparison.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes TigerScore, which is a text generation evaluation metric, which follows human instructions and provides the evaluation scores with the error analysis.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors could better elaborate the contributions and novelty in Introduction and related work.\n\n2. Please make sure all the comparisons are fair.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XYnBb9Vu9K", "reviewer": "Reviewer_hH1D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper presents a valuable metric for evaluating model performance among a wide range of text generation tasks. It has the advantage of generality, explainability, and free from reference.\n\n2. The paper is generally well written and easy to follow. The motivation is clear and the background is presented in detail. The dataset and approach are also described clearly.\n\n3. The analysis is thorough. When reading the paper, I have raised a few question. While reading forward, it is good to see most of concerns have been well addressed.\n\nWeaknesses:\n\nMy major concern is that some implementation details are not clarified. \n\n1. In Section 2.2, how the penalty score is determined? It seems that the penalty score is directly given by prompting GPT-4. However, as there would be bias in GPT-4 output, it remains unclear whether the absolute value of the score is convincing or not.\n\n2. This work adopts Kendall’s correlation with human ratings. It is better to justify the choice. Besides, it is interesting to see whether the results may vary with different correlation measures.\n\n3. The key claim may be incorrect. This work positions the scope as the \"evaluation of natural language generation tasks\". However, the actual contribution seems to be something like the evaluation of text generation \"models\" instead of \"tasks\". \n\nMinor Comments:\n\nThere are a few writing mistakes. A more careful proofreading is needed. Here are some examples.\n\n1. The caption of Figure 1: \"moel-generated\" -> \"model-generated\"\n\n2. Section 3.2: a running-on sentence,\"representing 2 main components of MetricInstruct\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a trained metric, TIGERScore, which can be used for evaluating the ability of text generation models. The key characteristics of the proposed metric lie within that TIGERScore is reference-free and explainable. It can not only provide evaluation scores but also pinpoint the mistakes with informative error analysis. The metric is fine-tuned on LLaMA-2, based on meticulously curated instruction-tuning dataset MetricInstruct which covers 6 text generation tasks and 23 text generation datasets. Analysis shows that  TIGERScore correlates well with human ratings on 5 held-in datasets, 2 held-out datasets. It is evident that TIGERScore can achieve the open-source SoTA correlation with human ratings across these datasets and almost approaches GPT-4 evaluator. This paper also provides a thorough discussion on the error cases and hallucination analysis. Overall, this work makes a solid contribution of an effective metric for evaluating text generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A few clarifications. Please see the weaknesses.", "broader_impact_concerns": "The ethics statements are informative.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "p5yz3MVO9C", "reviewer": "Reviewer_hP4V", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The human evaluation of TIGERScore's explanations provides valuable insights into the metric's interpretability and usefulness, validating its effectiveness in generating understandable error analyses.\n2. The paper's claim regarding the potential for TIGERScore to serve as a universal metric for text generation evaluation.\n\nWeaknesses:\nWhile the paper claims near parity with the GPT-4 evaluator, a deeper analysis is warranted, especially in comparison to GPT-based few-shot or fine-tuned models. Table 4 indicates that GPT-based zero-shot evaluation outperforms TIGERScore on certain tasks. This observation prompts further investigation into how TIGERScore fares against GPT-based few-shot or fine-tuned models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces TIGERScore, an innovative metric for evaluating text generation tasks that prioritizes explainability and reference-free assessment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Take comprehensive comparison on GPT-based metrics with the TIGERScore on both accuracy and efficiency.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a trained, reference-free, and explainable metric, TIGERScore, for a wide range of text generation tasks. The reviewers generally feel that the paper is well-written and contains solid and thorough experiments. After author's rebuttal, the concerns raised by the reviewers have been mostly resolved. The remaining concerns are: (1) the human evaluation is relatively small-scale; (2) the similarity to InstructScore; (3) some parts of the writing are unclear (e.g., experiment setup and implementation details). The AE believes that these concerns are quite minor and don't offset the merits of the paper. However, the authors are encouraged to consider reviewers' comments when preparing the final version.
Decision: Accept (poster)