INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback
Authors: Wenda Xu
Danqing Wang
Liangming Pan
Zhenqiao Song
Markus Freitag
William Yang Wang
Lei Li
Venue: EMNLP 2023 Main (2023)
Keywords: Text generation evaluation
Explainable metric

Abstract: Automatically evaluating the quality of language generation is critical. Although recent learned metrics show high correlation with human judgement, these metrics do not provide explicit explanation of their verdict, nor associate the scores with defects in the generated text. To address this limitation, we present INSTRUCTSCORE, a fine-grained explainable evaluation metric for text generation. By harnessing both explicit human instruction and the implicit knowledge of GPT-4, we fine-tune a text evaluation metric based on LLaMA, producing both a score for generated text and a human readable diagnostic report. We evaluate INSTRUCTSCORE on a variety of generation tasks, including translation, captioning, data-to-text, and commonsense generation. Experiments show that our 7B model surpasses all other unsupervised metrics, including those based on 175B GPT-3 and GPT-4. Surprisingly, our INSTRUCTSCORE, even without direct supervision from human-rated data, achieves performance levels on par with state-of-the-art metrics like COMET22, which were fine-tuned on human ratings.

INSTRUCTSCOR::
Explainable Text Generation Evaluation with Fine-grained Feedback
Wenda Xu', Danqing Wang1, Liangming Pan", Z Song", Markus Freitag'. William Yang Wang Lei Li
Tnniversity of California, Santa Barbara, 'Google Research, *Carnegie Mellon University {wendaxu, dangingwang, liangmingpan, zhenqiao, wi william) @ec.iammes.uusb........ udu leili@cs. cmu. edu
frei taggoogog com
Abstract
Promat: You are evaluating model output based on reference.
BER TScoee 0.925
Automatically evaluating the quality of lan- guage generation critical. Although recent learned metries show high correlation with hu- man judgement, these metrics do not provide explicit explanation of their verdict, ciate the scores with defects in the generated text. To address this limitation, we present IN- STRUCTSCORE, fine-grained explainable evaluation metric for text generation. By har- nessing both explicit human instruction and the implicit knowledge of GPT-4, we fine-tune text evaluation metric based on LLaMA, pro- ducing both a score for generated text and human readable diagnostic report. We eval- uate INSTRUCTScORE on variety of gener- ation tasks, including translation, captioning, data-to-text text, and commonsense generation. Ex- periments show that our 7B model surpasses all other unsupervised metries, including those based on 175B GPT-3 and GPT-4. Surprisingly, our INSTRUCTSCORE, even without direct su- pervision from human-rated data, achieves per- formance levels on par with state-of-the- metrics like COMET22, which were fine-tuned on human ratings.
Reference: Normalyy the administration office downstairs would call there's delivory.
COMET: 0,711
BLEURT:
asso-
Output: Usually when there takesway, the managoment office downstairs will call.
SEScore2: -5.43
Ion's + understand what those. scores meon
save you. am intelligent InstruciSSoore.
Human
Error Location
Exploration
InstructScore
Type
Ennor Type: Incorrect transiation has stylistic problems Sevenity Majer
Error Location: Usually when there is taksowey.
Explanation: The trunslation uses on awlword phrosing "Usually when there takeaway. instead of "Usuolly, when there's s delivery.'
Score: -
Figure INSTRUCTSCORE generates a comprehensive error diagnostic report for text generation tasks, including error type, location, severity label, and explanation. Based on this report, INSTRUCTSCORF counts the number of major errors (each worth -5) and minor error (cach worth -1), ultimately assigning a final score.
SE.Scor (Xu et al., 2022b,a) show higher corre- lation with humans on text generation tasks. How- ever, all these metrics produce a single numerical score. These learned metrics lack interpretation of predictions nor link the scores with individual defects in the candidate text.
1 Introduction
Although large language models (LLMs) have led to significant progress in various natural language tasks (Brown et al., 2020; Ouyang al., 2022; Touvron et al., 2023), remains challenge au- tomatically evaluate the quality of text generation across versatile tasks. Traditional word overlap metrics, such as n-gram matching, BLEU (Pap- ineni et al., 2002), and chrF (Popović, 2015), along with distance- -baend metrics like TER (Snover et al., 2006) do not best align with human experts' judge- ments (Freitag et al., 2021a). They primarily fo- cus on surface form differences between reference and candidate texts (Freitag et al., 2020). On the other hand, recent learned metrics such as BERTScore (Zhang et al., 2019), BLEURT (Sel- lam et al., 2020a), COMET (Rei et al., 2022) and
How can we devise a fine line-graieed explanation- based text generation metric capable of pinpointing concrete error locations, identifying error types, assigning severity labels, and justifying the fi- nal score-at all simultaneously without relying on human-annotate data. In this paper, we propose INSTRUCTSCORE, method to learn explain- able text generation metric without using human annotated ratings. InstructScore provides both a numerical score and a natural language error ex- planation. To this end, we first extract latent eval- uation knowledge from an instruction-followin

---

model, such as GPT-4 (OpenAl, 2023), con- struct a synthetic dataset with a predetermined ex- planation structure. Next, we determine a range explanation failure modes and devise automated feedback to meta-evaluate error explanations. Fi- nally, we further fine-tune INSTRUCTSCORE model on self- seff-generated outputs that optimize feedback scores, resulting in diagnostic reports that are better aligned with humans.
either pre-trained natural language inference or multilingual masked prediction model to attach er- ror severity score. Supervised metrics can arguably attain higher correlations with human judgments (Freitag et al., 2021b, 2022), while unsupervised metrics, such as SEScore (Xu et al., 2022b) and BERTScore (Zhang et al., 2019), exhibit greater levels of generalization. However, none of these approaches offer explanation for the resulting scores, rendering the deecision-maing processes obscure and less trustworthy. In this paper, we generate a diagnostic report to provide detailed ex- planations to support metric's final decisions.
We have conduct experiments on a variety of text generation tasks: machine translation, table- to-text, image captioning, commonsense genter- ation, and keyword-to-dialogue generation. Our experimental findings show that the unsupervised INstRuCTScore outperforms prior strong base- lines on all these tasks. It achieves the best results for the unseen keyword-to-diillgge generation task. Surprisingly, INSTRUCTSCORE surpasses the su- pervised BLEURT in 6 out of 9 directions and closely matches statc-of-the-art COMET22 in ma- chine translation. Furthermore, we identify range of failure modes and design automatic pipeline to pinpoint explanation failures. Our refinement step improves human score by 13.7%, leading more accurate alignment with human judgment. Our INSTRUCTSCORE enjoys the following advantages: (i) Compact yet competitive: IN- STRucTSCORE's 7B version displays strong perfor- mance compared to metrics based on closed-source 175B LLMs. (ii) Explainable: INSTRUCTSCORE provides natural language explanations justify numerical scores. (iii) Generalizable: The unsu- pervised training pipeline does not require human- annotations, making it easily adaptable to different domains and tasks.
Explainable Evaluation Metric. Recent de- mand for explainability in evaluation metrics has grown significantly. Freitag et al. (2021a) introduce a multi-dimensional human evaluation (MQM) framework for machine translation, while Leiter et al. (2022) investigates key characteristics of ex- plainable metries. Several metrics derived from those frameworks enhance explainability by differ- entiating error severity (Lu et al., 2022; Xu et al.. 2022b,a; Perrella al., 2022). Other efforts fo- cus on explanatory aspects of text generation met- rics, like error locations (Zerva al., 2022) and muuti-ddimensionl assessment (Zhong et al., 2022). Despite progress, explamations remain unclear. Re- searchers also explore LLMs' potential in evalua- tion, demonstrated by Fu et al. (2023), but suffers from lack of explanation. Koemi and Federmann (2023) and Liu et al. (2023) find large models like GPT-3.5 on system-level can correlate humans and generate rationales. However, these generated rationales are free-form and may not necessarily align with human judgements (Zheng et al., 2023). In our work, we explicitly refine INSTRUCTSCORF to produce explanations that align with human.
2
Related Work
Learned Evaluation Metrics Supervised metrics optimize performance by directly fine-tuning hu- man rating data, such as COMET (Rei et al., 2020) and BLEURT (Sellam et al., 2020a), as shown by Rei et al. (2020) and Sellam et al. (2020a). How- ever, human rating data is often unavailable. Unsu- pervised metrics use different learning objectives or heuristics on embeddings, such BERT for greedy matching and coverage scoring (Zhang et al., 2019), requence-to-sequenee models for probability es- timation (Thompson and Post, 2020; Yuan al., 2021). SEScore (Xu et al., 2022b) and SE.core2 (Xu et al., 2022a) train a regression model by syn- thesizing human-like -lke errors from raw text and using
3 Problem Definition
Our goal is to learn an explainable metric model that not only predicts the quality score of candidate text comparing to a reference but also generates a diagnostic report in natural language. Specifically, INsTrucTScORE assesses the quality of x regard- ing reference r by generating an informative diag- nostic report, which includes the details about error location 1, error type I, severity level se, and expla- nation e that associated with the identified error. INSTRUCTSCORF consists of score predictor and an explanation generator (Exp-Ged (xxp-Generator which learns a function f: (x,y) + {[I,t, se, e),) )

---

Guided
Feed real model generated output reference
Sample
Q1: Is it consistent with the given error type.
erro--and-explanatoon synthesis
diagnostic
Q2:
phase.
outputs
Q3: Is incorrect phrase semutticaly different from
correct phrase?
Auto-identifying Failure Modes
Pass explanations with query GPT4
GPT4
Finetune
Refinement with Meta-Feedback
Example seed
Synthetic data with generated error explanation
ii. Automatic/Human Feedback
Good: The translation uses an awkward phrasing "Usually when there is takeaway," " instead of "Usually, when there's a delivery." Bad: The translation uses "there is takcaway" instcad of there is takeaway," which alters the meaning of the sentence.
For Diugnostic Al: Yes; A2: [Usually when there takeuwayy, Usually, when there's delivery A3: Yes For Diagnostic output 2 (Alignment Score: 3)
(Alignment Score:
Yes, A2: Ithere takeaway, A3: No
lakesway')
Figure 2: Our INSTRUCTSCORE pipeline consists of three components: First, we construct synthetic data from GPT-4 and use it to fine-tune a 7B LLAMA model. Second, we sample from real-world machine-generatee distribution to trigger INSTRUCTSCORe's failure modes. We query GPT-4 on each failure mode and gather automatic feedback. Third, we select explanations that most aligned with human to further fine-tune I.I aMA model. Step 2 and can he repeated to iteratively refine the model output.
with n number of errors. However, such human annotated mapping data for most text generation tasks scarce due to limited human resources and high annotation costs. To this end, we propose a data construction method to automatically generate high-quality pseudo data to learn f.
vious practices (Freitag al., 2021a), our metric identifies up to five errors per sentence.
4.1 Learning with Guided Error and Explanation Synthesis
We leverage GPT-4 to extract representative ex- plainable knowledge that can greatly contribute to the subsequent Exp-Generator learning process. Specifically, we collected raw sentences in the target language from diverse domains and topics via GPT-4 (Details are included in Section 5.1 and Appendix A), resulting in data across diverse tasks. This corpus is used as the starting point to inject errors. Then, we prompt GPT-4 to synthesize des- ignated generation errors, as shown Table 1.
4 The INSTRUCTSCORE Approach
INSTRUCTSCORE assesses the quality of generated texts based on an explainable diagnostic report. Building upon this report, INSTRUCTŚCORE pro- vides an intuitive way to comprehend a model's generation capability, resulting in easier compar- ison among different models. In particular, begin by extracting concise yet representative ex- plainable knowledge from large-scale instruction- following model, which is then utilized to train our Exp-Generator. After carefully analyzing the diag- nostic reports produced by our Exp-Generator, we summarize common failure modes in diagnostic re- port and ask GPT-4 to identify them. Then we trans- form the GPT-4's feedback into alignment scores using our predefined criteria. Finally, diagnostic reports that have the highest alignment scores, and further finetune our Exp- Exp-Generator on those self-refined outputs. The overall framework is illustrated Figure 2.
For cach text, we specify the number of errors, error types, and severity labels, and ask GPT-4 to generate a candidate output with the specified error descriptions and 2) an explanation for this error an- notation. If an evaluation task is multi-dimenniinal. error types will be separately assigned each di- mension (An example is included in the Appendix). Benefiting from the large-scale pre-training pro- cess, GPT-4 is able to generate diverse errors and meet the requirements with specified instructions. To avoid the model's over-reliance on the lexical and structural similarities between the candidate and raw text, we request GPT-4 to rephrase the raw text sentence to construct pscudo-referenee tence. By specifying error type I, severity label se, and text, GPT-4 is able generate synthetic error sentence with annotated error location / and a pseudo reference y with explanation e. There- fore, we can construct synthetic data that reflects
select
The quality score s for each candidate y is de- termined based on the number of errors and their severity labels in the diagnostic report. Minor er- rors are given score of and major errors are given a score of -5. These penalties for errors are weighted to calculate the final score. Similar to pre-

---

Prompt for GPT4:
where 0 is the trainable parameter of the Exp- Generator.
The correct translation is. "The art writing for edu- cational publications involves striking delicate bal- ance between providing enough detail be useful and overwhelming the reader with too much informa- tion." " Please give minor errors and 2 major errors. For each error, give me an error type, a major/minor label, an error location, and an explanation. The format is Paraphrase correct translation: Incorrect Translation:
4.2 Auto-Identifying Failure Modes of Metric Output
The diagnostic report plays an important role in text quality explanation. However, the above trained model is not guaranteed to produce sensible expla- nations those incorrect explanations referred to failure modes. We categorize failure modes into global and local levels. A global failure in- validates all four fields: error type, error location, major/minor and explanation. A local failure only affects a specific field, like error type.
Error type 1: Translation includes information not present in the correct translation
Mujor/minor: Major
Error location I:
Explanation for error 1:
Error type 2: Prohlems with grammar, other than orthography
Major/minor: Major
Error location 2:
Explanation for error 2:
In Table 2, we define six scenarios M1-M6 for local failures and four scenarios G1-G4 for global failures. We demonstrate one failure mode M4 in Table Concrete examples for cach failure mode are included in the Appendix Table 13, 14, 15, 16, 17, 18, 19, 20, 21, 22. A special case is when the method outputs an annotation containing error. In this situation, we need to verify if this is accurate (A concrete example can be found in Appendix Table 35). If errors are present, the diagnostic report is incorrect.
GPT4 Oulput:
Error type 1: Translation includes information not present in the correct translation
Mujor/minor: Major
Error location I: "..iiiiin enough examples..." Explanation for error I: The correct translation talks about providing enough dotail, not specific examples, which changes the meaning of the sentence.
Error type 2: Problems with grammar, other than orthog- raphy
Major/minor: Major
Error location 2: "...making the reader bored with too many details."
Explanation for error 2: The correct translation mentions overwhelming the reader with too much information, not making them bored with too many details. This change in phrasing alters the meaning of the sentence.
Ideally, a human annotator can provide the most accurate judgment for detecting each failure mode. However, obtaining annotations from humans for every instance of a diagnostic report infeasible. As an alternative, we leverage GPT-4's capabilities in information extraction, parsing, and semantic understanding (OpenAI, 2023) to convert complex requirement queries into simple Yes/No questions. Specifically, we prompt GPT-4 to parse the ex- planation into incorrect and correct phrase pairs and extract the error span from the error location. To address hallucinations from error location (M3) and explanation (M4), we verify our parsed error span is present in the candidate sentence. If one error annotation contains multiple incorrect-correct phrase pairs, it indicates multiple errors in one error location (G4). To address G1, we first check if the incorrect phrase indeed an error. Additionally, we verify if the revision suggestion is in the out- put. For the remaining M and G aspects, we design specific prompt queries for the GPT-4. A detailed example of the prompt query for checking M1-M6 and G -G4 can be found in Table 4.
Table 1: This prompt was used to generate synthetic data to fine-tune the I.LAMA model, specifically GPT-4. The model was provided with the number of errors in sentence. For each error annotation, we pre-define the error type and severity label. GPT-4 was then required to identify the error locations and provide explanations for each error annotation. The texts highlighted are variables for each synthesized generation. The complete example be found Appendix Table 29.
the relationship between (æ, v) and (t, I, se, e). We train Exp-Generator by utilizing the con- structed data. Particularly, we use LLaMA as the initialization of the Exp-Generator since it open- sourced and performs well in both understanding and generation tasks. Then we train our Exp- Generator by taking the pseudo reference and candidate as the input, and the diagnostic report including the corresponding error type t, error lo- cation 1, severity label se and the explanation as the output. A concrete example can be found in Figure 2. Accordingly, our training objective can be defined as follows:
A After obtaining local and global failure modes from GPT-4's feedback, we can convert those feed- back into alignment scores. We assign a binary
C(t, I, se, e, y) = -log P(t, se, ely. 0; (1)

---

Fields
Explanation Failure Mode
Description
Local Failure Mode
Error Type
Inconsistency to explanation MI: Error type descriptions are not consistent with explanation
Inconsistency to explanation M2: Error locations are not consistent with the explanation
Error Location
Error location hallucination M3: Error locations are not referred in the output text
Major/Minor Major/Minor disagreement M5: Major and minor labels do correspond the correct severity levels
Error location hallucination M4: Error locations can not refer to the output text
Explanation
Explanation failure
M6: The explanation is wrong. However, error specified location does exist
Global Failure Mode
False negative error
GI: Firor described the explanation
error
Repetition Phrase misalignment
G2: One error is mentioned more than once among explanations
All 4 Fields
G3: Incorrect phrase and correct phrase are correctly aligned
Mention multiple errors
G4: One error span mentions multiple errors
Table 2: This table shows common failure modes of the explanation output of first step Exp-Generanl Exp-Generator (Fine-tuned LLaMA synthetic data without refinement). Local errors are field-speciee specific, which only correspond to the error of the errors can affect all four fields, such as error type, error location, major/minor, and explanation. The observation of the failures modes at first step Generator is the main motivation for to perform refinement with automatic feedback (See Section 4.3)
local field. Global
INSTRUCTŚCORE Input:
Prompt: Ref: ..... entered the revolutionary base area of south Jiangxi. Out: .... entered the old revolutionary district of southern Jiangxi.
Reference: Activities include the "Cool Maker Loves Family" carnival children's future learning, the first children's visual art festival, the "Book Carnival" event and the "Whole Family Loves Reading" Picture Book Reading activity.
Error location 1: "old revolutionary district"
Error type 1: Terminology is non-standard
Explanation 1: The correct term should "new revolu- tionary base area". ......
Output: Activities include the "love family and create cool" children's 's future learning carniaal, the first chil- dren's visual arts festival, the "book carnival" activity, the "family love reading" picture book series activity.
Q1: For each error location, extract the incorrect error location.
Q2: Parse the explanation into either one of the four forms: fincorrect phrase, correct phrasel l....el..... Q3: If A2 is "incorrect phrase to correct phrase". is A2 correct alignment for reference and output?
INSTRUCTSCORE Output:
Your Translation contains errors:
Error type 5: Incorrect translation includes information not present in the correct translation
Q4: According to the sentence context, is it no-error or minor-error or major-error?
Major/minor: Major
Q5: Is the explanation consistent with the given error type?
Error location "whole family loves reading" Explanation for error The incorrect translation adds the phrase "whole family loves reading." which is not present in the correct translation and changes the focus from "families" to "whole families."
Q6: Is error location in explanation?
Q7: Do two error locations mention the same location?
Table 4: Prompt for GPT4 feedback: We asked following questions to determine the correctness and consistency of the explanation. This is simplificd prompt of how we ohtained GPT-4 feedback for Table 2 (Complete examples can be found in Appenfix Table 34 and 35).
Table 3:
This is the case where error locations in explanation can not refer to the output text (M4). "whole family loves reading" is never mentioned in the output text. Instead, it was mentioned in the reference text. The complete example can be found in Appendix Table 16.
corresponding alignment score will be 1 11/16.
score in each field of the diagnostic output. If one local error is observed, the corresponding field will receive score of 0. If one global error is observed, all four fields corresponding to that annotation will receive Os. Each diagnostic report has a score of floorrectfiedds yiclding a alignment score between /iodal
4.3 Refinement with Meta-Feed
We then leverage the well-learned Exp-Generator to produce high-quality pseudo training data to iteratively refine the model performance, which can further benefit final quality score calcula- tion. Specifically, we use hypothsis ht with ref- erence k; as the model input and employ sampling strategies to generate diverse diagnostic reports. Due to discrepancies between synthetic data and real world hypothesis-referrnce pairs {a, y and
0 and 1. For example, one candidate sentence can be given four error annotations. For each error an- notation, there are four fields: error type, location, major/minor, and explanation, 16 fields in total. one global and one failure modes are observed, the

---

{h (Sellam et al., 2020a; Xu al., 2022b), we anticipate real world model hypothesis can trigger diverse failure modes from Table For each input pair (hs, ks), we use top p samping to sample n possible diagnostic outputs, deno- tated as for. .2..... on) Based on the feedback scores from GPT-4, we keep the diagnostic out- put that optimizes the alignment score, Oaligned (faligmeds faligned: SCaligned, Caligned), and further fine-tune our Exp- -xx-Geerrraor. This automatic cri- tiques and self- self-training pipeline can further encour- age our Exp- Exp-Generator to generate more accurate diagnostic reports. Based on the human evaluation, the final quality score can reduce failure modes and align outputs with humans better.
5.1 Experiment Setup
Baseline and Benchmark We tested our IN- STRUCTSCORE at 1) WMT22 shared metric task, in two target languages: English and German, WMT22 uses MQM-based human evaluation procedure (Freitag al., 202 WMT22 has 14 English, 16 German participating Machine Trans- lation systems, with 26,250 and 21,040 human- annotated outputs respectively; 2) WebNLG20: The input of the task contains Wikipedia triples, and the output is natural language text. This benchmark contains 16 participating WebNLG systems with 2832 human-annotated outputs; 3) Flicker3K-CF: This benchmark contains 145K bi- nary quality judgment gathered from CrowdFlower over 48K (image, caption) pairs with IK unique images; 4) Commongen: This benchmark con- tains 2796 model outputs from 6 participating sys- tems. All human annotations are done in pairwise rankings for the same source input. Therefore, we compute ranking accuracy estimate the metric's performance: 5) BAGEL: The input of this task contains keywords and the output is fluent human conversation. This benchmark contains 202 model outputs. We included top- oopperrforming baseline met- rics which joined each challenge, including n-gram based metrics: BLEU (Papineni et al., 2002), chrF (Popović, 5), METEOR (Banerjee and Lavie, 2005) and CIDEr (Vedantam et al., 2015); Un- supervised learned metries: PRISM (Thompson and Post, 2020), BARTScore (Yuan al., 2021).
[lallggned F, v) log P(oatigned y, 2; (2)
5 Experiments
In the experiment section, we aim to answer the following research questions: 1) What is the per- formance across various tasks within the English language? 2) What is the performance across dif- ferent domains within the same task? 3) What is the performance across different evaluation dimen- sions? 4) What the performance at unseen tasks? 5) Given that LLaMA is predominantly trained in English texts, can it effectively evaluate generations in other languages? 6) Can we align the diagnostic report with human expectations without requiring extensive human efforts?
BERTScore (Zhang et al., 2019), and SEScore2
(Xu et al., 2022a); LLM-based metrics: GPT3- Dav3 and GPT4(Koemi and Federmann, 2023);
To answer Q1, we tested INSTRUCTSCORE at various tasks, including WMT22 (Machine Trans- lation) (Freitag et al., 2022), WebNLG (Table-to- text) (Castro Ferreira et al., 2020), Flicker3k (Cap- tioning) (Hodosh et al., 2013), BAGEL (Keyword- to-text) (Mairesse al., 2010) and Commongen (Commonsense text generation) (Lin et al., 2020). To address Q2, we examed our method at four di- verse domains: News, Conversation, Social, and E- E-Commerce at WMT22. For Q3, we evaluated our method at five evaluation dimensions at WebNLG For Q4, we evaluated INSTRUCTSCORE at BAGEL benchmark, a task and evaluation dimensions that are unseen in the synthetic data. For Q5, eval- uated our approach to English-to-German transla- tions in order to investigate its multilingual evalua- tion capabilities. Lastly, we demonstrate our metric with automatic critique and refinement can achieve higher human ratings regarding failure modes.
Supervised learned metrics: BLEURT-20 (Sellam et al., 2020a), MaTESe (Perrella et al., 2022), UnITE (Wan et al., 2022) and MetricX XXL.
Implementation We utilize GPT-4 as our im- plicit evaluation knowledge base and LLaMA-7B as our trained initialization'. To ensure coverage of diverse text domains, we separately collect a dataset of 10k raw sentences from 100 different do- mains using GPT-4 for both English and German. Details on the construction of this dataset are pro- vided in Appendix Sec. C. To adapt to specific task domains, we separately gathered 10k sentences for the training datasets of WebNLG17 (Gardent et al..
We include InstructScore based on LLaMA2-7B (Touvron et al., 2023) results in the Appendix Table 12. We demon- strate that varying pertaining initialization can lead to different results at diverse NLG tasks.

---

Seen Tasks
Unseen Task BAGEL TIP
WMT22(Zh +En)
WebNLG20 Tpp
Flicker3k T/p
Commongen Acc
Rank Avg
7/p
Supervised
MATESE
38.9
Metric XXI.
42.7 58.11
COMET-22
42.8/ 58.5
BLEUUR-200
36.1
40.2 63.5
24.3 15.1
39.5
22.9 32.3
2.8
BLEU
14.5 / 17.5
20.1/
13.8/21.6
26.8
10.9 / 16.8
10.2
Without Supervision
ChrF
15.4/ 14.7
20.6/40.0
14.3
33.0
10.8 16.8
9.0
METEOR
16.5 18.8
25.6/36.3
13.4/23.1
33,9
12.6 14.5
9.4
CIDEr
18.0
26.1/31.8
29.8
32.6
13.7
7.8
BERTScore
37.6/3..6
32.8/50.4
17.4/24.6
38.8
/
5.6
BARTScore
22.2/24.9
33.1/56.8
17.9/22.2
37.0
20.3/
6.2
PRISM
25.0/27.9
37.6/
16.1 /23.8
38,0
21.7/30.7
5.1
GEMBA-GPT4
38.2/37.4
SEScore2
33.0/46.4
36.8/48.4
22.2
34.4
23.3
4.9
INSTRUCTSCORE
40.3/ 51.9
39.5/59.0
30.1 / 34.6
58.2
25.6 34.2
2.0
Table 5: We applied segment-level Kendall and Pearson correlation on WMT22, WebNLG20, Flicker3k, and BAGEL, and ranking accuracy for Commongen. INstRUCTScoRe significantly outperforms all unsupervised metrics in 8/9 directions using William's pairwise significance test (p 0.05). The top supervised and unsupervised metrics bolded. Pearson and Kendall correlations ranked per ranking details cach correlation and task.
overall performance being the average rank across tasks, Appendix Table contains
2017), CoCo Captioning (Chen et al., 2015), and CommonGen (Liu et al., 2022). We apply the re- spective prompts defined in Appendix Tables 29, 30, 31, and 32 to generate synthetic data.
alignment of our model before and after refinement. In particular, the human raters will estimate a binary score based on M1 to M6 and G1 to G4 criteria for each field in the diagnostic report. The total score of diagnostic report is calculated #correct /ielde The final score is averaged among #otal fields raters. Details of human evaluation procedures included in the Appendix Sec and Table 8.
We define four evaluation scenarios: 1) eval- uation with reference only; 2) evaluation with reference and additional data; 3) evaluation with reference where the source has different modali- ties; 4) evaluation with reference and world knowl- edge. For each scenario, obtain 10k candidate- reference pairs input and structured diagnos- tic reports output. We train a separate check- point for each evaluation scenario, resulting in four checkpoints in total. All models are fine-tuned with language modeling loss with 10k synthetic data. Each model is trained for three epochs, with learning rate, batch size, and weight decay of 2e-5, 128, and O, respectively. During the evaluation of each model, set the temperature 0 for greedy decoding. Details of our automatic critique and self-training are included the Appendix See
chuF
0.4
MATESE
BERTS
BLEURT-P 20
UniTE
COMET-22
OENBA-OFT
News
Eoom
Chat
Social
Figure Segment-level Kendall (r) correlation Zt-En for
differentt domains of WMT22. We connect points highlight InstructScore with other top performing metrics in the figure.
Meta-evaluation We assess the performance of INSTRUCTSCORE using Segment-level Kendall and Pearson correlations between human and met- ric output. Kendall Tau-b might favor tie pairs, possibly giving unfair advantage certain sys- tems (Deutsch al., 2023). Pearson, on the other hand, measures linear association. By reporting both complementary results, we can comprehen- sively understand the metric's performance. We employed three human annotators to assess the
5.2 Main Results
Robust Performance across Tasks We assess the primary performance of INSTRUCTSCORE for five diverse NL.G tasks. As shown in Table 5, IN- STRUCTSCORE significantly outperforms all other unsupervised metrics in 8 out of directions, achieving the best overall ranking. All improve-
Fach human rater paid at $16.0 0 per hour, which is above the minimum wage of the local region.

---

ments are statistically significant by William's pair- wise significant test (Graham and Baldwin, 2014) w Surprisingly, INSTRUCTŠCORE even outperforms prior supervised learned metrics that trained over direct assessment data (DA), leading BLEURT20 in 6 out of 9 directions. Compared to GPT4 baseline, INSTRUCTSCORF outperforms GEMBA-GPT4 with 0.021 in Kendall and 0.145 in Pearson correlation. The larger gap in Pearson correlation can be explained by a large set of ties that GEMBA-GPT4 producing. This will lead to false positive in Kendall correlation. Lastly, we demonstrate that INSTRUCTSCORE can achieve close performance to the supervised learned met- rics, MATESE, COMET22 and Metric XXL. that have trained over comprehensive human rating data (DA and MQM), with average 0.012 gap in Kendall correlation and 0.045 in Pearson correlation.
that INSTRUCTSCORE outperforms all unsuper- vised learned metrics across five different dimen- sions. Compared to BLEURT, which trained over WebNLG human rating data, INSTRUCTSCORE outperforms best performed BLEURT in three out of five evaluation dimensions. This signifies that INSTRUCTSCORE can be extended to assign single quality score but extending into multi- dimensional NLG evaluation.
Generalization over Unseen Task Since each NLG task has distinct evaluation criteria, one nat- ural question to ask: Is INSTRUCTSCORE gener- alizable to the task with unseen data format and evaluation criteria? To verify this question, we use BAGEL benchmark unseen task since it contains distinct data formats from our training data and con- tains distinct criteria. From Table and Figure 5, we demonstrate that despite never seen the evalau- tion criteria of keywords to text, INSTRUCTSCORF achieves the higher Kendall and Pearson correla- tion compared to BLEURT as well as two of three unseen evaluation dimensions.
Robust Performance across Domains In Fig- ure 3, we further evaluate the performance of IN- STRUCTSCORE across different domains. From Kendall correlation analysis. INSTRUCTSCORE surpasses all unsupervised metrics in four do- mains except GPT-3.5 and GPT-4 baselines at Chat. It achieves comparable performance Ecom- merce, Chat, and Social domains when compared to the leading supervised metries COMET22 and Metric-XXI. However, its performance is notice- ably worse in the News domain compared to SOTA COMET22 and Metric-XXI. This gap can be pri- marily attributed to their supervised data distribu- tion at News domain DA and MQM data.
0.3
BLEU
chrF
METEOR
0.2
CIDEr
BERTScore
0.1
BARTScore
BLEURT
Informativnness
Naturaheess
Quality
InstructScore
Figure 5: Segment-level Kendall Correlation on different di- mensions of BAGEL dialogue generation. We connect points highlight INs INstruCtScORe with two other top performing metries in the figure.
0.4
BLEURT
BLEU
ChiF
METEOR
CIDEr
5.3 Quantitative Analysis
BERTScam
0.2
BARTScom
Performance at Non-English Language. In Fig- ure 6, INstRUCTScore outperforms most unsu- pervised metrics, but not 175B GPT3.5 models at WMT22 English-to-GGerann or supervised coun- terparts like COMET22 and BLEURT20. We hy- pothesize that multiple factors contribute to the ob- served LLaMA performance on non-English texts: (1) limited pretraining data, resulting in weaker pre- trained knowledge for other languages, and (2) the task setup requiring alignment between languages due mixed code text generation (see Appendix Sec D and Table 24). Future research could explore multilingual alignment warmup methods before training on evaluation data.
PRISM
8F3mmm2
0.1
Cor
Cov
Flu
Rel
Str
InstuusScora
Figure 4: Segment-level Kendall Correlation on WebNLG Data-to-Text generation. Cor, Cov, Flu, Rel, and Str repre- sent Correctness, Coverage, Fluency, Relevance, and Text Structure respectively. We connect points to highlight IN- STRECTSCORE with two other top performing metries in the figure.
Robust Performance across Dimensions Un- like most of metrics which only output single score, INSTRUCTSCORE can output a score in each evaluation dimension. In Figure 4, we demonstrate

---

WMT22 En-De Segment-level Kendall Correlation
Automatic critique and Self-training can im- prove precision and recall of INSTRUCTSCORE We conduct a human evaluation of the quality of INStRUCTScORE) annotations (with three annota- tors). We calculated the precision and recall of the annotations before and after refinement. Precision # of correctly annotated error fields / # of IN- STRUCTSCORE'S labeled error fields. Recall # of correctly annotated error fields / (# of correctly annotated error fields+the number of error fields that INSTRUCTSCORF missed). Error fields con- sist of error type, severity label, error location, and explanations. In Table 7, INSTRUCTSCORE can achieve 77.8% precision and 82.4% recall before refinement. After refinement, INSTRUCTSCORF can improve precision and recall by 1 11.6% and 3.2%, respectively. In addition, study the field of explanation in particular, the refinement step can improve the precision of explanation from 75.6% % and improve the recall of explanation from 81.9% .9% to 85.0%. Overall, our finding suggests that 89.4% % of InstructScore's output after re- finement is correct and it can identify 85.6 % of errors produced by the translation system.
64
50
acarBoon
COMPT oz
Segment-level Kendall correlation of IN- STRUCTSCORE WMT22 English-to- Go-Gaan Purple in- dicates unsupervised learned metries, while Blue indicates supervised learned metrics.
Figure
Automatic critique and Self-training can im- prove human Alignment We conduct human evaluation to assess metric's alignment before and after self-training. From Figure 7, we demon- strates that INSTRUCIScORE after automatic- critique and refinement can significantly reduce most of global and local failure modes, In par- ticular, all global failures have more than 50% decreases in occurrences. This signifies that IN- STRuctScORe has improved over its phrase align- ment, error identification and error formats. More- over, consistency between four fields have all im- proved, demonstrated by improvements over all M occurrences. We observed that M6 has slight in- crease. This due to some of the conversions from global failures into the local failures. Table 6, we demonstrated that INSTRUCTSCORE can achieve 0.106 absolute human score gains with on par per- formance at Kendall and Pearson correlation. The detailed human evaluation procedure, individual rater scores and Appendix Sec F, 10 and 23.
Human Evaluation PAR
RAB
Pgep
Rgap
refinement
0.778
0.824
0.756
0.819
After refinement
0.894
0.856
0.861
0.850
Table report precision and recall hefore and after re- finement on annotation fields (error type, location, severity label and explanation) by INSTRUCTŠCORE, annotated Pau Rau. addition, we include precision and recall on the explanation field only, annotated by PEup and Rgup-
study are included in the
6 Conclusion
# Failure Modes per Sentence
Before Refinement
After Refinement
In this paper, we present novel framework for explainable text generation evaluation, addressing the existing black-box limitations associated with learned metrics. We define set of failure modes regularize the explanations. We empirically demon- strate that INRR can be generalized to different domains, tasks and evaluation dimen- sions, achieving the best ranking compared to other general text generation metrics. Lastly, our refine- ment from automatic feedback can further improve human alignment score, precision, and recall, by 13.7%. 11.6%, and 3.2%. respectively, leading more accurate alignment with human requirements. We released the INStRUCTSCORE model for public use and open-source the data and codes.
0.25
0.20
0.15
0.00
M2
M3
M4
M5
M6
G4
Figure 7: Failure mode occurrence per sentence hefore and after refinement. Results are averaged by three human raters,
INsTRUCTSCORE
Kendall
Pearson
Human
Fine-tune
0.404
0.515
0.773
Pene-tunn+Reeinemeee
0.403
0.519
0.879
Table 6: We report Segment- Kendall, Pearson correlation and human score before and after refinement.

---

Limitations
The quality improvements that may stem from using InstructScore could be instrumental in di- verse applications such as translation services, chat- bots, and content creation. Nonetheless, it is vital to monitor these advancements to ensure that they do not inadvertently suppress linguistic diversity. Additionally, the biases that may have been passed on to InstructScore from pre-existing models like GPT4 should be critically examined, and efforts must be made to alleviate biases that could impact language, dialect, or cultural representation.
While we have not yet been able to test IN- STRUCTScORE in a multilingual setting due to the limited availability of human annotation and significant label costs, it has shown promising re- sults in leveraging smaller synthetic and feedback data to fine-tune and refine its performance. In fact, INSTRUCTSCORE has demonstrated superior performance compared to unsupervised baselines, such as BERTScore, BARTScore, and PRISM in high-resource non-English language, such as German. Going forward, we aim to assess IN- STRUCTSCORE'S multilingual evaluation capabil- ities across high, medium, and low-resource lan- guages. As our instructions are in English and the evaluation target is in other language, we plan to en- hance INSTRUCTSCORF'S mixed code generation and multilingual word alignment abilities by ex- ploring more pretraining and warm-up techniques. Although our current computing resources re- strict our ability to confirm the impacts of model size performance, future research should investi- gate model size utilizing scaling law (Kaplan et al., 2020) to uncover potential improvements in failure modes related to larger model sizes.
Finally, the impact of InstructScore on educa- tional and professional writing practices should not be overlooked. As writers and educators might adapt their styles based on algorithmic evaluations, it is essential to balance the quest for higher scores with the preservation of human creativity and the diversity of expression. InstructScore has the po- tential be a powerful tool in the evaluation of text generation, but is imperative that ethical con- siderations surrounding transparency, accessibility, bias, and societal impact are vigilantly monitored and addressed.
We hired three human raters to annotate IN- STRUCTSCORL's diagnostic reports. We ran- domly sampled 100 candidate-reference pairs from WMT22 Chinese-to-Ennlissh direction. All eval- uated data does not contain sensitive or explicit languages. There is no risk of exposing raters' identities and they have full knowledge of data us- age. All annotators are well-trained with evaluation protocols and are proficient in English. The salary rate is above minimum wage at the local region. All human rating data will be released upon paper acceptance. The detailed human evaluation process included in the Appendix Sec F.
In the present framework, we introduce a straightforward but efficient refinement process to enhance the alignment of our metric with hu- man judgements. Future research can investigate more advanced techniques, such as incorporating human feedback through reinforcement (Ouyang et al., 2022), for more effective integration of feed- back into the training pipeline. More sophisticated approach holds promising potential to further boost the performance of this pipeline.
Ethics Statement
Acknowledgement
INSTRUCTSCORE, an open-source and explain- able evaluation metric for text generation, empha- sizes transparency and accountability in the evalu- ation of natural language processing systems. By generating interpretable evaluations and diagnos- tic reports, it fosters trust among developers and end-users. Moreover, its introduction could pro- pel further innovation in the field of explainable evaluation metrics and make high-quality evalua- tion tools more accessible. However, it is crucial ascertain that the interpretations provided by InstructScore do not harbor biases present in the training data, and data privacy and security mea- sures are observed.
This work was supported by the National Science Foundation award #2048 122. The views expressed are those of the author and do not reflect the official policy or position of the US government.
References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.

---

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:7
Natural Language Generation, pages 124-133, San- tiago de Compostela. Spain. Association for Compu- tational Linguistics.
Yette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172-176, Doha, Qatar. Association for Computational Linguistics.
Thiago Castro Ferreira, Claire Gardent, Nikolai llinkkh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020). In Proceed- ings of the 3rd International Workshop on Natu- ral Language Generation from the Semantic Web (WebNLG+), ). pages 55-76, Dublin, Ireland (Virtual). Association for Computational Linguisties.
Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853 899.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.
Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakr- ishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server.
Tom Koemi and Christian Federmann. 2023, Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302. 14520.
Daniel Deutsch, George Foster, and Markus Freitag. 2023. Ties matter: Modifying kendall's tau for mod- ern metric meta-evaluation.
Christoph Leiter, Piyawat Lertvitayakkmjjrn, Ma- rina Fomicheva, Wei Zhao, Yang Gao, and Steffen Eggrr 2022. Towards explainable evaluation met- ries for natural language generation. arXiv preprint arXiv:2203.1133:
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transac- rions of the Association for Computational Linguis- fics, 9:1460-1474,
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text gen- eration challenge for generative commonsense rea- soning. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 1823-1840, Online. Association for Computational Linguistics.
Markus Freitag, David Grangier, and Isaac Caswell. 2020. Bleu might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 61-71.
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Pe- West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. 2022. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 3154-312 3169, Dublin, Ireland. Association for Compu- tational Linguistics.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results WMT22 metrics shared task: Stop using BLEU - neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Yang Liu, Dan Iter. Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu, 2023. G-eval: Nig evaluation using gpt-4 with better human align- ment.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondfej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceed- ings of the Sixth Conference on Machine Translation, pages 733-774, Online. Association for Computa- tional Linguistics.
Qingyu Lu, Liang Ding, Liping Xie, Kanjian Zhang, Derek F Wong, and Dacheng Tao. 2022. To- ward human-like evaluation for natural language generation with error analysis. arXiv preprint 1rXiv:2212.10119.
Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04
François Mairesse, Milica Gašić, Filip Juresček, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010. basaee statistical language generation using graphical models and active learning. In Pro- ceedings of the 48th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1552- 1561, Uppsala, Sweden. Association for Computa- tional Linguistics.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura BeltracBll..cini 2017, The WebNLG challenge: Generating text from RDF data. In Pro- ceedings of the 10th International Conference on

---

OpenAI. 2023. Gpt-4 technical report.
Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223-231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow in- structions with human feedhack. arXiv preprint arXiv:2203.02155.
Brian Thompson and Matt Post. 2020. Automatic ma- chine translation evaluation in many languages via zero-shot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 90-121, Online. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- national Linguistics, pages 3-3138, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397 13911.
Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Niccolò Campolungo, and Roberto Navigli. 2022.
MaTESe: Machine translation evaluation as se- quence tagging problem. In Proceedings of the Sev- enth Conference on Machine Translation (WMT). pages 569-577, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image de- scription evaluation.
Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. 2022. UniTE: Unified translation evaluation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 81 17-8127, Dublin, Ireland. Association for Computational Linguistics.
Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Martins. 2022. COMET-22: Unbabel-ISTT IST 2022 submission for the metries shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT). pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Wenda Xu, Xian Qian, Mingxuan Wang. Lei Li, and William Yang Wang. 2022a. Sescore2: Retrieval augmented pretraining for text generation evaluation. arXiv preprint arXiv:2212.00
Alon Lavie, Luisa Coheur, and André
Wenda Xu, Yi-Lin Tuan, Yujie Lu, Michael Saxon, Lei Li, and William Yang Wang. 2022b. Not all errors are equal: Learning text generation metrics using stratified error synthesis. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022, pages 6559-6574, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2703 Online. Association for Computational I Linguistics.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. In Advances in Neural Information Processing Systems, volume 34, pages 27263-27277. Curran As- sociates, Inc.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. Learning robust metrics for text generation. In Proceedingg of the 58th Annual Meet- ing of the Association for Computationnl Linguistics. pages 7881-7892, Online. Association for Computa- tional Linguisties.
2020a. BLEURT:
Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvitayaakumjor, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Oräsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 shared task on quality estimation. of the Seventh Conference on Machine Translation (WMT), pages 69- 19-999, Abu Dhabi, United Arab Emi- rates (Hybrid). Association for Computational Lin- guistics.
Thibault Sellam, Amy Pu, Hyung Won Chung, Sebas- tian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, and Ankur Parikh. 2020b. Learning to evaluate translation beyond english: Bleurt submissions to the wmt metrics 2020 shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 921-927, Online. Association for Computational Lin- guistics.
Proceedings
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study trans- lation edit rate with targeted human annotation. In
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evalu- ating text generation with bert.

---

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Fric. P Xing. Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023, Judging llm-as-a-judge with mt-bench and chatbot arena.
A Prompting for Data Generation
In this section, we will discuss the data generation processes using prompts for GPT-4 API. We start from the seed domains, such as News, Technical, Legal and Medical, etc. We query GPT-4 to aug- ment seed domains to 100 domains (See prompt example in Table 25). For each domain, query GPT-4 to generate 100 topics (See prompt ex- ample in Table 26). Therefore, we have obtained 10,000 different topics for this process. For each topic, we generate 5 distinct sentences, each with different length and structure (See a prompt ex- ample in Table 27). For each topic, we randomly select one sentence out of five candidates, yield- ing 10,000 raw text sentences from distinct topics. We start from each text to synthesize sentence errors.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards unified multi- dimensional evaluator for text generation. In Pro- ceedings of the 2022 Conference Empirical Meth- ods in Natural Language Processing, pages 2023- 2038, Abu Dhabi, United Arab Emirates. Association for Computational I Linguistics.
Based on the guidelines provided by (Freitag et al., 2021a), we arbitrarily decide the range errors from 1 to 5. For each raw text, we randomly select number of errors from 1 to 5. Each synthe- sized error will have error types that are predefined from MQM guidelines (Freitag al., 2021a). For each synthesized error, randomly select ror type that is defined in Table 28 and randomly select major or minor severity labels. Therefore, based on the given raw text, error type, and sever- ity label, GPT-4 needs to generate the location of the error and explanation for the error annotation (See prompt example in Table 29). To disen- tangle the model's reliance sentence structure and lexical overlap, paraphrase the raw text yield the pseudo reference. The generated synthetic erroneous sentence will become pseudo model gen- erated text. This completes our synthetic data con- struction. During the fine-tuning process on the LLAMA model, we input the model with pseudo reference and pseudo candidate text, LLAMA is optimized to generate error type, error location. severity label, and explanation.
B Prompting for LLM Feedback
To obtain GPT-4 feedback for INSTRUCTSCORE'S generated diagnostic report, we developed two dif- ferent prompts. If our diagnostic report contains eπror annotations, asked 7 questions listed in Table 34 determine the correctness and consis- tency of the explanation output. This is how we collected responses for failure modes that we de- fined Table 2. if our diagnostic report does not contain error annotation, we directly query GPT-4

---

E Implementation on Refinement pipeline Human Evaluation Instructions:
MI: Error type descriptions are not consistent explanation
M2: Error locations are consistent with explanation
output text M4: Fror locations can not refer to the output text
Error locations are not referred
M5: Major and minor labels not correspond the correct severity levels
M6: The explanation is wrong. However, error at specified location does exist.
GI: Error described in the explanation is not an error
G2: One error mentioned more than once among explanations
G3: Incorrect phrase and correct phrase are not correctly aligned
G4: One error span mentions multiple errors
The rule is following: local failure mode will only impact local field in error annotation, like error type, error location or explanation. However, global failure mode between different error annotations.
For example, sentence contains 2 errors. will fields total
Error type
Major/minor:
Error location
Explanation for error 1:
Error location 2:
Major/minor:
Error location 2:
Explanation for error 2:
If one MI and one M2 occur, you will recoive score 6/8. If GI occurs, the entire error annotation or (like error type2. major/minor, location and explanation all become invalid). You will receive score 4/8. If global failure has overlap with local failure, you will choose global failure as annotation.
Your annotations will begin from here:
Table 8: This is the human evaluation instruction for human raters.
to validate this claim. If GPT-4 reconfirms with INSTRUCTSCORE 's claim, the feedback score is 1, Otherwise, 0.
Chinese-to-English parallel sentences. We ran- domly select one of the 20 MT systems to translate each source sentence and obtain 2,000 translation outputs in the end. We pair each translation with reference as input to INSTRUCTŠCORE and use top sampling with temperature 0.8 and p = 0.9 to generate 8 candidate outputs for each input. We ob- tained GPT-4'ss feedback on those 16,000 candidate outputs and formed 35,932 ranking pairs 3 We selected 4.777 diagnostic outputs which achieved the highest alignments scores to further fine-tune INSTRUCTSCORE.
C Raw Sentence Generation from GPT-4
For the data generation pipeline, we start by prompting GPT-4 with 12 seed domains, such as medical, technology News, etc. We used GPT-4 to augment them to 100 distinct domains. For each do- main, we further used GPT-4 to generate 100 topics. In the end, we generate 10k sentences with distinct topics, lengths, and sentence structures (See Ap- pendix Table 25).
F Human Evaluation Procedure
D Implementation of INStRUCTSCORe at Non-English Language
We conduct a human evaluation on WMT22 Zh-En testing set. We randomly select 100 system out- puts from 14 participating systems. Following the prior practice (Freitag et al., 2021a), we hired three graduate students who proficient in English lan- guage annotations. Each rater is trained with our annotation procedure for two hours before the anno- tations. The detailed human evaluation instruction is included in Table 8. Each rater will give a binary score for each field of error type, error location, ma- jor/minor label and explanation. If global failure
We trained a German reference only metric to inves- tigate INstruCTSCORe' multilingual capability. In this case, our metric will output explanations in English but quote error locations in German. See example in Appendix Table 33.
We used 18 participating system outputs WMT20 (Sellam et al., 2020b) to approximate the distribu- tion of the model- gederreee output. We use 2,000
We removed tie ranking pairs

---

Seen Tasks Flicker3k Rank(r/p)
Unseen Task Rank Avg
MT -En) Rank(r/p)
WebNLG Rank(r/p)
Commgen Rank(Ace)
BAGEL Rank(r/p)
Supervised
MATESE
4/3
Metrie XXL
2/2
COMET-22
1/1
BLEURT-20
6/6
2/1
3/3
2.8
BLEU
14/13
10/10 10
8/10 10
10
9/
10.2
Without Supervision
ChrF
13/14
7/7
10/4
8
10/8
9.0
METEOR
12/12
9/9
9/9
7
8/
9.4
CIDEr
11/11
8/8
713
9
7/6
7.8
BERTSScore
8/7
6/5
4/6
3
6/5
5.6
BARTScore
10/
514
317
5
517
6.2
PRISM
9/9
3/2
6/5
4
4/4
5.1
GEMBA-GPT4
5/8
SEScore2 INSTRUCTŠCORE
715
4/6
5/7
2/2
4.9
3/4
2/3
1/2
1/1
2.0
Table We rank metrics based Meta evaluation such Kendall correlation (r). Pearson correlation (p) and Ranking accuracy (Acc) for each task. The final ranking the average rankings of all Meta-evaluations five tasks.
INSTRUCTSCORE
Raterl
Rater2
Rater3
G Qualitative Analysis
Fine-tune
0.818
0.698
0.804
In this section, we will display a case study of our generated explanations from INSTRUCTSCORE.
Reefnemmn
0.849
0.887
0.902
Table 10: The human scores from three different raters before and after refinement.
G.1 Case study of INSTRUCTSCOOR'S Output Failure Modes of INSTRUCTSCORE'S Output Please check ten failure modes (M1-M6, GI-G4) for INSTRUCTSCORE without refinement (See Ta- ble 13, 14, 15, 16, 17, 18, 20, 21, 22).
has an overlap with local failure, rater will choose global failure as annotation. For each annotated ex- ample, rater will count the number of correct fields and divide the total number of fields to obtain alignment score. The final score of 100 examples is the average of 100 alignment scores. We ob- tained Kappa index among three raters, 0.388, for inter-rater agreement. Table 10. is evident that all raters agree on the improvement alignment after the refinement process.
Fields
Explanation Failure Mode
Percent%
Local Failure Mode
Error Type
MI: Consistency explanation
5.2%
M2: Consistency explanation
1.2%
Error Location
M3: Error location hallucination
8.2%
M4: Error location hallucination
4.5%
Explanation
M5: Major/Minor disagreement
27.6%
Global Failure Mode
G1: No-error
2.3%
Error Type, G2: Repetition
1.8%
Location and
G3: Phrase misalignment
Explanation
G4: Mention multiple errors
GS: Phrase inconsistency
2.2%
Table This table shows explanation failure modes and their corresponding failure occurrence ratios.

---

Seen Tasks
Unseen Task BAGEL T/p
WMT22/Zh-FE
WebNL-G20 7/p
Flicker3k 7/p
Commongen Acc
T/p
INSTRUCTSCORE (LLaMA) INSTRUCTSCORF (LLaMA2)
40.3/519
39.5 59.0
30,1 34,6
58.2
25.6/ 34.2 31.0 / 46.2
39.6/ 48.9
39.0 54.1
21.1
21.8
69.9
Table 12: We applied segment-level Kendall and Pearson correlation WMT22, WehNI Flickeräk, and BAGEL. and ranking accuracy for Commongen. INSTRUCTŠCORE with LLaMA initialization outperforms LLaMA2 initialization at Translation, data-to-text and captioning. However, INSTRUCTSCORF hased on LLaMA2 initialization significantly outperforms LLaMA initialization at Commonsense generation and unseen task texi-to-dialguee (BAGEL).
INSTRUCTSCORE Input:
Reference: Recommendation: Swiss legacy: wear art on your wrist; OMEGA elegant simple, low-key but attractive, suitable for work and formal situations.
Output: Recommended reason: Inheriting Switzerland, the art time is the wrist, Omega is dignified and elegant, simple and simple, not ostentatious but charming, and can worn on commuting and formal occasions.
INSTRUCTSCORE Output:
Your Translation contains 5 errors:
Error type Incorrect translation does not accurately represent correct translation
Major/minor: Major
Error location 1: "Inheriting Switzerland"
Explanation error The phrase "Inheriting Switzerland" does not accurately convey the meaning of "Swiss heritage." "
Error type 2: Incorrect translation includes information not present in the correct translation
Major/minor: Major
Error location 2: "the art of time is the wrist"
Explanation for error 2: The phrase "the art of time is in the wrist" introduces information that is not present in the correct translation, which is about the art of time being on the wrist. Error type 3: Incorrect translation is missing content from the correct translation
Major/minor: Major
Error location 3: "simple and simple"
Explanation for error 3: The phruse "simple and simple" is missing the word "clegant" from the correct translation, which should be "simple and clegant."
Error type 4: Incorrect translation has stylistic problems
Major/minor: Major
Error location 4: "not ostentatious but charming"
Explanation for error 4: The phrase "not ostentatious but charming" has stylistic problems, as it should be "understated but charming" to match the correct translation.
Error type 5: Wrong grammatical register (eg. inappropriately informal pronouns). Internal inconsistency (not related terminology)
Major/minor: Major Error location 5: "commuting"
Explanation for error 5: The word "commuting" is not consistent with the correct translation's 's use of "work" and changes the meaning of the sentence.
Table 13: This is the case where error type descriptions are not consistent with explanation (M1). Error type 5 should be word choice error. However, model predicts that this is related to grammatical register.

---

INSTRUCTSCORE Input:
Reference: The advantage of building inspection shafts using concrete modules is that the four sides of the modules are of concave groove structures, which are built to form chain locks.
Output: The advantage of using concrete modules to build inspection wells is that the upper, lower, left and right sides of the modules are all concave-convex groove structure, and the combined masonry forms chain lock.
INSTRUCTSCORE Output:
Your Translation contains 4 errors:
Error type I: Incorrect translation does not accurately represent the correct translation
Major/minor: Major
Error location I: "inspection wells" instead of "inspection shafts"
Explanation for error I: The term 'inspection wells" is not the same as "inspection shafts" and may confuse the reader.
Error type 2: Incorrect translation includes information not present in the correct translation
Major/minor: Major
Error location 2: "inspection wells" instead of "inspection shafts"
Explanation for error 2: The addition of "convex" changes the meaning of the structure, which is not present in the correct translation.
Error type 3: Incorrect translation is missing content from the correct translation Major/minor: Major
Error location 3: "combined masonry" instead of "combined masonry work"
Explanation for error 3: The phrase ccomiined masoury work" missing, which changes the meaning of the sentence.
Error type 4: Incorrect translation has stylistic problems
Major/minor: Minor
Error location 4: "The advantage of using conerete modules to build inspection wells is that the upper, lower, left and right sides the modules are concave-convex groove structure,"
Explanation for 4: The sentence structure awkward and clear the correct translation.
Table 14: This is the where error locations consistent with the explanation (M2).
INSTRUCISCORE Input:
Reference: 1. [Baijiu Distillery Direct Sales] Ningcheng Balihan Aged Cellar 42 Degrees 500 ML, Rich Flavor Sorghum Original Baijiu, Aged Spirit Tasting Mellow, Four Bottles in One Box Output: 1. [Directly run by winery] Ningcheng Balihan Laojiao 42 degree 500ML lush-flavored sorghum liquor with soft and refreshing taste, four bottles in the whole box
INSTRUCTSCORE Output:
Your Translation contains 5 errors:
Error type I: Incorrect translation does not accurately represent the correct translation
Major/minor: Major
Error location 1: "winery" instead of "distiller""
Explanation for error I: 'Tee translation incorrectly states that the business is a winery instead of distillery, which changes the meaning of the sentence.
Error type 2: Incorrect translation includes information not present in the correct translation
Major/minor: Major
Error location 2: "lush-flavored" instead of "rich-flavored"
Explanation for error 2: The translation adds the word "lush," which is not present in the correct translation and changes the meaning of the description.
Error type 3: Incorrect translation is missing content from the correct translation
Major/minor: Major
Error location 3: "aged spirit" instead of "aged baijiu"
Explanation for error 3: The translation omits the specific type of alcohol being described, which is important for understanding the product being sold.
Error type 4: Incorrect translation has stylistic problems
Major/minor: Major
Error location 4: "whole hox" instead of "box" Explanation error 4: The translation uses the phrase "whole box," which not a standard way describe box may confuse reader.
Error type Incorrect translation does not accurately represent the correct translation
Major/minor: Major
Error location 5: sooft and refreshing taste" instead "smooth and refreshing taste"
Explanation for error The translation changes the adjective "smooth" to "soft," " which alters the description the taste the alcohol.
Table This the case where error locations are not referred the ourput text (M3). "aged spirit" not never mentioned in the output text.

---

INSTRUCTSCORE Input:
Reference: It understood that activities held during "Children's Day" by China Children's 'ss Center this year include the "Cool Maker Loves Family" curniaal children's future learning. the first children 's visual art festival, the "Bnok Carnival" event and the "Whole Family Loves Reading" Picture Book Reading activity.
Output: is understood that activities held by the China children's is center during this year's "June children's day include the "love family and create cool" children's 's future learning carnival, the first children's 's visual arts festival, the "book carnival" activity, the "family love reading" picture book series activity, etc.
INSTRUCTSCORE Output:
Your Translation contains 5 errors:
Error type 1: Incorrect translation includes information not present the correct translation
Major/minor: Major
Error location 1: "June 1"
Explanation for error I: The incorrect translation specifies the date of Children's Day as "June 1," while correct translation says "Children's 's Day," which does not specify the date.
Error type Incorrect translation has stylistic problems Major/minor: Major
Error location 2: "love family and create cool"
Explanation for error 2: The phrase "love family and create cool" is awkward and does not convey the intended
meaning of "celebrating families and creativity."
Error type 3: Incorrect translation is missing content from the correct translation
Major/minor: Major
Error location 3: "Children's 's Day"
Explanation for error 3: The incorrect translation omits the term "Children's Day." which is an important aspoct of the correct translation.
Error type 4: Incorrect translation does not accurately represent the correct translation
Major/minor: Major
Error location 4: "picture book series activity"
Explanation for error 4: The incorrect translation changes the meaning of the "picture book reading activity" imply a series of books rather than single event.
Error type S: Incorrect translation includes information not present in the correct translation
Major/minor: Major
Error location 5: "whole family loves reading"
Explanation for error 5: The incorrect translation adds the phrase "whole family loves reading," which is not present in the correct translation and changes the focus from "families" to "whole families."
Table 16: This is the case where error locations in explanation not refer to the output text (M4), "whole family loves reading" is never mentioned in the output text. Instead, was mentioned in the reference text.
INSTRUCTSCORE Input:
Reference: Therefore, Zhang Zhongmou thinks it ironic for Intel to work on wafer foundry.
Output: So, Zhang Zhongmou feels ironic about Intel's entry into wafer foundry.
INSTRUCTSCORE Output:
Your Translation contains 2 errors:
Error type 1: Wrong grammatical register (eg, inappropriately informal pronouns), Internal inconsistency (not related to terminology)
Major/minor: Major Error location 1: "So"
Explanation for error 1: The use of "So" is informal and inconsistent with the formal tone of the rest of the sentence.
Error type 2: Incorrect translation is missing content from the correct translation
Major/minor: Minor
Error location 2: "entry" instead of "entry into"
Explanation for error 2: The incorrect translation omits the word "into," which is necessary to convey the meaning of Intel's entry into the wafer foundry market.
Table 17: This is the case where major and minor labels do not correspond to the correct severity levels (M5). The use of "So" is not a severe crror.

---

INSTRUCTSCORE Input:
Washing hands with air, diagnosing disease through hreath, accurately seeking for source of smog; last night, a scries of creative academic achievements were awarded at the opening ceremony the 2016 Annual Academic Conference of Zhejiang University.
Reference:
Output: Wash hands with air, judge diseases, accurately search for the source of fog Yesterday evening, at the opening ceremony the 2016 academic annual meeting Zhejiang University, series academic achievements were recognized.
INSTRUCTSCORE Output:
Your Translation contains 3 errors:
Error type 1: Incorrect translation is missing content from correct translation
Major/minor: Major
Error location 1: "judge diseases"
Explanation error The incorrect translation is missing the word "diagnose" and changes the meaning from accurately determining the cause of disease to simply judging diseases.
Error type 2: Incorrect translation has stylistic problems
Major/minor: Major
Error location 2: "accurately search for the source of fog"
Explanation for error 2: The incorrect translation uses "fog" instead of "smog." which changes the meaning and makes the sentence less clear.
Error type 3: Incorrect translation includes information not present in the correct translation
Major/minor: Minor
Error location 3: "academic annual mecting"
Explanation for error 3: The incorrect translation adds the word "annual" to the phrase "academic meeting," " which is not present in the correct translation. However, this does not significantly change the meaning of the sentence.
Table 18:
This is the case where the explanation is wrong. However, error at specified location does exist (M6). "academic annual meeting" minor crror, However, explanation is wrong because the word "annual" is also present in the reference.
INSTRUCTSCORE Input:
Reference: This set of biological sensor uses the new material of graphene; its target is chemical elements, and its keen "sense of smell" enables it to reflect the health status of the body more in-depth and accurately. Output: This biosensor uses graphene, a new material, and its target is chemical elements. Its keen sense of smell enables it to reflect the hcalth status more deeply and accuratcly.
INSTRUCTSCORE Output:
Your Translation contains 2 errors:
Error type I: Incorrect translation is missing content from the correct translation
Major/minor: Major
Error location 1: "and its target is chemical elements"
Explanation for error The incorrect translation omits the information about the biosensor's target being "chemical elements" and instead only mentions "chemical elements." Error type 2: Incorrect translation has stylistic problems Major/minor: Minor
Error location 2: "biosensor" instead of "biological sensor"
Explanation for error 2: The incorrect translation uses "hioscnsor" instead of "biological sensor," which a minor stylistic error that does not significantly change the meaning but may be noticed by the reader.
Table 19: Error described in the explanation is not an error (G1). The incorrect phrase and suggested revision are exactly the same.

---

INSTRUCTŠCORE Input:
Reference: And how much longer does it take for the restaurant to prepare it? Could could help me ask them? Output: Or how long before the restaurant is ready? Can you ask for me?
INSTRUCTSCORE Output:
Your Translation contains 3 errors:
Error type I: Incorrect translation is missing content from the correct translation
Major/minor: Major
Error location I: "Or how long before the restaurant is ready?"
Explanation for error 1: The incorrect translation omits the information about the wait time for the food, which is an important detail in the correct translation.
Error type 2: Incorrect translation has stylistic problems
Major/minor: Major
Error location 2: "Can you ask for me?"
Explanation for error 2: The incorrect translation changes the meaning of the sentence implying that the person is asking the question behalf of someone else, rather than themselves. Error type 3: Problems with grammar, other than orthography Major/minor: Minor
Error location 3:
"could help ask them?"
Explanation for error 3: The incorrect translation uses "could help ask them" instead of "could you ask them for me?" The grammarr is slightly off. but the meaning is still clear.
Table 20: One error is mentioned more than once among explanations (G2).
INSTRUCTSCORE Input:
Normally, people wash hands by flushing hands with water, with little soap and sanitizer at most; however, Li Qizhang and his classmates invented set of "equipment to wash hands with air".
Reference:
Output: Ordinary people wash their hands with water, at most, wipe some soap and squeeze some hand sanitizer, while Li Qizhang and other students invented set of "air hand washing device".
INSTRUCTŚCORE Output:
Your Translation contains 4 errors:
Error type I: Incorrect translation is missing content from the correct translation
Major/minor: Major
Error location I: "with air"
Explanation for error 1: The incorrect translation changes the method of hand washing from "with water" to "with air,"
which significantly alters the meaning of the sentence. Error type 2: Incorrect translation has stylistic problems Major/minor: Major
Error location 2: "Ordinary people wash their hands with water, most, wipe some soap and squeeze some hand sanitizer."
Explanation for error 2: The incorrect translation has choppy and awkward sentence structure, making it difficult understand the intended meaning.
Error type 3: Incorrect translation includes information present in the correct translation
Major/minor: Major
Error location 3: "other students"
Explanation for error 3: The incorrect translation adds information about other students, which is not present the conrect translation.
Error type Incorrect translation does accurately represent correct translation Major/minor: Minor
Error location 4; "air hand washing device"
Explanation error The incorrect translation uses "air hand washing device" instead of "air hand washing equipment." which a less accurate representation of the correct translation.
Table This case where two phrases are misaligned in explanation (G3). "With water" and "With air" are incorrectly aligned. "with air" is the correct generation output and is an emor.

---

INSTRUCIŠCORE Input:
Reference: Wang Cheng pointed that hosting the talent summit important measure implement the spirit of the talent working conference of the central govemmment and spirit of the 12th Party Congress of Hunan, showing the sincerity of Hunan
of putting emphasis talents, being cager for talents and making full of talents. Output: Wang Cheng pointed out that the talent summit important initiative implement the central talent work conference and the spirit of 12th Party Congress the province, which shows the sincere sincerity Hunan attaching importance to talents, thirsting for talents and using talents well.
INSTRUCTSCORE Output:
Your Translation contains 2 errors:
Error type I: Incorrect translation does not accurately represent the correct translation
Major/minor: Major
Error location I: "the central talent work conference" and "the 12th Party Congress the province" Explanation for
The incorrect translation changes the meaning by referring different events and locations
than the translation.
Error type Incorrect translation has stylistic problems
Major/minor: Minor
Error location 2: "thirsting talents"
Explanation error 2: The phrase "thirsting for talents" awkward and not commonly used, which makes stylistic issue.
Table 22: One error span mentions multiple errors (G4)
INSTRUCISCORE Input:
Reference:
Washing hands with air, diagnosing disease through breath, and accurately seeking for source of smog; last night, a series of creative academic achievements were awarded at the opening ceremony of the 2016 Annual Academic Conference of Zhejiang University.
Output: Wash hands with air, judge diseases, accurately search for the source of fog Yesterday evening, at the opening ceremony of the 2016 academic annual meeting of Zhejiang University, series of academic achievements were recognized.
INSTRUCTSCORE before refinement's output:
Your Translation contains 3 crrors:
Error type 1: Incorrect translation is missing content from the correct translation
Major/minor: Major
Error location 1: "judge diseases" Explanation for error I: The incorrect tramslation is missing the word "diagnose" and changes the meaning from accurately determining the cause of a disease to simply judging diseases. Error type 2: Incorrect translation has stylistic problems Major/minor: Major
Error location 2: "accurately search for the souree of fog"
Explanation for error 2: The incorrect translation uses "fog" instead of "smog," which changes the meaning and makes the sentence less clear.
Error type 3: Incorrect translation includes information not present in the correct translation Major/minor: Minor
Error location 3: "academic annual meeting"
Explanation for error 3: The incorrect translation adds the word "annual" to the phrase "academic meeting," which is not present in the correct translation. However, this does not significantly change the meaning of the sentence.
INSTRUCTSCORE after refinement's output:
Your Translation contains 3 errors:
Error type I: Incorrect translation is missing content from the correct translation
Major/minor: Major
Error location 1: "judge discases" Explanation for error 1: The incorrect translation is missing the information about
"diagnosing discases" from the correct translation.
Error type 2: Incorrect translation has stylistic problems Major/minor: Major
Error location 2: "accurately search for the source of fog"
Explanation for error 2: The incorrect translation uses "fog" instead of "smog." which changes the meaning and makes it less clear.
Error type 3: Incorrect translation does not accurately represent the correct translation
Major/minor: Minor
Error location 3: "academic annual meeting"
Explanation for error 3: The incorrect translation uses "academic annual meeting" instead of "academic annual confer- ence." which is minor that does not significantly change the meaning but will be noticed.
Table 23: In this case, INstrUCTScORE hefore refinement is able correctly identify the errrr location, type and major/minor label. However, its explanation over the emor is incorrect. INStRUCTSCORE after refinement improves model's rationable and output the correct explanation.

---

INSTRUCTSCORE Input:
Reference:
Diese Vakuumpumpe mit Mehrfachbetrich hat einen speziellen, in den Pumpenkopf integrierten Griff, der dem Benutzer eine exzellente Kontrolle über den Pump- und Saugvorgang gibt."
Output:
Diese mehrfach angetriebene Vakuumpumpe verfügt über einen speziellen Griff, der in den Pumpenkopf eingebaut ist und dem Benutzer eine hervorragende Kontrolle über den Pump- und Saugvorgang gibt.
INSTRUCISCORE Output:
Your Translation contains 2 errors:
Error type 1: Incorrect translation is missing content from the correct translation
Major/minor: Major
Error location I: "mit Mehrfachbetrieb" is missing in the incorrect translation
Explanation for error I: The incorrect translation omits the important information that the vacuum pump is multifune- tional, which changes the meaning of the sentence. Error type 2: Incorrect translation has stylistic problems Major/minor: Minor
Error location 2: "hervorragende" instead of "exzellente"
Explanation for error 2: The use of "hervorragende" is incorrect, but it is less commonly used in this context and may be considered stylistic choice. The meaning is still clear. but it may be noticed by the reader
Table 24:
Candidate and reference texts are both in German. However, instruction and diagonostic report are in English to enhance user's understandings. From this example, can the generation of the Non-English diagnostic report requires mixed code output and multilingual understandings between English and target language.
Find 100 major domains machine translation following format like this: News, Technical, Legal, Medical, Financial, Gaming, E-commerce, Tourism and hospitality, Marketing and advertising, and Scientific (make sure all domains are very distinct from another)
Prompt:
GPT-4:
"News", "Technical", "Tccnical",""Lgaa""" "Legal", "Medical", "Financcil""" "Gaming", "E-commerce"; ""aourrrc""""""iiii "Toourism and hospitality", "Marketing and advertising" "Scientifiei""" ""cientiie"."dduuatioo "Energy", "Energy,,"Agricultrr" "Agriculture" "Aerospacc", "Agrooulure""AArrspaeee"" "Art and culture", "aauure",""Butty"""" "Beauty and personal care", "Construction ", "CConsumer "Government"" "Heaaat "Health and fit-
"Education" ". "Social "oadia,,"Ettrtaiinnnen "nrttt""""""taaaa" "Automotive"."" "iarroooivve,,SSottt ". "Retail",
goods". "Defense". "Electronicc"" "Deleense"""llecrnniis"" ", "Eavironment"., "Food and beverngg"."Governmeen" ness". "Insuraccc""""" "Internet of things (IoT)", " "Logistics". "Manufacturing" "iaiiictturin""""ussiiii "Music", "Non-profit" "Phamacceticcala" "Real estate". estate"."Recrea "Recruitment". "Religion" "ww.eiioon."Teeeemummuniaia "Telecomununication" "rrpuuiiiiciiin""""rrttrrttrrai "Trrasportation" "aaiiioonn""""iiiii""i" "Utilities" "Venture and augmented reality"," "Web development", "Wine and spirits",
capital", " "eriiss"""Woeee""" "Women's issues". "ssues".."Worplla" "Workplace and HR". "Youhh issues", ""oologyy, "Roboties and automation". "Urhan planning and development", "Parenting and family"."Mentt Mentall health and wellness". "iireesss""Biieeet""" ""Biotcchnology". "Cybersecurity". """turrttt"""""""""""""" "Film and television" ". "Language and "Photiggs" and "atbogrrppy"""Poliicci" and diplomacy", "Psychology and neuroscience", "PPbblic health and and journalism" "Radio and podeasts", "Renewable energy and sustainability". "Reearcch and development", ""thipppp""""hppiinne and maritime", "Space and astronomy": "iaimny"""""ttt" "Start-ups and entrepreneurship.. "Taxation and accounting" "Textiles and fashion", "Theatre and performing arts". "Travel and exploration" ", "Weather and meteorology", "Wildlife and conservation". "Animation and graphic design". "Artificial intelligence and machine learning" "Bllockchinn and cryptocurrency" "Cannabis and hemp industries" "iaistries"""hheeeee" "Chemical engineering and materials science". "Coaching and personal development". "Comic books and graphic novels". "Dance and choreography" "Diet and nutrition" "Digital marketing and SEO", "Disaster relicf and emergency management", "F-llamming and online courses", "Event planning and management" "Forensie science and criminology", "Genealogy and family history", "Geology and earth sciences" "Home and garden", "Industrial design and product design", "Intellectual property and patents", "Interior design and architce- ture", "Library and information science", "Marine biology and oceanography" "Meteorology and climatology" "Mining and minerals", " "Pet care and animal husbandry"
Table 25: This is one example of how we generate 100 distinet domains from GPT-4

---

Prompt: Find 100 topics in the Technical domain (make sure all topics very distinct from one another)
GPT-4: 1. Artificial Intelligence 2. Machine learning 3. Data Science 4. Natural Language Processing 5. Robotics
6, Internet of Things 7. Big Data 8. Cyber Security 9. Cloud Computing 10, Virtual Reality 11. Augmented Reality
12. Blockchain 13, Cryptocureencies 14, Quantum Computing 15. 5G Technology 16. Wireless Communication 17.
Human-Compute Computer Interaction 21. User Experience Design 22. Software Development 23, Web Development 24, Mobile App Development 25, Game Development 26. Computer Graphics 27. Computer Animation 28. Computer Networks 29. Operating Systems 30. Database Management 31. Information Retrieval 32. Computer Architecture 33. Microprocessors 34. Embedded Systems 35. Control Systems 36. Mechatronics 37. Electrical Engineering 38. Electronic Engineering 39. Mechanical Engincering 40, Civil Engineering
Computer Vision 18, Neural Networks 19. Deep Learning 20. Human-
41. Chemical Engineering 42. Materials Science 43. Nanotechnology 44. Biomedical Engineering 45. Bioinformatics 46.
Computational Biology 47. Geneties 48. Neuroscience 49. Psychology 50. Linguistics 51. Computational Linguisties
52. Digital Signal Processing 53. Image Processing 54, Audio Processing 55. Video Processing 56. Speech Recognition
57. Speech Synthesis 58. Texc-to-Speech 59. Optical Character Recognition 60. Computer Vision Applications 61.
Face Recognition 62. Object Detection 63, Object Tracking 64. Image Segmentation 65. Image Classification 66.
Image Retrieval 67. Image Compression 68. Video Compression 69. Video Streaming 70. Multimedia Systems 71.
Human- Robot Interaction 72. Centnard Computing 73. Ubiquitous Computing 74, Wearable Computing 75. Smart Homes 76. Smart Cities 77. Intelligent Transportation Systems 78. Autonomous Vehicles 79. Drones 80.
Industrial Automation 81. Manufacturing Systems 82. Quality Control 83. Supply Chain Management 84. Logistics
85. Health Informatics 86. Telemedicine 87. Electronic Health Records 88. Medical Imaging 89. Medical Devices 90.
Biomechanics 91. Rehabilitation Engineering 92. Assistive Technology 93. Education Technology 94, E-Learning 95. Learning Analyties 96. Gamification 97. Social Media Analytics 98. Web Analytics Search Engine Optimization 100. Digital Marketing.
Table 26: This one example how we generate 100 topics within certain domain. Here are 100 topics generated by GPT-4 for the Technical domain.
Give English sentences about 5G Technology (Fach has distinct topic, length, and sentence structure). Only return English text.
Prompt:
GPT-4:
1. With the introduction of 5G technology, we can expect significantly faster download speeds, enabling quicker access high-quality content such as 4K videos and virtual reality experiences.
2. One of the most exciting aspects of 5G technology is its potential for improving communication between smart
devices in the Internet of Things, paving the way for more connected and efficient society.
3. Enhanced safety features and real-time data transfer in 5G-powered autonomous vehicles will revolutionize the way
we commute, offering faster, greener, and more reliable transportation.
4. 5G networks are anticipated make dramatic improvements in telemedicine; for example, doctors could conduct remote consultations and perform surgeries with haptic feedback. ultimately increasing accessibility to healthcare. Although there are concerns surrounding the potential health risks and privacy issues associated with 5G technology. many experts argue that the benefits far outweigh the drawbacks, heralding new era of innovation and connectivity.
Table 27: This one example how we generate 5 distinct sentences for distinct sentence length and structure from GPT-4 for
given topic.

---

Firor Category
Description
Addition
Translation includes information not present in the source.
Omission
Translation is missing content from the source.
Accuracy
Mistranslation
Translation does not accurately represent the source.
Untranslated text
Source been left untranslated.
Spelling
Incorrect spelling capitalization.
Grammar
Problems with grammar, other than orthography.
Fluency
Register
Wrong grammatical register (eg, inappropriately informal pronouns).
Inconsistency
Internal inconsistency (not related to terminology).
Character encoding
Characters are garbled due incorrect encoding.
Inappropriate for context
Terminology is non non standdddddd or does not fit the context.
Terminology
Inconsistent
Terminology used inconsistently.
Style
Awkward
Translation has stylistic problems.
Address format
Wrong format addresses.
Currency format
Wrong format for currency. Wrong format for dates.
Date format
Locale convention
Name format
Wrong format names.
Telephone format
Wrong format for telephone numbers.
Time format
Wrong format for time expressions.
Table 28: We the above types perform data synthesis GPT-4. These definitions from Freitag al. (2021a). For each annotation, randomly select of the above error types and query GPT4 for the generation. Please check the prompt example Table 29.
Prompt: You are evaluating model output based on a reference, which describes semantics about text. The correct translation is. "The art of writing for educational publications involves striking a delicate bulance between providing enough detail to be useful and overwhelming the reader with too much information." Your translation needs to contain two non-overlapping emors overlapeepppnggggge errors based the given error types. Please give 0 minor errors and 2 major errors. Major errors can significantly alter the meaning of the sentences. while minor errors will not change the meaning. For each eroo, give
error type, major/minor label, error location, and an explanation. The format is
Paraphrase correct translation:
Incorrect Translation:
Error type Translation includes information present in the correct translation
Major/minor: Major
Error location 1:
Explanation for emror 1:
Error type 2: Problems with grammar, other than orthography
Major/minor: Major
Error location 2:
Explanation for error
GPT-4: Paraphrase correct translation: "The art writing for educational publications involves striking delicate
balance between providing enough detail to be useful and overwhelming the reader with too much information." Incorrect Translation: "Writing for educational publications about finding balance between giving enough examples useful and making the reader bored with many details."
Eror type Translation includes information present the correct translation
Major/minor: Major
Eroo location "...giving enough examples..."
The correct translation talks about providing enough detail, specific examples, which changes the meaning of the sentence.
Explanation for
Error type 2: Problems with grammar, other than orthography
Major/minor: Major
Error location "...making the reader bored with many details."
Explanation for error 2: The correct translation mentions overwhelming the reader with too much information, not making them bored with too many details. This change in phrasing alters the meaning of sentence.
Table This prompt was used generate synthetic data to fine-tune the I.IAMA model, specifically GPT-4. The model was error annotation, pre-define emrrr type severity label.
provided with the number errors a sentence. GPT-4 was then required to identify error locations provide explanations for each error annotation. The prompts red are variables for each synthesized generation.

---

Prompt: You evaluating model output based on reference, which describes semantics about another modality, such as image. The correct generation is, "A very large kite is laying the beach.". Your generation needs to contain non-overlapping errors based the given error types. Please give 2 minor errors and 0 major error. Major errors can confuse mislead the reader due significant change in meaning, while minor are still noticeable. For each error, give an error type, major/minor label, error location and explanation. If some error types with major/minor labels do not exist incorrect generation, put N/A at error location and explanation. The format is
don't 't lead loss of meaning but
Paraphrased correct generation:
Incorrect Generation:
Error type The caption wrongly describes motion or movement in the correct generation
Major/minor: Minor
Error location
Explanation for error 1:
Error type 2: The caption wrongly describes object relationships the correct generation
Major/minor: Minor
Error location 2:
Explanation for error 2:
GPT-4: Paraphrased correct generation: A huge kite rests the sandy beach.
Incorrect Generation: huge kite is flying towards sandy beach. Error type The caption wrongly describes motion or movement in the correct generation
Major/minor: Minor
Error location "flynng towards"
Explanation for error 1: The kite described as laying on beach the correct generation, while in the incorrect
generation is described as flying towards the beach.
Error type 2: The caption wrongly descrihes object relationships in the correct generation Major/minor: Minor
Error location "towards"
Explanation for error 2: The incorrect generation suggests that the kite is moving towards the beach, while in the correct generation, the kite is already on the beach. This changes the relationship between the objects (kite and beach).
Table 30: This prompt was used to generate synthetic data to fine-tune the LLAMA model, specifically GPT-4, The model was provided with the number of errors a sentence. For each error annotation, we pre-detine the error type and severity lahel. GPT4 was then required to identify the error locations and provide explanations for each error annotation. The prompts in green are variables for each synthesized generation.
Prompt: You are evaluating model output based on commonsense knowledge. Use "enter, exit, shopper, store" to write a sentence which follows common sense. One example generation is, "shoppers enter and exit a store .". Your generation needs to contain two non-overlapping errors hased on the given error types. Please give 0 minor error and 2 major errors. Major errors can confuse or mislcad the reader due to significant change in meaning, while minor errors don't lead to loss of meaning but crrors are still noticcable. For cach error, give me an error type, major/minor label, an error location and an explanation. If some error types with major/minor labels do not exist at incorrect generation, put N/A al error location and explanation. The format is Paraphrased correct generation:
Incorrect Generation:
Error type I: Actions/beliefs contradicting known personality Major/minor: Major
Error location 1:
Explanation for error 1:
Error type 2: Inaccurate characteristics/allltte of objects
Major/minor: Major
Error location 2:
Explanation for error 2:
GPT-4: Paraphrased correct generation: Customers come in and out of the store.
Incorrect Generation: Shoppers fly in and stay forever in the store.
Error type 1: Inaccurate characteristics/abllttes of objects
Major/minor: Major
Firor location I: Shoppers fly
Explanation for error I: Shoppers cannot fly: they walk move using other means such wheelchairs or carts. Error type 2: Actions/belief's contradicting known personality
Major/minor: Major
Eroo location 2: stay forever in the store Explanation for error 2: Shoppers do stay in the store forever, they eventually leave after making their purchases browsing.
Table 31: This prompt was used to generate synthetic data to fine-tune the I.AMMA model, specifically GPT-4, The model was provided with the number of errors sentence. For each annotation, pre-define the error type and severity label. GPT-4 was then required identify the emor locations and provide explanations for each error annotation. The prompts in green variables for each synthesized generation.

---

Prompt: You are evaluating model output based on a structured data input a reference. The input '1634:The Bavarian Crisis author Eric Flint', '1634: The Bavarian Crisis mydiaType Hardcover', '1634 The Bavarian Crisis numberOfPages "448"" The correct generation is, "1634: The Bavarian Crisis, written by Eric Flint, has 448 pages available hardonver.". Your generation needs contain one non-overlapping error based on the given error types. Please give minor error and major eror. Major errors can confuse or mislead the reader due to significant change meaning, while minor errors don't lead to loss of meaning but emrors are noticcable. For each error, give me an error type, a major/minor label, an error location and an explanation. If some error types with major/minor labels do not exist at incorrect generation, put N/A at error location and explanation. The format is:
Paraphrased correct generation:
Incorrect Generation:
Error dimension 1: Data Coverage
Error type I: The text only partially deserihes predicate present in the data
Major/minor: Minor
Error location Explanation for error 1:
GPT-4: Paraphrased correct generation: "1634; The Bavarian Crisis, written by Eric Flint, has 448 pages and available in hardcover."
Incorrect Generation: "1634: The Bavarian Crisis, written by Eric Flint, is available in hardcover."
Erorr dimension I: Data Coverage
Error type 1: The text only partially describes predicate present in the data
Major/minor: Minor
Error location I: author and numberOfPages
Explanation for error I: The incorrect gencration omits the predicate "numherOOPages", which results in incomplete information and noticeable errors in data coverage.
Table 32: This prompt was used to generate synthetic data to fine-tune the LLAMA model, specifically GPT-4. The model was provided with the number of errors in a sentence. For each error annotation, we pre-detine the error type and severity lahel. GPT-4 was then required to identify the error locations and provide explanations for each error annotation. The prompts in green are variables for each synthesized generation.

---

You translating from English-to- German. The correct translation is, "Die Anzahl der COVID- 19-Falle Deutschland hat cinen neuen Höchstttand erreicht.". ". First, paraphrase the correct translation. Then, generate translation contains overlappa errors based on the given error types. Please give minor errors and major errors. Major errors can confuse or mislead reader loss meaning but explanation. some etror types with major/minor labels not exist incorrect translation, put N/A at error location explanation. The format is:
Prompt:
significant change meaning. while minor errors don't noticed. For each error, give me an error type, major/minor label, an error location an
Paraphrased correct translation:
Incorrect Translation:
Error type Terminology is non-standard does not fit context. Terminology is used inconsistently
Major/minor: Major
Error location
Explanation for error 1:
Error type 2: Incorrect translation includes information not present in the correct translation (Not the case of opposing
the meaning from correct translation)
Major/minor: Major
Error location 2:
Explanation for error 2:
Error type 3: Incorrect translation does not accurately represent the correct translation (Not the case of opposing the meaning from correct translation)
Major/minor: Minor
Error location 3:
Explanation for error 3:
Error type 4: Incorrect translation has stylistic problems
Major/minor: Minor
Error location 4:
Explanation for error 4:
GPT-4:
Paraphrased correct translation: "In Deutschland ist die Zahl der COVID- 1O-D--9--- 19-Fäl auf einen neuen Rekord gestiegen." Incorrect Translation: "Dic Summe 1O- in Deutschland hat einen neuen Höchstwert erreichen." Error type 1: Terminology is non-standard does not fit context. Terminology is used inconsistently
Major/minor: Major
Error location "COVID-19-Apfel"
Explanation for error 1: The term COVID--1-AAffe incorrect does not the context. The correct term should LOVID-19-Fälle"
Error type 2: Incorrect translation includes information not present in the correct translation (Not the case of opposing the meaning from correct translation)
Major/minor: Major
Error location 2: "Summe"
Explanation for error 2: The use of the word "Summe" adds information not present in the correct translation. The correct word should be "Anzahl" or "Zahl". Error type 3: Incorrect translation does not accurately represent the correct translation (Not the case of opposing the meaning from correct translation)
Major/minor: Minor
Error location 3: "erreichen"
Explanttion for error 3: The verb "erreichen" is in the infinitive form, while it should be in the past participle form "erreicht" to represent the correct translation accurately. Error type 4: Incorrect translation has stylistic problems Major/minor: Minor
Error location 4: "von"
Explanation for error 4: The preposition "von" is not necessary in this context, and it creates stylistic problem. The correct translation should be "Die Anzahl der COVID-19-Falle
Table 33: This prompt was used to generate synthetic data to fine-tune the LLAMA model, specifically GPT-4, The model was provided with the number of errors in sentence. For each error annotation, pre-define the error type and severity label. GPT-4 was then required to identify the error locations and provide explanations for each error annotation. The prompts in green are variables for each synthesized generation.

---

Prompt (For diagnostic report which contains enror annotations):
Ref: State-owned enterprises and advantageous private enterprises entered the revolutionary base area of south Jiangxi.
Out: State-owned enterprises and dominant private enterprises entered the old revolutionary district of southern Jiangxi. Errorl:
Error location 1: "old revolutionary district"
standard or does not fit the context. Terminology is used inconsistentl Explanation 1: The correct term should be "new revolationary hase area" which refers to the area where the revolution started in 1927, not "old revolutionary district" which does not convey the meaning.
Error type Terminology
Error2:
Error location 2: "dominant"
Error type Incorrect translation includes information present in the correct translation Explanation The word "dominant" is used instead of "privileged, which adds information that is present the translation, but does change the overall meaning.
For cach error location, extract the incorrect error location.
Al:
Q2: Parse explanation into either one the four forms: [incorrect phrase, correct phrase]. [incorrect phrase, None], INone, correct phrase], [None, None].
A2:
Q3: "incorrect phrase to correct phrase", is A2 a correct alignment reference and output? (Answer Yes/No)
A2 contains None, Answer None.
A3:
Q4: According to the sentence context, if A2 is "incorrctt phrase to correct phrase" and incorrect and correct phrases can be used interchangeably, it no-error. If an error can be fixed by reading the rest of the output context is a minor-error. changes the sentence's 's meaning (Don't consider sentence formality changes) and is unable fixed by reading the rest of the output it is a major-error. Answer, no-error or minor-error or major-error A4:
Q5: Is the explanation consistent with the given error type?
A5:
Q6: Does the explanation talk about the given error location?
A6:
Q7: Do two error locations mention the same location in two different ways? (Answer Yes/No) How many pairs of repetitions? (Answer in number)
A7: Yes/No, number
The output format will be in ISON [(Errl: (QI: AI, Q2: A2, Q3: A3, Q4: A4, Q5: A5. Q6: A6), Err: (QI: AI, Q2: A2, Q3: A3, Q4: A4, Q5: AS, Q6: A6). Q7: A7)
GPT-:: | "Errl"; { "QI": "old revolutionary district", "Q2": ['old revolutionary district", "new revolutionary base
area"], "Q3": "Yes", "Q4": "major-error", "Q5": "Yes", "Q6": "Yes" ), "En2": ("QI": "dominant", "Q2": ["dominant", "privileged"] "Q3": "Yes", "Q4": "minor-error.. "Q5": "Yes". " "Q6": "Yes" J. "Q7": "No, 0"
Table 34: Jumman-instructed annotation: if our diagnostic report contains error annotations, we asked following questions to determine the correctness and consistency of the explanation output. This is a prompt example of how we obtained GPT-4 feedback for Table 2.
Prompt (For diagnostic report which contains no-error):
Reference: Summarizing foreign media reports, the IAEA report states that Iran's Natanz Nuclear Facility's advanced centrifuge is "accumulating or ready to accumudate enriched uranium". Output: According to comprehensive foreign media reports, the IAEA report pointed out that the advanced centrifuges at Iran's 's Natanz nuclear facility are "accumulating or are ready to accunulate enriched uranium:". Compared to the reference, does the output contain any error? (Answer in Yes/No)
GPT-4:
Table Human-instructee annotation: our diagnostic report does not contain error annotation, we directly query GPT-4 validate this claim. GPT-4 reconfirms INSTRUCTSCORE claim, the feedback score is 1. Otherwise,


Reviews:
[{"review_id": "DD3NDJOxqe", "reviewer": "Reviewer_uTpq", "summary": "", "questions": "A: from the formulation of the Exp-Generator (L157) is looks like the system will always produce a report consisting of at least 1 error even for error-free hypotheses. Is it possible for the system to determine that a hypothesis is error-free?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback on our work! \n\nQ.A: Error-free cases?\n\nAnswer: To address the question of the \"no-error\" case, our InstructScore can produce a no-error result if there are no major and minor errors present.\n\nQ.B: Direct error report using GPT-4?\n\nAnswer: In our preliminary experiments, we found that just prompting GPT-4 is far from optimal and would clearly underperform our model. Similar behavior has also been reported by (https://arxiv.org/pdf/2303.13809.pdf). Moreover, the experiments of running GPT-4 over our entire testing set at WMT22 will cost around 1500 US dollars and 5000 US dollars for all five testing sets, which exceeds our research budget. Instead we include the existing best performed GPT-4 results (GEMBA-GPT4) on WMT22’s testing set to demonstrate our performance against prompt-based GPT4 methods. We would like to clarify that our approach does not heavily rely on GPT-4. Instead the goal is to finetune a small LLM to overcome the limitation of GPT-4. \n\nQ.C: Organization of the pipeline\n\nWe will improve the description of the data construction pipeline and move some parts of the appendix to the main body of the paper.\n\nQ.D: Explanation aspect of InstructScore? \n\nWe conduct human evaluation on the quality of explanations (with three annotators). We calculated precision and recall of the explanations before and after refinement. Precision = the number of correctly annotated error fields / total number of instructScore's labeled error fields. Recall = the number of correctly annotated error fields / (the number of correctly annotated error fields+the number of error fields that InstructScore missed). Based on the human evaluation result, 89.4% of InstructScore’s output after refinement is correct and it can identify 85.6% of errors produced by the translation system. To the field of explanation in particular, 86.1% of InstructScore’s output after refinement is correct according to humans and it can explain 85.0% of errors produced by the translation system. We will add these numbers to the revision.\n\n| All fields | Before refinement | After refinement |\n| ---------- | ----------------- | ---------------- |\n| Precision  | 0.778             | 0.894            |\n| Recall     | 0.824             | 0.856            |\n\n| Explanation field | Before refinement | After refinement |\n| ----------------- | ----------------- | ---------------- |\n| Precision         | 0.756             | 0.861            |\n| Recall            | 0.819             | 0.850             |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think the idea of trying to produce MQM-style reports using an automated metric is really nice as it provides a lot more fine-grained information than just an opaque score.\n\nThe work shows a clever way to leverage the implicit knowledge of large language models for evaluation.\n\nI think it is really impressive that the method consistently works well for a diverse range of text generation tasks (Table 2).\n\nI appreciate the human study showing the alignment improvements in Fig. 7.", "reasons_to_reject": "The procedure as described relies extremely heavily on synthetic data from GPT4. I think a reasonable question is whether one couldn't just prompt GPT4 to produce error reports directly, which I think might have been a good baseline for comparison.\n\nThe exact procedure for collecting synthetic data is very hard to understand without carefully studying the Appendix.\n\nOverall the described process is very involved and I am happy that the code is already available, as the process of reproducing this seems very labor-intensive.\n\nI think the explanation aspect of the work is a little under-explored. The error reports include explanations but it is not clear how helpful they will be in practice. Indeed, the human evaluation shows that they are wrong in many cases (Fig. 7).", "questions_to_authors": "A: from the formulation of the Exp-Generator (L157) is looks like the system will always produce a report consisting of at least 1 error even for error-free hypotheses. Is it possible for the system to determine that a hypothesis is error-free?", "paper_topic_and_main_contributions": "This work introduces InstructScore, a metric for text generation evaluation. Given a generated hypothesis and a reference text it generates an a list of error reports containing error type, location, severity, and an explanation. A score is assigned based on the severity of all errors.\nThis procedure is inspired by MQM annotations for evaluation. InstructScore is based on a LLaMa backbone which is fine-tuned on synthetic data generated by GPT-4. GPT-4 produces both the initial training data of hypotheses, references and error reports, as well as corrective feedback making sure that error reports are consistent. InstructScore is evaluated on a diverse set of text generation tasks and shows impressive performance.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "L124: to my knowledge the MQM framework was introduced by Lommel et al. in 2014 (see https://doi.org/10.5565/rev/tradumatica.77)"}, "extra_text": {}}, {"review_id": "ywxakMSjp2", "reviewer": "Reviewer_p5F9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your review of our work.\n\nQ.A: OpenAI restrictions:\n\nOpenAI’s terms state that “OpenAI prohibits using their services to develop models that compete with OpenAI”. Our InstructScore does not aim to compete against OpenAI in any of their products or redistribute GPT4’s output data. Rather, our objective is to provide an open-sourced and explainable evaluation metric and explore LLMs for fine-grained categorization of errors. Therefore it is acceptable under their terms. \n\nQ.B: Comparison set up:\n\nWe agree that model size and pre-training data plays an important role for the performance of the learned evaluation metric. However, the main focus of the paper is to propose a novel framework of explainable evaluation metric rather than extensively examining the scaling properties. We evaluate this metric in two settings: 1) its final score’s correlation to the human ratings 2) human evaluation at its diagnostic output. For the first setting, we include BLEURT and COMET as comparison points, which indicates InstructScore’s correlation compared to SOTA metrics at each task. However, we do not claim InstructScore can outperform BLEURT and COMET under the same training data and model size. For the second setting, we leverage fine-grained human annotation to meta-evaluate InstructScore’s outputs. We further include InstructScore’s precision and recall of its annotations and explanations to illustrate its usefulness to humans. Our human evaluation results indicate that, after refinement, 89.4% of InstructScore's output fields are correct, and it can identify 85.6% of error fields produced by the translation system. Furthermore, regarding explanations, 86.1% of InstructScore's output after refinement is deemed correct by humans, and it can explain 85.0% of errors produced by the translation system. We will revise our claim in the paper. \n\nQ.C: InstructScore to assist human evaluation:\n\nThanks for your great suggestion. We believe using InstructScore to support human labeling is a great idea and future work.  This would exceed the scope of this work and would require more additional budget to run human evaluation with professional translators (the standard in MT).\n\n| All fields | Before refinement | After refinement |\n| ---------- | ----------------- | ---------------- |\n| Precision  | 0.778             | 0.894            |\n| Recall     | 0.824             | 0.856            |\n\n| Explanation field | Before refinement | After refinement |\n| ----------------- | ----------------- | ---------------- |\n| Precision         | 0.756             | 0.861            |\n| Recall            | 0.819             | 0.850             |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe paper further utilizes LLM’s feedback as a syntetic data (human-like feedback) to improve NLG evaluation with human-like reasoning and error analysis.\n\n-\tThe authors discuss failure modes of explanation-based evaluation and provide solutions through the proposed metric.\n\nAlthough, I still have a concern for relying on GPT4’s feedback to meta-evaluate the provided list of failure modes (Table 1). But, if such feedback will be made publicly available as meta-evaluation data for future studies, then at least the authors help advancing current limitations in NLG evaluation field by sharing the data publicly.\n\n-\tThe paper demonstrates the successful utilization of the proposed evaluation metrics on smaller LLM (7B model compared to larger LLMs, such that 175B GPT3 and GPT4). \n\n-\tThe by-products: synthetic finetuning data and LLM’s feedback can be useful for future studies.\n\n-\tThe implication of the study can be expanded to tackle the subjectivity issue of human evaluation in NLG. For example, to increase the objectivity in human evaluation setup by simplifying the task to binary classification of error analysis.\n", "reasons_to_reject": "\n-\tThe proposed InstructScore is highly dependent on the synthetic training data from OpenAI GPT4 (to finetune LLAMA or the backbone of InstructScore) and GPT4’s feedback (to further finetune LLAMA given its failure modes).\n\n-\tThe utilization of the proposed automatic metric may be constrained by the API’s particular usage permission.\n\nCould the authors double-check whether the OpenAI license permission (restriction) allows users to use the outputs based on their API to compare with their models (GPT3 and GPT4)? – \n\nThis issue might not be the biggest concern if future studies are not using the proposed metric to evaluate GPT3 or GPT4 (or any OpenAI LLMs).\n\n-\tSome comparison is rather unfair and over-claimed.\n\nLine 076: Comparison with BLEURT. \nBLEURT scores are known to be noisy, which can be linked to the model capacity (BERT, a Transformers MLM model with less number of parameters and generalization capability – to be compared to LLMs with 7B parameter size in the current proposal). Even though BLEURT is supervisedly trained or aligned with constrained human ratings, I would not be surprised that the proposed LLM-based metric outperforms BLEURT, considering the benefit of scaling (model size and pretraining data) from proprietary API-based LLMs.\n\nThe metric evaluation (in Table 2) is also rather apple-to-orange comparison. Most of the metrics, such as BLEURT, BERTScore, BARTScore, etc are not automatic metrics that are optimized to reason the model performance with an error analysis. For each metric, the correlation level to the human evaluators is calculated by a different setup. For example, human ratings is often done by voting based on relative comparison, while in InstructScore, human is asked to focus more on binary classification of each error type.\n\nI would be more interested if in addition to the final correlation score level between automatic metrics and human raters, the authors also highlight the increase/decrease of human agreement level given the different human evaluation setup above (I see that in Appendix line 991, the agreement level between three raters is defined as 0.388). I would have an educational guess that supporting human evaluation with an error analysis objective may increase the agreement level between raters because the evaluators are becoming more objective, compared to when they are subjectively comparing between two different generation outputs based on predefined rating scale.    \n\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "\nThe paper proposes a new automatic metric with an error analysis and explanation capability for text generation tasks. The proposed metric is based on Llama-7B as a backbone, which is further finetuned on pseudo-generation outputs (synthetic data from GPT4). After a finetuning stage, the proposed InstructScore is further refined by GPT4 feedback to reduce the risks or failure modes in smaller LLM (Llama-7B) due to its constrained generalization capability compared to when utilizing larger LLMs.\n\nThe paper demonstrates that the proposed InstructScore is superior on seen tasks: Machine Translation (WMT22), Wikipedia triples-based text generation (WebNLG20), binary classification of image captioning (Flicker3K-CF) and common sense reasoning (CommonGen); and unseen task: keyword-based text generation (Bagel); compared to currently available automatic evaluation metrics in NLG. The evaluation is done by comparing the correlation level between automatic metrics and human evaluation outcomes.\n\nThe contribution of the paper is NLG engineering experiment with an extended analysis (failure mode analysis, ablation study of human evaluation before and after the refinement of the proposed InstructScore).\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0mBr3YjBGZ", "reviewer": "Reviewer_k7RG", "summary": "", "questions": "Can the method generalize to NLU tasks?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and suggestions.\n\nQ.A: English to German experiment\n\nWe also run experiments with InstructScore for English-to-German (see Figure 6). InstructScore outperforms all unsupervised with learned metrics except GPT3-based GEMBA metric.  \n\nQ.B: Extension to NLU\n\nApplying InstructScore to NLU is an interesting research direction. However, NLU mostly uses classification-based metrics, such as accuracy. The explanation for NLU tasks tend to focus on explaining the model behaviors after making predictions. Different from this work, our explanation focuses on explaining the evaluation process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The results are good and make a good fit into the current trend on LLMs. This is good research with LLMs and evaluations on generation tasks.", "reasons_to_reject": "The experiments on MT consider Zh-En only.", "questions_to_authors": "Can the method generalize to NLU tasks?", "paper_topic_and_main_contributions": "This paper proposes to fine-tune LLMs for explainable scoring on generation tasks. The correlation with human judgments is on-par with SOTA models and provides additional explanations on scoring for generation tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a metric that attempts to identify errors in generated text using LLaMA-7B. The authors show that their metric works across a variety of tasks and also validate with human judgments.

Reviewers uTpq and p5F9 were both impressed with the idea of using GPT-4 to help fine-tune LLaMA. Reviewer uTpq also points out that there's very nice work to show the alignment with human judgments. In the rebuttal the authors allayed some of the issues that the reviewers had by explicitly stating that they will clarify the pipeline and include the refinement table.
Decision: Accept-Main