High Quality HumanEvaluation of NLG
Ehud Reiter
University of Aberdeen

Dr. Ehud Reiter, Computing Science, University of Aberdeen

1

Contents▪ High quality human evaluation▪ Old work
▪ Detecting accuracy errors▪ Evaluating real-world utility of summaries▪ Enhancing replicability▪ Final thoughts

Dr. Ehud Reiter, Computing Science, University of Aberdeen

2

Evaluation in Medicine
Focus on high-quality expensive evaluations ofclinical outcomes (RCTs)▪ Sometimes can use cheaper/quicker surrogate
endpoints for clinical outcome▪

» Eg, viral load instead of mortality» Much quicker/easier to measure» Only use if high correlation with clinical outcome» Best studies avoid surrogate, use clinical outcomes

Dr. Ehud Reiter, Computing Science, University of Aberdeen

3Evaluation in NLP▪ Dominated by metrics (BLEU, etc)

» Metrics are surrogate endpoints» Used even if limited corr with human eval
» Used everywhere, including top studies▪ Human evaluations often limited» Random crowdworkers as subjects» Measure opinion rather than task outcome▪ Need more high-quality human evals» Analogous to RCT in medicine?
Dr. Ehud Reiter, Computing Science, University of Aberdeen

4

BLEU-human corr in NLG▪ Meta-analysis across papers in ACL

Anthology (Reiter 2018)

Dr. Ehud Reiter, Computing Science, University of Aberdeen

5

Human eval: subjects▪ Most human evaluations in NLP use

crowdworkers (eg Mechanical Turk)▪ Freitag et al (2021): WMT human evals(based on monolingual crowdworkers)
do NOT correlate well with structured
evaluations by professional translators.

Dr. Ehud Reiter, Computing Science, University of Aberdeen

6

Human eval: opinion vs outcome▪ Most human evaluations in NLG solicit

ratings or opinions▪ Usually what we really care about iswhether NLP system helps people» Task outcome (extrinsic eval)▪ Rating/opinions may NOT correlate withtask effectiveness» Eg Law et al (2005)
Dr. Ehud Reiter, Computing Science, University of Aberdeen

7

Vision: High Qual Human Eval
▪ Do high-quality human eval of NLP» Subjects with domain knowledge» Objective/task outcome instead of opinion▪ Use these for key experiments▪ Use these to ground/validate metrics

and cheaper human evals

Dr. Ehud Reiter, Computing Science, University of Aberdeen

8

Contents▪ High quality human evaluation▪ Old work
▪ Detecting accuracy errors▪ Evaluating real-world utility of summaries▪ Enhancing replicability▪ Final thoughts

Dr. Ehud Reiter, Computing Science, University of Aberdeen

9

Smoking cessation▪ NLG system generated stop-smoking

leaflets based on user questionnaire▪ Evaluated in medical-grade RCT» 2500 subjects!
▪ Result: Simple fixed letter as effective

as NLG letters▪ Reiter et al (2003)
Dr. Ehud Reiter, Computing Science, University of Aberdeen

10Clinical Decision Support▪ NLG system summarized patient datafor babies in neonatal ICU, to help
clinicians decide on interventions▪ Evaluation» show clinicians NLG sum and visualisations» asking them to make treatment decisions» Compare decisions against gold stand▪ Result: small diff, not stat significant▪ Portet et al (2009)
Dr. Ehud Reiter, Computing Science, University of Aberdeen

11

Nursing Shift Handover▪ NLG system generated nurse shifthandover rep, for NICU babies▪ Eval:» System deployed, used on ward
» Researcher vets texts for errors» Nurses say whether test useful▪ Result: No serious errors, useful▪ Hunter et al (2012)
Dr. Ehud Reiter, Computing Science, University of Aberdeen

12

Contents▪ High quality human evaluation▪ Old work
▪ Example: Detecting accuracy errors▪ Evaluating real-world utility of summaries▪ Enhancing replicability▪ Final thoughts

Dr. Ehud Reiter, Computing Science, University of Aberdeen

13

Evaluating Accuracy▪ Accuracy (hallucination) is big problem» Especially in neural NLG» Especially in longer texts▪ Users expect NLG texts to be accurate!

» Lose trust if sys produces inaccurate texts▪ How do we evaluate accuracy?

▪ Part of Craig Thomson’s PhD

Dr. Ehud Reiter, Computing Science, University of Aberdeen

14

Craig’s work
▪ Accuracy of summaries of basketballgames» Produced from “box score” game data
» 300 words on average

Dr. Ehud Reiter, Computing Science, University of Aberdeen

15

Team & Player Data
TEAMW

L

H1-PTS

H2-PTSPTS

FG%

Grizzlies

50

46

56

102.486

Suns

3252

3991.559

PlayerTEAM

PTS

REB

ASTBLKSTL

Marc Gasol Grizzlies

185604IsaiahThomas

151201Suns

Dr. Ehud Reiter, Computing Science, University of Aberdeen

16

Partial game summary
The Memphis Grizzlies (5-2) defeated the Phoenix
Suns (3-2) Monday 102-91 at the Talking StickResort Arena in Phoenix. The Grizzlies had a strong
first half where they out-scored the Suns 59-42.
Marc Gasol scored 18 points, leading the Grizzlies.
Isaiah Thomas added 15 points, he is averaging 19
points on the season so far.

Dr. Ehud Reiter, Computing Science, University of Aberdeen

17

Partial summary with errors
The Memphis Grizzlies (5-2) defeated the Phoenix
Suns (3-2) Monday 102-91 at the Talking StickResort Arena in Phoenix. The Grizzlies had astrong first half where they out-scored the Suns
59-42. Marc Gasol scored 18 points, leading theGrizzlies. Isaiah Thomas added 15 points, he isaveraging 19 points on the season so far.

Dr. Ehud Reiter, Computing Science, University of Aberdeen

18

Mistake categories
Name

Player, Team, day of week, etc.

NumberNumber, in any form.

Word

Word or phrase that is not
Name/Number.
Something that is contextually wrong.

Context

Not
Impossible/time-consuming to check.
Checkable
Other
Any other error.

Dr. Ehud Reiter, Computing Science, University of Aberdeen

19Gold standard protocol▪ High-quality human eval to find mistakes» Thomson and Reiter (2020)▪ Subjects» Selected Mechanical Turk workers» Know basketball, do well on vetting task
▪ Task» Find and categorise mistakes» More objective than 1-5 accuracy rating
Dr. Ehud Reiter, Computing Science, University of Aberdeen

20Gold standard protocol▪ Procedure» 3 Turkers annotate each text
» Researcher combines (majority opinion)▪ Process worked» High interannotator agreement» Various checks, including with domain experts▪ Expensive» US$30 for each 300-word summary
Dr. Ehud Reiter, Computing Science, University of Aberdeen

21

Cheaper Eval: Shared Task
▪ Created shared task to find cheaper andquicker techniques» Should correlate with gold standard▪ Cheaper human eval
▪ Metrics▪ Thomson and Reiter (2021)

Dr. Ehud Reiter, Computing Science, University of Aberdeen

22Quicker Human Eval
▪ Garneau and Lamontagne (2021):

quicker and cheaper human eval
» Used metric to pre-annotate simple
mistakes (not complex ones)» Significant reduction in time/cost» High agreement with gold stand– Recall of .84– Precision of .88
Dr. Ehud Reiter, Computing Science, University of Aberdeen

23

Metrics▪ Kasner et al (2021) proposed metric» Generate synthetic data with rule-based NLG» Train language model to detect errors (usingreal and synthetic data)▪ Works well for simpler errors▪ Not great for complex errors

Dr. Ehud Reiter, Computing Science, University of Aberdeen

24Kasner et al metric
TypeRecallPrecision

Name0.75

0.85

Number0.78

0.75

Word0.51

0.48

Context0--

Not checkable0--

Other0--Overall

0.69

0.76

Dr. Ehud Reiter, Computing Science, University of Aberdeen

25Summary▪ Identify area where good eval needed» Evaluating accuracy is very important▪ Created gold-standard human eval

» US$30 per text (expensive)▪ Used gold standard to development

metrics and cheaper human eval

Dr. Ehud Reiter, Computing Science, University of Aberdeen

26

Contents▪ High quality human evaluation▪ Old work
▪ Detecting accuracy errors▪ Evaluating real-world utility of summaries▪ Enhancing replicability▪ Final thoughtrs

Dr. Ehud Reiter, Computing Science, University of Aberdeen

27

Evaluating Utility▪ How evaluate if generated texts helpusers do tasks better or more quickly?
» Depends on task (and user)▪ Part of Francesco Moramarco’s PhD» Task: summarizing patient-doctor
consultations» working with Babylon HealthDr. Ehud Reiter, Computing Science, University of Aberdeen

28

Use case▪ GP (doctor) talks to patient 5-10 mins

» Called “consultation”▪ Needs to write summary of consultation» For medical records, patient can see▪ Currently done by GP▪ Goal: NLP system gen draft summary▪ Doctor “post-edits” to fix mistakes
Dr. Ehud Reiter, Computing Science, University of Aberdeen

29

Example
Consultation
Doctor: Hello? Good morning, Tim. Um, how can I help you thismorning?
Patient: Um, so I’m having some, some pain, uh, in my tummy,
like the lower part of my tummy. Um and I’ve just been feeling,quite, hot and sweaty.
Doctor: OK. Right, I’m sorry to hear that. When, when did yoursymptoms all start?
Patient: About two days ago.

Summary
Two days of lower abdominal pain.
Dr. Ehud Reiter, Computing Science, University of Aberdeen

30How measure usefulness?
▪ Time spent post-editing NLP summary?

» Compared to time to write from scratch▪ Quality of post-edited summary?

» determined by experienced clinician▪ Number of mistakes in NLP summary▪ Doctor satisfaction?
▪ Impact on workflow?
Dr. Ehud Reiter, Computing Science, University of Aberdeen

31Not just averages▪ Differences between doctors» Post-editing time (and what is edited)» Satisfaction▪ Worst-case as well as average case» No tolerance for medically misleadingsummaries

Dr. Ehud Reiter, Computing Science, University of Aberdeen

32

High Quality Human Eval
▪ Developing protocol▪ Current version» Doctors write their own summary» Doctors shown NLP summary» Doctors post-edit NLP summary to make itacceptable» Measure time to post-edit» Also identify accuracy problems in NLP
Dr. Ehud Reiter, Computing Science, University of Aberdeen

33

High Quality Human Eval
▪ Moramarco et al (2021) describes firstversion of protocol▪ Refined since» Post-edit UI is critical

Dr. Ehud Reiter, Computing Science, University of Aberdeen

34

Correlation with exist metrics▪ Preliminary work, not yet published▪ Levenshtein (character edit distance)

better than ROUGE, BertScore, etc
MetricCorr with post-edit time

Corr with num error

ROUGE-2

0.38

0.73

METEOR

0.41

0.71

BertScore

0.50

0.74

Levenshtein

0.55

0.76

Dr. Ehud Reiter, Computing Science, University of Aberdeen

35

Levenshtein is best?
▪ Surprising that Levenstein (characterlevel edit distance is best)▪ Bertscore, etc, mostly justified by corrwith crowdworker opinion (eg, WMT)» Freitag: Corr between WMT and proftranslators can be negative…
» Good corr with WMT not guarantee goodcorr with high-quality outcome-based
human evals!
Dr. Ehud Reiter, Computing Science, University of Aberdeen

36Summary▪ Working towards high quality eval ofreal-world utility» Work in progress» Expensive (need lots of doctor time)▪ Explore which metrics have best corr

» So far 1960s Leven dist beats all of themodern metrics used in NLP

Dr. Ehud Reiter, Computing Science, University of Aberdeen

37

Contents▪ High quality human evaluation▪ Old work
▪ Detecting accuracy errors▪ Evaluating real-world utility of summaries▪ Enhancing replicability▪ Final thoughtrs

Dr. Ehud Reiter, Computing Science, University of Aberdeen

38

Reproducibility▪ Scientific experiments (including eval ofAI systems) should be reproducible!
▪ If someone else does the same exper,
should get similar results» Not identical if people are involved▪ Major concern in many areas of science

Dr. Ehud Reiter, Computing Science, University of Aberdeen

39

Reproducibility in NLP▪ Some work on reproducing automatic(metric) evals» Ensure all details published, data sets andsoft available, preprocessing clear, etc▪ What about reproducing human eval?

» Poorly understood

Dr. Ehud Reiter, Computing Science, University of Aberdeen

40ReproGen: Human NLG Eval
▪ Shared task where people reproduced

human evaluations of NLG systems» Belz et al (2021)▪ Mixed results» Some reproductions had similar results,
some did not» Unclear why (small sample size) (4 replic)

Dr. Ehud Reiter, Computing Science, University of Aberdeen

41

ReproHum▪ New EPSRC project on reproducibilityof human evaluations of NLP» Will start in early 2022
▪ Much larger scale than ReproGen» 20 partner labs will reproduce a selected
set of NLP evaluations» Identify key factors for replication» Develop theoretical framework» Make recommendations
Dr. Ehud Reiter, Computing Science, University of Aberdeen

42ReproHum▪ New partner labs are welcome!
▪ Contact Anya Belz (PI) or me ifinterested

Dr. Ehud Reiter, Computing Science, University of Aberdeen

43

Contents▪ High quality human evaluation▪ Old work
▪ Detecting accuracy errors▪ Evaluating real-world utility of summaries▪ Enhancing replicability▪ Final thoughts

Dr. Ehud Reiter, Computing Science, University of Aberdeen

44

Final Thoughts▪ Too much focus on quick/cheap evals inNLP!
▪ If we’re doing science (as opposed tokeeping score in contests), we needhigh-quality human evals» Ground/validate metrics» Confidence in key findings

Dr. Ehud Reiter, Computing Science, University of Aberdeen

45

Final Thoughts▪ I’d love to see more high-quality humanevaluations in NLP▪ Feel free to contact me if I can help!

Dr. Ehud Reiter, Computing Science, University of Aberdeen

46

References
Belz et al (2021). The ReproGen Shared Task on
Reproducibility of Human Evaluations in NLG: Overview andResults. Proc of INLG-2021
Freitag et al (2021). Experts, Errors, and Context: A LargeScale Study of Human Evaluation for Machine Translation. Arxiv
Garneau and Lamontagne (2021). Shared Task in Evaluating
Accuracy: Leveraging Pre-Annotations in the Validation Process.
Proc of INLG-2021
Hunter et al (2012). Automatic generation of natural language
nursing shift summaries in neonatal intensive care: BT-Nurse.
Artificial Intelligence in Medicine 56:157–172
Kasner et al (2021). Text-in-Context: Token-Level Error
Detection for Table-to-Text Generation. Proc of INLG-2021
Dr. Ehud Reiter, Computing Science, University of Aberdeen47

References
Law et al (2005). A comparison of graphical and textual
presentations of time series data to support medical decision
making in the neonatal intensive care unit Journal of Clinical
Monitoring and Computing 19:183-94.
Moramarco et al (2021). A preliminary study on evaluating
Consultation Notes with Post-Editing. Proc of EACL-2021
workshop on Human Evaluation of NLP Systems
Portet et al (2009). Automatic Generation of Textual Summaries
from Neonatal Intensive Care Data. Artificial Intelligence 173:789816
Reiter et al (2003). Lessons from a Failure: Generating Tailored
Smoking Cessation Letters. Artificial Intelligence 144:41-58

Dr. Ehud Reiter, Computing Science, University of Aberdeen

48

References
Reiter (2018). A Structured Review of the Validity ofBLEU. Computational Linguistics 44:393-401
Thomson and Reiter (2020). A Gold Standard Methodology forEvaluating Accuracy in Data-To-Text Systems. Proc of INLG-2020
Thomson and Reiter (2021). Generation Challenges: Results ofthe Accuracy Evaluation Shared Task. Proc of INLG-2021

Dr. Ehud Reiter, Computing Science, University of Aberdeen

49