Towards Trustworthy Evaluation andInterpretation:
Summarization & Dialogue
Chien-Sheng (Jason) Wu

2021.11.10 Eval4NLP @ EMNLPSalesforce AI Research Team
Palo Alto, CAOutline●Re-evaluating metrics and models for text summarization○●

SummEval: Re-evaluating Summarization Evaluation

Visualize and open the black box for model prediction○

SUMMVIS: Interactive Visual Analysis of Models, Data, and Evaluation for Text
Summarization●

Factual Consistency in Summarization and Dialogue

SummEval: Re-evaluating
Summarization Evaluation
Fabbri et al., 2020

Summarization Evaluation●

ROUGE-N (Lin, 2004): Lexical overlap with a reference summary (or summaries)

Need for Consistent Summarization Evaluation●Model evaluation○

Recent papers vastly differ in their evaluation protocol (e.g., different
ROUGE packages) as noted in Hardy et al. (2019)○●

Most papers compare to only several other models

Metric evaluation○

New Metrics proposed but not widely adopted.

○Metrics are evaluated on DUC and TAC shared tasks, notrepresentative of modern systems (Peyrard, 2019).

Re-evaluating Metrics and Models●Re-evaluating metrics○

14 automatic evaluation metrics○Toolkit with extensible and uniﬁed API○

Largest and most diverse, in terms of model types, collection of humanjudgments of model-generated summaries on the CNN/DM dataset●

Re-evaluating models○

Consistently benchmark 23 recent summarization models (2017 to 2019)○

Largest collection of summaries on the CNN/DM news dataset for easiercomparison.

Evaluation Metrics●

ROUGE-based: ROUGE (Lin, 2004b); ROUGE-WE (Ng and Abrecht, 2015); S3 (Peyrardet al., 2017)●Contextual Embedding-Based: BertScore (Zhang* et al., 2020), MoverScore (Zhao etal., 2019), Sentence Mover’s Similarity (Clark et al., 2019); SummaQA (Scialom et al.,2019)○●

Reference-less: BLANC (Vasilyev et al., 2020); SUPERT (Gao et al., 2020)

Machine translation, text generation metrics: BLEU (Papineni et al., 2002); CHRF(Popović, 2015); METEOR (Lavie and Agarwal, 2007); CIDEr (Vedantam et al., 2015)●

Data Statistics: Grusky et al. (2018)

Summarization Models
Extractive Models
NEUSUM (Zhou et al., 2018)

BanditSum (Dong et al., 2018)LATENT (Zhang et al., 2018b)

REFRESH (Narayan et al., 2018)

RNES (Wu and Hu, 2018)JECS (Xu and Durrett, 2019)STRASS (Bouscarrat et al., 2019)

Summarization Models
Non-pretrained Abstractive Models
Pointer Generator (See et al., 2017)

ROUGESal (Pasunuru and Bansal, 2018)

Fast-abs-rl (Chen and Bansal, 2018)

Multi-task (Ent + QG ) (Guo et al., 2018)

Bottom-Up (Gehrmann et al., 2018)

Closed book decoder (Jiang and Bansal, 2018)

Improve-abs (Kryściński et al., 2018)

SENECA (Sharma et al., 2019)

Unified-ext-abs (Hsu et al., 2018)

NeuralTD (Böhm et al., 2019)

Summarization Models
Pretrained Abstractive Models
T5 (Raffel et al., 2019)

UniLM (Dong et al., 2019)

BertSum-abs (Liu and Lapata, 2019)

BART (Lewis et al., 2019)

GPT-2 (Ziegler et al., 2019)

Pegasus (Zhang et al., 2019a)

Human Judgments●

100 articles from CNN/DM dataset; 16 models; 3 expert and 5 crowdsourced judgments●4 quality dimensions (rated from 1 to 5, higher better)●○

Coherence - the structure and organization of all summary sentences○Consistency - the factual alignment between summary and input○

Fluency - the grammatical quality of individual sentences○

Relevance - selection of important content from the source.

Two rounds of expert annotations for better agreement (0.71 Krippendorf's alpha),
problems with crowdsourced judgments

Problems with Crowdsourced Judgements
ExpertCrowdworker

Metric Re-evaluation●

Kendall’s tau rank for system-level ranking as inLouis and Nenkova, 201313

Metric Re-evaluation●

Strong correlation with consistency and ﬂuency perhaps
due to extractive nature of dataset.

Metric Re-evaluation●Weaker correlations potentially to inherent
subjectiveness of the dimension and the difﬁcultyof collecting consistent human annotations

Model Re-evaluation●

Scores from 1 to 5 (best)16Model Re-evaluation●

Reference summaries
are far from ideal

17

Model Re-evaluation●

Reference summaries
are far from ideal●

Improvements withpretrained models

18Model Re-evaluation●

Reference summaries
are far from ideal●

Improvements withpretrained models●

Coherence and relevancecan still be improved on
this dataset

19

SummEval Toolkit●Install> pip install summ-eval●Import> from summ_eval.rouge_metric import RougeMetric> rouge = RougeMetric()●Evaluate!
> summaries = ["This is one summary", "This is another summary"]> references = ["This is one reference", "This is another"]> rouge_dict = rouge.evaluate_batch(summaries, references)SummVis: Interactive Visual
Analysis of Models, Data, andEvaluation
Vig et al., 2021

SummVis: Motivation●Novel neural architectures, training
strategies, and the availability oflarge-scale corpora haven been the driving
force behind recent progress inabstractive text summarization.

●However, due to the black-box nature ofneural models, uninformative evaluation
metrics, and scarce tooling for modeland data analysis, the true performanceand failure modes of summarization
models remain largely unknown.

Tool Design(a) Conﬁguration panel(b) Source document (or referencesummary)(c) Generated summaries (and/orreference summary)(d) Scroll bar with global view ofannotations.

Text Comparison●

Lexical Overlap○●N-grams

Semantic Overlap○

Cosine similarity between word embeddings■■●

Static word embeddings provided by spaCy(Honnibal et al., 2020)
Contextual embeddings from a pretrained
RoBERTa (Liu et al., 2019) modelTaxonomy○

Two dimensions, four quadrants of behavior

Tool Design●

Colored underlines align n-grams between source document and the selectedsummary.

●Novel words in the summary that do not appear in the source document are bolded,
while novel entities are bolded in red.

●Stopwords are grayed out and are not used in the matching algorithms.

●

Dotted underlines indicate tokens that are semantically similar to a token in the source
document (above the threshold speciﬁed in the conﬁguration panel).

●The user may hover over a token to see the most semantically similar tokens in thesource document, or click on the token to auto-scroll the source document to the mostsimilar token.

Case Study: Debugging Hallucination

Case Study: Similarity

Case Study: Similarity

Case Study: Similarity

Case Study: Similarity

Case Study: Similarity

https://github.com/robustness-gym/summvis

Factual Consistency inSummarization / Dialogue
Preprint / OngoingFactual Evaluation: Summarization
1.

Many interesting human evaluation datasets are being collected upon which evaluation metrics can be compareda.
FRANK factuality benchmarkb.
FFCI: for focus, faithfulness, coherence, informativenessc.
SummEval: coherence, consistency, fluency, relevance
FALKE, Go Figure!, FEQA, QAGs, FactCC, On Faithfulness and Factuality, Entity-level factual consistency,
ScarecrowDifferences in metric implementations lead to different conclusionsa.
QA > NLI: FEQAb.
NLI > QA: On Faithfulnessc.
BertScore-based metrics > QA: FFCIi.
For each summary sentence, uses average of top 3 sentences in the source. Then average oversummary sentences.
d.
QA > BertScore-based metrics: QuestEval
i.
Likely uses reference-based BertScoree.
BertScore and FactCC perform best: FRANKi.
BertScore over source article but doesn’t do sentence division/averaging over top sentencesd.

2.

Factual Evaluation: Summarization●Are QA/QG approaches sensitive to diﬀerent QA/QG models?
○●MixQG: Neural Question Generation with Mixed Answer Types

What is the best way to select NLI-style comparison?
○Average? Top-K? Sentence-level or token-level?

●Which metric is better under which setting?

●How to combine and leverage theses metrics for better summaries?

Factual Evaluation: Dialogue●

Fact-checking has been explored to verify formal
single-sentence claims instead of casualconversational claims.

●

Not all responses in a dialogue are carrying veriﬁableinformation.

●

Kim et al. (2021) proposed the task of veriﬁcation ofcolloquial claims which are stylistic modiﬁcations of
the FEVER (Thorne et al., 2018) claims. However,their generated claims are not contextualized within
dialogue.

Factual Evaluation: Dialogue●●●

Release a new evaluation dataset for factual
evaluation in dialogue.
Three sub-tasks: 1) veriﬁable claim detection, 2)
evidence retrieval, and 3) claim veriﬁcation.
Point out the weaknesses of existing fact-checking
models trained on non-dialogue domains.

wu.jason@salesforce.com
Twitter: @SFResearch @jasonwu0731

Thank You