TrueTeacher: Learning Factual Consistency Evaluation with Large Language
  Models
Authors: name: Zorik Gekhman; name: Jonathan Herzig; name: Roee Aharoni; name: Chen Elkind; name: Idan Szpektor
Published: 2023-05-18 17:58:35

Abstract: Factual consistency evaluation is often conducted using Natural Language
Inference (NLI) models, yet these models exhibit limited success in evaluating
summaries. Previous work improved such models with synthetic training data.
However, the data is typically based on perturbed human-written summaries,
which often differ in their characteristics from real model-generated summaries
and have limited coverage of possible factual errors. Alternatively, large
language models (LLMs) have recently shown promising results in directly
evaluating generative tasks, but are too computationally expensive for
practical use. Motivated by these limitations, we introduce TrueTeacher, a
method for generating synthetic data by annotating diverse model-generated
summaries using a LLM. Unlike prior work, TrueTeacher does not rely on
human-written summaries, and is multilingual by nature. Experiments on the TRUE
benchmark show that a student model trained using our data, substantially
outperforms both the state-of-the-art model with similar capacity, and the LLM
teacher. In a systematic study, we compare TrueTeacher to existing synthetic
data generation methods and demonstrate its superiority and robustness to
domain-shift. Using the the mFACE dataset, we also show that our method
generalizes to multilingual scenarios. Finally, we release a large-scale
synthetic dataset with 1.4M examples generated using TrueTeacher.