Before revision:  In this work, we perform an extensive evaluation of a QA-based metric for summary content quality , calculating its performance with today's state-of-the-art models as well as estimating its potential upper-bound performance. We analyze a proposed metric, QAEval , which is more widely applicable than previous work.
After revision:  In this work, we propose a metric to evaluate the content quality of a summary using question-answering (QA). QA-based methods directly measure a summary's information overlap with a reference, making them fundamentally different than text overlap metrics. We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval , which is more widely applicable than previous work.