|
--- |
|
license: bsd-3-clause |
|
datasets: |
|
- mocha |
|
language: |
|
- en |
|
--- |
|
|
|
# Answer Overlap Module of QAFactEval Metric |
|
|
|
|
|
This is the span scorer module, used in [RQUGE paper]() to evaluate the generated questions of the question generation task. |
|
The model was originally used in [QAFactEval]() for computing the semantic similarity of the generated answer span, given the reference answer, context, and question in the question answering task. |
|
It outputs a 1-5 answer overlap score. The scorer is trained on their MOCHA dataset (initialized from [Jia et al. (2021)]()), consisting of 40k crowdsourced judgments on QA model outputs. |
|
|
|
The input to the model is defined as: |
|
``` |
|
[CLS] question [q] gold answer [r] pred answer [c] context |
|
``` |
|
|
|
|
|
# Generation |
|
|
|
You can use the following script to get the semantic similarity of the predicted answer given the gold answer, context, and question. |
|
|
|
``` |
|
from transformers import AutoModelForSequenceClassification, AutoTokenizer |
|
sp_scorer = AutoModelForSequenceClassification.from_pretrained('alirezamsh/quip-512-mocha') |
|
tokenizer_sp = AutoTokenizer.from_pretrained('alirezamsh/quip-512-mocha') |
|
sp_scorer.eval() |
|
|
|
pred_answer = "" |
|
gold_answer = "" |
|
question = "" |
|
context = "" |
|
|
|
input_sp = f"{question} <q> {gold_answer} <r>" \ |
|
f" {pred_answer} <c> {context}" |
|
|
|
inputs = tokenizer_sp(input_sp, max_length=512, truncation=True, \ |
|
padding="max_length", return_tensors="pt") |
|
|
|
outputs = sp_scorer(input_ids=inputs["input_ids"], attention_mask=inputs["attention_mask"]) |
|
print(outputs) |
|
``` |
|
|
|
# Citations |
|
|
|
``` |
|
@inproceedings{fabbri-etal-2022-qafacteval, |
|
title = "{QAF}act{E}val: Improved {QA}-Based Factual Consistency Evaluation for Summarization", |
|
author = "Fabbri, Alexander and |
|
Wu, Chien-Sheng and |
|
Liu, Wenhao and |
|
Xiong, Caiming", |
|
booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies", |
|
month = jul, |
|
year = "2022", |
|
address = "Seattle, United States", |
|
publisher = "Association for Computational Linguistics", |
|
url = "https://aclanthology.org/2022.naacl-main.187", |
|
doi = "10.18653/v1/2022.naacl-main.187", |
|
pages = "2587--2601", |
|
abstract = "Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFactEval, that leads to a 14{\%} average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost.", |
|
} |
|
|
|
@misc{mohammadshahi2022rquge, |
|
title={RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question}, |
|
author={Alireza Mohammadshahi and Thomas Scialom and Majid Yazdani and Pouya Yanki and Angela Fan and James Henderson and Marzieh Saeidi}, |
|
year={2022}, |
|
eprint={2211.01482}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
