File size: 3,719 Bytes
45caa02 1222138 45caa02 1222138 14a5ed8 1222138 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 |
---
license: bsd-3-clause
datasets:
- mocha
language:
- en
---
# Answer Overlap Module of QAFactEval Metric
This is the span scorer module, used in [RQUGE paper]() to evaluate the generated questions of the question generation task.
The model was originally used in [QAFactEval]() for computing the semantic similarity of the generated answer span, given the reference answer, context, and question in the question answering task.
It outputs a 1-5 answer overlap score. The scorer is trained on their MOCHA dataset (initialized from [Jia et al. (2021)]()), consisting of 40k crowdsourced judgments on QA model outputs.
The input to the model is defined as:
```
[CLS] cand. question [q] gold answer [r] pred answer [c] context
```
# Generation
You can use the following script to get the semantic similarity of the predicted answer given the gold answer, context, and question.
```
from transformers import AutoModelForSequenceClassification, AutoTokenizer
sp_scorer = AutoModelForSequenceClassification.from_pretrained('alirezamsh/quip-512-mocha')
tokenizer_sp = AutoTokenizer.from_pretrained('alirezamsh/quip-512-mocha')
sp_scorer.eval()
pred_answer = ""
gold_answer = ""
question = ""
context = ""
input_sp = f"{question} <q> {gold_answer} <r>" \
f" {pred_answer} <c> {context}"
inputs = tokenizer_sp(input_sp, max_length=512, truncation=True, \
padding="max_length", return_tensors="pt")
outputs = sp_scorer(input_ids=inputs["input_ids"], attention_mask=inputs["attention_mask"])
print(outputs)
```
# Citations
```
@inproceedings{fabbri-etal-2022-qafacteval,
title = "{QAF}act{E}val: Improved {QA}-Based Factual Consistency Evaluation for Summarization",
author = "Fabbri, Alexander and
Wu, Chien-Sheng and
Liu, Wenhao and
Xiong, Caiming",
booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
month = jul,
year = "2022",
address = "Seattle, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.naacl-main.187",
doi = "10.18653/v1/2022.naacl-main.187",
pages = "2587--2601",
abstract = "Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFactEval, that leads to a 14{\%} average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost.",
}
@misc{mohammadshahi2022rquge,
title={RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question},
author={Alireza Mohammadshahi and Thomas Scialom and Majid Yazdani and Pouya Yanki and Angela Fan and James Henderson and Marzieh Saeidi},
year={2022},
eprint={2211.01482},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
``` |