README.md

---
license: bsd-3-clause
datasets:
- mocha
language:
- en
---

# Answer Overlap Module of QAFactEval Metric


This is the span scorer module, used in [RQUGE paper]() to evaluate the generated questions of the question generation task.
The model was originally used in [QAFactEval]() for computing the semantic similarity of the generated answer span, given the reference answer, context, and question in the question answering task. 
It outputs a 1-5 answer overlap score. The scorer is trained on their MOCHA dataset (initialized from [Jia et al. (2021)]()), consisting of 40k crowdsourced judgments on QA model outputs.

The input to the model is defined as:
```
[CLS] cand. question [q] gold answer [r] pred answer [c] context
```


# Generation 

You can use the following script to get the semantic similarity of the predicted answer given the gold answer, context, and question.

```
from transformers import AutoModelForSequenceClassification, AutoTokenizer
sp_scorer = AutoModelForSequenceClassification.from_pretrained('alirezamsh/quip-512-mocha')
tokenizer_sp = AutoTokenizer.from_pretrained('alirezamsh/quip-512-mocha')
sp_scorer.eval()

pred_answer = ""
gold_answer = ""
question = ""
context = ""

input_sp = f"{question} <q> {gold_answer} <r>" \
                   f" {pred_answer} <c> {context}"

inputs = tokenizer_sp(input_sp, max_length=512, truncation=True, \
                                   padding="max_length", return_tensors="pt")

outputs = sp_scorer(input_ids=inputs["input_ids"], attention_mask=inputs["attention_mask"])
print(outputs)
```

# Citations

```
@inproceedings{fabbri-etal-2022-qafacteval,
    title = "{QAF}act{E}val: Improved {QA}-Based Factual Consistency Evaluation for Summarization",
    author = "Fabbri, Alexander  and
      Wu, Chien-Sheng  and
      Liu, Wenhao  and
      Xiong, Caiming",
    booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jul,
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.naacl-main.187",
    doi = "10.18653/v1/2022.naacl-main.187",
    pages = "2587--2601",
    abstract = "Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFactEval, that leads to a 14{\%} average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost.",
}

@misc{mohammadshahi2022rquge,
    title={RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question},
    author={Alireza Mohammadshahi and Thomas Scialom and Majid Yazdani and Pouya Yanki and Angela Fan and James Henderson and Marzieh Saeidi},
    year={2022},
    eprint={2211.01482},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```