Package: rquge
RQUGE🤗: Reference-Free Metric for Evaluating Question Generation by Answering the QuestionWe propose RQUGE, aReference-freeQUestionGenerationEvaluation metric that can compute the quality of the candidate question without requiring the access to the reference question. Given the corresponding context and answer span, our metric calculates the acceptability score by applying a general question-answering module, followed by a span scorer. You can find more detail inthe paper(ACL2023).ContentsHuggingface Evaluate🤗InstallationCalculationCitationHuggingFace Evaluation 🤗RQUGE score is available onHuggingface Evaluate. It can be used as:from evaluate import load
rqugescore = load("alirezamsh/rquge")
generated_questions = ["how is the weather?"]
contexts = ["the weather is sunny"]
answers = ["sunny"]
results = rquge.compute(generated_questions=generated_questions, contexts=contexts, answers=answers)
print(results["mean_score"])
>>> [5.05]The demo and further details are also available onhere.InstallationYou should have the following packages:transformerspytorchsentencepieceInstall from pypi with pip bypip install rqugeInstall latest unstable version from the master branch on Github by:pip install git+https://github.com/alirezamshi/RQUGEInstall it from the source by:git clone https://github.com/alirezamshi/RQUGE
cd RQUGE
pip install .Note: you need to download the pre-trained model for the span scorer module (available onHuggingface🤗alirezamsh/quip-512-mocha):wget https://storage.googleapis.com/sfr-qafacteval-research/quip-512-mocha.tar.gz
tar -xzvf quip-512-mocha.tar.gz
rm quip-512-mocha.tar.gzRQUGE CalculationPython FunctionThe RQUGE class is provided inrquge_score/scorer.py. We also provide a python function inrquge_score_cli/scorer_cli.pyto use different features of RQUGE metric.Command Line Interface (CLI)We provide a command line interface (CLI) of RQUGE, you can use it as follows:rquge --input_type #MODE --sp_scorer_path #PATH_TO_SPAN_SCORER --qa_model_path #PATH_TO_QA_MODEL --context #CONTEXT_FILE --question #QUESTION --answer #ANSWER --output_path #OUTPUT

#MODE: The type of input (sample or offline). In the sample mode, "--context", "--question", and "--answer" commands contain string, while in offline mode, they contain path to files including contexts, corresponding questions and answers
#PATH_TO_SPAN_SCORER: path to the local checkpoint of span scorer model or "alirezamsh/quip-512-mocha"
#PATH_TO_QA_MODEL: name of QA model on Huggingface or local path 
#CONTEXT_FILE: a text file containing one context per line (directly in the input in "sample" mode)
#QUESTION_FILE: a text file containing one question per line (directly in the input in "sample" mode)
#ANSWER_FILE: a text file containing one answer per line (directly in the input in "sample" mode)
#OUTPUT: local path for saving RQUGE scores for each (context,question,answer) pairHere is a sample score computation for the interactive moderquge --input_type sample --sp_scorer_path ckpt/quip-512-mocha --qa_model_path 'allenai/unifiedqa-v2-t5-large-1363200' --context "the weather is sunny" --question "how is the weather?" --answer sunnyNote:the rquge score is between 1 to 5.CitationIf you use this code for your research, please cite the following work:@misc{mohammadshahi2022rquge,
    title={RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question},
    author={Alireza Mohammadshahi and Thomas Scialom and Majid Yazdani and Pouya Yanki and Angela Fan and James Henderson and Marzieh Saeidi},
    year={2022},
    eprint={2211.01482},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}Have a question not listed here? Opena GitHub Issueor
send us anemail.