Package: qa-metrics
QA-Evaluation-MetricsQA-Evaluation-Metrics is a fast and lightweight Python package for evaluating question-answering models. It provides various basic metrics to assess the performance of QA models. Check out our paperPANDA, an efficient QA evaluation that retains competitive evaluation performance of transformer LLM models.InstallationTo install the package, run the following command:pipinstallqa-metricsUsageThe python package currently provides four QA evaluation metrics.Exact Matchfromqa_metrics.emimportem_matchreference_answer=["The Frog Prince","The Princess and the Frog"]candidate_answer="The movie\"The Princess and the Frog\"is loosely based off the Brother Grimm's\"Iron Henry\""match_result=em_match(reference_answer,candidate_answer)print("Exact Match: ",match_result)'''Exact Match:  False'''F1 Scorefromqa_metrics.f1importf1_match,f1_score_with_precision_recallf1_stats=f1_score_with_precision_recall(reference_answer[0],candidate_answer)print("F1 stats: ",f1_stats)'''F1 stats:  {'f1': 0.25, 'precision': 0.6666666666666666, 'recall': 0.15384615384615385}'''match_result=f1_match(reference_answer,candidate_answer,threshold=0.5)print("F1 Match: ",match_result)'''F1 Match:  False'''PANDA Matchfromqa_metrics.pedantimportPEDANTquestion="Which movie is loosley based off the Brother Grimm's Iron Henry?"pedant=PEDANT()scores=pedant.get_scores(reference_answer,candidate_answer,question)max_pair,highest_scores=pedant.get_highest_score(reference_answer,candidate_answer,question)match_result=pedant.evaluate(reference_answer,candidate_answer,question)print("Max Pair:%s; Highest Score:%s"%(max_pair,highest_scores))print("Score:%s; PANDA Match:%s"%(scores,match_result))'''Max Pair: ('the princess and the frog', 'The movie "The Princess and the Frog" is loosely based off the Brother Grimm\'s "Iron Henry"'); Highest Score: 0.854451712151719Score: {'the frog prince': {'The movie "The Princess and the Frog" is loosely based off the Brother Grimm\'s "Iron Henry"': 0.7131625951317375}, 'the princess and the frog': {'The movie "The Princess and the Frog" is loosely based off the Brother Grimm\'s "Iron Henry"': 0.854451712151719}}; PANDA Match: True'''print(pedant.get_score(reference_answer[0],candidate_answer,question))'''0.7122460127464126'''Transformer MatchOur fine-tuned BERT model is on 🤗Huggingface. Our Package also supports downloading and matching directly.distilroberta,distilbert,roberta, androberta-largeare also supported now! 🔥🔥🔥fromqa_metrics.transformerMatcherimportTransformerMatcherquestion="Which movie is loosley based off the Brother Grimm's Iron Henry?"tm=TransformerMatcher("bert")scores=tm.get_scores(reference_answer,candidate_answer,question)match_result=tm.transformer_match(reference_answer,candidate_answer,question)print("Score:%s; bert Match:%s"%(scores,match_result))'''Score: {'The Frog Prince': {'The movie "The Princess and the Frog" is loosely based off the Brother Grimm\'s "Iron Henry"': 0.6934309}, 'The Princess and the Frog': {'The movie "The Princess and the Frog" is loosely based off the Brother Grimm\'s "Iron Henry"': 0.7400551}}; TM Match: True'''If you find this repo avialable, please cite:@misc{li2024panda,title={PANDA (Pedantic ANswer-correctness Determination and Adjudication):Improving Automatic Evaluation for Question Answering and Text Generation},author={Zongxia Li and Ishani Mondal and Yijun Liang and Huy Nghiem and Jordan Lee Boyd-Graber},year={2024},eprint={2402.11161},archivePrefix={arXiv},primaryClass={cs.CL}}Updates[01/24/24] 🔥 The full paper is uploaded and can be accessedhere. The dataset is expanded and leaderboard is updated.Our Training Dataset is adapted and augmented fromBulian et al. Ourdataset repoincludes the augmented training set and QA evaluation testing sets discussed in our paper.Now our model supportsdistilroberta,distilbert, a smaller and faster matching model than Bert!Now our model supportsroberta,roberta-large, a larger and more robust matching model than Bert!LicenseThis project is licensed under theMIT License- see the LICENSE file for details.ContactFor any additional questions or comments, please contact [zli12321@umd.edu].