Package: factsumm
FactSumm: Factual Consistency Scorer for Abstractive SummarizationFactSummis a toolkit that scoresFactualy ConsistencyforAbstract SummarizationWithout fine-tuning, you can simply apply a variety of downstream tasks to boththe source articleandthe generated abstractive summaryFor example, by extractingfact triplesfrom source articles and generated summaries, you can verify that generated summaries correctly reflect source-based facts (See image above)InstallationWith andPython 3(>= 3.8), you can installfactsummsimply usingpip:pipinstallfactsummOr you can installFactSummfrom source repository:gitclonehttps://github.com/huffon/factsummcdfactsumm
pipinstall.Usage>>>fromfactsummimportFactSumm>>>factsumm=FactSumm()>>>article="Lionel Andrés Messi (born 24 June 1987) is an Argentine professional footballer who plays as a forward and captains both Spanish club Barcelona and the Argentina national team. Often considered as the best player in the world and widely regarded as one of the greatest players of all time, Messi has won a record six Ballon d'Or awards, a record six European Golden Shoes, and in 2020 was named to the Ballon d'Or Dream Team.">>>summary="Lionel Andrés Messi (born 24 Aug 1997) is an Spanish professional footballer who plays as a forward and captains both Spanish club Barcelona and the Spanish national team.">>>factsumm(article,summary,verbose=True)<SourceEntities>LineNo.1:[[('Lionel Andrés Messi','PERSON'),('24 June 1987','DATE'),('Argentine','NORP'),('Spanish','NORP'),('Barcelona','ORG'),('Argentina','GPE')]]LineNo.2:[[('one','CARDINAL'),('Messi','PERSON'),('six','CARDINAL'),("Ballon d'Or",'WORK_OF_ART'),('European Golden Shoes','WORK_OF_ART'),('2020','DATE'),("Ballon d'Or Dream Team",'WORK_OF_ART')]]<SummaryEntities>LineNo.1:[[('Lionel Andrés Messi','PERSON'),('24 Aug 1997','DATE'),('Spanish','NORP'),('Barcelona','ORG')]]<SourceFacts>('Lionel Andrés Messi','per:origin','Argentine')('Lionel Andrés Messi','per:employee_of','Barcelona')('Barcelona','org:country_of_headquarters','Spanish')('Lionel Andrés Messi','per:date_of_birth','24 June 1987')('Lionel Andrés Messi','per:countries_of_residence','Argentina')('Spanish','org:top_members/employees','Lionel Andrés Messi')('Barcelona','org:top_members/employees','Lionel Andrés Messi')<SummaryFacts>('Lionel Andrés Messi','per:employee_of','Barcelona')('Lionel Andrés Messi','per:date_of_birth','24 Aug 1997')('Barcelona','org:country_of_headquarters','Spanish')('Spanish','org:top_members/employees','Lionel Andrés Messi')('Barcelona','org:top_members/employees','Lionel Andrés Messi')('Lionel Andrés Messi','per:origin','Spanish')('Lionel Andrés Messi','per:countries_of_residence','Spanish')<CommonFacts>('Barcelona','org:top_members/employees','Lionel Andrés Messi')('Lionel Andrés Messi','per:employee_of','Barcelona')('Barcelona','org:country_of_headquarters','Spanish')('Spanish','org:top_members/employees','Lionel Andrés Messi')<DiffFacts>('Lionel Andrés Messi','per:date_of_birth','24 Aug 1997')('Lionel Andrés Messi','per:origin','Spanish')('Lionel Andrés Messi','per:countries_of_residence','Spanish')FactScore:0.5714285714285714AnswersbasedonSource(QuestionsaregeneratedfromSummary)[Q]WhoisthecaptainoftheSpanishnationalteam?[Pred]<unanswerable>[Q]WhenwasLionelAndrésMessiborn?[Pred]24June1987[Q]LionelAndrésMessiisaprofessionalfootballerofwhatnationality?[Pred]Argentine[Q]LionelMessiisacaptainofwhichSpanishclub?[Pred]BarcelonaAnswersbasedonSummary(QuestionsaregeneratedfromSummary)[Q]WhoisthecaptainoftheSpanishnationalteam?[Pred]LionelAndrésMessi[Q]WhenwasLionelAndrésMessiborn?[Pred]24Aug1997[Q]LionelAndrésMessiisaprofessionalfootballerofwhatnationality?[Pred]Spanish[Q]LionelMessiisacaptainofwhichSpanishclub?[Pred]BarcelonaQAGSScore:0.3333333333333333Avg.ROUGE-1:0.4415584415584415Avg.ROUGE-2:0.3287671232876712Avg.ROUGE-L:0.4415584415584415<BERTScoreScore>Precision:0.9760397672653198Recall:0.9778039455413818F1:0.9769210815429688You can use the GPU with thedevice. If you want to use GPU, passcuda(default iscpu)>>>factsumm(article,summary,device="cuda")Sub-modulesFromhere, you can find various way to scoreFactual Consistency levelwithUnsupervised methodsTriple-based Module (closed-scheme)>>>fromfactsummimportFactSumm>>>factsumm=FactSumm()>>>factsumm.extract_facts(article,summary,verbose=True)<SourceEntities>LineNo.1:[[('Lionel Andrés Messi','PERSON'),('24 June 1987','DATE'),('Argentine','NORP'),('Spanish','NORP'),('Barcelona','ORG'),('Argentina','GPE')]]LineNo.2:[[('one','CARDINAL'),('Messi','PERSON'),('six','CARDINAL'),("Ballon d'Or",'WORK_OF_ART'),('European Golden Shoes','WORK_OF_ART'),('2020','DATE'),("Ballon d'Or Dream Team",'WORK_OF_ART')]]<SummaryEntities>LineNo.1:[[('Lionel Andrés Messi','PERSON'),('24 Aug 1997','DATE'),('Spanish','NORP'),('Barcelona','ORG')]]<SourceFacts>('Lionel Andrés Messi','per:origin','Argentine')('Lionel Andrés Messi','per:employee_of','Barcelona')('Barcelona','org:country_of_headquarters','Spanish')('Lionel Andrés Messi','per:date_of_birth','24 June 1987')('Lionel Andrés Messi','per:countries_of_residence','Argentina')('Spanish','org:top_members/employees','Lionel Andrés Messi')('Barcelona','org:top_members/employees','Lionel Andrés Messi')<SummaryFacts>('Lionel Andrés Messi','per:employee_of','Barcelona')('Lionel Andrés Messi','per:date_of_birth','24 Aug 1997')('Barcelona','org:country_of_headquarters','Spanish')('Spanish','org:top_members/employees','Lionel Andrés Messi')('Barcelona','org:top_members/employees','Lionel Andrés Messi')('Lionel Andrés Messi','per:origin','Spanish')('Lionel Andrés Messi','per:countries_of_residence','Spanish')<CommonFacts>('Barcelona','org:top_members/employees','Lionel Andrés Messi')('Lionel Andrés Messi','per:employee_of','Barcelona')('Barcelona','org:country_of_headquarters','Spanish')('Spanish','org:top_members/employees','Lionel Andrés Messi')<DiffFacts>('Lionel Andrés Messi','per:date_of_birth','24 Aug 1997')('Lionel Andrés Messi','per:origin','Spanish')('Lionel Andrés Messi','per:countries_of_residence','Spanish')FactScore:0.5714285714285714The triple-based module counts the overlap of fact triples between the generated summary and the source document.QA-based ModuleIf you ask questions about the summary and the source document, you will get a similar answer if the summary realistically matches the source document>>>fromfactsummimportFactSumm>>>factsumm=FactSumm()>>>factsumm.extract_qas(article,summary,verbose=True)AnswersbasedonSource(QuestionsaregeneratedfromSummary)[Q]WhoisthecaptainoftheSpanishnationalteam?[Pred]<unanswerable>[Q]WhenwasLionelAndrésMessiborn?[Pred]24June1987[Q]LionelAndrésMessiisaprofessionalfootballerofwhatnationality?[Pred]Argentine[Q]LionelMessiisacaptainofwhichSpanishclub?[Pred]BarcelonaAnswersbasedonSummary(QuestionsaregeneratedfromSummary)[Q]WhoisthecaptainoftheSpanishnationalteam?[Pred]LionelAndrésMessi[Q]WhenwasLionelAndrésMessiborn?[Pred]24Aug1997[Q]LionelAndrésMessiisaprofessionalfootballerofwhatnationality?[Pred]Spanish[Q]LionelMessiisacaptainofwhichSpanishclub?[Pred]BarcelonaQAGSScore:0.3333333333333333ROUGE-based Module>>>fromfactsummimportFactSumm>>>factsumm=FactSumm()>>>factsumm.calculate_rouge(article,summary)Avg.ROUGE-1:0.4415584415584415Avg.ROUGE-2:0.3287671232876712Avg.ROUGE-L:0.4415584415584415Simple but effective word-level overlap ROUGE scoreBERTScore Module>>>fromfactsummimportFactSumm>>>factsumm=FactSumm()>>>factsumm.calculate_bert_score(article,summary)<BERTScoreScore>Precision:0.9760397672653198Recall:0.9778039455413818F1:0.9769210815429688BERTScorecan be used to calculate the similarity between each source sentence and the summary sentenceCitationIf you apply this library to any project, please cite:@misc{factsumm,
  author       = {Heo, Hoon},
  title        = {FactSumm: Factual Consistency Scorer for Abstractive Summarization},
  howpublished = {\url{https://github.com/Huffon/factsumm}},
  year         = {2021},
}ReferencesThe Factual Inconsistency Problem in Abstractive Text Summarization: A SurveyAssessing The Factual Accuracy of Generated TextAsking and Answering Questions to Evaluate the Factual Consistency of SummariesFEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization