Package: scale-score
Fast and Accurate Factual Inconsistency Detection Over Long DocumentsBarrett Martin Lattimer, Patrick Chen, Xinyuan Zhang, Yi Yangblattimer@asapp.comEMNLP 2023https://arxiv.org/abs/2310.13189OverviewIntroducing SCALE, an reference-free NLI based factual inconsistency detection method, and ScreenEval, the longest dialogue based dataset for factual inconsistency detection presently available.
Both can be found in our paper Fast and Accurate Factual Inconsistency Detection Over Long Documents.SCALE uses a novel chunking strategy to achieve state-of-the-art factual inconsistency deteciton performance across many NLG domains, tasks, and over long documents (>6k tokens). SCALE's chunking approach enables fast relevant source text retrival over long documents.SCALEThis metrics outputs the estimated probablility that a hypothesis is supported by a given premiseSCALE(premise, hypothesis). Commonly the hypothesis is generated text and the premise is some ground truth text. For example, a premise may be a document and the hypothesis may be a language model generated summary sentence. The score is bounded as follows 0≤SCALE(premise, hypothesis)≤1. A higher score signifies a higher probability the hypothesis is factually consistent with the premise. A lower score signifies the hypothesis is more likely to be factually inconsistent with the premise. It is recommended to use Flan_T5_XL as the base model for the best results.InstallTo use the evaluation metric, first pip install the python module.pip install scale-scoreor install from sourcepip install -e .ScoreRunning the MetricImport the score function and load your premises, hypothesies. For scoring, the premise is a list of entire document strings while the hypothesis are single sentences represented as is a list of list of strings. Each premise has a list of associated hypothesis with a one to one mapping based on index (premise_0 -> ['hypothesis_0_0', 'hypothesis_0_1'], premise_1-> ['hypothesis_1_0', 'hypothesis_1_1', 'hypothesis_1_2']).fromscale_scoreimportscorepremise=['premise_0','premise_1',]hypothesis=[['hypothesis_0_0','hypothesis_0_1'],['hypothesis_1_0','hypothesis_1_1','hypothesis_1_2']]results=score(premise,hypothesis)Where the results correspond to each hypothesis scored with it's respecitve premiseresults=[SCALE(premise_0,hypothesis_0_0),SCALE(premise_0,hypothesis_0_1),SCALE(premise_1,hypothesis_1_0),SCALE(premise_1,hypothesis_1_1),SCALE(premise_1,hypothesis_1_2),]You can also use thescorerobject to prevent loading the model at every call like so,fromscale_score.scorerimportSCALEScorerscorer=SCALEScorer(size='small',device='cuda')results=scorer.score(premise,hypothesis)ArgumentsThese arguments are the exact same for bothscoreandscorer.scorefunctions exceptscorer.scoredoes not take in asizeordeviceas that is set up when building the scorer object.ArgumentTypeDefaultDescriptionpremiseList[str]requiredpremise text, the ground truthhypothesisList[List[str]]requiredhypothesis text, usually the text predicted by a model being evaluatedchunk_sizeint1000The size of the chunks used to perform chunking on the premisewindow_sizefloat0.25The percentage of overlap between chunks. 0≤window_size<1sizestr'xl'Size of Flan-T5 model, options are 'small', 'base', 'large', 'xl', 'xxl'devicestr'cuda'torch device to send the model to.model_pathstrNoneOptional path to a Flan-T5 model to load. Note the corresponding size must be specified in thesizeargument.modelT5ForConditionalGenerationNoneOptional model to use for scoringtokenizerT5TokenizerNoneOptional tokenizer to use for scoringEvaluationAfter scoring, use theevaluate_scalefunction to evaluate the results.fromscale_score.evalimportevaluate_scalefromscale_score.scorerimportSCALEScorerscorer=SCALEScorer(size='small',device='cuda')results=scorer.score(premise,hypothesis)metrics=evaluate_scale(results)The arguments forevaluate_scaleare as followsArgumentTypeDefaultDescriptionresultsList[float]requiredOutput from scale_score score or scorer runincorrectList[int]requiredList of labels for summary sentences, 1 for incorrect and 0 for correctthresholdfloat0.5Threshold used to calculate binary, micro, macro, and weighted f1 scoresout_filestrNoneOptional json filepath to write the metrics toprint_outputsboolTrueWhether to print the metricsThe metrics that are output are described below.MetricDescriptionpearsonPearson correlationspearmanSpearman correlationkendalltauKendall Tau correlationmajority_class_accuracyAccuracy if we always predict correctbest_accuracyBest predicted accuracy possible after threshold tuningbest_detection_precisionBest predicted precision possible after threshold tuning f1 scorebest_detection_recallBest predicted recall possible after threshold tuning f1 scorebest_detection_f1Best predicted f1 possible after threshold tuningaccuracy@90%Accuracy achieved if we want to keep 90% of all correct sentencesaccuracy@70%Accuracy achieved if we want to keep 70% of all correct sentencesthreshold_f1Threshold used to calculate best_detection_f1threshold_@90%Threshold used to calculate accuracy@90%threshold_@70%Threshold used to calculate accuracy@70%f1_binaryF1 score of incorrect sentence detectionf1_macroAverage F1 score between correct and incorrect sentence detectionf1_microCalculate F1 globally by counting the total true positives, false negatives and false positivesf1_weightedCalculate F1 for each label, and find their average weighted by supportRetrieveRunning RetrievalImport the retrieve function and load your premises, hypothesies.NOTE: Premises are lists of lists in retrieval. Both premises and hypothesis are split down to the sentence or utterance level.Each premise list has an associated hypothesis list with a one to one mapping based on index.fromscale_scoreimportretrievepremise=[['premise_0_utt_0','premise_0_utt_1','premise_0_utt_2'],['premise_1_utt_0','premise_1_utt_1'],]hypothesis=[['hypothesis_0_0','hypothesis_0_1'],['hypothesis_1_0','hypothesis_1_1','hypothesis_1_2']]results=retrieve(premise,hypothesis)Where the results correspond to a list which has the most relevant premise utterance/sentence and the corresponding score.You can also use thescorerobject to prevent loading the model at every call like so,fromscale_score.scorerimportSCALEScorerscorer=SCALEScorer(size='small',device='cuda')results=scorer.retrieve(premise,hypothesis)ArgumentsThese arguments are the exact same for bothretrieveandscorer.retrievefunctions exceptscorer.retrievedoes not take in asizeordeviceas that is set up when building the scorer object.ArgumentTypeDefaultDescriptionpremiseList[str]requiredpremise text, the ground truthhypothesisList[List[str]]requiredhypothesis text, usually the text predicted by a model being evaluatedbranchesint2The number of branches to have in the search treesizestr'xl'Size of Flan-T5 model, options are 'small', 'base', 'large', 'xl', 'xxl'devicestr'cuda'torch device to send the model to.model_pathstrNoneOptional path to a Flan-T5 model to load. Note the corresponding size must be specified in thesizeargument.modelT5ForConditionalGenerationNoneOptional model to use for scoringtokenizerT5TokenizerNoneOptional tokenizer to use for scoringScreenEvalScreenEval is located in thedatafolder stored as a json file. The following keys are important for the use of ScreenEval.KeyTypeDescriptionoriginal_convoList[str]The source document that is to be summarized as a stringconvoList[List[str]]The source document that is to be summarized split into a list of utterancesinferred_summaryList[str]The summary sentence that is paired with the given source documentsummary_idList[str]The source model for the summary sentenceconvo_idList[int]The ID of the source documentannotated_summaryList[str]The entire associated summary, with the focus summary sentence surrounded by<mark><\mark>prediction_annotated_source_docList[str]Raw source documentagreementList[float]Annotator agreement on summary sentence facutal inconsistency labelagg_labelList[bool]Factual inconsistency label (true -> factually consistent, false -> factually inconsistent)rel_uttList[List[int]]The indices of related utterances in the correspondingconvolist.