Package: factscore
FActScoreThis is the official release accompanying our EMNLP 2023 paper,FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. FActScore is available as a PIP package as well.If you find FActScore useful, please cite:@inproceedings{ factscore,
    title={ {FActScore}: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation },
    author={ Min, Sewon and Krishna, Kalpesh and Lyu, Xinxi and Lewis, Mike and Yih, Wen-tau and Koh, Pang Wei and Iyyer, Mohit and Zettlemoyer, Luke and Hajishirzi, Hannaneh },
    year={ 2023 },
    booktitle = { EMNLP },
    url={ https://arxiv.org/abs/2305.14251 }
}InstallMake a new Python 3.7+ environment usingvirtualenvorconda.pipinstall--upgradefactscore
python-mspacydownloaden_core_web_smDownload the datapython-mfactscore.download_data--llama_7B_HF_path"llama-7B"This command does the following.Download the knowledge source and example data.Take the LLAMA 7B model and reconstruct Inst-LLAMA. This requires having access to HuggingFace weights of the LLAMA-7B model, which are added to the--llama_7B_HF_pathflag. Followthis guidein order to obtain those weights. Skip the--llama_7B_HF_pathif you would only like to use the ChatGPT version of FActScore.Optional flags:--data_dir: directory to store the knowledge source and example data..cache/factscoreby default.--model_dir: directory to store Inst-LLAMA weights..cache/factscoreby default.Troubleshooting:If you get aERROR 429: Too Many Requestserror while downloading the DB file, please download the DB fromthis Google Drive linkand place it under--data_dir(.cache/factscoreby default).If everything else fails, consider downloading the files manually fromthis linkand placing them in--data_dirand--model_dir, seefactscore/download_data.pyfor more details.Running FActScore using a command lineWe expect running FActScore costs about $1 of the API cost per 100 sentences. For instance, if you have 100 generations, each with 5 sentences on average, it costs $5 in total.python-mfactscore.factscorer--input_path{input_path}--model_name{estimator_name}--openai_key{openai_key}--input_pathcan be something likedata/unlabeled/InstructGPT.jsonl. It should be a.jsonlformat where each line containstopic(a topic entity that corresponds to the Wikipedia title) andoutput(a generation from the model).--model_name:retrieval+ChatGPTandretrieval+llama+npm(You can also useretrieval+ChatGPT+npmorretrieval+llamabut we recommend the former two.)--openai_key: File containing OpenAI API Key.Optional flags:--data_dir: Directory containing knowledge source, etc..cache/factscoreby default.--model_dir: Directory containing Inst-LLAMA weights. Skip if yourmodel_namedoesn't includellama..cache/factscoreby default.--cache_dir: Directory containing cache from API/models..cache/factscoreby default.--use_atomic_facts: If specified, it uses model-generated atomic facts released as part of our data instead of running the atomic fact generator. This will allow reproducing our results with no (or little if it still uses ChatGPT) cost. You can't specify it if you are running new model generations.--gamma: A hyperparameter for length penalty.10by default. It penalizes the score if the number of facts is less thangamma.10roughly corresponds to 2 sentences, so would penalize if the generation has less than 2 sentences. Usually, this would not change the ranking between systems unless some systems generate overly short responses all the time (e.g., models trained on NLP datasets without long-form generation tasks may do so). If you would like to turn off the length penalty completely, specify--gamma 0.--n_samples: If specified, it runs the model on a subset of the data.--verbose: If specified, it shows the progress bar.--print_rate_limit_error: It specified, it prints out rate limit errors from OpenAI API.--cost_estimate: This flag decides the type of OpenAI API cost estimation that we provide before calling it. It can be"consider_cache"(default) or"ignore_cache".--abstain_detection: This flag optionally enables automatic detection of abstained responses. By default this is disabled, but it is recommended to add your own function tailored to your model. The currently supported detectors are"generic"and"perplexity_ai", and their implementations can be found infactscore/abstain_detection.py. There are two methods to add your own abstain function: a) clone our GitHub repository to installfactscorelocally (pip install --editable .), and then add your function tofactscore/abstain_detection.pydirectly; b) process your abstain detection outside our package, and use empty strings in theoutputkey for the JSONL file used in--input_path.--knowledge_source: In case the default knowledge source (Wikipedia - 2023/04/01) will not be used, preprocess it using theinstructions below, and then specify the knowledge_source name under this flag.To evaluate your own LMThere're two sets of prompt entities,data/labeled/prompt_entities.txt(183 entities) anddata/unlabeled/prompt_entities.txt(500 entities). Each line contains the name of the person (which is also a corresponding Wikipedia title). You can use the labeled version if you want to be compatible with the data underdata/labeled(Section 3 and Section 4.2 in the paper), and use the unlabeled version if you want to be compatible with the data underdata/unlabeled(Section 4.3 in the paper).You can prompt your LM with your own prompt (we usedQuestion: Tell me a bio of <entity>.) and use the following code.fromfactscore.factscorerimportFactScorerfs=FactScorer(openai_key="...")# topics: list of strings (human entities used to generate bios)# generations: list of strings (model generations)out=fs.get_score(topics,generations,gamma=10)print(out["score"])# FActScoreprint(out["init_score"])# FActScore w/o length penaltyprint(out["respond_ratio"])# % of responding (not abstaining from answering)print(out["num_facts_per_response"])# average number of atomic facts per responseAlternatively, you can create a .jsonl file, where each line hastopic(entity name, exactly same as the one from.txtfile) andoutput(generation from LM), and then use a command lineabove.We recommend using (A)FactScorer(model_name="retrieval+ChatGPT")(default) or (B)FactScorer(model_name="retrieval+llama+npm"). They have 0.99 Pearson correlation. Here're results of a range of models, which you can easily reproduce throughthese command lines.Model% respond# factsFActScore from (A)FActScore from (B)GPT-488.260.873.159.9ChatGPT84.237.071.660.4Alpaca 65B100.017.155.646.3InstructGPT99.827.752.841.7Alpaca 13B100.016.647.740.3Vicuna 13B76.650.946.640.7Alpaca 7B100.017.439.736.5Vicuna 7B91.045.638.936.9MPT Chat 7B88.837.330.127.9Oasst Pythia 12B100.039.725.120.8Dolly 12B100.024.621.717.1StableLM tuned 7B66.638.017.316.3% respond(% of responding instead of abstaining from answering) and# facts(# of atomic facts per valid response) indicate "factual recall" (how many pieces of information the model gives) and FActScore indicates "factual precision" (how accurate each piece of information the model gives is).To use a custom knowledge sourceBy default, FActScore uses Wikipedia dump from 2023/04/01. But you can also use your own knowledge source!The knolwedge source should be ready in a.jsonlformat, where each line is a dictionary containingtitleandtext.textcan either be a string or a list of strings (e.g., sections).fromfactscore.factscorerimportFactScorerfs=FactScorer()# this will create a database using your file# for English Wikipedia (18GB)), it takes ~8 hours# once DB file is created, you can reuse it by only specifying `db_path`fs.register_knowledge_source(name_of_your_knowledge_source,data_path=path_to_jsonl_file,db_path=path_to_output_db_file)# now, when you compute a score, specify knowledge source to useout=fs.get_score(topics,generations,knowledge_source=name_of_your_knowledge_source)print(out["score"])# FActScoreprint(out["respond_ratio"])# % of responding (not abstaining from answering)print(out["num_facts_per_response"])# average number of atomic facts per responseTo see an example of constructing the ACL anthology knowledge source, seepreprocessing/preprocess_acl.py.