Package: instruct-qa
Evaluating Correctness and Faithfulness of Instruction-Following Models for Question AnsweringQuick StartInstallationMake sure you have Python 3.7+ installed. It is also a good idea to use a virtual environment.Show instructions for creating a Virtual Environmentpython3-mvenvinstruct-qa-venvsourceinstruct-qa-venv/bin/activateYou can install the library viapip:# Install the latest releasepip3installinstruct-qa# Install the latest version from GitHubpip3installgit+https://github.com/McGill-NLP/instruct-qaFor development, you can install it in editable mode with:git clone https://github.com/McGill-NLP/instruct-qa
cd instruct-qa/
pip3 install -e .UsageHere is a simple example to get started. Using this library, use can easily leverage retrieval-augmented instruction-following models for question-answering in ~25 lines of code. The source file for this example isexamples/get_started.py.frominstruct_qa.collections.utilsimportload_collectionfrominstruct_qa.retrieval.utilsimportload_retriever,load_indexfrominstruct_qa.prompt.utilsimportload_templatefrominstruct_qa.generation.utilsimportload_modelfrominstruct_qa.response_runnerimportResponseRunnercollection=load_collection("dpr_wiki_collection")index=load_index("dpr-nq-multi-hnsw")retriever=load_retriever("facebook-dpr-question_encoder-multiset-base",index)model=load_model("flan-t5-xxl")prompt_template=load_template("qa")queries=["what is haleys comet"]runner=ResponseRunner(model=model,retriever=retriever,document_collection=collection,prompt_template=prompt_template,queries=queries,)responses=runner()print(responses[0]["response"])# Halley's Comet Halley's Comet or Comet Halley, officially designated 1P/Halley, is a short-period comet visible from Earth every 75–76 years. Halley is the only known short-period comet that is regularly visible to the naked eye from Earth, and the only naked-eye comet that might appear twice in a human lifetime. Halley last appeared...You can also check the input prompt given to the instruction-sollowing model that contains the instruction and the retrieved passages.print(responses[0]["prompt"])"""Please answer the following question given the following passages:- Title: Bill Haleythen known as Bill Haley's Saddlemen...- Title: C/2016 R2 (PANSTARRS)(CO) with a blue coma. The blue color......Question: what is haleys cometAnswer:"""Detailed documentation of different modules of the library can be foundhereGenerating responses for entire datasetsOur library supports both question answering (QA) and conversational question answering (ConvQA) datasets. The following datasets are currently incorporated in the libraryNatural Questions (Open-domain)HotpotQATopiOCQAHere is an example to generate responses for Natural Questions using DPR retriever and Flan-T5 generator.pythonexperiments/question_answering.py\--prompt_typeqa\--dataset_namenatural_questions\--document_collection_namedpr_wiki_collection\--index_namedpr-nq-multi-hnsw\--retriever_namefacebook-dpr-question_encoder-multiset-base\--batch_size1\--model_nameflan-t5-xxl\--k8By default, aresultsdirectory is created within the repository that stores the model responses. The default directory location can be overidden by providing an additional command line argument--persistent_dir <OUTPUT_DIR>More examples are present in theexamplesdirectory.Download model responses and human evaluation dataWe release the model responses generated using the above commands for all three datasets. The scores reported in the paper are based on these responses. The responses can be downloaded with the following command:pythondownload_data.py--resourceresultsThe responses are automatically unzipped and stored as JSON lines in the following directory structure:results
├── {dataset_name}
│   ├── response
│   │   ├── {dataset}_{split}_c-{collection}_m-{model}_r-{retriever}_prompt-{prompt}_p-{top_p}_t-{temperature}_s-{seed}.jsonlCurrently, the following models are included:fid(Fusion-in-Decoder, separately fine-tuned on each dataset)gpt-3.5-turbo(GPT-3.5)alpaca-7b(Alpaca)llama-2-7b-chat(Llama-2)flan-t5-xxl(Flan-T5)We also release the human annotations for correctness and faithfulness on a subset of responses for all datasets. The annotations can be downloaded with the following command:pythondownload_data.py--resourcehuman_eval_annotationsThe responses will be automatically unzipped in the following directory structure:human_eval_annotations
├── correctness
│   ├── {dataset_name}
│   │   ├── {model}_human_eval_results.json
|
├── faithfulness
│   ├── {dataset_name}
│   │   ├── {model}_human_eval_results.jsonEvaluating model responses (Coming soon!)Documentation to evaluate model responses and add your own evaluation criterion coming soon! Stay tuned!LicenseThis work is licensed under the Apache 2 license. SeeLICENSEfor details.CitationTo cite this work, please use the following citation:@article{adlakha2023evaluating,
      title={Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering}, 
      author={Vaibhav Adlakha and Parishad BehnamGhader and Xing Han Lu and Nicholas Meade and Siva Reddy},
      year={2023},
      journal={arXiv:2307.16877},
}ContactFor queries and clarifications please contactvaibhav.adlakha (at) mila (dot) quebec