Package: fanoutqa
FanOutQARead the paper! |Download the dataset!FanOutQA is a high quality, multi-hop, multi-document benchmark for large language models using English Wikipedia as its
knowledge base. Compared to other question-answering benchmarks, FanOutQA requires reasoning over a greater number of
documents, with the benchmark's main focus being on the titular fan-out style of question. We present these questions
in three tasks -- closed-book, open-book, and evidence-provided -- which measure different abilities of LLM systems.This repository contains utilities to download and work with the dataset in Python, along with implementations of the
evaluation metrics presented in our paper. Alternatively, you can download the dev and test sets in JSON format and
generate completions to submit to us for evaluation.To view the leaderboards and more documentation about how to use this dataset, check out our website
athttps://fanoutqa.com!Requirements and InstallationThefanoutqapackage requires Python 3.8+.To work with just the data, usepip install fanoutqa.
Usepip install "fanoutqa[all]"and read the following section to include a baseline retriever and evaluation metrics.OptionalTo include a baseline BM25-based retriever, usepip install "fanoutqa[retrieval]".To run evaluations on the dev set, you will need to run a couple more steps:pipinstall"fanoutqa[eval]"python-mspacydownloaden_core_web_sm
pipinstall"bleurt @ git+https://github.com/google-research/bleurt.git@master"wgethttps://storage.googleapis.com/bleurt-oss-21/BLEURT-20.zip
unzipBLEURT-20.zip
rmBLEURT-20.zipQuickstartUsefanoutqa.load_dev()orfanoutqa.load_test()to load the dataset.Run your generations.Usefanoutqa.wiki_search(title)andfanoutqa.wiki_content(evidence)to retrieve the contents of
Wikipedia pages for the Open Book and Evidence Provided settings.Evaluate your generations withfanoutqa.eval.evaluate(dev_questions, answers)(see below for the schema).Data FormatTo load the dev or test questions, simply usefanoutqa.load_dev()orfanoutqa.load_test(). This will return a list
ofDevQuestionorTestQuestion, as documented below.Common ModelsPrimitive=bool|int|float|strclassEvidence:pageid:int# Wikipedia page IDrevid:int# Wikipedia revision ID of page as of dataset epochtitle:str# Title of pageurl:str# Link to pageDev SetThe development set is a JSON file containing a list of DevQuestion objects:classDevQuestion:id:strquestion:str# the top-level question to answerdecomposition:list[DevSubquestion]# human-written decomposition of the questionanswer:dict[str,Primitive]|list[Primitive]|Primitivenecessary_evidence:list[Evidence]categories:list[str]classDevSubquestion:id:strquestion:strdecomposition:list[DevSubquestion]answer:dict[str,Primitive]|list[Primitive]|Primitive# the answer to this subquestiondepends_on:list[str]# the IDs of subquestions that this subquestion requires answering firstevidence:Evidence|None# if this is None, the question will have a decompositionTest SetThe test set contains a slightly different format, as the answers are not provided. We include links to all the evidence
used in the human-written decompositions for our Evidence Provided task.classTestQuestion:id:strquestion:strnecessary_evidence:list[Evidence]categories:list[str]Wikipedia RetrievalTo retrieve the contents of Wikipedia pages used as evidence, this package queries Wikipedia's Revisions API. There
are two main functions to interface with Wikipedia:wiki_search(query)returns a list of Evidence (Wikipedia pages that best match the query)wiki_content(evidence)takes an Evidence and returns its content (as of the dataset epoch) as Markdown.To save on time waiting for requests and computation power (both locally and on Wikipedia's end), this package
aggressively caches retrieved Wikipedia pages. By default, this cache is located in~/.cache/fanoutqa/wikicache.
We provide many cached pages (~9GB) you can prepopulate this cache with, by using the following commands:mkdir-p~/.cache/fanoutqa
wget-O~/.cache/fanoutqa/wikicache.tar.gzhttps://datasets.mechanus.zhu.codes/fanoutqa/wikicache.tar.gztar-xzf~/.cache/fanoutqa/wikicache.tar.gzEvaluationTo evaluate a model's generation, first ensure that you have installed all the evaluation dependencies (see above).To use the GPT-as-judge metric, you will need to provide your OpenAI API key. We intentionally do not read
theOPENAI_API_KEYenvironment variable by default to prevent accidentally spending money; you must set theFANOUTQA_OPENAI_API_KEYenvironment variable instead. You can useexport FANOUTQA_OPENAI_API_KEY=$OPENAI_API_KEYto
quickly copy it over.You should record your model/system's outputs as a list of dicts with the following schema:{"id":"The ID of the question (see test set schema) this is a generation for.","answer":"The model's generation."}Finally, to evaluate your generations on the dev set,
callfanoutqa.eval.evaluate(dev_questions, answers, llm_cache_key="your-model-key"). This will run all of the metrics
and return anEvaluationScoreobject, which has attributes matching the following structure:{"acc":{"loose":0,"strict":0},"rouge":{"rouge1":{"precision":0,"recall":0,"fscore":0},"rouge2":{"precision":0,"recall":0,"fscore":0},"rougeL":{"precision":0,"recall":0,"fscore":0}},"bleurt":0,"gpt":0}(to access this in this dictionary form, usedataclasses.asdict().)Test Set EvaluationTo evaluate your model on the hidden test set, please email your generations
toandrz@seas.upenn.eduwith the subject "FanOutQA Test Evaluation". Your generations
should be in the form of a JSONL file, with each line being a JSON object with the following schema for each test
question:{"id":"The ID of the question (see test set schema) this is a generation for.","answer":"The model's generation."}In the email body, please include details about your system, including at least:the name of your systemthe list of authorsa link to your paper and recommended short citation, if applicablewhether it is a new foundation model, a fine-tune, a prompting approach, or otherAdditional ResourcesAlthough this package queries live Wikipedia and uses the Revisions API to get page content as of the dataset epoch,
we also provide a snapshot of English Wikipedia as of Nov 20, 2023. You can download this
snapshothere(23G)
and its
indexhere.