FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on
  Technical Documents
Authors: name: Nandan Thakur, name: Jimmy Lin, name: Sam Havens, name: Michael Carbin, name: Omar Khattab, name: Andrew Drozdov

We introduce FreshStack, a reusable framework for automatically building
information retrieval (IR) evaluation benchmarks from community-asked questions
and answers. FreshStack conducts the following steps: (1) automatic corpus
collection from code and technical documentation, (2) nugget generation from
community-asked questions and answers, and (3) nugget-level support, retrieving
documents using a fusion of retrieval techniques and hybrid architectures. We
use FreshStack to build five datasets on fast-growing, recent, and niche topics
to ensure the tasks are sufficiently challenging. On FreshStack, existing
retrieval models, when applied out-of-the-box, significantly underperform
oracle approaches on all five topics, denoting plenty of headroom to improve IR
quality. In addition, we identify cases where rerankers do not clearly improve
first-stage retrieval accuracy (two out of five topics). We hope that
FreshStack will facilitate future work toward constructing realistic, scalable,
and uncontaminated IR and RAG evaluation benchmarks. FreshStack datasets are
available at: https://fresh-stack.github.io.