Title: KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval
Benchmark: Kitab
Summary: The paper introduces KITAB, a large‑scale benchmark that evaluates how well large language models (LLMs) can satisfy multi‑way constraint‑based information‑retrieval queries (e.g., “list all books by Toni Morrison published between 1970‑1980”). KITAB contains > 600 authors and 12 989 natural‑language queries with automatically verified ground‑truth answer sets, plus an evaluation protocol that separates failure modes such as irrelevance, constraint violations and incompleteness.
Contribution: - (i) A fully‑released dataset with rich metadata (author popularity, constrainedness, constraint type) and a data‑generation recipe that can be re‑run on disjoint author sets.
- (ii) A set of generous, automatically computed metrics (irrelevant‑rate, satisfaction‑rate, completeness, etc.) and prompt templates that isolate parametric‑knowledge, retrieval‑augmented and chain‑of‑thought settings.
- (iii) An extensive empirical study showing GPT‑4 and GPT‑3.5 still fail badly on constraint satisfaction, even when given perfect context which highlights a key open problem.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Constraint‑satisfaction in factual information‑retrieval
Phenomenon defined by authors: Yes
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: Evaluates both parametric knowledge and reasoning over supplied context
Task definition: Given an author plus 1–2 natural‑language constraints on book titles (lexical, temporal, named‑entity), output all books by that author that satisfy the constraints.
Task item definition: A single row consists of a single query: author metadata, constraint(s), ground‑truth answer set, metadata (popularity, constrainedness, constraint type).
Task source: Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: The queries are instantiated automatically by applying structured templates over cleaned book metadata from OpenLibrary and WikiData. Authors filtered authors/books using criteria (e.g., popularity, date of birth, number of works), and controlled for constraint type balance and constrainedness.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 12,989 queries (8,239 single-constraint, 4,750 double-constraint)
Dataset metadata available: Yes
Dataset metadata detail: author birth‑year, WikiData sitelinks (popularity), constraint‑type, constrainedness, satisfiable/unsatisfiable flag.
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: By constraint type, popularity bin, constrainedness bin
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Manual spot‑check (< 5 % missing titles) to verify data quality.
Taxonomy: Instruction Following / 
New taxonomy: General Capability / Instruction Following
Other notes: KITAB deliberately bundles unsatisfiable queries and reports separate failure modes, making it especially useful for probing hallucination and constraint‑aware reasoning beyond mere fact recall.