Title: STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases
Benchmark: STaRK (Semi-structure retrieval benchmark on Textual and Relational Knowledge Bases)
Summary: STaRK is a large-scale benchmark for evaluating LLM-based retrieval systems on semi-structured knowledge bases (SKBs) that integrate textual and relational information. It covers product search, academic paper search, and precision medicine domains. A novel pipeline synthesizes realistic queries and ground truth answers, supplemented by human-generated queries, revealing significant challenges for current retrieval systems.
Contribution: Key contributions include the first large-scale benchmark specifically for retrieval on SKBs integrating text and relations, a novel query synthesis pipeline using LLMs, the construction of three domain-specific SKBs and corresponding datasets, and extensive experiments evaluating various retrieval models including LLMs.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications), Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: LLM retrieval capability on semi-structured knowledge bases (SKBs), involving reasoning over combined textual and relational information.
Phenomenon defined by authors: Yes
Phenomenon definition: The task is defined as: Given a semi-structured knowledge base (SKB) comprising a knowledge graph G=(V,E) and associated text documents D, and a query Q, the goal is to retrieve a set of nodes (entities) A ⊆ V that satisfy both the relational requirements implied by G and the textual requirements specified in Q, based on their associated documents.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: The benchmark specifically targets the gap left by prior work that treated textual and relational retrieval separately, aiming to evaluate systems on more realistic, integrated knowledge sources.
Task definition: Given a query combining textual descriptions and relational constraints, retrieve the correct entities (nodes) from a semi-structured knowledge base (SKB) that satisfies both aspects.
Task item definition: A single item consists of a natural language query (potentially simulating different user roles or contexts) and a set of ground-truth entity identifiers (nodes) from the corresponding SKB that correctly answer the query.
Task definition detail: Queries are designed to be natural-sounding, incorporate diverse relational patterns (including multi-hop) and textual properties, cover three distinct domains, and include both synthesised and human-generated questions.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: A novel pipeline samples relational templates extracts textual properties from a 'gold' entity using LLMs, synthesizes natural language queries using LLMs (incorporating roles and context), and filters candidate answers using LLMs to create the synthesized dataset. Additionally, human participants generated queries using an interactive platform exploring the SKBs.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: The benchmark simulates queries from different user roles (customers, researchers, doctors, patients)  and includes complex contexts. Human evaluations confirmed the naturalness, diversity, and practicality of the synthesized queries.
Dataset size: Test set sizes: Synthesized: STARK-AMAZON ≈ 1638, STARK-MAG ≈ 2665, STARK-PRIME ≈ 2801. Human-generated: STARK-AMAZON = 81, STARK-MAG = 84, STARK-PRIME = 98. Total Test Queries ≈ 7367. 
Dataset size notes: Synthesized Train/Validation sizes: STARK-AMAZON: Train≈5915, Val≈1547; STARK-MAG: Train≈7994, Val≈2665; STARK-PRIME: Train≈6162, Val≈2241. 
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number)
Response format detail: Systems are expected to return a ranked list of entity nodes (V) from the knowledge base that satisfies the query's textual and relational constraints.
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring), Distribution (perplexity, calibration, correlation)
Metric definition detail: Primary metrics are Hit@k (k=1, 5), Recall@k (k=20, chosen because max answer set size ≤ 20), and Mean Reciprocal Rank (MRR).
Metric aggregation: Simple Mean
Metric statistics: Simple mean/average of Hit@k, Recall@k, and MRR over the test sets.
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Conducted human evaluation with 63 participants validating synthesized query naturalness (94.1% ≥ neutral), diversity (85.3% ≥ neutral), and practicality (89.4% ≥ neutral). Analyzed dataset statistics: query/answer lengths, lexical diversity (Shannon Entropy, TTR), and ratio of relational/textual information. Assessed the precision of the LLM-based answer filtering step in the synthesis pipeline (high verification rates for gold answers). Compared synthesized vs. human-generated queries.
Taxonomy: Retrieval / 
New taxonomy: General Application / Document Retrieval
Procedural notes: Human query generation involved volunteers acknowledged in the paper. Detailed prompts and LLM versions used for the synthesis pipeline are documented in the appendix. Data sources and licenses are mentioned. An interactive data explorer is provided.
Other notes: The benchmark demonstrates that even advanced LLM-based retrieval and re-ranking systems face significant challenges with complex SKB retrieval, indicated by relatively low performance on metrics like Hit@1 and Recall@20 across all domains, especially STARK-PRIME. Retrieval latency is identified as a major practical hurdle for the best-performing (re-ranker) models.