LLM-guided Hierarchical Retrieval
Authors: name: Nilesh Gupta, name: Wei-Cheng Chang, name: Ngot Bui, name: Cho-Jui Hsieh, name: Inderjit S. Dhillon
Organization: Google

Modern IR systems are increasingly tasked with answering complex,
multi-faceted queries that require deep reasoning rather than simple keyword or
semantic matching. While LLM-based IR has shown great promise, the prevailing
retrieve-then-rerank paradigm inherits the limitations of embedding-based
retrieval; parametric generative approaches are difficult to update with new
information; and long-context methods that place the entire corpus in context
are computationally infeasible for large document collections. To address these
challenges, we introduce LATTICE, a hierarchical retrieval framework that
enables an LLM to reason over and navigate large corpora with logarithmic
search complexity by imposing a semantic tree structure on the corpus. Our
approach consists of two stages: (1) an offline phase that organizes the corpus
into a semantic hierarchy via either a bottom-up agglomerative strategy or a
top-down divisive strategy using multi-level summaries and (2) an online
traversal phase where a search LLM navigates this tree. A central challenge in
such LLM-guided search is that the model's relevance judgments are noisy,
context-dependent, and unaware of the hierarchy, making cross-branch and
cross-level comparisons difficult. To overcome this, we propose a traversal
algorithm that estimates calibrated latent relevance scores from local LLM
outputs and aggregates them into a global path relevance metric. Our
training-free framework achieves state-of-the-art zero-shot performance on the
reasoning-intensive BRIGHT benchmark, demonstrating up to 9% improvement in
Recall@100 and 5% in nDCG@10 over the next best zero-shot baseline.
Furthermore, compared to the fine-tuned SOTA method DIVER-v2, LATTICE attains
comparable results on BRIGHT subsets that use a static corpus for evaluation.