Self-Taught Agentic Long Context Understanding
Authors: name: Yufan Zhuang; name: Xiaodong Yu; name: Jialian Wu; name: Ximeng Sun; name: Ze Wang; name: Jiang Liu; name: Yusheng Su; name: Jingbo Shang; name: Zicheng Liu; name: Emad Barsoum
Published: 2025-02-21 20:29:36

Abstract: Answering complex, long-context questions remains a major challenge for large
language models (LLMs) as it requires effective question clarifications and
context retrieval. We propose Agentic Long-Context Understanding (AgenticLU), a
framework designed to enhance an LLM's understanding of such queries by
integrating targeted self-clarification with contextual grounding within an
agentic workflow. At the core of AgenticLU is Chain-of-Clarifications (CoC),
where models refine their understanding through self-generated clarification
questions and corresponding contextual groundings. By scaling inference as a
tree search where each node represents a CoC step, we achieve 97.8% answer
recall on NarrativeQA with a search depth of up to three and a branching factor
of eight. To amortize the high cost of this search process to training, we
leverage the preference pairs for each step obtained by the CoC workflow and
perform two-stage model finetuning: (1) supervised finetuning to learn
effective decomposition strategies, and (2) direct preference optimization to
enhance reasoning quality. This enables AgenticLU models to generate
clarifications and retrieve relevant context effectively and efficiently in a
single inference pass. Extensive experiments across seven long-context tasks
demonstrate that AgenticLU significantly outperforms state-of-the-art prompting
methods and specialized long-context LLMs, achieving robust multi-hop reasoning
while sustaining consistent performance as context length grows.