Title: PilotRAG: Teaching LLMs Multi-Turn Hybrid RAG via Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Retrieval-Augmented Generation, Reinforcement Learning

Abstract: Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by incorporating external knowledge, typically from unstructured texts or structured graphs. While recent progress has extended text-based RAG to multi-turn settings, graph-based RAG remains limited to one-shot retrieval. This limitation hinders the ability of graph-based RAG systems to incrementally integrate supplementary evidence as reasoning unfolds, thereby reducing their effectiveness on complex multi-hop questions. To address this limitation, we introduce PilotRAG, a reinforcement learning-based framework that enables LLMs to perform multi-turn and adaptive graph-text hybrid RAG by dynamically interleaving reasoning, hybrid retrieval, and answer formulation. PilotRAG jointly optimizes the entire generation process via reinforcement learning, allowing the model to learn when to reason, what to retrieve from either unstructured texts or structured graphs, and when to produce final answers, all within a unified generation policy. To guide this learning process, we design a two-stage training framework with a reward function that accounts for both task outcome and retrieval efficiency. By rewarding answer accuracy and efficient retrieval while penalizing redundant retrieval operations, the model learns to retrieve selectively and reason effectively. Experiments on both simple and multi-hop question answering benchmarks demonstrate that PilotRAG significantly outperforms existing RAG baselines, highlighting the benefits of end-to-end reinforcement learning for enabling adaptive and iterative retrieval in complex reasoning scenarios.

TL;DR: We propose PilotRAG, an RL-based RAG framework that enables LLMs to perform multi-turn hybrid RAG with both high accuracy and efficiency.