Title: Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs
Primary area: foundation or frontier models, including LLMs
Keywords: reinforcement finetuning, proactive conversation model, large language model

Abstract: Large Language Models (LLMs) excel as passive responders, but teaching them to be proactive, goal-oriented partners—a critical capability in high-stakes domains—remains a major challenge. 
Current paradigms either myopically optimize single-turn attributes or rely on brittle, high-cost user simulators, creating a persistent ``reality gap''.
To bridge this gap, we introduce \texttt{Learn-to-Ask}, a general, simulator-free framework for learning and deploying proactive dialogue agents \textit{directly from offline expert data}, bypassing the need to model complex user dynamics.
Our key insight is to reframe the offline policy learning problem by leveraging the \textbf{observed future} of each expert trajectory. 
This allows us to infer a dense, turn-by-turn reward signal grounded in the expert's revealed strategy, decomposing the intractable long-horizon problem into a series of supervised learning tasks, and training a policy to output a structured \texttt{(action, state\_assessment)} tuple, governing both \textbf{what to ask} and, crucially, \textbf{when to stop}. 
To ensure reward fidelity, our Automated Grader Calibration pipeline systematically purges noise from the LLM-based reward model with minimal human supervision.
Empirically, we demonstrate the efficacy of \texttt{Learn-to-Ask} in a real-world medical dataset, using LLMs of varying sizes up to 32B. Our approach culminates in the successful deployment of LLMs into a live, large-scale online AI service. In rigorous in-house evaluations, our model was launched and achieved performance even superior to human experts, proving our framework's ability to translate offline data into tangible, real-world impact. We hope this work provides a practical and economically viable blueprint for transforming passive LLMs into proactive, goal-oriented LLM applications.

TL;DR: This work introduces Learn-to-Ask, a framework that enables LLMs to learn both what-to-ask and when-to-stop from offline dialogue data, and has demonstrated human-superior performance in large-scale real-world deployment.