Title: Teaching LLMs When to Stop Seeking and Start Acting
Primary area: foundation or frontier models, including LLMs
Keywords: termination, information seeking, reasoning, counterfactuals, large language models, decision making, medical diagnosis, mathematical problem solving, efficiency, test-time compute

Abstract: Many tasks require machine learning models to strategically gather relevant information over multiple rounds of interaction before actually acting on a task. Strategic information gathering requires models to know not only how to effectively acquire information, but also when to stop gathering information and make a decision, in order to avoid overthinking or getting derailed when acting. In this paper, we formalize this problem and introduce Counterfactuals and Reasoning for Termination (CaRT), an approach for teaching LLMs when to stop seeking information. To appropriately learn when to terminate, CaRT fine-tunes LLMs using counterfactual pairs of trajectories, one where termination is appropriate and a minimally modified version of the same trajectory where it is not. It trains the LLM to explain the rationale for the termination decision in either case via verbal reasoning, and imbues this capability into the base LLM via fine-tuning. We instantiate CaRT in two domains: interactive medical diagnosis and math problem solving. In both domains, we find that CaRT improves the efficiency of information gathering and task success rate compared to other fine-tuning methods.

TL;DR: A framework that teaches LLMs to decide when to stop thinking by training on counterfactual termination pairs with reasoning traces, enabling more efficient and reliable problem solving in various domains.