Title: MiGrATe: Mixed-Policy GRPO for Adaptation at Test-Time
Primary area: foundation or frontier models, including LLMs
Keywords: search, TTT, test-time training, GRPO, RL, discovery, LLMs

Abstract: Large language models (LLMs) are increasingly being applied to black-box optimization tasks, from program synthesis to molecule design. Prior work typically leverages in-context learning to iteratively guide the model towards better solutions. Such methods, however, often struggle to balance exploration of new solution spaces with exploitation of high-reward ones. Recently, test-time training (TTT) with synthetic data has shown promise in improving solution quality. However, the need for hand-crafted training data tailored to each task limits feasibility and scalability across domains. To address this problem, we introduce MiGrATe—a method for online TTT that uses GRPO as a search algorithm to adapt LLMs at inference without requiring external training data. MiGrATe operates via a mixed-policy group construction procedure that combines on-policy sampling with two off-policy data selection techniques: greedy sampling, which selects top-performing past completions, and neighborhood sampling (NS), which generates completions structurally similar to high-reward ones. Together, these components bias the policy gradient towards exploiting promising regions in the solution space, while preserving exploration through on-policy sampling. We evaluate MiGrATe on four challenging domains—word search, molecule optimization, hypothesis+program induction on the Abstraction and Reasoning Corpus (ARC), and natural-language hypothesis search on DiscoveryBench—and find that it consistently outperforms both inference-only and TTT baselines, demonstrating the potential of online TTT as a solution for complex search tasks without curated training data.

TL;DR: MiGrATe is an online test-time training method for LLMs that adaptively balances exploration and exploitation via mixed-policy GRPO, outperforming inference-only and TTT baselines across black-box optimization tasks without requiring external data.