Title: Unified Latent Steering and Residual Refinement for Online Improvement of Diffusion Policy Models
Primary area: reinforcement learning
Keywords: Offline-to-online Reinforcement Learning, Diffusion Policy, Robotic Manipulation

Abstract: Imitation learning has driven major advances in robotic manipulation by exploiting large and diverse demonstrations, yet policies trained purely by imitation remain brittle under distribution shift and novel scenarios, making online improvement essential. Directly finetuning the parameters of modern large policies is prohibitively sample inefficient and computationally expensive, while recent finetuning-free adaptation methods either fail to exploit the multimodal distributions learned by pretrained policies or remain confined to the coverage of demonstrations. We propose USR, a Unified framework for latent Steering and residual Refinement that enables efficient online improvement of diffusion policy models. A lightweight actor jointly outputs latent noise to steer the diffusion process toward promising modes and residual corrections to adapt beyond the diffusion policy's support, combining stable mode selection with flexible refinement. This unified design stabilizes training and fully leverages both components. Experiments on two standard benchmarks and our proposed MultiModalBench show that USR achieves state-of-the-art online performance, setting a new paradigm for sample-efficient adaptation of diffusion-based manipulation policies.

TL;DR: We propose USR, a unified framework that steers and refines diffusion policy models through a lightweight actor, enabling sample-efficient online adaptation for robotic manipulation.