Exploring Conditions for Diffusion models in Robotic Control
Authors: name: Heeseong Shin, name: Byeongho Heo, name: Dongyoon Han, name: Seungryong Kim, name: Taekyung Kim
Organization: NAVER AI Lab

While pre-trained visual representations have significantly advanced
imitation learning, they are often task-agnostic as they remain frozen during
policy learning. In this work, we explore leveraging pre-trained text-to-image
diffusion models to obtain task-adaptive visual representations for robotic
control, without fine-tuning the model itself. However, we find that naively
applying textual conditions - a successful strategy in other vision domains -
yields minimal or even negative gains in control tasks. We attribute this to
the domain gap between the diffusion model's training data and robotic control
environments, leading us to argue for conditions that consider the specific,
dynamic visual information required for control. To this end, we propose ORCA,
which introduces learnable task prompts that adapt to the control environment
and visual prompts that capture fine-grained, frame-specific details. Through
facilitating task-adaptive representations with our newly devised conditions,
our approach achieves state-of-the-art performance on various robotic control
benchmarks, significantly surpassing prior methods.