Self-Adapting Improvement Loops for Robotic Learning
Authors: name: Calvin Luo, name: Zilai Zeng, name: Mingxi Jia, name: Yilun Du, name: Chen Sun

Video generative models trained on expert demonstrations have been utilized
as performant text-conditioned visual planners for solving robotic tasks.
However, generalization to unseen tasks remains a challenge. Whereas improved
generalization may be facilitated by leveraging learned prior knowledge from
additional pre-collected offline data sources, such as web-scale video
datasets, in the era of experience we aim to design agents that can
continuously improve in an online manner from self-collected behaviors. In this
work we thus propose the Self-Adapting Improvement Loop (SAIL), where an
in-domain video model iteratively updates itself on self-produced trajectories,
collected through adaptation with an internet-scale pretrained video model, and
steadily improves its performance for a specified task of interest. We apply
SAIL to a diverse suite of MetaWorld tasks, as well as two manipulation tasks
on a real robot arm, and find that performance improvements continuously emerge
over multiple iterations for novel tasks initially unseen during original
in-domain video model training. Furthermore, we discover that SAIL is
surprisingly robust regarding if and how the self-collected experience is
filtered, and the quality of the initial in-domain demonstrations. Through
adaptation with summarized internet-scale data, and learning through online
experience, we thus demonstrate a way to iteratively bootstrap a
high-performance video model for solving novel robotic tasks through
self-improvement.