Experience is the Best Teacher: Grounding VLMs for Robotics through
  Self-Generated Memory
Authors: name: Guowei Lan, name: Kaixian Qu, name: René Zurbrügg, name: Changan Chen, name: Christopher E. Mower, name: Haitham Bou-Ammar, name: Marco Hutter

Vision-language models (VLMs) have been widely adopted in robotics to enable
autonomous planning. However, grounding VLMs, originally trained on internet
data, to diverse real-world robots remains a challenge. This paper presents
ExpTeach, a framework that grounds VLMs to physical robots by building a
self-generated memory of real-world experiences. In ExpTeach, the VLM
autonomously plans actions, verifies outcomes, reflects on failures, and adapts
robot behaviors in a closed loop. The self-generated experiences during this
process are then summarized into a long-term memory, enabling retrieval of
learned knowledge to guide future tasks via retrieval-augmented generation
(RAG). Additionally, ExpTeach enhances the spatial understanding of VLMs with
an on-demand image annotation module. In experiments, we show that reflection
improves success rates from 36% to 84% on four challenging robotic tasks and
observe the emergence of intelligent object interactions, including creative
tool use. Across extensive tests on 12 real-world scenarios (including eight
unseen ones), we find that grounding with long-term memory boosts single-trial
success rates from 22% to 80%, demonstrating the effectiveness and
generalizability of ExpTeach.