IGOR: Image-GOal Representations are the Atomic Control Units for
  Foundation Models in Embodied AI
Authors: name: Xiaoyu Chen; name: Junliang Guo; name: Tianyu He; name: Chuheng Zhang; name: Pushi Zhang; name: Derek Cathera Yang; name: Li Zhao; name: Jiang Bian
Published: 2024-10-17 13:41:16

Abstract: We introduce Image-GOal Representations (IGOR), aiming to learn a unified,
semantically consistent action space across human and various robots. Through
this unified latent action space, IGOR enables knowledge transfer among
large-scale robot and human activity data. We achieve this by compressing
visual changes between an initial image and its goal state into latent actions.
IGOR allows us to generate latent action labels for internet-scale video data.
This unified latent action space enables the training of foundation policy and
world models across a wide variety of tasks performed by both robots and
humans. We demonstrate that: (1) IGOR learns a semantically consistent action
space for both human and robots, characterizing various possible motions of
objects representing the physical interaction knowledge; (2) IGOR can "migrate"
the movements of the object in the one video to other videos, even across human
and robots, by jointly using the latent action model and world model; (3) IGOR
can learn to align latent actions with natural language through the foundation
policy model, and integrate latent actions with a low-level policy model to
achieve effective robot control. We believe IGOR opens new possibilities for
human-to-robot knowledge transfer and control.