WildLMa: Long Horizon Loco-Manipulation in the Wild
Authors: name: Ri-Zhao Qiu; name: Yuchen Song; name: Xuanbin Peng; name: Sai Aneesh Suryadevara; name: Ge Yang; name: Minghuan Liu; name: Mazeyu Ji; name: Chengzhe Jia; name: Ruihan Yang; name: Xueyan Zou; name: Xiaolong Wang
Published: 2024-11-22 18:56:56

Abstract: `In-the-wild' mobile manipulation aims to deploy robots in diverse real-world
environments, which requires the robot to (1) have skills that generalize
across object configurations; (2) be capable of long-horizon task execution in
diverse environments; and (3) perform complex manipulation beyond
pick-and-place. Quadruped robots with manipulators hold promise for extending
the workspace and enabling robust locomotion, but existing results do not
investigate such a capability. This paper proposes WildLMa with three
components to address these issues: (1) adaptation of learned low-level
controller for VR-enabled whole-body teleoperation and traversability; (2)
WildLMa-Skill -- a library of generalizable visuomotor skills acquired via
imitation learning or heuristics and (3) WildLMa-Planner -- an interface of
learned skills that allow LLM planners to coordinate skills for long-horizon
tasks. We demonstrate the importance of high-quality training data by achieving
higher grasping success rate over existing RL baselines using only tens of
demonstrations. WildLMa exploits CLIP for language-conditioned imitation
learning that empirically generalizes to objects unseen in training
demonstrations. Besides extensive quantitative evaluation, we qualitatively
demonstrate practical robot applications, such as cleaning up trash in
university hallways or outdoor terrains, operating articulated objects, and
rearranging items on a bookshelf.