Theia: Distilling Diverse Vision Foundation Models for Robot Learning
Authors: name: Jinghuan Shang; name: Karl Schmeckpeper; name: Brandon B. May; name: Maria Vittoria Minniti; name: Tarik Kelestemur; name: David Watkins; name: Laura Herlant
Published: 2024-07-29 17:08:21

Abstract: Vision-based robot policy learning, which maps visual inputs to actions,
necessitates a holistic understanding of diverse visual tasks beyond
single-task needs like classification or segmentation. Inspired by this, we
introduce Theia, a vision foundation model for robot learning that distills
multiple off-the-shelf vision foundation models trained on varied vision tasks.
Theia's rich visual representations encode diverse visual knowledge, enhancing
downstream robot learning. Extensive experiments demonstrate that Theia
outperforms its teacher models and prior robot learning models using less
training data and smaller model sizes. Additionally, we quantify the quality of
pre-trained visual representations and hypothesize that higher entropy in
feature norm distributions leads to improved robot learning performance. Code
and models are available at https://github.com/bdaiinstitute/theia.