Theia: Distilling Diverse Vision Foundation Models for Robot Learning
Authors: Jinghuan Shang
Karl Schmeckpeper
Brandon B. May
Maria Vittoria Minniti
Tarik Kelestemur
David Watkins
Laura Herlant
Venue: CoRL 2024 (2024)
Keywords: visual representation
robot learning
distillation
foundation model
TL;DR: We build a robot vision foundation model by distilling existing vision foundation models, which improves downstream robot learning performance.

Abstract: Vision-based robot policy learning, which maps visual inputs to actions, necessitates a holistic understanding of diverse visual tasks beyond single-task needs like classification or segmentation. Inspired by this, we introduce Theia, a vision foundation model for robot learning that distills multiple off-the-shelf vision foundation models trained on varied vision tasks. Theia's rich visual representations encode diverse visual knowledge, enhancing downstream robot learning. Extensive experiments demonstrate that Theia outperforms its teacher models and prior robot learning models using less training data and smaller model sizes. Additionally, we quantify the quality of pre-trained visual representations and hypothesize that higher entropy in feature norm distributions leads to improved robot learning performance. Code, models, and demo are available at https://theia.theaiinstitute.com.

Theia: Distilling Diverse Vision Foundation Models for Robot Learning
Jinghuan Shang!?, 12, Karl Schmeckpeper', Brandon B. May', Maria Vittoria Minniti', Tarik Kelestemur', David Watkins', Laura Herlant
'The Al Institute Stonn Brook University {jshang, kschmeckpeper, bmay, mminniti, tkelestemur, dvatkins, 1herlant [herlant]@theai jtthaaiisttttttteeee
Abstract: Vision-based robot policy learning, which maps visual inputs to ac- tions, necessitates a holistic understanding of diverse visual tasks beyond single- task needs like classification or segmentation. Inspired by this, we introduce Theia, a vision foundation model for robot learning that distills multiple off-the- shelf vision foundation models trained on varied vision tasks. Theia's rich visual representations encode diverse visual knowledge, enhancing downstream robot learning. Extensive experiments demonstrate that Theia outperforms its teacher models and prior robot learning models using less training data and smaller model sizes. Additionally, we quantify the quality of pre-trained visual representations and hypothesize that higher entropy in feature norm distributions leads to im- proved robot learning performance. Code, models, and demo are available here.
Keywords: Visual representation, Robot learning, Distillation, Foundation model
Distill Visual Understanding dense visual conrespondencce
segmentation
ceepth prediction
Theia-5
80
DINOV2
Depth- Anything
SAM
Thels
R3M
E-RADIO
CLIP
VIT
RADIO-H
Performance
DINDv2:L
laiounlongugge
classifcction
26
DINOV2
WITH
DINDD225 S
Theia
Robot Vision Foundation Model
14
VIL-B
VIT.5
VC1-8-sp
VC1-L-sp
72
VITT
VIP
CDV
10
10a
10
Reboot Learning
MACs (G. log!
Figure I: We introduce Theia, a model that distills multiple vision foundation models (VFMs) to provide better representations for robot learning (left). Theia achieves superior performance on robot learning tasks with less computation compared to standard VFMs and pre-trained models for robotics (right). Results shown are from the MuJoCo subset of tasks in CortexBench.
Introduction
Visual understanding, i.c.. the process of abstracting high- sigh-dimensional visual signals like images and videos, includes many different sub-problems, from depth prediction [1] and vision-language correspondence [2, 3], to tasks ranging from coarse to fine granularity such as classification [4, 5] and object grounding [6, 7, 8], as well as tasks defined along spatial and temporal axes like segmentation [9, 10] and tracking [11]. Given this diversity, a long-standing effort in the vision community has been develop models tailored to one a few specific types of visual understanding tasks. In recent years, several models [4, 5, 10, 2, 3, 12, 13, 6. 14, 15, 10, 7, 16] have achieved remarkable generalizability to unseen domains and new tasks, and are commonly referred to as vision foundation models (VFMs).
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Vision-based robot policy learning, which learns action policies from visual inputs, requires strong and diverse visual comprehension. These policies involve many implicit vision tasks such as object recognition and semantic grounding, where off-the-shed VFMs corresponding to some well-defined tasks can be easily found, but there no single model for all of vision tasks. Studies have shown that off the--hhelf VFMs such as CLIP [2] usually under- perffrr relative visual representation models tailored for specific tasks in robot learning [17, 18, 19, 20, 21]. This fact reveals a gap between the needs of robot learning and the limited visual understanding capabilities of any individual VFM. Our motivation is different than all prior works on learning visual representation models for robotics, where those works focused primarily on improving training data [17, 21, 20J. designing objective functions [17, 18], and directly taking advances of vision architectures [19]. We uniquely focus on improving the visual representation from the angle of solving multiple implicit visual understanding tasks, which will benefit the downstream robot learning.
In this work, we propose combining multiple large VFMs into single, smaller model for robot learning that leverages diverse visual understanding abilities from VFMs. We achieve this via knowl- edge distillation [22]. Unlike conventional distillation from a larger model to a smaller model on the same task, we distill VFMs tailored for varied vision tasks to improve visual representation for robot learning, which is an unseen task for VFMs.
We introduce Theia, robot vision foundation model that simultancously distills off tfflthe-shelf VFMs such as CLIP [2]. DINOv2 [7], and ViT [5]. Compared to off-the-shelf VFMs [2, 5, 7] and prior works [19, 20], Theia offers both better pre-trained visual representations for higher downstream robot learning performance and reduced computational costs. Furthermore, training Theia only requires ImageNet [23] and about 150 GPU hours, in contrast to prior works which necessitate substantially more compute [19, 20, 17, 18]. To understand what makes a good visual representation for robot learning, we observe multiple factors that relate to the performance of downstream robot learning tasks. We hypothesize that higher entropy in representation norms [24] correlates with improved robot learning performance.
In summary, our contributions are:
We introduce Theia, a model that combines knowledge from multiple VFMs into a single, smaller model using knowledge distillation with low training cost.
Through extensive simulated and real-world experiments, we confirm that Theia's visual repre- sentations lead to better downstream robot learning with improved computational efficiency.
We identify key factors relevant to robot learning performance, such as model size, the use of spatial tokens, and the entropy of representation norms, offering valuable insights for guiding future research on optimizing visual representations for robot learning.
2 Related Work
2.1 Visual Representations for Robot Learning
Visual representations are important for vision-based robot policies to parse high-dimenssional visual signals. Visual representation learning can happen different stages, including pre- training [17, 18, 20, 21], joint-learning with robot tasks [25, 26, 27, 28). or combination of both using either trainable or frozen visual representations [29, 30]. Off-the-shelf visual en- coders [4, 5, 2, 31, 32] can also provide visual representations for robot learning. Additionally, an important factor in training visual representations is the choice of data. ImageNet [23], as suggested by Dasari et al. [21], is a particularly effective pre-training dataset, while video datasets [33, 34] are also widely used. Training objectives and auxiliary tasks for visual representation learning vary, including data augmentation [25, 26], prediction tasks [28], contrastive learning [27, 35, 36), and self-supervised learning [19, 20]. Specifically, handle invariance and equivariance in visual obser- vations, inductive biases [37, 38] and constraints [39, 40] can be introduced into neural networks to improve visual representation quality. Unlike prior works, we build a robot vision foundation model from the perspective of merging the visual understanding abilities of VFMs via distillation. Concur- rent work OLA [32] uses pre-trained SigLIP [41] and DINOv2 [7] encoders, which suggests the benefit of diverse visual understanding in robot tasks.

---

Theia- Representation
Distillation
Theia
reature
VFM
Z
Same Input X
((x)
X
More
Feature
More VFMs...
Policy
g: (z)
h hx(x))
Images
Downstream
Robot Tasks
Figure 2: Theia distills multiple VFM features into one rich representation for robot learning. The feature translators (z) are supervised by the features from pretrained VFMs hx(x) during training time, then the distilled representation z is used as input to the policy head for robot learning tasks.
2.2 Vision Foundation Models
Vision foundation models trained on large-scale data exhibit strong task-speciff performance and transferability to unseen domains and new tasks. VFMs can focus on single tasks, multiple tasks, or
remain task-agnostic at the pre-training stage. For example, ViT [5], DeiT [42], and ConvNeXt [43] are designed for image classification, SAM [10] for semantic segmentation, and Depth-Anything [1] for depth prediction. When combined with Large Language Models (LLMs) [44, 45, 46), these models can solve multiple visual tasks including referring segmentation, visual question answering (VQA), and image editing. Task-agnostic models like the DINO series [6, 7] are trained through self- distillation, while CLIP [2] is trained by aligning image-text pairs, Given the strong generalization capability of VFMs, robot learning should also benefit from leveraging the latent representations of pre-trained VFMs. which is the primary motivation for our research.
2.3 Knowledge Distillation in Vision Models
Knowledge distillation [22] compresses the knowledge from one or more larger models into sin- gle smaller model. To leverage VFMs, which are usually computationally intensive, several studies have explored distilling them into more compact models. For example, some works distill a single VFM like SAM [10] into smaller variants [47, 48, 49]. There are also relevant works on combining two models: SAM-CLIP [50] merges SAM [10] and CLIP [2] into one model, while ViLD [51] and GrO [52] combine vision and language models to perform object detection. RA- DIO [53] an agglomerative model that simultaneously distills CLIP, SAM, and DINOv2. These studies show that combined models can improve performance downstream tasks enable new applications. Similarly, in this work, we investigate whether combining VFMs will benefit robot learning. Notably, RADIO [53] is the closest approach to ours. The key differences between Theia and RADIO [53] are that (1) We aim to use our representation for robot learning tasks that none of the VFMs have covered their pretraining tasks, (2) we distill only spatial tokens rather than both spatial and [CLS] tokens, (3) we choose a different set of teacher models, and (4) we show how each teacher model contributes robot learning performance.
3 Method
Overview. We introduce Theia, framework that distills the knowledge of multiple VFMs into smaller model, producing rich spatial representations for downstream vision-basse based robot learning- Figure shows the overall design of Theia Our model comprises a visual encoder (backbone) and a set feature translators for distillation. Note that only the visual encoder used to produce latent representations for downstream robot learning tasks.
Given input image the visual encoder f(-) produces rich representation 2 fxx (called the Theia-representation), which is utilized for downstream robot learning tasks. We focus on backbone models that are smaller than typical VFMs, specifically using ViT-Tiny, Small, and Base [5, 42]' motivated by the limited computing resources on robotic systems. We use Theia-Tiny (Theia-T), Theia-Smal Small (Theia-S) and Theia-Base (Theia-B) to refer to Theia models
Architecture.
For simplicity, we denote DeiT-Tiny and DeiT-Small [42] by ViT-Tiny and ViT-Small DeiT was intro- duced Touvron al. [42] and the original VIT [5] does not have Tiny and Small models.

---

using a vision transformer backbone with the corresponding size. To train the Theia- reeieeeeprssnntation,, we perform feature distillation with the help of feature translators, which will be described below.
3.1 Rich Spatial Representation
The Theis-repressenttiion is a set of encoded tokens corresponding to input image patches. We choose spatial tokens because spatially-dense representations are the foundation for diverse visual understanding, as evidenced by the powerful per-patch features DINOv2 [7]. Therefore, we aim distill all spatial tokens and leave the [CLS] token untouched.
Feature Translators. Our goal is supervise Theia-rrpressnntations with teacher representations from various VFMs. We extract teacher representations h2 (x) of VFMs at the last layer for CLIP [2],
VIT [5] and DINOv2 [7], or before the decoders for SAM [10] and Depth-Anything [1]. Since a single representation cannot be learned to match all the teacher representations directly, feature translators (:).) are used to map the reeeetrepresnntationn, z, to each teacher representation. Feature
translators are shallow CNNs to ensure that knowledge distilled primarily into Theia's visual
encoder. Details are available in the Appendix.
3.2 Training
Distillation Objective. Our training objective is matching the outputs of the feature translators with their corresponding teacher VFM representations. To achieve this, we use a combination of cosine and smooth-L1 losses [53] to match each pair of predicted and ground truth representations for the same image, taking their weighted average. Formally, our loss is
M Σ ass
L(x; 0)
+ (1 - Ea (o))))))))))))))))))))))))))
(I)
where x is the input image, M is the number of teacher VFMs, a; is the loss weight for each teacher, and /3 is the weight for balancing cosine loss and smooth-L1 loss respectively. In general, we set a; = 1/M such that the loss weights each teacher equally. We empirically set 3 = 0.9 [53].
Feature Normalization. To properly accommodate the different scales of teacher representations, we first perform a normalization step. This helps us scale the loss different teacher features evenly and avoid biasing (collapsing) a teacher model with extremely larger norms. We perform the normalization on the teacher representations over each latent dimension, where mean and variance calculated from all ImageNet training samples. Details are in Appendix A.
Dataset. We train model on the ImageNet [23] training set for 50 epochs. We opt to use ImageNet because of its greater diversity compared to human videos [33, 54, 55, 56] and robot datasets (57, 58, 59, 60, 61] within the same number of images. This diversity has been experimen- tally shown to improve visual representation learning [21].
4 Experiments
4.1
Benchmark and Settings
To evaluate pre-trained visual representations, we use simulation tasks in CortexBench [20], which combines MuJoCo tasks (Adroit [62, 63], DeepMind Control Suite (DMC) [64], and Meta- World [65]), Habitat [66, 67] tasks (ImageNav [68], ObjectNav [69], and MobilePick [70].), and Trifinger [20] tasks². ImageNav and MobilePick are reinforcement learning (RL) tasks and others are imitation learning (IL) tasks. We follow the experiment settings of Majumdar et al. [20] and re- port aggregated scores for rewards (DMC tasks) and success rates (all other tasks). For DMC tasks, raw rewards are divided by 10 to be in a scale consistent with the success rate. We also conduct real robot experiments, introduced in Section 4.3. We use the same policy heads for the each type of representations (vector or spatial tokens). Full experimental details are available the Appendix.
Due reproducibility issues, not able to evaluate Move Cube, ObjectNav, and MobilePick tasks

---

Opening
Toy- Microwave Cooking
Adroit
Door Opening
DeesMind Control Suite
Reach Cube
Pick-and-Place
Metaworld
ImageNav
Figure 3: Simulation and real- wol-w (labeled in blue) tasks used in this work. For simulated environ- ments we show one image per task. For real-world tasks, we show images of key steps throughout the task labeled by numbers. A third-person view image shows the setup in Drawer Opening.
4.2 Simulation Results
We comprchensively evaluate Theia and baseline pre-trained models on the MuJoCo subset Cor- texBench [20] to provide an overall assessment pre-trained visual representations. We consider prior works on visual representations for robot learning, including R3M [17]. VIP [18], MVP [19], and VC-1 [20], well as agglomerative models for vision tasks RADIO and E-RADIO [53], and off-the-shelf vision foundation models ViT [5], DINOv2 [7], and CLIP [2]. We also test naive con-
catination of three VFMs (CLIP, ViT, and DINOv2) which are used to train Theia, referred as CDV.
All pre-trained representations are frozen in this experiment. Throughout this section, we answer the following questions:
How does Theia perform compared to baselines?
Which is more effective for visual representations: [CLS] or spatial tokens?
How does robot learning performance scale with the size of the visual encoder?
Theia Performance. As shown in Figure 4, Theia outperforms all evaluated models, surpassing the per- formance of the best prior models, R3M and MVP, as well agglomerative models for vision tasks RA- DIO and E-RADIO. We also tested naive approach of using multiple VFMs (CLIP, DINOv2, and ViT) simultancously by concatenating their spatial tokens channel-wise (CDV in Figure 4), but this performed much worse than using just the individual VFMs. Theia models scale effectively from tiny to base sizes, with Theia-S and Theia-B being the only models break scores of 80 on this subset of CortexBench, even though they use only a small fraction of the inference computation required by comparable models. Theia's training is very efficient, using only the 1.2M images in ImageNet with training time of about 150 GPU hours on NVIDIA H100 GPUs, compared to approxi- mately 5M images used in prior works [17, 18, 19, 20] and IB images used by RADIO [53].
Thiis
Thwas
80
PADIC
DINDV2.8
DINOV--L
DINOv2
VI=L
WITH
VIT5
Vit-8
Vel4
CL CLP
CDV
VC14
V014
CL
DINOV2 cb
DINOv2 dids
Wibteck
VIT
(Spatial
Transforme (CLLS) Toben)
ResNet/Vector)
18a
10'
19'
MACs [G]
Figure 4: Performance MuJoCo tasks vs. inference computation. Theia achieves the best performance with much less com- pute (MACs (G): Multiply-Accumulle operations in billions, log-scale).
Spatial Tokens vs. [CLS] Token. We evaluate Transformer- bassd models (all models in Figure except R3M and VIP) using either their [CLS] token or spatial tokens [5] for downstream robot learning. To accommodate spatial tokens in the robot policy, policy, we introduce extra shallow CNN layers the input of the policy network, known as "compression layer" [71, 20]. Figure shows the results of all models we evaluated, clearly showing that for Transformer-basee models, providing spatial tokens is consistently better than using the [CLS] token for robot learning. This finding applies to both off-the-shelf VFMs and prior pre-trained representations, including MVP and VC-1.
5

---

Mean Corch score across 14 tasks (2 in Adroit, 5 in MetaWorld, 5 in DMC, 1 in Trifinger (Reach Cube), and in Habitat (ImageNav)).
Table 1:
Model Theia-B (Ours) VC-I-L--p [20]
MVP-L.-p [19]
R3M [17]
VIP [18]
E-RADIO [53]
Score 79.79 + 0.14
69.56 + 0.80
77.42 + 3.14 76.51 + 0.79 71.18 + 0.34 73.50 + 1.69
Table 2: Real robot behavioral cloning results measured by success rate. Door Opening and Toy- Microwave Cooking evaluated for 50 trials, and the others are evaluated for 20 trials. We report results for policies trained with either frozen ()) or fine-tuned (4) visual encoders. For tasks having key stages (see Figure 3), we measure the success rates of achieving each stage separately.
Model
Parameters (M)
Door Opening C
Pick-and- Place $
Toy-Microwave Cooking A
Drawer Opening
Open Z Fully Open
? Place
Pick
2 Place
Theia-8 (Ours)
86
92%
66%
85%
75%
58%
52%
40%
85% 1005
E-RADIO [ss]
390)
72%
46%
75%
55%
58%
54%
42%
15% KIF
MVP-1-50 [19]
303
32%
2%
55%
35%
4016
26%
18%
30% 659
VC-1-L-a [20]
303
45%
R3M
24
48%
36%
26%
18%
(1%
DINDv241(7]
303
45%
12%
10%
08
2096
10%
285
35% 20%8
WITH [S]
633
18%
4%
15%
0%
52%
44%
40%
45% / 85%
Scaling with Model Size. We observe that most models, including Theia, achieve better perfor- mance with larger sizes. The scaling effect is more obvious when using the [CLS] token, probably because the [CLS] token encodes less information, making the size of the feature vector more crit- ical. Different models that use spatial tokens also scale varying rates. VC-1 shows only minor improvements when scaling from base large, while VIT shows much larger improvements when scaling from tiny to huge model.
CortexBench Results. We compare Theia-B against baselines over the CortexBench evaluation suite 2. The results Table confirm that Theia-B outperforms all other models.
4.3 Real World Robot Learning
Based on simulation performance, we test Theia-B and the best-performing baseline models: MVP- [19]. R3M [17]. VC-I-L [20], DINOv2-L [7], VIT-H [5], and E-RADIO-L [53] for evaluation on real-world tasks. We employ four tasks (Figure 3): Door Opening, Pick-and-Pllas and Toy- Microwave Cooking with WidowX 250s arm, and Drawer Opening with Boston Dynamics Spot. We train behavioral cloning policies top of visual representations using conventional policy net- works composed of CNNs and MLPs in the WidowX setup and diffusion policy [72] in the Spot setup. During testing, we vary the robot position for Door Opening and Drawer Opening, random- ize the object position for Pick-and-Place and randomize both the object positions and object types Toy-Microwave Cooking. Full experimental settings, including details about the number of col- lected demonstrations, the interface used collect them, and the policy architecture, are available in the Appendix (Sec. D.3).
Table shows the success rates on these real-world tasks. Theia-B achieves the highest success rate across all tasks except Toy-Microwave Cooking. The results also highlight that the Theia- representation useful for both conventional and diffusion-based policy heads, and for either freez- ing or fine-tuning the visual representation. E-RADIO the most competitive model in this setting amongst all models compared, likely due to its similar distillation of VFMs and much larger training dataset. VC-1 has high task variance, performing poorly on Door Opening but adequately on Pick- and-Place. VIT-H works much better when being fine-tuned but DINOv2 does not, which could be caused by some fundamental differences in VFMs.
HujoCo Performance
72
Di
AII-V
AII-
AlI-DI
AIL-
All
CDIV
CDIS
Combination of Teacher Models
Figure 5: MuloCo subset performance with respect to different combinations of teacher models to train Theia-T. Abbreviations of teacher models: V=ViT-H, C=CLIP-L, S=SAM-H, Di=DINOv2-L, De=DDeph--nythhngg, All=all of five models (CDeDiSV), and All- -Itt-iaking X out of All.

---

Table 3: Ablation studies on model design. All experiments are based on Theia-T unless otherwise labeled. Scores are the average performance from the MuloCo subset of CortexBench.
(a) Distillation losses
(b) Number of register tokens (Theia-B)
(c) Tokens to Distill. SP=Spatial Tokens
MSC Cos L1
8
Distill SP
Distill [CLS] SP
78.2 + 1.0 78.9 + 0.8 79.2 + 1.2 80.3 0.7 79.4 0.8 79.6 0.9
Use [CLS]
50.0 + 0.4
Use SP
78.9 + 0.8
77.3 + 2.4
Selection of Teacher Models. Theia is motivated by distilling many vision foundation models, but not all the models can be effectively integrated into one model nor do they contribute equally to downstream robot learning. In these experiments, we empirically identify the most effective combination of teacher models among five candidate VFMs: CLIP (referred to as C) [2], Depth-
Anything (De) [1]. DINOv2 (Di) [7]. SAM (S) [10], and ViT (V) [5]. We select these candidates because they have been designed to perform well across various important vision tasks.
To select the best combination, we train Theia-T using different sets of teachers and evaluate the learned representation on the MuJoCo subset of tasks. Figure 5 shows the results of different teacher combinations. We start with single VFM teachers and observe that C and are the most beneficial. We then distill All teacher models altogether and discover that it performs better than four out of five single-teacher distillations, but performs worse than only distilling V, suggesting possible negative effects from some of the teacher models. We then remove each teacher model from All and distill the remaining four (AII-X). The results show that removing V and C from the teachers causes the most significant performance drops while taking out S and De leads to either similar or improved performance. Given the (nearly) negative effects observed, we distill the rest of the three models (CDiV), which results in the best performance. We also try the teacher combination that was performed in RADIO [53] (CDiS), but it does not outperform CDiV.
Model Design. We conducted ablation studies on distillation loss choices and model architecture. We compared two kinds of distillation losses: MSE and Cos+L1 from Ranzinger et al. [53]. As shown in Table 3a, we find that Cos+1.1 performs better. For model architecture, we examined the effect of "register tokens" introduced by Darcet et al. [24]. These extra tokens are introduced at the input without supervision and are discarded when using the representation. With the same amount of spatial tokens supervised by distillation, we varied the number of register tokens of between 0, 1, 4, and 8. Note the unsupervised [CLS] token is removed in zero register token case, and it counts for 1 register token in other cases. According to Table 3b, we find that using I register token worked the best, while having no register tokens performed the worst. Compared to RADIO [53], which distills both [CLS] and spatial tokens, our use of the [CLS] token as a register token provides an advantage. We also evaluated distilling both [CLS] and spatial tokens in the Theia model, and the results shown in Table confirm the advantage of exclusively distilling spatial tokens.
4.5 Qualitative Visualizations
Inout Image
DINOV2
SAM
Depth-Anycting
We present qualitative visualizations to demon- strate how Theia-roprsssnntations can be trans- formed into teacher representations through fea- ture translators. Using Theia trained with all teachers (CDeDiSV, All,, we applied PCA for visualizing predicted DINOv2 [7] features, used the SAM [10] decoder produce segmentation results, and used the Depth-Anything [1] head to produce estimated depth. Results are shown in Figure 6 with more examples in the Appendix. The visualizations indicate that our predicted rep- resentations can be decoded by the original VFM and produce reasonable results. Encouragingly, we find that predicted depth maps from the Theia-predicted representation appear to be more accu- rate compared to the original Deyth-Anything model, particularly in the stove-top area and at the microwave door. This shows the potential benefit of combining different visual understandings.
Original
VFMS
Figure 6: Visualization of VFM outputs from Theia predicted representations (top) and origi- nal VFM representations (bottom).
7

---

Reguiar Models
VC2, v.
Trmm
VCL
MYP
0.343
76
kacropy
Distileed Hiddels
Fann 023
thela, base cdiv
Fumn C33
Yone 003
theie, bate. cdll
Forwo
20
2
223
70
Figure 7: Left: Correlations between feature norm distribution entropy and robot learning perfor-
mance; Right: Visualizations of spatial token feature norms from VC-I-L-sp and Theia-B.
What Makes Visual Representations Good for Robot Learning?
Traditionally, the quality of the pre-trained visual representations is evaluated through downstream robot learning like II. or RI. However, it is unclear why different visual representations lead to varying robot learning performance outcomes. In this section, we quantify the quality of visual representations and analyze how they correlate with downstream robot learning performance.
Feature Norm Distributions and Entropy. Darcet et al. [24] analyzed the norm of spatial tokens vision Transformers and found that high-norm outlier tokens are detrimental to vision task perfor- mance. Following this, investigate whether similar phenomenon arises in visual representations for robot learning. We inspect the feature norms of Theia with different teacher combinations and baseline models evaluated in Section 4.2, and their corresponding performance on the MuJoCo sub- set tasks. We sample 1 %% of the MuloCo task training set and calculate the 1.2-norm of each spatial token after encoding images, We measure the entropy of the feature norm distribution across all samples and patches per model and use it as a quantitative metric. To calculate the entropy of the distribution, we first discretize the distribution by a histogram. Then, we use the following formula obtain the entropy
E,(r: * log(p:)), where p; is the probability of each discretized bin.
We confirm that similar outlier tokens also appear in VC-1 corresponding to the image patches that are not task-relevant, shown in the visualizations of feature norms on the right of Figure 7. In contrast, Theia has very few or no outlier tokens, and the tokens with higher norms are more task- relevant even though Theia-representationn are not trained on these robot images. In our quantitative analysis (Figure 7, left), we divide the models into distilled and regular based on the observation that distilled models generally have higher entropy (fewer outliers, Figure 4(c) Darcet et [24]). We find that there is strong correlation (R=0.943) between entropy and robot learning performance among regular models, and high correlation (R=0.638) among distilled models. We hypothesize that spatial token representations with high entropy (better feature diversity) encode more informa- tion that aids policy learning, while less diverse representations (low entropy) may hinder it. In the Appendix G, we discuss the results of other quantitative measurements, including feature similarity and eep-explained variance ratios, where no strong correlations are found.
Conclusion
In this work, introduced Theia, novel robot vision foundation model specifically distilled from multiple VFMs to enhance robot learning tasks. Theia builds rich visual representation from diverse VFM teachers, preserving spatial features ensure detailed visual comprehension. Through extensive evaluations on CortexBench and in the real world, Theia consistently outperforms state- of-the-art models, including all prior models for robot learning, off-the-shelf VFMs, and similarly distilled models for vision tasks. Our results highlight the effectiveness of distilling multiple VFMs into a compact model for superior performance a variety of robot learning scenarios. Furthermore, answer a key question about what kinds of visual representations lead better robot learning by finding strong correlation between the entropy of feature norms and enhanced downstream performance, offering insights for future research optimizing visual representations for robotics.

---

Acknowledgments
The authors thank Deva Ramanan for inspiring discussions the analysis of trained models, Osman Dogan Yirmibesoglu for his help with the Spot grippers, and Joe St. Germain and Jien Cao for their assistance in setting up the WidowX robots. We also extend our gratitude to Erica Lin and Ahmet Gundogdu for their work on Spot's policy learning infrastructure, as well as Nathan Williams and Brennan Vanderlaan for DevOps support.
References
[1] I.. Yang, B. Kang, 7. Huang, X. Xu, J. Feng, and H. Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. arXiv preprint arXiv:2401.1089) 2024.
[2] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, al. Learning transferable visual models from natural language supervi- sion. In International Conference on Machine Learning, pages
PMIR, 2021.
[3] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[4] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 770-778, 2016.
[5] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[6] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 9650-9660, 2021.
[7] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
[8] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf. Object-centric learning with slot attention. Advances in Neu- ral Information Processing Systems, 33:11525-1152 2020.
[9] Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3431-3440, 2015.
[10] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[11] Yang, M. Gao, Z. Li, S. Gao, F. Wang, and F. Zheng. Track anything: Segment anything meets videos. arXiv preprint arXiv:2304.11968 2023.
[12] D. P. Kingma and M. Welling Auto-encoding variational bayes. 1rXiv:1312.6114 14, 2013.
arXiv preprint
[13] M. Caron, I. Misra, Mairal, Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 12991229924, 2020.

---

[14] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[15] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022.
[16] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next-generation image-text models. Advances in Neural Information Processing Systems, 35: 25294,, 2022.
[17] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. arXiv preprint (tii:2203./2600 2022.
[18) Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.000300 2022.
[19] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.061 2022.
[20] A. Majumdar, K. Yadav, S. Arnaud, J. Ma, C. Chen, S. Silwal, A. Jain, V.P. Berges, T. Wu,
J. Vakil, et al. Where are we in the search for artificial visual cortex for embodied intelli- gence? Advances in Neural Information Processing Systems, 36, 2024.
[21] S. Dasari, M. K. Srirama, U. Jain, and A. Gupta. An unbiased look at datasets for visuo-motor pre-training. In Conference on Robot Learning, pages 1183-1198. PMLR, 2023.
[22] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 5.5500223//, 2015.
[23] Deng, W. Dong, R. Socher, I. Li, K. Li, and I., Fei-Fei. Imagenet: large-scale hierarchi- cal image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages
2009.
[24] Darcet, M. Oquab, Mairal, and P. Bojanowski. Vision transformers need registers. The Twelfth International Conference on Learning Representations, 2024.
[25] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data, Advances in Neural Information Processing Systems, 33:19884-19899 2020.
[26] D. Yarats, I. Kostrikov, and R. Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2020.
[27] M. Laskin, A. Srinivas, and P. Abbeel. Curl:
Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[28] X. Li, Shang, Das, and M. Ryoo. Does self-supervised learning really improve reinforce- ment learning from pixels? In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 30865-30881. Curran Associates, Inc., 2022.
[29] R. M. Shah and V. Kumar. Rrl: Resnet as representation for reinforcement learning. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9465-9476.
PMLR, 18-24 Jul 2021.
10

---

[30] C. Wang, X. Luo, K. W. Ross, and D. Li. VRL3: A data-drives framework for visual deep reinforcement learning. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
[3]] X. Li, C. Mata, Park, K. Kahatapitiya, Y. Jang, J. Shang, K. Ranasinghe, R. Burgert, M. Cai, Y. J. Lee, and M. S. Ryoo. Llara: Supercharging robot learning data for vision- language policy. arXiv preprint arXiv:2406. 240.09400000095,, 2024.
[32] M. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, Liang, and C. Finn. Openvla: An open-source aarlon-lnngagg-aceeo action model. arXiv preprint arXiv:2406.0922 2024.
[33] K. Grauman, A. Westbury, E. Bymee Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[34] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pages 720-736, 2018.
[35] Sermanet, C. Lynch, Y. Chebotar, Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 1134-1141. IEEE, 2018
[36] J. Shang and M. S. Ryoo. Self-supervised disentangled representation learning for third-person imitation learning. In 202/ IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 214-221. IEEE, 2021.
[37] Shang, S. Das, and M. Ryoo. Learning viewpoint-agnostic visual representations by recover- ing tokens in 3d space. Advances in Neural Information Processing Systems, 35:31031-31044, 2022.
[38] Shi, Qian, Y. J. Ma, and D. Jayaraman. Composing pre-trained object-centric rep- resentations for robotics from "what" and "where" foundation models. arXiv preprint arXiv:2404.0 13474, 2024.
[39] D. Wang, R. Walters, X. Zhu, and R. Platt. Equivariant g learning in spatial action spaces. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings the the 5th Conference on Robot Learn- ing, volume 164 of Proceedings of Machine Learning Research, pages 1713-1723. PMLR, 08-11 Nov 2022.
[40] K. Chen, X. Chen, Z. Yu, M. Zhu, and H. Yang. Equidiff: A conditional equivariant diffusion model for trajectory prediction. In 2023 IEEE 26th International Conference Intelligent Transportation Systems (ITSC), pages 746 751. IEEE, 2023.
[41] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975-1 1986, 2023.
[42] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data- efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[43] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11976-11986 2022.
11

---

[44] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Milli- can, M. Reynolds, et al. Flamingo: A visual language model for few-shot learning. Advances in Neural Information Processing Systems, 04223716-23385 2022.
[45] J. Chen, D. Zhu, X. Shen, X. Li, Z. Liu, P. Zhang. R. Krishnamoorthi, V. Chandra, Y. Xiong, and M. Elhoseiny. Minigpt-v2: Large language model as unified interface for vision-language multi-task learning, arXiv preprint arXiv:2310.09477 2023.
[46] Xu, L. Xu, Y. Yang, X. Li, Y. Xie, Y.J. Huang, and Y. Li. u-llava: Unifying multi-modal tasks via large language model. arXiv preprint arXiv:2311.0534 2023.
[47] Z. Chen, G. Fang. X. Ma, and X. Wang. 0.1% data makes segment anything slim. arXiv preprint arXiv:23/2.05284, 2023.
[48] Y. Xiong, B. Varadarajan, L. Wu, X. Xiang, F. Xiao, C. Zhu, X. Dai, D. Wang, F. Sun, F. Ian- dola, et al. Efficientsam: Leveraged masked image pretraining for efficient segment anything. arXiv preprini 1rXiv:23/2.00863 2023.
[49] Z. Zhang, H. Cai, and Han. Efficientvit-sam:: Accelerated segment anything model without performance loss. arXiv preprint arXiv:2402.050008, 2024.
[50] H. Wang, P. K. A. Vasu, F. Faghri, R. Vemulapalli, M. Farajtabar, S. Mehta, M. Rastegari, O. Tuzel, and H. Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. arXiv preprint arXiv:2310. 15308, 2023.
[51] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and lan- guage knowledge distillation. arXiv preprint arXiv:2/04.1399 2021.
[52] Liu, Z. Zeng. Ren, Li, H. Zhang, J. Yang. C. Li, J. Yang. H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arX1v:2303.05499 2023.
[53] M. Ranzinger, G. Heinrich, J. Kautz, and P. Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12490-12500, June 2024.
[54] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al. The "something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision, pages 5842-5850, 2017.
[55] D. Shan, J. Geng, M. Shu, and D. Fouhey. Understanding human hands in contact at internet scale. In 2020 IEEF/CVE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[56] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, Moltisanti, J. Munro, Perrett, W. Price, and M. Wray Scaling egocentric vision: The epic-kitchens dataset. In European Conference Computer Vision (ECCV), 2018
[57] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651-673. PMLR, 2018.
[58] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and
C. Finn. Robonet: Large-scale multi-robot learning, arXiv preprint arXiv: 2019.
[59] Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and
S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109. 13390, 2021.
12

---

[60] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023.
[61] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x mod- els. arXiv preprint arXiv:2310.0886, 2023.
[62] V. Kumar. Manipulators and manipulation in high dimensional spaces. PhD thesis, University of Washington, Seattle, 2016.
[63] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. arXiv preprini arXiv: 1177009.0077 2017.
[64] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancg, et al. Deepmind control suite. arXiv preprint arXiv:1811.00990 2018.
[65] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1100. PMLR, 2020.
[66] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, Liu, V. Koltun, Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339-9347, 2019.
[67] A. Szot, A. Clegg, E. Undersander, Wijmans, Y. Zhao, Turner, N. Maestre, M. Mukadam, D. S. Chaplor, O. Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 34:251-266, 2021.
[68] L. Mezghan, S. Sukhbaatar, T. Lavril, O. Maksymets, D. Batra, P. Bojanowski, and K. Alahari. Memory-augmented reinforcement learning for image-goal navigation. In 2022 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 3316-3323, IEEE, 2022.
[69] K. Yadav, R. Ramrakhya, S. K. Ramakrishnan, T. Gervet, J. Turner, A. Gokaslan, N. Maestre, A. X. Chang, D. Batra, M. Savva, et al. mabitat-matterpont 3d semantics dataset. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4927-4936, 2023.
[70] Gu, D. S. Chaplot, H. Su, and J. Malik. Multi-skill mobile manipulation for object rear- rangement. arXiv preprint arXhv:2209.02778. 2022.
[71] K. Yadav, A. Majumdar, R. Ramrakhya, N. Yokoyama, A. Baevski, Z. Kira, O. Maksymets, and D. Batra. Ovrl-v2: A simple state-of-art baseline for imagenav and objectnav. arXiv preprint 23Xv:22333.07798 2023.
[72] C. Chi, S. Feng, Y. Du, 7. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[73] S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smymis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
[74] M. El Banani, Raj, K.-K. Maninis, Kar, Li, M. Rubinstein, Sun, L. Guibas, John- son, and V. Jampani. Probing the 3D Awareness of Visual Foundation Models. In CVPR, 2024.
[75] A. Xie, L. Lee, T. Xiao, and C. Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation, 2023.
13

---

A Theia Model Architecture
Backbone. We use the DeiT-Tiny, DeiT-Small, and DeiT-Bare models [42] as our backbone archi- tectures. We keep the [CLS] token in the model and in the forward pass, but there no supervisory signal provided for it. As result, the [CLS] token serves a "register token" [24], which provides
some benefits for learning high quality representations. We train Theia from scratch (no pre-trained DeiT [42] weights are applied).
Feature Translators. The feature translators are composed primarily of CNNs, with a linear layer appended the end match the teacher's representation dimension. Pure linear transforms might not be able to map rheiereeprssnntationn to all three teacher representations well, resulting in failure of learning (See Table 6a). Thus, we use three CNN layers to account for the fact that each teacher model's representations are very different from one another. Details are listed in Table 4, where we show the architectural details of the translators used for our (student, teacher)-feature pairs.
Table Feature Translator configurations
Student d, x 14 x 14 (Theia backhome= -S, -B) to Teacher d, x 16 x 16 (CLJP, DINOv2, ViT) Con/Transpose2ddd.... d,, kerne stride=1, output.padding=0) LayerNorm
Conv2d(d,, d,, kernel size=3, padding=1)
Rel.U+LayerNorm
Conv2d(d.,. d,, kernelsiize-) padding=1)
ReLU+LayerNorm
Flatten and Linear(d,, d,)
Student d, 14 14 to Teacher 64 64 (SAM and Depth-Anything
ConyTranspose2dddd d,, sim=3, stride=2, padding=1 LayerNorm
Com/Transpos:2d(d,, d., kemne stride=2, output.padding=)) ReLU+LayerNorm
Conv2d(d,, d,, kernel size=3, padding=1)
ReLU++LayrrNorm
Flatten and Lincar(d,, de)
Feature Normalization. Formally, the normalization is:
h(x;). h((x)c= the Oc Mhe E h(xx)) de
Eg (1((x)) - 112
where is the channel index for the feature, j is the index of image sample, and N is the number of samples in ImageNet.
B
Training
We train Theia on 8 NVIDIA HI00 GPUs. The main bottlenecks in training are the data transfer speed between devices and the GPU memory bandwidth load large spatial feature tensors, for example, of size 1280x 16x16 for ViT-H and 256x64x64 for SAM. We pre-compute the features from all teacher models instead of doing inference the fly. This approach requires extra storage space to save all the features extracted from the VFMs, but significantly saves training time and avoids loading models with high GPU memory usage during training, such as Depth-Anything or SAM (a batch size of 16 cannot into 80GB of GPU memory). All training configurations are listed Table
Teacher VFM Features. We the output representations at the last layer of VIT [5], CLIP [2], and DINOv2 [7]. For SAM [10], use its encoder output. For Depth-Anything [1], since it is
14

---

Table 5: Theia Training Configuration
Hyperparameters
GPUs
Batch size
16 / GPU (128 effective)
Learning (LR)
5e-4
LR Schedule
Constant
Weight decay
0.01
Optimizer
AdamW W [0.9, 0.999]
Betas
Epochs
50
Warm-up epochs
Warm-up LR schedule
Linear (1e-2*LR)
Gradient clipping
None
Image augmentation
None 152
Total GPU hours
Table More ablation studies on model design. All experiments are based on Theia-T. Scores are the average performance from the MuJoCo subset of CortexBench. (a) Feature Translator Architecture
(b) Training from Scratch vs Pre-trained Backhone
CNNs
Linear
Training from Scratch Pre-trained Backbone
78.9 0.8 41.7 + 1.6
78.9 0.8
80.8 + 1.5
initialized from DINOv2, we use the latent representation before the final convolution layer. When decoding SAM and Depth-Anything results from Theia-predicted representations, we send the pre- dicted representations through the remaining layers of original models and obtain the output.
C Additional Ablation Studies
We conduct two additional ablation studies to verify design choices in the Theia model. The first is comparison between the current CNN-based feature translator and linear feature translator. In Table 6a, we find that using a Linear feature translator leads to a significant performance drop. The second ablation studies whether Theia should be trained from scratch or be initialized using the pre-trained DeiT [42] weights. In Table 6b, we find that using pre-trained weights improves the downstream performance. This could be interpreted as the positive effect of incorporating knowl- edge from an additional useful model into the distillation process. We would expect to similar performance improvements as more informative models are included during training.
D Full Experimental Settings
Table 7: Comparison of model architectures, training datasets, total number of images, objectives, and training duration (epochs or GPU hours) across the models used in this paper. We use the numbers reported in their original papers and stands for we could not find such information.
Mokd
Archdaate
Dinastol
Toul lnaoo
Otjecire
Thening Durzins
RADXO [SRAAI01S1
VITS2R NTnSsk-driggdd driggdd
DuraCterp 18P1
1.
Destlovve
HHRRIGH
MITMAR [14
InugsNo-14 N 14
1982
[ig
1630 caacts
BJM [T]
NoNaa
Resio Payi
Thez Conzadoc [-]j- Valll/aasssLaaaaeec.. Alaanen: VIPTIS)
wige
VIP 4
Bocke
BIND
VI
DKDAR2M
143M HEMM L0MI 700M
2001 GPO BINOs koan
Image uct conmartia kaminn
TmageNae 334 [1)
D:1 [44]
VI
Inagchislk
1-2
300 cooepddo 255 CPU
Leenves V100
15

---

D.1 Baseline Models
Theia and baseline models are trained different sizes of datasets using different objectives. We organize these details in Table 7 to provide a comprehensive comparison between them.
D.2 CortexBench
For all of our CortexBench experiments, we use the original setup [20], except for a few modifica- tions to produce more reliable results. The modifications include:
We increase the number of evaluation roll-outs from 10 (original) to 25 (ours) in DMC tasks. The mean scores reported are from a total of 75 rums (25 per seed x 3).
We remove the noise added to the policy network output the CortexBench code base. The noise causes minor performance degradation (about 1.0 on overall mean score for MuJoCo tasks) compared to the version without noise.
We modify the policy networks to take spatial feature inputs for MuloCo and Trifinger tasks (details follow and are presented in Table 8).
Note that prior models including R3M, VIP, MVP, and VC-1 all re-run using the same settings in MuloCo tasks for the purposes of making a fair comparison when evaluating against Theia.
Policy Networks. For MuJoCo and Trifinger tasks we utilize three-layer MLP for vector-based representations, including ResNet models and Transformer models that use the [CLS] token. For models that generate spatial feature maps, such Transformers using spatial tokens, introduce three-layer CNN before the MLP. For Habitat tasks, exclusively benchmark models that produce spatial feature maps and adopt the same policy network as used by Majumdar et al. [20]. Details can be found in Table 8.
Table 8: Policy Networks for MuJoCo Tasks
Spatial Representation dimension x x W
Conv2d(d, 256, kernel size=4, stride=2, padding=1)
ReLU
Conv2d(256, 256, kernel_size=3, strideee=
ReLU
Conv2d(256, 256, kernel-size=3, stride=1)
Flatten and Linear(256, 256)
Linear(256. 256) Li5ear(256, action dimension)
Vector Representation dimension d
Linear(d, d, 256)
Linear(256, 256)
Linear(256, action dimension)
For ImageNav task, we use the provided policy network from VC-1 [20] without modification. The policy network is composed by compression-layer (a CNN layer) to convert the spatial feature map into vector representation, followed by 2-layer LSTM. Details available in Appendix A.2 in [20].
D.3 Real World Robot Learning
D.3.1 WidowX Arm Experiments
WidowX Arm Setup. The robot used for these experiments is 6-degree-of of-freedooomm-- (DOF) Wid- owX 250s arm. The data collection and evaluation framework is based on [59].
16

---

Table 9: Number of demonstrations and data collection method for the real-world experiments.
Task
Door-Opening
Pick-and-Place
Toy-Microwave Cooking
Drawer Opening
# Demos
48
63
101
50 Scripted Policy
Method
Teleoperation
Teleoperation
Teleoperation
Table 10: WidowX Policy Training Configuration
Hyperparameters
Batch size
16
Learning rate
le-4
Weight decay
0.01
Optimizer
AdamW
Betas
[0.9, 0.999]
Epochs
400
Loss
SmoothL1
We train behavior-cloning policy for each of the four evaluated setups and for each evaluated baseline (see Table 2 and Figure 3); the training hyperparameters are shown in Table 10. To train each of the three tasks performed with the WidowX robot, i.e., Door Opening, Pick-and-Place, and Toy-Microwave Cooking, we collected human demonstrations by teleoperating the robot with a Vir- tual Reality (VR) controller using the setup introduced in [59]; the number collected demonstra- tions for each of the tasks is reported in Table 9. The policy's observations are RGB images and robot joint states. Images are encoded by pre-trained visual encoder and randomly initialized, unfrozen feature neck ("compression layer" [71].. We use the same feature neck as we did for the previously discussed MuJoCo tasks in Section 4.2. The encoded vector is concatenated with the robot's joint states, which is fed into a 3-layer MLP with a hidden dimension of size 256. The policy outputs end-effector commands, consisting of the end-effector's delta positions (Cartesian coordi- nates), delta rotations (Euler angles), and the gripper's 's opening/closing command; such commands are tracked by the robot at a frequency of 5 Hz. addition to the hyperparameters listed in Table 10, we vary the policy action prediction horizon depending of the difficulty of the task, i.e., at each step the policy predicts the next 10, 10 and actions for Door Opening, Microwvve Cooking, and Pick-and- Place, respectively.
In the following, we give more details about the WidowX tasks showcased in our work.
Door Opening. In this task the robot has to open a fridge door in a toy-kitchen setup; we identify two stages to evaluate the task's success: Open and Fully Open (see Figure 3). We place the robot in front of the fridge and collect 63 demonstrations to train the behavioral cloning policy. We vary the height (z-axis) of the robot base between 40-46cm, and the position (x-axis, parallel to the toy kitchen) of the robot base; samples from the initial state distribution of the demonstrations are shown in Figure 8. At inference time, we vary the height of the robot base among {40, 42, 43, 44, 46) cm, and select between 5 randomly-picke positions along the x-axis (for all policies). For each robot position, we evaluate the policy twice, for a total of 50 runs.
Pick-and-Place. In this task, the robot has to pick up pink cup from a toy-sink and drop into drying rack located on the left of the sink. We collected 48 demonstrations train the policy, where we varied the initial pose of the objects, as shown in Figure 8. During evaluation, vary the cup's starting position amongst total of 10 positions, of which 8 positions are equally distributed about the perimeter, and 2 are in the center of the sink. We also roughly vary the direction of the cup handle towards the left or the right. In total, we evaluate this task for 20 runs. There are two key stages for which we measure the success rate: picking up the cup and successfully releasing into the drying rack.
-ic-wiie Cooking. In this task, the robot has to pick up object from within the pot the stovetop, putting the object into toy-microwave, and closing the microwave. In each test,
17

---

Figure 8: Samples from the initial state distribution of the collected demonstrations for fridge door- opening (top), pick-and-place (middle), and toy-microwave cooking (bottom). For each of these tasks, the factors of variation are, respectively: the robot height and position with respect to the door, the initial pose of the cup, and the initial position of the objects and object types.
Figure 9: Samples from the initial state distribution of the collected demonstrations for the Spot drawer opening experiments. For this task, the factors of variation are the position and orientation of the robot with respect to the drawer.
initialize the environment with the mierowave door open. In this task, we collected 100 demonstra- tions across 10 different toy-food objects (10 demonstrations per object) with randomized object positions; examples from the initial state distributions of the collected demonstrations can be seen in Figure 8. During evaluation, we test 40 runs on 10 seen, in-distribution objects (4 runs per object), and 10 runs on 5 unseen, -uttt--diittibuttin objects (2 each), for a total of 50 runs. Furthermore, we vary the position the pot that holds the object. We point out that this task is characterized by longer horizon with respect the other two; in fact, it has three different steps: 1. picking up the ob- ject, 2. placing the object into the microwave, and closing the microwave. In addition, the policy needs to generalize the different poses of the objects the stove-top and out-of-distribution objects. The longer horizon and the variety of objects in this task make it particularly challenging, so using frozen visual encoders was not effective (0% success rate). However, with fine-tuning, the policies performed reasonably well.
D.3.2 Spot Experiments
Spot Sctup. For the Spot experiments, we train diffusion policy [72] conditioned the encoded image. The diffusion policy outputs the desired absolute positions and rotations of the end-effector and the gripper state. Hyperparameters for the policies shown in Table 11.
18

---

Table 11: Spot Policy Training Configuration
Hyperparameters
Batch size
32
Learning rate
3e-4
Weight decay
cosine
Optimizer
Adam
Betas
[0.9. 0.999]
Training Iterations
3000
Loss
MSE
Policy Horizon
4
Drawer Opening. In this task, the robot has to open the top drawer of a cabinet. The policy receives color images from single forward facing camera mounted the body of the Spot. A fiducial marker is used to enable the robot to walk to random position and orientation for each trial. Samples from the distribution of random initial states are shown in Figure 9. The starting locations vary by #5 em in and and +0.2 radians in orientation. 50 successful demonstrations were collected using a scripted policy and we evaluate each policy for 20 trials. A trial considered successful the drawer is opened at least cm. After each trial, a scripted policy uses the fiducial marker reset the environment and the robot moves new random location.
E
More Visualizations of Translating Teacher Model Features
Depth-
Depth- Anything
Input
DINOV2
SAM
Anything
Input
DINOV2
SAM
Theia
Original
VFMs
Theia
Original VFMs
Figure 10: More examples of decoding receiserpresnntttioon VFM outputs using feature translator and original VFM decoders. We select robot images from our experiment recordings. Theia and VFMs are not trained on these images.
We attach examples of decoding Theia-representationn of 4 frames from robot video into VFM outputs in Figure 10. Note that Theia and VFMs are not trained on the robot images on which we run this evaluation.
Per-Task CortexBench Results
In Table 12 we report per-task scores of the models evaluated in Figure 4, over the MuJoCo subset of tasks. In Table 13, report per-task scores of all 14 tasks evaluated in Cortexbench, corre- sponding to Table 1. Note that perform the evaluation following the original Cortexbench [20]
19

---

protocol, where there are a total 75 runs per MuJoCo task (we increased it from 30 to 75 for DMC tasks), 100 runs in Reach Cube, and 4200 runs in ImageNav.
Table 12: Per Task Results on the MuJoCo Subset.
Model
Ascceniy
Sorposiig
peto
Checab-ras
Readec-ceay
Willz--tad
Walke-waak
Hasueer
Pcs
Rehosse
Thuser
14.5045-00 o0
53//611
05 10.7
156 527
192
1000030110
58.0042.85
16:0001381
54.072022
70
13.00a14.22
51.375225 25
19
1400
10
10000:00 00
57.378480 60
113322.3 31
50.6766111
DINOVD4
01.33-8
50.00a4
35e2.3
45:665.00
1097-022 25
14.5
10000
00000
3334
40000000
DINOXS-B
03.0058
76 N.m=144 1:43
1000 00
15.073771
10.77-159
02 06514133 81
53
100 10
05.672333 51
00000:00 10
53 315.3.31
02.3728.25
01:13412222
45.415-25
181111:00
102003100
VIEH
94.7729
18101:240 12:0
5585534
47.09230.00
09.5421.16
3585-45
36
19.9985:24
1030022100
54.022235 51
56 00000030000
11534461
WELL
7530385.4 56
5418615-46
50.3285.2
02 07.3431100
2.63 25
141458233 30
77:458120 30
10003201000
950083-2 27
88 08:006227
41 M123-22
VITS
01.3348
37.3143.11
35e6
42.98a1
06666.2 59
07.53+100
10503-0.00
(0-100
VIST
07.3348
67
66 13410010 17
790741 87
71 1/211110 07
71 13-11.4
78 701
100 10
14.00M4 95
60 000011055 10
VC-14-4p
10000.00
56.584610 66
14-0::7
83 63-78+3.00 2:40
16303-91 10
11
16:000000
1B-40000
75.35624.44
RADIO
95.9221.99
14
1267412250
153221 59
71992144
78 03.665339
2M1662422
51.358491
105002000
95.612251
(8.0055000
ERADIO
9461221 24
82692251
5110
57.378227 27
01881116
75 29.5961:29
01-732233
51.2222115
102012000
1001020000
05 0567212556
43.362331
53.4484-1
MVN.
41.77933
12.6744 31
16674585
6161-3:0
99-118
08:041.17
10500-0:00
15.0046
62
3100-100
RTM
96.936600 95
00.001101 m
110-
15.0541 13
3065-034 14
17-1770
13 92-3-70
06 174444
1620)-0000
08.6747.1 51
75 33333-331
VIP
91.3148
1000-000
45/041
02
10
6319341.00
068763.31
46.034100
75356611
76-34410
G Analysis of Visual Representations
Entropy of the Representation Norm Distribution. Given N representations produced by en- coding N images per model, where each representation contains P spatial tokens, we discretize the distribution of token norms over all NxP tokens by using a histogram. We normalize the count of each bin in histogram by the total number of tokens to obtain the probabilities of each bin. We then calculate the Shannon entropy, given by Σ; Pi log(pt). We find that the distilled models have higher entropy than the regular models, so we divide them into two distinct groups Results are plotted as model performance vs entropy on the MuJoCo tasks. We attach the full version of plot presented on the left of Figure here in Figure 11, including two plots corresponding to each category of models and one plot for all models.
At the top of Figure 12, find that both CLIP and VC-1 have high-norm outlier tokens. To better visualize the values of normal tokens, we use the median of norm values to clip the values. Specifically, we clip the norm values to range [0, 2 * median] and visualize the clipped norm values on the bottom of Figure 12. We find that the high-norm tokens are still not task-relevant.
In Figure 13, we attach the feature norm map of all other models. Among those, we find that MVP [19], which performs well on CortexBench, also produces features without outlier tokens. Feature norms from Depth-Anything [i] and SAM [10], in contrast, have low diversity.
We investigated the distributions of the feature norms that could give more information. Plots in Figure 14 we show the histogram of the feature norm entropy of 4 selected models: VIT-B, DINOV2- B, VC-1-B, and Theia-B (refered to as ryfm deit basttaaaeeeeee 4features in the legend).
We find that Theia has the most balanced distribution of the feature norm, while the others are long- tailed. VIT-B has two peaks in the histogram. VC-1 is extremely centered and heavily long-tailed. DINOv2 is relatively less long-tailed than ViT-B and VC-1, and has a relatively nicer distribution around the median value.
PCA Explained- Explained-Variance Ratio of Representations. Similar to the entropy analysis, given NxP spatial token representations, we apply PCA to them and extract the explained-variance ratio (EVR) of each latent dimension. We calculate and plot the cumulative sum of EVRs, well calculate the Area Under the Curve (AUC) of cumulative sum of the EVR. When comparing Theia-B with VIT-B, DINOv2-B and VC-1-B (Figure 15), we find that Theia-B has the lowest AUC and the best MuJoCo performance, while VC-1 has the highest AUC and the worse MuJoCo performance amongst these models. The higher AUC is caused by one or few principle components that have very high EVRs, indicating that these components are capturing the majority of the variance of the feature representations. This means that less information is encoded within such representations. In contrast, the reeasseepresnnnttoonn has a low AUC which we believe is due to the rich information that has been encoded within the latent space.
20

---

Table 13: Per-task results on CortexBench (excluding Move Cube, ObjectNav, and MobilePick due to reproducibility issues).
Maadi
Aswerdty
Chetton
Walaseds
Houdusds
Thoast KL.00k.0
828115 11
40402155
00.3223
91.002.11
on Pottr
90
30302221
13.722122
566/11122
Sostan11
3001117
MVFSLPP
6002121
210121122
0134012
2000-100
(8:00-10)
32.324100
3838633
Shar
73 13:160333
15.00-191
64636333
90.0-110
73.3
N..0+15.00
However, when extending the scope to encompass all the models we evaluated (Figure 16 left), find that the AUC of the EVR does not have strong correlation with robot learning performance.
Cosine Similarity of Representations. We also use cosine similarity to analyze the representa- tions from different models by first calculating the mean of all representations and then computing cosine similarity between each representation and this mean representation. Results are shown the right of Figure 16, which shows very weak correlation between cosine similarity and perfor- mance on ConcexBench.
H Linear Probing on ImageNet
In addition to robot learning, we evaluate the Theia--epressentatinn on vision tasks to show to how well such abilities are maintained after the distillation process. For example, to evaluate image classification performance
apply linear probing the Theia-representation to classify images from ImageNet- 1k [23]. We use mean pooling of the theia-representattinn (i.e. spatial tokens) and the same training schedule MAE [14]. Results shown in Table 14, where find that Theia outperforms MAE [14] at the same model size, but is not comparable to SOTA results from models like DINOv2 [7].
Table 14: ImageNet- Ik [23] evaluation accuracy using linear probing.
Model
Accuracy
Theia-B
72.1% %
Theia-B (initialized from DeiT-B [42] weights)
75.2%
MAE (ViT-B) [14]
67.5%
DINOv2 [7] (ViT-L)
84.5%
I Probing 3D-awareness of Theeia-representation
We evaluate reeeiirepresentaiion 3D awareness to show the learned diverse vision knowledge in the representation. In particular, we adopt probe3D [74], an evaluation toolbox to investigate the 3D-awareness of pretrained models. This evaluation includes Monocular Depth Estimation, Surface Normal Estimation, and Multiview Correspondence. We present the evaluation results in Tables 15, 16 and 17, and we summarize them here:
Theia achieves the third best Depth Estimation performance among over 10 types of vision models (only worse than DeiT-III and DINOv2 models) (Table 2 in Appendix of probe3D [74], NYU dataset).
Theia achieves comparable performance as the best performing model, DINOv2, on Multiview Correspondence on ScanNet (Table 4, Block3 in Appendix of probe3D [74]).
Theia has better Surface Normal Estimation performance than CLIP (Table Appendix of probe3D [74], NYU dataset).
We note that the Theia model is trained only on ImageNet with 224x224 resolution images, while many baseline models in this benchmark are trained billions of images and higher resolutions. Theia achieves better performance than models with similar training data and resolution.
21

---

Regular Models
DINOv2
NVP
MuloCo Per refoomence
78
MVP-L-5p
DINOV2-L
0.943
MC1--e5p LC
72
79.
5.5
6.0
6.5
7.0
7.5
8.0
Entropy
Distilled Models
82
Theio
Theie-T_cdiv cdive
Thein-8 cdiv
Theia Butdisv
DINOV2
RABIO
E-RADIO
ediv RADIO
& cdiiv
MoloCo Pei feforance
78-
Thea ediov
The vil ADIO-H
Thela-T clip GINOV2.B
DINOV2-5
74-
Theia sam
Theja Binov2 M5
72
TReig-T depthanything
7g. 5.0
5.5
6.0
6.5
7.0
7.5
8.0
Entropy
Thela VIT
ThelaTT coivr
Theig B_ddiv Theie-B -diivv
80
Theis S cdisv
The _ai RADIO
MujoCo Performence
78
E-RADID
Theia-T_call caiiv MVP
Theis L
Theia T. rlip
DiNDv2-L
MT-H
DINOX--B 8
VTTL
DINOvZ-5
74
CLIP-L VC1-B-Bp VCLLsp VT-B
Theia-T serVIT-
Mieli-T_ddinve
VIT-T Theio-T depthanything
70.0
5.5
7.0
8.0
Entropy
Figure 11: Full results of feature norm entropy.
Table 15: Depth Estimation Table 16: ScanNet Multi-view Table 17: Surface Normal Esti- mation on NYU dataset
NYU dataset
Correspondence
Model
6g
6
RMSE
Model
8/
600
Model
11.25
22.5"
30'
RMSE
Theia
0.8420
0.9699
0.9929
0.4665
47.88
37.47
Theia
30.22
54.22
65.13
33,59
22

---

Hond.
Figure 12: Feature norn map visualizations of CLIP and VCI, original (top) and with high-norm outlier values clipped (bottom)
Figure 13: Feature map visualizations of VIT [5], DINOv2 [7], MVP [19], Depth-
Anything [1], SAM [10], RADIO [53], and E-RADIO [53].
Generalization Evaluation
We evaluate Theia-B and baseline models on simulated benchmark, Factor-World [75], test the generalization ability of the pre-trained visual representations, We use the Door-Open task from Factor- World benchmark environment, for which we collected 100 trajectories across 20 randomized environments. We trained the policies using the same policy head that we use in CortexBench MuJoCo tasks (Table 8), and we evaluated the generalization ability when varying each of 6 factors of variations over 20 different runs controlled by one factor at a time. In Table 18, we report the success rates of the baselines when varying each factor. These results show that Theia- has the best generalization abilities compared R3M, E-RADIO, and MVP.
23

---

Trature Doer Patch
17500
dra/b
2500
189
350
260
Pasiare Norn 2striction vitch
14300
1230
4000
3500
26
Photiara Morm direntt
12300
1200
4300
1200
Testure kerm Datritution.. Patch vel Vit.
15300
1000
1000
6f's JoMe, Phatur
4200
1200
-000
2230
Figure 14: Distrubtions of feature norms. From top to bottom: 4 models on the same plot, ViT-B,
DINOv2-B, VC-I-B, and Theia-B.
24

---

PCA Expaired Varaince Ratio Cumsum
0.0
0.5
VTb
dinov2 vel
600
Figure 15: Cumulative sum of PCA Explained Variance Ratio of features from ViT-B, DINOv2-B, VC-1-B, 1-B, and Theia-B.
PCA Explained Variance Ratio AUC vs CortexBench Performance
Similarity CortexBench Performance
gheia_base_com
base, cdisv
thela wwt_bbaeeecooo
80
pheaa small cdiiv
Treiassmall_oo _disv
radio
radao
hhii cndiy greia _ii_innyyy vit
70
theiattny tiny.yccyyyyyvvvoo
radio
mmpl_sp
updio
@heie_ting vi
pvpl
tiny Ith dinov2 dinov2b
dnovatheia wtwn _lppp Ano
76
0.085
dEnoVES
R: -0.207
74
@11 vitl
vel.
ytts theia tiny sam
vRb Sp
#c1, @ct_vitb_gg
theia tiny Sam
dinov
72
tiny. nov2
theia tiny, depthanything
utt
theia, wwwpjiinn_eeppee dewthanytffing
70
0.4
0.6
Figure 16: PCA explained variance ratio-AUC (left) and cosine similarity (right) vs MuJoCo per- formance of many models evaluated.
Table 18: Generalization evaluation Factor-World
Model
Light
Table Texture
Table Pose
Camera Pose
Floor Texture
Arm Pose
Theia-B
100%
90%
100%
90%
100%
55%
E-RADIO
100%
85%
80%
90%
100%
40%
MVP-L
100%
50%
20%
70%
100%
45%
R3M
100%
35%
70%
90%
100%
50%
25


Reviews:
[{"review_id": "nV8d5DsGFc", "reviewer": "Reviewer_CYjq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 1, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents a simple approach for distilling existing pretrained visual representation models such as CLIP, DINOv2, and SAM into a unified, smaller model. Through straightforward application of knowledge distillation techniques, the resulting family of Theia models (at different parameter counts) show marginally stronger results on policy learning (behavioral cloning and RL) in both simulation and the real-world compared to individual representations and naive agglomeration strategies.", "summary_of_recommendation": "Based on the above weaknesses and lack of important experimental details, I recommend rejection at this time.", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Q1. Given that you’re learning the feature translators anyway, why is the feature normalization step (Section 3.2) necessary? Doesn’t this by nature change the feature norm distribution across patches (nullifying the analysis at the end of the paper)?\n\nQ2. The paper is currently missing a lot of important details about the evaluation. While many of the task settings are from the VC-1 paper, it would be worth going into more detail about the exact setup here — especially since in the Appendix it seems that some details of the VC-1 evaluation were tweaked for this work. For the real world experiments, how many demonstrations were collected, and with what interface? Who collected them? What does the initial state distribution actually look like? How did you choose the policy architecture? What hyperparameters did you sweep over?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VCIGQTOTiR", "reviewer": "Reviewer_TPg9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This submission introduces Theia, a vision foundation model for robot learning applications, distilled for multiple foundation models for primitive vision tasks. During training Imagenet samples are processed through several VFMs (Clip, Dino,ViT etc) and their resulting spatial features are used for distillation-based training of another ViT-based transformer encoder, exploiting some learnable feature translator modules, to conclude to a unified representation. Additionally, an interesting insight that correlates the entropy of features norms with its effectiveness to robot learning tasks.", "summary_of_recommendation": "Overall, the manuscript contributes a new state-of-the-art vision foundation model for robotics, achieving state of the art accuracy on robot learning tasks. The paper is well-written and the results are convincing. The proposed distillation scheme and visual representation evaluation approach, although particularly effective, can be considered incremental to existing approaches.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "- Do the authors have any insights, or qualitative observations, that justify the poor performance of Segment-Anything and Depth-Anything models? Such visual cues can be considered very important for robot manipulation, and it is surprising that their participation in the proposed distillation scheme downgrades accuracy. \n\n- Would a more naïve fusion method (e.g. feature vector averaging or appending) on the distillation of different models be comparably effective with the proposed feature translators? Experiments indicated that this is not a good option for off-the-shelve VFM fusion, but under the finetuning scheme of the proposed distillation approach, the contribution of the feature translators remains unclear.\n\n- It is unclear whether the authors intend to open-source the weights of the proposed Theia model. Since the SoTA accuracy of the model is in the reviewers opinion the most important contribution of this submission, sharing it with the community would enhance the argument for paper acceptance. \n\n- How could the insights obtained by the proposed Feature Norm entropy metric be fed back to the training process, to improve its accuracy or data efficiency ?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fDDZCPJuwf", "reviewer": "Reviewer_GPK7", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 1, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper produces a new VFM that distills knowledge from multiple existing VFMs such as CLIP, SAM, DINO, etc.", "summary_of_recommendation": "Overall the novelty against RADIO is limited.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Line 131-132: how are the teacher representations normalized? Is it batch normalization or layer normalization?\n2. Could you explain more on the spatial tokens?\n3. Why does the CLS token encodes less information than the spatial tokens?\n4. Line 216: as far as I understand ViT is more like an architecture than a special model.\n5. The formula for the entropy of the feature norms needs to be written down.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents Theia, a distilled vision foundation model for robot learning, combining features from multiple pre-trained models. The reviewers acknowledge the paper's timeliness and extensive evaluation across simulated and real-world environments, showing competitive performance and improved inference efficiency. However, concerns are raised about the novelty of the approach, given its similarity to previous work like RADIO. The evaluation strategy is questioned, with suggestions for more robust generalization tests and diverse robot learning tasks beyond simple policy learning. While the feature norm entropy analysis is interesting, reviewers note it could be further developed.
Following the rebuttal, two of the three reviewers engaged in discussions with the authors, and the majority of their concerns were addressed, with both leaning towards accepting the paper.
Decision: Accept