Token Bottleneck: One Token to Remember Dynamics
Authors: Taekyung Kim
Dongyoon Han
Byeongho Heo
Jeongeun Park
Sangdoo Yun
Venue: NeurIPS 2025 poster ()
Keywords: Self-supervised Learning
Representation Learning
Primary area: deep_learning
TL;DR: We introduces a simple yet effective pipeline called Token Bottleneck that facilitates conservative summarization of the observed scene into a bottleneck token while enable capturing of dynamic transitions through the bottleneck token.

Abstract: Deriving compact and temporally aware visual representations from dynamic scenes is essential for successful execution of sequential scene understanding tasks such as visual tracking and robotic manipulation. In this paper, we introduce Token Bottleneck (ToBo), a simple yet intuitive self-supervised learning pipeline that squeezes a scene into a bottleneck token and predicts the subsequent scene using minimal patches as hints. The ToBo pipeline facilitates the learning of sequential scene representations by conservatively encoding the reference scene into a compact bottleneck token during the squeeze step. In the expansion step, we guide the model to capture temporal dynamics by predicting the target scene using the bottleneck token along with few target patches as hints. This design encourages the vision backbone to embed temporal dependencies, thereby enabling understanding of dynamic transitions across scenes. Extensive experiments in diverse sequential tasks, including video label propagation and robot manipulation in simulated environments demonstrate the superiority of ToBo over baselines. Moreover, deploying our pre-trained model on physical robots confirms its robustness and effectiveness in real-world environments. We further validate the scalability of ToBo across different model scales. Code is available at https://github.com/naver-ai/tobo.

Token Bottleneck: One Token to Remember Dynamics
Taekyung Kim
Dongyoon Han'
Byeongho Heo' Jeongeun Park
Sangdoo Yun
'AVVRR AI Lab Korea University
(taekyung. dongyoon. han, bh. heo, sangdoo, baoooo..ya yun./mavercorppcoo
baro09060korea.co
Abstract
Deriving compact and temporally aware visual representations from dynamic scenes is essential for successful execution of sequential scene understanding tasks such as visual tracking and robotic manipulation. In this paper, we introduce Token Bottleneck (ToBo), simple yet intuitive self-supervised learning pipeline that squeezes a scene into bottleneck token and predicts the subsequent scene using minimal patches hints. The ToBo pipeline facilitates the learning sequential scene representations by conservatively encoding the reference scene into compact bottleneck token during the squeeze step. In the reconstruction step, we guide the model to capture temporal dynamics by predicting the target scene using the bottleneck token along with few target patches hints. This design encourages the vision backbone to embed temporal dependencies, thereby enabling understanding of dynamic transitions across scenes. Extensive experiments in diverse sequential tasks, including video label propagation and robot manipulation in simulated environments demonstrate the superiority ToBo over baselines. Moreover, deploying pre- tre-trained model physical robots confirms its robustness and effectiveness in ToBo across different model scales. Code is available at https: //g: thub. com/ naver- ai/tobo.
environments. We further validate the scalability of
1
Introduction
With the increasing interest in deploying machines in real-world environments, ensuring seamless perception and interaction with their surroundings has emerged a crucial challenge. These operations are inherently sequential in nature, requiring the ability trace objects (e.g., visual tracking) and predict future actions (e-g., manipulation) based on current and immediate past observations. Such understanding of the surrounding environments primarily depends on vision backbones. Therefore, strong and robust backbone capable of generalizing across diverse tasks and environments is essential for effective sequential scene understanding.
Self-supervised learning (SSL) of visual representations has been highlighted as pivotal research in vision domains, with the pre-trained models being widely adopted for effective backbone deployment. A series of studies have introduced promising recipes for learning image [3, 4, 5, 6, 8, 14, 18] and video representations [36, 44] without labeled data. However, these studies primarily focus on understanding entire scenes or videos, which poses limitations for sequential scene understanding, as requires capturing temporal changes across consecutive scenes and conservatively encoding the visual states of observed scenes.
To address the challenges, sequence of studies [12, 16, 22] have attempted incorporate correspon- dence learning into the MAE [18] framework, aiming to retain its strong localization capability while enabling the model to match corresponding regions across consecutive scenes. However, we observe that such additional considerations have a limited impact on the quality of scene representations and may result suboptimal performance in sequential scene understanding tasks, such as robotic manipulation ($3.2). This limitation arises since recognizing temporal changes alone is insufficient;
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

DeepMind Control 73.5
Noto vs
SinCLe
MetaWorld
DRC
Adroit
SiamM4E
CropMAE
RLBench 41.6
Ours
Franka
Reference Scene
Target Scene Xt-k
Kitchen
Conventional SSL encoders
Token Bottleneck
66.6
Trifinger
74.8
JHMDB
60.6
DAVIS
(a) Illustration of Token Bottleneck (ToBo)
(b) Overall performance comparison
Figure 1: (a) We describe the underlying mechanism our Token Bottleneck (ToBo) pipeline during pre-training, which conservatively encode reference scene into bottleneck token and predict the subsequent target scene based on a scarce target patches and the bottleneck token. ToBo facilitates learning the capability of temporal progression recognition and preservation of observed information (top). Therefore, using bottleneck tokens from the current and recent past observations enables the robot to better understand its current state (bottom). (b) Our method significantly surpasses previous self-supervised visual representation learning methods designed for static [4, 8, 18] and dynamic scenes [16, 22, 49] on various robot manipulation and locomotion tasks.
these tasks require the ability to summarize the essential information from each scene without loss, while preserving temporal cues within the summarized representation.
In this paper, we introduce Token Bottleneck (ToBo), a simple yet effective SSL approach that intuitively facilitates the conservative summarization of observed scenes while enabling effective recognition temporal evolution within the summarized representations. As illustrated in Fig. la, into a bottleneck token and then predicts the subsequent target
ToBo squeezes a reference scene using only a minimal set patches as hints. This design enforces strong reliance on the bottleneck token, encouraging the vision backbone to capture essential scene information. Moreover, predicting the target scene from the bottleneck token implicitly embeds temporal dependencies, guiding the vision backbone to generate representations capable of capturing dynamic transitions across consecutive scenes.
We conduct comprehensive experiments assess the effectiveness of our pre- pre-training pipeline in comparison with existing self-supervised learning methods. We evaluate our method on various sequential understanding tasks, including manipulation tasks in simulated environments and video label propagation tasks, surpassing baselines [4, 5, 8, 12, 16, 18, 22, 49] with significant gaps (see Fig. 1b). Furthermore, deploy our pre-trained models real-world robots, demonstrating strong generalization performance in unseen physical environments. Finally, we validate the scalability of approach by observing consistent performance gains across various model scales.
2 Related Work
Self-supervised learning on a static scene. Self-supervised learning (SSL) approaches have been widely explored the image domain. Contrastive learning approaches [3, 5, 8, 17] aim learn useful representations by maximizing the similarity between positive pairs derived from static scene through strong augmentations. Although these methods excel in facilitating cohesive understanding of images, they suffer from limited localization capabilities [27], essential for action prediction in roboties. On the other hand, masked image modeling (MIM) [1, 2, 18, 27, 28, 50] has recently gained attention for its promising capacity to learn visual representations through predictive learning. Inspired by masked language modeling (MLM) in transformers [10], BEiT [2] extends MLM into the vision domain, adopting external offline tokenizer. MAE [18] and SimMIM [50] showcase efficient MIM by directly reconstructing masked input pixels without any tokenizer. However, these approaches do not incorporate mechanisms for capturing temporal progression during pre-training.

---

Predicted d Scene Yeek
100
SiamMAE
Decoder d
Ours
Mio
Success Rates
Encoder fe
20
Knab1
Gdoor Micro
Refer Scene X₁
Target Xt+k
Figure Comparative analysis for motivation. We compare robot manip- ulation performance using MAE and SiamMAE visual backbones. While SiamMAE employ temporal correspon- dence to the limitation of MAE, its im- provement over MAE remains limited.
Figure 3: Overview of our Token Bottleneck (ToBo). Our ToBo reconstructs the masked patches from the bot- tleneck token representation of the reference scene xf and extremely searce patches from the target scene x Such extreme scarcity leads the decoder dæ to rely heavily on reference scene x facilitating the preservation of observed information in the bottleneck token.
Self-supervised learning on dynamic scenes. Recent studies have focused on enhancing the recognition of dynamic transitions. SiamMAE [16] proposes visual representation learning methods that utilize dynamic scenes. CropMAE [12] introduce a simple augmentation strategy that enables the generation of dynamic scenes even from single static image. On the other hand, RSP employs stochastic frame prediction tasks along with masked autoencoding. Several works have also explored applying these techniques to embodied agents and robotic manipulation. For example, VC-1 [33], MVP [40], and Dasari et al. [9] adopt MAE objectives for visual pretraining, while STP [51] builds on SiamMAE with reference masking strategy. On the other hand, some prior works investigate representation learning with annotated supervision. Theia [42] distills representations from large scale pre-trained teacher networks, some of which are trained with annotation supervision, into student models. MPI [24], Voltron [25], and R3m [34] explore language-drivee representation learning, leveraging auxiliary textual guidance through manually annotated data. In contrast, focus self-supervised learning directly from raw dynamic scenes without any guidance from annotations.
3 Method
3.1
Preliminary
Masked autoencoding, Given a scene image, we patchify the image into N non-overlapping pxp- size patches (x:) where E R33p We randomly select masked patch set M C 2 N). N) with a ratio r E (0, where [ = [rN]. The remaining patches [x: NE fed into the encoder fa, becoming spatial representations {u:JieM- where u, E Rd for encoder dimension d. Note that learnable CLS token e(CLs) is also encoded with spatial representations
part of the encoding process. The encoded tokens are reconstructed to N tokens by substituting masked positions to mask token m Rd i.e. u, + m for i M. The decoder de gets {u, ) input and predicts the masked image patches (Ee using encoded tokens.
3.2 Motivation
Real-world robots must see and act seamlessly in complex dynamic environments. This fundamental challenge motivates our central research question: how should a vision backbone learn capture spatiotemporal relationships across a sequence of observations? While conventional self-supervieed vision encoders [16, 18, 22, 50] have been widely employed for this purpose, they possess significant limitations from the perspective of sequential scene understanding. In this section, we discuss the strengths and limitations of these prior approaches, thereby providing the insights for our proposed method.

---

Limits of self-supervised learning on static scenes Self-supervised learning approaches on static scenes such as MAE [18] and SimMIM [50] are effective for appearance modeling and localization, which leads to their adoption in several robotics studies [9, 33, 40]. These strengths stem from the design that enforces the autoencoder to predict missing information from available prior information (e.g., visible patches). This pipeline implicitly encourages the encoder to facilitate interactions among the remaining sparse tokens, thereby enhancing localization capabilities. However, since their predictive learning is performed exclusively within single static scenes, the encoder is never explicitly optimized to compare consecutive frames, leaving them ineffective at modeling temporal dynamics. Moreover, a recent study reveals that they struggle with learning broader contexts [27], leading to representations with a limited cohesive understanding. These limitations further constrain their potential for sequential scenes understanding. Consequently, MAE shows limited sequential understanding and thus underperforms some manipulation task, as depicted in Fig. 2.
Limits of self-supervised learning with patch-wise temporal correspondence To alleviate the chromic limitations of static scene-based SSL approaches, SiamMAE [16] builds a non-tri non-trivial correspondence matching problem by randomly sampling two dynamic scenes from sequential data. The core principle involves propagating patches from the reference scene to their corresponding locations the target scene. Applying this guidance using a cross-attention layer-based decoder encourages patch-level correlation between target patches and reference patches. However, while the training objective successfully enforces patch-level correspondences, it overlooks the crucial step of interpreting what these collective matches represent from a holistic perspective. Consequently, despite being built upon the MAE framework, its performance gain over MAE is marginal or even negative in some sequential scene-based tasks, as shown in Fig. 2. This suggests that fine-gra fine-grained recognition of temporal evolution is insufficient for sequential scene understanding, and a conservative summarization of the observed scenes is essential.
Computational inefficiency of combinatorial architectures A common approach to achieve com- prehensive capabilities to construct a combinatorial architecture that integrates separate pipelines specialized for each desired capability. For examplc, RSP [22] combines masked autoencoding, global representation alignment, and target scene reconstruction for localization, global understanding. and patch-level correlation, respectively. However, this combinatory design leads to a substantial increase in computational overhead. As a result, RSP requires more than double the computation cost of competing methods, as reported in Table 9.
3.3 The Proposed Method Token Bottleneck (ToBo)
Our claim Our goal is to achieve representations optimized for resolving sequential scene- basee-baeed tasks. In light of the discussions in $3.2, we extend our focus beyond simply recognizing tempo- ral evolution; we consider the conservative summarization of observed scenes in way that also effectively embeds temporal dynamics within the summarized representation.
To this end, we present Token Bottleneck (ToBo), self-supervised visual representation learning pipeline that enables these capabilities through a token bottleneck mechanism. ToBo consists of two key steps: squeezing scene into a single token, which we denote as the bottleneck token, and reconstructing information from this token. Suppose a reference scene and a target scene are given. In the squeeze step, visual information from the reference scene is compactly encoded into the bottleneck token. Subsequently, in the reconstruction step, we guide the model to predict the target scene using the bottleneck token, with only minimal set of patches from the target scene provided as hints. In this situation, the model cannot precisely reconstruct the target scene based solely on the limited hints, which strengthens the reliance of the reconstruction step the bottleneck token. This design yields two advantages: (1) the bottleneck token should preserve essential information from the reference scene, and (2) such information should be encoded in a way that enables recognition of temporal dynamics when interleaved with the hints from the target scene. Eventually, our goal can be achieved by optimizing the objective of the Token Bottleneck pipeline. The overall description of our pipeline is depicted in Fig. 3.
Overall pipeline formulation Suppose we sample a reference scene x² E x and a target scene x E [33HXW HXW with a temporal gap k. We patchify x' and x ++k into N non-overlapping patches [x()-V] and (x Ji=1- respectively. The reference scene patches {x()_V] are fed into an encoder fo. yielding spatial representations (uf).M.e. We use the CLS token output from this encoding process as the bottleneck token Urobo which will be guided to compactly summarize the

---

Table 1: Experimental results vision-based robot policy learning on Franka Kitchen. We report the performance of imitation learning agents on Franka Kitchen [15], which are trained upon representations from the VIT-S/16 model pre-trained on Kinetics-400 [26] dataset. The success rates (%) are reported for all the tasks. We underline the second-best performance. We report the gains of our method over the second-best baseline.
Tasks
SimCLR
MoCo v3
DINO
MAE
SiamMAE
RSP
CropMAE
ToBo
Knobl on 2.
11,5+39
27.0=332
12.0+3.3
16.8+4.4
31.0+24
31,5453
57.0=20
Light on
55.8=664
24.3+5.0
44,3=665
24.3++4.
36,5+7.0
44.5+5.6
54.0=11.2
82.0=166
Sdoor open 72.3+28
66,5+3.2
77.0=5.0
71.5+43
68.0+79
82,5+27
77.0+8.1
95.0=7.1
Ldoor open 17.0:22
10.312.1
16.5:22
12.8:3.9
17.3133
28.84448
25.5:15.7
51.0=1.4
Micro open 23.3::28
14.31:2.5
28.5:448
10.0
13.5
30.3:15..6
32.5
55.0=1.4
Table 2: Experimental results on vision-based robot policy learning on CortexBench. The performance of imitation learning agents CortexBench [33] is reported, where the agents trained upon representations from the ViT-S/16 model pre-trained on the Kinetics-400 [26] dataset. We report the normalized score for DeepMind Control Suite (DMC) and success rates (%) for other tasks. We report the gains of our method over the second-best baseline.
Tasks
SimCLR
MoCo
DINO
MAE
SiamMAE
RSP
CropMAE
ToBo
Adroit 30.4
39,6
45.6#622
39.6
44.0=6.6
45.6+44.6
50.0+5.1
2.24
MetaWorld
78.4::5.2
65.4
82.4
65.4
81.1
84.5-66.6
82.4
87.8
DMC
39.7::2.9
43.7:t3.2
50.9
43.7+:3.2
56.0:229
61.6413.4
46.41:111
73.5
TriFinger
63.3:13.3
53.3:E1.6
64.241..5
53.311..
52.1
66.2:10.88
46.3:11.7
66.5
reference scene. The target scene [x] is masked with an extremely high ratio r E (0, 1). where J. The unmasked target patches {x! JieM- are processed by
M C (1, N and [M] = [rr]] the same encoder fo. producing (u.+*) JEEM for the target scene. We then concatenate the bottleneck token Usobo with the target representations (ut+** and fill mask tokens for missing regions i M. These are passed the decoder dg. which predicts the masked image patches {x ).EM by using Utobo and (ue+*) Tee Due to the extremely high masking ratio applied the target scene,
the decoder de proactively rely on Urobo, which enable the encoder fa to conservatively summarize the reference scene way that facilitates temporal reasoning when compared to the target hints. We minimize the reconstruction loss throughout the training as follows:
LTOBO
Σ
d(x
***)
SEM
where d(()) is a distance function; we use cosine distance for the pre-training.
Decoder structure Previous methods in dynamic SSL [12, 16, 22] utilize cross-attention layers as core component for learning temporal evolution awareness, placing them within the decoders to guide the encoder to learn representations that effectively capture correspondences. These approaches leverage hybrid structure of cross-attention layers, self-attention layers, and multi-layer perceptron (MLP) layers. In contrast, ToBo employs self-attention layers ensure that the decoder exclusively attends to the given information during the reconstruction step, with MLP layers for progressive transformation from representation embedding spaces into the pixel space.
4 Experiment
In this section, we focus on demonstrating the effectiveness of our pre-training pipeline through fair comparisons with existing self-supervised learning methods. To this end, we evaluate our method on sequential tasks, including video label propagation tasks [23, 38, 55] and vision- based policy learning for robotic manipulation and locomotion across various simulated environments [15, 21, 33). We extend our validation to real-world settings by deploying our pre-trained model on physical robots, showcasing its transferability. We further investigate the scalability of our method. In the appendix, we validate our claim regarding the importance of extremely high masking ratios to the target scene, present qualitative comparisons manipulation processes against baseline methods, and show provide demonstrations of real-world manipulation tasks.
5

---

Table 3: Experimental results vision-based robot policy learning RLBench. We report the performance of imitation learning agents on RLBench [21], which are trained upon representations from the ViT-S/16 model pre-traine traindried on Kinetics-400 [26] dataset. The success rates (%) are reported for all the tasks. We report the gains of our method over the second-best baseline.
Tasks
SimCLR
MoCo v3
DINO
MAB
SiamMAB
RSP
CropMAE
ToBo
Button
7.4
11,414.1
24.7=1.5
6,412.
6.1
28.44:3.0
26.916.7
41.2
Phone
34.626.6
36.2434
32.0-555
37.7+19
5,4
48.0
16.6+3.8
52.343.2
Umbrella 5.8+33.
13.2
28.1=14
10.0
4.0=0.0
37.3++3.0
37.5+
42.2+6.9
Wine 11.0+2.11
8.740.7
31.4=155
10.042.1
8.7+0.8
31.9422
33.2+:02
35,4
Rubbish 5.2+1.2
6.740.8
12.9=1.5
6.243.2
3.5+0.9
18.5
20.6++1.7
37.0+6..1
Table 4: Performance on real-world vision-based
Figure 4: Real-world robot trajectories. Initial, intermediate. and final states of the robot during (a) Cabinet Opening, (b) Drawer Closing, and (c) Cup Stacking.
robot policy learning. Success rates (%) of imitation learning agents on three manipulation tasks: Cabinet Opening, Drawer Closing, and Cup Stacking. Agents are trained with ViT-S/16 representations pre-trained on Kinetics-400 [26] for 400 epochs. The results demonstrate generalizability of ToBo real-world.
(a)
Cabinet Opening
Drawer Closing
Cup Stacking
Method
SiamMAE
20.0
55.0
50.0
RSP
25.0
65.0
55.0
CropMAE
0.0
25.0
20.0
(c)
ToBo (ours)
65.0
75.0
80.0
4.1 Experimental Setup
Implementaion details We follow the evaluation protocol of Jang al. [22] for both video label propagation and vision-based policy learning on simulated environments. To ensure fair comparisons with the baselines, we also pre-train our method Kinetics-400 for 400 epochs. Detailed explanation for both pre-training and evaluation are provided in the Appendix.
Baselines We compare the performance of our method with conventional self-supervised learning (SSL) methods for visual representations including SimCLR [5], MoCo v3 [8], DINO [4], and MAE [18] We also consider previous dynamic scene SSL methods, i.e., SiamMAE [16], RSP [22], and CropMAE [12]. We validate the impacts of explicitly learning state representations over these approaches.
4.2 Vision-based robot policy learning in simulated environments
We evaluate method through imitation learning robot manipulation and locomotion tasks across various simulated environments. Specifically, we evaluate five tasks from both the Franka Kitchen and RLBench benchmarks. Moreover, consider two, five, five, and two tasks from Adroit [41], MetaWorld [53], DeepMind Control Suite (DMC) [43], and TriFinger [46] from the CortexBench benchmark, respectively.
Franka Kitchen. We present a comparison between our method and the baselines on vision-based robot policy learning in the Franka Kitchen environment in Table 1. The results demonstrate that our method significantly outperforms all the baselines across all tasks. Notably, our method achieves over 20% improvements in success rates on all tasks, except for the Light task. This highlights the effectiveness of explicitly encoding visual state representation for Assionbased robot policy learning.
CortexBench. We compare our method with the baselines for the iision-based robot manipulation and locomotion tasks the Adroit, MetaWorld, DeepMind Control (DMC), and Trifinger environments in Table 2. The results show that our method achieves superior performance compared to the baselines across all tasks. In partieular, our method surpasses the second-best performance with success rate gains of 11.9% DMC and 10.4%p Adroit.

---

Table 5: Results on video label propagation. We report performances on video segmentation, video part segmentation, and pose tracking tasks from DAVIS [38], VIP [55], and JHMDB [23] benchmarks, respectively. For all methods, we report the performance with the representations pre-trained on the Kinetics-400 [26] dataset for 400 epochs.
DAVIS
VIP
JHMDB
Method
F&Fm
Im
Fm
mloU
0.1
PCK @0.2
SimCLR
53.9
51.7
56.2
31.9
37.9
66.1
MoCo
57.7
54.6
60.8
32.4
38.4
67.6
DINO
59.5
56.5
62.5
33.4
41.1
70,3
MAE
53.5
50.4
56.7
32.5
43.0
71.3
SiamMAE
58.1
56.6
59.6
33.3
44.7
73.0
RSP
60.1
57.4
62.8
33.8
44.6
73.4
CropMAE
58.6
55.8
61.4
33.7
42.9
71.1
ToBo (ours)
60.6
58.4
63.0
34.0
47.0
74.8
(a) Semantic Part Propagation
(b) Object Propagation
(c) Pose Tracking
Figure 5: Qualitative results for video label propagation. We provide examples of predicted propagation of our model on video object segmentation, video part segmentation, and pose tracking benchmarks. The leftmost images indicate the ground-truth annotations. We visualize the propagated results corresponding to 25, 50, and 100% ratio of the videos.
RLBench. Table showcases the robot manipulation performance on five demonstration tasks the RLBench environment. Notably, our method consistently exceeds all baselines across the five tasks. Moreover, the degraded performance of MAE and SiamMAE further highlights the significance of state representation learning for the robot backbones.
4.3 Vision-based Robot Policy Learning in Real-world Environments
Quantitative comparison. To validate the robustness of our method in real-world environments, further investigate SSL methods real-world robot manipulation tasks. Specifically, design three demonstration tasks: Cabinet Opening, Drawer Closing, and Cup Stacking. For each task, We collect 50 demonstration episodes for training and 10 demonstration episodes for evaluation for imitation learning. Following the training protocol used in simulated environments, we train the policy network using a standard behavior cloning loss. The experimental results for each individual task are reported in Table 4. We first observe that our method exceeds SiamMAE [16], RSP [22]. and CropMAE [12] on all three tasks. Specifically, our method improves 40%p. 10%p. and 25%p over the baselines on the Cabinet Opening, Drawer Closing, and Cup Stacking tasks, respectively. While previous SSL methods on dynamic scenes struggle with tasks that require relatively high precision, like cabinet opening tasks, our method even successfully executes the task with considerable success rate. This showcases that models pre-trained by our method can be robustly transferred to real- world environments.
7

---

Table Comparison with robot representation learning models. We compare the performance of our method with robot representation learning methods across multiple simulated manipulation tasks. We categorize the methods into seff-supervise learning, supervised learning through foundation models, and supervised learning with auxiliary language guidance. Despite the unbalanced training and evaluation setup, ToBo surpasses the RRI. models on MetaWorld. Moreover, ToBo exceeds self-supervised RRL models despite of smaller model with significantly smaller amount of data. These results demonstrate the effectiveness the representations learned by ToBo diverse robot manipulation tasks.
Method #aram
Datasct
# Seen frames
Adroit
MctaWorld
Franka Kitchen
Supervision through Foundation Models
Theia
52.9M
Theia datasel
14.4B*
66.0
86.
Supervision with Auxiliary Language Guidance
R3M
25.6M
Ego4D
0.8B
65.0
69.2
53.1
MVP
21.7M
MVP dataset
4.8B
84.6
Voltron
21.7M
SS-v2
0.3B
68.7
70.5
MPI
21.7M
Ego4D
0.1B
85.7
76.5
Self-supervised Learning
R3M
25.6M
Ego4D
0.8B
45.6
67.0
47.2
data4robotics
86.0M
Kinetics-700
0.5B
87.0
55.0
VC-1
86.0M
Ego4D+N
1.0B
50.0
86.4
ToBo (ours)
21.7M
Kinetics-400
0.2B
60.4
87.8
68.0
Uses udditiinal compression layers. Uses multi head attention pooling layers integrating spatial tokens.
Excludes language guidance from the vanilla recipe. IIclluees data for distillation models.
Qualitative comparison To illustrate the actual manipulation processes, present the robot trajec- tories from successful demonstrations for three real-world manipulation tasks in Fig. 4. Specifically, the initial states of the physical robot are depicted in the left scenes, while the right scenes show the final states of the demonstrations. The middle scenes illustrate the intermediate states of the demonstrations. Our model clearly succeeds in all the tasks. We also compared the trajectories with the baselines in the Appendix.
4.4 Video Label Propagation
We perform comparative analyses the video label propagation tasks. We consider the video object segmentation, video part segmentation, and pose tracking tasks from DAVIS [38], VIP [55], and JHMDB [23]. We follow the evaluation protocol in Jang et. al [22]. We present the quantitative evaluation Table 5. Our method demonstrates superior performance compared to all the baselines across the video label propagation tasks. We also provide qualitative results in Fig. 5, where method effectively traces visual appearances across various video label propagation tasks. These visualizations highlight that our method maintains robust object identity, part consistency, and pose continuity. The strong performance both quantitative and qualitative evaluations further demonstrate the effectiveness of our approach in capturing the temporal evolution of visual appearance across consecutive scenes.
Discussion
Comparison with robot representation learning models We further compare our method with recent robot representation learning (RRL) models, categorized by their supervision types: self- supervised learning [9, 33], supervision via foundation model outputs [42], and supervision with auxiliary language annotations [24, 25, 34, 40]. Table shows the reported performance of RRL models across several simulated robot manipulation benchmarks [15, 41, 53]. Here, our model is based on ViT-Small architecture trained on Kinetics-400 for 400 epochs. Notably, despite having the smallest number of parameters and the second smallest amount of training data, and using no annotation- supervision, our method achieves the highest score on MetaWorld. In particular, Theia trained by distilling knowledge from five large-scale foundation models (CLIP large [39], Depth Anything large [52], DINOv2 large [35], Segment Anything huge [29], and VIT huge [45]),

---

Table 7: Performance with vision-language models. We compare the performance of method with vision- langunla models on Franka Kitchen. Despite using a smaller model, significantly less pre-training data, and no auxiliary textual guidance from manually annotated data, ToBo consistently outperform the other models across all tasks.
Method
# #Param
Dataset
#Sen frames Knobl on
Light on
Sdoor open
Ldoor open
Mícro open
CLIP
149.3M
WebImage
12.8B
23.0
29.5
69.5
13.5
22.0
DINOv2
22.1M
LVD-142M
4.3B
25.5
38.0
82.0
15.5
20.0
SigLIP'
203M
WebLI
2.1B
17.5
38.5
75.0
8.5
16.5
SigI.IP2
375M
Wehl.I
408
11.0
23.5
58.5
11.0
18.0
ToBo (Ours)
21.7M
Kinetics-400
0.2B
57.0
82.0
95.0
51.0
55.0
Gain
+1.5
+ 43.5
+ 13.0
+:35.5
+ +3.0
The model is trained using textual guidance with manually annotated data.
which are collectively trained on 14.3 billion annotated samples. It also employs convolution-based compression layers during evaluation. Surpassing Theia under such an unbalanced training and evaluation setup is noteworthy. Furthermore, the performance gap between R3M with and without language guidance highlights the substantial benefit of auxiliary language supervision. Even with such unfairness in the training setup, our method outperforms R3M, MVP, Voltron, and MPI on Meta World. It also surpasses R3M on Franka Kitchen, despite significant differences in training data and model size. Compared to self- eef-suuer RRL models, our method outperforms all the models. It surpasses much larger models such as VC-1 and data4roboticc,, despite being trained on significantly smaller amount of data. Given the minimal number of parameters and training scale, these results demonstrates the effectiveness and efficiency of our proposed method for robot manipulation tasks.
Comparison with Vision-Language Models We compare our method with vision-language models widely used either backbones across various domains as vision towers in large language models. For fair evaluation, we follow the same evaluation protocol used in the main paper. We evaluate CLIP [39], DINOv2 [35], SigLIP [54]. and SigLIP2 [47] in the Franka Kitchen benchmark, as shown in Table 7. Despite having the smallest number of learnable parameters and being exposed to the smallest number of seen frames during pre-training, ToBo achieves consistently superior performance, outperforming the baselines by margins at least 13.0%p to the maximum 43.5%p. These performance gaps are notable given that all baselines except DINOv2 use language supervision from manually annotated data. These results demonstrate the effectiveness of ToBo in summarizing visual observations for sequential scene understanding tasks.
Ablation Study on Mask Ratio of Target Scenes To verify our claim that extremely scarce information from target scenes forces the decoder to rely highly on the stored visual seene information of the refer- ence scene, we conduct an ablation study varying the mask ratio of target scenes. We pre-train the models on the Kinetics 400 [26] dataset for 100 epochs and evaluate five tasks Franka Kitchen. As shown in Figure 6, the effectiveness of our proposed method in- creases the masking ratio of target scenes increases until 0.9, verifying our claim that scarce target scene information facilitates the exploitation of the com- pressed reference information. Besides, the models pre-trained with a masking ratio of 0.95 yield de- graded performance in some tasks, demonstrating that minimal clues are necessary for the prediction the missing information.
100
Knob1 on
Light
Success Rores (%6
Ldver open
Micro open
20
0.50
0.60
0.70
0.80
0.90
Nask Ratio Scene
Figure 6: Varying the masking ratio of tar- get scenes. We vary the masking ratio from 0.5 to 0.95 and pre-train ViT-S/16 models on the Kinetics-400 [26] dataset for 100 epochs.
Scalability We investigate the scalability of our ToBo beyond ViT-S/16 by pre- pre-training ViT- B/16 and ViT-L/16 on Kinetics-400 [26] for 100 epochs. We evaluate the pre-trained models on vision-based robot policy learning on Franka Kitchen [15] using three different seeds. We compare our method with MAE, SiamMAF, and RSP. Table 8 presents the mean and standard deviation across all seeds. We observe that models pre-trained with ToBo consistently achieving the best

---

Table 8: Scalability of our method. We report the performance of vision-based robot policy learning on Franka Kitchen [15], which are trained upon representations from the ViT-B/16 and ViT-L/16 model pre-trained on Kineties-400 [26] dataset for 100 epochs. The success rates (%) reported for all the tasks. We underline the second-bett performance. We report the gains of our method over the second-best baseline. We conduct evaluations using three different seeds.
Arch.
Method
Knobl on
Light
Sdoor open
Ldoor open
Micro open
MAE
18.7
21.3444
70.0
17.3+2.
15.3:22
SiamMAE
18.01220
34.0:20
80.7
18.71:12
19.3:6.1
ViT-B/16
RSP
24.7.13.11
$1.749.1
87.3
23.3:47.66
26.7,42.3
ToBo (ours)
46.7=66.
78.7
95.3+112
47.3+5.0
37.3=4.6
Gain
+.20
4/20.
4.8.0
+ +.20
+ 10.6
MAE
19.3+7.6
33.3+23
61.3#66
16.0+2.0
14.0=20
SiamMAE
20.7=33.1
34.0=4.0
76.0=2.0
12.7+64
22.0=0.0
ViT-I/16
RSP
26.7.12.3
48.0422
88.0
22.748..3
23.3:142
ToBo (ours)
54.7::5.0
75.3144
94.0
50.0
42.7
Gain
+28.0
27.3
6.0
+ 27.3
+ 19.4
performance across all five tasks, exhibiting significant improvements over the second-best results. These demonstrate the scalability of our method.
Comparison of training and inference flops We conducted FLOPs evaluation for both train- ing and inference to quantitatively compare the computational cost each model, as summarized in Table 9, During inference, all models use the same backbone architecture and input resolution without any input masking, resulting in identical inference FLOPs at the same model scale (e.g., 4.6 GFLOPs for ViT-Small During training. ToBo, MAE [18], and SiamMAE [16] show similar com-
Table 9: Comparison of training FLOPs and downstream performance in Franka Kitchen.
Training FLOPs (GFLOPs)
Franka Kitchen (%)
Method
MAE SiamMAE
13.0
26.1
13.1
30.4
RSP
32.5
43.4
ToBo
15.9
68.1
computation of 32.5 GFLOPs due to its complex decoding mechanisms. When considering computational costs with downstream perfor- mance (e.g., performance in Franka Kitchen [15]), these results further support the effectiveness of ToBo, which achieves a strong balance between efficiency and performance.
putational costs while RSP [22] requires substantially
6 Conclusion
We have introduced Token Bottleneck (ToBo), self-supervised visual representation learning method designed for sequential scene understanding. The backbones for sequential scene-based tasks should effectively preserve visual information from observations while facilitating the recognition of temporal progression across sequential scenes. While conventional self-supervised learning (SSI.) methods have proven promising impacts in visual representation learning, they primarily focus on understanding static images or entire videos, often lacking embeds for handling dynamic transitions in sequential tasks. Recent SSLs aim to address this by adapting correspondence learning in dynamic scenes. However, their patch-wise representations observations are often suboptima for subsequent policy networks, especially in tasks like robotic manipulation. To this end, ToBo introduces simple yet effective pipeline that facilitates conservative summarization of the observed scene into a bottleneck token while enable capturing of dynamic transitions through the bottleneck token. Through extensive experiments in various sequential understanding tasks including manipulation tasks and video label propagation tasks, we verified the superiority of ToBo over conventional SSL methods and previous dynamic scene SSL. methods. Furthermore, applying ToBo in real-world settings demonstrates its robustness and generalization capability.
Limitation Due to the resource constrains, we did not check the scalability of our method beyond huge scale and explore beyond the commonly used input resolution. Additionally, our study focused on simplest setting involving two dynamic scenes learn temporal dynamics.
10

---

References
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vee: general framework for self-supervised learninnn learning in speech, vision language. ICML. pages 1298-1312.
PMLR, 2022.
[2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2021. 2
[3]
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. Proceedings of NewrIPS. 2020.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, 2021. 2,
[5] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, 2020. 1,
[6] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, pages 15750-15755 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 04297. 2020.
[8] Xinlei Chen, Saining Xie, and Kaiming Hc. An empirical study training self-supervised vision transformers. arXiv preprint
2021. 1,
[9] Sudeep Dasari, Mohan Kumar Srirama, Unnat Jain, and Abhinav Gupta. An unbiased look datasets for visuo-motor pre-training. In Conference Robot Learning. PMLR, 2023, 8.8
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv: 1810.04805, 2018. 2
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. 15
[12] Alexandre Eymaël, Renaud Vandeghen, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck. Efficient image pre-training with siamese cropped masked autoencoders. In European Conference on Computer Vision, pages 166..66.666 Springer, 2025. 1. 2, 3. 6,
[13] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaimning He. A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings of the IEEE/CVP Conference on Computer Vision and Puttern Recognition, pages 3299-3309, 2021.
[14] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray Kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap Your Own Latent A New Approach to Self- Self-Supervised Learning. In NewrIPS, pages 21271-2128. Curran Associates. Inc., 2020.
[15] Abhishck Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long- horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910. 19966, 2019. 5, 8, 9, 10, 14, 15, 17
[16] Agrim Gupta. Jiajun Wu, Jia Deng, and Li Fei-Fei. Siamese masked autoencoders. In Advances in Neural Information Processing Systems, 2023. 2, 3, 4, 5, 10, 14. 15
[17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05777 2019. 2
[18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022. 1, 6, 10, 14,

---

[19] Elad Hoffer, Tal Ben-Nun, hay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference on Computer Vixion and Pattern Recognition, 2020.
[20] Allan Jabri, Andrew Owens, and Alexei A. Efros. Space-time correspondence contrastive random walk. In Advances in Neural Information Processing Systems, 2020.
[21] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark& & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-300 2020. 5, 6,
14, 16, 17
[22] Huiwon Jang, Dongyoung Kim, Junsu Kim, Jinwoo Shin, Picter Ahbeel, and Younggyo Seo. Visual repre- sentation learning with stochastic frame prediction. In Proceedings of the 41st International Conference Machine Learning, pages 21289-21305. PMLR, 2024. 1, 2, 3, 4, 7, 8, 10, 14, 15, 16
[23] Hucihan Jhuang, Juergen Gall, Silvia Zufti, Cordelia Schmid, and Michael Black. Towards understanding action recognition. Proceedings of the IEEE international conference on computer vision, 2013. 5, 7,7,5 18
[24] Zeng Jia, Bu Qingwen, Wang Bangjun, Xia Wenke, Chen Li, Dong Hao, Song Haoming, Wang Dong, Hu Di, Luo Ping, Cui Heming, Zhao Bin, Li Xuelong, Qiao Yu, and Li Hongyang. Learning manipulation by predicting interaction. In Proceedings of Robotics: Science and Systems (RSS), 2024. 3, 8
[25] Siddharth Karamcheti, Suraj Nair, Annie S. Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. In Robotics: Science and Systems (RSS), 2023. 3,8 8
[26] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaynnarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kineties human action vidco dataset. arXiv preprint or arXiv:1005.06955 2017. 5, 6, 7, 9, 10, 14, 15
[27] Tackyung Kim, Sanghyuk Chun, Byeongho Heo, and Dongyoon Han, Learning with unmasked tokens drives stronger vision learners. European Conference on Computer Vision (ECCV), 2024. 2, 4
[28] Taekyung Kim, Bycongho Heo, and Dongyoon Han. Morphing tokens draw strong masked image models. In The Thirteenth International Conference on Learning Representations, 2025. 2
[29] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tcte Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015 2023. 8
[30] Z. Lai and W. Xie. Self-supervised learning for video correspondence flow. In BMVC, 2019. 18
[31] Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang. Jan Kautz, and Ming-Hsuan Hsuan Yang. Joint-task self-supervised learning for temporal correspondence. In NeurlPS, 2019. 18
[32] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. 15
[33] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma. Claire Chen, Sneha Silwal, Aryan Jain, Fincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? In Advances in neural information processing systems, 2023. 3, 4,
5, 8, 14, 16, 17
[34] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for rohot manipulation. In 6th Annual Conference on Robot Learning, 2022. 3, 8, 15
[35] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint wxXiv:2304.07992 2023. 8,
[36] Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, and Wei Liu. Vidcomoco: Contrastive video representation learning with temporally adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 1 11005-12144 2021.
[37] Simone Parisi, Aravind Rajeswaran. Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. In Proceedings of the 39th International Conference Machine Learning, pages 17359 17371. PMLR, 2022. 15
12

---

[38] Jordi Pont-Tuset, Federico Perazzi. Sergi Caelles, Pablo Arbeláez, Alex Morkine-Hormunn and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint 17Xv:1100.00075. 2017. 5. 17
[39] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PmLR, 2021. 9
[40] Iliaa Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Proceedings of The 6th Conference on Robot Leurning. pages 416-426. PMLR, 2023. 3, 4,
[41] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani. John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018. 6, 8
[42] Jinghuan Shang. Karl Schmeckpeper, Brandon B. May, Maria Vittoria Minniti, Tarik Kelestemur, David Watkins, and Laura Herlant. Theia: Distilling diverse vision foundation models for robot learning. 2024. 3.
[43] Yuval Tassa. Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez. Timothy Lillicrap. and Nicolas Heess. _onttol:::: Software and tasks for continuous control. arXiv preprint arXiv:2006. /2983, 2020. 6, 16
[44] Zhan Tong. Yibing Song. Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self- self-supervised video pre-training. In Advances in neural information processing systems. 2022. 1
[45] Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. Computer Vision-ECC ECcn 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV. pages 516-533. Springer, 2022. 8
[46] Frederik Träuble, Andrea Dittadi, Manuel Wuthrich, Felix Widmaier, Peter Vincent Gehler, Ole Winther, Francesco Locatello, Olivier Bachem, Bernhard Schölkopf. and Stefan Bauer. The role of pretrained representationns for the OOD generalization of RL agents. In International Conference on Learning Representations, 2022. 6
[47] Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Naeem, Ibrahim Alabdulmohsia. Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision- lsiounlanguage encoders with improved semantic undorstanding, localization, and dense features. arXiv preprint 25022 14786, 2025. 9
[48] Xiaolong Wang, Allan Jabri, and Alexci A. Efros. Learning correspondence from the eycle-consistency of time. In CVPR, 2019. 18
[49] Philippe Weinzaepfel. Vincent Leroy, Thomas Lucas, Romain Brégier. Yohann Cabon, Vaibhav Arora, Lconid Antsfeld, Boris Chidlovskii, Gabricla Csurka, and Revaud Jérôme. Croco: Sclf-supervised pre-training for 3d vision tasks by cross-view completion. In NeurlPS, 2022. 2
[50] Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In ICCV, 2022. 2,3,4
[51] Jiange Yang. Bei Liu, Jianlong Fu, Bocheng Pan, Gangshan Wu, and Limin Wang. Spatiotemporal predictive pre- pre-training for robotic motor control, 2024. 3
[52] Lihe Yang. Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10371-10381, 2024. 8
[53] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-tash and meta reinforcement learning. In Conference on Robot Learning, 2020. 6. 8, 16
[54] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 11975-11986, 2023.
[55] Qixian Zhou, Xiaodan Liang, Ke Gong, and Liang Lin. Adaptive temporal encoding network for video instance- level human parsing. In Proceedings of the 26th ACM international conference on Multimedia, 2018. 5, 7, 8, 17
13

---

Appendix
AA Further analysis on the hyperparameter choices in the Token Bottleneck pre-training pipeline
B Manipulation trajectory visualization of real-world demonstrations
SC: Implementation details for pre-training and evaluation
A Further Analysis
In this section, examine how hyperparameter choices in the Token Bottleneck (ToBo) pre-training pipeline affect performance, regarding the number of bottleneck tokens, impact of temporal difference, We pre-train ViT-B/16 for 100 epochs Kinetics-400 [26] throughout the ablation studies. The comparisons are done on five Franka Kitchen imitation-learning tasks [15].
Ablation study on the number of bottleneck tokens in {1, 2, 4, 8). As shown Table Aa, using single token consistently yields the best performance across tasks. This demonstrates that conservative summarization without auxiliary storage better captures the current observation and thus improves action prediction roboties.
We vary the number of bottleneck tokens
Ablation study on temporal difference We further vary the maximum temporal gap between frames among (48, 96, 144). As shown in Table Ab, moderate temporal differences encourage the model to learn dynamic scene evolution, since shorter gaps lack meaningful change whereas overly longer gaps disrupt temporal coherence
Ablation study with no temporal difference We apply our method using the same frame for both source and target scenes. As shown in Table Ac, our method still works even without temporal difference, surpassing other baselines (e.g., MAE [18], SiamMAE [16], and RSP [22]) with significant margin across tasks. However, its overall performance degrades compared to original ToBo, reflecting the loss of supervision from temporal change. This highlights the importance of temporal contrast for effective pre-training of ToBo.
Ablation study multiple source frames We compare ToBo variant pre-trained with multiple source frames. Specifically, we randomly sample four source frames and pre-train for 100 epochs under the same recipe ToBo. As shown in Table Ad, this multi-frame frramf variant surpasses prior baselines (e.g., MAE [18], SiamMAE [16], and RSP [22]) in most of the tasks. However, despite requiring higher pre-training cost, it underperforms compared to ToBo across all robotics tasks. These results suggest that while is possible to extend ToBo to multi-frame settings, such naive extension may encounter potential new challenges, leading suboptimal performance.
B Manipulation trajectory visualization of Real-world Demonstrations
We showcase the robot manipulation trajectories for the SiamMAE [16], RSP [22], and our model as robot backbones in the same episode on the real-world environment for each task. In Figure A, Specifically, the leftmost scenes depicts the initial states of the physical robot, while the rightmost scenes show the final states of the demonstrations. As shown in Fig. A. while SiamMAE and RSP fail execute the manipulation tasks, method successfully completes them within the episode. We also provide videos these demonstrations in the supplementary material.
C Implementation Details
provide implementation details for pre-training and evaluation. Specifically, we present evaluation protocols for vision-based robot policy learning each simulated environment (i.e., Franka Kitchen [15], CortexBench [33], RLBeecch [21]) and real-world environment. Then, we explain experimental setups for video label propagation tasks.
14

---

Table A: Ablation studies Token Bottleneck pre-training. We vary hyperparameters of the Token Bottleneck (ToBo) pre-training pipeline. We report the success rates (%) on five imitation learning tasks from the Franka Kitchen benchmark [15]. All models are ViT-B/16 and pre-trained for 100 epochs on Kinetics-400 [26]. We mark our default settings in gray
(a) Number of bottleneck tokens
# bottleneck tokens
Knobl
Light on
Sdoor open
Ldoor open
Micro open
Mean
46.7
78.7
95.3
47.3
37.3
61.1
2
31.0
54.0
74.0
26.0
24.0
41.8
28.0
24.3
78.0
28.0
22.0
36.1
8
10.0
20.0
56.0
26.0
9.3
24.3
(b) Temporal difference
Maximum temporal difference
Knobl
Light on
Sdoor open
Ldoor open
Micro open
Mean
40.7
78.7
96.0
44.0
35.3
46.7
78.7
95.3
47.3
37.3
61.1
144
36.0
97.3
46.7
39.3
57.7
(c) Ablation with temporal difference
Method
Knobl
Light on
Sdoor open
Ldoor open
Micro open
Mean
MAF [18]
18.7
21.3
70.0
17.3
15.3
28.5
SiamMAE [16]
18.0
34.0
80.7
18.7
19.3
34.1
RSP [22)
24.7
51.7
87.3
23.3
26.7
42.7
ToBo (no temporal difference)
41.0
72.0
32.7
32.0
ToBo
78.7
95.3
47.3
37.3
61.1
(d) Ablation multiple source frames
Method
Knobl
Light
Sdoor open
Ldoor open
Micro open
Mean
MAR [18]
18.7
21.3
70.0
17.3
15.3
28.5
SiamMAE [16]
18.0
34.0
80.7
18.7
19.3
34.1
RSP [22]
24.7
51.7
87.3
23.3
26.7
42.7
ToBo multi-frame)
60.7
92.7
20.7
32.0
ToBo
46.7
78.7
95.3
47.3
37.3
61.1
C.1 Pre-training
We pre-train ViT-S/16 [11] on Kinetics-400 [26] for 400 epochs for the main comparison, while we pre-train ViT-S/16, ViT-B/16, and ViT-L/16 for 100 epochs for analyses. We employ repeated sampling [19, 13] with a factor
so that the models are indeed pre-trained for 200 epochs. We use AdamW optimizer [32] with batch size of 1536, comprising dynamic scenes with resolution of 224x These scenes are randomly sampled from videos at a rate of FPS, with a temporal index gap ranging from 4 to 96. We simply apply random resized crop and horizontal flip to the scenes, aligning the cropping region across the reference and target scenes. To drive the learning mechanism of our proposed method, masking ratio 0.9. Our decoder is composed of eight vision transformer blocks, i.c., each block contains self-attention layers and multi-layer perceptrons. We follow the default hyperparameters: of the baselines for their pre-training on Kinetics-400 [26]. We adopt a siamese masked autoencoding loss [16] auxiliary objective to enhance learn patch-level correspondence learning.
randomly mask the target scenes with extremely high
C.2 Vision-based Robot Policy Learning
Franka Kitchen. We validate models pre-trained by our method and other baselines in five imitation learning tasks from the Franka Kitchen benchmark [15]. Our experiments mainly follow the imitation learning evaluation setup in Jang et. al. [22], which builds upon [34, 37]. Specifically, we employ an agent comprising a frozen backbone initialized with pre-trained models and a policy network consisting of two-layer MLP, with batch normalization layer applied at the input stage. We define the state representation for the policy network as the combination of the visual representation and the robot's proprioception. For the perception, we employ either left or right camera with 224x224 resolution while omitting depth. The policy network is trained with standard behavior cloning loss. Training for each demonstration task progresses for 20,000 steps, with periodic online
15

---

SiamMAE
RSP
Oura
(a) Cabinet Opening
SimMAE
RSP
Ours
(b) Drawer Closing
RSP
Ours
(c) Cup Stacking
Figure A: Sampled Trajectories from Real World Experiment. We visualize the manipulation trajectories of ToBo, RSP, and SiamMAE physical robot manipulation tasks in real-world vironments (i.e., Cabinet Opening, Drawer Closing, and Cup Stacking). Our ToBo successfully demonstrates all tasks, which aligns with the quantitative performance comparisons results.
evaluation in the simulated environment every 1,000 steps. We evaluate the highest success rates of each demonstration across four different seeds and report its average with 95% confidence interval.
RLBench. We consider five manipulation tasks from RLBench [21]. Follow the evaluation setup in Jang et. al. [22], we generate 100 demonstrations and utilize them for training the agent. We employ a front camera with a 224x224 resolution. Point cloud information excluded throughout all experiments. We employ the end-effector controller with path planning. We evaluate the highest success rates of each demonstration across four different seeds.
CortexBench. We evaluate the models on four simulated environments from CortexBench [33]. We consider two, five, five, and two demonstrations from Adroit, DeepMind Control (DMC) [43], Meta- World [53], and Trifinger, respectively. Proprioceptive data is utilized except the DMC benchmark. We mainly follow the experimental setups in Jang et. al. [22], which builds upon [33]. For each task, train the agent for 100 epochs, with periodic online evaluation in the simulated environment
16

---

Simulated environments
(b) Real-world
Figure B: Visualization of environments for robot policy learning evaluation. We validate the ef- fectiveness of our method on (a) simulated environments (e.g., Franka Kitchen [15], CortexBench [33]. RLBench [21] and (b) real-world environments. We design real-world environments with physical robots to evaluate how the algorithm handles given tasks.
(a) Cabinel Opening
(b) Drawer Closing
(c) Cup Stacking
Figure C: Task Description for Real-world Environments. We illustrates the objectives of physical robot manipulation tasks in the real-world Yellow arrows indicate the target actions for each task.
every 5 epochs. We report the normalized score for DMC and the highest success rates for other tasks. We conduct demonstration tasks for five different seeds and report average with 95% confidence interval.
Real-world Environments. We evaluate our proposed method in real-world robotic imitation learning tasks using UR5e manipulator equipped with parallel gripper. The policy operates at control frequency of 5 Hz, executing actions defined as delta end-effector poses and gripper's state, with specific parameterizations for each task: (dx, dy) for drawer closing, (dx, dy, gripper open/close) for cabinet opening, and (dx, dy, dz, gripper) for cup stacking. The system employs joint position control at 50 Hz, with numerical inverse kinematics (IK) solver running in the background calculate the end-effector's pose to the joint position. Our training dataset consists of 50 demonstrations for cabinet opening and cup stacking and 30 demonstrations for drawer closing. train the two-layer MLP policy for 100 epochs without incorporating proprioceptive states, using top-front camera view with resolution of 224x224. The final performance is evaluated based on the reported average success rate across tasks. Figure C provides visual examples of the three tasks under consideration.
Video label propagation. conduct comparative analyses for video label propagation on video object segmentation DAVIS [38], video part segmentation VIP [55], and pose tracking
17

---

JHMDB [23]. Following the evaluation protocols in the previous studies [48, 31, 30, 20], we employ k-nearest neighbor inference, maintain queue of length m provide temporal context, and restrict the set of source nodes within spatial radius r. Additionally, we perform grid search to optimize evaluation hyperparameters for each method and report the best results.
18

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state our claims in the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations at the end of the paper
19

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We explain the implementation details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide link our source code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/ Suuides/CodeSuumiee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: CCte///ooeummiiic Poooeeubmisssioooooi.
/ //nips... /tippscc/pppii wttpc//cccpppcccccccceo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
21

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We explain them in the implementation details in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the experimental results with error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We explain them in the implementation details in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
22

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://///euus...eeeee wwtrriisscccpppeeeo hewpoo/ii/tthiisuiieeeeeee
Answer: [Yes]
Justification: Our research conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
23

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cited all the papers from which we used code, data, and models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https : //tperrpss...e/eee wttpiipps..../ooooooee cc/Conferences:2222 /LLM) for what should or should be described.
25


Reviews:
[{"review_id": "pHWi78XeFI", "reviewer": "Reviewer_W34j", "summary": "The paper proposes a method for learning visual representations. The method is based on MAE and called Token Bottleneck. The basic idea is to learn representations from a pair of images of the same scene (two different views / video frames). The method trains a model to compress the patches of the first image into a single bottleneck token and, from that token plus visible patches from the second image, predict the masked patches of the second image. The results are evaluated on several simulated robotic tasks and davis label propagation.", "questions": "Please see the weaknesses above.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- I reviewed an earlier version of this paper for ICCV 2025. Compared to that version, the present submission is considerably improved. The writing, method name, and description are much clearer. Some of the missing baselines are included too. I appreciate the updates and thank the authors for incorporating the feedback from the previous review.\n- The method is simple, easy to understand, and shows promising signal in tested settings.\n\nWeaknesses:\n- 1) Key design choices are not ablated which makes it difficult to understand different aspects of the proposed method. Some examples:\n    - a) How many patches to squeeze the information into? Single token vs multiple? How does the tradeoff look like?\n    - b) How does the temporal difference impact the performance?\n    - c) What would happen if there was no temporal difference between the frames? Would the task be trivial or still useful?\n    - d) ...\n- 2) It would be good to include comparisons between different methods (e.g., vanilla MAE, SimSiam, etc.) in terms of flops. Both training time flops and inference flops.\n- 3) Comparisons on standard benchmarks like ImageNet and Kinetics missing. I understand that the main focus is on simulated robotic evaluation and tracking but it would still be helpful to see how the learnt representations compare to standard approaches like MAE on standard benchmarks.\n- 4) It would be good to see straightforward baselines like running vanilla MAE across both views/images.", "was_revised": true, "final_justification": "The author rebuttal addressed by questions and updated the score accordingly.", "rebuttal": "We sincerely thank Reviewer W34j for the thoughtful comments and constructive suggestions. Below, we provide detailed responses to each point.\n\n> ### **W1. Ablation studies on key design choices**\n\n\nWe agree that the commented factors are important for understanding the behaviors of ToBo. Thus, we conduct ablation studies on the suggested aspects. All the ablation studies are conducted using models pre-trained for 100 epochs.\n\n* **Ablation study on the number of bottleneck tokens**\n    * We conducted an ablation study on the number of bottleneck tokens, varying it among {1, 2, 4, 8}. As shown in **Table R1**, we observe that using a single token generally yields the best performance across tasks. These results demonstrate that conservative summarization without separate storage is beneficial for understanding the current observation, thereby improving action prediction in robotics\n\n\n* **Ablation study on the impact of temporal difference**\n    * We conducted additional experiments where the maximum frame interval is varied among (48, 96, 144). As shown in **Table R2**, the models are encouraged better to learn to capture dynamic scene evolution when trained with moderate temporal differences, not too short to include meaningful changes and not too long to break temporal coherence.\n\n* **What if there is no temporal difference between the frames?**\n    * Following the comment, we applied our method using the same frame for both the source and target scenes. As shown in **Table R3**, we found that our method still works even without temporal difference, surpassing other baselines (e.g., MAE [1], SiamMAE [2], and RSP [3]) with significant gaps across the tasks. However, its overall performance degrades compared to original ToBo since it loses the opportunity to learn how to capture dynamic evolutions from consecutive scenes. This highlights the importance of temporal contrast for effective pre-training of ToBo.\n    \n* **What if multiple source frames are given?**\n    * We further compare ToBo with the model pre-trained using multiple source frames. Specifically, we randomly sample 4 frames as the source frames and pre-train the model for 100 epochs  under the same training recipe to ToBo. As shown in **Table R4**, the multi-frame based model surpasses other baselines (e.g., MAE [1], SiamMAE [2], and RSP [3]) in most of the tasks. However, despite requiring higher pre-training cost, it underperforms compared to ToBo across all robotics tasks. These results suggest that while it is possible to extend ToBo to multi-frame settings, such naive extension may encounter potential new challenges, leading to suboptimal performance.\n\n\n\n[**Table R1.** Ablation study on the number of bottleneck tokens]\n\n|  Number of bottleneck tokens |    Knob1 on   |   Light on    |   Sdoor open    |   Ldoor open    |  Micro open  |\n|----------------------------------------|-------------------|----------------|----------------------|---------------------|------------------|\n| 1                                             |         **46.7**      |        **78.7**       |       **95.3**          |        **47.3**          |        **37.3**       |  \n| 2     |         31.0      |        54.0       |        74.0          |        26.0          |        24.0       |  \n| 4       |         28.0      |        24.3       |        78.0          |        28.0          |        22.0       |  \n| 8     |         10.0     |        20.0       |        56.0          |        26.0          |         9.3       |  \n\n\n[**Table R2.** Ablation study on the impact of temporal difference]\n\n| Maximum temporal difference |    Knob1 on   |   Light on    |   Sdoor open    |   Ldoor open    |  Micro open    |\n|--------------------------|-------------------|-----------------|---------------------|---------------------|---------------------|\n| 48                          |         40.7      |        78.7       |        96.0         |        44.0          |        35.3          |  \n| 96                          |         **46.7**      |        **78.7**       |        95.3         |        **47.3**          |        37.3          |  \n| 144                        |         36.0      |        69.3       |        **97.3**         |        46.7          |        **39.3**          |  \n\n                                \n[**Table R3.** Analysis on the importance of temporal difference]\n\n|                                         |    Knob1 on   |   Light on    |   Sdoor open    |   Ldoor open    |  Micro open    |\n|----------------------------------|------------------|-----------------|---------------------|---------------------|---------------------|\n| MAE                |       18.7       |       21.3       |       70.0       |       17.3       |       15.3       |\n| SiamMAE        |       18.0       |       34.0       |       80.7       |       18.7       |       19.3       |\n| RSP                 |       24.7      |       51.7       |       87.3       |       23.3       |       26.7       |\n| |\n| ToBo (no temporal difference)    |         41.0     |        72.0       |        89.3          |        32.7          |        32.0          |  \n| ToBo (with temporal difference)  |         **46.7**      |        **78.7**       |        **95.3**         |        **47.3**          |        **37.3**          |  \n\n\n[**Table R4.** Analysis on the multiple source frame]\n\n|                             |    Knob1 on   |   Light on    |   Sdoor open    |   Ldoor open    |   Micro open    |\n|------------------------|------------------|-----------------|---------------------|---------------------|---------------------|\n| MAE                |       18.7       |       21.3       |       70.0       |       17.3       |       15.3       |\n| SiamMAE        |       18.0       |       34.0       |       80.7       |       18.7       |       19.3       |\n| RSP                 |       24.7      |       51.7       |       87.3       |       23.3       |       26.7       |\n| |\n| ToBo (multi-frame) |       28.7     |         60.7    |          92.7        |           20.7       |             32.0     | \n| ToBo                      |        **46.7**      |        **78.7**       |        **95.3**          |        **47.3**          |        **37.3**       | \n\n\n> ### **W2. Comparison of training and inference flops**\n\nWe conducted FLOPs evaluation for both training and inference to quantitatively compare the computational cost of each model, as summarized in **Table R5**. \n* During **inference**, all models use the same backbone architecture and input resolution without any input masking, resulting in identical inference FLOPs at the same model scale (e.g., 4.6 GFLOPs for ViT-Small). \n* During **training**, ToBo, MAE [1], and SiamMAE [2] show similar computational costs while RSP [3] requires substantially more computation of 32.5 GFLOPs due to its complex decoding mechanisms. When considering computational costs with downstream performance (e.g., performance in Franka Kitchen [4]), these results further support the effectiveness of ToBo, which achieves a strong balance between efficiency and performance.\n\n[**Table R5.** Comparison of training and inference FLOPs and downstream performance in Franka Kitchen ]\n\n| Method    | Training FLOPs (GFLOPs) | Inference FLOPs (GFLOPs) | Franka Kitchen (%) |\n|-----------|-------------------------|---------------------------|---------------------|\n| MAE      | 13.0                    | 4.6                       | 26.1                |\n| SiamMAE  | 13.1                    | 4.6                       | 30.4                |\n| RSP     | 32.5                    | 4.6                       | 43.4                |\n| |\n| ToBo      | 15.9                    | 4.6                       | 68.1                |\n\n\n> ### **W3. Comparisons of ToBo and MAE on Kinetics-400**\n\nFollowing the comment, we compare the action classification accuracy of ToBo and the baselines (i.e., MAE [1], SiamMAE [2], and RSP [3]) on the Kinetics-400 dataset [6]. To this end, we fine-tune all models under a reduced setting of 40 epochs, due to the limited rebuttal period. As shown in **Table R6**, while the performance gaps are relatively smaller than those appears in our main downstream tasks in the main paper, ToBo achieves the highest accuracy among all models, demonstrating its effectiveness on this standard video classification benchmark.\n\n[**Table R6.** Comparison of ToBo and other baselines in action recognition in Kinetics-400]\n\n|                  |   Kinetics-400 (%)   |   \n|---------------|-----------------|\n| MAE     |  39.9          |\n| SiamMAE  |  38.1          |\n| RSP           | 38.2           |\n| |\n| ToBo         |    **40.8**         |\n\n\n \n> ### **W4. Comparison with straightforward baselines**\n\nWe apologize for the confusion. We would like to clarify that MAE [1] in the tables of the main paper is indeed a straightforward baseline that performs masked image modeling independently on both input views. We will improve the description for the baselines.\n\n\n### **References**\n\n[1] He et. al., \"Masked Autoencoders are Scalable Vision Learners,\" CVPR 2022.\n\n[2] Gupta et. al., \"Siamese masked autoencoders,\" NeurIPS 2023.\n\n[3] Jang et. al., \"Visual representation learning with stochastic frame prediction\", ICML 2024.\n\n[4] Gupta et. al., \"Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. CoRL 2019.\n\n[5] Russakovsky et. al., \"Imagenet large scale visual recognition challenge,\" IJCV 2015.\n\n[6] Kay et. al., \"The kinetics human action video dataset,\" arXiv:1705.06950, 2017."}, {"review_id": "EJacyV9QOM", "reviewer": "Reviewer_GhiU", "summary": "This paper investigates how to use the dynamics information in video to pretrain an image encoder, extending MAE. The proposed ToBo(Token Bottleneck) has a unique pretext task of compressing an image into a single token, then decoding a later frame based on the bottleneck token and sparse target frame patches. The method is mainly evaluated on video label propagation and robotics tasks, yielding especially large improvements in robotics manipulation.", "questions": "Please see the weakness section. To summarize, I have questions regarding the variance of performances, more careful discussion of limitations, and direct video pretraining. I am open to increasing the score if these items are clarified.", "limitations": "The limitation discussion is too short, probably due to paper length limitation. But it should be revisited more carefully.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The proposed approach is simple but effective. The pretext task intuitively makes sense and is grounded in previous literature.\n2. The improvements on especially robotics tasks is impressive.\n3. Robotics experiments include error bars, which is informative.\n\nWeaknesses:\n1. The variance on the proposed approach seems especially large, especially in Table 3, compared to the baselines. Could the author provide some insight into why that is the case?\n2. More careful discussion of the limitation should be included in the final paper.\n3. Since we are learning representation from video, is it possible to take the VideoMAE approach to pretrain a video model, then somehow adapt it to the robotics use case?", "was_revised": true, "final_justification": "I believe this paper makes good progress on representation learning and it is meaningful to the community. My final rating is borderline accept.", "rebuttal": "We sincerely thank Reviewer GhiU for the thoughtful comments and constructive suggestions. Below, we provide detailed responses to each point.\n\n> ### **W1. Clarification on concerns regarding variance**\n\n* While addressing your comment, we found that the variance of ToBo in the DMC benchmark [7] was incorrectly reported as 8.6, when it should have been 0.86. We will correct this error (**8.6 --> 0.9**) in Table 2 of the main paper, as reflected in **Table R7**.\n* ToBo may tend to exhibit relatively high variance in RLBench [10]. However, we believe this reflects a broader trend where model variances vary significantly depending on the evaluation benchmark. In fact, on other benchmarks such as CortexBench [8] (**Table R7**) and Franka Kitchen [4] (**Table R8**), ToBo shows lower or comparable variance than other methods except “Sdoor Open”. Moreover, even in the \"Sdoor Open\" task, ToBo records a variance of 7.1, which is lower than that of SiamMAE (7.9) and CropMAE (8.1). These results suggest that variance is task-dependent, and ToBo maintains stable performance in several benchmarks despite relatively higher sensitivity in certain RLBench scenarios\n\n\n[**Table R7.** Experimental results on vision-based robot policy learning on CortexBench.]\n\n|Task|SimCLR|MoCo v3|DINO|MAE|SiamMAE|RSP|CropMAE|ToBo     |\n|----|-------|--------|-----|-----|--------|-----|--------|--------|\n|Adroit|40.4±3.3|39.6±4.3|45.6±6.2|39.6±4.3|44.0±6.6|45.6±4.6|50.0±5.1| 60.4±2.2 |\n|MetaWorld|78.4±5.2|65.4±8.0|82.4±5.8|65.4±8.0|81.1±6.3|84.5±6.6|82.4±5.8| 87.8±4.6|\n|DMC|39.7±2.9|43.7±3.2|50.9±1.5|43.7±3.2|56.0±2.9|61.6±3.4|46.4±1.1| 73.5±**0.9** |\n|TriFinger|63.3±3.3|53.3±1.6|64.2±3.5|53.3±1.6|52.1±7.6|66.2±0.8|46.3±1.7|66.5±1.0|\n\n[**Table R8.** Experimental results on vision-based robot policy learning on Franka Kitchen.]\n\n| Tasks | SimCLR | MoCo v3     | DINO    | MAE     | SiamMAE | RSP     | CropMAE   | ToBo     |\n|-------|---------|----------|----------|----------|----------|----------|----------|-----------|\n| Knob1 on      | 25.3±2.1 | 11.5±3.9 | 27.0±3.2 | 12.0±3.3 | 16.8±4.4 | 31.0±2.4 | 31.5±5.3  | 57.3±2.3 |\n| Light on      | 55.8±6.4 | 24.3±5.0 | 44.3±6.5 | 24.3±4.2 | 36.5±7.0 | 44.5±5.6 | 54.0±11.2 | 82.0±1.6 |\n| Sdoor open    | 72.3±2.8 | 66.5±3.2 | 77.0±5.0 | 71.5±4.3 | 68.0±7.9 | 82.5±2.7 | 77.0±8.1  | 95.0±7.1 |\n| Ldoor open    | 17.0±2.9 | 10.3±2.1 | 16.5±2.5 | 12.8±3.9 | 17.3±3.7 | 28.8±4.8 | 25.5±5.7  | 51.0±1.4 |\n| Micro open    | 23.3±2.8 | 14.3±2.5 | 28.5±4.8 | 10.0±2.8 | 13.5±4.8 | 30.3±5.6 | 32.5±4.1  | 55.0±1.4 |\n\n\n\n> ### **W2. Supplement to the limitation part**\n\nThank you for the advice. We improved the discussion in the limitation part: Due to the resource constraints, we both did not check the scalability of our method beyond huge scale and did not explore beyond the commonly used input resolution of 224x224. Additionally, our study focused on a simplest setting involving two dynamic scenes to learn temporal dynamics. Extending our pipeline to multiple frames setting would be an interesting direction for future work, solving new potential challenges emerging from multi-frame setting.\n\n\n> ### **W3. Extension of ToBo to multi-frame-based approach**\n\nFollowing the comment, we compare ToBo with the model pre-trained using multiple source frames. Specifically, we randomly sample 4 frames as the source frames and pre-train the model for 100 epochs under the same training recipe to ToBo. As shown in **Table R9**, the multi-frame based model surpasses other baselines (e.g., MAE [1], SiamMAE [2], and RSP [3]) in most of the tasks. However, despite requiring higher pre-training cost, it underperforms compared to ToBo across all robotics tasks. These results suggest that while it is possible to extend ToBo to multi-frame settings, such naive extension may encounter potential new challenges, leading to suboptimal performance.\n\n[**Table R9.** Comparison with the naive extension of ToBo to multi-frame-based setting]\n\n|                             |    Knob1 on   |   Light on    |   Sdoor open    |   Ldoor open    |   Micro open    |\n|------------------------|------------------|-----------------|---------------------|---------------------|---------------------|\n| MAE                |       18.7       |       21.3       |       70.0       |       17.3       |       15.3       |\n| SiamMAE        |       18.0       |       34.0       |       80.7       |       18.7       |       19.3       |\n| RSP                 |       24.7      |       51.7       |       87.3       |       23.3       |       26.7       |\n| |\n| ToBo (multi-frame) |       28.7     |         60.7    |          92.7        |           20.7       |             32.0     | \n| ToBo                      |        **46.7**      |        **78.7**       |        **95.3**          |        **47.3**          |        **37.3**       | \n\n\n\n### **References**\n\n[1] He et. al., \"Masked Autoencoders are Scalable Vision Learners,\" CVPR 2022.\n\n[2] Gupta et. al., \"Siamese masked autoencoders,\" NeurIPS 2023.\n\n[3] Jang et. al., \"Visual representation learning with stochastic frame prediction\", ICML 2024.\n\n[4] Gupta et. al., \"Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. CoRL 2019.\n\n[7] Tassa et. al., \"dm_control: Software and tasks for continuous control,\" arXiv:2006.12983, 2020.\n\n[8] Majumdar et. al., \"Where are we in the search for an artificial visual cortex for embodied intelligence?,\" NeurIPS 2023.\n\n[9] Eymaël et. al., \"Efficient image pre-training with siamese cropped masked autoencoders,\" ECCV 2025.\n\n[10] James et. al., \"Rlbench: The robot learning benchmark & learning environment,\" IEEE Robotics and Automation Letters, 2020."}, {"review_id": "LUQ2gNy5Fx", "reviewer": "Reviewer_dztw", "summary": "The authors propose a self-supervised vision pretraining method for sequential scene based tasks by bottlenecking the representation using a single token. The authors demonstrate the effectiveness of this approach in simulation and real world robotics tasks, with significant improvements over other methods like SimCLR and MAE(and its variants). The authors also demonstrate superior performance of their proposed method in label(mask,pose) propagation tasks over these baselines.", "questions": "see weaknesses", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The authors propose a novel self-supervised pretraining method to capture both temporal dynamics and summarize scene information in the same embedding.\n2.  The authors demonstrate the superior performance of their proposed method in robotics tasks and label propagation when compared to other SSL methods like MAE variants and SimCLR.\n\n\n\nWeaknesses\n1. The authors do not compare with newer self supervised vision pretraining methods like DINO v2 (except franka-kitchen) and I/V-JEPA on all the tasks. This is required to contextualise the  performance of the proposed method in the current SSL landscape. \n2. The authors need to demonstrate the scaling property of the method, when trained on more data and not just a larger model.\n3. The improvements in label propagation are much smaller than the improvements in robotics policy learning tasks. This needs to be explained/investigated. Is this an artefact of the agent's policy architecture being more suited to the proposed SSL method as opposed to the other SSL methods? Is the agent policy architecture the same for all the robotics tasks ? \n4. While the authors focus on bottlenecking the representation using a single embedding, it would be interesting to see the effect of increasing the number of embeddings in the bottleneck effectively relaxing the constraint gradually.", "was_revised": true, "final_justification": "The proposed SSL method beats SoTA methods like I/V-JEPA, DinoV2 on robotics tasks and can scale with increase in training data.", "rebuttal": "We sincerely thank Reviewer dztw for the thoughtful comments and constructive suggestions. Below, we provide detailed responses to each point.\n\n> ### **W1. Comparison with DINO v2 (beyond franka-kitchen), I-JEPA, and V-JEPA**\n\nFollowing the comments, we provide the comparison results of DINOv2 [11], I-JEPA [12], V-JEPA [13]. We provide the performance of I-JEPA and V-JEPA for five tasks of Franka Kitchen in **Table R10**. Moreover, since we have provided the performance of DINOv2 in Franka Kitchen in the supplementary material (Table A), we additionally compare ToBo with DINOv2 on 'relocate' and 'pen' tasks in Adroit in **Table R11**. As shown in **Table R10**, our ToBo surpass DINOv2, I-JEPA, and V-JEPA on all the tasks in Franka Kitchen. Moreover, ToBo also outperforms DINOv2 on all the tasks in Adroit. These results further support the effectiveness of ToBo. We promise to provide the performance of DINOv2, I-JEPA, V-JEPA in other benchmarks in the revised version.\n\n\n[**Table R10.** Comparison of ToBo with DINOv2, I-JEPA, and V-JEPA on Franka Kitchen]\n\n|                             |    Knob1 on   |   Light on    |   Sdoor open    |   Ldoor open    |   Micro open    |\n|------------------------|------------------|-----------------|---------------------|---------------------|---------------------|\n| DINOv2        |         27.3     |        38.7      |         82.0        |        16.0           |        20.0         |\n| I-JEPA           |         26.0     |        40.0      |         78.7        |         21.3         |         17.3         |\n| V-JEPA          |        21.3      |        41.3      |         74.7        |         28.0          |         12.7          |   \n| |\n| ToBo (ours)         |        **46.7**      |        **78.7**       |        **95.3**          |        **47.3**          |        **37.3**       | \n\n\n[**Table R11.** Comparison of ToBo and DINOv2 on Adroit]\n\n|                             |   Pen    |   Relocate    |\n|------------------------|-----------|------------------|\n| DINOv2         |  69.6    |       24.0       |\n| ToBo (ours)         |  **81.6**    |       **39.2**       |\n\n\n\n\n\n\n> ### **W2. Scalability of our method when trained on more samples**\n\nFollowing the advice, we verify the scalability of ToBo when trained on more samples. To this end, we compare ToBo models pre-trained on Kinetics-400 and Kinetics-600, a larger dataset that includes Kinetics-400, for 100 epochs. **Table R12** showcases that the ToBo model pre-trained on Kinetics600 surpasses the ToBo model pre-trained on Kinetics-400in all tasks except for the \"Sdoor Open\" task, where both models perform comparably. This results validate the scalability of ToBo for more samples.\n\n\n[**Table R12.** Scalability of our method when trained on more samples]\n\n|                             |    Knob1 on   |   Light on    |   Sdoor open    |   Ldoor open    |   Micro open    |\n|------------------------|------------------|-----------------|---------------------|---------------------|---------------------|\n| ToBo pre-trained on Kinetics-400   |        46.7      |        78.7       |        **95.3**          |        47.3          |        37.3       | \n| ToBo pre-trained on Kinetics-600 |        **51.0**     |        **80.3**      |        94.7          |        **48.7**          |        **45.3**       | \n\n\n> ### **W3. Clarification for much smaller improvements on label propagation**\n\nThe improvements in video label propagation are smaller because it requires dense and spatially distributed predictions, which do not directly align with our SSL objective. In contrast, the strong gains in robotics stems from the nature of our SSL objective, which enforces conservative summarization of observed scenes into a bottleneck token, a representation better suited for action prediction under partial observability.\n\n\nThe following are responses to the sub-questions: \n\n\n**Response to \"Is the agent's policy architecture being more suited to the proposed SSL method?\"**\nThe observed improvements in robotics tasks are not due to any special advantage in policy architecture. Our method directly adopts the policy network design and the evaluation protocol from prior works [3], without any modification specific to our SSL objective. Importantly, all compared SSL methods share the same policy architecture, ensuring a fair comparison. Therefore, the gains are not attributable to an architectural bias toward our method.\n\n**Response to \"Is the agent policy architecture the same for all the robotics tasks?\"**\nYes, the same policy network architecture is used across all robotics tasks. This uniform setup eliminates architectural variation as a source of performance gaps. Although the training mechanisms of models vary across SSL methods, all models share the same structure, which enables the use of a single policy design across the pre-trained models.\n\n\n\n\n\n### **References**\n\n[1] He et. al., \"Masked Autoencoders are Scalable Vision Learners,\" CVPR 2022.\n\n[2] Gupta et. al., \"Siamese masked autoencoders,\" NeurIPS 2023.\n\n[3] Jang et. al., \"Visual representation learning with stochastic frame prediction\", ICML 2024.\n\n[6] Kay et. al., \"The kinetics human action video dataset,\" arXiv:1705.06950, 2017.\n\n[11] Oquab et. al., \"DINOv2: Learning Robust Visual Features without Supervision\", arXiv:2304.07193.\n\n[12] Bardes et. al., \"Revisiting Feature Prediction for Learning Visual Representations from Video\", arXiv:2404.08471.\n\n[13] Assran et. al., \"Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture,\" ICCV 2023."}, {"review_id": "9t78IYScyN", "reviewer": "Reviewer_3Edo", "summary": "This work depicts that a single token is sufficient to compress the spatial and temporal dynamics of a video. Specifically, this work demonstrates that utilizing the class token of the reference frame in conjunction with a few target patches facilitates the learning of sequential scene representations.", "questions": "See weaknesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:-\n\n- The paper is predominantly well-written.\n- Extensive experiments on various downstream tasks such as robot policy learning have been conducted.\n- The main insight presented in the paper that utilizing a single token from the reference frame instead of utilizing all the patches is more beneficial, is very interesting.\n\nWeakness:-\n\n- The decoder architecture is not clear. While the visualization in figure 3 indicates attention from the bottleneck token to the unmasked patches, it is mentioned in L-156 that self-attention is employed. A better description would be helpful.\n- While the experiments are predominantly comprehensive, more comparisons with works performing masked image modeling on latent space such as i-BOT, DINOv2 and T-LORE could have been conducted.", "was_revised": true, "final_justification": "The authors have addressed many of my concerns and I believe that this paper can be accepted. So I’m maintaining my rating.", "rebuttal": "We sincerely thank Reviewer 3Edo for the thoughtful comments and constructive suggestions. Below, we provide detailed responses to each point.\n\n> ### **W1. Clarification on the description of the decoder architecture**\n\nWe apologize for the confusion. During the decoding process, the visible and masked target patches are concatenated with the bottleneck token and then fed into the self-attention layer. We promise to address the confusions in the descriptions.\n\n> ### **W2. Comparison with i-BOT, DINOv2 and T-CORE**\n\nWe provide the comparison results of DINOv2 [11], iBoT [15], and T-CORE [16] in five tasks in Franka Kitchen. Since we have provided the performance of DINOv2 in Franka Kitchen in the supplementary material (Table A), we directly report the results in **Table R13**. Moreover, we further evaluate the performance of iBoT and T-LORE. As shown in **Table R13**, our ToBo surpasses DINOv2, iBoT, and T-LORE on all the five tasks. We promise to provide the performance of DINOv2, iBoT, T-LORE in other benchmarks in the revised version.\n\n\n[**Table R13.** Comparison of ToBo with DINOv2 and T-CORE on Franka Kitchen]\n\n\n\n|                             |    Knob1 on   |   Light on    |   Sdoor open    |   Ldoor open    |   Micro open    |\n|------------------------|------------------|-----------------|---------------------|---------------------|---------------------|\n| DINOv2         |         27.3     |        38.7      |         82.0        |        16.0           |        20.0         |\n| iBoT             |         37.3    |        55.3      |         75.3          |        12.7          |        47.3          |\n| T-CORE         |       7.3        |        17.3      |         69.3        |          11.3         |       8.0             |\n| |\n| ToBo (ours)          |        **57.3**      |        **82.0**      |        **95.0**          |        **51.0**          |        **55.0**         |\n\n\n\n### **References**\n\n[11] Oquab et. al., \"DINOv2: Learning Robust Visual Features without Supervision\", arXiv:2304.07193.\n\n[15] Zhou et., al., iBOT: Image BERT Pre-Training with Online Tokenizer, ICLR 2022.\n\n[16] Liu et. al., \"When the Future Becomes the Past: Taming Temporal Correspondence for Self-supervised Video Representation Learning\", CVPR 2025."}]

Author rebuttal:

Meta-review: The paper introduces a Token Bottleneck approach that allows to learn a self-supervised representation of the sequential scene by squeezing it into a token bottleneck. The model is trained to reconstruct a scene solely from the token bottleneck, together with a few target “hints” (patches of the target scene). The usefulness of the approach is demonstrated on real-world robot manipulation tasks.

The reviewers praise that the approach is simple but effective, particularly performing well on the robotics tasks. During the rebuttal, the authors also extended the ToBo method to a multi-frame approach, which added a valuable comparison. The authors also provided extensive comparisons and ablations to other existing self-supervised methods and demonstrated that token bottleneck remains superior.

The recommendation is Accept.
Decision: Accept (poster)