Title: Token Bottleneck: One Token to Remember Dynamics
Venue: NeurIPS 2025
Accepted: yes
Keywords: Self-supervised Learning

Representation Learning
Abstract: Deriving compact and temporally aware visual representations from dynamic scenes is essential for successful execution of sequential scene understanding tasks such as visual tracking and robotic manipulation. In this paper, we introduce Token Bottleneck (ToBo), a simple yet intuitive self-supervised learning pipeline that squeezes a scene into a bottleneck token and predicts the subsequent scene using minimal patches as hints. The ToBo pipeline facilitates the learning of sequential scene representations by conservatively encoding the reference scene into a compact bottleneck token during the squeeze step. In the expansion step, we guide the model to capture temporal dynamics by predicting the target scene using the bottleneck token along with few target patches as hints. This design encourages the vision backbone to embed temporal dependencies, thereby enabling understanding of dynamic transitions across scenes. Extensive experiments in diverse sequential tasks, including video label propagation and robot manipulation in simulated environments demonstrate the superiority of ToBo over baselines. Moreover, deploying our pre-trained model on physical robots confirms its robustness and effectiveness in real-world environments. We further validate the scalability of ToBo across different model scales. Code is available at https://github.com/naver-ai/tobo.


Peer reviews:
comments: SUMMARY:
This work depicts that a single token is sufficient to compress the spatial and temporal dynamics of a video. Specifically, this work demonstrates that utilizing the class token of the reference frame in conjunction with a few target patches facilitates the learning of sequential scene representations.

STRENGTHS AND WEAKNESSES:
Strengths:-

- The paper is predominantly well-written.
- Extensive experiments on various downstream tasks such as robot policy learning have been conducted.
- The main insight presented in the paper that utilizing a single token from the reference frame instead of utilizing all the patches is more beneficial, is very interesting.

Weakness:-

- The decoder architecture is not clear. While the visualization in figure 3 indicates attention from the bottleneck token to the unmasked patches, it is mentioned in L-156 that self-attention is employed. A better description would be helpful.
- While the experiments are predominantly comprehensive, more comparisons with works performing masked image modeling on latent space such as i-BOT, DINOv2 and T-LORE could have been conducted.

QUESTIONS:
See weaknesses

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a self-supervised vision pretraining method for sequential scene based tasks by bottlenecking the representation using a single token. The authors demonstrate the effectiveness of this approach in simulation and real world robotics tasks, with significant improvements over other methods like SimCLR and MAE(and its variants). The authors also demonstrate superior performance of their proposed method in label(mask,pose) propagation tasks over these baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The authors propose a novel self-supervised pretraining method to capture both temporal dynamics and summarize scene information in the same embedding.\n2.  The authors demonstrate the superior performance of their proposed method in robotics tasks and label propagation when compared to other SSL methods like MAE variants and SimCLR.\n\n\n\nWeaknesses\n1. The authors do not compare with newer self supervised vision pretraining methods like DINO v2 (except franka-kitchen) and I/V-JEPA on all the tasks. This is required to contextualise the  performance of the proposed method in the current SSL landscape. \n2. The authors need to demonstrate the scaling property of the method, when trained on more data and not just a larger model.\n3. The improvements in label propagation are much smaller than the improvements in robotics policy learning tasks. This needs to be explained/investigated. Is this an artefact of the agent's policy architecture being more suited to the proposed SSL method as opposed to the other SSL methods? Is the agent policy architecture the same for all the robotics tasks ? \n4. While the authors focus on bottlenecking the representation using a single embedding, it would be interesting to see the effect of increasing the number of embeddings in the bottleneck effectively relaxing the constraint gradually.\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper investigates how to use the dynamics information in video to pretrain an image encoder, extending MAE. The proposed ToBo(Token Bottleneck) has a unique pretext task of compressing an image into a single token, then decoding a later frame based on the bottleneck token and sparse target frame patches. The method is mainly evaluated on video label propagation and robotics tasks, yielding especially large improvements in robotics manipulation.

STRENGTHS AND WEAKNESSES:
Strength:
1. The proposed approach is simple but effective. The pretext task intuitively makes sense and is grounded in previous literature.
2. The improvements on especially robotics tasks is impressive.
3. Robotics experiments include error bars, which is informative.

Weaknesses:
1. The variance on the proposed approach seems especially large, especially in Table 3, compared to the baselines. Could the author provide some insight into why that is the case?
2. More careful discussion of the limitation should be included in the final paper.
3. Since we are learning representation from video, is it possible to take the VideoMAE approach to pretrain a video model, then somehow adapt it to the robotics use case?

QUESTIONS:
Please see the weakness section. To summarize, I have questions regarding the variance of performances, more careful discussion of limitations, and direct video pretraining. I am open to increasing the score if these items are clarified.

LIMITATIONS:
The limitation discussion is too short, probably due to paper length limitation. But it should be revisited more carefully.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for learning visual representations. The method is based on MAE and called Token Bottleneck. The basic idea is to learn representations from a pair of images of the same scene (two different views / video frames). The method trains a model to compress the patches of the first image into a single bottleneck token and, from that token plus visible patches from the second image, predict the masked patches of the second image. The results are evaluated on several simulated robotic tasks and davis label propagation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- I reviewed an earlier version of this paper for ICCV 2025. Compared to that version, the present submission is considerably improved. The writing, method name, and description are much clearer. Some of the missing baselines are included too. I appreciate the updates and thank the authors for incorporating the feedback from the previous review.\n- The method is simple, easy to understand, and shows promising signal in tested settings.\n\nWeaknesses:\n- 1) Key design choices are not ablated which makes it difficult to understand different aspects of the proposed method. Some examples:\n    - a) How many patches to squeeze the information into? Single token vs multiple? How does the tradeoff look like?\n    - b) How does the temporal difference impact the performance?\n    - c) What would happen if there was no temporal difference between the frames? Would the task be trivial or still useful?\n    - d) ...\n- 2) It would be good to include comparisons between different methods (e.g., vanilla MAE, SimSiam, etc.) in terms of flops. Both training time flops and inference flops.\n- 3) Comparisons on standard benchmarks like ImageNet and Kinetics missing. I understand that the main focus is on simulated robotic evaluation and tracking but it would still be helpful to see how the learnt representations compare to standard approaches like MAE on standard benchmarks.\n- 4) It would be good to see straightforward baselines like running vanilla MAE across both views/images.\n\nQUESTIONS:\nPlease see the weaknesses above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}