Title: Object Concepts Emerge from Motion
Venue: NeurIPS 2025
Accepted: yes
Keywords: Pretraining; Visual Foundation Models; Object Discovery
Abstract: Object concepts play a foundational role in human visual cognition, enabling perception, memory, and interaction in the physical world. Inspired by findings in developmental psychology—where infants are shown to acquire object understanding through observation of motion—we propose a biologically inspired framework for learning object-centric visual representations in an unsupervised manner.
We were inspired by the insight that motion boundary serves as a strong signal for object-level grouping, which can be used to derive pseudo-instance supervision from raw videos.
Concretely, we generate motion-based instance masks using off-the-shelf optical flow and clustering algorithms, and use them to train visual encoders via contrastive learning. Our framework is fully label-free and does not rely on camera calibration, making it scalable to large-scale unstructured video data.
We evaluate our approach on three downstream tasks spanning both low-level (monocular depth estimation) and high-level (3D object detection and occupancy prediction) vision. Our models outperform previous supervised and self-supervised baselines and demonstrate strong generalization to unseen scenes. These results suggest that motion-induced object representations offer a compelling alternative to existing vision foundation models, capturing a crucial but overlooked level of abstraction: the visual instance.
The implementation can be found here: https://github.com/yulemao/Object_Concepts_Emerge_from_Motion


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInspired by insights from classic computer vision and developmental psychology regarding the importance of motion boundaries for inferring object boundaries and grouping image elements with similar motion into objects, the authors propose a system for unsupervised learning of \"object-centric\" visual representations. They leverage an off-the-shelf optic flow algorithm to generate motion-defined object instance masks and use them for contrastive visual representation learning. The approach is evaluated extensively and shown to outperform previous supervised and self-supervised baselines on three downstream tasks (monocular depth estimation, 3D object detection, occupancy prediction).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe approach is novel and outperforms several baselines on a set of diverse tasks comprising low-level (monocular depth estimation) and high-level (3D object detection, occupancy prediction) ones. I find it noteworthy that the approach even outperforms some supervised baselines. I feel the work is likely to lead to many follow-up works. As a work demonstrating strong representations emerging from unsupervised learning from large video datasets it is very timely and interesting.\n\nI enjoyed the generalization to indoor scenes outside of the training domain.\n\nWeaknesses:\n\n1) I think some claims need to be toned down:\n1a) Abstract:\n\"Our key insight is that motion boundary serves as a strong signal for object-level grouping\". This is not at all a new insight. You are just using this very old insight in a novel way.\n\n1b) Introduction:\n\"we are the first to leverage motion as an unsupervised supervisory signal to guide the emergence of object-level semantics in visual\nrepresentations.\" This statement also seems rather bold. There is a long history of works (pre-dating the deep learning era) of using motion to segment objects or infer the 3-D structure of the world. I think you should replace \"emergence of object-level semantics in visual\nrepresentations\" with something more specific.\n\n2) You frequently use \"Developmental Neuroscience\" when \"Developmental Psycholog\" is more correct. (Developmental Neuroscience is typically studying the developing brains of animals with invasive techniques that cannot be applied to human infants and children.)\n\n3) Occasional grammar problems\nline 124: missing \"that\" or similar\nline 141: used to separate or used for separating\nline 181: exhibit\nline 195: that -> whose. Or use: for which computational costs scale...\n\nQUESTIONS:\nFor future work I'd invite the authors to think about replacing the off-the-shelf optic flow algorithm with a learning-based framework.\n\nLIMITATIONS:\nI did not see a particular mentioning of potential negative societal impact of this work. But I also don't see this as necessary.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

comments: SUMMARY:
This paper proposes a motion-induced object-centric pretraining method for vision foundation models. The proposed method uses an optical flow model to extract motion clues from raw videos, and derive object pseudo labels through clustering on optical flow map. Then a simple contrastive learning objective is proposed to learn the concept of object from the annotated images. Extensive experiments on depth estimation, 3D object detection and 3D occupancy prediction validate the effectiveness of the proposed method.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Good motivation. The proposed method originates from the biological perspective and aligns with the development of human cognition.
2. Simple yet effective method. The proposed method consists of a simple contrastive loss, but is verified to surpass other advanced pretraining methods.
3. Extensive experiments. The paper conducts extensive experiments on three downstream tasks, on all of which the proposed method demonstrate good performance.

Weaknesses:
1. The performance of the proposed method is very close to SimMIM under large model parameters.
2. The types of downstream tasks is limited, all of which is closely related to 3D perception of the environment. According to the math derivation, the pseudo labels inherently correlate with the depth of the image, which might render 3D-related downstream tasks unfair for other pretraining methods.

QUESTIONS:
1. Could the authors explain the similar performance with SimMIM in depth estimation and occupancy prediction?
2. Could the authors provide the results for SimMIM under smaller model sizes for more comprehensive comparison?
3. How does the method perform in other 2D downstream tasks, such as 2D semantic/instance segmentation?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to learn semantically separated features from motion signals. The method uses an optical flow model to predict the flow for a large set of driving scenes. This flow is then used in a clustering step to create pseudo-labels for all videos. A network is trained on those pseudo-labels and evaluated on various tasks, such as Depth Estimation, BEV Prediction, and Occlusion Prediction. The results show the method can be effective for these tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The idea of exploiting coherent motion as as representation of objectness is very interesting and appealing\n- The fact that the method is designed to work on a large unlabeled data pool of videos makes it scalable\n- The presented quantitative results are somewhat consistent and improve upon the SOTA\n- The qualitative results are nicely presented and show the effectiveness of the pseudo-labeling\n\nWeaknesses:\n- The choice of datasets seems to be restricted to driving videos. I wonder why this choice was made since the method could also work on other types of natural videos? This choice fails to fully exploit the potential of the proposed method\n- The proposed method is rather simple and seems a bit handcrafted. However, it is somewhat effective, so this is not critical.\n- Lack of ablations: In the main paper, the method’s components and parameter choices are barely ablated. I think this is crucial for the paper. There would be a lot of value in seeing how effective the pseudo-labeling is under different parameter choices like the manually chosen thresholds in L183, different optical flow models like RAFT [1] or SEA-RAFT [2], different choices for the margin parameter L209, and so on\n- More model comparisons: It would be great to also see other video self-supervised models compared, for example [3]. Furthermore, DINOv2 was only compared with a ViT-S, while the proposed method uses up to a Swin-L. It would be nice to also see other sizes for DINOv2.\n\n[1] Teed, Zachary, and Jia Deng. \"Raft: Recurrent all-pairs field transforms for optical flow.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020.\n[2] Wang, Yihan, Lahav Lipson, and Jia Deng. \"Sea-raft: Simple, efficient, accurate raft for optical flow.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n[3] Salehi, Mohammadreza, et al. \"Time does tell: Self-supervised time-tuning of dense image representations.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\nQUESTIONS:\nWhile I think the idea proposed by this paper is valuable, there are various aspects that need to be addressed.\n- Extension beyond just driving scenes: How does the method behave when trained on other natural scenes?\n- How does the approach behave under different parameter combinations & optical flow models (see above)?\n- How does it compare to other video SSL approaches and other DINOv2 sizes?\n\nIf the authors can address my concerns, I'm willing to update my score.\n\nLIMITATIONS:\nBarely, but yes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a framework to learn object-centric visual features from motion inspired by findings from developmental neuro-science. An off-the-shelf flow model is used to detect motion in videos. The flow is then clustered to generate object pseudo-labels. The labeled clusters are then used to train an encoder using contrastive loss.  The learnt features are evaluated on the tasks of depth-estimation, 3D object detection and occupancy prediction. State-of-the-art performance is demonstrated with respect to several relevant baselines.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n1. **Label free method**: The proposed approach is label-free and does not require additional annotations like camera parameters. This is particularly useful to extract structured information from a large collection of videos.\n2. **Quantitative results**: The approach outperforms DINOv2 and other relevant baselines on 3D tasks.   \n\n\n## Weaknesses:\n1. **Novelty**: The core idea of using motion to learn objectness has been explored before (points on a object have 'common fate'). The approach uses an off the shelf flow estimation model \n2. **Dependence on optical flow estimator**: The base of the approach relies on clustering the motion representation obtained from an off-the-shelf optical flow estimator thus inherits all the limitation of the chosen base model. In particular, optical flow models fail to do well on flat textured/ untextured regions of a video. Since the clustering and training starts from this step, is there a tendency for the errors to cascade across the stages? \n3. **Qualitative examples**: Providing more qualitative results would be beneficial to help appreciate the effectiveness of the approach.\n4. **Writing**: There are number of typos and language errors throughout the manuscript. The paper would benefit from a thorough proof reading and streamlining of certain sections.\n\nQUESTIONS:\n1. The clustering is done based on flow information. What if the data contains video with low motion/ or videos of scenes with no specific objects, how do instances like these affect the clustering?\n2. Is there a curation process that the data has to go through to be able to generate reasonable clusters.\n\nLIMITATIONS:\nThe limitations of the framework have been adequately discussed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a self-supervised, instance-aware framework for learning dense visual representations. Their supervision signal is derived from pseudo-segmentation maps obtained by clustering optical flow. An encoder is trained using a triplet ranking loss to produce dense feature maps where pixel-wise similarities align with the inferred region maps. For evaluation, the authors present region-similarity visualizations and quantitative benchmarks on depth estimation and 3D object detection, demonstrating that their approach outperforms contrastive-based self-supervised learning methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n(1) Learning instance-level representations in a self-supervised manner remains a challenging task. The authors present a simple yet effective framework that outperforms state-of-the-art contrastive learning approaches on selected benchmarks. \n(2) Visualizations of the learned pixel-wise affinity maps indicate that the model captures meaningful instance-level embeddings.\n(3) Good paper presentation.\n\nWeakness:\n(1) The method relies on externally estimated optical flow as input, but the authors do not include ablation studies across different flow estimators to assess the robustness of their approach.\n(2) While the method is evaluated on depth estimation and 3D object detection benchmarks, it lacks several key metrics: specifically, evaluations of the learned feature maps on semantic and instance segmentation tasks via fine-tuning, linear probing, or unsupervised clustering. Comparisons to baseline methods, such as DINO, on image classification tasks are also missing.\n(3) The backbone used in the proposed method differs from those in baseline approaches (e.g., DINO), as it is designed to produce higher-resolution feature maps. This architectural difference raises questions about whether performance gains in high-resolution tasks like depth estimation are due to the proposed method or simply the model architecture. For instance, could a DINO model with a similar FPN-style decoder yield comparable results?\n(4) The paper omits comparisons with relevant prior work that also targets instance-aware embeddings [1] or learns objectness from video signals [2].\n\n[1] Zhang et al., Self-Supervised Visual Representation Learning from Hierarchical Grouping\n[2] Liu et al., The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos\n\nQUESTIONS:\nI have questions about the robustness of the proposed methods for different flow estimations and their performance on the standard benchmark. Please see the weakness for details.\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-09"}