Object Concepts Emerge from Motion
Authors: Haoqian Liang
Xiaohui Wang
Zhichao Li
Ya Yang
Naiyan Wang
Venue: NeurIPS 2025 poster ()
Keywords: Pretraining; Visual Foundation Models; Object Discovery
Primary area: deep_learning

Abstract: Object concepts play a foundational role in human visual cognition, enabling perception, memory, and interaction in the physical world. Inspired by findings in developmental psychology—where infants are shown to acquire object understanding through observation of motion—we propose a biologically inspired framework for learning object-centric visual representations in an unsupervised manner.
We were inspired by the insight that motion boundary serves as a strong signal for object-level grouping, which can be used to derive pseudo-instance supervision from raw videos.
Concretely, we generate motion-based instance masks using off-the-shelf optical flow and clustering algorithms, and use them to train visual encoders via contrastive learning. Our framework is fully label-free and does not rely on camera calibration, making it scalable to large-scale unstructured video data.
We evaluate our approach on three downstream tasks spanning both low-level (monocular depth estimation) and high-level (3D object detection and occupancy prediction) vision. Our models outperform previous supervised and self-supervised baselines and demonstrate strong generalization to unseen scenes. These results suggest that motion-induced object representations offer a compelling alternative to existing vision foundation models, capturing a crucial but overlooked level of abstraction: the visual instance.
The implementation can be found here: https://github.com/yulemao/Object_Concepts_Emerge_from_Motion

Object Concepts Emerge from Motion
Haoqian Liang', Xiaohui Wang', Zhichao Li, Ya Yang', Naiyan Wang
Beijing University of Posts and Telecommunications lianghgubupt. edu. en, nekomi wwwpom.i@uupe... ed..pt.dduccom en, leeisabug@gmaill com, yangyalbupt edu. cn, winsty@gmail com
Original Image
DINOv2
CLIP
MAE
Ours
Figure Comparisons with feature maps learned by our method and different visual foundation models. Our method focuses the unity of object instance, in contrast to other methods emphasize on object class more.
Abstract
Object concepts play a foundational role in human visual cognition, enabling per- ception, memory, and interaction in the physical world. Inspired by findings in developmental psychology- where infants are shown to acquire object understand- ing through observation of motion-we propose biologically inspired framework for learning object-centric visual representations in an unsupervised manner. We were inspired by the insight that motion boundary serves as strong signal for object-level grouping. which can be used to derive pseudo-instance supervision from raw videos. Concretely, off the-shelf optical flow and clustering algorithms, and use them to train visual encoders via contrastive learning. Our framework fully label-free and does not rely on camera calibration, making it scalable to large-scale unstructured video data. We evaluate our approach on three downstream tasks spanning both low-level (monocular depth estimation) and high-level (3I) object detection and occupancy prediction) vision. Our models outperform previous supervised and self-supervised baselines and demonstrate strong generalization to unseen scenes. These results suggest that motion-inducee object representations offer a compelling alternative to existing vision foundation models, capturing crucial but overlooked level abstraction: the visual instance. The implementation can be found here: https://gitbbbcommyylllm//////pp Concepts Emerge from Motion
generate motion-based instance masks using
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Physical Al aims to develop intelligent agents capable of perceiving and interacting with the physical world. A fundamental cognitive capacity required for such agents is the ability to recognize and understand the concept of "object" a core unit of perception and reasoning. In the human visual system, the importance of object concepts is well-established in neuroscience. As noted by Kellman and Spelke [27], "this cognitive ability not only supports object recognition and classification, but also plays crucial role in spatial perception, memory formation, and the interaction between objects and their environment." Understanding how object concepts are formed and represented in biological systems provides critical insights for building more robust and generalizable visual agents in artificial systems.
However, what makes an object look like an object? This is a non-trivial question, as objects can vary drastically appearance, shape, and motion patterns. Early studies in developmental psychology [27 have demonstrated that the ability to perceive object unity is not innate, but learned during infancy. Infants begin to exhibit evidence of understanding object cohesion from around two months of age, with robust performance observed by four months. These findings suggest that object perception
a learned capacity grounded in sensory experience. Subsequent research [24] 40] has shown that motion cues- purt-particllarly common or coherent motion object boundaries and unity. As the visual system matures, this dynamic understanding is gradually internalized into the ventral visual stream [29 [53, enabling object recognition from static visual inputs alone. Inspired by this developmental trajectory, our work aims to design an unsupervised computational model that mimics this learning process: beginning with motion-basee interactions and evolving toward abstract, appearance- based object concepts.
a powerful signal for infants to infer
Recently, learning universal visual representations through self- eeff-supervise or weakly supervised paradigms has gained significant attention, due to their strong performance across a wide range of vision tasks, Among self-supervised approaches, notable examples include the DINO 7 41 and MAE 123
64 families, which rely on self-distillation and self-reconstruction mechanisms, respectively, to learn robust feature representations. Another influential direction leverages web-scale image-text pairs, as exemplified by CLIP [45), to align visual and language representations. To better understand what these models capture, compare the low-dimensional PCA projections of features extracted by DINO, CLIP, and our model (see Fig. DT We observe that DINO and CLIP tend to focus semantic categories. However, neither method captures the concept of semantic instance- a distinct, coherent object autit-adeequaee) We argue that existing visual foundation models overlook this crucial level of abstraction, which is fundamental for understanding the physical world.
In this work, we propose a biologically inspired framework for learning visual features that encode object-level semanties. As a first step, we explore this approach in outdoor driving scenarios, which provide rich motion cues arising from both ego-motion and independently moving objects. The key observation that inspires our method is that motion boundaries often align with object boundaries (detailed in Sec. 3.I. which echoes the discoveries in neuroscience that common motion is crucial to the early development of object unity. Based on this, we employ an off-the-shelf optical flow estimation algorithm, followed by simple clustering technique, generate pseudo instance masks without human supervision. These instance labels are then used to supervise representation learning via a contrastive objective. Importantly, unlike previous approaches [74] 4 method does not require camera calibration parameters, allowing it to scale to large and diverse unlabeled video datasets.
Our sudiond learning paradigm naturally bridges low-level and high-level vision. We validate method three downstream tasks: monocular depth estimation (low-level), 3D object detection and occupancy prediction (high-level). Across all model sizes, our method consistently outperforms supervised pretraining on ImageNet-22K and other self-supervised learning methods, demonstrating the effectiveness of learning object-centric representations from motion cues. Moreover, we find that our features are complementary to those from existing foundation models such DINO fusing them leads to further performance gains. Interestingly, although our model is trained only on outdoor scenes, it generalizes well to unseen indoor environments. This suggests that the learned features capture object composition and structure, rather than merely memorizing training-set appearances.
To summarize, our key contributions are follows:

---

We propose a biologically inspired paradigm for object-centric visual representation learning. Motivated by studies of infant cogmition, we are the first to demonstrate the effectiveness and scalability of using motion as an unsupervised supervisory signal on a large-scale dataset and modern model architectures.
We introduce a computationally efficient framework that implements this paradigm using off-the-shelf optical flow and simple clustering. Our approach scales naturally to large-scale outdoor driving datasets without requiring camera calibration or manual labels, and supports models of varying capacities.
We extensively evaluate our models on three downstream tasks-monoculaa depth estima- tion (low-level), 3D object detection and occupancy prediction (high-level). Our method outperforms supervised and self-supervised methods across all model sizes, and shows strong generalization unseen indoor scenes, highlighting the robustness and transferability of the learned object-centric features.
2 Related Work
2.1 Object Discovery
The central aim of object discovery is the identification and localization of objects within visual data, including images and videos, without the prerequisite of instance-level annotations for specific object classes. This paradigm significantly mitigates the need for large-scale, high-quality labeled datasets. Early approaches to object discovery included methods based object occurrence frequency [25] 26 [44, and techniques utilizing region proposals to select key object bounding boxes through combinatorial optimization [52] 55
62 75 More recently, researchers have proposed numerous learning-based methods built upon the Transformer architecture. These approaches leverage features obtained from powerful pre-trat pre-trained image models (c.g., DINO) to identify and segment objects via graph-basee rraphbased or spectral clustering techniques [46] 55 59, 60 73). Another line of research adopts an reconstruction methodologies to derive learning signals, that involve decomposing scenes into their constituent parts (e.g.. objects, background) and learning their respective representations (5] [8 36 38). Similarly, another class of methods utilizes motion and multi-modal information as supervision, using the motion consistency
56
object-centric perspective, frequently utilizing scene generation
of 2D or 3D points as a cue to distinguish objects from the background [44 |1 61 72]. To address
the high demand for input dependencies of these works, our method only requires the simplest
optical flow and clustering process acquire object masks in an unsupervised manner. These masks subsequently serve as pseudo labels for angle-image representation learning.
2.2 Visual Foundation Models
Visual foundation models aim to learn broadly applicable and transferable visual representations by pre-training on large-scale data. These learned general representations are intended to be transferred to downstream tasks by fine-tuning or prompting. Various self supervide learning paradigms have been proposed for visual foundation models. Contrastive-lcarning based methods pull together representations of different augmented views of the same image (positive samples) in an embedding space, while pushing apart representations of different images (negative samples) 10 14 22 70)
Building upon this, subsequent self-distillation methods utilize "teacher" signals moving averaged by the student model itself for guided training, achieving excellent performance without relying on
negative sample pairs 7 12 19 41). On the other hand, inspired by masked language modeling in methods learn representations by randomly
natural language processing,
masking portions of an image and training the model to reconstruct the masked content 2 23 58
64). There is also some works that jointly learn the embeddings and predictions for more semantically
rich and general features [ 3. Furthermore, despite different supervisory approaches, methods employing weak supervision, such as through text, have also made significant progress in the field of foundation models [45). However, as illustrated in Fig. [ all these pretrained models provide semantic class features rather than semantic instance features. We argue that semantic instance features could be beneficial for downstream tasks that require instance- separation, such as object detection.
3 Method
To build an object-centric visual representation that generalizes across tasks and environments, we propose biologically inspired learning framework centered on motion cues. Rooted in cognitive
3

---

Pixel Cluster
Video Flow
Labeling
Backbone
PUSH
Training
Figure 2: Pipeline of the proposed method.
developmental insights, our approach leverages the observation that coherent motion often indicates objecthood- an idea supported by infant perception studies and geometric reasoning in dynamic scenes. In this section, we introduce our method, which consists of three key components: (1) geometric analysis revealing how motion boundaries correlate with object boundaries, (2) data processing pipeline that extracts motion- motion-induced pseudo-labels from large-scale video data, and (3) an unsupervised training objective designed learn robust and transferable features from these labels. Together, these components form scalable and calibration-free paradigm for learning object-level semantics from raw videos. The whole pipeline shown in Fig.
3.1 Geometric Insights
central insight of our approach is that motion boundaries often align with object boundaries. is obvious that if the object itself moves, its flow boundary naturally serve the object boundary. In this section, we provide a geometric and mathematical justification for why ego-motion can also be used separate different objects under the assumption of rigid scenes.
Let p = (u, v)) denote a pixel the image domain, and D(u, v) corresponding depth. Assuming pinhole camera model with intrinsic matrix K, p the pixel projected by the 3D point P. Under rigid motion, the 3D scene point undergoes a transformation via camera pose change (R, t) E SE),., resulting in new image projection p' in the next frame. Projecting ' back into the image plane yields:
K K R DR, K
U
+t
The optical flow is then computed as the pixel displacement. This means that the optical flow F(u, v) function of the depth (u, v), the camera motion (R. t). and the camera intrinsies K. We summarize this dependency as:
F(u, v)
(D((u,)))
Taking the spatial gradient of the flow field gives:
VF(u,v) do dD VD(uv)
This expression indicates that discontinuities in the flow field-i.e. motion boundaries -an arise from large gradients in the depth map. Under the assumption of rigid motion, these motion boundaries serve as reliable proxies for object boundaries. This geometric insight underpins our approach of utilizing motion cues derive instance-level supervision.
This concept aligns with foundational theories in computer vision. David Marr, in his seminal book [39], articulated that:

---

Input
Flow
Label
Feature
Figure 3: Examples of the pseudo-label generation results and the output features.
"...the velocity field of motion the image varies continuously almost everywhere, and if it is ever discontinuous at more than isolated point, then failure of rigidity (like an object boundary) is present in the outside world. In particular, if the direction of motion is ever discontinuous more than one point-along line, for example- then an object boundary is present."
notable advantage of our method is its independence from camera calibration. The necessary geometric information is inherently encoded within the optical flow, enabling us to train large-scale, uncalibrated video datasets. This approach enhances scalability and broadens the applicability of our framework across diverse real-world scenarios.
3.2 Data Processing
Data Sources. We use two datasets in our approach: OpenDV- YouTube [67 and nuPlan 20]. Both datasets provide large amount of high-quality and diverse unlabeled video data. OpenDV-You YouTube contains videos collected from more than 244 cities all over the world, resulting a total of 1747 hours of front-view videos. nuPlan provides 8 different camera views. It collects 1200 hours of driving data from cities, 120 hours of which were recorded with 8 different camera views. We merged the two datasets and obtained approximately 2,700 hours of raw video data in total.
extract optical flow informa- tion from videos. The model takes five frames as input and outputs the optical flow for the middle three frames. We sample each clip with 0.3s intervals, and for each clip, we select the first frame within two consecutive frames as input, which spans second.
Optical Flow Estimation. We apply the pipeline of VideoFlow
Pixel Cluster. For all optical flow data generated by VideoFlow, perform simple Breadth- First Search (BFS) to cluster objects. Our algorithm takes the optical flow, the forward-backwand consistency check result, and two thresholds 0, and 0,, input. For each pixel that satisfies the howard-backwart consistency check, all neighboring pixels with a flow difference smaller than 0 are considered belong the same object. 02 the minimum number of pixels to form a cluster. Pscudo-code of our algorithm is provided in the appendix.
Results. We set two thresholds 8, 1.5. A. 100. Fig. shows some example results. As illustrated the pseudo-label visualizations, the proposed algorithm successfully segments objects exhibiting significant movement, such as moving cars and pedestrians. Furthermore, because objects at different depths exhibit different apparent motion in the image even when they are stationary, the proposed algorithm is also able to segment foreground instances such trees and signs. The pseudo-labels exhibit under segmentation due to weak motion cues or errors in optical flow estimation. Such cases explicitly handled in the design of the loss function. We retained all samples with least two labeso-.abessiie.. at least one foreground cluster) and successfully obtained total of 48 million images along with their corresponding pseudo-labels for pre-training.
3.3 Pre-training
Overall Structure. Our network architecture follows the design proposed (28]. The network takes one image as input. Due to the need for high-resolution feature maps, we choose backbone
5

---

networks(e.g. ResNet 21 Swin 335 whose computational cost scales linearly with input size. These features are then processed by a Feature Pyramid Network (FPN). Similar the semantic segmentation branch in [8)] the information from all levels of the FPN pyramid is merged into simple output. The resulting feature map has a spatial resolution of 1/4 the input size and a channel dimension of 64. A 2x bilinear upsampling is then applied, and each feature vector is normalized to yield the final output features.
Training Loss. Based on the labels derived from the optical flow and the output feature map, we design a simple yet effective loss function. As discussed in Sec. 3.2 the pseudo-labels can only extract a subset of the instances, making it inappropriate to cluster all background regions together. Since the number of background pixels is usually significantly larger than that of instance pixels, we treat the label with the highest pixel count as the background. The loss between any two background pixels is ignored. The loss function between two pixels and j is defined as follows:
|| - 22 L(I,)) max | 033
= + 0 VA + yj y = yj =0
(4)
where F; and F3 are the feature vectors of the final output feature map of the network, m is margin parameter, y represents the instance label derived from the optical flow. y = 0 denotes the background. We set the margin parameter to 1.0 in our implementation.
The total loss over all sampled pixel pairs is defined as:
1
Ltotal N
(5)
4 Experiments
To validate the effectiveness of our method across the vision spectrum, conduct comprehensive experiments on both low-level and high-level vision tasks. Our core hypothesis is that the model, by learning from lew-level cues, develops internal understanding of object composition, which subsequently benefits high-level semantic reasoning. Conversely, this object-centric representation also enhances performance on low-level tasks by providing richer contextual cues. We evaluate models on three representative downstream tasks: monocular depth estimation (low-level), 3D object detection, and 3D occupancy prediction (high-level).
4.1 Implementation Details
We implement the proposed method using Py 43 and mmPretrain 11. We train models on Swin Transformer (35] (Tiny to Large) and ResNet-50 21. All Swin models use a window size of 7, while the B and L variants of SimMIM (44 and Semantic-SAM (30], which are used for comparison, adopt larger window size of 12. This larger window is usually beneficial due to the increased context, at the expense of higher computational cost. Adam W optimizer (37] with weight decay of 0.05 is adopted. All models are trained for 200 epochs using cosine decay learning rate scheduler and 10 epochs of linear warm-up. The initial learning rate is set to 0.001 and batch size is set to 2048. All input images augmentation strategy that includes random flipping, brightness, and gamma adjustment. We sample 200 labeled pixels from each image for training. We further fine-tune the models for 20 epochs with an initial learning rate of 2 x 10 and a weight decay of 10 During fine-tuning, two random crops extracted from each input image, and the loss is calculated both within each crop and between the two. This fine-tuning process further enhances the separation of distant objects in large images. All downstream models are trained with official open-sourced code for comparison. During fine-tuning on downstream tasks, only the pretrained weights of the backbone are utilized for fair comparison.
cropped and resized to a resolution of 224 224. We employ data
4.2 Qualitative Results
The fourth column in Fig. 3 shows PCA projections of our model's 's features. Thanks to the generaliza- tion the backbone network, the features reveal a key strength: the model distinguishes many objects not annotated in the pseudo-labels- such as distant cars, pedestrians, and even static structures like buildings and poles. This suggests that our model goes beyond mimicking pseudo-labels and learns a more general, object-centric representation. Fig. further visualizes similarity maps from selected

---

Figure 4: Similarity visualization for set of reference points.
Table 1: Quantitative evaluation of DCDepth [57] on the KITTI Eigen split using different pretraining.
Method
Backbone
SILog +
Abs Rel
Sq Rel 1
RMSE + RMSE log J
51
82
83
ImageNet-22KK
Swin-T 7.455
0.055
0.165
2.182
0.082
0.969
0.996
0.999
Semantic-SAN
30
Swin-T 7.346
0.055
0.165
2.169
0.082
0,971
0.996
0.999
DINOv2
A1
ViT-S
7.119
0.052
0.158
2.153
0.079
0.974
0.997
0.999
ImageNet-22K
Swin-I.
6.891
0.051
0.145
2.044
0.076
0.977
0.997
0.999
Semantic-SAM
30
Swin-L.
6.713
0.049
0.137
2.007
0.074
0.979
0.998 1.000
SimMIM
64
Swin-L.
6.542
0.048
0.130
1.941
0.073
0.979
0.998
0.999
Ours
Swin-T
6.991
0.051
0.145
2.016
0.077
0.975
0.997
0.999
Ours
Swin-S 6.736
0.049
0.138
1.981
0.075
0.978
0.997
0.999
Ours
Swin-B 6.598
0.048
0.131
1.939
0.073
0.981
0.997
0.999
Ours
Swin-L. 6.558
0.047
0.129
1.929
0.072
0.981
0.997
0.999
reference points. The sharp boundaries and clear object separation confirm that our features capture consistent, instance-level semantics, even without explicit supervision.
4.3
Monocular Depth Estimation
We evaluate our model on the KITTI dataset
16 using the standard Eigen split [5]. with DCDepth [57] the decoder. As shown in Tab. D our model consistently outperforms both su- pervised ImageNet-22K pretraining and models pretrained on the Semantic-SAM (30), which is weakly supervised method utilizing large-scale pseudo segmentation annotations.
Our approach achieves superior performance across all backbone sizes. For instance, with Swin- Swin-Tiny, model reduces the RMSE to 2.016 (compared to 2.169 from Semantic-SANe SAM) and improves the d1 accuracy to 0.975. These results are even comparable to Swin-Large with ImageNet-222 pretraining. As the backbone scale increases, the performance of method improves steadily.
As shown Tab, combining our features with DINO leads to consistent and significant performance improvements. While our method alone already outperforms using either DINO or ImageNet-22K pretrained features in isolation (rows 1-3), the best result is achieved when concatenating our features with DINO pretrained features (row 6), reaching the lowest SILog (6.796) and competitive RMSE (2.014).
This highlights the complementary nature of the two representations: DINO focuses more on semantic category-Ieve cues, while our method emphasizes instance-level object structure derived from motion cues. Fusing them allows the model to leverage both semantic context and object-centric information, leading to improved depth estimation performance.
Tab. 2 further shows the results on the official KITTI online leaderboard. Our method outperforms other methods in the primary metric (SILog) and also achieves competitive performance across the other evaluation metrics.
4.4 3D Object Detection
We evaluate our learned visual representations on the nuScenes dataset [6 for the 3D object detection task, using BEVFormer V2 [31 66] as the detection framework. We compare our method against diverse set of pretraining strategies. including supervised ImageNet-22K and COCO, as well as self-supervised approaches such as MoCo [10] and SimMIM [64].
As shown in Tab. 3 our approach achieves consistent and substantial improvements in both mean Average Precision (mAP) and NuScenes Detection Score (NDS) across multiple backbones. For instance, with a Swin- Tiny backbone, our model achieves mAP of 43.01% and NDS of 52.41%,

---

Table Quantitative results on the official split of KITTI dataset. All metrics reported here are from the KITTI online leaderboard.
Method
Backbone
Pretrain
SILog
Abs Rel 4
Sq Rel 4
iRMSE $
NeW CRFs
Swin-Large
ImageNet-22K
10.39
8.37
1.83
11.03
VA-DepthNet 32
Swin-Large
ImageNet-22%%
9.63
7.96
1.66
10.44
IEBins 48
Swin-v2-
MIM 665
9.84
7.82
1.60
10.68
NDDepth
47
Swin-v2-Large
MIM 665
9.62
7.75
1.59
10.62
DCDepth
57
Swin-Large
Semantic-SAne
30
9.60
7.83
1.54
10.12
DCDepth 57
Swin-Large
Ours
9.54
7.76
1.55
10.37
Table Quantitative evaluation of BEVFormerV2 666 ing methods.
on nuScenes val set using different pretrain-
Method
Backbone
NDS f mAP f
mATE +
mASE 4
mAOE
mAVE
mAAE
COCO
Res50
51.82
41.99
66.89
28.14
39.15
38.34
19.28
ImageNet-IK
Res50
51.99
42.51
65.90
27.79
42.12
37.70
19.20
MoCo v3 [T]
Res50
52.42
42.94
67.13
27.70
35.84
39.91
19.95
Ours
Res50
52.55
43.22
66.30
27.56
37.76
38.47
20.53
ImageNet-22K
Swin-T
51.69
42.12
67.69
28.07
38.39
40.89
18.68
Ours
Swin-T
52.41
43.01
65.81
28.30
41.43
37.23
18.15
ImageNet-22K
Swin-S
53.62
45.22
64.94
27.75
36.97
40.07
20.18
Ours
Swin-S
54.22
45.49
65.22
27.73
37.61
35.61
19.04
ImageNet-22K
Swin
53.98
45.48
65.94
28.10
35.82
38.75
19.01
SimMIM 64
Swin-B
54.03
45.18
63.81
27.53
38.02
37.04
19.22
Ours
Swin-B B
55.68
47.54
62.74
27.84
33.79
36.77
19.81
ImageNet-22K
Swin-I.
54.59
45.91
65.39
27.44
34.31
37.64
18.87
SimMIM
Swin-L
54.98
46.52
64.80
28.06
33.72
35.87
20.36
Ours
Swin-L 55.80
47.29
62.83
27.16
33.20
36.50
18.77
outperforming ImageNet-22KK pretrained counterpart (mAP 42. 12%, NDS 51.69%). the backbone scales up to Swin-Large, our model further improves to 47.29% mAP and 55.80% NDS, still outperforming the compared supervised and self-supervised methods.
To compare with more methods based on ViT architectures whose computational cost are not affordable for high input resolution, we also tested various methods a resolution of 704 256. As shown in Tab. 4 our Swin-based models achieve competitive superior performance compared to DINOv2, while using significantly fewer parameters and lower computational costs. For instance, our model pretrained with the Swin-L backbone attains an NDS of 52.03% and an mAP of 41.79%. These results are comparable to those achieved by DINOv2 with the ViT-L backbone.
Notably, these improvements are not limited to Transformer-based architectures. With ResNet backbones such as R50, model also outperforms COCO-supervised models, indicating that the benefit of our pretraining is architecture-agnosic This broad compatibility with both convolutional and Transformer backbones highlights the generality of the learned features.
These gains can be attributed to the object-centric and geometry-aware priors introduced by our object-based visual representation. Unlike traditional supervised pretraining, our approach enables the model to internalize compositional structure and spatial relationships between objects. This proves partieularly valuable in 3D detection tasks, where reasoning about object placement, extent, and occlusion is critical.
4.5
3D Occupancy Perception
We evaluate our method on the nuScenes validation set using SparseOce [33 as the occupancy prediction framework. As shown in Tab. 6 our pre-trained models outperform both supervised (mmageNet222K) and self-supervised (SimMIM) counterparts across all Swin backbone variants. Crucially, the strong performance of our method can be attributed to the underlying geometric insight described in S.1 By leveraging this property, our method is able encode spatial structures that semantically meaningful, even without direct instance-level annotations.

---

Table 4: Quantitative evaluation of BEVFormerV2 666 on nuScenes val set using different pretrain- ing methods at a resolution of 704 x 256
Method
Backbone
NDS t mAP1 f
mATE +
mASE +
mAOE
mAVE
mAAE 4
DINOv2
VIT-S
46.24
34.88
71.65
28.45
49.97
42.70
18.84
DINOv2
ViT-B
49.08
38.36
69.74
28.21
41.81
42.40
18.84
DINOv2
VIT-L
51.91
42.05
65.04
27.35
36.45
43.79
18,51
ImageNet-22K
Swin-T
47.42
36.34
70.90
28.40
48.36
40,47
19.40
Ours
Swin-T
48.24
37.08
70.61
27.99
44.45
40.54
19.45
ImageNet-22K
Swin-S
48.78
38.00
70.65
28.35
41.20
42.95
19.01
Ours
Swin-S
50.87
40.23
68.88
27.85
40.45
36.67
18.61
ImageNet-22K
Swin-B
50.42
40.71
68.56
27.80
40.60
42.81
19.52
Ours
Swin-B
51.69
41.36
66.01
28.03
37.98
39.73
18.10
ImageNet-22K
Swin-L
50.48
40.09
68.01
27.90
40.69
40.67
18.38
Ours
Swin-I.
52.03
41.79
66.10
28.12
36.84
40.13
17.51
Table 6: Quantitative evaluation of SparseOcc (33 on nuScenes set using different pretraining.
Table 5: Ablation studies the KITTI depth estimation task. We evaluate the impact of different pretraining strategies: DINO refers to DINOv2 [41], and IN denotes ImageNet-
Method
BB
RayloU
RayloU Im. 2m, 4m
MoCo v3 100
R50
34.4
28.3
35.1
39.9
ImageNet-IK
R50
35.0
28.8
35.6
40.5
22K [33 supervised pretraining. Ours and IN use Swin-T as the backbone, while DINO
Ours
R50
36.4
30.2
37.1
ImageNet-22K
Sw-T
35.5
29.4
36.3
40.9
uses ViT-S.
Ours
Sw-T
37.0
31.1
37.8
42.2
DINOv2 [4
VIT-S
35.9
29.5
36.8
41.4
ImageNet-22K
Sw-S
36.8
30.4
37.6
42.3
Ours DINO IN SILog
AbsRel
RMSE
Ours
Sw-S
38.1
32.0
39.1
43.4
6.991
0.051
2.016
DINOv2 41
VIT-B
37.1
31.0
37.9
42.4
7.119
0.052
2.153
ImageNet-22K
Sw-B
37.6
38.4
43.1
7,455
0.055
2.182
SimMIM
64
Sw-B
38.0
31.7
38.7
43.4
7,071
0.052
2.117
Ours
Sw-B B
38.3
32.1
39.1
43.7
6.927
0.050
2.009
DINOv2 41
VIT-L
39.0
32.8
39.9
44.3
6.796
0.050
2.014
ImageNet-225
Sw-L
37.6
31.4
38.4
43.0
SimMIM
64
Sw-L
38.6
32.6
39.4
43.7
Ours
Sw-L
38.7
32.6
39.5
43.8
Discussion and Future Work
5.1 Generalization to out of domain scenes
As preliminary investigation, we train our method on outdoor driving videos. To demonstrate the generalization of our learned features, we also visualize the features on daily life videos from the Ego4D 17 and the robot manipulation dataset RTX 42, ]. shown in Fig. 5 Though not perfect, our model is capable of distinguishing different objects that did not appear in the training set. The first row shows some common scenes in indoor scenes. Our method can segment the unseen objects, including windows, tools, even cats, and hands. We observe similar results the second row for robot manipulation. These results illustrate that our method does not overfit the objects in the training set, but indeed learns the essential composition of an object.
5.2 Further scaling up and extensions
We also attempted to apply method to broader scenarios, such ego-centric videos and uncon- strained videos from the web. However, we found that the performance of our method is greatly limited by the performance of optical flow. Fortunately,
find that recent closely related work in monocular depth estimation 668 69 improves significantly with the help of large-scale synthetic data. We hope similar paradigm can also benefit the performance of optical flow.
Our method can be further extended representation, we can easily endow the model with temporal prediction ability. It is essentially world model that could predict the dynamics of the world. We will pursue these directions in our future work.
temporal setting. Based on the compact object instance

---

Figure 5: Examples of feature maps in out of domain scenes.
5.3 Improving precise localization ability
As seen in the visualization, method focuses the whole object, which means the features of the parts within an object are indistinguishable. This makes our method unsuitable for applications that need precise localization, such keypoint matching. Our method can be improved by combining previous works that emphasizes local feature learning, such as CroCo 63]. to get the best of two worlds.
Conclusions
In this work, we present a biologically inspired framework for learning object-centric visual rep- resentations, drawing motivation from developmental psychology studies on how infants acquire the concept of objects through motion cues, By leveraging the natural correlation between motion boundaries and object boundaries, our method derives instance-level pseudo labels from raw videos, enabling unsupervised representation learning without human annotations or camera calibration.
Through extensive experiments across three diverse vision tasks, demonstrate that approach not only matches but surpasses the supervised and self-supervised pretraining baselines. Our learned features capture object-level semanties that complementary those in existing vision foundation models such as DINO and MAF
These results highlight the potential of integrating biologically inspired mechanisms-such as motion- guided grouping- into the design of scalable, general-purposs visual pretraining frameworks. We hope this work encourages further exploration of cognitive principles in building more robust and luman-aligned vision systems.
References
[1] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. CVPR, 2023.
[2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BETT: BETR pre-training of image transformers. In ICLR, 2022.
[3] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. TMLR, 2024.
[4] Jiawang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming- Ming Cheng, and Ian Reid. Unsupervised scale-consistent depth and ego-motion learning from monocular video. In NewrIPS, 2019.
[5] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerehner. MONet: Unsupervised seene decomposition and representation. arXiv preprint arXiv:190m 3900, 2019.
[6] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu. Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
10

---

[7] Mathilde Caron. Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, In ICCV, 2021.
[8] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[9] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint 20Xiv:2003304297,, 2020.
[10] Xinlci Chen, Saining Xie, and Kaiming Hc. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
[11]
MrPieTrain Contributors. OpenMMLab's pre- pre-training toolbox and benchmark. https: /ct. open mm1 ab/mpretraaaaannnaaeaeaaaaaaaaat 2023.
[12] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR, 2024.
[13] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hicrarchical image database. In CVPR, 2009.
[14] Dehidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In ICCV, 2021.
[15] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In NeurlPS, 2014.
[16] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
[17] Kristen Grauman, Andrew Westhury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamhurger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
[18] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In ICML, 2019.
[19] Jean-Baste Man-BBatienn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to sell-supervised learning. In NeurlPS, 2020.
[20] K. Tan et al. H. Caesar, J. Kabzan. NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles. In CVPR ADP3 workshop, 2021.
[21] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[22] Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[23] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[24] Scott Johnson and Susan Johnson. Development of perceptual completion originates early infancy. Psychological Science, 14(6):553 559, 2003.
[25] Armand Joulin, Francis Bach, and Jean Ponce. Discriminative clustering for image co-segmentation. In CVPR, 2010.
[26] Armand Joulin, Francis Bach, and Jean Ponce. Multi-class cosegmentation. In CVPR, 2012.
[27] Philip J Kellman and Elizabeth S Spelke. Perception of object unity in young infants. Infant Behavior and Development, (1))1111-,,0 1983.
[28] Alexander Kirillov. Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019.
[29] Dwight Kravitz, Kadharbatcha S Saleem, Chris Baker, and Mortimer Mishkin. A new neural framework for visuospatial processing. Nature Reviews Neuroscience, 12(4):217-230, 2011.
11

---

[30] Feng Li. Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, Lei Zhang, and Jianfeng Gao. Segment and recognize anything any granularity. In ECCV, 2024.
[31] Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. BEVForee:: learning bird's seyee--eee representation from LiDAR- camera via spatiotemporal transformers. TPAMI, 47(3):2020-255 2036. 2024.
[32] Ce Liu, Suryansh Kumar, Shuhang Gu, Radu Timofte, and Luc Van Gool. VA-DepthNet: A variational approach to single image depth prediction. ICLR, 2023.
[33] Haisong Liu, Yang Chen. Haiguang Wang, Zetong Yang. Tianyu Li, Jia Zeng, Li Chen, Hongyang Li, and Limin Wang. Fully sparse 3D occupancy prediction. In ECCV, 2024.
[34] Runtao Liu, Zhirong Wu, Stella Yu, and Stephen Lin. The emergence of objectness: Learning zero-shot segmentation from videos. In NewrlPS, 2021.
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[36] Francesco Locatello, Dirk Weissenbom, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakoh Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurlPS, 2020.
[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019.
[38] Rundong Luo, Hong-Xing Yu, and Jiajun Wu. Unsupervised discovery of object-centric neural fields. arXiv preprint
2024.
[39] David Marr. Vision: Computational Investigation into the Human Representation and Processing of Visual Information. 1982.
[40] Amy Needham. Object exploration and object knowledge in young infants: A view from developmental psychology. Cognition, Brain, and Consciousness, 2001.
[41] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaekdin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. TML.R, 2023.
[42] Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open X-embodiment: Robotic learning datasets and RT-X models. In ICRA, 2024.
[43] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, per-peerformanc deep learning library. NewrIPS, 2019.
[44] Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BETT v2: Masked image modeling with vector-guantized visual tokenizers. arXiv preprint arXiv:2208.06.80 06366, 2022.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML. 2021.
[46] Mohammadreza Salchi, Efstratios Gavves, Coes GM Snock, and Yuki M Asano. Time does tell: Self- supervised time-tuning of dense image representations. In ICCV, 2023.
[47] Shuwei Shao, Zhongeai Pei, Weihai Chen, Xingming Wu, and Zhengguo Li. NDDepth: Normal-distanee assisted monocular depth estimation. In ICCV, 2023.
[48] Shuwei Shao, Zhongeai Pei, Xingming Wu, Zhong Liu, Weihai Chen, and Zhengguo Li. IEBins: Iterative clastic bins for monocular depth estimation. In NeurlPS, 2023.
[49] Xiaoyu Shi. Zhaoyang Huang, Weikang Bian, Dasong Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. VideoFlow: Exploiting temporal cues for multi- optical flow estimation. In ICCV, 2023.
[50] Oriane Siméoni, Gilles Puy, Huy V Vo. Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self supervised transformers and no labels. In BMVC, 2021.
12

---

[51] Silky Singh, Shripad Deshmukh, Mausoom Sarkar, and Balaji Krishnamurthy. Locate: self-supervised object discovery via flow-guided graph-cut and bootstrapped self-training. In BMVC, 2023.
[52] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. IJC,, 104:154-117 171, 2013.
[53] Lestie G Ungerleider and Mortimer Mishkin. Two cortical visual systems. Analysis of Visual Behavior, 1982.
[54] Sara Vicente, Carsten Rother, and Vladimir Kolmogorov, Object cosegmentation. In CVPR, 2011.
[55] Huy Vo, Francis Bach, Minsu Cho, Kai Han, Yann LeCun, Patrick Pérez, and Jean Ponce. Unsupervised image matching and object discovery as optimization. In CVPR, 2019.
[56] Van Huy Vo, Elena Sizikova, Cordelia Schmid, Patrick Pérez, and Jean Ponce. Large-scale unsupervised object discovery. In NeurlPS, 2021.
[57] Kun Wang, Zhiqiang Yan, Junkai Fan, Wanlu Zhu, Xiang Li. Jun Li, and Jian Yang. DCDepth: Progressive monocular depth estimation in discrete cosine domain. In Neur/PS, 2024.
[58] Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorek. Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as foreign language: BEiT pretraining for vision and vision- lanounla tasks. In CVPR, 2023.
[59] Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L Crowley, and Dominique Vaufreydaz. Self- supervised transformers for unsupervised object discovery using normalized cut. In CVPR, 2022.
[60] Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley, and Dominique Vaufreydaz. TokenCut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. TPAMI, 45(12):15790-1558,, 2023.
[61] Yugi Wang, Yuntao Chen, and Zhao-Xiang Zhang. 4D unsupervised object discovery. In NeurlPS, 2022.
[62] Xiu-Shen Wei, Chen-Lin Zhang, Jianxin Wu, Chunhua Shen, and Zhi-Hua Zhou. Unsupervised object discovery and lo-ooaalization by deep descriptor transformation. PR. 88:113- 126, 2019.
[63] Philippe Weinzaepfel. Vincent Leroy. Thomas Lucas, Romain Brégier. Yohann Cabon, Vaibhav Arora. Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. CroCo: Self-supervised pre-training for 3D vision tasks by cross-view completion. In NeurlPS, 2022.
[64] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. SimMIM: A simple framework for masked image modeling. In CVPR, 2022.
[65] Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. In CVPR, 2023.
[66] Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. BEVFormer v2: Adapting modern image backbones to bird's seey-- view recognition via perspective supervision. In CVPR, 2023.
[67] Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, Jun Zhang, Andreas Geiger, Yu Qiao, and Hongyang Li. Generalized predictive model for autonomous driving. In CVPR, 2024.
[68] Lihe Yang, Bingyi Kang, Zilong Huang, Xiangang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlaheled data. In CVPR, 2024.
[69] Lihe Yang. Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. NeurlPS, 2024.
[70] Chun-Hsiao Yeh, Cheng- Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yuhei Chen, and Yann LeCun. Decoupled contrastive learning. In ECCV, 2022.
[71] Weihao Yuan, Xiaodong Gu. Zuozhuo Dai, Siyu Zhu, and Ping Tan. Neural window fully-connected CRFs for monocular depth estimation. In CVPR, 2022.
[72] Xiao Zhang and Michael Maire. Self-supervised visual representation learning from hierarchical grouping. In NewrlPS, 2020.
13

---

[73] Xin Zhang, Jinheng Xie, Yuan Yuan, Michael Bi Mi, and Robby T Tan. HEAP: unsupervised object discovery and localization with contrastive grouping. In AAAI, 2024.
[74] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, 2017.
[75] C Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges. In ECCV, 2014.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the contributions, and the experi-
ments in Sec. 4 are conducted support these contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work are discussed in Sec. 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Ys]]
Justification: All details are specified in Sec. and Sec. Pseudo-code is provided in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: The code and data associated with this paper will be made publicly available the link provided in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https i
//nips. //nips.c//pul wwtppbllcggguueeiocccce
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and test details are specified in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We report evaluation metrics following prior work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All training and test details are specified in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification:
Our research followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and related prior work are properly cited in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets at this time.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Pseudo-codes for Pixel Cluster
For all optical flow data generated by VideoFlow, we perform a simple Breadth-First Search(BFS) to segment moving objects. Alg. Alg.-providie takes the optical flow, the provides pseudocode description of our algorithm. The algorithm consistency check result, and two thresholds fy and 8, as input. y is used to determine when the optical flow of two adjacent pixels, being sufficiently close, is considered to belong to the same object. 9g controls the minimum number of pixels that an object should have.
Algorithm 1 Pixel Cluster
Input: flow(optica flow), valid(ccnsistency check), 0 0, 1: Initialization:: t- 0, [[i]b]] + false, +-0
2: for <- to H do
3: for / + 1 to W do
4: if [[>][y]] = true valid [ false then
5: continue
6: end if
7:
f empyy queue, C + 0
8: Enqueue(Q,, (z, y))
9:
while Q * do
(x,y) Dequeue(())
11:
C B-CCUE (cx(()))
for (i. j) in (x, y)'s 4 neighbors do
13:
if ||flow [i] flow [ow[=]|blll2 S Oy and r[]][i] false and valid[z] [>]]]]
= true then
u|i][i] true
15:
Enqueue(Q. (i.5))
end if end for end while
17:
18:
19:
if C > 0g then
20:
S 5 SUSSUE (C)
21:
end if end for
22:
23:
end for
Output: S
B Data Augmentation Details
All input images are first randomly resized to a resolution between 512 x 288 and 1024 576. They then randomly cropped 224 x 224. During cropping, up 10 attempts made ensure that the cropped region contains at least two distinct labels. Afterward, each image has a 50% chance of being horizontally flipped. Additionally, gamma, brightness, and color augmentations applied with a 50% probability, each sampled within the range of (0.9, 1.1).
C More Qualitative Results
Fig. shows additional qualitative results of the pseudo-label generation and the visualizations of the output features. As illustrated in the pseudo-label visualizations, the proposed algorithm successfully segments objects exhibiting significant movement, well foreground instances exhibiting motion patterns distinct from the background. The feature visualizations shows that the model distinguishes many objects annotated in the pseudo-labels. This suggests model goes beyond mimicking pseudo-labels, but learning more general, object-centric representation.
22

---

FOR
Input
Flow
Label
Feature
Figure 6: Examples of the pseudo-label generation results and the output features.
23


Reviews:
[{"review_id": "kKAAKpO0x9", "reviewer": "Reviewer_h9dF", "summary": "The authors introduce a self-supervised, instance-aware framework for learning dense visual representations. Their supervision signal is derived from pseudo-segmentation maps obtained by clustering optical flow. An encoder is trained using a triplet ranking loss to produce dense feature maps where pixel-wise similarities align with the inferred region maps. For evaluation, the authors present region-similarity visualizations and quantitative benchmarks on depth estimation and 3D object detection, demonstrating that their approach outperforms contrastive-based self-supervised learning methods.", "questions": "I have questions about the robustness of the proposed methods for different flow estimations and their performance on the standard benchmark. Please see the weakness for details.", "limitations": "See weakness.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n(1) Learning instance-level representations in a self-supervised manner remains a challenging task. The authors present a simple yet effective framework that outperforms state-of-the-art contrastive learning approaches on selected benchmarks. \n(2) Visualizations of the learned pixel-wise affinity maps indicate that the model captures meaningful instance-level embeddings.\n(3) Good paper presentation.\n\nWeakness:\n(1) The method relies on externally estimated optical flow as input, but the authors do not include ablation studies across different flow estimators to assess the robustness of their approach.\n(2) While the method is evaluated on depth estimation and 3D object detection benchmarks, it lacks several key metrics: specifically, evaluations of the learned feature maps on semantic and instance segmentation tasks via fine-tuning, linear probing, or unsupervised clustering. Comparisons to baseline methods, such as DINO, on image classification tasks are also missing.\n(3) The backbone used in the proposed method differs from those in baseline approaches (e.g., DINO), as it is designed to produce higher-resolution feature maps. This architectural difference raises questions about whether performance gains in high-resolution tasks like depth estimation are due to the proposed method or simply the model architecture. For instance, could a DINO model with a similar FPN-style decoder yield comparable results?\n(4) The paper omits comparisons with relevant prior work that also targets instance-aware embeddings [1] or learns objectness from video signals [2].\n\n[1] Zhang et al., Self-Supervised Visual Representation Learning from Hierarchical Grouping\n[2] Liu et al., The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos", "was_revised": true, "final_justification": "After reading the author's and other reviewers' comments, I kept my rating to weak acceptance. Learning instance-wise representation without any human supervision is a very challenging task, and the authors propose a simple and effective framework. I wish there will be following-up work to test this approach at larger scale and provide more extensive benchmark.", "rebuttal": "Dear Reviewer h9dF,  \nWe greatly appreciate your valuable comments and questions. Our responses are as follows.\n\n***Flow models***\n\nWe chose VideoFlow specifically because it performs joint multi-frame estimation, which provides better temporal consistency compared to two-frame-based models such as RAFT. This is particularly beneficial for long-term motion tracking in videos. \n\nHowever, our method is not limited to a specific optical flow model — it can benefit from improvements in flow estimation. For example, we are actively exploring more robust approaches like CoTracker \\[3\\], which offer enhanced pixel-level tracking across time.\n\n***Evaluations on other tasks***  \nThe following table presents the results of semantic segmentation using UperNet, 2D detection and instance segmentation using Mask R-CNN on the nuImages dataset.\n\n| Method | Backbone | mAP(detection bbox) | mAP(instance seg) | mIoU(semantic seg) |\n| :---- | :---- | :---- | :---- | :---- |\n| ImageNet | Swin-T | 53.1 | 44.3 | 85.74 |\n| Ours | Swin-T | 53.7 | 44.7 | 83.27 |\n| ImageNet | Swin-S | 53.9 | 44.8 | 87.16 |\n| Ours | Swin-S | 54.8 | 45.5 | 84.82 |\n| ImageNet | Swin-B | 54.4 | 45.1 | 87.51 |\n| Ours | Swin-B | 55.0 | 45.8 | 85.55 |\n| ImageNet | Swin-L | 54.7 | 45.3 | 88.49 |\n| Ours | Swin-L | 55.2 | 45.5 | 85.69 |\n\nWe conducted a linear probe evaluation on Cityscapes dataset with ResNet-50 as backbone. \n\n| Method | mIoU |\n| :---- | :---- |\n| ImageNet | 44.14 |\n| MoCo v3 | 46.25 |\n| Ours | 33.33 |\n| Ours \\+ ImageNet | 48.40 |\n| Ours \\+ MoCo v3 | 49.50 |\n\nAs shown in the table, our method achieves comparable performance on 2D detection and instance segmentation tasks, but also shows noticeable shortcomings in semantic segmentation. This is because the features learned by our model are instance-discriminative rather than category-discriminative. However, as demonstrated in paper’s Table 3 and the table above, our features are complementary to class-level features. \n\n***The architectural difference***  \nAs described in lines 233–234, although our method is pretrained with an FPN, during fine-tuning on downstream tasks, only the pretrained weights of the backbone are used. This is because our pretraining task involves pixel-level supervision, which benefits from multi-scale features.  We also conducted additional experiments at lower resolutions, as shown in Table 2 of the supplementary material. The results demonstrate that our method remains effective even at lower resolutions.\n\n***Comparisons with relevant works***  \nWe found it quite difficult to make fair comparisons with other relevant works under our experimental setup. Many works do not open-source their code \\[1\\] or fail to provide downloadable pretrained weights \\[2\\]. Nevertheless, these works remain valuable. We will include them in the discussion section of the paper and conduct comparisons once the official code is available.  \nWe tested a video self-supervised learning model TimeT\\[4\\] on depth estimation and BevFormer v2(704 × 256 resolution). The official TimeT GitHub repository only offers the ViT-S model. The following table shows the comparison results for the Small model.\n\n|  |  | BEVFormer v2 |  | Depth Estimation |  |  |  |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| **Method** | **Backbone** | **mAP ↑** | **NDS ↑** | **SILog ↓** | **Abs Rel ↓** | **Sq Rel ↓** | **RMSE ↓** |\n| DINOv2 | ViT-S | 46.24 | 34.88 | 7.119 |  0.052 | 0.158 | 2.153 |\n| TimeT | ViT-S | 41.02 | 28.63 | 8.551 | 0.062 | 0.219 | 2.484 |\n| Ours | Swin-S | 50.87 | 40.23 | 6.736 | 0.049 | 0.138 | 1.981 |\n\nAs shown in the table, since the TimeT model was not designed or trained for large-scale pretraining, it exhibits a significant gap compared to existing methods on both tasks.  \n\n\\[1\\] Zhang et al., Self-Supervised Visual Representation Learning from Hierarchical Grouping\\. NeurIPS 2020\n\n\\[2\\] Liu et al., The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos\\. NeurIPS 2021\n\n\\[3\\] Karaev, Nikita, et al. Cotracker: It is better to track together. ECCV 2024.\n\n\\[4\\] Salehi, Mohammadreza, et al. Time does tell: Self-supervised time-tuning of dense image representations. ICCV. 2023."}, {"review_id": "4Szobsn8P5", "reviewer": "Reviewer_6akB", "summary": "The authors propose a framework to learn object-centric visual features from motion inspired by findings from developmental neuro-science. An off-the-shelf flow model is used to detect motion in videos. The flow is then clustered to generate object pseudo-labels. The labeled clusters are then used to train an encoder using contrastive loss.  The learnt features are evaluated on the tasks of depth-estimation, 3D object detection and occupancy prediction. State-of-the-art performance is demonstrated with respect to several relevant baselines.", "questions": "1. The clustering is done based on flow information. What if the data contains video with low motion/ or videos of scenes with no specific objects, how do instances like these affect the clustering?\n2. Is there a curation process that the data has to go through to be able to generate reasonable clusters.", "limitations": "The limitations of the framework have been adequately discussed", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths:\n1. **Label free method**: The proposed approach is label-free and does not require additional annotations like camera parameters. This is particularly useful to extract structured information from a large collection of videos.\n2. **Quantitative results**: The approach outperforms DINOv2 and other relevant baselines on 3D tasks.   \n\n\n## Weaknesses:\n1. **Novelty**: The core idea of using motion to learn objectness has been explored before (points on a object have 'common fate'). The approach uses an off the shelf flow estimation model \n2. **Dependence on optical flow estimator**: The base of the approach relies on clustering the motion representation obtained from an off-the-shelf optical flow estimator thus inherits all the limitation of the chosen base model. In particular, optical flow models fail to do well on flat textured/ untextured regions of a video. Since the clustering and training starts from this step, is there a tendency for the errors to cascade across the stages? \n3. **Qualitative examples**: Providing more qualitative results would be beneficial to help appreciate the effectiveness of the approach.\n4. **Writing**: There are number of typos and language errors throughout the manuscript. The paper would benefit from a thorough proof reading and streamlining of certain sections.", "was_revised": true, "final_justification": "The rebuttal satisfactorily answers most concerns raised and the additional results help strengthen the claims of the paper.", "rebuttal": "Dear Reviewer 6akB,  \nWe greatly appreciate your valuable comments and questions. Our responses are as follows.\n\n***Novelty***  \nOur idea builds upon numerous prior works on object discovery, and we did not intend to claim this as a novel insight. Our main contribution lies in demonstrating the effectiveness and scalability of this idea on a large-scale dataset and achieving comparable performance to foundation models such as DINO on downstream tasks.  \nWe will revise some sentence in our paper, for example:  \n1. \"Our key insight is that motion boundary serves as a strong signal for object-level grouping.\" will be revised to “we were inspired by the insight that motion boundary serves as a strong signal for object-level grouping, which can be used to derive pseudo instance supervision from raw videos.”\n2. \"we are the first to leverage motion as an unsupervised supervisory signal to guide the emergence of object-level semantics in visual representations.\" We will revise it to “We are first to demonstrate the effectiveness and scalability of using motion as an unsupervised supervisory signal on a large-scale dataset and modern model architectures”\n\n***Is there a curation process that the data has to go through to be able to generate reasonable clusters?***\n\nSince the nuPlan dataset provides camera distortion parameters, we applied undistortion to all images when processing the nuPlan data. Apart from this, all procedures are exactly as described in the paper(Lines 173–191) and Section A of the supplementary material, with no additional processing involved. We retained all samples with at least two pseudo-label(i.e. at least one foreground cluster) since samples that cluster all pixels into a single group have a loss of 0 and are not useful for training.\n\n***The clustering is done based on flow information. What if the data contains video with low motion/ or videos of scenes with no specific objects, how do instances like these affect the clustering?***\n\nThe video contains low motion or no objects which failed to cluster instances will be filtered out during preprocessing, as described in Lines 189–191. Given the large scale of the dataset, the model still learns generalizable representations, even with these less informative frames removed.\n\n***Qualitative examples***  \nAdditional Qualitative results are provided in the supplementary material. Unfortunately, due to system limitations, we are unable to include more during the rebuttal phase.\n\n***Writing***  \nWe appreciate your suggestions. We will continue to improve our paper and address these writing errors."}, {"review_id": "DrVoWa9ge3", "reviewer": "Reviewer_9fwV", "summary": "The paper proposes to learn semantically separated features from motion signals. The method uses an optical flow model to predict the flow for a large set of driving scenes. This flow is then used in a clustering step to create pseudo-labels for all videos. A network is trained on those pseudo-labels and evaluated on various tasks, such as Depth Estimation, BEV Prediction, and Occlusion Prediction. The results show the method can be effective for these tasks.", "questions": "While I think the idea proposed by this paper is valuable, there are various aspects that need to be addressed.\n- Extension beyond just driving scenes: How does the method behave when trained on other natural scenes?\n- How does the approach behave under different parameter combinations & optical flow models (see above)?\n- How does it compare to other video SSL approaches and other DINOv2 sizes?\n\nIf the authors can address my concerns, I'm willing to update my score.", "limitations": "Barely, but yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The idea of exploiting coherent motion as as representation of objectness is very interesting and appealing\n- The fact that the method is designed to work on a large unlabeled data pool of videos makes it scalable\n- The presented quantitative results are somewhat consistent and improve upon the SOTA\n- The qualitative results are nicely presented and show the effectiveness of the pseudo-labeling\n\nWeaknesses:\n- The choice of datasets seems to be restricted to driving videos. I wonder why this choice was made since the method could also work on other types of natural videos? This choice fails to fully exploit the potential of the proposed method\n- The proposed method is rather simple and seems a bit handcrafted. However, it is somewhat effective, so this is not critical.\n- Lack of ablations: In the main paper, the method’s components and parameter choices are barely ablated. I think this is crucial for the paper. There would be a lot of value in seeing how effective the pseudo-labeling is under different parameter choices like the manually chosen thresholds in L183, different optical flow models like RAFT [1] or SEA-RAFT [2], different choices for the margin parameter L209, and so on\n- More model comparisons: It would be great to also see other video self-supervised models compared, for example [3]. Furthermore, DINOv2 was only compared with a ViT-S, while the proposed method uses up to a Swin-L. It would be nice to also see other sizes for DINOv2.\n\n[1] Teed, Zachary, and Jia Deng. \"Raft: Recurrent all-pairs field transforms for optical flow.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020.\n[2] Wang, Yihan, Lahav Lipson, and Jia Deng. \"Sea-raft: Simple, efficient, accurate raft for optical flow.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n[3] Salehi, Mohammadreza, et al. \"Time does tell: Self-supervised time-tuning of dense image representations.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "was_revised": true, "final_justification": "While the authors did address some of my concerns convincingly, I remain in doubt about how general the approach can be and what the effect of the optical flow model is on the overall approach. I will keep my score of borderline reject.\n\nOverall, the paper is on the right track, but needs some more improvements in the cited aspects.", "rebuttal": "Dear Reviewer 9fwV,  \nWe greatly appreciate your valuable comments and questions. Our responses are as follows.\n\n***Extension beyond just driving scenes: How does the method behave when trained on other natural scenes?***\n\nWe use driving data because current optical flow models perform better in such environments. In real-world datasets like Ego4D, where high-quality optical flow data is lacking, flow estimation remains unreliable. However, as discussed in paper's Sections 5.1 and 5.2, our method shows reasonable generalization to natural scenes.\n\n***How does the approach behave under different parameter combinations & optical flow models?***\n\nThe following table shows the depth estimation results of the Swin-T model after 200 epochs of pretraining with different values of $\\\\theta\\_f$​. \n\n| $\\\\theta\\_f$ | SILog ↓ | Abs Rel ↓ | Sq Rel ↓ | RMSE ↓ | RMSE log ↓ | δ1 ↑ | δ2 ↑ | δ3 ↑ |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| 0.5 | 7.069 | 0.051 | 0.146 | 2.050 | 0.078 | 0.974 | 0.997 | 0.999 |\n| 1.5 | 7.022 | 0.051 | 0.146 | 2.051 | 0.079 | 0.974 | 0.997 | 0.999 |\n\nHere, we present the results of some downstream tasks with the margin set to 0.5, 1.0, and 1.5, trained for 100 epochs on one-quarter of the dataset using Swin-T as backbone.\n\n|  | BEVFormer v2 |  | Depth Estimation |  |  |  |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| **Margin** | **mAP ↑** | **NDS ↑** | **SILog ↓** | **Abs Rel ↓** | **Sq Rel ↓** | **RMSE ↓** |\n| 0.5 | 42.41 | 51.87 | 7.329 | 0.54 | 0.157 | 2.079 |\n| 1.0 | 42.81 | 51.72 | 7.284 | 0.53 | 0.153 | 2.090 |\n| 1.5 | 42.54 | 52.23 | 7.244 | 0.53 | 0.154 | 2.086 |\n\nAccording to the ablation study, our model demonstrates robustness to hyperparameter variations when trained on large-scale datasets, except in extreme cases—such as setting the margin to 0, which indicates infinite clusters.\n\n---\n\nWe chose VideoFlow specifically because it performs joint multi-frame estimation, which provides better temporal consistency compared to two-frame-based models such as RAFT. This is particularly beneficial for long-term motion tracking in videos. \n\nHowever, our method is not limited to a specific optical flow model — it can benefit from improvements in flow estimation. For example, we are actively exploring more robust approaches like CoTracker \\[4\\], which offer enhanced pixel-level tracking across time.\n\n***How does it compare to other video SSL approaches and other DINOv2 sizes?***  \nAdditional results on DINOv2 are provided in the supplementary materials.\n\nWe tested the TimeT\\[3\\] model on depth estimation and BevFormer v2(704 × 256 resolution). The official TimeT GitHub repository only offers the ViT-S model. The following table shows the comparison results for the Small model.\n\n|  |  | BEVFormer v2 |  | Depth Estimation |  |  |  |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| **Method** | **Backbone** | **mAP ↑** | **NDS ↑** | **SILog ↓** | **Abs Rel ↓** | **Sq Rel ↓** | **RMSE ↓** |\n| DINOv2 | ViT-S | 46.24 | 34.88 | 7.119 |  0.052 | 0.158 | 2.153 |\n| TimeT | ViT-S | 41.02 | 28.63 | 8.551 | 0.062 | 0.219 | 2.484 |\n| Ours | Swin-S | 50.87 | 40.23 | 6.736 | 0.049 | 0.138 | 1.981 |\n\nAs shown in the table, since the TimeT model was not designed or trained for large-scale pretraining, it exhibits a significant gap compared to existing methods on both tasks.  Nevertheless, these works remain valuable, and we will include a discussion of them in the paper.\n\n\\[4\\] Karaev, Nikita, et al. Cotracker: It is better to track together. ECCV 2024."}, {"review_id": "9Xaax9iwRc", "reviewer": "Reviewer_dAst", "summary": "This paper proposes a motion-induced object-centric pretraining method for vision foundation models. The proposed method uses an optical flow model to extract motion clues from raw videos, and derive object pseudo labels through clustering on optical flow map. Then a simple contrastive learning objective is proposed to learn the concept of object from the annotated images. Extensive experiments on depth estimation, 3D object detection and 3D occupancy prediction validate the effectiveness of the proposed method.", "questions": "1. Could the authors explain the similar performance with SimMIM in depth estimation and occupancy prediction?\n2. Could the authors provide the results for SimMIM under smaller model sizes for more comprehensive comparison?\n3. How does the method perform in other 2D downstream tasks, such as 2D semantic/instance segmentation?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Good motivation. The proposed method originates from the biological perspective and aligns with the development of human cognition.\n2. Simple yet effective method. The proposed method consists of a simple contrastive loss, but is verified to surpass other advanced pretraining methods.\n3. Extensive experiments. The paper conducts extensive experiments on three downstream tasks, on all of which the proposed method demonstrate good performance.\n\nWeaknesses:\n1. The performance of the proposed method is very close to SimMIM under large model parameters.\n2. The types of downstream tasks is limited, all of which is closely related to 3D perception of the environment. According to the math derivation, the pseudo labels inherently correlate with the depth of the image, which might render 3D-related downstream tasks unfair for other pretraining methods.", "was_revised": true, "final_justification": "Post rebuttal, the additional experiments have resolved my concerns about downstream tasks and performance. Thus I am leaning to accept the paper.", "rebuttal": "Dear Reviewer dAst,  \nWe greatly appreciate your valuable comments and questions. Our responses are as follows.\n\n***Questions about SimMIM***  \nFor all methods compared in our paper, we utilize the official pretrained weights. Unfortunately, SimMIM does not provide models smaller than the Base version. Here we report results for the Base version of SimMIM on depth estimation.\n\n| Method | Backbone | SILog ↓ | Abs Rel ↓ | Sq Rel ↓ | RMSE ↓ | RMSE log ↓ | δ1 ↑ | δ2 ↑ | δ3 ↑ |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| SimMIM | Swin-B | 6.835 | 0.050 | 0.142 | 2.034 | 0.076 | 0.975 | 0.997 | 0.999 |\n| SimMIM  | Swin-L | 6.542 | 0.048 | 0.130 | 1.941 | 0.073 | 0.979 | 0.998 | 0.999 |\n| Ours | Swin-B | 6.598 | 0.048 | 0.131 | 1.939 | 0.073 | 0.981 | 0.997 | 0.999 |\n| Ours | Swin-L | 6.558 | 0.047 | 0.129 | 1.929 | 0.072 | 0.981 | 0.997 | 0.999 |\n\nSimMIM adopts a dense pretraining strategy, which shows advantages on tasks like depth estimation and occupancy prediction. However, on 3D object detection, our model significantly outperforms SimMIM.\n\nOur model leverages all available autonomous driving data. We believe that access to more diverse datasets could further improve its performance and generalization ability.\n\n***How does the method perform in other 2D downstream tasks, such as 2D semantic/instance segmentation?***  \nThe following table presents the results of semantic segmentation using UperNet, 2D detection and instance segmentation using Mask R-CNN on the nuImages dataset.\n\n| Method | Backbone | mAP(detection bbox) | mAP(instance seg) | mIoU(semantic seg) |\n| :---- | :---- | :---- | :---- | :---- |\n| ImageNet | Swin-T | 53.1 | 44.3 | 85.74 |\n| Ours | Swin-T | 53.7 | 44.7 | 83.27 |\n| ImageNet | Swin-S | 53.9 | 44.8 | 87.16 |\n| Ours | Swin-S | 54.8 | 45.5 | 84.82 |\n| ImageNet | Swin-B | 54.4 | 45.1 | 87.51 |\n| Ours | Swin-B | 55.0 | 45.8 | 85.55 |\n| ImageNet | Swin-L | 54.7 | 45.3 | 88.49 |\n| Ours | Swin-L | 55.2 | 45.5 | 85.69 |\n\nAs shown in the table, our method achieves comparable performance on 2D detection and instance segmentation tasks, but also shows noticeable shortcomings in semantic segmentation. Consequently, features learned by our model are instance-discriminative rather than category-discriminative. However, as demonstrated in paper's Table 3, our features are complementary to class-level features."}, {"review_id": "O04UutftWK", "reviewer": "Reviewer_Rb8Z", "summary": "Inspired by insights from classic computer vision and developmental psychology regarding the importance of motion boundaries for inferring object boundaries and grouping image elements with similar motion into objects, the authors propose a system for unsupervised learning of \"object-centric\" visual representations. They leverage an off-the-shelf optic flow algorithm to generate motion-defined object instance masks and use them for contrastive visual representation learning. The approach is evaluated extensively and shown to outperform previous supervised and self-supervised baselines on three downstream tasks (monocular depth estimation, 3D object detection, occupancy prediction).", "questions": "For future work I'd invite the authors to think about replacing the off-the-shelf optic flow algorithm with a learning-based framework.", "limitations": "I did not see a particular mentioning of potential negative societal impact of this work. But I also don't see this as necessary.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe approach is novel and outperforms several baselines on a set of diverse tasks comprising low-level (monocular depth estimation) and high-level (3D object detection, occupancy prediction) ones. I find it noteworthy that the approach even outperforms some supervised baselines. I feel the work is likely to lead to many follow-up works. As a work demonstrating strong representations emerging from unsupervised learning from large video datasets it is very timely and interesting.\n\nI enjoyed the generalization to indoor scenes outside of the training domain.\n\nWeaknesses:\n\n1) I think some claims need to be toned down:\n1a) Abstract:\n\"Our key insight is that motion boundary serves as a strong signal for object-level grouping\". This is not at all a new insight. You are just using this very old insight in a novel way.\n\n1b) Introduction:\n\"we are the first to leverage motion as an unsupervised supervisory signal to guide the emergence of object-level semantics in visual\nrepresentations.\" This statement also seems rather bold. There is a long history of works (pre-dating the deep learning era) of using motion to segment objects or infer the 3-D structure of the world. I think you should replace \"emergence of object-level semantics in visual\nrepresentations\" with something more specific.\n\n2) You frequently use \"Developmental Neuroscience\" when \"Developmental Psycholog\" is more correct. (Developmental Neuroscience is typically studying the developing brains of animals with invasive techniques that cannot be applied to human infants and children.)\n\n3) Occasional grammar problems\nline 124: missing \"that\" or similar\nline 141: used to separate or used for separating\nline 181: exhibit\nline 195: that -> whose. Or use: for which computational costs scale...", "was_revised": true, "final_justification": "My impression of the paper is even stronger after the authors have thoroughly addressed various concerns in their rebuttal.", "rebuttal": "Dear Reviewer Rb8Z,  \nWe greatly appreciate your valuable comments and questions. Our responses are as follows.\n\n***Abstract: \"Our key insight is that motion boundary serves as a strong signal for object-level grouping\". This is not at all a new insight. You are just using this very old insight in a novel way.***  \nOur idea builds upon numerous prior works on object discovery, and we did not intend to claim this as a novel insight. We will revise the sentence to: “We were inspired by the insight that motion boundary serves as a strong signal for object-level grouping, which can be used to derive pseudo instance supervision from raw videos.”\n\n***Introduction: \"we are the first to leverage motion as an unsupervised supervisory signal to guide the emergence of object-level semantics in visual representations.\" This statement also seems rather bold. There is a long history of works (pre-dating the deep learning era) of using motion to segment objects or infer the 3-D structure of the world. I think you should replace \"emergence of object-level semantics in visual representations\" with something more specific.***  \nWe agree that the current statement in the introduction may appear too strong. We fully acknowledge the rich body of prior work leveraging motion cues for object segmentation and 3D structure inference. We will revise it to “we are first to demonstrate the effectiveness and scalability of using motion as an unsupervised supervisory signal on a large-scale dataset and modern model architectures”\n\n***You frequently use \"Developmental Neuroscience\" when \"Developmental Psychology\" is more correct.***  \nSince our work mainly addresses cognitive aspects, we agree that 'Developmental Psychology' would be a more appropriate term. We will revise the text accordingly.\n\n***Occasional grammar problems***  \nWe are grateful to the reviewer for pointing out these grammar issues. We will thoroughly review the manuscript and make the necessary corrections.\n\n***For future work I'd invite the authors to think about replacing the off-the-shelf optic flow algorithm with a learning-based framework.***  \nWe sincerely appreciate your valuable suggestions regarding future work directions. In our current method, optical flow estimation is separated from model training. This was a deliberate choice to first test the basic effectiveness of our framework using a simple and reliable setup.\nHowever, we agree that an end-to-end learning framework is a promising direction. We are exploring ways to train the optical flow estimator together with our model, so that the main task can help improve flow estimation, and better flow can in turn improve the model’s performance and generalization."}]

Author rebuttal:

Meta-review: The paper introduces a self-supervised, object-centric representation learning method that uses optical flow clustering to generate pseudo-instance masks for training a dense encoder via contrastive loss. Inspired by developmental psychology, the method learns pixel-wise embeddings aligned with motion-derived regions from unlabeled videos. It is evaluated on downstream 3D tasks such as monocular depth estimation, 3D object detection, and occupancy prediction, demonstrating strong performance and outperforming both supervised and self-supervised baselines.

Reviewers praised the method’s simplicity, effectiveness, and scalability, noting that it leverages motion cues in a biologically inspired way. The label-free approach performs well across tasks and surpasses prior methods like SimMIM and DINOv2. Visualizations of learned pixel affinities were compelling, and one reviewer highlighted its generalization beyond driving scenes. The work was seen as timely, with potential for broader impact in unsupervised video representation learning.

Key concerns include the method’s reliance on external optical flow, with no ablations on flow models or sensitivity to thresholds and hyperparameters. Evaluation focused mostly on 3D driving datasets, leaving open questions about generalizability to 2D or non-motion tasks. Some novelty claims were seen as exaggerated, and differences in model backbones raised fairness concerns in comparisons. Reviewers also noted missing evaluations (e.g., segmentation, linear probes), limited related work citations, and called for clearer writing and terminology.

The authors’ rebuttal addressed many core concerns, including ablations and architectural comparisons, and the additional experiments on 3D downstream tasks helped solidify the method’s empirical claims. Reviewers noted the strong performance across multiple benchmarks and appreciated the paper’s clarity and potential for follow-up work.

However, one reviewer maintained a borderline reject, expressing concern over the method’s limited generalization beyond driving scenes, which restricts its broader applicability. While the authors defended their domain choice by citing optical flow quality, the reviewer found this reasoning unconvincing given recent work using natural scene videos (non-driving datasets like DAVIS) with off-the-shelf flow. Additionally, the lack of ablations on different optical flow estimators was seen as a missed opportunity to understand the method’s robustness.  These concerns suggest that while the framework is promising, its broader applicability beyond driving scenarios is still uncertain.

In terms of novelty, the authors clarified during the rebuttal that their intent was not to claim motion boundaries as a new insight, but rather to demonstrate the scalability and modern implementation of this classical idea. They acknowledged prior work on motion-based object discovery and committed to revising phrasing in the paper to more accurately reflect this, e.g., changing “our key insight” to “we were inspired by the insight...” and reframing their novelty as demonstrating effectiveness on large-scale datasets with modern architectures. With these adjustments, and given the paper’s solid technical execution and relevance, the consensus (out of 2x accepts, 2x borderline accepts, 1x borderline reject) is to accept.
Decision: Accept (poster)