PooDLe🐩: Pooled and dense self-supervised learning from naturalistic videos
Authors: Alex N Wang
Christopher Hoang
Yuwen Xiong
Yann LeCun
Mengye Ren
Venue: ICLR 2025 Poster (2025)
Keywords: computer vision
representation learning
self-supervised learning
egocentric video
visual representation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We uncover challenges of applying self-supervised learning to naturalistic, dense video data and propose a unified dense and pooled objective alongside architectural improvements to learn an effective visual representation.

Abstract: Self-supervised learning has driven significant progress in learning from single-subject, _iconic_ images.
However, there are still unanswered questions about the use of minimally-curated, naturalistic video data, which contain _dense_ scenes with many independent objects, imbalanced class distributions, and varying object sizes.
In this paper, we propose PooDLe, a self-supervised learning method that combines an invariance-based objective on pooled representations with a dense SSL objective that enforces equivariance to optical flow warping.
Our results show that a unified objective applied at multiple feature scales is essential for learning effective image representations from naturalistic videos.
We validate our method with experiments on the BDD100K driving video dataset and the Walking Tours first-person video dataset, demonstrating its ability to capture spatial understanding from a dense objective and semantic understanding via a pooled representation objective.

Published as a conference paper at ICLR 2025
POoDLE POOLED AND DENSE SELF-SUPERVISER LEARNING FROM NATURALISTIC VIDEOS
Alex N. Wang'", Christopher Hoang'', Yuwen Xiong, Yann LeCun 12, Mengye Ren NNww York University, Meta
(anw2067, ch3451, mengye| @nyu edu https: httpss//aggeticceemaan www.ing.aappoooo poodle/
T ai
ABSTRACT
Self-supervised learning has driven significant progress in learning from single- subject, iconic images. However, there are still unanswered questions about the use of minimally-curated. naturalistic video data, which contain dense scenes with many independent objects, imbalanced class distributions, and varying object sizes. In this paper, we propose PooDLe, a superviee learning method that combines an anvarianc--bsee objective on pooled representations with a dense SSL objective that enforces equivariance to optical flow warping. Our results show that unified objective applied at multiple feature scales is essential for learning effective image representations from naturalistic videos. We validate our method with experiments on the EDDI00K driving video dataset and the Walking Tours first-person video dataset, demonstrating its ability capture spatial understanding from dense objective and semantic understanding via a pooled representation objective.
INTRODUCTION
Humans and other animals learn visual understanding from a continuous stream of inputs with little explicit supervision. Self-supervised learning (SSL) (Chen et al., 2020; Grill et al., 2020; Chen & He, 2021; Caron et al., 2021; Bardes et al., 2021; He et al., 2022; Assran et al., 2023; Bardes et al., 2023a; He al., 2020) has made great strides in learning without human annotations, becoming competitive with supervised learning. However, many methods still revolve around ImageNet (Deng al., 2009), which is implicitly supervised through iconic images that contain a single, clear subject and balanced class distribution. In contrast, naturalistic data like egocentric videos contain cluttered scenes, imbalanced classes, and objects of varying sizes, making them ill-suited for iconic methods.
Nevertheless, these naturalistic videos are still valuable for their information density and ease of collection, while also mimicking the real-life perspective of humans. Unfortunately, iconic methods, which pool global image representations, may perform poorly dense scenes produce views containing independent subjects that are semantically incompatible (Figure 1b, red boxes). Recent works have attempted to address this weakness by introducing 1) cropping (Selvaraju et al., 2021) or attention (Venkataramanan et al., 2024) mechanisms to account for multiple subjects, and 2) "dense SSL" objectives (Xiong et al., 2021; Wang et al., 2021b) with losses defined over regions of unpooled image representations.
While dense SSL methods avoid semantic mismatches, discover that they are susceptible to spatial imbalance where larger background classes like the sky dominate the representation, while smaller classes like pedestrians objects should be prioritized over low-detail, repetitive background classes. Furthermore, this can be dangerous in applications like self-driving (Yu al.. 2020) where critical objects like pedestrians occupy less than 0.3% of a video frame (Figure 1b, green boxes and Ic). This contrasts with ImageNet (Deng et al., 2009) training, where models can easily learn semantics from iconic images with clear, single-subjec views and balanced class distribution. Surprisingly, dense methods like FlowE (Xiong et al., 2021) and supervised ImageNet pretraining achieve similar downstream performance while converging very different solutions; the former prioritizes large background classes while the latter captures many small and rare classes, but with relatively poor specificity.
underrepresented. This is undesirable because smaller foreground
"Equal contribution

---

Published as a conference paper at ICLR 2025
2014
2 2a
1
CH
(a) Iconic image
(b) Dense scene with global crops and subcrops
(c) Class distribution
Figure I: Challenges of SSL on dense, naturalistic video include training crowded scenes with many objects of varying scale and class imbalance. (a) Iconic image from ImageNet. (b) Dense scene from BDD100K. Global crops (red boxes) used for iconic data contain multiple, disjoint sets of objects. Subcrops (green boxes) provide pseudo-iconic views of a single subject. (c) Naturalistic video like BDD100K have long-tain long-tailed class distribution of class sizes and frequency; classes ordered by percentage of pixels. Example suberops capture smaller classes like "Traffic Light" and "Person."
Some dense SSL methods (Wang et al., 2021b; Parthasarathy et al., 2023; Venkataramanan et al., 2024) include losses that optimize a global, pooled representation learn semantic information from dense scenes, but do not explore how to integrate these two objectives through means of architecture and augmentation strategies.
To address the challenges of cluttered scenes and spatial imbalance, we propose joint Pooled and Dense Learning (PooDLe) method that optimizes dense SSL. objective over full images and a pooled objective on smaller, both high-level semantics and fine-grained details, effectively representing both small objects and scene-level understanding. Our dense objective is adopted from FlowE, using optical flow warping to align dense feature maps. To adapt the pooled objective to dense scene data, we introduce a flow-informed cropping procedure that generates pairs of smaller "subcrops"" with high alignment. These subcrops serve as pseudo-iconic views of foreground objects, functionally increasing the prevalence of smaller objects (Figure 1). Finally, introduce lightweight spatial decoder module (SDM) with top-down layers and UNet-like lateral connections (Ronneberger et al., 2015) to upsample high-level semantic representations and preserve smaller objects in the dense objective. We show that both objectives, combined with the SDM, is essential for capturing the semantics of smaller objects and achieving strong downstream task performance.
views. The combination of objectives captures
We pretrain on BDD100K, dataset of dashcam driving videos as well as Walking Tours, a dataset of first-person walking videos (Venkataramanan et al., 2024). PooDLe achieves state-of-the-art performance semantic segmentation and object detection benchmarks, with notable gain on recognizing small objects. We also introduce Walking Tours Semantic (WT-Sem) as a new in- distribution semantic segmentation evaluation for Walking Tours. In ablations, show that our joint objective formulation and the SDM are critical for success. Finally, we study the effect of crop area, input resolution, number of subcrops and temporal stride.
In summary, our contributions are as follows:
We introduce PooDLe, new SSL method which overcomes the challenges of spatial imbalance and cluttered scenes by unifying flow equivariance, dense SSL objective and a pooled objective over pseudo-iconic subcrops alongside a spatial decoder module to effectively learn from natu- ralistic video. PooDLe achieves ofte-of-thheart performance on BDD100K (Yu et al., 2020) and Cityscapes (Cordts et al., 2016) semantic segmentation and BDD object detection. also obtains the highest mIoU on ADE20K (Zhou et al., 2017), and on WT-Sem, our new, in-distribution semantic segmentation task for Walking Tours.
We deconstruct the BDD100K semantic segmentation task, identifying class categories by fre- quency and size within the dataset. We show that existing dense SSL methods and supervised ImageNet training produce different results across these categories, while PooDLe learns a balanced semantic and spatial representation to achieve strong, consistent performance.
3. We study the effects of global and subcrop area, input resolution and temporal stride between paired frames. We show the importance of maintaining pixel density by adjusting crop area when training with larger resolutions for learning visual representations. We also verified that smaller subcrop areas are able to better capture smaller classes. We believe these observations will be helpful in guiding future work on dense, naturalistic data.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Self-supervised learning with iconic images. Representation learning on iconic image datasets has a long history, from denoising autoencoders (Vincent et al., 2010) to joint embedding methods (Chen et al., 2020; He et al., 2020; Grill et al., 2020; Zbontar et al., 2021; Bardes et al., 2021; Caron et al., 2021) to joint-embedding predictive architectures (Assran et al., 2023; Bardes et al., 2023b). Joint embedding methods learn representation invariance to visual changes produced by augmentations using contrastive (Chen et al., 2020; Oord et al., 2018). mean squared error (Grill et al., 2020), or classification (Caron et al., 2021; 2020) losses between corresponding pairs, pushing SSL to new heights on ImageNet classification. Later works extend these methods to curated, internet-scale data (Oquab et al., 2023) and include other modalities like text (Radford et al., 2021). Separately, MAE (He et al., 2022) learns via reconstruction of masked image regions. iBOT (Zhou et al., 2021) combines joint embedding methods with token reconstruction achieve impressive results on ImageNet classification. The methods above have been primarily designed for iconic images and contain assumptions that may not transfer well to uncurated datasets, e.g. dense scenes. Methods leveraging multi-crop (Caron et al., 2020; 2021; Oquab et al., 2023; Zhou et al., 2021) generate small crops optimized to predict the representations of global crops for training on iconic images with little additional compute. In contrast, our subcrop strategy yields small, aligned crops pseudo-iconic, paired views from otherwise dense scenes.
Training using dense multi-subject images. Following the success of SSL on ImageNet, other works seek to learn from dense, multi-subject images where augmented views may not contain corresponding subjects for invariance learning. Wang et al. (2021b); Xie et al. (2021); Chen et al. (2021) extend joint embedding methods by leveraging feature similarity bootstrapped from standard invariance learning to identify positive pairs across dense, unpooled feature maps. Hénaff et al. (2021); Wang et al. (2021a) optimize dense losses, contrasting pixels belonging to different semantic classes; these methods require off-the-shell segmentation modules. Ziegler & Asano (2022); Guo et al. (2023) utilize DINO (Caron et al., 2021) attention maps to identify training pairs, while ADCLR (Zhang et al., 2023b) identifies pairs using small "query" crops and the patches that attend to them. These methods advance the ability to learn from dense images with multiple objects, but still have limitations. Some rely on learning objectives that make assumptions about iconic data, while others struggle with the spatial imbalance problem that is especially prevalent in naturalistic data.
Learning image representations from video data. Extending beyond images, other works have sought to capture the variance of objects through time by training on pairs of video frames. Gordon al. (2020) adapts contrastive learning to use correlated frames positive examples, while Jabri et al. (2020); Parthasarathy et al. (2023) identify positive pairs based on high similarity in representation space. FlowE (Xiong et al., 2021) builds on BYO (Grill et al., 2020) and identifies positive spatial regions between frames using off-the-shelf flow. MC-JEPA (Bardes et al., 2023b) learns motion using video data by aligning latent representations throughout the feature pyramid while performing representation learning on ImageNet. Most recently. DoRA (Venkataramanan et al., 2024) proposes a new dense video dataset and extends DINO by clustering over many frames to identify and track objects for representation learning. In the MAE paradigm, Tong et al. (2022); Feichtenhofer et al. (2022) directly reconstruct sequences of frames while Weinzaepfel et al. (2022); Gupta et al. (2023) perform reconstruction given a corresponding overlapping frame. While PooDLe learns rich image representation from video data similar to these existing methods, it distinguishes itself by leveraging unified dense and pooled objective architecture, specifically designed to tackle the challenges posed by naturalistic data.
P POOLED AND DENSE LEARNING FROM NATURALISTIC VIDEOS
We present PooDLe, a self-supervised method for learning visual representations using paired frames from naturalistic, first-person videos. PooDLe combines two SSL objectives: a dense objective for learning representations of dense, crowded scenes; and a pooled objective on small suberops sampled using flow-aware cropping augmentations. We also propose a lightweight spatial decoder module (SDM) that uses top-down decoder layers and UNet-like lateral connections to earlier encoder representations to both upsample the high-level representations and resurface fine- tne-ggained details and small objects that may get lost in downsampling operations. For high-level overview of PooDLe, see Figure 2.
3

---

Published as a conference paper at ICLR 2025
predictor
Apool
augpool
Stop gradiont
avgpool
emi
encoder fe
spatial decoder
spatial decoder
encoder
go
Bt
predictor
dense
subcrop Xxa
Ap subcrop
frame X,
flow M2-1
frame Xeear
Figure 2: PooDLe, a self-supervised learning method that combines pooled and dense objectives. Green path: dense objective performing flow-equivariance learning on the output of the decoder g()). Orange path: pooled objective encoding K subcrops sampled with flow-informed cropping. Projector modules are not shown. Offline weights are the exponential moving average of online weights 0.
Preliminaries. Inputs to the model are video frame pairs It, Fe+AAe with dimensions H W. and dense optical flow Mr-tt-tt: Randomly sampled augmentations A₁ and A2 are applied to each example to create positive training pairs. In a similar setup to BYOL, the encoder and projector are denoted as a function p = f(x) using either online weights or offline, EMA-updated weights ç. The predictor module (-) only has online weights 0. Separate projector and predictor modules used for the pooled and dense objectives, but are not annotated for simplicity. We use a ResNet-50 backbone, well as projectors and predictors following FlowE (Xiong et al., 2021) and BYOL (Grill et al., 2020), that are discarded after pretraining.
Dense SSL with flow equivariance. The dense objective follows FlowE (Xiong et al., 2021) by using optical flow Mr align paired feature projections p: and Pr+Ar. At high-level, this objective minimizes differences in representation between corresponding regions. More specifically, the inverse augmentation functions A-1 and optical flow are used to align the representations p and after upsampling to input resolution H x W, the objective the squared error:
Ldense
9o (p:)) - (Mc+++ (p)))))))))))) 2
(1)
HW
where normalization is applied after the predictor and flow warping.
Pooled objective with flow-informed subcrops. First, we identify K pseudo-iconic subcrop pairs. Unlike for iconic data, random crops from paired frames are unlikely to contain common subject. To mitigate this problem, we once again use optical flow in flow-informed cropping procedure identify aligned training pairs. For each subcrop pair, we sample a random point (u, v) in the target frame Et+ Arr to serve as the crop center. It is then warped into the earlier frame using flow Mi-+++ plus random jitter (6a, 5,) for paired center (u', e²). crop made around each center, with area sampled from U[sminn 5max] of the global crop for subcrops Lt,i and Tr+Ar,
As we require both crop centers to land within the bounds the image, subcrops tend to center- biased (Peng et al., 2022) and lack diversity. To remedy this, we employ a grid-sampling procedure for selecting the initial crop center (u, v). Each global crop x is divided into a grid with cells of side length dgnd = min H, W) / (Smin + mmmxm// for a H /dggri x W/dgii grid. Each cell is selected without replacement, and center (u, v) is then uniformiy sampled within the cell.
After K pairs (Ir,R, -+,A are generated, they are encoded by the backbone and the pooled objective projector. Unlike the dense objective, alignment or upsampling is performed, and each projection p is averaged-pooled over its spatial dimensions before computing the loss:
Cpool Σ (g,(Pr.)) Pr+
where denotes average pooling over spatial dimensions followed by normalization. Our objective has each suberop to predict its corresponding pair, which contains the same object in different frame.
4

---

Published as a conference paper at ICLR 2025
This differs from multi-crop (Caron et al., 2020), where local crops predict global crops, which would be less effective for dense scenes as local crops only capture a subset of the objects in a frame.
Spatial Decoder Module (SDM). We introduce the SDM (Fig- ure 3b) to upsample high-level encoder features and preserve information from lower layers, particularly smaller foreground objects that may be lost during pooling operations. Its design draws inspiration from a convolutional UNet (Ronneberger et al., 2015) and FPN (Lin et al., 2017) and improves upon FlowE's use of dilated convolutions to replace pooling by ef- ficiently maintaining high-resolution representations while re- ducing activations and memory usage.
(a) Baseline
The SDM utilizes decoder blocks, each consisting of upsam- ple operation, a computation block of processing layers g(-), and UNet-like lateral connection. The output of each block is computed as:
z+41 ((upsamplee2 + lateral(='),
where 20)) is the representation after the 1th encoder stage and 201) is an earlier feature map of the same spatial dimensions as ((+1)_ The use of computation blocks and lateral connections ablated in Table 4. Figure 3 contrasts a naive implementation that places both objectives at the top encoder level and Poo- DLe, which uses the SDM to integrate the two objectives in a complementary fashion.
(b) PooDLe using SDM
Figure 3: a) Baseline: both losses combined at final encoder layer; D)PooDLe: SDM incorporates ear- lier feature maps, upsampling for Leeence-
4 EXPERIMENTS
We pretrain PooDLe on raw videos from BDDI0OK (Yu et al., 2020) and Walk- ing Tours (WT) (Venkataramanan al., 2024) and evaluate them on semantic segmentation and object detection benchmarks. The BDD100K pretrained model is evaluated on in-distribution tasks as well as Cityscapes (Cordts al., 2016), and the Walking Tours model on ADE20K (Agrawal et al., 2015) and our newly proposed Walking Tours Semantic benchmark. We also ablate our combination of loss functions and decoder components, as well as the effects of crop area and input resolutions.
4.1 EXPERIMENT SETUP
Pretraining datasets. 1) BDD (Yu et al.. 2020) consists of 100,000 dashcam driving videos collected in various weather conditions and times of day from New York and the San Francisco Bay Area. Each video is -40 seconds long at 720p and 30 fps. We pretrain with the 70,000 videos in the training split and evaluate on the dataset's 's semantic segmentation and object detection tasks. 2) Walking Tours (WT) (Venkataramanan et al., 2024) is a dataset of first-person Tube videos of continuous walkaround through various cities of Europe, Asia, and wildlife safari. There are 10 videos, ranging from 59 minutes to 2 hours 55 minutes, at 720p and 30 fps. Each video contains large numbers of unique objects per frame and natural transitions in lighting and location. We use either the Venice video veTnen) or all 10 videos (WT.I) following DoRA (Venkataramanan al., 2024).
Technical details. We use ResNet-50 (R50) (He al., 201 as our feature encoder, with the dense projector and predictor networks following FlowE (Xiong et al., 2021) and pooled counterparts following BYOL (Grill et al., 2020). For SDM, we use two decoder stages, with each consisting of 2x upsample, a ResNet Bottleneck Block (He et al., 2016a), and 2-layer convolutional MLP for the lateral connection. When training on BDD, we sample two frames that are 0.5 1 seconds apart (At E 31) from each video. We then take two large crops from the same image coordinates of area 10.16, 0.45) of the original image and resize them to 512 x 1024 pixels before applying augmentations. For each training epoch on WT, we divide each video into 10-second clips and randomly sample two frames 0.5 seconds apart from each clip, and use crop area range [0.65, 1.0]. For both datasets, we apply color distortion and Gaussian blurring independently to each frame following BYOL (Grill et al., 2020). For the dense objective, we also apply random reversible affine transformations similar to FlowE (Xiong et al., 2021): random scaling of 0.9-1.1x and rotation of -10-10 degrees. For the local objective, we sample K = subcrop pairs with a crop area of
5

---

Published as a conference paper at ICLR 2025
Table BDD and CityScapes semantic segmentation (SemSeg) and object detection (Det) readout evaluations. All settings are conducted with frozen backbone. "Pretrained on BDD, initialized with supervised IN1K weights.
BDDIOOK
Cilystupes
Method
Arch
Ep.
Pretruin
mloU
Aec
mloU
UperrNet
Ace
mAP
MAP
mloUi
Acc
mloU
UperNet
Acc
Scraich
R30
9.7
55.0
26.1
81.2
0.0
7.7
9.8
58.0
30.7
$4.1
(Coron 2020)
VITSS
300
BDD
2021)
DoRA (Venkatammmmmm
2024)
VIT-S
200
BDD
33.2
88.1
43.3
37.4
88.7
DINO (Caron et al,, 2023)
R50
100
BDD
13.1
64.7
25.6
S0.3
11.9
14.9
69.4
29.2
81.4
PisPro (Xie et al., 2021)
R50
100
BDD
21.8
80.0
37.3
$5.0
0.7
18.4
25.5
81.0
44.3
89.5
DenseCL. (Wang all, 202
R550
100
BDD
24.2
84.9
41.8
90.0
20.3
26.5
85.6
53.2
91.9
Fookk (Niong
K30
BDD
23.8
Supervised
R50
600
INIK
84.7
55.2
92.0
24.9
46.8
87.4
63.4
93.7
PooDLe
R30
100
BDD*
44.7
90.7
54,1
92.7
3,9
28.0
$2.0
91.5
65.1
94.4
Image
Ground Truth
PooDLe
FlowE
DenseCL
DINO
DoRA
Supervised IN1K
Figure 4: Visualization of BDD semantic segmentation linear readout. PooDLe is able to identify
smaller objects and generate cleaner object boundaries.
[0.05, 0.3] of the initial crop, resized to 192 x 192 for both BDD and WT. For subcrops, random spatial jitter applied as =10% of the initial crops' height and width.
Baselines. We official implementations of DenseCL, PixPro, DINO, iBOT, and DoRA, and our own implementation of FlowE for pretraining on BDD. We use torchvis ion for supervised DINO. We obtain weights
ImageNet (INIK) and weights released online for ImageNet- from the authors of DoRA for iBOT, DINO-ViT, and DoRA pretrained on WT and use official implementations of DINO-R5O, MAE, and PixPro for pretraining on WT. For PixPro, its FPN decoder or high-resolution crops for pretraining and report results from the best-performing setting. We use 512 1024 crops to train all R50 baselines for more accurate comparisons.
either
Evaluation. We adopt the evaluation protocol from FlowE (Xiong al., 2021) for BDD and Cityscapes. We use DeepLab (Chen et al., 2018) as the "lincar" readout header and UperNet (Xiao al., 2018) the heavier readout head for semantic segmentation and Faster R-CNN with ResNet-C4 and Faster R-CNN with FPN (He et al., 2016b) as the standard and heavier readout headers for object detection. We do not include VIT object detection due to the lack of an established recipe. For semantic segmentation on ADE20K, we perform both linear readout following BDD and UperNet finetuning as described in iBOT (Zhou et al., 2021). We retain the SDM when evaluating PooDLe on semantic segmentation with linear readouts but discard it when using UperNet. We report mean intersection-over-unioon (mIoU), pixel-level accuracy (Acc), and mean average precision (mAP) as our evaluation metrics. Additional details on implementation and hyperparameters are provided in Appendix A.
4.2 MAIN RESULTS
EDD100K-pretrain models. We report results semantic segmentation and object detection on the BDD100K benchmark in Table 1. PooDLe achieves superior performance on all readout tasks compared prior methods, outperforming the strongest baseline FlowE by 3.5% mloU linear

---

Published as a conference paper at ICLR 2025
Table 2: ADE20K and WT-Sem semantic segmentation linear readout and finetuning evaluations. Linear readout is performed with frozen backbone while in finetuning. backbone parameters are trainable. DINO- VIT and iBOT results are taken from DoRA (Venkataramanan al., 2024).
ADE20K Sem. Seg.
WT-Sem Sem. Seg.
SemSeg Linear
Finetune
SemSee Linear mloU
Finetune mloU
Method
Arch
Epoch
Pretruin
mioU
Acc
DINO (Caron et 2021)
R50
800
INIK
15.7
61.5
43.0
80.5
8.8
76.7
17.8
87.6
DINO (Caron et al,, 2021)!
VIT.S
100
INIK
33.9
DINO
al, 2021)
VII-S
WTTeese
7.8
29.2
74.7
73.7
11.0
ROT (2hou al., 2021y!
VITS
100
WTwwes
33.9
MAE ot
VII-S
WTeees
7.4
55.1
71.4
72.6
DoR (Venkataramatann et al., 2024)
VIT-
100
WTwaea
14.1
63.5
35.2
77.7
6.2
76.9
13.6
85.7
DINO (Caron et 2021)
R50
WTwsss
6.9
48.2
35.7
77.4
69.0
12.3
84.7
PixPro (Xie al,, 2021)
WTweer
48.6
77.6
69.3
11.5
PooDLe
R50
20
WTwaes
14.6
59.0
36.6
77.9
6.4
75.7
13.7
85.4
DINO (Caron ot al., 2021>
VIT-S
100
WT.
34.1
MAR (Ho
VIT-S
WTa
75.9
77.7
12.7
85.2
DoRA (Venkataramunun al., 2024)
VIT-S
100
WTa
13.9
64.4
38.3
79.3
7.8
79.4
15.9
87.5
PooDLe
R50
20
WT.
16.5
63.9
41.0
79.6
11.2
81,3
17.0
86.9
Table 3: Breakdowns of mloU over different class groupings. Linear readout mloU is computed over various groupings of the 19 classes in BDD semantic segmentation. *Pretrained on BDD, initialized with supervised ImageNet weights.
Method
Pretrain
All
Small
Large
Rare
Common
DINO
BDD
29.6
8.4
42.0
1.0
42.8
DenseCL
BDD
24.2
1.6
37.4
0.0
35.4
DoRA
BDD
33.2
11.9
45.6
47.3
FlowE
BDD
35.7
12.2
49.3
10.7
47.2
PooDLe
BDD
39.2
18.3
51.4
12.0
51.8
Supervised
IN1K
36.7
27.2
42.2
16.1
46.2
PooDLe
3DD
44.7
25.2
56.1
17.9
57.1
and 2.6% mloU on UperNet for semantic segmentation, and 1.7% mAP on C4 and 1.4% mAP on FPN for object detection. We find that that PooDLe's improved performance (Table 3) is attributed to better recognition of small and rare object classes. We also evaluate the transfer of PooDLe representations new tasks by evaluating on the Cityscapes benchmark, where PooDLe outperforms all baselines. Figure 4 shows predicted segmentation masks, and Figure 14 and Figure 15 show additional evaluation results.
PooDLe also outperforms supervised IN1K pretraining, despite the latter's advantage in learning small and rare classes present BDD (spatial imbalance shown in Figure 1) due to ImageNet being class-balanced dataset with iconic views of objects. In addition, we pretrain PooDLe BDD100K with weights initialized from the supervised IN1K checkpoint, improving mloU by 8% and Acc by 6% over the initialization weights on linear semantic segmentation. In Appendix G, show PooDLe remains competitive against INIK areetrrede baselines despite being trained in the challenging naturalistic video setting.
WT-pretrained models. We also train PooDLe WT and WT Table 2 shows results on ADE20K (Zhou et al., 2017) semantic segmentation using linear readout and finetuning follow- ing (Venkataramanan al., 2024). Notably, when pretrained on WT PooDLe obtains 2.6% higher mloU than DoRA on ADE20K linear readout, and 2.7% mIoU on UperNet finetuning. PooDLe also performs better WT Venice. with gain of 1.4% mIoU over DoRA and 0.6% mloU over PixPro on ADE20K UperNet finetuning. We note that PooDLe uses a smaller ResNet-50 backbone and is trained for fewer epochs than DoRA, the strongest baseline. Despite these differences, these results show that PooDLe learns strong representations from naturalistic video captured in the open world. Figure 16 shows predicted segmentation masks for ADE20K.
Walking Tours Semantic benchmark. While ADE20K is challenging benchmark, it contains mixture of indoor and outdoor scenes that can be out-of-distribun from scenes in Walking Tours. Therefore, introduce Walking Tours Semantic (WT-Sem) provide in-distribution bench-
7

---

Published as a conference paper at ICLR 2025
Table 4: Ablation studies PooDLe components, reporting mloU on BDDI00K semantic segmenta- tion linear readout. Rows without top-down follow FlowE (Xiong et al., 2021), replacing pooling with dilated convolutions to maintain spatial extent. FFlow model trained without supervised labels.
Variant
Dense
Pool
Top-Down
Lateral
Flow
All
Small
Large
Rare
Common
FlowE
RAFT
8.7
40.5
29.2
RAFT
7.2
41.6
28.7
/
RAFT
30.3
6.8
44.0
4.3
30.2
/
RAFT
30.3
10,9
41.7
2.4
31.1
V
RAFT
31.8
12.8
42.8
8.3
31.7
PoooDLef PooDDe
/
UFlow
33.7
14.1
45.1
8.9
33.8
/
/
RAFT
34.2
15.0
45.5
9.0
34.5
classes large
Subcrop
small
all
0,04 0.18
16.0
41.6
34.6
0.18 0.36
14.2
41.2
33.7
30
0.36 0.54
13.6
39.5
32.4
0.54 1.00
12.3
38.8
31.5
286.26
Table 5: Choice of suberop area on small, large and all classes.
Figure 5: Varying input resolution and global crop
fraction of the full frame. Large resolutions prefer larger crops.
mark to accompany the WT dataset. We find that when pretrained on WT - PooDLe outperforms DoRA (Venkataramanan al., 2024) by 3.4% and 1.1% mIoU linear readout and UperNet finetuning, respectively. To generate the dataset, we use OpenSeeD (Zhang et al., 2023a), strong open-vocabulary segmentation model, to generate semantic segmentation masks for all videos in WT as well as 3 new walkaround videos. We use the Swin-L (Liu et al., 2021) variant of OpenSeeD finetuned on ADE20K semantic segmentation with a vocabulary of the 150 class labels from ADE20K to generate masks. See Appendix D for visualizations and details of WT-Sem.
Class-based performance and INIK initialization. Naturalistic videos have imbalanced class representation and object sizes (Figure Ie; e.g., "road" occupies 21% of pixels while "bicycle" only occupies 0.05% of pixels). Capturing information on these underrepresented classes is very challenging. To further demonstrate this phenomenon, we categorize BDD classes as "small" they occupy 1% of pixels and "large" for those that occupy 1%. Separately, we define "rare" as classes that appear < 20% of images and "common" as those that appear shows linear readout mloU for different class groupings, highlighting the impact of class and spatial imbalance. Full class-level statistics and designations are in Appendix H. We observe that FlowE performs well large and common classes due to its dense loss, but struggles small rare classes. Meanwhile, supervised IN1K, benefiting from balanced pretraining data, effectively learns about smaller classes. PooDLe, with its unified objectives and spatial decoder module, significantly outperforms other BDD-pretrained models across all class groupings, particularly on small and rare classes. PooDLe, initialized from supervised INIK weights, significantly improves upon supervised INIK on large classes, from 42.2% to 56.1% due to the dense objective, while remaining competitive with supervised INIK on small classes.
20%. Table
4.3 ABLATION STUDIES
Table 4 shows ablation experiments, testing each of our contributions beginning from FlowE. Models trained without the decoder use dilated convolutions place pooling operations, as in FlowE (Xiong al., 2021). Figure 3 shows how the dense and pooled objectives composed with and without the decoder. For the ablations, models are trained for 40 epochs on BDD and use a reduced 256 512 resolution and [0.04, 0.11] area for the initial crops; evaluate BDD semantic segmentation using linear readout.
We observe that adding Epoot alone has little benefit (row 2) and including either the decoder spatial upsampler (row 3) or only the UNet-style lateral connections (row 4) also does not yield much benefit. Row 5 achieves +3% mloU, showing that the top-down decoder only effective when
8

---

Published as a conference paper at ICLR 2025
mial
X-
100%
50%
Figure 6: Varying initial crop percent of the full frame. As crop area decreases, the views transition from global views of a dense number of suberops K. scene pseudo-iconic views, sometimes depicting singular subjects.
Figure 7: mloU when varying
32
30
28
34-00 (0s)
(0.5g
ÁI 30 (1s)
A1.55
teriporal stride (MI
Figure 8: Overlaid frames with varying At.
Figure 9: mloU when varying At.
combined with the lateral connections for the full SDM, suggesting that preserving tigh-resolution information as well as including some capacity for feature processing are both important. However, when re-adding the pooled loss addition to the decoder with lateral connections (row 7), we see substantial 5.4% mloU improvement. While the dense objective benefits from the full SDM, it has an even greater synergistic effect with the pooled loss. This may be because the pooled objective with subcrops can effectively learn about small objects while the full decoder helps propagate the semantic representations through to the dense loss.
We also demonstrate that PooDLe is able to perform well even with self-supervised flow. We train UFlow (Jonschkowski et al., 2020) model on KITTI (Geiger et al., 2013) and finetune on BDD resulting in only 0.5% mIoU loss compared pretraining with RAFT. See Appendix A and J for details and visualizations.
4.4 SPATIAL AND TEMPORAL CROPPING IN SELF- SULERVSUERRI VIDEO LEARNING
In this section, we study the effect of frame intervals and image cropping parameters used in data augmentation. Without a 1:1 mmage-to-concepp relationship like in iconic data, the visible area of each frame can greatly affect representation learning. study this, we perform 4 experiments varying: (1) subcrop area, (2) global crop area, (3) number of subcrops, (4) temporal stride At. Crop and subcrop area refer to the fraction of the frame taken during random-resiie mesized cropping. Figure depicts how crops transition from global to pscudo-iconic with decreasing crop area. Training recipe follows the ablations in section 4.3 and results should compared to row 7 in Table 4.
Varying suberop area. First, we study how subcrop area affects our learned representations. We train 4 PooDLes using different subcrop ranges and fixed global crop area of [0.125, 0.25) at resolution 256 x 512. Results are shown in Table for classes, as well as small and large class subgroupings. We observe that larger subcrop areas result in worse performance, with larger relative drop for smaller classes. This likely because larger crops contain multiple, smaller objects which breaks the pseudo-iconic assumption and produces false invariances.
Varying global crop area Next, we vary both the global crop area from the raw video frame along with the input resolution to study their effects on self-supervised pretraining. We select different resolutions, and sample the crop area from Gaussian truncated to [0, 1). with varying mean and 0.1. All three resolutions are trained with μ 0. 0.5 0.275, 0.425, 0.725 with the 2 smaller resolutions also trained at μ 0.05, 0.20 for higher granularity. Our results in Figure 5 show that larger crop areas and higher input resolutions, together, are important for maximizing performance. The largest model (512 1024) produces the best results and peaks at 0.425 while the other two peak at smaller crop areas. The 256 x 512 model degrades more slowly in performance as crop area increases in comparison the 128 x 256 model.

---

Published as a conference paper at ICLR 2025
Varying number of suberops. We also study how varying the number of suberops affects perfor- mance. We train PooDLes using K = 0, 3, 6, 9 suberops on BDD100K and evaluate using linear readout, with results shown in Figure 7. Using suberops gives initially large performance jump, and using additional subcrops provides more modest gains. We decide to use = 6 as our default option to balance between performance and computational efficiency.
Effect of temporal stride during frame sampling. We study the effect of temporal stride by training PooDLe with At = 0, 8, 15, 30, 45 on BDD100K, evaluated using linear readout (Figure 9). Performance peaks at At 15 and degrades only slightly at 8 and, 30 while dropping further at values and 45. When is small, there is limited variance in object appearance, diminishing the value of video data, and when it is too large, correspondence between frames decreases and optical flow becomes unreliable. Note that for At = O, we add jitter to the initial large crop by up to 10% of the image size. Figure 8 shows frame sequences from different videos, highlighting the high variability of motion in BDDIOOK.
4.5 SUBCROPS AS PSEUDO-ICONE TRAINING IMAGES
80
Probabiitty
20
Empirical Subcrap Hit
Poxol
20
40
4b
RD
100
Object size (% pisels of image)
Toy simulation the probability of subcrop acting hitting circular object. Suhcrop hits are green and non-hits are red.
(b) Suberop and pixel probabilities foreground objects of varying size from toy and empirical sim- ulation.
Figure 10: Analysis of subcrops pseudo-iconic views.
To understand the impact of pseudo-iconic subcrops on PooDLe's performance, particularly on small objects, we analyze their effect object prevalence in the pooled objective. Using a simulated circular object (Figure 10a), we calculate the probability of subcrop capturing it as pseudo- iconic view, i.e. subcrop "hit", and compare this the probability of pixel landing on the object, emulating pixel-Ievel dense SSL objective. We count a subcrop hit if it reaches at least 5% object coverage, which we justify as background classes generally have little visual variation and consequently, minimal impact the pooled representations. We extend this analysis to the BDD100K semantic segmentation dataset, empirically simulating subcrops, and computing subcrop hit and pixel probabilities for varying object sizes. The simulation results, illustrated in Figure 10b, show greater relative difference between suberop hit and pixel probabilities for smaller objects, indicating pseudo-iconic subcrops increase their prevalence in the pooled objective. This likely contributes towards PooDLe's improved performance on small object classes compared to dense SSL methods. Further details of the analysis be found in Appendix C.
5 CONCLUSION
Self-supervised learning on naturalistic videos presents many unsolved challenges, especially due to the presence of high-resolution, multi-object crowded scenes with severe spatial imbalance. Iconic methods rely on single-subject images, and dense methods struggle with the scale imbalance of objects. We propose PooDLe that combines pooled region- invariance learning and dense dow-equivariance learning objectives in unified framework. PooDLe achieves state-of of-the--ttttttt performance on down- stream semantic segmentation and object detection evaluations compared to prior methods pretrained on the same video datasets, particularly on recognizing small objects. Our study on the effects of crop area, input resolution, and temporal stride also offers key insights on the design choices for video self- supervsee learning.
10

---

Published as a conference paper at ICLR 2025
ACKNOW
We thank Jenny Zhu for her assistance in generating semantic segmentation labels for the WT-Sem dataset. The work is supported in part by the Institute of Information & Communications Technology Planning & Evaluation (IIPP) under grant 20244-02404442,, funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AI Frontier Lab International Collaborative Research. AW is supported by the NSERC PGS-D Scholarship. CH is supported by the DoD NDSEG Fellowship. The compute supported in part through the Microsoft Accelerating Foundation Model Research (AFMR) program, a Google Cloud Platform (GCP) award, and NYU IT's High Performance Computing resources, services, and staff expertise.
REFERENCES
Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving, In Proceedings of the IEEE International Conference on Computer Vision, 2015.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-super learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: aetaumee-inaarinne--ovaaaaaaaaee regularization for supervised learning. In International Conference on Learning Representations, 2021.
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-jepa: Latent video prediction for visual representation learning. arXiv preprint arXiv:2404.08444 2023a.
Adrien Bardes, Jean Ponce, and Yann LeCun. Mc-jepa: A joint-embedding predictive architecture for self-supervised learning of motion and content features. arXiv preprint 4rXiv:2307.12696 2023b.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision, 2021.
Kai Chen, Lanqing Hong. Hang Xu, Zhenguo Li, and Dit-Yan Yeung. Multisiam: Self-supervised multi-instance siamese representation learning for autonomous driving. In Proceedings of the IEFF/CVF International Conference on Computer Vision, 2021.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint weXiv:7706.05:0 2017.
Liang-Chiel Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscupes dataset for semantic urban scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
11

---

Published as a conference paper at ICLR 2025
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2009.
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. In Advances in Neural Information Processing Systems, 2022.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. International Journal of Robotics Research, 2013.
Daniel Gordon, Kiana Ehsani, Dieter Fox, and Ali Farhadi. Watching the world go by: Representation learning from unlabeled videos. arXiv preprint arXiv:2003.0799 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach to self- supervised learning. In Advances in Neural Information Processing Systems, 2020.
Qiushan Guo, Yizhou Yu, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Ping Luo. Multi-level contrastive learning for dense prediction task. arXiv preprint XXxv:3304.02010, 2023.
Agrim Gupta, Jiajun Wu, Jia Deng, and Fei-Fei Li. Siamese masked autoencoders. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 40676-40693. Curran Associates, Inc., 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016a.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016b.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Olivier Hénaff, Skanda Koppula, Jcan-Baptist Alayrac, Aaron Van den Oord, Oriol Vinyals, and Joao Carreira. Efficient visual pretraining with contrastive detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as contrastive random walk. Advances in Neural Information Processing Systems, 2020.
Rico Jonschkowski, Austin Stone, Jonathan Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova. What matters in unsupervised optical flow. In European Conference Computer Vision, 2020.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He. Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
Pengpeng Liu, Irwin King. Michael R Lyu, and Jia Xu. Ddflow: Learning optical flow with unlabeled data distillation. In Proceedings of the AAAI Conference Artificial Intelligence, 2019.
12

---

Published as a conference paper at ICLR 2025
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transforeer using shifted windows. In 2021 IEEE/CVF International Conference Computer Vision, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1nXivv:807..3333 2018.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions Machine Learning Research, 2023.
Nikhil Parthasarathy, SM Eslami, Joao Carreira, and Olivier Henaff. Self-supervised video pretraining yields robust and more auman-aligned visual representations. Advances in Neural Information Processing Systems, 2023.
Xiangyu Peng, Kai Wang, Zheng Zhu, Mang Wang, and Yang You. Crafting better contrastive views for siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assisted Intervention International Conference, 2015.
Ramprasaath R. Selvaraju, Karan Desai, Justin Johnson, and Nikhil Naik. Casting your model: Learning to localize improves self-supervised representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In 2019 IEEE/CVF International Conference on Computer Vision, 2019.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference Computer Vision, 2020.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders data-efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Systems, 2022.
Shashanka Venkataramanan, Mamshad Nayeem Rizve, João Carreira, Yuki Asano, and Yannis Avrithis. Is imagenet worth 1 video? learning strong image encoders from long unlabelled video. In International Conference on Learning Representations, 2024.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations deep network with local denoising criterion. Journal of Machine Learning Research, 2010.
Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Ex- ploring cross-image pixel contrast for semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021a.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for supf-super visual pre- pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021b.
Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. Croco: Self-supervised pre- trainigg for 3d vision tasks by cross-view completion. Advances in Neural Information Processing Systems, 2022.
13

---

Published as a conference paper at ICLR 2025
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision, 2018.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2021.
Yuwen Xiong, Mengye Ren, Wenyuan Zeng, and Raquel Urtasun. Self-supervised representation learning from flow equivariance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, 2021.
Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. A simple framework for open- vocabulaayy segmentation and detection. In Proceedings of the IEEF/CVF International Conference Computer Vision, 2023a.
Shaofeng Zhang. Feng Zhu, Rui Zhao, and Junchi Yan. Patch-level contrasting without patch correspondence for accurate and dense contrastive representation learning. In International Conference Learning Representations, 2023b.
Bolei Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang, Shen, Cihang Xie, Alan Yuille, and Tao Kong, ibot: Image bert pre-training with online tokenizer. In International Conference Learning Representations, 2021.
Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A IMPLEMENTATION DETAILS
Backbone. As discussed in the pretraining details, we use a Resnet-50 (He et al., 201 as our backbone architecture. The projector model is non-linear, 2-layer MLP (linear for pooled, 1 x 1 convolutions for dense) that has a 4096 hidden dimension and projects down to 256 dimensions. The predictor is the same network with 256 - 4096 256 channels. We follow BOL (Grill et al., 2020) with a momentum starting at 0.996 and increasing to 1 throughout training.
Decoder details. The decoder uses single Bottleneck block from the ResNet architecture with a 8x downsampling ratio in the number of channels. Upsampling in the decoder is 2 and the lateral connection is a single linear convolutional layer that up-projects the input latent to match the decoder channels (1024 + 2048 in the first decoder block and 512 - 2048 for the second block). As mentioned, decoder blocks are used to achieve total of 4x upsampling.
Supervised and self-supervised flow prediction. Flow is predicted using a supervised off aff-the-shelf RAFT model or an unsupervised UFlow (Jonschkowski et al., 2020) model that we train ourselves. For unsupervised training, we exactly follow UFlow and train on the KITTI (Geiger et al., 2013) dataset before finetuning on BDD 100k (Yu et al., 2020) for 100,000 steps on daytime-only videos. The training and inference resolutions were set to 256 x 512 to better match the inference setting. KITTI used adjacent frames (10Hz video) while BDD frames were sampled with temporal stride of 10 (30Hz video).
Local cropping details. K paired local crops are sampled using the methods described. Cropping is performed using RandomResizedCrop with an output resolution of 192 x 192. Jitter is 10% of the input image size and a standard aspect ratio range of [3/4, 4, 4/3] is used.
Loss details. We sum our 2 loss functions directly and give them equal weight. The loss computa- tion and warping function were applied to representations after reversing the affine transform and resizing to the input image resolution. This is to take full advantage of high resolution flow like in FlowE (Xiong et al., 2021). We also use flow-based occlusion to prevent misaligning occluded regions without correspondence. We use the same occlusion formulation as DDFlow (Liu al., 2019) and parameters 01 = 0.1, 02 = 0.5. We also mask out regions that are not visible after affine transformations for Eaense-
Our loss is symmetrical: we reverse the 2; and 2++4 so that both are encoded by the online weights and used for optimization each training step.
Optimization details. AdamW is used the optimizer and weight decay value of 0.01. A learning rate of 5e 4 is used with 32 GPUs and 4 image pairs per GPU for a batch size total of 128. Cosine learning rate decay is used with schedule for 300 epochs, despite early termination due compute limitations. LR warmup is used for 2 training epochs. Full float 32 precision is used during training.
Evaluation settings. For all BDD and Cityscapes semantic segmentation and object detection readout tasks, we follow the setup described in FlowE (Xiong et al., 2021) for ResNet-based methods. For ViT-based methods, we adopt those settings, but use AdamW for the optimizer with learning rate of 3e - 5 and weight decay of 0.05, and a crop size of 512 x 512 rather than the normal 512 x 1024 to accommodate the square aspect ratio used in ViT pretraining, following the semantic segmentation linear readout setup described in iBOT (Zhou et al., 2021). In addition, ViT-based methods require sliding window inference in order achieve performance that is competitive with convolution- methods.
For ADE20K and WT-Sem linear readout, we simply use the respective BDD linear readout settings for ResNet and VIT methods. For ADE20K and WT-Sem UperNet finetuning, we follow the procedure described in iBOT (Zhou et al., 2021) except we use a batch size of for WT-Sem finetuning.
Ablation data sampling. For all ablation experiments, we employ repeated sampling like in MAE- st (Feichtenhofer et al., 2022) which samples R frames each time a video is encountered for faster data loading. Therefore, each pass through every video in the dataset counts as R epochs.
15

---

Published as a conference paper at ICLR 2025
B COMPUTE RESOURCES
The full model trained on 16 A100s and takes about 30h for 100 epochs on BDD100K or 18min per epoch. Walking Tours takes longer at 40min per epoch, the number of training samples per epoch is larger.
Ablation-sized experiments 20 20-40h depending on the configuration.
2 or 4 H100/A100 GPUs for total of 40 epochs, taking
C SUBCROP ANALYSIS
For the toy simulation of suberops, we place foreground object centered circle of varying size within a 256 x 512 frame. We then simulate all possible subcrops of area A E [0.02, 0.04, 0.06, 0.08]. For each subcrop area, we compute subcrop hits, i.e. whether least 5% of the subcrop contains the object, using numerical grid-based integration. We compute the subcrop hit probability, or subcrop hits over valid subcrops, averaged across subcrop areas, well the pixel probability, or object pixels over total image pixels.
We also emulate our training procedure for our empirical simulation of subcrops. For each of the 7,000 images in the BDDI00K semantic segmentation training dataset, we sample two global crops with area sampled from 10016,, 0.45) and for each global crop, 4096 subcrops with area sampled from (0002, 0.03]. We compute subcrop probability and pixel probability independently for the pixels of each foreground class: pole, traffic light, traffic sign, person, rider, car, truck, bus, train, motorcycle, bicycle. We then group the results into 10% quantile bins by object size (i.e. pixel proportions) and average the subcrop and pixel probabilities. We utilize slightly different subcrop area range in the empirical simulation because two-step global crop and suberop procedure results in logarithmic-lin distribution.
350%
Foreground
Background
300%
250%
Relative change
250%
200%
187%
200N
167%
160%
150%
138%
147%
108%,116% 116%
108%
100%
68%
50%
17%
-27N-557 -50%
0%
-50%
16%.25%
25%%
-0%
-22%
Road
Sky
Building
Vegetation
Car
Sidewalk
Fence
Terrain
Truck
Pole
Bus
Wall
Traffic Sign
Person
Traffic Light
Bicycle
Rider
Matorcycle
Train
Figure 11: Relative change in local crop region class assignments relative to per-pixel class distribu- tion.
We hypothesize that PooDLe's improvement on spatially underrepresented classes, as shown in Table 10, is due to this suberop effect. To quantify this effect real data, perform similar exercise as above on the BDD100K semantic segmentation training set. We sample subcrops following method and assign class label to each subcrop. If over 10% of the suberop is foreground class (not road, sky, building, vegetation, sidewalk, fence, terrain), then we label the subcrop as the majority foreground class. Otherwise, the majority background class label is assigned. In Figure II, we show the relative change in class distribution when using this subcrop class assignment. Foreground classes (green) increase in occurrence while background classes (blue) decrease frequency. besides road.
16

---

Published as a conference paper at ICLR 2025
D WALKING TOURS SEMANTIC BENCHMARK
20-
W1-3-mm ADE20K
aT ADX20K
uaiquu clasos
%uf pads
60
ADE20K
20
150
cank
rank
Frequency of classes by pixel
(b) Frequency of classes by frame
(c) Unique classes over time
Analysis of WT--Sem in comparison to ADE20K (Zhou et al., 2017) by frequency of each class by pixels occupied (a) frames present (b) and number of unique classes in each frame (c).
Figure 12:
We create the WT-Sem benchmark by sampling a frame every seconds from each of the 10 videos in WT as well as new walkaround videos. The new walkaround videos are filmed in Rome, Torun, and Poznan, sourced from the same YouTube channel as WT (Venkataramanan et al., 2024) under the Creative Commons (CC-BY) license. The Swin-L variant of OpenSeed (Zhang et al., 2023a), pretrained on COCO (Lin et al., 2014) and Objects365 (Shao al., 2019) and finetuned on ADE20K, used to generate semantic segmentation masks. We utilize the 25,910 frames sourced from WT as the training set and the 6,170 frames sourced from the 3 new videos as the validation set. Figure 12 shows analysis of WT-Sem in comparison ADE20K (Zhou et al., 2017), where observe that both datasets have long-tailed class distributions and WT-Sem has slightly higher number unique classes per frame. We also visualize examples from the WT-Sem benchmark in Figure 13.
17

---

Published as a conference paper at ICLR 2025
Figure 13: Visualizations of images and associated semantic segmentation masks from the WT--Sem benchmark.
18

---

Published as a conference paper at ICLR 2025
E ADDITIONAL VISUALIZATIONS
We provide additional visualizations of results on our evaluated benchmarks: BDD100K (Yu et al., 2020) semantic segmentation (Figure 14), object detection (Figure 15) and ADE20K semantic segmentation (Figure 16). Once again, we note that PooDLe produces segmentation maps with clearer boundaries while also effectively capturing small objects.
Image
Ground Truth
PooDLe
FlowE
DenseCL
DINO
DoRA
Supervised
BDD UperNet
Cityscapes Linear
Cityscapes UperNet
Figure 14: Visualizations of semantic segmentation masks for BDD linear readout, Cityscapes linear readout, and Cityscapes UperNet readout.
19

---

Published as a conference paper at ICLR 2025
Ground Truth
PooDLe
FlowE
Scratch
DenseCL
Supervised IN1K
Figure 15: Visualizations of object detection bounding boxes for BDD FPN readout.
Tani
Ruldd ta
brlding
entar
Image
Ground Truth
PooDLe (WT
DoRA (WT.))
DINO (IN1)
Figure 16: Visualizations of semantic segmentation masks for ADE UperNet finetuning.
F ACCURACY VALUES FOR CLASS BREAKDOWN AND ABLATIONS
Table 6: Accuracy values for Table class breakdowns
Method
Pretrain
All
Small
Large
Rare
Common
BDD
86.8
12.3
88.3
2.3
87.8
DenseCI.
86.6
86.0
DoRA
BDD
88.1
19.3
89.5
7.2
89.1
FlowE
BDD
88.5
18.2
89.9
32,0
89.2
PooDLe
BDD
89.2
33.6
90.3
34.2
89,9
Supervised
INIK
84.7
36.9
85.3
23.8
85.1
PooDLe
BDD
90.7
35.6
91.2
46.9
91.2
Table 7: Accuray values for Table 4, ablations
Variant
Dense
Pool
Top-Down
Lateral
Flow
Small
Large
Rare
Common
FlowF
RAFT
85.0
22.8
86.3
6.1
86.0
RAFT
86.2
14.2
87.6
6.3
87.1
1
RAFT
86,8
11.9
87.7
13.6
87.7
4
/
/
RAFT
86.6
22.1
87.9
16.9
87.5
/
RAFT
84,2
25.5
85.3
28.2
84.9
6 PooDLet
/
/
UFlow
86.0
26.4
87.2
29.6
86.7
PooDLe
/
/
RAFT
86.5
26.6
87.7
28.5
87.1
20

---

Published as a conference paper at ICLR 2025
G ADDITIONAL EVALUATION RESULTS
Table 8: Additional BDD semantic segmentation (SemSeg) and object detection (Det) readout evaluations. All settings are conducted with a frozen backbone. BYOL results are taken from FlowE (Xiong et al., 2021) and used DeepLab v3 (Chen et al., 2017) in-place of Upernet (Xiao et al., 2018). *Pretrained on BDD, initialized with supervised ImageNet weights.
BDD1OKK
BDD100K Obj.
Cilyscapes Seg
Method
Arch
Ep-
Pretrain
mloU
Linear
Acc
UperNet mloU
Ace
Det C4
MAP
MAP
FPN
mloU
Linear
Aoe
UperNet mloU
Acc
Scratch
R50
55.0
26.1
81.2
0.0
9.8
58.0
30.7
84.1
PoolLee
R30
100
BDD
39.2
89.2
49.9
91.8
4.9
25.2
47.2
90.2
60.7
93.5
Supervised
R50
600
INIK
36.7
84.7
55.2
92.0
3.6
24.9
46.8
87.4
63.4
93.7
PYOL-GGil al, m00mf
28.3
52.4
26.0
60.3
DenseCl. (Wang ot al., 20211)
R50
200
INIK
21.3
82.7
$2.8
91.6
0.3
25.0
27.3
84.0
63.7
93.7
Supervised
VIT-S
300
INIK
41.9
88.5
50.9
91.4
46.8
87.4
63.4
93.7
DINO
38.5
88.1
IBOT et 2021)
800
INIK
91.5
PoolXL
R30
100
BDD
44.7
90.7
54.1
92.7
3.9
28.0
52.0
91.5
65.1
94.4
We compare PooDLe against Imageeett-reetrained baselines in Table 8 and observe that PooDLe outperforms most baselines except iBOT and ImageNet supervised VIT. This result is encouraging, as pretraining on naturalistic video is more challenging due to spatial and class imbalance, yet is also a more realistic setting that enables the use of broader sets of usable data. Furthermore, we note that pretraining on dlass-balanced data such as ImageNet particularly benefits mloU, which weighs all classes equally despite some classes only appearing in a tiny proportion of pixels evaluation. Finally, PooDLe pretrained on BDD with weights initialized from the ImageNet supervised checkpoint surpasses all ImageNet-pretrainen baselines on linear semantic segmentation.
H PER-CLASS EVALUATION RESULTS
Table 9: loU per class on BDD semantic segmentation linear readout, *Pretrained on BDD, initialized with supervised ImageNet weights.
DENO
B010
68.6
4311
12.3
77.1
15.3
0.5
11.7
5%%
5.1
10.5
77%4
$1
0
19.6
134
17.7
DerneT1.
67.0
830
7.3
0
12.9
11.0
13.3
23.5
Fhont
800
968
72.9
75.8
79.5
50.8
11.9
23.5
22.7
15.3
43.7
32.4
12.9
11.8
26.7
4.1
13.9
12.1
Podllc
800
91.3
03.5
17.0
50.4
$1.7
14.
294
243
172
40.5
36.1
28.
18.0
352
29
25.6
21.2
Poalta
800
92.8
9401
843
362
949
324
118
022
25.7
27.1
41.
300
35.2
Table 10: Defined groupings and statisties of classes in the BDD semantic segmentation dataset. L-Large, S-Small, C=Common, R-Rare.
Trock
Will
S-mal
Tersin
Pol:
Bicyde
M-esck
Rider
Light
leg Phc 4 Tim. 13.0
18.2
150
14.4
84
37
34
33
311
3.1
2.8
3.1
1.0
0.8
8.7
0.6
0.
0ut
04
Total Soo Frag. Gep.
05
10
5.1
83
60
03
1.
C
R
C
R
R
We provide a breakdown of IoU per class on BDD semantic segmentation linear readout in Table 9. In Table 10, we also provide dataset-level statistics for each class computed over the training split of 7,000 images in the BDD semantic segmentation dataset, namely average pixel percentage per image, total percentage of pixels over the dataset and total percentage of images that they appear over the dataset. Size and frequency groupings are then independently defined using these statistics and used Table A class is considered 'Largge (I.) if its average pixel percentage per image is > 1% and 'Small' (S) otherwise. Separately, define class 'Common' (C) if the total percentage of images it appears is > 20% and 'Rare' (R) otherwise. Notably, PooDLe achieves significant gains small classes such 'Pole'. 'Bicycle', "Traffic Sign', "Traffic Light'. Methods trained on BDD underperform supervised INIK on classes rare in BDD such as 'Rider,, ; likely because IN1K offers both abundant and iconic images of these object categories.
21

---

Published as a conference paper at ICLR 2025
I BACKBONE COMPUTATION COST
We provide a table detailing the number of FLOPs associated with various SSL methods and backbones. We note that our SDM is by far the most efficient upsampling approach for dense representation learning methods.
Architecture
Associated Methods
GFLOPs
ResNct-50
DINO-R50
43.3
ResNet-50 SDM
PooDLe
60.5
ResNet-550 FPN decoder
PixPro
124.4
ResNet-50 dilated convolutions ViT-S/16
FlowE, DINO, DenseCL
200.7
DINO, iBOT, DoRA, MAE
82.9
Table 11: Comparison of different backbones, their associated methods, and computational require- ments in GFLOPs.
22

---

Published as a conference paper at ICLR 2025
J
FLOW VISUALIZATIONS
Source
Target
RAFT
UFlow
Figure 17: Comparison of predicted optical flow from RAFT (supervised) and UFlow (unsupervised).
In Figure 17, we compare the predicted flow maps generated from RAFT (Teed & Deng, 2020), an off-the-shelf supervised model, and our own unsupervised UFlow (Jonschkowski et al., 2020) model. The frame pairs are randomly sampled with At E [15, 30). We do note that self-supervised flow, particularly on BDD100K. may exhibit noisy or splotchy results. This possibly due the inconsistent motion and large dark regions that do not offer sufficient photometric supervisory signal. This is contrast to RAFT (Teed Deng, 2020) which learns sharp edges like from supervised labels. Nevertheless, we find that this self-supervised flow is sufficient for training PooDLe.
23


Reviews:
[{"review_id": "7H1rX7mLJn", "reviewer": "Reviewer_NLbx", "summary": "This paper introduces PooDLe (Pooled and Dense Learning), a self-supervised learning method for learning visual representations from naturalistic videos. PooDLe is a joint framework combining dense flow-equivariance learning with a pooled objective using pseudo-iconic subcrops. It shows state-of-the-art performance on multiple benchmarks while demonstrating strong performance on small objects and rare classes.", "questions": "1. What is the computational overhead of the SDM compared to simpler upsampling approaches?\n2. How sensitive is the method to video frame rate or motion blur?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The authors propose a novel approach to handling spatial imbalance in dense scenes through unified pooled and dense objectives. The creative use of flow-informed cropping to generate meaningful subcrops is insightful. \n2. The paper provides comprehensive experimental validation across multiple benchmarks, and the method demonstrates obvious improvements on challenging cases (small objects, rare classes).", "weaknesses": "1. Optical flow computation is typically computationally intensive, requiring expensive pixel-wise correspondence matching. It is important to discuss the computational costs of flow estimation. Limited discussion of real-time performance implications\n2. Limited analysis of inference time requirements. e.g. how inference time scales with input resolution.\n3. Missing discussion of potential memory bottlenecks from joint objectives. e.g. GPU memory requirements for different input sizes, and comparison with baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GxSY9Kbo7m", "reviewer": "Reviewer_HmA1", "summary": "This paper proposes an approach for self-supervision from video. The authors highlight the discrepancy between image datasets which typically contain a single central object and more challenging video data which is composed of complex scenes filled with many objects of differing sizes. The simpler image setting lends itself well to global embedding-based self-supervised methods, but such methods might not be ideal for representation of larger multi-object scenes.\n\nThe key idea of the paper is to mix dense and global objectives. For dense feature supervision, the authors follow FlowE (Xiong et al 2021) with one key difference being the use of a U-net style architecture to upsample and connect higher resolution features from earlier layers. The more \"global\" pooled-objective is applied to subcrops. Similar to how FlowE will match optical flow-warped dense features, subcrops are adjusted across frames by predicted optical flow to have a higher chance of containing the same subject. The motivation for learning on subcrops is for smaller, salient objects to take up a larger fraction of pixels. \n\nThe model is pretrained on BDD and Walking Tours and then segmentation and detection heads are trained on top of the frozen backbone. The resulting model outperforms other self-supervised approaches as well as ImageNet pretraining.", "questions": "Overall the paper is solid so this is again fairly minor, but I have a bunch of cropping related questions as that's the one area I had the most confusion:\n- Where exactly does the global cropping fit into all of this, what's the point of defining a global crop at all?\n- In Figure 7, I'm not sure what it means to change the number of subcrops, is it changing the proportion of correlated samples in a given minibatch? What is the method with 0 subcrops - is it using the global crop as input into the encoder?\n- Maybe I missed it, but is there ever an ablation on the importance of the subcrops being flow matched? Intuitively seems like a reasonable thing to do but does it matter much?\n- In Table 4, is the FlowE result trained with the proposed cropping approach? How does this compare to FlowE without using subcrops?\n- Is the training done on 192x192 crops but testing at 512x512, is there any issue with this discrepancy or does the model generalize well enough to the larger test time resolution?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a solid paper, well motivated introduction, straightforward idea, outperforming other methods with the appropriate ablations to break down different design decisions. Each of the key ideas behind the approach are sensible, I am surprised it is not already common to perform dense feature self-supervision on U-net/FPN style features.\n- I appreciate the level of specific, little details throughout the paper to understand exactly how things were implemented\n- it's interesting to see that both losses only start to play a role after a clearer distinction has been made between the pooled \"global\" and upsampled dense features (Table 4)\n- also cool to see that the method performs well with imperfect self-supervised flow", "weaknesses": "This is minor, but I am a bit confused about the subcrops. When reading I was expecting some mechanism for smarter choice of crops such that there's a higher probability of including an object of interest. For example, some sampling based on saliency or flow that would ensure more crops included people/bikes/etc, but it doesn't seem like that's the case? Instead, crops are chosen randomly and uniformly across the image so they are just as likely to sample a patch from the sky as from the street. The key difference as far as I understand is that by virtue of doing any cropping at all an object of interest will now take up a greater fraction of pixels, and using flow ensures the object will likely be included in both crops. Feels somewhat at odds with how the paper has been written and motivated which emphasizes putting extra attention on smaller objects as opposed to background context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YcTyi4htjB", "reviewer": "Reviewer_TTAd", "summary": "The paper introduces PooDLe, a self-supervised learning (SSL) method tailored for complex, naturalistic video data with densely packed scenes and diverse object scales. Unlike standard SSL methods trained on iconic images, PooDLe addresses the spatial imbalance in cluttered video data by combining a dense SSL objective that maintains equivariance through optical flow warping with a pooled objective that captures semantic information from smaller, aligned subcrops. This dual-objective approach, augmented with a spatial decoder module (SDM), is shown to effectively represent both large and small objects.\n\nPooDLe’s performance is validated on driving and first-person video datasets, achieving state-of-the-art results in semantic segmentation and object detection tasks, particularly excelling with small object recognition. Additionally, the paper introduces the Walking Tours Semantic (WT-Sem) segmentation task, presents a detailed analysis of class size and frequency on BDD100K, and studies the impact of global crop area, input resolution, and temporal stride on learning efficacy, highlighting key factors for SSL with naturalistic data.", "questions": "- The explanation for why combining SDM with Pool helps is not clear. Also, it is not clear why just pool should not help, its even hurting the small objects. SDM has additional parameters, can that be a reason (of course the fine-grained features will help too), SDM is not even connected with Pool.\n\n- In figure 4, it shows finer boundaries for smaller objects, but boundaries for bigger objects, such as car, is getting worse in comparison to other methods, also performance seems similar to the baseline approach FlowE.\n\n- L075: \"These subcrops serve as pseudo-iconic views of foreground objects, functionally increasing the prevalence of smaller objects\". It is not clear why taking random crops will increase prevalence of smaller objects, parts of bigger objects or backgrounds will have higher probability among these crops.\n\n- Ablations are not comprehensive, only one dataset, one metric, and one task is shown. It will be good to see accuracy metric, and also how these components perform with any other dataset. BDD is street-view, maybe Ego or some other natural videos, CCTV, etc. It will be good to see the ablations on the walkingtour dataset along with BDD. It seems to be working fine with driving videos, since scene will not change a lot, but what about other videos? Like ego-centric? Or static camera?\n\n- What about performance just with pool? It will be good to see if just learning from the crops are enough. SDM improves, but it comes with additional parameters too.\n\n- Since this work is build on top of FlowE, there should be results for FlowE on other datasets too (Table 2).\n\n- It is not clear why results were shown only with ResNet, why not ViT?\n\n- Will this method work on other simple tasks like classification?\n\n- It will be good to see accuracy in Table 3 and 4.\n\n- Using additional crops for training adds training overhead, there is no analysis shown on this. \n\n- Figure 9, it will be good to see t between 0 and 15.\n\n- The role of predictor and projector is not clearly explained, projector is not even shown. It is not clear.\n\n- L205: why crop features are averaged-pooled over spatial dimension? Fine-grained constraint should be better?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written with a clear structure to demonstrate a clear and logical progression of motivation and ideas. The authors use precise and well-chosen words that enhance readability and engagement.\n\n2. The motivation to combine pooled representations and dense local object representations is well justified. The proposed method aligns well with the motivation.\n\n3. The paper provides comprehensive/detailed analysis, testing the proposed method across various scenarios and datasets. This thorough experimental validation successfully demonstrates the method’s effectiveness on small and rare objects.", "weaknesses": "1. The main weakness of this work is lack of enough ablations for the proposed method. It is not clear (with the shown ablations) if local crops really are useful. Considering this is the main contribution in terms of novel idea, it should have been made clear. Results with just local crops might help, since adding these local crops on dense does not improve, but hurt smaller objects. It will be good to see accuracy on this and on datasets other than BDD as well.\n\n2. Although there are several experiments presented, the technical contributions seems little on the weaker side, the method is build on top of FlowE, local crops is novel (but its contribution not clear), the improvement on dense consistency is expected due to use of features from earlier layers, \n\n3. The proposed method shows some improvement over existing methods, but looking at a comparison with the baseline (FlowE), mostly its incremental (less than 1%) if we look at accuracy. That is why it is important to loot at this metric in the ablations. Also, this baseline is not shown for the other dataset (WalkingTour), where the proposed method do not outperform other methods across the board.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1yXkfjR7wg", "reviewer": "Reviewer_jDP3", "summary": "Self-supervised learning for naturalistic videos is challenging with the presence of independent objects of various sizes, imbalanced class distribution, etc.\nThe paper aims to tackle these problems by combining global and dense supervision, as well as spatial decoder modules (SDM).\nThe dense objective could help capture different object instances from the videos while the global objective could help learn the semantics of the objects using contextualized signals.\nThe proposed method, PooDLe, was evaluated on representative video benchmarks, e.g. BDD100K, Walking Tours, ADE20K, Cityscapes, demonstrating superior performance on dense prediction tasks, i.e. semantic segmentation and object detection.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "i) Method. The proposed method, PooDLe, is technically sound.\n\nii) Performance. Compared to prior global or dense-only works, PooDLe demonstrated superior performance on dense prediction tasks like semantic segmentation and object detection.\n\niii) Ablation. The paper further verified that PooDLe improved the performance of small and rare objects, justifying the advantage of combining global and local supervisions, as well as the spatial decoder modules (SDM). It also provided ablations on the pretraining datasets, different components, and augmentation strategies, e.g. crop area, resolutions, etc.", "weaknesses": "The reviewer has several concerns and questions about the evaluation, ordered by their impact on the final score.\n\n---\nConcerns:\n\ni) About SDM. SDM introduced extra operations on higher resolution feature maps or images. From L306-307, SDM was also used in the inference stage for semantic segmentation. In this case, the inference overhead is determined by both the backbone architectures (e.g. ViT-S, ResNet50), and the UNet based SDM. The reviewer wonders if the baseline approaches also have SDMs during inference? Were they of the same architectures, complexities (e.g. FLOPs)? If they are not of the same complexity, could the authors provide the FLOPs of each baseline and PooDLe. This could show if the comparisons of different methods on semantic segmentation are apples-to-apples, especially for inference.\n\nii) Some comparisons are apples-to-oranges. At L358-L360, it was claimed that PooDLe outperformed DoRA across different settings. If the reviewer understood correctly, the comparisons there were between PooDLe_ResNet50_epoch20 and DoRA_ViT-S_epoch100, which were not fully convincing. The authors are encouraged to present apples-to-apples comparisons under the same setting, e.g. backbone, #epoch, or remove the claims otherwise.\n\n---\nQuestions\n\niii) Tasks beyond dense prediction. It was shown that augmenting dense supervisions with global objectives could help improve dense prediction tasks. The reader may also be curious if augmenting global supervisions with dense objectives could help improve global prediction tasks, e.g. action recognition, etc.\n\niv) About the \"relationship between global crop area and input resolution\". It was claimed in Contribution 3 (L101-102). However, there seems to be no explicit analysis of the \"relationship\" between \"crop area\" and \"resolution\", or the relationship could not be easily interpreted from Figure 5. To reveal the relationship between \"crop area\" and \"resolution\", the authors could consider analyzing extra metrics derived from them, one example could be the pixel-scale, i.e. crop_area/input_resolution, defined in [1].\n\nv) Patch size used for ViT-S, which determines the FLOPs of some baseline approaches\n\nvi) The videos of Walking Tours (WT) are at 4K+60fps, instead of 720p+30fps (L248-249)?\n\n[1] Effective Self-supervised Pre-training on Low-compute Networks without Distillation. ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)