PooDLe🐩: Pooled and dense self-supervised learning from naturalistic videos

## Abstract

Abstract Self-supervised learning has driven significant progress in learning from single-subject, iconic images.
However, there are still unanswered questions about the use of minimally-curated, naturalistic video data, which contain dense scenes with many independent objects, imbalanced class distributions, and varying object sizes.
In this paper, we propose a novel approach that combines an invariance-based SSL objective on pooled representations with a dense SSL objective that enforces equivariance to optical flow warping.
Our findings indicate that a unified objective applied at multiple feature scales is essential for learning effective image representations from high-resolution, naturalistic videos.
We validate our approach on the BDD100K driving video dataset and the Walking Tours first-person video dataset, demonstrating its ability to capture spatial understanding from a dense objective and semantic understanding
via a pooled representation objective.

## 1 Introduction

Humans and other animals learn visual understanding from a continuous stream of inputs with little explicit supervision.
Recently, self-supervised learning (SSL) [, , , , , , , , ] has made great strides in learning without human annotations, becoming competitive with supervised learning. However, most methods still revolve around ImageNet [] which is implicitly supervised through *iconic* images that contain an unambiguous subject sampled from a balanced class distribution.
In contrast, naturalistic vision data such as egocentric videos, contain cluttered scenes, imbalanced
classes and objects of varying size, making them ill-suited for existing iconic methods.

Nevertheless, these naturalistic videos are still valuable for their information density and ease of collection, while also mimicking the real-life perspective of humans.
Unfortunately, the aforementioned iconic methods, which learn to align pooled representations of an image, may perform poorly as dense scenes often produce views depicting unrelated subjects that are semantically incompatible (Figure 1, red boxes).
Recent works have attempted to address this weakness by introducing mechanisms to account for multiple subjects, such as new cropping [] or attention [, ] strategies.
Others [, ] have also proposed variations using a “dense SSL” objective, which does not pool image representations and instead identifies correspondences between views before computing an SSL loss for each corresponding region.

While dense SSL methods avoid the problem of aligning semantically incompatible views,
and can learn fine-grained object boundaries [], they are susceptible to other issues.
Computing the loss at each spatial location
presents a spatial imbalance problem, where larger background classes such as the sky dominate the representation while smaller classes such as pedestrians are underrepresented.
This effect is undesirable because foreground objects, which are often smaller, should be favored over low-detail, repetitive background classes which often occupy more space.
Furthermore, ignoring small foreground objects
can be acutely hazardous in applications like self-driving [] where critical objects, such as traffic lights and pedestrians occupy less than $0.3\%$ of a video frame (Figure 1, green boxes and 1).
Preserving these crucial details is particularly necessary in the SSL, where there are no labels to assign importance.
This is in contrast to ImageNet [] training, where models that can easily learn semantics from iconic data with clear, single-subject views and a balanced class distribution.
Some dense SSL methods [, , ]
also include losses that optimize a global, pooled representation
to learn semantic information from dense scenes, but do not explore how to integrate these two objectives through means of architecture and augmentation strategies.

To address problems of cluttered scenes and spatial imbalance when learning from dense, naturalistic videos,
we propose a joint Pooled and Dense Learning (PooDLe) method that optimizes a dense SSL objective
over representations without pooling and a secondary pooled objective applied to smaller, content-aligned views.
The combination of dense and pooled objectives captures both scene-level understanding and fine-grained boundaries, as well as high-level semantics and small objects.
Like FlowE [], our dense objective encourages the alignment of dense feature maps using optical flow warping.
We also use a flow-informed cropping procedure to generate pairs of smaller views, or “subcrops”, with improved subject correspondence between each subcrop pair when training the pooled loss.
Depicted in Figure 1,
these subcrops often serve as pseudo-iconic
views of foreground objects, which functionally increase the prevalence of smaller foreground classes in our dataset.
We also introduce a lightweight spatial decoder module (SDM), comprised of top-down decoder layers and UNet []-like lateral connections to upsample high-level semantic representations to preserve smaller objects in our dense objective.
We find the inclusion of both objectives along with the SDM is essential to capture the semantics of smaller objects and to achieve overall strong downstream task performance.

We conduct experiments on the BDD100K [] dataset of dashcam driving videos and the recent WalkingTours dataset of first-person walking videos [].
PooDLe achieves state-of-the-art performance on downstream semantic segmentation and object detection benchmarks, with a notable gain on recognizing small objects.
In our ablations, we demonstrate that our joint objective formulation and the SDM are critical for success.
In addition, our approach maintains strong performance
over a wide range of crop areas and input resolutions while the pooled or dense objective alone quickly degrades

In summary, our contributions are as follows:

- 1.
We introduce PooDLe, a method combining a dense SSL objective based on flow equivariance and a pooled objective using small pseudo-iconic subcrops. PooDLe unifies the two objectives with a spatial decoder module
to combine the strengths of both objectives.
- 2.
When pretrained on naturalistic videos, PooDLe achieves state-of-the-art downstream performance on BDD100K [], ADE20K [] and Cityscapes [] semantic segmentation, and BDD100K object detection tasks. PooDLe outperforms prior dense SSL methods [, ],
particularly on small objects, and is competitive with ImageNet pretraining.
- 3.
We study the effects of crop area, resolution and temporal stride between paired frames. We demonstrate the importance of maintaining pixel density when increasing image resolution for the dense SSL objective.
We also perform analysis to show that subcrops can increase prevalence of small objects in the pooled objective by serving as pseudo-iconic views, which supports our experimental findings.

Figure: (a) Iconic image
Refer to caption: x2.png

## 2 Related Work

#### Self-supervised learning with iconic images.

Representation learning on iconic image datasets has a long history from denoising autoencoders [] to joint embedding methods [, , , , , ] to joint-embedding predictive architectures [, ].
Joint embedding methods learn representation invariance to visual changes created by augmentations using contrastive [, ], mean squared error [], or classification [, ] losses between corresponding pairs, pushing SSL to new heights on ImageNet classification.
Later works have extended these methods to internet-scale data [] and included other modalities like text [].
Separately, the MAE [] family of methods learn via reconstruction of masked image regions.
iBOT [] combines joint embedding methods with token reconstruction to produce impressive results on ImageNet classification.
[, , , ] all generate small, low-resolution crops (‘multi-crop’) that are optimized to predict the representations of global crops for iconic training examples for little additional compute.
This is different from our subcrop strategy which produces aligned image pairs of small crops as pseudo-iconic views from otherwise dense scenes.

#### Training using dense multi-subject images.

Following the success of SSL on ImageNet, other works seek to learn from dense, multi-subject images where augmented views may not contain corresponding subjects for invariance learning.
[, , ] build upon joint-embedding methods, relying on similarity of features bootstrapped from typical invariance learning to identify positive pairs across dense, unpooled feature maps.
[, ] optimize dense losses, contrasting pixels belonging to different semantic classes; these methods require off-the-shelf segmentation modules.
[, ] utilize DINO [] attention maps to identify training pairs, while ADCLR [] identifies pairs using small ‘query’ crops and the patches that attend to them.
All of these methods make strides towards learning from dense images with multiple objects.
However, these methods do not consider the effects of a dense objective on per-class accuracy and the skewed class and spatial distributions of more naturalistic data.

#### Learning image representations from video data.

Extending beyond images, other works have sought to capture the variance of objects through time by training on pairs of video frames.
Gordon et al. [] adapts contrastive learning to use correlated frames as positive examples, while [, ] identify positive pairs using high representational similarity.
FlowE [] builds on BYOL [] and identifies positive spatial regions between frames using off-the-shelf flow.
Most recently, DoRA [] proposes a new dense video dataset and extends DINO by clustering over many frames to identify and track objects for representation learning.
In the MAE paradigm,  [, ] directly reconstruct sequences of frames while [, ] perform reconstruction given a corresponding overlapping frame.
MC-JEPA [] learns motion using video data by aligning latent representations throughout the feature pyramid while performing representation learning on ImageNet while VJEPA [] directly aims to learn video representations via latent video prediction.
Like some methods above, PooDLe learns a rich image representation from video data and does so using a dense and pooled objective to capture both scene-level spatial understanding and high-level semantics.

Figure: Figure 2: PooDLe combines pooled and dense learning objective for video self-supervised learning. Green path: dense objective performing flow-equivarience learning on the output of the decoder $g(\cdot)$. Orange path: pooled objective encoding $K$ subcrops sampled with flow-informed local cropping. Projector modules are omitted for simplicity. Offline parameter $\xi$ are the exponential moving average (ema) of online parameters $\theta$.
Refer to caption: x5.png

## 3 PooDLe : Poo led and D ense Le arning from naturalistic videos

We present PooDLe, a self-supervised method for learning dense visual representations using paired frames from naturalistic, first-person videos.
PooDLe combines two SSL objectives: a *dense* objective for learning representations of dense, crowded scenes; and a *pooled* objective on small subcrops sampled using flow-aware cropping augmentations.
We also propose a lightweight spatial decoder module (SDM) that uses top-down decoder layers and UNet-like lateral connections to earlier encoder representations to both upsample the high-level representations and resurface fine-grained details and small objects that may get lost in downsampling operations.
For a high-level overview of PooDLe, see Figure 2.

#### Preliminaries.

Inputs to the model are video frame pairs $\bm{x}_{t},\bm{x}_{t+\Delta t}$ with dimensions $H\times W$, and dense optical flow $M_{t\rightarrow t+\Delta t}$ predicted by an external optical flow model.
Randomly sampled augmentations $A_{1}$ and $A_{2}$ are applied to each example to create positive training pairs. In a similar setup to BYOL, the encoder and projector are denoted as a function $\bm{p}=f(\bm{x})$ using either online weights $\theta$ or offline, EMA-updated weights $\xi$.
The predictor module $q_{\theta}(\cdot)$ only has online weights $\theta$.
Separate projector and predictor modules are used for the pooled and dense objectives, but are not annotated for simplicity.
We use a ResNet-50 backbone, as well as projectors and predictors following FlowE [] and BYOL [], that are discarded after pre-training.

#### Dense SSL with flow equivariance.

The dense objective follows FlowE [] by using optical flow $M_{t\rightarrow t+\Delta t}$ to align paired feature projections $\bm{p}_{t}$ and $\bm{p}_{t+\Delta t}$.
At a high-level, this objective minimizes differences in representation between corresponding regions.
More specifically, the inverse augmentation functions $A^{-1}$ and optical flow are used to align the representations $\bm{p}$ and after upsampling to input resolution $H\times W$, the objective is the squared error:

$$ $\displaystyle\mathcal{L}_{\text{dense}}=\frac{1}{HW}\left\lVert q_{\theta}(A_{ 1}^{-1}(\bm{p}_{t}))-(M_{t\rightarrow t+\Delta t}\circ A_{2}^{-1})(\bm{p}_{t+ \Delta t})\right\rVert^{2}_{2},$ (1) $$

where normalization is applied after the predictor and flow warping.

#### Pooled objective with flow-informed subcrops.

First we identify $K$ pseudo-iconic subcrop pairs.
Unlike for iconic data, random crops from paired frames are unlikely to contain a common subject.
To mitigate this, we once again use optical flow in a flow-informed cropping procedure to identify aligned training pairs.
For each subcrop pair, we sample a random point $(u,v)$ in the target frame $\bm{x}_{t+\Delta t}$ to serve as the crop center.
It is then warped into the earlier frame $\bm{x}_{t}$ using flow $M_{t\rightarrow t+\Delta t}$ plus random jitter $(\delta_{u},\delta_{t})$ for paired center $(u^{\prime},v^{\prime})$.
A crop is made around each center, with an area sampled from $U[s_{\text{min}},s_{\text{max}}]$ of the global crop for subcrops $\bm{x}_{t,i}$ and $\bm{x}_{t+\Delta t,i}$.

As we require both crop centers to land within the bounds of the image, subcrops tend to be center-biased [] and lack diversity.
To remedy this, we employ a grid-sampling procedure for selecting the initial crop center $(u,v)$.
Each global crop $\bm{x}$ is divided into a grid with cells of side length $d_{\text{grid}}=\text{min}(H,W)\times\sqrt{(s_{\text{min}}+s_{\text{max}})/2}$ for a $H/d_{\text{grid}}\times W/d_{\text{grid}}$ grid.
Each cell is selected without replacement and a center $(u,v)$ is then uniformly sampled within the cell.

After $K$ pairs $(x_{t,k},x_{t+\Delta t,k})$ are generated, they are encoded via the backbone and the pooled objective projector.
Unlike in the dense objective, no alignment or upsampling is performed, and each projection $\bm{p}$ is averaged-pooled over its spatial dimensions before computing the loss:

$$ $\displaystyle\mathcal{L}_{\text{pool}}=\frac{1}{K}\sum_{k}^{K}\left\lVert q_{ \theta}(\bar{\bm{p}}_{t,k})-\bar{\bm{p}}_{t+\Delta t,k}\right\rVert^{2}_{2},$ (2) $$

where $\bar{\cdot}$ denotes average pooling over spatial dimensions with normalization performed after.
Our approach trains each subcrop to predict its corresponding pair, likely containing the same objects in dense scenes.
This differs from multi-crop [], which has local crops predict global crops — potentially less effective for dense scenes where local crops may only capture a subset of objects.

#### Spatial Decoder Module (SDM).

Figure: (a) Baseline
Refer to caption: x7.png

We introduce SDM (Figure 3(b)) to both upsample high-level encoder features and preserve information from lower layers, particularly any smaller foreground objects that may be lost due to pooling operations.
Its design draws inspiration from convolutional UNet [] and FPN [] and improves upon FlowE’s use of dilated convolutions to replace pooling by efficiently maintaining high-resolution representations while significantly reducing activations and memory usage.

The SDM utilizes decoder blocks, each consisting of an *upsample* operation, a computation block of processing layers $g(\cdot)$, and a UNet-like *lateral* connection.
The output of each block is computed as:

$$ $\displaystyle\bm{z}^{l+1}=g(upsample(\bm{z}^{(l)}))+lateral(\bm{z}^{j}),$ (3) $$

where $\bm{z}^{(l)}$ is the representation after the $l^{\text{th}}$ encoder stage and $\bm{z}^{(j)}$ is an earlier feature map of the same spatial dimensions as $\bm{z}^{(l+1)}$.
The use of computation blocks and lateral connections are ablated in Table 4.2.
Figure 3 depicts a naive implementation that places both objectives at the top encoder level and PooDLe, which uses the SDM to integrate the two objectives in a complementary fashion.

## 4 Experiments

We pretrain PooDLe on raw videos from BDD100K [] and Walking Tours (WT) [] and evaluate them with semantic segmentation and object detection benchmarks.
The BDD100K pretrained model is evaluated on the corresponding in-domain benchmarks as well as Cityscapes [] and the Walking Tours model on ADE20K [].
We also ablate our combination of loss functions and decoder components as well as the effects of crop area and input resolutions.

### 4.1 Experiment Setup

#### Pretraining datasets.

1) BDD [] consists of 100,000 dashcam driving videos collected in various weather conditions and times of day from New York and the San Francisco Bay Area.
Each video is  40 seconds long at 720p and 30 fps.
We pretrain with the 70,000 videos in the official training split and evaluate on the semantic segmentation and object detection tasks proposed alongside the dataset.
2) Walking Tours (WT) [] is a dataset of first-person egocentric videos collected from YouTube of a continuous walkaround through various cities of Europe, Asia, and a wildlife safari.
There are 10 videos, ranging from 59 minutes to 2 hours 55 minutes, at 720p and 30 fps.
Each video contains large numbers of unique objects per frame and natural transitions in lighting and location.
For self-supervised pretraining, we use either the Venice video ($\text{WT}_{\text{Venice}}$) or all 10 videos ($\text{WT}_{\text{all}}$) following DoRA [].

#### Technical details.

We use ResNet-50 (R50) [] as our feature encoder, with the dense projector and predictor networks following FlowE [] and pooled counterparts following BYOL [].
For SDM, we use two decoder stages, with each consisting of a $2\times$ upsample, a ResNet Bottleneck Block [] for computation, and a 2-layer convolutional MLP for the lateral connection.
When training on BDD, we sample two frames that are $0.5\sim 1$ seconds apart ($\Delta t\in\{15...30\}$) from each video.
We then take two large crops from the same image coordinates of area
$[0.16,0.45]$ of the original image and resize them to $512\times 1024$ pixels before applying augmentations.
For each training epoch on WT, we divide each video into 10-second clips and randomly sample two frames $0.5$ seconds apart from each clip, and use crop area range $[0.65,1.0]$.
For non-spatial augmentations on both datasets, we apply color distortion and Gaussian blurring independently to each frame following BYOL [].
For the dense objective, we also apply a random reversible affine transformations similar to FlowE []: random scaling of 0.9–1.1$\times$ and rotation of -10–10.
For the local objective, we sample $K=6$ subcrop pairs and use a crop area of $[0.05,0.3]$ of the initial dense crop for both BDD and WT, resized to $192\times 192$.
When sampling supcrops, random spatial jitter is $\pm 10\%$ of the large crops’ height and width.
We attempt experiments with ViT-S, but observe a collapse phenomenon (see Appendix G).

#### Baselines.

We use official implementations of DenseCL, PixPro, DINO, iBOT and DoRA, and our own implementation of FlowE for pretraining on BDD.
We use torchvision for ImageNet supervised and weights released online for ImageNet-pretrained DINO.
We obtain weights from the authors of DoRA for iBOT, DINO - ViT, and DoRA pretrained on WT and use official implementations of DINO - R50 and PixPro for pretraining on WT.
For PixPro, we use its FPN decoder only in pretraining, which we find is best for performance.
We use 512x1024 crops for training FlowE and DINO-R50; we are unable to train the other baselines with high-resolution crops due to prohibitive GPU memory requirements.

#### Evaluation.

We adopt the evaluations described in FlowE [] for BDD and Cityscapes.
We use DeepLab v1 [] as the ‘linear’ readout header and UperNet [] as the heavier readout head for semantic segmentation and Faster R-CNN with ResNet-C4 and Faster R-CNN with FPN [] as the standard and heavier readout headers for object detection.
We do not include ViT object detection due to the lack of an existing reliable recipe.
For semantic segmentation on ADE20K, we perform both linear readout following BDD and Upernet finetuning as described in iBOT [].
We retain the SDM when evaluating PooDLe on semantic segmentation with linear readouts.
We report mean intersection-over-union (mIoU), pixel-level accuracy (Acc), and mean average precision (mAP) as our evaluation metrics.
Additional details on implementation and hyperparameters are provided in Appendix A.

### 4.2 Main Results

**Table 1: BDD semantic segmentation (SemSeg) and object detection (Det) readout evaluations. All settings are conducted with a frozen backbone. *Pretrained on BDD, initialized with supervised ImageNet weights.**
|  |  |  |  | BDD100K Sem. Seg. | BDD100K Obj. Det. | Cityscapes Sem. Seg |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  | Linear | UperNet | Det C4 | FPN | Linear | UperNet |  |  |  |  |
| Method | Arch | Ep. | Pretrain | mIoU | Acc | mIoU | Acc | mAP | mAP | mIoU | Acc | mIoU | Acc |
| Scratch | R50 | - | - | 9.7 | 55.0 | 26.1 | 81.2 | 0.0 | 7.7 | 9.8 | 58.0 | 30.7 | 84.1 |
| DenseCL [] | R50 | 800 | BDD | 21.3 | 82.6 | 46.9 | 91.4 | 0.9 | 23.4 | 25.1 | 84.8 | 57.2 | 93.0 |
| FlowE [] | R50 | 100 | BDD | 35.7 | 88.5 | 47.3 | 91.5 | 3.2 | 23.8 | 43.1 | 89.5 | 57.7 | 93.1 |
| PixPro [] | R50 | 100 | BDD | 21.8 | 80.0 | 37.3 | 88.0 | 0.7 | 18.4 | 25.5 | 81.0 | 44.3 | 89.5 |
| DINO [] | R50 | 100 | BDD | 13.1 | 64.7 | 25.6 | 80.3 | 0.3 | 11.9 | 14.9 | 69.4 | 29.2 | 81.4 |
| DINO [] | ViT-S | 300 | BDD | 29.6 | 86.8 | 41.1 | 90.1 | - | - | 35.1 | 87.9 | 51.5 | 91.9 |
| iBOT [] | ViT-S | 800 | BDD | 27.2 | 85.4 | 35.5 | 88.7 | - | - | 32.0 | 86.2 | 44.0 | 90.3 |
| DoRA [] | ViT-S | 200 | BDD | 33.2 | 88.1 | 43.3 | 90.7 | - | - | 37.4 | 88.7 | 50.8 | 92.0 |
| PooDLe | R50 | 100 | BDD | 39.2 | 89.2 | 49.9 | 91.8 | 4.9 | 25.2 | 47.2 | 90.2 | 60.7 | 93.5 |
| Supervised | R50 | 600 | IN1K | 36.7 | 84.7 | 55.2 | 92.0 | 3.6 | 24.9 | 46.8 | 87.4 | 63.4 | 93.7 |
| PooDLe | R50 | 100 | BDD* | 44.7 | 90.7 | 54.1 | 92.7 | 3.9 | 28.0 | 52.0 | 91.5 | 65.1 | 94.4 |

Figure: Figure 4: Visualization of BDD semantic segmentation linear readout. PooDLe is able to identify smaller objects and generate cleaner object boundaries.
Refer to caption: x9.png

#### BDD100K-pretrained models.

We evaluate the learned representations on semantic-segmentation and object-detection on the BDD in-distribution, labeled benchmark (Table 1).
PooDLe achieves superior performance on all readout tasks compared to other models pretrained on BDD, outperforming the strongest baseline FlowE by $3.5\%$ mIoU on linear and $2.6\%$ mIoU on UperNet for semantic segmentation, and $1.7\%$ mAP on C4 and $1.4\%$ mAP on FPN for object detection.
This indicates that PooDLe is able to learn better representations for dense prediction tasks.
While we keep the decoder in linear readout, it is discarded during UperNet probing, showing that the representation improves substantially over all other BDD-trained models ($49.9$ mIoU compared to next-best $47.3$).
We demonstrate in Table 3 that our performance improvements are derived from better recognition of small classes.
We also evaluate the downstream transfer of PooDLe representations by performing linear and UperNet readout on the Cityscapes benchmark with models trained on BDD.
PooDLe readily outperforms all baselines.
We provide visualizations of predicted segmentation masks in Figure 4 and additional evaluation visualizations in Figure 12 and Figure 13.

PooDLe outperforms ImageNet supervised pretraining as well, despite the latter’s advantage in learning small and rare classes present in BDD100K (spatial imbalance shown in Figure 1) due to ImageNet being a class-balanced dataset with iconic views of objects.
In addition, we may pretrain PooDLe on BDD100K with weights initialized from the ImageNet supervised checkpoint to further improve performance, notably boosting mIoU by $8\%$ and Acc by $6\%$ on linear semantic segmentation.
We provide additional comparisons to ImageNet-pretrained baselines in Appendix E, where PooDLe remains competitive despite being pretrained in the challenging naturalistic video setting.

**Table 2: ADE20K semantic segmentation linear readout and finetuning evaluations. Linear readout is performed with a frozen backbone while in finetuning, backbone parameters are trainable. ^† DINO-ViT and iBOT results are taken from DoRA [].**
| Method | Arch | Epoch | Pretrain | SemSeg Linear | Finetune |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| mIoU | Acc | mIoU | Acc |  |  |  |  |
| DINO [] | R50 | 800 | IN1K | 15.7 | 61.5 | 43.0 | 80.5 |
| DINO []^† | ViT-S | 100 | IN1K | - | - | 33.9 | - |
| PixPro [] | R50 | 100 | $\text{WT}_{\text{Venice}}$ | 4.6 | 48.6 | 36.0 | 77.6 |
| DINO [] | R50 | 100 | $\text{WT}_{\text{Venice}}$ | 6.9 | 48.2 | 35.7 | 77.4 |
| DINO [] | ViT-S | 100 | $\text{WT}_{\text{Venice}}$ | 7.8 | 57.7 | 29.2 | 74.7 |
| iBOT []^† | ViT-S | 100 | $\text{WT}_{\text{Venice}}$ | - | - | 33.9 | - |
| DoRA [] | ViT-S | 100 | $\text{WT}_{\text{Venice}}$ | 14.1 | 63.5 | 35.2 | 77.7 |
| PooDLe | R50 | 20 | $\text{WT}_{\text{Venice}}$ | 14.6 | 59.0 | 36.6 | 77.9 |
| DoRA [] | ViT-S | 100 | $\text{WT}_{\text{all}}$ | 13.9 | 64.4 | 38.3 | 79.3 |
| PooDLe | R50 | 20 | $\text{WT}_{\text{all}}$ | 16.5 | 63.9 | 41.0 | 79.6 |

#### WT-pretrained models.

We also train PooDLe on $\text{WT}_{\text{Venice}}$ as well as the entire dataset, $\text{WT}_{\text{all}}$.
Table 2 shows linear readout and finetuning experiments on ADE20K [] semantic segmentation following Venkataramanan et al. [].
Notably, when pretrained on $\text{WT}_{\text{all}}$, PooDLe surpasses DoRA by $2.6\%$ mIoU on linear readout and $2.7\%$ mIoU on UperNet finetuning over DoRA.
PooDLe also shows some improvement trained on $\text{WT}_{\text{Venice}}$-pretraining setting, with a gain of $0.5\%$ mIoU on linear readout over DoRA and $0.6\%$ mIoU on UperNet finetuning over PixPro.
This shows that PooDLe also learns strong representations from naturalistic video captured in the open world.
We show visualizations of predicted segmentation masks for ADE20K evaluations in Figure 14.

**Table 3: Breakdowns of mIoU over different class groupings. Linear readout mIoU is computed over various groupings of the 19 classes in BDD semantic segmentation. *Pretrained on BDD, initialized with supervised ImageNet weights.**
| Method | Pretrain | All | Small | Large | Rare | Common |
| --- | --- | --- | --- | --- | --- | --- |
| DINO | BDD | 29.6 | 8.4 | 42.0 | 1.0 | 42.8 |
| DenseCL | BDD | 21.7 | 1.6 | 33.4 | 0.0 | 31.7 |
| DoRA | BDD | 33.2 | 11.9 | 45.6 | 2.8 | 47.3 |
| FlowE | BDD | 35.7 | 12.2 | 49.3 | 10.7 | 47.2 |
| PooDLe | BDD | 39.2 | 18.3 | 51.4 | 12.0 | 51.8 |
| Supervised | IN1K | 36.7 | 27.2 | 42.2 | 16.1 | 46.2 |
| PooDLe | BDD* | 44.7 | 25.2 | 56.1 | 17.9 | 57.1 |

**Table 4: Ablation studies on PooDLe components, reporting mIoU on BDD100K semantic segmentation linear readout. Rows without top-down follow FlowE [], replacing pooling with dilated convolutions to maintain spatial extent. $\dagger$Flow model trained without supervised labels.**
| Variant | Dense | Pool | Top-Down | Lateral | Flow | All | Small | Large | Rare | Common |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 FlowE | ✓ |  |  |  | RAFT | 28.8 | 8.7 | 40.5 | 1.8 | 29.2 |
| 2 | ✓ | ✓ |  |  | RAFT | 28.9 | 7.2 | 41.6 | 2.2 | 28.7 |
| 3 | ✓ | ✓ | ✓ |  | RAFT | 30.3 | 6.8 | 44.0 | 4.3 | 30.2 |
| 4 | ✓ | ✓ |  | ✓ | RAFT | 30.3 | 10.9 | 41.7 | 2.4 | 31.1 |
| 5 | ✓ |  | ✓ | ✓ | RAFT | 31.8 | 12.8 | 42.8 | 8.3 | 31.7 |
| 6 PooDLe$\dagger$ | ✓ | ✓ | ✓ | ✓ | UFlow | 33.7 | 14.1 | 45.1 | 8.9 | 33.8 |
| 7 PooDLe | ✓ | ✓ | ✓ | ✓ | RAFT | 34.2 | 15.0 | 45.5 | 9.0 | 34.5 |

#### Class-based performance and IN1K initialization.

Naturalistic videos have severe class and size imbalance (Figure 1; e.g., for semantic segmentation, the "road" occupies $21\%$ of pixels and appears in $96\%$ of images while "bicycle" only occupies $0.05\%$ in $6\%$ of images.
Capturing information on severely underrepresented classes is very challenging.
While it is unfeasible to balance our training data, we demonstrated the composability of  PooDLe with iconic pretraining by initializing with supervised ImageNet weights (Table 1).
This variant improves upon the semantic class understanding of IN1K-pretraining while also significantly improving on spatial boundaries (accuracy), which we hypothesize is due to our dense loss $\mathcal{L}_{\text{dense}}$.
In particular, training PooDLe on BDD improves performance on linear readout by $8\%$ mIoU and $6\%$ Acc over the supervised IN1K weights used for initialization.

To further evidence this phenomenon, we designate BDD classes as "small" if they on average occupy $<1\%$ of pixels and "large" as those that occupy $>1\%$.
We also independently define "rare" as classes that appear in $<20\%$ of images and "common" as those that appear $>20\%$.
Table 3 shows linear readout mIoU with different class groupings to further illustrate the class and spatial imbalance effect.
Full class-level statistics and designations are in Appendix F.
We observe that FlowE performs well on large and common classes due to its dense loss, but struggles on small or rare classes.
Meanwhile, the supervised ImageNet model effectively learns about smaller classes via it’s balanced pretraining data as evidenced by its performance on semantic segmentation.
PooDLe, with its unified objectives, and spatial decoder module shows significant improvement over other BDD-trained models in all class designations and particularly on small and rare classes.
We also train a PooDLe on top of supervised ImageNet weights.
It drastically improves on large classes, from $42.2\%$ to $56.1\%$, due to the dense objective, while remaining competitive with supervised ImageNet on small classes.j

### 4.3 Ablation studies

Table 4.2 shows our ablation experiments, testing each of our contributions beginning from FlowE.
Models trained without the decoder use dilated convolutions in place of pooling operations, as in FlowE [].
How the dense and pooled objectives are composed with and without the decoder is shown in Figure 3.
For this table, models are trained for $40$ epochs on BDD and use a reduced $256\times 512$ resolution and $[0.04,0.11]$ area for the initial crops; we evaluate on BDD semantic segmentation using linear readout.

We observe that adding $\mathcal{L}_{\text{pool}}$ alone has little benefit (row 2) and including either the decoder as a spatial upsampler (row 3) or only the UNet-style lateral connections (row 4) also does not yield much benefit.
Row 5 achieves +3% mIoU, showing that the top-down decoder is only effective when combined with the lateral connections for the full SDM, suggesting that preserving high-resolution information as well as including some capacity for feature processing are both important.
However, when re-adding the pooled loss in addition to the decoder with lateral connections (row 7), we see a substantial 5.4% mIoU improvement.
While the dense objective benefits from the full SDM, it has an even greater synergistic effect with the pooled loss.
This may be because the pooled objective with subcrops can effectively learn about small objects while the full decoder helps propagate the semantic representations through to the dense loss.

Figure: (a) Global crop area
Refer to caption: x10.png

Figure: Figure 6: Varying initial crop area as a percent of the full frame. As crop area decreases, the views transition from global views of a dense scene to pseudo-iconic views, sometimes depicting singular subjects.
Refer to caption: x13.png

We also demonstrate that PooDLe is able to perform well even with self-supervised flow.
We train our own unsupervised UFlow [] model on KITTI [] and finetune it on BDD resulting in only a 0.5% mIoU loss compared to pretraining with RAFT.
See Appendix H and A for visualizations and details.

### 4.4 Spatial and temporal cropping in self-supervised video learning

In this section, we study the effect of image cropping parameters used during data augmentation.
As there is no longer a 1:1 image-to-concept relationship like in iconic data, how much of a scene is visible and assigned to each representation can greatly affect learning.
Specifically, we perform 5 experiments varying: (1) global crop area, (2) subcrop area, and (3) global crop area and resolution to preserve pixel density (4) number of subcrops (5) temporal stride $\Delta t$.
Crop area refers to the fraction of an image taken by the crop, and pixel density is the ratio of source frame pixels to model input frame pixels.
Figure 6 visually depicts how crops transition from global to pseudo-iconic with decreasing crop area.
Training recipe follows the ablations in section 4.3 and should be compared to row 7 in Table 4.2.

Figure: Figure 8: Overlaid frames with varying $\Delta t$.
Refer to caption: x15.png

#### Varying global crop area when training PooDLes.

We train 4 PooDLes on non-overlapping crop ranges $(a_{\text{min}},a_{\text{max}})=(0.05,0.25),(0.25,0.50),(0.50,0.75),(0.75,1.0)$, and 2 different resolutions $256\times 512$ and $384\times 768$.
Results are shown in Figure 5.
We find that directly increasing crop area, thereby increasing the content seen by each vector in $\mathcal{L}_{\text{dense}}$, does not directly improve performance.
Rather, there is a sweet spot for each resolution, $[0.05,0.25]$ for $256\times 512$ and $[0.25,0.5]$ for $384\times 768$ with the larger resolution performing slightly better (34.0% v. 34.4%).
While larger global crop areas improve PooDLe representations, they must be paired with a larger input resolution, otherwise performance degrades.
We believe this is due to loss of information from downsampling.

#### Varying subcrop area.

Next, we study how subcrop area affects our learned representations.
Again, we train 4 PooDLes on subcrop ranges, using a fixed global crop range [0.125, 0.25] at resolution $256\times 512$.
Results are shown in Figure 5.
We show the mIoU for all classes, as well as the large and small class subgroupings.
We observe that larger subcrop areas result in poorer performance, which we hypothesize is due to how larger subcrops are more likely to contain multiple subjects, producing false invariances.
In addition, when crops are too large, subcrops may not serve well as pseudo-iconic views.

#### Varying input resolution with fixed pixel density.

With the above observation, we train another set of models while controlling the pixel density.
The frame crop area is set to $[0.125,0.25]$ and the base crop size, $256\times 512$ making the average pixel density $\rho_{\text{pixel}}=1.32$.
Maintaining this pixel density, we extrapolate 3 additional scales with input resolution $H\times W=(192\times 384),(320\times 640),(384\times 768)$ and crop range $(0.07,0.14),(0.195,0.39),(0.281,0.563)$.
We pretrain a PooDLe as well as dense-only, pooled-only, and FlowE models for each scale on BDD100K and evaluate using linear readout. As subcrops are taken from global crops, we adjust the local crop area following Figure 5 for $320\times 640$ and $384\times 768$ to $(0.032,0.192)$ and $(0.022,0.133)$.

The results in Figure 5 confirm our hypotheses.
PooDLe trained with both objectives is more robust to different input scopes given a fixed pixel density whereas a dense-only model requires high resolution, and pooled-only and FlowE are less competitive on all resolutions.

#### Varying number of subcrops

We also study how varying the number of subcrops affects performance.
We train 4 PooDLes using $K=0,3,6,9$ subcrops on BDD100K and evaluate using linear readout, with results shown in Figure 7.
Using 3 subcrops gives an initial large performance jump and using additional subcrops provides more modest gains.
We decide to use $K=6$ as our default option to balance between performance and computational efficiency.

#### Effect of temporal stride during frame sampling.

To study the effect of varying $\Delta t$, we train PooDLe with values $\Delta t=0,15,30,45$ on BDD100K and once again evaluate using linear readout (Figure 9).
Performance degrades when temporal stride is either too small or too large.
When it is small, there is limited variance in object appearance, making the training objective too easy.
On the other hand, as temporal stride grows, correspondence between frames decreases and optical flow may become unreliable.
Note that for $\Delta t=0$, we jitter the initial large crop by up to $10\%$ of the image size. We also visualize frame sequences from 3 different videos, Figure 9 showing the high variability of motion in BDD100K.

### 4.5 Subcrops as pseudo-iconic training images

We find that the addition of a pooled loss on pseudo-iconic subcrops improves PooDLe representations, particularly on smaller objects.
To better understand this effect, we analyze how subcrops may increase the prevalence of small objects when they are assumed to be pseudo-iconic.
First, we simulate a circle as our foreground object and compute the probability that a subcrop “hits” the object as a pseudo-iconic view, as illustrated in Figure 10(a).
A subcrop counts as a hit if at least 5% of the subcrop is occupied by the object.
We argue that this is a reasonable assumption because background classes typically contain little visual variation and thus, have a relatively modest effect on the learned pooled representations.
We also empirically simulate subcrops on the BDD100K semantic segmentation training dataset, and compute subcrop hit and pixel probabilities for foreground objects of varying size.
As shown in Figure 10(b), the relative difference between the subcrop probability and pixel probability is greater for smaller objects.
This illustrates how subcrops can boost prevalence of small foreground objects in the pooled objective in comparison to the pixel-level dense SSL objective.
Appendix C contains additional details about our setup and analyses.

## 5 Conclusion

Figure: (a) Toy simulation to compute probability of subcrop acting as a pseudo-iconic view, i.e. hit, of circular object. Subcrop hits are green and non-hits are red.
Refer to caption: x17.png

Self-supervised learning on naturalistic videos present many unsolved challenges, especially due to the presence of high-resolution multi-object crowded scenes with severe spatial region imbalance.
Iconic methods rely on single-subject images and dense methods struggle with scale-imbalance of objects.
We propose PooDLe that combines pooled region-invariance learning and dense flow-equivariance learning objectives in a unified framework.
PooDLe achieves state-of-the-art performance on downstream semantic evaluations compared to prior methods pretrained on the same video datasets, particularly on recognizing small objects.
Our study on the effects of crop area, resolution density, and temporal stride also offers key insights on the design choices for video self-supervised learning.

#### Limitations and future work.

Our work has several limitations and areas for future work.
First, PooDLe relies on an external flow prediction network.
Although we demonstrate that an unsupervised flow predictor also works well, it would be more elegant to incorporate motion prediction in a joint framework like in MC-JEPA [].
Additionally, while PooDLe achieves state-of-the-art performance, we find that it does not currently work well with ViTs and this limitation warrants further investigations.
In the future, we would also like to explore ways to zoom-in and attend to interesting regions and regions that contain rare objects rather than relying on random subcrops, e.g., using priors such as motion [, ] to identify foreground objects.

## Acknowledgment

We would like to thank Adrien Bardes and Jean Ponce for helpful discussion, and the Microsoft Accelerating Foundation Models Research program for providing cloud compute credits for our experiments.
The project was also supported by the NYU High Performance Computing resources, services, and staff expertise.

## Appendix

## Appendix A Implementation details

#### Backbone.

As discussed in the pretraining details, we use a Resnet-50 [] as our backbone architecture.
The projector model is a non-linear, 2-layer MLP (linear for pooled, $1\times 1$ convolutions for dense) that has a $4096$ hidden dimension and projects down to $256$ dimensions.
The predictor is the same network with $256-4096-256$ channels.
We follow BYOL [] with a momentum starting at $0.996$ and increases to $1$ throughout training.

#### Decoder details.

The decoder uses a single Bottleneck block from the ResNet architecture with a $8\times$ downsampling ratio in the number of channels.
Upsampling in the decoder is $2\times$ and the lateral connection is a single linear convolutional layer that up-projects the input latent to match the decoder channels ($1024\rightarrow 2048$ in the first decoder block and $512\rightarrow 2048$ for the second block).
As mentioned, 2 decoder blocks are used to achieve a total of $4\times$ upsampling.

#### Supervised and self-supervised flow prediction.

Flow is predicted using a supervised off-the-shelf RAFT model or an unsupervised UFlow [] model that we train ourselves.
For unsupervised training, we exactly follow UFlow and train on the KITTI [] dataset before finetuning on BDD100k [] for 100000 steps on daytime-only videos.
The training and inference resolutions were set to $256\times 512$ to better match the inference setting.
KITTI used adjacent frames (10Hz video) while BDD frames were sampled with temporal stride of 10 (30Hz video).

#### Local cropping details.

$K=6$ paired local crops are sampled using the methods described. Cropping is performed using RandomResizedCrop with an output resolution of $192\times 192$. Jitter is 10% of the input image size and standard aspect ratio range of $[3/4,4/3]$ is used.

#### Loss details.

We sum our 2 loss functions directly and give them equal weight.
The loss computation and warping function were applied to representations after reversing the affine transform and resizing to the input image resolution, this is to take full advantage of high resolution flow like in FlowE [].
We also use flow-based occlusion to prevent misaligning occluded regions without correspondence.
We use the same occlusion formulation as DDFlow [] and parameters $\alpha_{1}=0.1,\alpha_{2}=0.5$.
We also mask out regions that are not visible after affine transformations for $\mathcal{L}_{\text{dense}}$.

Our loss is symmetrical: we reverse the $x_{t}$ and $x_{t+\Delta}$ so that both are encoded by the online weights and used for optimization at each training step.

#### Optimization details.

AdamW is used as the optimizer and a weight decay value of $0.01$.
A learning rate of $5e-4$ is used with batch size of 32 GPUs and $4$ image pairs per GPU for a batch size total of $128$.
Cosine learning rate decay is used with a schedule for 300 epochs, despite early termination due to compute limitations.
LR warmup is used for 2 training epochs.
Full float32 precision is used during training.

#### Evaluation settings.

For all BDD and Cityscapes semantic segmentation and object detection readout tasks, we follow the setup described in FlowE [] for ResNet-based methods.
For ViT-based methods, we adopt those settings, but use AdamW for the optimizer with learning rate of $3e-5$ and weight decay of $0.05$, and a crop size of $512\times 512$ rather than the normal $512\times 1024$ to accommodate the square aspect ratio used in ViT-pretraining, following the semantic segmentation linear readout setup described in iBOT [].
In addition, ViT-based methods require sliding window inference in order to achieve performance that is competitive with convolution-based methods.

For ADE20K linear readout, we simply use the respective BDD linear readout settings for ResNet and ViT methods. For ADE20K UperNet finetuning, we follow the procedure described in iBOT [].

#### Abaltion data sampling.

For all ablation experiments, we employ repeated sampling like in MAE-st [] which samples $R$ frames each time a video is encountered for faster data loading.
Therefore, each pass through every video in the dataset counts as $R$ epochs.

## Appendix B Compute resources

The full model is trained on 16 A100s and takes about 30h for 100 epochs on BDD100K or 18min per epoch. Walking Tours takes longer at 40min per epoch, as the number of training samples per epoch is larger.

Ablation-sized experiments were run 2 or 4 H100/A100 GPUs for a total of 40 epochs taking 20-40h depending on the configuration.

## Appendix C Subcrop analysis

For the toy simulation of subcrops, we place a foreground object as a centered circle of varying size within a $256\times 512$ frame.
We then simulate all possible subcrops of area $A\in[0.02,0.04,0.06,0.08]$.
For each subcrop area, we compute subcrop hits, i.e. whether at least 5% of the subcrop contains the object, using numerical grid-based integration.
We compute the subcrop hit probability, or subcrop hits over valid subcrops, averaged across subcrop areas, as well as the pixel probability, or object pixels over total image pixels.

We also emulate our training procedure for our empirical simulation of subcrops.
For each of the 7,000 images in the BDD100K semantic segmentation training dataset, we sample two global crops with area sampled from $U[0.16,0.45]$ and for each global crop, 4096 subcrops with area sampled from $U[0.02,0.03]$.
We compute subcrop probability and pixel probability independently for the pixels of each foreground class: pole, traffic light, traffic sign, person, rider, car, truck, bus, train, motorcycle, bicycle.
We then group the results into 10% quantile bins by object size (i.e. pixel proportions) and average the subcrop and pixel probabilities.
We utilize a slightly different subcrop area range in the empirical simulation because our two-step global crop and subcrop procedure results in a logarithmic-like distribution.

Figure: Figure 11: Relative change in local crop region class assignments relative to per-pixel class distribution.
Refer to caption: x19.png

We hypothesize that PooDLe’s improvement on spatially underrepresented classes, as shown in Table 7, is due to this subcrop effect.
To quantify this effect on real data, we perform a similar exercise as above on the BDD100K semantic segmentation training set.
We sample subcrops following our method and assign a class label to each subcrop.
If over 10% of the subcrop is a foreground class (*not* road, sky, building, vegetation, sidewalk, fence, terrain), then we label the subcrop as the majority foreground class.
Otherwise, the majority background class label is assigned.
In Figure 11, we show the relative change in class distribution when using this subcrop class assignment.
Foreground classes (green) increase in occurrence while background classes (blue) decrease in frequency, besides road.

## Appendix D Additional visualizations

We provide additional visualizations of results on our evaluated benchmarks: BDD100K [] semenatic segmentation (Figure 12), object detection (Figure 13) and ADE20K semantic segmentation (Figure 14).
Once again, we note that PooDLe produces segmentation maps with clearer boundaries while also effectively capturing small objects.

Figure: Figure 12: Visualizations of semantic segmentation masks for BDD linear readout, Cityscapes linear readout, and Cityscapes UperNet readout.
Refer to caption: x20.png

Figure: Figure 13: Visualizations of object detection bounding boxes for BDD FPN readout.
Refer to caption: x21.png

Figure: Figure 14: Visualizations of semantic segmentation masks for ADE UperNet finetuning.
Refer to caption: x22.png

## Appendix E Additional evaluation results

**Table 5: Additional BDD semantic segmentation (SemSeg) and object detection (Det) readout evaluations. All settings are conducted with a frozen backbone. ^‡ BYOL results are taken from FlowE [] and used DeepLab v3 [] in-place of Upernet []. *Pretrained on BDD, initialized with supervised ImageNet weights.**
|  |  |  |  | BDD100K Sem. Seg. | BDD100K Obj. Det. | Cityscapes Sem. Seg |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  | Linear | UperNet | Det C4 | FPN | Linear | UperNet |  |  |  |  |
| Method | Arch | Ep. | Pretrain | mIoU | Acc | mIoU | Acc | mAP | mAP | mIoU | Acc | mIoU | Acc |
| Scratch | R50 | - | - | 9.7 | 55.0 | 26.1 | 81.2 | 0.0 | 7.7 | 9.8 | 58.0 | 30.7 | 84.1 |
| PooDLe | R50 | 100 | BDD | 39.2 | 89.2 | 49.9 | 91.8 | 4.9 | 25.2 | 47.2 | 90.2 | 60.7 | 93.5 |
| Supervised | R50 | 600 | IN1K | 36.7 | 84.7 | 55.2 | 92.0 | 3.6 | 24.9 | 46.8 | 87.4 | 63.4 | 93.7 |
| BYOL []^‡ | R50 | 1000 | IN1K | 28.3 | - | 52.4 | - | 2.8 | 26.0 | 39.9 | - | 60.3 | - |
| DenseCL [] | R50 | 200 | IN1K | 21.3 | 82.7 | 52.8 | 91.6 | 0.3 | 25.0 | 27.3 | 84.0 | 63.7 | 93.7 |
| Supervised | ViT-S | 300 | IN1K | 41.9 | 88.5 | 50.9 | 91.4 | - | - | 46.8 | 87.4 | 63.4 | 93.7 |
| DINO [] | ViT-S | 800 | IN1K | 38.5 | 88.1 | 52.3 | 92.0 | - | - | 47.1 | 90.3 | 63.6 | 94.0 |
| iBOT [] | ViT-S | 800 | IN1K | 44.4 | 89.6 | 54.2 | 92.2 | - | - | 52.1 | 91.5 | 65.3 | 94.3 |
| PooDLe | R50 | 100 | BDD* | 44.7 | 90.7 | 54.1 | 92.7 | 3.9 | 28.0 | 52.0 | 91.5 | 65.1 | 94.4 |

We compare PooDLe against ImageNet-pretrained baselines in Table 5 and observe that PooDLe outperforms most baselines except iBOT and ImageNet supervised ViT.
This result is encouraging, as pretraining on naturalsitic video is more challenging due to spatial and class imbalance, yet is also a more realistic setting that enables the use of broader sets of usable data.
Furthermore, we note that pretraining on class-balanced data such as ImageNet particularly benefits mIoU, which weighs all classes equally despite some classes only appearing in a tiny proportion of pixels in evaluation.
Finally, PooDLe pretrained on BDD with weights initialized from the ImageNet supervised checkpoint surpasses all ImageNet-pretrained baselines on linear semantic segmentation.

## Appendix F Per-class evaluation results

**Table 6: IoU per class on BDD semantic segmentation linear readout.**
| Method | Pretrain | Rd | Sky | Bldg | Veg | Car | Bus | Fence | Truck | Wall | S-walk | Terrain | Train | Pole | Bicycle | Person | M-cycle | Tr. Sign | Rider | Tr. Light |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DINO | BDD | 88.6 | 93.0 | 72.3 | 77.3 | 73.7 | 0.8 | 11.7 | 5.3 | 5.1 | 38.5 | 37.5 | 0 | 8.1 | 0 | 19.6 | 0 | 13.4 | 0 | 17.7 |
| DenseCL | BDD | 82.0 | 88.2 | 68.3 | 72.6 | 63.0 | 0 | 1.5 | 0.5 | 0 | 17.7 | 7.2 | 0 | 0.3 | 0 | 0.1 | 0 | 1.1 | 0 | 9.4 |
| DoRA | BDD | 89.9 | 93.6 | 75.4 | 79.9 | 76.6 | 5.1 | 17.9 | 11.0 | 10.8 | 44.6 | 42.7 | 0 | 13.3 | 0.7 | 25.2 | 0 | 20.5 | 0 | 23.9 |
| FlowE | BDD | 90.6 | 92.9 | 75.8 | 79.6 | 80.8 | 32.9 | 23.5 | 22.7 | 15.3 | 45.7 | 32.4 | 0 | 12.9 | 11.8 | 28.7 | 4.1 | 15.9 | 0 | 12.1 |
| PooDLe | BDD | 91.3 | 93.5 | 77.0 | 80.4 | 81.7 | 34.0 | 29.4 | 24.3 | 17.2 | 49.6 | 38.1 | 0 | 24.3 | 18.0 | 35.2 | 2.9 | 26.6 | 0 | 21.2 |
| Supv. | IN | 79.8 | 88.8 | 70.0 | 77.2 | 72.6 | 24.9 | 21.8 | 14.4 | 7.2 | 18.4 | 31.8 | 0 | 22.8 | 36.8 | 40.2 | 19.5 | 31.8 | 8.2 | 31.2 |
| PooDLe | IN+BDD | 92.6 | 94.0 | 80.3 | 82.2 | 84.8 | 54.7 | 34.9 | 33.4 | 17.8 | 56.3 | 42.2 | 0 | 25.7 | 27.1 | 41.5 | 7.7 | 39.0 | 0.1 | 35.2 |

**Table 7: Defined groupings and statistics of classes in the BDD semantic segmentation dataset. L=Large, S=Small, C=Common, R=Rare.**
|  | Rd | Sky | Bldg | Veg | Car | Bus | Fence | Truck | Wall | S-walk | Terrain | Train | Pole | Bicycle | Person | M-cycle | Tr. Sign | Rider | Tr. Light |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Avg Pix. % / Im. | 22.0 | 18.2 | 15.0 | 14.4 | 8.4 | 3.7 | 3.4 | 3.2 | 3.1 | 3.1 | 2.8 | 2.1 | 1.0 | 0.8 | 0.7 | 0.6 | 0.5 | 0.4 | 0.4 |
| Total % of Pix. | 21.3 | 17.3 | 13.2 | 13.2 | 8.1 | 0.6 | 1.0 | 1.0 | 0.5 | 2.0 | 1.0 | 0.0 | 0.9 | 0.1 | 0.3 | 0.0 | 0.3 | 0.0 | 0.2 |
| Total % of Im. | 96.5 | 94.8 | 88.4 | 91.7 | 97.3 | 15.0 | 30.6 | 30.5 | 15.4 | 66.7 | 36.7 | 0.7 | 95.0 | 6.4 | 34.7 | 3.8 | 75.3 | 5.2 | 47.1 |
| Size Grp. | L | L | L | L | L | L | L | L | L | L | L | L | S | S | S | S | S | S | S |
| Freq. Grp. | C | C | C | C | C | R | C | C | R | C | C | R | C | R | C | R | C | R | C |

We provide a breakdown of IoU per class on BDD semantic segmentation linear readout in Table 6.
In Table 7, we also provide dataset-level statistics for each class computed over the training split of 7,000 images in the BDD semantic segmentation dataset, namely average pixel percentage per image, total percentage of pixels over the dataset and total percentage of images that they appear in over the dataset.
Size and frequency groupings are then independently defined using these statistics and used in Table 3.
A class is considered ‘Large’ (L) if its average pixel percentage per image is $>1\%$ and ‘Small’ (S) otherwise.
Separately, we define a class as ‘Common’ (C) if the total percentage of images it appears in is $>20\%$ and ‘Rare’ (R) otherwise.
Notably, PooDLe achieves significant gains on small classes such as ‘Pole’, ‘Bicycle’, ‘Traffic Sign’, ‘Traffic Light’.
Methods trained on BDD underperform supervised IN1K on classes rare in BDD such as ‘Rider’, likely because IN1K offers both abundant and iconic images of these object categories.

## Appendix G Spatial token collapse in vision transformers

Figure: Figure 15: Comparison of L2 norm of last feature layer activations. The visualizations show both the raw L2 norm values as well as the L2 norm values clamped at the 80th percentile.
Refer to caption: x23.png

Initial attempts at using a ViT [] backbone for PooDLe results in poor downstream performance. We hypothesize this is because of unwanted interaction between the model architecture and the FlowE dense objective.
In our initial analysis, we have found that ViTs trained with only the FlowE dense objective results in background tokens receiving very high feature activations relative to other regions, as shown in Figure 15.
This does not occur for our ResNet-based model as well as DINO-ViT which we believe is contributing to the performance drop.
In the future, we would like to investigate this phenomenon further and formulate possible remedies to enable PooDLe to work with ViTs.

## Appendix H Flow visualizations

Figure: Figure 16: Comparison of predicted optical flow from RAFT (supervised) and UFlow (unsupervised)
Refer to caption: x24.png

In Figure 16, we compare the predicted flow maps generated from RAFT [], an off-the-shelf supervised model, and our own unsupervised UFlow [] model.
The frame pairs are randomly sampled with $\Delta t\in[15,30]$.
We do note that self-supervised flow, particularly on BDD100K may exhibit noisy or splotchy results.
This is possibly due to the inconsistent motion and large dark regions that do not offer sufficient photometric supervisory signal.
This is in contrast to RAFT [] which learns sharp edges like from supervised labels.
Nevertheless, we find that this self-supervised flow is sufficient for training PooDLes.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: 1. What is the computational overhead of the SDM compared to simpler upsampling approaches?
2. How sensitive is the method to video frame rate or motion blur?
soundness: 4
strengths: 1. The authors propose a novel approach to handling spatial imbalance in dense scenes through unified pooled and dense objectives. The creative use of flow-informed cropping to generate meaningful subcrops is insightful. 
2. The paper provides comprehensive experimental validation across multiple benchmarks, and the method demonstrates obvious improvements on challenging cases (small objects, rare classes).
summary: This paper introduces PooDLe (Pooled and Dense Learning), a self-supervised learning method for learning visual representations from naturalistic videos. PooDLe is a joint framework combining dense flow-equivariance learning with a pooled objective using pseudo-iconic subcrops. It shows state-of-the-art performance on multiple benchmarks while demonstrating strong performance on small objects and rare classes.
weaknesses: 1. Optical flow computation is typically computationally intensive, requiring expensive pixel-wise correspondence matching. It is important to discuss the computational costs of flow estimation. Limited discussion of real-time performance implications
2. Limited analysis of inference time requirements. e.g. how inference time scales with input resolution.
3. Missing discussion of potential memory bottlenecks from joint objectives. e.g. GPU memory requirements for different input sizes, and comparison with baselines.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Overall the paper is solid so this is again fairly minor, but I have a bunch of cropping related questions as that's the one area I had the most confusion:
- Where exactly does the global cropping fit into all of this, what's the point of defining a global crop at all?
- In Figure 7, I'm not sure what it means to change the number of subcrops, is it changing the proportion of correlated samples in a given minibatch? What is the method with 0 subcrops - is it using the global crop as input into the encoder?
- Maybe I missed it, but is there ever an ablation on the importance of the subcrops being flow matched? Intuitively seems like a reasonable thing to do but does it matter much?
- In Table 4, is the FlowE result trained with the proposed cropping approach? How does this compare to FlowE without using subcrops?
- Is the training done on 192x192 crops but testing at 512x512, is there any issue with this discrepancy or does the model generalize well enough to the larger test time resolution?
soundness: 3
strengths: This is a solid paper, well motivated introduction, straightforward idea, outperforming other methods with the appropriate ablations to break down different design decisions. Each of the key ideas behind the approach are sensible, I am surprised it is not already common to perform dense feature self-supervision on U-net/FPN style features.
- I appreciate the level of specific, little details throughout the paper to understand exactly how things were implemented
- it's interesting to see that both losses only start to play a role after a clearer distinction has been made between the pooled "global" and upsampled dense features (Table 4)
- also cool to see that the method performs well with imperfect self-supervised flow
summary: This paper proposes an approach for self-supervision from video. The authors highlight the discrepancy between image datasets which typically contain a single central object and more challenging video data which is composed of complex scenes filled with many objects of differing sizes. The simpler image setting lends itself well to global embedding-based self-supervised methods, but such methods might not be ideal for representation of larger multi-object scenes.

The key idea of the paper is to mix dense and global objectives. For dense feature supervision, the authors follow FlowE (Xiong et al 2021) with one key difference being the use of a U-net style architecture to upsample and connect higher resolution features from earlier layers. The more "global" pooled-objective is applied to subcrops. Similar to how FlowE will match optical flow-warped dense features, subcrops are adjusted across frames by predicted optical flow to have a higher chance of containing the same subject. The motivation for learning on subcrops is for smaller, salient objects to take up a larger fraction of pixels. 

The model is pretrained on BDD and Walking Tours and then segmentation and detection heads are trained on top of the frozen backbone. The resulting model outperforms other self-supervised approaches as well as ImageNet pretraining.
weaknesses: This is minor, but I am a bit confused about the subcrops. When reading I was expecting some mechanism for smarter choice of crops such that there's a higher probability of including an object of interest. For example, some sampling based on saliency or flow that would ensure more crops included people/bikes/etc, but it doesn't seem like that's the case? Instead, crops are chosen randomly and uniformly across the image so they are just as likely to sample a patch from the sky as from the street. The key difference as far as I understand is that by virtue of doing any cropping at all an object of interest will now take up a greater fraction of pixels, and using flow ensures the object will likely be included in both crops. Feels somewhat at odds with how the paper has been written and motivated which emphasizes putting extra attention on smaller objects as opposed to background context.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - The explanation for why combining SDM with Pool helps is not clear. Also, it is not clear why just pool should not help, its even hurting the small objects. SDM has additional parameters, can that be a reason (of course the fine-grained features will help too), SDM is not even connected with Pool.

- In figure 4, it shows finer boundaries for smaller objects, but boundaries for bigger objects, such as car, is getting worse in comparison to other methods, also performance seems similar to the baseline approach FlowE.

- L075: "These subcrops serve as pseudo-iconic views of foreground objects, functionally increasing the prevalence of smaller objects". It is not clear why taking random crops will increase prevalence of smaller objects, parts of bigger objects or backgrounds will have higher probability among these crops.

- Ablations are not comprehensive, only one dataset, one metric, and one task is shown. It will be good to see accuracy metric, and also how these components perform with any other dataset. BDD is street-view, maybe Ego or some other natural videos, CCTV, etc. It will be good to see the ablations on the walkingtour dataset along with BDD. It seems to be working fine with driving videos, since scene will not change a lot, but what about other videos? Like ego-centric? Or static camera?

- What about performance just with pool? It will be good to see if just learning from the crops are enough. SDM improves, but it comes with additional parameters too.

- Since this work is build on top of FlowE, there should be results for FlowE on other datasets too (Table 2).

- It is not clear why results were shown only with ResNet, why not ViT?

- Will this method work on other simple tasks like classification?

- It will be good to see accuracy in Table 3 and 4.

- Using additional crops for training adds training overhead, there is no analysis shown on this. 

- Figure 9, it will be good to see t between 0 and 15.

- The role of predictor and projector is not clearly explained, projector is not even shown. It is not clear.

- L205: why crop features are averaged-pooled over spatial dimension? Fine-grained constraint should be better?
soundness: 3
strengths: 1. The paper is well-written with a clear structure to demonstrate a clear and logical progression of motivation and ideas. The authors use precise and well-chosen words that enhance readability and engagement.

2. The motivation to combine pooled representations and dense local object representations is well justified. The proposed method aligns well with the motivation.

3. The paper provides comprehensive/detailed analysis, testing the proposed method across various scenarios and datasets. This thorough experimental validation successfully demonstrates the method’s effectiveness on small and rare objects.
summary: The paper introduces PooDLe, a self-supervised learning (SSL) method tailored for complex, naturalistic video data with densely packed scenes and diverse object scales. Unlike standard SSL methods trained on iconic images, PooDLe addresses the spatial imbalance in cluttered video data by combining a dense SSL objective that maintains equivariance through optical flow warping with a pooled objective that captures semantic information from smaller, aligned subcrops. This dual-objective approach, augmented with a spatial decoder module (SDM), is shown to effectively represent both large and small objects.

PooDLe’s performance is validated on driving and first-person video datasets, achieving state-of-the-art results in semantic segmentation and object detection tasks, particularly excelling with small object recognition. Additionally, the paper introduces the Walking Tours Semantic (WT-Sem) segmentation task, presents a detailed analysis of class size and frequency on BDD100K, and studies the impact of global crop area, input resolution, and temporal stride on learning efficacy, highlighting key factors for SSL with naturalistic data.
weaknesses: 1. The main weakness of this work is lack of enough ablations for the proposed method. It is not clear (with the shown ablations) if local crops really are useful. Considering this is the main contribution in terms of novel idea, it should have been made clear. Results with just local crops might help, since adding these local crops on dense does not improve, but hurt smaller objects. It will be good to see accuracy on this and on datasets other than BDD as well.

2. Although there are several experiments presented, the technical contributions seems little on the weaker side, the method is build on top of FlowE, local crops is novel (but its contribution not clear), the improvement on dense consistency is expected due to use of features from earlier layers, 

3. The proposed method shows some improvement over existing methods, but looking at a comparison with the baseline (FlowE), mostly its incremental (less than 1%) if we look at accuracy. That is why it is important to loot at this metric in the ablations. Also, this baseline is not shown for the other dataset (WalkingTour), where the proposed method do not outperform other methods across the board.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weaknesses.
soundness: 3
strengths: i) Method. The proposed method, PooDLe, is technically sound.

ii) Performance. Compared to prior global or dense-only works, PooDLe demonstrated superior performance on dense prediction tasks like semantic segmentation and object detection.

iii) Ablation. The paper further verified that PooDLe improved the performance of small and rare objects, justifying the advantage of combining global and local supervisions, as well as the spatial decoder modules (SDM). It also provided ablations on the pretraining datasets, different components, and augmentation strategies, e.g. crop area, resolutions, etc.
summary: Self-supervised learning for naturalistic videos is challenging with the presence of independent objects of various sizes, imbalanced class distribution, etc.
The paper aims to tackle these problems by combining global and dense supervision, as well as spatial decoder modules (SDM).
The dense objective could help capture different object instances from the videos while the global objective could help learn the semantics of the objects using contextualized signals.
The proposed method, PooDLe, was evaluated on representative video benchmarks, e.g. BDD100K, Walking Tours, ADE20K, Cityscapes, demonstrating superior performance on dense prediction tasks, i.e. semantic segmentation and object detection.
weaknesses: The reviewer has several concerns and questions about the evaluation, ordered by their impact on the final score.

---
Concerns:

i) About SDM. SDM introduced extra operations on higher resolution feature maps or images. From L306-307, SDM was also used in the inference stage for semantic segmentation. In this case, the inference overhead is determined by both the backbone architectures (e.g. ViT-S, ResNet50), and the UNet based SDM. The reviewer wonders if the baseline approaches also have SDMs during inference? Were they of the same architectures, complexities (e.g. FLOPs)? If they are not of the same complexity, could the authors provide the FLOPs of each baseline and PooDLe. This could show if the comparisons of different methods on semantic segmentation are apples-to-apples, especially for inference.

ii) Some comparisons are apples-to-oranges. At L358-L360, it was claimed that PooDLe outperformed DoRA across different settings. If the reviewer understood correctly, the comparisons there were between PooDLe_ResNet50_epoch20 and DoRA_ViT-S_epoch100, which were not fully convincing. The authors are encouraged to present apples-to-apples comparisons under the same setting, e.g. backbone, #epoch, or remove the claims otherwise.

---
Questions

iii) Tasks beyond dense prediction. It was shown that augmenting dense supervisions with global objectives could help improve dense prediction tasks. The reader may also be curious if augmenting global supervisions with dense objectives could help improve global prediction tasks, e.g. action recognition, etc.

iv) About the "relationship between global crop area and input resolution". It was claimed in Contribution 3 (L101-102). However, there seems to be no explicit analysis of the "relationship" between "crop area" and "resolution", or the relationship could not be easily interpreted from Figure 5. To reveal the relationship between "crop area" and "resolution", the authors could consider analyzing extra metrics derived from them, one example could be the pixel-scale, i.e. crop_area/input_resolution, defined in [1].

v) Patch size used for ViT-S, which determines the FLOPs of some baseline approaches

vi) The videos of Walking Tours (WT) are at 4K+60fps, instead of 720p+30fps (L248-249)?

[1] Effective Self-supervised Pre-training on Low-compute Networks without Distillation. ICLR 2023.

Meta-review: This paper proposes a self-supervised learning method for video with dense scenes. It combines an invariance-based objective on pooled representations and a dense SSL objective that enforces equivariance to optical flow warping. This dual-objective approach, augmented with a spatial decoder module (SDM), is shown to effectively represent both large and small objects. The experiments on first-person video datasets have shown better results than other SSL methods, particularly on small object recognition. 

All reviewers are unanimously positive on this submission, and like the contributions including 1) the approach is somehow novel and well motivated; 2) the experiments are extensive/good and support the proposed claims; 3) the paper is well written with sufficient details and precision. The common concerns from the reviewers are 1) complexity overhead; 2) missing further ablation/comparison/discussion; 3) technical contributions are kind of incremental given the previous works. However, most of the concerns were well addressed after the rebuttal. The AC agrees with the reviewers' unanimous decision on accept.

Decision: Accept (Poster)