Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video
Authors: Shashanka Venkataramanan
Mamshad Nayeem Rizve
Joao Carreira
Yuki M Asano
Yannis Avrithis
Venue: ICLR 2024 oral (2024)
Keywords: self-supervised image-pretraining
egocentric video
Walking Tour dataset
multi-object tracking
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Self-supervised learning has unlocked the potential of scaling up pretraining to billions of images, since annotation is unnecessary. But are we making the best use of data? How more economical can we be? In this work, we attempt to answer this question by making two contributions. First, we investigate first-person videos and introduce a ``Walking Tours'' dataset. These videos are high-resolution, hours-long, captured in a single uninterrupted take, depicting a large number of objects and actions with natural scene transitions. They are unlabeled and uncurated, thus realistic for self-supervision and comparable with human learning. 

Second, we introduce a novel self-supervised image pretraining method tailored for learning from continuous videos. Existing methods typically adapt image-based pretraining approaches to incorporate more frames. Instead, we advocate a ``tracking to learn to recognize'' approach. Our method called DoRA, leads to attention maps that **D**isc**O**ver and t**RA**ck objects over time in an end-to-end manner, using transformer cross-attention. We derive multiple views from the tracks and use them in a classical self-supervised distillation loss. Using our novel approach, a single Walking Tours video remarkably becomes a strong competitor to ImageNet for several image and video downstream tasks.

Published as a conference paper at ICLR 2024
Is IMAGENET WORTH 1 VIDEO? LEARNING STRONG IMAGE ENCODERS FROM I LONG UNLABELLED VIDEO
Shashanka Venkataramanan Inria, Uniy Rennes, CNRS, IRISA
Mamshad Nayeem Rizve University of Central Florida
João Carreira Google DeepMind
Yuki M. Asano" University of Amsterdam
Yannis Avrithis*
Institute of Advanced Research on Artificial Intelligence (IARAI)
ABSTRACT
Self-suue learning has unlocked the potential of scaling up pretraining to billions of images, since annotation is unnecessary. But are we making the best use of data? How more economical can we be? In this work, we attempt to answer this question by making two contributions. First, we investigate first-person videos and introduce a "Walking Tours" dataset. These videos are high-resolution, hours- long, captured in single uninterrupted take, depicting large number of objects and actions with natural scene transitions. They are unlabeled and uncurated, thus realistic for self-supervision and comparable with human learning.
Second, we introduce a novel seff-supe image pretraining method tailored for learning from continuous videos. Existing methods typically adapt image- based pretraining approaches to incorporate more frames. Instead, we advocate "tracking to learn recognize" approach. Our method called DoRA, leads to attention maps that Discover and IRAck objects over time in an end-to-end manner, using transformer rross-attention. We derive multiple views from the tracks and use them in a classical self self-supervised distillation loss. Using our novel approach, a single Walking Tours video remarkably becomes a strong competitor to ImageNet for several image and video downstream tasks. Dataset and code can github. ii/ dora.
be found https: https:///shahhahaasee
INTRODUCTION
(To the question "Have you read all the books in here?") No, only four of them. But I read those very, very carefully.
Jacques Derrida
Learning from large scale datasets has been at the core of great progress. In particular, the field of self-supervised learning has allowed pretraining of neural networks to scale beyond the size of labelled datasets. By avoiding costly annotation, strong performance has been demonstrated by increasing the training dataset sizes into billions of images.
But how well those images really used? At rate of one image per second, dataset of 1B im- ages would take 317 years to watch. Yet, humans develop functioning visual systems much facheaster." Besides potential genetic visual priors in humans, one stark difference the type of data. Humans observe their visual surroundings in one continuous stream, only interrupted by sleep. Indeed, learn- ing visual representations of images from videos is not new. However, previous works have found significant gaps in performance to mmage-pretrained models. They have mostly used object-centric videos scraped from the internet, and adapted image-based pretraining methods to use different frames an extra form of data augmentation (Gordon al., 2020; Parthasarathy et al., 2022).
"Equal last authors. Order determined randomly
'Humans develop face recognition (de Haan al., 2001) and color sensitivity (Adams, 1987) in three months, depth perception in five months (Campos et al., 1978) and visual acuity in months (Sokol, 1978).

---

Published as a conference paper at ICLR 2024
Figure I: Examples of frames from the Walking Tours dataset, containing hours-long, continuous egocentric 4K videos from urban scenes in different cities, under CC-BY license. There are large number of objects and actions in a variety and natural transition of places, e.g. residential area, park, market, waterfront, etc., with natural transition of lighting conditions and object augmentations.
In this work, we investigate two directions. First, in the direction of data, we introduce a new dataset of open-source first-person videos, recorded for the purpose of virtual "walking tours", inspired by (Wiles et al., 2022). These videos have several advantages. Not only are the individual frames dense in semantic categories much more so than movies, as we analyze but these videos also directly represent the viewpoint of human, contain few or no shot cuts nor special effects and are long (1-3h). Another benefit their transparency: indeed, one can watch the whole dataset one setting. The dataset we create contains 10 Walking Tours (WT) videos with CC-BY license.
Second, in the direction of the method, we develop a new sulf-supe peeee-prtraining method that is uniquely suited for learning from natural, non-object-eentric videos. Our approach is inspired by observing toddlers first learn to track objects and animals, then to recognize and differentiate them (Bomba & Siqueland, 1983; Quinn et al., 1993; Spelke & Kinzler, 2007). Our method, called DORA, is an end-to-end training approach that "tracks to learn to recognize": given a video clip, objects in an initial frame are implicitly Discovered and tRAcked across time. The tracked objects clustering of patch embeddings; the
are incentivized to be diverse by introducing a Sinkhom- tracked instances are used as a learning signal for a classical multi-view SSL loss.
Surprisingly, contrary to previous works, we find that our novel method obtains ImageNet-level performances by training on single WT video, evidenced by performances on segmentation and object detection downstream tasks. While humorously intentioned, Derrida's quote rings true to this finding and our results give some hope for alternative directions in SSL that depart from blind dataset scaling towards more efficient and smarter use of existing video data.
To summarize, our key contributions in this work are as follows:
1. We introduce new dataset of 10 WT videos, with single-video and mixed-video splits. The latter is conveniently equal in size to ImageNet. We analyze their usefulness compared to existing video and image datasets.
2. We propose a new end-to-ent self-supervised visual pretraining method called DoRA. It builds upon DINO but is tailored to promote tracking of multiple objects across frames. We use it to learn strong image encoders and trace the source of its improvements through extensive ablations.
3. We obtain strong performance on ADE20k segmentation and MS COCO detection, outper- forming SongeNeet-pretriine DINO, while instead pretraining on a single long video.
2 RELATED WORK
Self-supervised learning of image encoders from video data is very active area of research. Video, and more generally temporal streams, have long been theorized to be ideal signals for unsupervised learning (Wiskott & Sejnowski, 2002). In computer vision, early methods have been very diverse and included pretext tasks such as egomotion prediction (Agrawal et al., 2015; Jayaraman & Grau- man, 2015), active recognition (Jayaraman & Grauman, 2016), pose estimation (Chakraborty & Namboodiri, 2017), unsupervised object discovery (Croitoru et al., 2017), dense prediction (Pathak al., 2017; Li et al., 2019). optical flow (Mahendran et al., 2018; Xiong et al., 2021), frame order prediction (Misra et al., 2016), view-poin matching (Sermanet et al., 2018; Pirk et al., 2020) or learning visual correspondences (Wang et al., 2019).
More recently, there have been considerable advances in supervupere learning using ImageNet, with the main theme being extracting multiple augmentations of an image (Chen et al., 2020; Caron
2

---

Published as a conference paper at ICLR 2024
DATASET
DOMAIN
EGD
PRE
BAI
ANNOT
#VIDEOS
(SBC)
(IR))
RESOLUTION
Diverse Pretroiming
Kinetics 400 (Kay el al., 2017)
255
WehVid-2M (Rain et al,, 2021)
Open
X
/
X
Weak
18
13k
320 x 240
HoWTo100M (Miech al., 2019)
Instructions
/
Weak
135k
Egocentrie
Epic Kitchens (Damen el al., 2022)
Cooking
x
X
Loc.
510
100
37
1920 x 1080
Ego-4D (Grauman al,, 2022)
Daily
931
1920
Meccano (Ragusa al., 2023)
Industry
x
Loc.
1247
849
20
1920
1080
Assembly- 101 (Sener et al., 2022)
Assembly
Loc.
426
167
362
1920 x 1080
ImageNet-aligned
R2V2 (Gordon et al., 2020)
ImageNet
/
/
Class
467 280
VideoNet (Parthusarathy al., 2022)
ImageNet
Class
3055
Walking Tours (ours)
Urban
X None
3880
23
10
3840 x 2160
Table I: Walking Tours vs. existing video datasets. EGO: egocentric; PRE: used for pretraining: BAL: class balance control; ANNOT: annotation type. Weak: associated data per clip (text other modality); Class: class label per frame or clip; Loc: localization per frame (e.g. bounding box, segmentation, mask 3D pose). AvG. DUR: average duration per video; DUR: total duration.
al., 2021) and training models pull them together/apart. These methods have since percolated to learning from video frames (Gordon et al., 2020; Parthasarathy et al., 2022; Tschannen et al., 2020; Wang & Gupta, 2015; Orhan al., 2020). Similar this work, TimeTuning (Salchi et al., 2023) leverages the passage of time in videos by not treating it as simple augmentations. However, in contrast to our work, it requires an already image-pretrained backbone. VITO (Parthasarathy al., 2022) improves performance relative to ImageNet, by using VideoNet, large YouTube dataset of 10s videos from a similar class distribution and the same number of examples as ImageNet. this paper, show that it is possible to obtain strong results from single long video, with very different visual distribution compared to ImageNet/ VideoNet.
WALKING TOURS DATASET
3.1 DATASET COLLECTION AND PROPERTIES
dataset of urban scenes called "Walking Tours" (WTours, or WT) comprising 10 egocentric vidcos of a person walking in different cities in Europe and Asia. The cities include Amsterdam, Bangkok, Chiang Mai, Istanbul, Kuala Lampur, Singapore, Stockholm, Venice, and Zurich. We also include a video from a Wildlife safari. Examples are shown in Figure 1. These videos are captured in 4K resolution (3840 2160 pixels) 60 prameonper-second and under Creative Commons License (CC-BY). The minimum video duration is 59 minutes (Wildlife safari), the maximum is 2 hours 55 minutes (Bangkok) and the average videos are particularly interesting for visual learning because of the following properties:
We collect from YouTube
hour 38 minutes. Such
1. Large number of objects and actions. Each frame or clip taken from a video depicts several objects and actions, e.g. walking, riding bike, sitting, drinking etc.
2. Natural transition in lighting conditions. In some videos, the lighting gradually transitions from bright (late afternoon) dim (dusk) then to dark (post sunset).
3. Natural transition in scenes. The videos depict transitions between places, e.g. from city center to market place to residential areas to parks to water fronts etc.
4. Natural object augmentations. Continuous variation e.g. of pose, deformatiin,, viewpoint, perspective distortion, relative object position, occlusion, background clutter.
The abundance information within these videos, encompassing a multitude of objects and com- plex scenes, presents formidable challenge for manual annotation curation, making it appro- priate for unsupervised pretraining. To the best of our knowledge, we are the first to propose an egocentric video dataset for pretraining and evaluate
wealth of downstream tasks.
3

---

Published as a conference paper at ICLR 2024
Wittao
BPC Kicckn
200A
0.5
102
Apt
102
Chinn Hevina
250
40
100
100
Men
Grac ORiNn
lxnd (rea)
(a) Lightness
(b) #Ohjects per frame
(c) Total #classes
(d) Shots
Figure 2: Dataset analysis of WTours video compared with videos from Epic-Kitched (Damen et al., 2022), AVA (Gu al., 2018) and two entire movies, concatenated or cropped to match the duration the WTours video. (a) Lightness vs. time. (b) Number objects per frame vs, time. (c) Frequency of classes in entire video. (d) Number of shots, Objects detected by Detic (Zhou et al., 2022b), trained on ImageNet-21k.
3.2 COMPARISON WITH OTHER VIDEO DATASETS
In Table I, we compare WTours with existing video datasets. In summary, self-supervised pre- training on videos has been mostly limited to short, low-resolution datasets that rely weak notation in the form of video-text pairs (Bain et al., 2021; Miech et al., 2019) or are curated, e.g. their class balance is controlled, even if their annotation is unused (Kay al., 2017). ImageNer- aligned datasets (Gordon et al., 2020; Parthasarathy et al., 2022) contain short videos that are (semi- automatically) curated and annotated with the same distribution and classes as ImageNet. Egocentric video datasets (Damen et al., 2022; Grauman et al., 2022: Sener et al., 2022) have long, high-quality videos, but are the result of significant manual work. In this paper we aim learn from videos publicly available online.
WTTours videos are continuous, longer and higher-resolutie than even other egocentric datasets. Using object detectors, we find that the average number of object classes is close to that of ImageNet and there is a high number of objects per frame, making WTours appropriate for representation learning. WTTorrs is not curated and does not rely on search terms. It is data-first and more open- ended, thus well suited for the self-supervised setting. It is scalable since requires no human labeling effort and more videos be easily downloaded or even made. We are inspired by 10k walking tours videos created by Wiles et al. (2022), which however is not publicly released and not studied for self-supervised learning. A more detailed discussion is given in subsection A.I.
3.3 DATASET ANALYSIS
In Figure 2, we analyse the properties of a single WTTous video compared with videos of the same length from two other datasets, as well as two movie videos. In summary, our findings are as follows. From Figure 2(a), WT may exhibit gradual shifts in lightness, transitioning from bright to dim dark, while Epic-King Ditc-Kitchens and AVA videos exhibit random brightness fluctuations. Lightness variations are not well expored in self-supervise pretraining. From Figure 2(b,c), unique classes appear more frequently and there are more unique objects per frame in WTours than the other datasets. This makes WTours semantically richer. From Figure 2(d), WTours and OpiccKKtchenns videos contain only one or two shots per entire video on average, while the other datasets contain hundreds. In subsection 5.2 and in Appendix C, show that WTours significantly outperforms movies in downstream tasks, which is partially attributed to the absence of cuts. More detailed discussion of dataset analysis is given in subsection A.2.
4 ATTENTION-BASE MULTI-OBJECT TRACKING
Our goal is to build robust representations by leveraging the rich information in video frames. Stan- dard SSL frameworks (Chen et al., 2020; Caron et al., 2020) often assume correspondences between different views. This is true whether using dense (Zhou et al., 2022a) global representations by pooling (Caron et al., 2021). While it is relatively straightforward to establish correspondences in images, it becomes more challenging when dealing with temporal deformations, requiring some form of object tracking (Salchi et al., 2023). In videos with a large field of view or ego-motion, obtaining correspondences becomes even more difficult.
4

---

Published as a conference paper at ICLR 2024
Kene (Kg)
Teachee
conEgnedens
load (K)
sancher
sseool Feaks Nerove (CU)
restaffoee (A2)
Student
bandher 2rrkel (2:)
Chooee
FORTON (TA)
16l sdeed
muBictjed Tradder
Figure 3: DORA, our self-supervised image pretraining method from video. (Left) From an input frame Xt., the output of the second-last layer of the teacher model is used by multi-object tracker to generate cross-attention maps T` with frame X,, We use those to mask X, (7), feed it to the student model and apply a distillation loss L3 between [CLS] token embeddings (8). (Right) In the tracker, we obtain the query Q. key K and output Z embeddings. From the multi-head attention maps A (1), we draw a subset I of heads and form object prototypes P by pooling over patch queries Q (2). We refine them into p' to discover distinct objects, using Sinkhorn- Knopp (SK) to establish correspondences M between and patch embeddings 2 (4) and pooling over Z (5). We then track the objects over frames X by cross- cross-attention Tý with patch key embeddings K; (6).
High-level idea We introduce DORA, based on multi-object Discovery and tRAcking. As shown in Figure 3, it leverages the attention from the [CLS] token of distinct heads in a vision transformer to identify and consistently track multiple objects within a given frame across temporal sequences. On these, deacher-student distillation loss is then applied. Importantly, we do not use any off-the- shelf object tracker or optical flow network. This keeps our pipeline simple and does not require any additional data or training. It also ensures that the learned representation robust.
Preliminaries We are given a video clip consisting of T frames X, E Axxxx for t E (1..... T) where h w is the spatial resolution and c is the number of channels. Each frame is split into n = hw/pp² non-overlapping patches of resolution x p. The patches are linearly projected into embeddings of dimension and [CLS] token embedding is prepended. This representation is input to a transformer encoder (Dosovitskiy et al., 2020). The outpul embeddings are Z, = ge (X,) E g(++1)xd, where mapping ge includes the tokenizer and encoder, while 0 denotes its learnable parameters. Given an embedding Z E R(x+1)xd, we write Z = [ZicLsi; 2}. where ZiCLs) E R is the [CLS] token embedding and 2 E Raxd are the patch embeddings.
Following DINO (Caron et al., 2021), there is a student network with parameters 0 and teacher network with identical architecture and parameters 6' obtained as the exponential moving average (EMA) of f according to 6' <- ad' + (1 0. The encoder is followed by head that includes an MLP and scaled softmax, such that the output token embeddings can be interpreted as probabilities. We denote by So the mapping that includes the tokenizer, encoder and head.
Discovering objects with multi-head attention Starting at a first frame X we obtain the query and key embeddings Q, K E [x++l)xdd from the last transformer layer of the teacher network'. According to multi-head attention, these embeddings are partitioned as Q = [Q'..... Q*I, K [K',..., Kh1, where Q', K* 1)x++)/d/h for l,..., h and h is the number of heads. For each head i. the self-attention matrix A ((++1)x(+++) is based on the dot-product similarity between the query and key embeddings:
A' softmax (Q'(K)) E 1x(++))x(++))
Given attention matrix A E ((++1)x(++1) let AlCLs] = [01 (01.2.-.., [n Rixn be the [CLS]] attention vector between the [CI.S] and patch embeddings. where dg,j is the element (i,j) of A. We draw at random a subset I := {ir..... in} of < h heads and collect their [cLs]-attention vectors into A := [[AAh) ) :...; (A4)))css) E _kxn Intuitively, as expressed in rows of matrix A², the different heads attend to different objects in the frame (Caron et al., 2021).
For simplicity, we drop to from the notation.
5

---

Published as a conference paper at ICLR 2024
t=1
t= 2
3
1=4 4
t= 5
X
T,
TV
Figure 4: For each input frame t of a video clip (top), cross-attention map T E Rk (3) (middle) and refined cross-attention map TV k (6) (bottom), using Sinkhorn-Knopp algorithm. For each object, one row of T, or TV is reshaped as h/p x w/p and upsampled to an h x w attention map overlaid on the input frame for k = 3 objects encoded in blue, red and green channel. Mixed colors yellow and cyan for T; (middle, in red circle) indicate spatial overlap of two objects, while TV (bottom) yields three well separated objects shown in primary colors blue, red and green.
To represent the k objects in the embedding space, we use matrix A e F to form linear combinations of patch embeddings Q paxd obtaining object prototypes
P:= A E kxd
This can be seen the representation of different [CLS] tokens in the full embedding space, capturing i objects frame to- Then, given the key embeddings K, IR [x++ another frame i, we could track the objects by cross-attention
T = softmax (PRT/VA E Rx
(3)
where K, Ra Unfortunately, we observe in Figure that the k attention maps obtained this way are spatially overlapping, meaning that each attention map is not delineating single object.
Establishing object-patch correspondences To discover spatially distinct objects, propose to establish correspondences between prototypes and patch tokens. Let Z go (X,,) R R be the output embeddings of the teacher network, still at frame fg. We seek a correspondence between the rows of P E k and Ž E Ra where Ž are the patch token embeddings.
The goal is to find a transport plan M kx that minimizes the expected pairwise cost C : -P E REXE between prototypes and patches, while incorporating an entropic regular- izer with coefficient e. Matrix M is non-negative with row-wise sum 1/k and column-wise sum 1 1/n, representing a joint probability over and with uniform marginals. The minimal solution M is unique and can be found by forming the matrix e CC/e and then applying the Sinkhorn- Knopp (SK) algorithm (Cuturi, 2013), i.e., iteratively normalizing its rows and columns:
M -SK (exp [P2 (P2T/e) /c)) E Tx
Observe the similarity with (1) and (3), where scaling is by vd rather than c, exp is included in softmax and normalization on rows only rather than iterative. Then, similarly with (2), we use the
optimal transport plan M k to form linear combinations of patch embeddings Z E Raxd
obtaining the refined object prototypes
P' = M 2 E k
(5)

---

Published as a conference paper at ICLR 2024
Now, given the key embeddings K, 1)++ at another frame t, track the objects by the refined cross- attention, similarly with (3):
T = softmax
(P (P'RT/VO) E RAX"
(6)
where K, E IRnxdd Indeed, Figure 4 confirms that each of the k resulting attention maps is associ- ated with a spatially distinct object, thanks to the established correspondences.
In contrast to previous works that use SK in the context of self-supervised learning to force an equi- partitioning of images to cluster labels (Asano al., 2020; Caron al., 2020; Oquab et al., 2023), we rather use optimal transport to re-balance spatial correspondences to different objects.
Multi-object masking We use the cross- cross-attention (6) to mask the input video clip for the student network, such that each masked clip can be considered as a multi-object crop. This crop plays a similar role with local crops in DINO (Caron al., 2021), but it has arbitrary shape and tracks an object over video frames. In particular, given an input frame X E Rixwxe with cross-attention matrix 7" I (6) and an object
{I...., k). we reshape the i-th row of T" as h/p x w/p and upsample to h x w attention map to match the spatial resolution of X, as shown in Figure 4. We repeat along the channel dimension to form tensor T Rxxuxe and we mask X as
X X T
where @ the Hadamard product. Following DINO (Caron et al., 2021), given an input frame X., we generate two standard resolution augmented global views X; X We introduce mulli-object loss LE for frame t. applied to the [CLS] token between the teacher for output for one global view X and the student fe output for the masked version X of the other view X for E {1, {I...., k), where u, v E V = {a. b} and u * V:
L.
Σ u Σ for (Xx) log (Fo(X)
M
In addition, as detailed in subsection B.I, we apply a local loss, following multi-crop (Caron et al., 2020). The overall loss L is the sum of the two losses, averaged over all T frames.
EXPERIMENTS
5.1 SETUP
Tasks and methods We perform self-supervised pretraining on a single WT tour video in Venice (referred to WTvvnicee all 10 WT videos (referred as WT.n) and compare with other image and video datasets. To evaluate the quality of the learned representations, we use frozen features for classification, unsupervised object discovery and video object segmentation. We fine-tune for semantic segmentation, object detection and object tracking. We compare DoRA with SoTA SSL methods (da Costa et al., 2022) using our settings. We provide more details in individual sections per task. Implementation details and hyperparameters are given in Appendix B.
5.2 ABLATIONS
We examine the effect of using different pretraining video dataset and different options and param- eters for DORA, A, measuring performance of classification ImageNet-1k (Deng et al., 2009) by linear probing (LP) accuracy and unsupervised object discovery on PaScal-VOC 2012 (Everingham et al.) by correct localization (CorLoc) (Siméoni et al., 2021).
Pretraining video dataset We study the impact of pretraining on diverse video datasets, encom- passing object-centric videos such as Kinetics-400 (K-400) (Kay et al., 2017), egocentric videos like Epic-Kitchens (EK) (Damen et al., 2022) and a single movie, Moviemo (Central). To maintain uniformity in terms of the number of frames, we curate subset of videos from K-400 and EK, such that their total duration is the same as a single WT video. In Table 2a, we observe that although K-400 is object-centric, pretraining on WTours videos yields superior performance on ImageNet and Pascal- VOC 2012. Pretraining on a single movie yields is inferior to both WTours and K-400 by large margin. This is possibly due the presence of cuts, as studied in Appendix C.
7

---

Published as a conference paper at ICLR 2024
METHOD PRETRAIN #FRAMES
CoRL.cc:
METHOD
& 1.P CoRL.O
(M)
DINO DORA
Movisin
0.19
34.9
51.5
DINO
x
33.8
51.2
Movison
0.19
35.3
51.6
DoRA
1
39.9
53.9
DORA
2
43.1
$5.7
DINO
K-400
0.2
40.7
52.4
DORA
3 44.5
56.2
METROD SK MASK
I.P
CoRt.oc
DORA
K-400
0.2
43.0
55.2
DORA
4 39.2
53.8
DINO
33.8
51.2
DINO
EK"
0.2
38.6
53.5
DORA
5
50.3
DORA
Random
49.8
DORA
DORA
6 35.8
48.8
DORA
Object
42.5
55.3
DINO
WTvasiee
0.2
DORA
16
48.5
DORA
Random 29.9
46.7
DORA
WT
0.2
44.5
56.2
DORA
32
27.1
46.8
DORA
Object 44,5
56.2
(a) Video datasets
(b) #Objects on Vince
(c) SK and masking on Vonke
Table 2: Effect of parameters. ViT-S/16 pretrained, then frozen. (a) Different pretraining video
dataset, (b) Number
tracked objects. (c) Random or multi-object mask, without SK (3) and with
SK (6). ": subset of videos with same total duration as single WTours video. K-400: Kinetics-400, Epic-Kiite Kitchens. LP: top-1 accuracy (%) of linear probing on the validation set of ImageNet- Ik. CorLoc: correct localization validation set of Pascal VOC 2012.
EK:
SEMANTIC SEG.
(b) ORJECT
INSTANCE SEG.
MATHOD
EPOCHS
PRETRAIN
mioU
Accm
GAIN
GAIN
mloU
GAIN
VIT-S/16
none
25.1
33.3
IROT (7hon et al., 2022s)
100
WTwwiee
31.9
43.3
37.6
33.0
ArtMask (Kakogeonyiou al., 2022)
W WTveiiee
33.6
42.7
36.5
32.5
VITO (Parthasuruthy et 2022)
300
VideoNet
39.4
44.0
DINO (Caron et 2021)
33.9
44.3
39,9
DoRA (ours)
100
WTa
36.9
48.0
40.7
36.3
DINO (Carun et 2011)
100
W WTymiie
32.4
43.7
37.1
32.1
DORA (ours)
WTwnier
35.4
45.5
+1.8
39,5
+2.6
Table 3: Semantic segmentation, object detection and instance segmentation. ViT-S/16 pretrained, then fine-tuned. WT Walking Tours (ours), single video of Venice; WT all videos. IN-IK: ImageNet-Ik. Ik. (a) Semantic segmentation: fine-tuning ADE20k using UperNet. mloU: mean loU; Accm: mean-class accuracy. (b) Object detection and (c) Instance segmentation: fine-tuning on MS-COCO using Cascade RCNN. mAP: mean average precision; mloU: mean IoU.
Number of tracked objects We study the impact of the number objects. Objects are discov- ered using attention heads, where the total number of heads is in ViT-S/16 is h = 6. For A: >> h, we modify the MSA block as described in subsection B.2. In Table 2b, we observe that k = 3 works best. We hypothesize that this is compromise between the number of objects that can be tracked and the multi-object loss (8) attempting to match small objects with the global crop.
Choice of masking and Sinkhorn-Knopp We explore the effect of using a multi-object mask (7) vs. random block-wise (Zhou et al., 2022a) and the effect of improving object-patc correspondence through SK in refined cross-attention (6) vs. (3). In Table 2c, we observe that multi-object mask leads to a remarkable performance improvement even in the absence of SK. In fact, random block- wise mask undermines object- patch correspondence, making the effect of SK negative. By contrast, SK improves performance in the presence of multi-object mask.
5.3 COMPARISON WITH STATE-OF-TIEE-ART
Dense seene understanding Table 3(a) shows semantic segmentation by fine-tuning on ADE20k (Zhou al.. 2017) using UperNet (Xiao et al., 2018). DoRA outperforms DINO by 3% mloU, and 1.8% Acc,. It is interesting to note that DORA pretrained on 200k frames of a sin- gle WTours video outperforms DINO pretrained on 1.3M images of ImageNet- 1k by 1.5% mloU. A more comparable setting is DoRA pretrained on 1.5M frames of WT which outperforms DINO pretrained on ImageNet by 3% mloU. Table 3(b) shows object detection and instance segmentation by fine-tuning MS-COCO (Lin et al., 2014) using Cascade RCNN (Cai & Vasconcelos, 2019). DoRA outperforms DINO by 2.4% mAP and 2.6% mloU. DORA pretrained on WT outper- forms DINO pretrained on ImageNet by 0.8% mloU and 1.2% mAP. This shows that pretraining on WTours videos significantly improves the generality of DoRA to dense prediction tasks, requiring only one tenth of the total images.
8

---

Published as a conference paper at ICLR 2024
(a) VIneo ORIECT SEOMENTATION
(b) OBJECT TRACKING
MATHOD
EPOCHS PRETRAIN
(JheF) GAIN
Fn.
GAIN
Fm
GAIN
mAO
GAIN
SRo.5
GAIN
SRo 75 GAIN
VIT (Docwirskly et al. 2020)
100
Nane
26.9
25.4
28.3
23.1
19.0
3.4
iBOT (2hou et al., 2022a1
100
WT
57.4
56.7
58.0
41.5
47.5
16.6
(Caron et 2021)
100
61.4
46.4
DoRA (ours)
100
WT
57.6
55.1
60.2
45.9
53.4
23.7
DINO (Carun et al., 2021)
100
WTemes
54.6
53.0
56.2
37.4
414
134
DoRA (ours)
100
60.4
41.4
+4.0
47.
+5.8
18.2
+4.8
Table 4: Video object segmentation and object tracking. ViT-S/16 pretrained, then frozen fine- tuned. Venieee:: Walking Tours (ours), single video from Venice; WT all videos. IN-1k: ImageNet-Ile Ik. (a) Video object segmentation: frozen features DAVIS-2017. Tm: mean region similarity; Fm mean contour-based accuracy. (b) Multi-object tracking: fine-tuning on on GOT- 10k. mAO: mean average overlap: SR: success rate, threshold 50% and 75%.
#FRAMES
CLASSIRIATION
(b) OBJECT DISCOVERY
METHOD
EPOCHS
PRETRAIN
(M)
LP
R-NN
JACC.
CORLoC
SimCLR (Chen et al., 2020)
100
WTytatee
26.3
25,9
40.4
50.2
SwAV (Caron al., 2020)
100
WT WTWeicee
0.2
28.0
26.4
40.6
51.4
iBOT (Zhou ot al., 2022a)
100
W Vitiice
0.2
36.8
32.8
43.0
51.1
ArtMask (Kalogrongiou et 2002)
W WTitiicee
35.8
31,9
43.5
54.5
VicRog (Bardes et al., 2021)
100
W W Venice
0.2
36.5
30.1
42.7
52.1
(Carun al., 2021)
100
W WTiaiie
0.2
33.8
29,9
43.8
51.2
DoRA (ours)
100
W WTweeeee
45.4
+11.6
33.8
+3,9
44.0
+0.2
56.2
+5.0
DINO (Caron al., 2021)
100
WT
1.5
36.6
31.1
42.9
55.8
DoRA (ours)
WT
+8.7
35.7
+1.3
Table 5: Image classification and object discovery. ViT-S/16 pretrained, then frozen. WT Venice Walking Tours (ours), single video from Venice; WT all videos. (a) Classification top-1 accuracy (%) on validation set of ImageNet-Ik. LP: linear probing. (b) Unsupervised object discovery on validation set of Pascal- VOC 2012. Jacc.: Jaccard similarity; CorLoc: Correct Localization.
Video understanding Table 4(a) shows video object segmentation by using frozen features on DAVIS-2017 (Pont-Tuset et al., 2017), which assesses the ability to segment an object over its dy- namic temporal changes. DORA captures detailed temporal deformations and outperforms baseline DINO by 3.4% Im and 4.2% Fm- Using only single video for pretraining, DoRA achieves almost the same performance of DINO pretrained on ImageNet (56.4% 57.4% Jm.). Table 4(b) shows multi-object tracking by fine-tuning on GOT-1OK (Huang et al., 2021) 2023). GOT-10k assesses the ability to track extremely fast moving objects, objects with illumina- tion variation and low resolution. DoRA achieves significant gains between 4-6% over DINO.
1) using SeqTrack (Chen et al.,
Image classification and unsupervised object discovery We pretrain DORA WTours and then we keep it frozen on the downstream task, indicating the quality of the pretrained features. Table 5(a) shows image classification ImageNet- 1k, measuring accuracy for linear probing and k-nearest neighbor. Table 5(b) shows unsupervised object discovery on Pascal VOC 2012, using attention maps segmentation masks to measure Jaccard similarity and CorLoc.
On both tasks, non-contrastive methods (DINO, iBOT, VICReg) outperform contrastive methods (SimCLR, SwAV), when pretrained single WT video. Importantly, non-contrastive methods are also more efficient to train, since no negative pairs are used. Also on both tasks, DORA A outperforms DINO by a large margin, e.g. 11.6% LP and 3.9% k-NN on classification, when trained on a single WT video. Comparing DORA A on WT with the WT dataset, the improvement brought by the full dataset small when using DoRA, although is 10 times larger.
In Appendix D, we show that DoRA outperforms SoTA methods on all tasks on ImageNet- Ik.
CONCLUSIONS
We have introduced a dataset of 10 walking tour videos - first- videos taken by people touring city, with cuts, high resolution and that are hours long. We show that learning from clips taken from these videos is surprisingly powerful: with an appropriately tailored self-supervised learning method for videos, obtain representations that rival those obtained on ImageNet when

---

Published as a conference paper at ICLR 2024
transferring popular downstream image and video tasks. This differs from previous state-of-the- art approaches to learning image encoders from video, which also obtain such results but require large video datasets, following closely the ImageNet blueprint.
Our proposed learning method DoRA is inspired by DINO, generalizing it to video by incorporating implicit multi-object tracking across video clips. We observe that the method leads interesting emergent attention masks within the transformer model, that seem to latch on to particular objects, even through occlusions. This makes it uniquely suited our newly introduced dataset.
ACKNOWLEDGEMENTS
This work was in part supported by the ANR-19-CELT 0028 MEERQAT project and was performed using the HPC resources from GENCI-IDRIS Grant 2021 AD011012528. We also thank Dilara Gokay and Andrew Zisserman for their valuable feedback this paper.
REFERENCES
Russell Adams. evaluation of color preference early infancy, Infant Behavior and Development, 1987.
Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In ICCV, 2015. 2
Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-lahelling via simultaneous clustering and representation learning. In ICLR, 2020. 7
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021. 3, 4.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-iivariinec-covannes regularization for self- supervised learning. arXiv preprint arXiv:2105.04900 2021. 9
Paul C Bomba and Einar R Siqueland. The nature and structure of infant form categories. Journal of Experi- mental Child Psychology, 1983. 2
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance segmentation.
IEEE TPAMI, 2019. 8,
Joseph Campos, Susan Hiatt, Douglas Ramsay, Charlotte Henderson, and Marilyn Svejda. The emergence of fear on the visual cliff' The development of affect, 1978.
Mathilde Caron, Ishan Misra. Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurlPS, 2020. 4, 7,
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin, Emerging properties in self-supervised vision transformers. ICCV, 2021. 2, 5, 7, 9, Brandon Castellano. Pyscenedetect. https: httppp//ssttttacccommerattttrtte
World Movie Central. The night we met. https: http://wwtttooocccc////tttttttto
7,
2
Prabuddha Chakraborty and Vinay P. Namboodiri. Learning to estimate pose by watching videos. arXiv preprint arXiv: 774,040814 2017.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geolfrey Hinton. A simple framework for contrastive learning of visual representations. In /CML, 2020. 2, 4,
Xin Chen, Houwen Peng, Dong Wang. Huchuan Lu, and Han Hu. Seqtrack: Sequence to sequence learning for visual object tracking. In CVPR, 2023. 9, 4
Ioana Croitoru, Simion- Vlad Bogolin, and Marius Leordeanu. Unsupervised learning from video to detect foreground objects in single images. In ICCV, 2017. 2
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. NeurlPS, 2013. 6
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sehe, and Elisa Ricci, solo-leamm: A library of self-supervised methods for visual representation learning. JMI.R, 2022.
10

---

Published as a conference paper at ICLR 2024
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michacl Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epie-kitchens- 100, ICV,, 2022. 4, 7, 1,2
Michelle de Haan, Mark H Johnson, Daphne Maurer, and David Perrett. Recognition of individual faces and average face prototypes by 1-and 3-month-old infants, Cognitive development, 2001. 1
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words:
Transformers for image recognition scale. In ICLR, 2020. 5,9.3
M. Everingham, L. Van Gool, C. K. L. Williams, J. Winn, and A. Zisserman. The PAS- CAL Visual Object Classes Challenge 2012 (VOC2012) Results. network. wwwcchallnges//ooo/oe22//////oo/
http:/www.paseat
7,3
Daniel Gordon, Kiana Ehsani, Dieter Fox, and Ali Farhadi. arXiv, 2020. 1,
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022. 4,
Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijaya- narasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, al. Ava: video dataset of spatio- temporally localized atomic visual actions. CVPR, 2018. 4,
Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object
tracking in the wild. IEEE TPAMI. 2019. 4
Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high high-diversity benchmark for generic object tracking in the wild. IEEE TPAMI, 2021. 9
Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In ICCV, 2015.
Dinesh Jayaraman and Kristen Grauman. Look-ahead before leap: end-to-end active recognition by fore- casting the effect of motion. In ECCV, 2016.
Toannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. What to hide from your students: Attention-guided masked image modeling. In ECCV, 2022. 8,
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaannarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, al. The kinetics human action video dataset. arXiv preprint arXiv:1705.00 06950, 2017. 3,4, 7,
Abdullah Aman Khan, Jie Shao, Waqar Ali, and Saifullah Tumrani. Content-aware summarization sports videos: audio-visual feature extraction approach. Neural Processing Letters, 2020. 2
of broadcast
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine fine-grained catego- rization. In ICCVW, 2013. 6
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Xueting Li, Sifei Liu, Shalini Mello, Xiaolong Wang. Jan Kautz, and Ming-Hssaa Yang. Joint-task self- supervised learning for temporal correspondence. NeurlPS, 2019. 2
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrencc Zitnick. Microsoft coco: Common objects context. ECCV, 2014.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Aravindh Mahendran, James Thewlis, and Andrea Vedaldi. Cross pixel optical-flow similarity for self- supervised learning. In ACCV, 2018. 2
Antoine Miech. Dimitri Zhukov, Jean-Bapiss Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m Learning text-viden embedding by watching hundred million narrated video clips. In ICCV, 2019. 3, 4,
11

---

Published as a conference paper at ICLR 2024
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. ECCV, 2016."
Maria- Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008. 6
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fer- nandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po- Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. arXiv:2304.07193, 2023. 7
Emin Orhan. Vaibhav Gupta, and Brenden M Lake. Self-supervised learning through the eyes of a child. NeurIPS, 2020. 3
Nikhil Parthasarathy, SM Eslami, João Carreira, and Olivier J Hénaff. Self-supervised vidco pretraining yiclds strong image representations. arXiv preprint arXiv:2110064333 2022. 1, 4,
Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Iecrming fcatures by watching objects move. In CVPR, 2017. 2
Sören Pirk, Mohi Khansari, Yunfei Bai, Corey Lynch, and Pierre Sermanet. Online learning of object represen- tations by appearance space feature alignment. In ICRA, 2020. 2
Jordi Pont- Jusee, Federico Perazzi. Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint 4rXiv:1704.00675, 2017. 9,4
Paul Quinn, Peter Eimas, and Stacey Rosenkrantz. Evidence representations perceptually similar natural categories by 3-month-ole old and 4-month-oke old infants. Perception, 1993.
Francesco Ragusa, Antonino Furnari, and Giovanni Maria Farinella. Meccano: A multimodal egocentic dataset for humans hchavior understanding in the industrial-like domain. CVIU, 2023. 3
Mohammadreza Salchi, Efstratios Gavves, Cees G. M. Snock, and Yuki M. Asano. Time does tell: Self- supervised time-tuning of dense image representations. ICCV, 2023. 3,4
F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. CVPR, 2022. 3, 3,
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. ICRA, 2018.
Oriane Siméoni, Gilles Puy. Huy Vo. Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. In BMVC, 2021. 7.3
Skiptrace. Skiptrace. putps://wwet yow. ube com wat hattchhy--bbeEE20055.. 2
Samuel Sokol. Mcasurement of infant visual acuity from pattern reversal evoked potentials. Vision research, 18-1),33-39, 1978.
Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 2007. 2
Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil Houlsby, Sylvain Gelly, and Mario Lucie. Self-supervised learning of video-induced visual invariances. In CVPR, pp. 13806-13815, 2020.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018. 6
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015. 3
Xiaolong Wang, Allan Jabri, and Alexci A Efros. Learning correspondence from the cycle-consistency of time. In CVPR, 2019. 2
Olivia Wiles, Joao Carreira, lain Barr, Andrew Zisserman, and Mateusz Malinowski. Compressed vision for efficient video understanding. In ACCV, 2022. 2,
12

---

Published as a conference paper at ICLR 2024
Laurenz Wiskolt and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 2002.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Yuwen Xiong, Mengye Ren, Wenyuan Zeng, and Raquel Urtasun. repessptttttiddreeeeeeesttttit learning from flow equivariance. In ICCV, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. CVPR, 2017. 8. 3
Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. ICLR, 2022a. 4. 8, 6
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühh, and Ishan Misra. Detecting twonty-thousand classes using image-level supervision. In ECCV, 2022b. 4,
13

---

Published as a conference paper at ICLR 2024
Is ImageNet worth video? Learning strong image encoders from 1 long unlabelled video
Supplementary Material
Table of Contents
More on Walking Tours
Comparison with other video datasets
A.2 Dataset analysis
B More experimental setup
2
B.1 Multi-crop
2
B.2 Implementation details
3
B.3 Hyperparameters.
3
C More ablations
4
D More comparisons with state of the art
6
E More visualizations
7
A MORE ON WALKING TOURS
A.1 COMPARISON WITH OTHER VIDEO DATASETS
In Table 1, we compare WTTuuss with different types of existing video datasets. Self-supervised pretraining on videos has been mostly limited to video datasets that rely on weak annotation in the form of video-text pairs (Bain al., 2021; Miech et al., 2019) or even are curated, e.g. their class balance is controlled, even if their annotation is unused (Kay et al., 2017). Their average clip duration is small, e.g. less than 20 sec, and their resolution is also small, limiting the capacity to detect objects at a greater distance. By contrast, WTours videos are continuous, hours-long at high resolution and provide natural transitions of scenes and viewing conditions. They not curated and thus better suited for the self-supervised setting.
ImageeNet-aligned datasets such R2V2 (Gordon al., 2020) and VideoNet (Parthasarathy al., 2022) contain videos that are curated and annotated with the same distribution and classes as Ima- geNet, meant for pretraining image encoders. These videos are short, i.e. 10 seconds average. By contrast, WTours consists of continuous stream of egocentric video, where the average number of classes is close to that of ImageNet, as shown in subsection 3.3. The rich information contained in 4K resolution, together with high number of objects in frame, makes it appropriate for represen- tation learning. Importantly, the continuity and absence of curation make more realistic and more comparable with human learning. Our dataset does not rely on set of objects, human activities or other search terms but instead is data-first and more onen-ended.
Despite the large number of high-quality videos, egocentric video datasets (Damen al., 2022; Grauman et al., 2022; Sener et al., 2022) have been used only for downstream tasks and thus come with extensive annotation. In comparison, WTours has 4-10 times longer average duration and twice the frame resolution. While WToouss is smaller terms of total duration and number videos, is scalable under the self-supervised setting since it requires human labeling effort and more videos

---

Published as a conference paper at ICLR 2024
can be easily found, downloaded even made. This makes collecting more data simple as walk in the park.
Very long video datasets. A large dataset of 10k WTours videos created recently by (Wiles et al., 2022) but was not publicly released and not studied for self-supervised learning. Another dataset having hour-long videos is introduced in (Khan et al., 2020), in the context of sports analytics; has not been explored for self-supervised learning either.
A.2 DATASET ANALYSIS
Here we present more detailed discussion on the dataset analysis of subsection 3.3. We refer to Figure 2, where we analyse the properties of a single WToors video compared with videos of the same length from Epic- Kitchens (Damen et al., 2022) and AVA (Gu et al., 2018) datasets, as well as two movie videos, an action movie and romantic movie.
Variation in lightness We measure the change in perceived brightness using the lightness value (L) across consecutive frames. From Figure 2(a), we observe gradual shift at roughly 150 min into the WTours video, transitioning from bright to dim dark. By contrast, SpiccKKtchens and AVA videos exhibit random brightness fluctuations, alternating between dim and bright conditions. Typically, self-supervised pretraining happens on datasets with uniform brightness levels. Datasets featuring such brightness variations are less expored.
Variation in number of objects Using Detic (Zhou al., 2022b), a DETR-style object detector trained on ImageNet-21k, we detect objects in each frame. Figure 2(b) shows the number of unique objects per frame and Figure 2(c) shows their frequency the entire video. We observe that WTours contains 703 unique objects, while
has 373, AVA has 663 and Movie-2 has 259. The unique objects appear more frequently and there are more unique objects per frame WTours than in the other datasets. This makes WTours semantically richer, despite coming from one continuous stream of video. Using videos with a large number of objects can encourage the model to capture complex relations and variations in the data.
Variation shots Egocentric videos are typically captured single uninterrupted take, with exceptions being post-processed special effects or cuts. In Figure 2(d), we find that, on aver- age, WTours and Epic-Kitchens Kitchens videos contain only one or two shots per entire video, while AVA contains 406, action movie (Movicaa) (Skiptrace) contains 2000 and romantic movie (Movicum) (Central) contains 667. The substantial number of shots in movies and AVA poses chal- lenges for representation learning methods that rely object tracking or optical flow. In subsec- tion 5.2, we show that WTTurs significantly outperforms movies in downstream tasks, which may be attributed the absence of cuts.
B MORE ON EXPERIMENTAL SETUP
B.1 MULTI-CROP
Following DINO (Caron et al., 2021) and iBOT (Zhou et al., 2022a), we apply the multi-crop strat- egy (Caron et al., 2020). In particular, we generate m local crops X of smaller resolution for i E {I,...,m) m]. The local loss LtC for frame is applied to the [CLS] token between the teacher for output for global view X and the student fo output for the local crop X for {L....,m} m].:
T.C
For (x:) log (fo(x)))CCss)
The overall loss L is the sum of the multi-object loss 1T (8) and the local loss L.C (9), averaged over all T frames:
T L = T
Σ L + L.
(10)
t=1
2

---

Published as a conference paper at ICLR 2024
B.2 IMPLEMENTATION DETAILS
We use VIT-S/16 (Dosovitskiy et al., 2020) the backbone in all our experiments. For each mini- batch, we randomly sample clips from the video, consisting of T = 8 frames temporally separated by I second i.e. we sample one frame every 30. Objects discovered in the first frame are tracked over the following 7 frames. Since each frame contains several different objects, applying the standard multi-crop augmentation (Caron et al., 2020) to the entire frame would result in crops with very different visual content or noisy positive pairs. Instead, we apply multi-crop to a 300 x 300 crop that we first take from the frame. Following DINO (Caron et al., 2021), we obtain two global crops and six local crops. Masking (7) is applied to the global crops seen by the student for the multi-object
loss (8), while local crops are seen directly by the student for the local loss (9). We train for 100 epochs by default.
Objects are discovered using attention heads, where the total number of heads in ViT-S/16 is lim- ited to h = 6. For the purpose of the ablation of the number & of objects for k > h in Table 2b, we modify the MSA block in the final layer, resulting in configurations of 16 and 32 heads. Con- sequently, we can identify and track up to 16 and 32 objects within the video clip. To accomplish this, we decompose the query and key embeddings of dimension d' = 768 into 16 and 32 subvec- tors, resulting in new feature dimensions of 24 and 12 respectively, as opposed to 64 for 6 heads. In Table 2b, we observe that tracking 16 or 32 objects results in overall poor performance possibly due to the small feature dimension, which encodes poor representations.
B.3 HYPERPARAMETERS
ImageNet-Ik:: Linear probing and k--N We pretrain DORA in a self-supervised setting with VIT-S/16 using DINO for 100 and 300 epochs. We two global and six local crops for each clip and train on 8 A100 GPUs with global batch size of 16 x 8 = 128. We use LARS with a learning rate of x 10 10-4, minimum learning rate of x 10-6, global crop scale of [0.4, 1.0] and local crop scale [0.05, 0.4].
For linear probing, we follow (Caron et al., 2021) and use the frozen features of the transformer backbone to train linear classifier in a supervised setting. We use global batch size of 1024 on the training set and evaluate on the validation set of ImageNet-lk. We use top-1 accuracy (%) as our evaluation metric. For k-NN, we freeze the backbone and extract features of training images, then use a nearett neighbour classifier with k- 20.
Pascal-VOC 2012: Object discovery We use the validation set of Pascal VOC 2012 (Everingham
al.), which comprises total of 1449 images. Following LOST (Siméoni et al., 2021), we use the averaged self-attention map, extracted from the final layer of our pretrained ViT-S/16, to retain 80% of the mass. We use the Jaccard similarity J measured as overlap between predicted mask P Gnt and the ground truth mask G as J(P, G) CUP. We also use CorLoc, which measures the number of correct predicted boxes, where a predicted box is said to correct if its IoU 2 0.5.
Semantic segmentation We evaluate DoRA on ADE20k (Zhou et al., 2017) for se- mantic segmentation. The dataset includes 20,000 images in the training set and 2,000 images in the validation set. We use UperNet (Xiao et al., 2018) as the segmentation model and use DoRA A pretrained on WT to initialize the backbone. Following the experimental settings in iBOT (Zhou et al., 2022a), we use Adam (Loshchilov & Hutter, 2019) with an initial learning rate of 6 x 10-5, weight decay of x 10-2, and linear warmup of I, 1,500 iterations. We fine-tune for 160,000 iterations with batch size of 4.
ADE20k:
Object detection We evaluate DoRA for object detection and instance segmentation on MS-COCO. We use Cascade Mask R-CNN (Cai & Vasconcelos, 2019), which produces bounding boxes and instance masks simultaneously on the COCO dataset. We use a multi-scale training strategy, where we resize images to have shorter side ranging between 480 and 800, ensuring that the longer side does not exceed 1,333 pixels. The learning rate is 1 x 10-4 and the weight decay is 0.05. During training, we fine-tune the entire network using 1x schedule, which involves 12 epochs with learning rate reductions by a factor of 10 at epochs 9 and 11. We explore different layer decay rates, specifically 0.65, 0.75, 0.8, 0.9, with rate of 1.0 indicating decay.
MS-COCO:
3

---

Published as a conference paper at ICLR 2024
CoRLac
Amsterdaam
45.4
54.5
Bangkok
42.1
54.3
Chiang Mai
44.9
55.5
Itambul
Kuala Lampur
43.9
54.1
Singapore
42.7
Stockholm
44.1
54.7
METHOD
PT
LP
CORLoC
Venice
44.5
56.2
DINO
33.8
51.2
Wildlife
44.0
DORA
Movic
35.3
51.6
Zurich
44.9
54.4
DORA
Movie
39.8
54.8
Mean
44.1
54.8
DORA
WT
44.5
56.2
(a) WT videos
(h) Cuts
Table 6: Effect of pretraining video and cuts. ViT-S/16 pretrained. then frozen. (a) Different WTours video, using DORA. (b) Effect of cuts. *: subset of videos with same total duration as a single WTours video. T: sampling without cuts. LP: top-1 accuracy (%) of linear probing on the validation set of ImageNet-Ik CorLoc: correct localization on validation set of Pascal- VOC 2012.
To generate hierarchical feature maps, we utilize the features produced by layers 4, 6, 8, and 12 of our network and apply two deconvolutions for layer 4, one deconvolution for layer 6, identity mapping for layer 8, and max moo-pooling for layer 12. These post-processing steps enable the creation of hicrarchical feature representations. It is important to note that we do not employ multi-scale testing in experiments.
D0V1S-2017: Video object segmentation We assess the performance of DoRA for video object segmentation on DAVIS 2017 dataset (Pont-Tuset et al., 2017), which involves segmenting between 2 to 4 objects within the video frames. We follow DINO (Caron et al., 2021) and evaluate on video frames with resolution of 480p. We apply label propagation on the attention map from our pretrained model and use mean region- based similarity Jm and mean contour- based accuracy Fi.. our evaluation metrics.
GOT-10k: Object tracking We evaluate the object-tracking performance of DoRA on the GOT- 10k dataset (Huang et al., 2019). This is a large-scale benchmark for object tracking that contains 563 categories of common moving objects. The training set contains around 10,000 videos and the test set contains 180 videos. Another challenging aspect of this dataset is that the object classes in the training and test set are non-overlapping. We use the SeqTrack (Chen et al., 2023) codebase to evaluate the performance of different methods this dataset. In particular, we initialize the encoder weights of Seq Track with the self-supervised weights and keep them frozen during training. While training, we only update the parameters of the lightweight decoder which consists of 2 transformer blocks. We use all the default hyperparameters. We report mean average overlap (mAO) and success rate (SR) different thresholds. The mAO measures the class-balanced average overlap between the ground truth and predicted bounding boxes whereas SR indicates the percentage of accurately tracked ground truth bounding boxes where the overlap crosses a certain threshold.
C MORE ABLATIONS
Blurring faces in videos We address the privacy concerns that may arise from potential lack of consent during recording by the creator. Given the significant presence of individuals in WTour videos, we employ Deface" to automatically detect and blur faces in WT videos. Specifically, we blur all faces in WT-Amsterdam and observe that DoRA achieves top-1 accuracy of 45.5% on linear probing, compared to 45.4% without any blur. This demonstrates that face blurring does not impact performance and can effectively mitigate potential privacy and safety issues.
Pretraining WT video We study the effect of pretraining on different videos of WTours. In Table 6a, we observe that the effect is minimal on both image classification and unsupervised object discovery. Notably, the fluctuation in illumination conditions within the Bangkok video influences the performance image classification. It is also interesting to note that, while pretraining on Amsterdam is best on image classification, pretraining on Vénice is best on object discovery. This
frtppa: / hwt.h cacaab..cn com/ (ORR- RD de face
4

---

Published as a conference paper at ICLR 2024
ORIECT DISC.
SEMANTIC SEG.
ORJECT
METROD
PRETRAIN
EPOCHS
JACC
CoRLoc
MIOU
ACCm..
MAP
MiOU
Jm
Fm
DINO (Caron et al, 1021)
IN-IK
100
44.5
59.6
33.9
44.3
39.9
35.1
57.4
61.4
DINO (Caron 2021)
IN-IK
45,9
64.0
49,4
42.4
41.6
63.4
DINO (Carom et al., 2021)
WT Vence
100
43.8
51.2
32.4
43.7
37.1
32.1
56.2
DoRA
WT WTwmin
100
44.0
56.2
35.4
45.5
39.5
34.7
56.4
60.4
DORA
Waeee
300
44,9
56.9
37.5
47.3
42.8
42.0
57.5
60.9
DINO (Caron et ol. 2021)
WT
100
42.9
55.8
34.1
44.0
38.3
33.2
53.3
57.6
DORA
WT
49.0
40.7
36.3
60.2
DORA
WT.
300
45.2
61.3
38.7
49.8
43.7
42.6
58.4
62.0
Table 7: Effect of longer pretraining ViT-S/16 pretrained, then frozen (object discovery and VOS, same settings as Table Table 5) fine-tuned (semantic segmentation and object detection, same settings as Table 3). IN-IK: ImageNet-IK.
CLASSIFICATION
OBJECT DISC.
SEMANTIC SEG.
# FRAMES
LP
k-NN
JACC
CORLOC
MIOU
ACCm
DoRA (T = 2)
43.1
32.6
41.9
53.3
32.0
43.5
DoRA (T 4)
44.0
32.9
42.7
54.6
32.6
44.1
DoRA (T = 6)
44.9
33.5
44.0
55.8
32.7
43.9
DoRA (T 8)
45.4
33.8
44.0
56.2
32.4
43.7
Table 8: Effect of frames in a video clip. ViT-S/16 pretrained on Veniees then frozen features used for classification on ImageNet-IK: KK and object discovery on PASCAL-VOC; features fine-tuned on ADE20K for semantic segmentation.
could be due to the large overlap of objects in these videos with respect to the downstream datasets. However, the consistency of our method across diverse videos indicates that DoRA is robust to variations in scenes, number of objects and lighting conditions.
Presence of cuts We now analyse the effect of cuts in representation learning. Cuts are defined as instant transitions from one shot to the next, which is frequent in movies. action movies, a single shot lasts around 4 seconds, while in romance movies, around 12 seconds average To understand the effect of cuts, we compare pretraining on WTours videos and a romance movie. We PySceneDetect (Castellano) to extract the timestamps in the movie and pretrain DoRA A by sampling clips that do not intersect cuts; cuts naturally do not exist in WT videos. In Table 6b, observe that the performance improves significantly in the absence of cuts, tracking in DoRA will fail across a cut.
Longer pretraining We investigate the impact of pretraining DoRA on single WT video and all WT videos over 300 epochs. The results presented in Table indicate a notable enhancement in per- formance when DoRA is pretrained for 300 epochs in general. Furthermore, we observe that DoRA pretrained on WT surpasses WTvenecee in VOS pretraining performance, in contrast to the situation where DoRA was pretrained for 100 epochs. It is worth highlighting that the extended pretraining duration also enables DoRA to achieve comparable results to DINO pretrained on ImageNet- IK, while demonstrating superior performance in dense scene understanding tasks.
Number of frames in a video clip We study the effectiveness of DoRA in classification, ob- ject discovery and semantic segmentation tasks using a lower number of frames in video clip, specifically when T {2,4, 6}. Using the frame-wise loss (8), our goal is to improve the training efficiency DoRA by achieving comparable performance through pretraining with fewer frames than the default T = 8 frames.
We observe from Table 11 that the performance of DoRA in classification and object discovery improves as the number of frames in video clip increases. Employing T = 2 enhances training throughput by 60%, albeit with a 2.3% decrease in linear probing accuracy. However, fine-tuning DoRA for semantic segmentation on ADE20K shows no additional performance gain when varying the number of frames the video clip. These insights contribute to a detailed understanding of
frtppa: pettps://ttephhtttie 1o9..co com y- - s wowttt-aaeeeeee mageeroooooo
5

---

Published as a conference paper at ICLR 2024
DoRA's performance across diverse tasks and provide valuable guidance in optimizing its training strategy to suit specific applications.
Using larger VIT architecture We evaluate DoRA (WT- Ven-ien for 100 epochs on ViT-B/16 for semantic segmentation on ADE20K and object detection on MS-COCO. The results are summarized in the Table below.
METHOD
ARCH
PRETRAIN ADE20K MS-COCO
DoRA (ours)
ViT-S/16 WT--eeice
35.4
39.5
DoRA (ours) ViT-B/16 WT--Veniee
40.3
41.7
Table 9: Using a larger model. ViT-B/16 pretrained and then fine-tuned for semantic segmentation on ADE20K and object detection MS-COCO. We report mloU ADE20K and mAP on MS- COCO.
We observe that with ViT-B/16, DoRA achieves 4.9% gain in terms of mloU on ADE20K and 2.2% in terms of mAP on MS-COCO as compared to using VIT-S/16. This shows that DoRA also scales well with the model size.
D MORE COMPARISONS WITH STATE OF THE ART
Pretraining on ImageNet-1k We pretrain DoRA on ImageNet- Ik and compare with SoTA meth- ods on multiple tasks. Unlike videos, we discover objects but do not track them. Instead, images in a mini-batch are processed independently. Given an input image X. we obtain refined object proto- types as usual (5), but the refined cross-attention (6) is with K, replaced by K of the same image X. The same image X is masked for the student (7). The loss is given again by (8) and (9) with X; replaced by X, averaged over the mini-batch. We refer to this version as DoRA without tracking or DORA*
DINO (Caron al., 2021) and iBOT (Zhou et al., 2022a) use only one global crop for the student, while DoRA uses & object crops. To compensate, we perform an experiment where we pretrain DORA* for 60 epochs and the competitors for 100, thus all methods having the same training time.
From Table 10, we observe that DORA outperforms state-of-the-art self-supervised learning (SSL) methods like DINO and iBOT on image downstream tasks. This demonstrates that the multi-object loss not only enhances performance when pretrained WTours videos but also achieves superior results when pretrained on ImageNet- Ik images.
Fine-tuning for classification We follow the evaluation protocol of iBOT (Zhou et al., 2022a) and fine-tune DINO (ImNet), DoRA WWT-Venice and DoRA (WT-all) using ViT-S/16 on CIFAR- 10/100 (C-10,C-100) (Krizhevsky al., 2009), iNaturalist18 (iNat 18) (Van Horn al., 2018), Oxford Flowers (Flwrs) (Nilsback & Zisserman, 2008), Stanford Cars (Cars) (Krause et al., 2013) and ImageNet-Iee 1k (ImNet) datasets.
From Table I, we observe that despite the class distribution of iN being closer to ImNet than WTours, DoRA (WT- (erive) is on-par with DINO (ImNet) when fine-tuned on C-10/C-100, iNat 8, Flowers and Cars. Furthermore, DoRA (WT-all) outperforms DINO (ImNet) on all fine-tuning tasks.
CLASSIFICATION
OBJECT DISC.
SEMANTIC SEG.
OBJECT DET.
LP
K-NN
JACC
CORLoC
MIOU
ACCm
MIOU
DINO (Caron al. 2021)
71.4
69.0
44,5
59.6
33,9
44.3
37.1
32.1
IBOT (Zhoa 2022a)
100
72.1
69.4
44,5
59.7
35.2
45.1
38.9
34.4
DoRA" (ours)
71.9
69.4
44.4
60.0
35.4
44.9
39.3
34.9
DORA" (ours)
100
72.2
69.6
44.8
60.2
35.8
45.1
39.9
35.1
Table 10: Pretraining on ImageNet- VIT-S/16 pretrained, then frozen (classification and object discovery, same settings Table 5) or fine-tuned (semantic segmentation and object detection, same settings as Table 3). DORA": DoRA without tracking; when pretrained for 60 epochs, it has the same training time DINO and iBOT.

---

Published as a conference paper at ICLR 2024
METHOD
PRETRAINED
C-10
C-100
INAT 18
FLWRS
CARS
IMNET
DINO
IN-Ik
98.7
89.8
71.5
98.3
92.2
81.3
DoRA (ours)
WT Venioe
98.5
89.4
69.8
94.0
92.5
80.8
DoRA (ours)
WTal
98.8
89.9
72.2
98.7
93.1
81.4
Table 11: Fine-tuning for classification. VIT-S/16 pretrained, then fine-tuned on different image-
based datasets. Veniee::: Walking Tours (ours), single video Venice; WTa.: all videos. IN-lK: ImageNet-Ik Ik. We report the top-1 accuracy (%).
We also observe such performance gains when fine-tuning on semantic segmentation on ADE20K and object detection on MS-COCO (Table 3).
E MORE VISUALIZATIONS
Figures 5 and 6 show example attention maps obtained using SK on different clips. These figures show that SK (6) leads to attention maps that exhibit spatial locality and are well aligned with objects in the scene. Remarkably, the masks seem be even robust to occlusions, shown in the sequence with bicycle moving behind traffic lights.

---

Published as a conference paper at ICLR 2024
2
3
5
X
(a)
(b)
X,
T
(c)
Figure 5: For each input frame X, of video clip, refined cross-attention map TV E kxn (6), using Sinkhorn- Knoopp. For each object, one row T is reshaped as h/p x w/p and upsampled to an h attention map overlaid the input frame for l objects encoded in blue, red and green channel. TV yields three well separated objects shown blue, red and green.
8

---

Published as a conference paper at ICLR 2024
t=1
t=2
t=3
t=4
T
(a)
X
TT
(b)
X,
Ty
(c)
Figure 6: For each input frame X, of video clip, refined cross-attention map Ty E Rxx (6), using Knoopp. For each object, one row of TV is reshaped as h/p x w/p and upsampled to an h attention map overlaid the input frame for le objects encoded in blue, red and green channel. TV yields three well separated objects shown blue, red and green.


Reviews:
[{"review_id": "ytDhdexGqR", "reviewer": "Reviewer_YYAs", "summary": "The authors consider how both (a) data and (b) method can improve training an image encoder in a self-supervised manner. Regarding data, they introduce an open-source dataset containing long, first-person videos, and propose several advantages of this over curated image (and video) datasets. Given this dataset contribution, the authors propose a self-supervised method which tracks objects to act as a signal for a classical multi-view self-supervised learning (SSL) loss. This method leverages some key properties from the dataset, specifically that the videos have natural scene transitions and are of a person-person view. Instead of using off-the-shelf object-trackers or optical flow to establish correspondence, the authors use the attention-map between the [CLS] token from a selection of heads and the patch-embeddings. Optimal transport is used to establish unique object-patch correspondence (i.e. non overlapping patches) and then given this multi-view correspondence, a technique based on DINO is used.\n\nOverall the authors show that for many downstream tasks, their method (DORA) pre-trained on (even one) video where the number of frames is comparable to imagenet-1K (IN-1K) achieves better performance than DINO pretrained on IN-1k. In comparison when DINO is pretrained on the same data, the peformance is worse than IN-1K which suggests it is the unique coupling of dataset and training-method which helps the authors achieve SOTA results", "questions": "- What is the stability like when using SK, aside from the entropy regularisation is some annealing schedule needed that transforms the coupling matrix from soft to hard gradually?\n- Is there any intuition why the k-attention maps obtained by projecting the attention map are spatially overlapping? Is there any possibility to use a simple heuristic to avoid it that can be ablated with SK?\n- What is the training cost of using SK for every forward-pass like this?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and the supplementary work provides a good level of detail and convincing ablations and visualisations\n- The contribution of the open-source dataset (10 videos) to replicate this work is very useful for the community\n- The statistical analysis of the contributed dataset also useful\n- This is a nice use-case of Sinkhorn–Knopp to avoid having to use non end-to-end approaches like optical-flow or off-the-shelf object-detectors and the motivation which shows overlapping spatial regions when linearly projecting the attention-map (instead) is a good argument for its use\n- Overall, the results with DORA are very impressive", "weaknesses": "- I'm a bit confused about Table 4: Video Object Segmentation (DAVIS-2017). In the DINO paper they report ViT-S/16 with INet getting 61.8, 60.2, 63.4 respectively (their Table 5), however your Table 4 reports DINO as getting 59.4, 57.4, 61.4 what accounts for this difference?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LfLYBJv4GP", "reviewer": "Reviewer_DfLs", "summary": "The paper proposed a dataset *Walking Tours (WT)* consisting of high definition street-walk videos of 23 hours total length, and DoRA, a multi-object-tracking-inspired framework to learn visual representation from different views of the same objects in adjacent frames. The proposed method largely follows the DINO framework, with the local crops in DINO replaced with a tracking of objects in a video. The method is tested on several mainstream visual tasks, including image detection and segmentation, video object segmentation, object tracking, image classification and object discovery. Compared to its baseline DINO, the proposed method outperforms it clearly when using the same WT dataset.", "questions": "* In addition to training epochs, it would be kind to also mention the actual training time as a more practical measurement of the training cost.\n\n* In Appendix D, are there any other differences between DoRA without tracking and DINO other than the crop generation method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The motivation to learn visual representation using of the appearance variation of object along time is sound and is probably worth exploring in the future as an additional source of information in self-supervised visual learning.\n\n* The proposed architecture to utilize temporal information by tracking the same object in different frames is novel, and the visualization can justify that the proposed method is working as expected.", "weaknesses": "* **Scalability concern.** Although the proposed method does outperform DINO on all reported experiments in controlled experiments (i.e., with the proposed WT dataset), the results with even WT_{all} is still not clearly better than DINO with ImageNet-1k. This leaves it unknown whether the WT dataset will eventually outperform ImageNet-1k (or the even larger ones like ImageNet-22k and LVD-142M) with the dataset at a reasonable scale. Also the experiments are mostly on ViT-S, which is relatively small compared to the well-known works in the field (which usually report at least ViT-B), so it is also hard to tell whether the proposed method scale well with the model size.\n\n* **Significantly worse results on image classification.** I have noticed that the image classification results of WT-pretrained models are lower than ImageNet-1k-pretrained by a fairly large margin (45LP / 36KNN on WT vs. 72LP / 70KNN on ImageNet). Although one can argue that this is because of the domain gap between WT and IN, I would consider the accuracy difference large enough to require some formal justification (e.g., run WT and IN pretrained models on a 3rd classification dataset like iNaturalist or Places) to conclude that the WT-pretrained models are not significantly weaker on image classification tasks.\n\n* **The potential privacy and safety issues of the dataset.** Also see *Details Of Ethics Concerns*. As the paper claims the dataset as a main contribution, I would expect more efforts in assessing the privacy and safety issues in the dataset (e.g., How many clear faces are detected and what are their resolutions? How many harmful scenes are detected to the best effort of the authors?) and clarifying the legal issues and usage restrictions of the dataset (e.g., Is it possible that some videos are taken down upon the request from people appearing in them? Is their usage in some jurisdictions not allowed / limited to non-commercial only? What are some possible negative effects if the models remember the private information in it? What are the possible effects of some common mitigations, like blurring the faces in Google street view?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vfRXQvOmOn", "reviewer": "Reviewer_CACu", "summary": "This paper proposed a new perspective on self-supervised learning (SSL). Instead of pretraining models on ImageNet-like object-centric datasets, the paper pretrained the models on egocentric videos (“Walking Tours” dataset) which depict numbers of objects and are comparable with human learning. Compared with other video datasets for SSL, Walking Tour dataset had more objects and classes and more gradual shifts in lightness. To pretrain on Walking Tours, the paper proposed a novel SSL method, based on DINO, to first discover objects and then track objects, named DoRA. In every batch, DoRA randomly sampled 8 frames temporally separated by 1 second, discovered objects in the first frame, and tracked them over the following 7 frames. In the default setting, objects are tracked by cross-attention in the multi-object tracker, which leads to spatially overlapping. The paper then proposed to establish object-patch correspondences using the Sinkhorn-Knopp algorithm to deal with the problem. After finding separated objects, the input video clip in the student branch would be masked to contrast with the clip in the teacher branch. Experiments on dense prediction tasks show that DoRA on Walking tours shows comparable performance with other SSL methods pretrained on ImageNet.", "questions": "1. In Table 4, why does DoRA perform worse when using WT_all than when using WT_Venice?\n2. DoRA shows inferior performance on ImageNet linear probe (LP) but superior performance on dense prediction tasks, would it perform better than other contrastive methods on the ImageNet fine-tuning task? Like MAE [1], lower on LP but higher on fine-tuning.\n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper proposed a new pretraining method on egocentric videos which are uncurated and comparable with human learning. As these videos do not involve human annotation, they can be easily obtained, leading to a promising way of SSL.\n2.\tThe proposed method is simple and neat. DoRA provides an intuitive but effective way to learn from frames that contain multiple objects.", "weaknesses": "1.\tThe paper mainly talks about how we can learn discriminative representations from egocentric videos, whereas what kind of egocentric videos are suitable for DoRA is not deeply discussed. It would contribute more to the community if we knew what properties a video should have to be worth learning.\n2.\tA good SSL method should be scalable, not only on the dataset but also on the model structure. It would be better for authors to show more results on larger ViTs.\n3.\tSome minor writing problems. (1) in Sec. 4 “Discovering objects with multi-head attention”, $\\widetilde{Q}$, $\\widetilde{K_t}$ are only defined in Fig.3 and are not defined in text. (2) In Fig.3 (Left), the input should be $X_t^{o_i}$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HgpHHR9IeZ", "reviewer": "Reviewer_krCr", "summary": "This work introduces a method for self-supervised learning of image representation models. It is based on a similar scheme to DINO (Caron et. al. 2021) that learns image representation by distilling a moving average teacher model's representation of the global image view to the representation of a student model's on multiple local views. The novel way of achieving this in this work is to use the feature correspondence provided in hours-long walking tour videos to provide tracking, which can identify the location of different objects in any video frame without annotation. This object-centric way of generating local views seems to lead to good learned representation, which is examined in the experimental section.\n\nIn total, the proposed method does not need curated image or video data for self-supervised learning and can learn effective representation from hours-long walking tour videos. The proposed multi-object masking approach is shown to be significant in learning effective visual representation.", "questions": "I have the following questions after reading the text:\n\n1. In Eq. (8) the learning is done on a single frame. Because the tracking has already provided corresponding between locations across multiple frames, is there a certain consideration to not use views from multiple frames in this loss function?\n\n2. The authors have presented 10 walking tour videos. The results in Table 5 suggest training on one video already achieves similar accuracy obtained by training on all videos. Does this mean one video is sufficient? Is there some point in further scaling up the training data? I would like to see a discussion on this topic.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "+ The fact that this method does not use curated data is a big plus for me. The way of collecting this type of data seems to be easily scalable, as it can be from walking tours or vehicle-mounted cameras. \n\n+ Using correspondence-based tracking to provide localization is sound and practical in representation learning. \n\n+ The experimental results show the model trained with the proposed method on the walking tour videos can outperform strong baselines trained on curated datasets such as ImageNet and Kinetics-400.", "weaknesses": "- One minor issue I have about the presentation is the introduction of the tracking module. The tracker is not learned, and it is only used to provide object locations. I would like further discussion on the potential use of the corresponding information. Also, a comparison on the effect of using different types of unsupervised trackers would also help strengthen the work as the major idea seems to be not dependent on a certain type of tracker.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UryWVVBzLM", "reviewer": "Reviewer_GQ4C", "summary": "This paper investigates self-supervised representation learning from video with a specific focus on the data distributions. In particular, the paper questions the needs of using large-internet scale image datasets and propose instead to learn representation by watching few long videos.\n\nThe paper makes two mains contributions:\n-\tThe WTtour datasets which composed by 10 long-videos \n-\tDORA, a self-supervised representation learning approach that learns to represent and track object in a video at the same time.\n\nThe paper evaluates the learned representations on various downstream tasks including ImageNet linear probing, Pascal VOC unsupervised object discovery, MS-COCO/ADE20K object detection/segmentation and DAVIS-2017 for video understanding. DORA pretrained on WTours demonstrates strong performances on ADE20K and MS-COCO", "questions": "I like the motivation and the novel exploration of the paper. However, I think the experimental evaluation could be improved to better support the claims of the paper. \n\nFirst, I think comparing with state-of-art image-baseline such as DINOv2 and CLIP on the different tasks would really highlight the importance of video pretraining. Second, I think it would be useful to discuss in depth the relation with the VITTO approach. Finally, the current approach currently falls short of image-pretrained model on ImageNet. It would be nice if the author could discuss this limitation in the manuscript.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- This paper explores the pretraining of visual representation using few long videos which is an original and novel empirical setting.\n\n- They propose a new datasets WTours which could be of interest to the representation learning community.\n\n-  Dora obtains reasonable performance when fine-tuning the performances on ADE-20K and MS-COCO.", "weaknesses": "- Performance on ImageNet linear probing seems low. While I understand that video-pretrained models have a disadvantage over image-pretrained model as they can’t be pretrain ‘in-distribution’ with respect to imagenet. However, ImageNet is a standard vision task. It is important to understand why there is such a gap between image and video models on this evaluation.\n\n- The DINO baseline is trained for 100 epochs only which is not the default setting. Additionally, DINO paper reports a performance of 61.8 with a VIT.S/16 on Davis while the paper reports of 54.6 for the same method. I would encourage the authors to report what are the performances of the DINO released models as those models are available.\n\n- DORA shares some similarity with the VITTO. Both approaches learn from video and use an 'unsupervised' pooling mechanism. However, DORA seems to underperform VITTO on the ADE20K and MS-COCO tasks. \n\n- Missing comparison with more recent baselines. It would be nice to add comparison with DINOv2 and a weakly-supervised baseline  OpenCLIP, which are both state-of-art methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)