Dense Unsupervised Learning for Video Segmentation
Authors: Nikita Araslanov
Simone Schaub-Meyer
Stefan Roth
Venue: NeurIPS 2021 Poster (2021)
Keywords: unsupervised learning
video object segmentation
representation learning from unlabelled videos

Abstract: We present a novel approach to unsupervised learning for video object segmentation (VOS). Unlike previous work, our formulation allows to learn dense feature representations directly in a fully convolutional regime. We rely on uniform grid sampling to extract a set of anchors and train our model to disambiguate between them on both inter- and intra-video levels. However, a naive scheme to train such a model results in a degenerate solution. We propose to prevent this with a simple regularisation scheme, accommodating the equivariance property of the segmentation task to similarity transformations. Our training objective admits efficient implementation and exhibits fast training convergence. On established VOS benchmarks, our approach exceeds the segmentation accuracy of previous work despite using significantly less training data and compute power.

Dense Unsupervised Learning for Video Segmentation
Nikita Araslanov
Simone Schaub-Meyer'
Stefan Roth
'Department of Computer Science, TU Darmstadt bessiann.A arastaa,,,,,, simone simone.schaub, stefan. (ooth@vvisiin tu tu-darmstadt.de
Abstract
We present a novel approach to unsupervised learning for video object segmenta- tion (VOS). Unlike previous work, our formulation allows to learn dense feature representations directly in fully convolutional regime. We rely on uniform grid sampling to extract set anchors and train our model to disambiguate between them on both inter- and intra-video levels. However, a naive scheme to train such model results in a degenerate solution. We propose to prevent this with a simple regularisation scheme, accommodating the equivariance property of the segmentation task to similarity transformations. Our training objective admits efficient implementation and exhibits fast training convergence. On established VOS benchmarks, our approach exceeds the segmentation accuracy of previous work despite using significantly less training data and compute power]
Introduction
Unsupervised learning of visual representations has recently made considerable and expeditious advances, in part already outperforming even supervised feature learning methods [8] [1]. Most of these works, however, require substantial computational resources [8 [1], and only few accommodate
one of the most ubiquitous types of visual data: videos [13] 33). In contrast to image sets, video data embeds ample information about typical transformations occurring in nature. Exploiting such may allow systems to learn more task-relevant invariances, instead of relying only on hand-engineered data augmentation (30 In this work, propose learn such invariances with a novel framework for the task of video object segmentation (VOS). In contrast to previous works [17 19], we learn dense representations efficiently
a fully convolutional manner, previously considered prone to degenerate solutions [17]. This is achieved by leveraging the assumption of feature equivariance to similarity transformations (e. g., multi-scale cropping) applied the input frame.
Fig. [ presents an overview of our approach. For each video sequence, we designate one of the frames the reference frame. We sample set of features, produced from the reference frame by fully convolutional network, on a spatially uniform grid, and refer to them as anchors. Using a contrastive formulation [12, approach learns represent the temporally proximate frames the reference in terms of these anchors. Importantly, this representation is learned to be equivariant to similarity transformations. Towards this goal, we devise self-training objective (20) that generates pseudo labels by leveraging the second, transformed view of the original video sequence. These pseudo labels contain dominant assignments of the features from the second view to the anchors. Following the self-training mechanism, we learn to assign the features in the original view consistently with the transformed view. Our self-supervised loss further disambiguates the anchors themselves spatially and between independent video sequences, while also ensuring their transformation equivariance.
Implementing this process in novel framework, attain new state of the in video segmentation accuracy, with compelling computational and data efficiency. Our framework can be trained on
Code (Apache-2.0 2.0 License) available htttps://ggitaab.cmmm//iisssisit
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

embedding space viden-level discrimination
reference frame
anchors
4444
anchor affinity self-supervisio
soft assignment
assignment
similarity transform
spatial discrimination
view
I
video
View
video B
Figure 1: Illustration of the main idea. We learn dense visual feature representations in an unsupervised way by (i) discriminating the features both spatially and at the video level, and (ii) embedding video representations in terms of temporally persistent set of video-specific anchors. We devise a simple and effective approach that sidesteps trivial solutions by exploiting feature equivariance to similarity transformations of the input frame, which allows to efficiently learn dense representations in a fully convolutional manner.
a single commodity GPU, yet achieves higher segmentation accuracy than previous work, while requiring orders of magnitude less data. Improving the accuracy even further with more data, our approach also exhibits favourable scalability in a more comprehensive testing scenario, which has not yet been shown in prior work [17 38).
2 Related work
The research domains most relevant to our work are video object segmentation (VOS) without supervision and representation learning.
VOS.
The inference setting of semi-supervised VOS, which is the task considered here, is to
densely track object masks provided in the first frame. It is typically approached by learning dense feature representations to establish temporal correspondences for propagating the semantic labels. While most previous methods are supervised by pixel-wise mask annotations [e. g.12 an emerging direction is to leverage unsupervised learning. Our work contributes to these research efforts. The Contrastive Random Walk (CRW) of Jabri al. [17 and its predecessor [39 exploit time-cycle consistency constraint, akin to the forward- backkarddddddddd consistency in unsupervised optical flow estimation (24) Other methods [88 19] learn the label propagation process at training time by substituting the target mask (which is unavailable at training time) with the image, an idea inspired by video colourisation (37). The algorithm for label propagation itself significantly affects the VOS accuracy. Mask propagation with naive pixel-wise correspondences usually benefits greatly from leveraging more sophisticated region-level propagation [19 which not the focus here.
Representation learning, Although previous work on unsupervised representation learning pursued spatially view-invariant objectives [5) 8 I
25 36). concurrent study [10] finds that equipping these methods with temporally invarrinn, or cmmporally-persister constraints, yields sizeable gains in terms of accuracy on action recognition benchmarks. These findings align well with previous and concurrent work tailored specifically to learning spatio-tempont representations [3] I 33 for video recognition and retrieval tasks. These works additionally distinguish the representations of frames from different videos (or, at different timesteps). The limitation of these works the context of VOS that they do not learn dense, but global feature representations at the image or video level, and typically require considerable computational budgets. By contrast, Pinhciro et al. [31] Iearn dense representations and exploit the equivariance constraint, as in our work. However, their approach is limited to image sets of static scenes, whereas we address videos of
14
15
(potentially) dynamic scenes here.
Semi-supervised representation learning. Recent works 13 leverage optical flow 34 to ensure the representation similarity of temporally corresponding features. Such an objective can be alternatively supervised by motion segments (28). Spatial equivariance has been also explored in part co-segmentation [16], a setup that requires semantically aligned image pairs, as well as in weakly supervised semantic segmentation [40] and unsupervised domain adaptation [. In contrast to this body of work, we learn establish temporal correspondences via a suitable feature representations learned in completely unsupervised way.

---

Clustering, Contrastive learning methods are inherently expensive due to the size of pairwise feature comparisons they consider training [8] [4]. Clustering provides a more computationally efficient alternative [6], since pairwise distances need to be computed only w. r. I. (moderately sized) set of pre-defined clusters. Combining these ideas, Caron et al. propose to cluster the embeddings and then enforce consistency between the cluster assignments by contrastive learning. The main challenge in this approach is to impose an expressive prior on the cluster assignments. For example, a uniform prior prevents trivial solutions where feature assignments collapse to single cluster.
Here, we neither explicitly define the clusters nor the prior. Instead, we sample attractor points, called anchors, directly from the available feature representations using spatial grid sampling strategy.
3
Method
Learning feature representations from unlabelled data is an inherently ill-posed problem and requires specifying assumptions about what constitutes a useful feature embedding with downstream task in mind. We begin with outlining the assumptions guiding our unsupervised learning method for video segmentation, before delving into the technical specifics of their implementation. We then look in more detail into the mechanisms that our method implements to comply with those assumptions.
3.1 Assumptions
Assumption 1: Non-local feature diversity. We assume that the distinctness of visual artefacts in natural scene exhibits spatial correlation: the more distant two visual phenomena are the pixel grid, the more likely they are to belong to different semantic entities, and vice versa. Dense representations of the images should mirror this property, i. e. be spatially distinguishable. Note that this assumption is weaker than that of spatial smoothness, which is problematic at object boundaries, where it would encourage the same representation of the object and the background. In contrast, non-local feature diversity is agnostic to the spatial arrangement of the objects and the background in the scene. On feature grid, where every cell corresponds its own receptive field in the image, we can discriminate the representation on all levels of the semantic hierarchy (provided sufficient grid resolution): between the objects and the background, well as between object parts. Jabri al. [17 implicitly relied on this assumption to spatially distinguish the nodes of the space-time graph.
Assumption 2: Temporal coherence. We posit that the feature representation extracted from one frame in a video should be closer in the embedding space (e. g., in terms of the cosine similarity) to a feature vector from another temporally close frame of the same sequence, rather than to feature embedding derived from another video. The premise for this assumption is the temporal coherence in videos: the changes in appearance within video sequence seldom significant as between two independent video shots. Also leveraged by Wang et al. [38] this is a desirable property for dense tracking, which is our focus in this work.
Assumption 3: Temporal persistence of semantic content. We further assume that the semantic content of video clips remains unchanged, at least for a short time span. Specifically, if we represent a given reference frame with a set of distinct features (following Assumption 1), the semantic content of temporally close frames can be faithfully represented with the same feature set. Note that the contrastive random walk (CRW) [17] makes a somewhat stronger assumption that every frame in a sequence must contain the same feature set as the reference frame. By contrast, under assumption every frame may comprise only a (proper) subset of the reference feature set. A Although the edge dropout technique improves model robustness partial occlusions in (17), following our assumption allows to handle full occlusions, provided the occluded object is visible in the reference frame.
Assumption 4: Equivariance to similarity transformations. Finally, require the feature represen- tation be equivariant to similarity transformations, i. e. scaling or flipping of an input video frame should produce correspondingly scaled or flipped feature embedding. This assumption has not been yet explored in dense unsupervised learning from videos, and is in contrast to the common practice of unsupervised learning from static image sets, where two views produced by random similarity transform need to yield the same global, hence invariant representation [].
While Assumptions and 2 have been studied independently in prior work their combination here, and further complement them with our occlusion-aware variant of Assumption 3 and the yet unexplored Assumption 4 on equivariance.
38
we investigate
3

---

Teatures
computing alfinities
Feature Extractor
T(- random crop & fip
cress-view consistency
T
main branch regulansingg branch
anchors
self-training
shared
Lev
Ost
Step
sampling
gradient
Feature Extractor
features
anchor affnnities
pscudo labels
Step
(a) Framework overview
(b) Anchor sampling
Figure 2: Overview. Our framework aa) consists of a single LNN-based feature extractor that processes video sequences in the main branch. The second regularising branch takes transformed version (random cropping and flipping) of the videos as input and produces pseudo labels á w. E f. the anchors y extracted from the features of the main branch. The self-training loss Lst learns space-time embeddings implementing our Assumptions 1-3 and, implicitly, Assumption 4 on feature equivariance. The cross-view consistency loss Lev further facilitates the latter explicitly. To sample the anchors [b] we first randomly select one frame per video sequence in the current batch (Step 1), and then sample the features at random on a uniform grid (here, with size x 3) in Step obtain q.
3.2 Overview
The core of our training procedure is a self-supervised loss imposing four assumed properties on the feature representation. First, learn to densely represent every video frame terms of spatially discriminative features (Assumption 1). Second, the set of discriminative features representing video is learned be distinguishable from the set of another video sequence (Assumption 2). Third, our method learns to represent every frame in a video sequence as a composite of discriminative features extracted from a single reference frame of the same sequence (Assumption 3). Last, we learn a feature representation satisfying these properties under the equivariance constraint (A (Assumption 4). Specifically, we minimise the distance between every pair of spatially corresponding features extracted from two views and related by similarity transform w: (. non-corresponding pairs.
3.3 Framework
The feature extractor in our framework, illustrated in Fig. 2a is a fully convolutional network [22]. It processes two copies of the input image batch, comprising the same set of video sequences, in the main and regularising branches. The purpose of the regularising branch is to prevent degenerate solutions observed in previous attempts to learn feature representations in fully convolutional manner 17. While the main branch receives the original video frames, we feed a transformed version of these frames to the regularising branch. Specifically, we extract random multi-scale crops and apply horizontal flipping at random. We denote this random similarity transform as T(-). Both the main and the regularising branch produce dense L2 normalieed K -ddimensional feature tensors k, k E K.T.KK respectively, where [| |l [.k.j....mlll the number of selected frames from video sequence, and h. are the spatial dimensions of these embeddings. Terms related to the regularising branch are denoted with the notation (") in the following. In the next step, compute pairwise feature distances to leverage contrastive learning [1]. Recall from Sec. 3.2 that our goal is to discriminate features (i) spatially within individual frames under the equivariance constraint, and (ii) temporally, to represent cach frame in video sequence in terms of the same feature set while distinguishing between independent video sequences.
1, B is the batch size, T is
Anchor sampling. We take a bi-level sampling approach based on clustering for improved training efficiency, illustrated in Fig. [6 First, for each video sequence in the batch, we sample one frame at random. We define it as the reference frame for the video sequence from which it originates. Relying on Assumption that the frames the same sequence share semantic content, we leverage the randomly chosen reference frame for extracting set of video-level features, i. encouraged to be shared between temporally close video frames. more detail, instead of computing the pairwise distances between every feature vector the batch, define spatially uniform grid of size N x N

---

on the feature tensor of the reference frame, and draw one sample per grid cell. This is to collect features that are spatially distinct (Assumption 1) and cover the full image. As a result, we obtain (B x N²) K K-dimensional feature embeddings, which we will refer to as anchors and denote as q. Defining this grid sampling operation as GN we can write this step as g == (k)).. We then compute pairwise distances between the features & and W. f. these anchors, rather than to the features themselves. The outcome is a distance matrix of size (B T h w)> x N, Compared to dense sampling, this reduces memory and the computational budget by a factor of OTT h- w N²).
Computing affinity to anchors. Following our cluster-based analogy, we compute affinities of the features I and k from the two branches f. the anchors q. Note that extract the anchors only from the main branch and share them with the regularising branch. Since the anchors q E BBN are sampled from k. they also have unit L2-norm, i. e. || = 1 for all i. Let v, E IR 1ggTher w BN² denote the affimities of k and k w. r. q. respectively, which we compute using softmax over cosine similarities w. f. all anchors, i.
ViJ Σ, exp(k)-117)
exp(k.-g,/7)
exp(k.99//7) exp(kk.-qr
(1)
where 7 IR+ is scalar temperature hyperparameter. Put simply, Va [o, 1] BThxXBN represents the similarity between feature k; and anchor (analogously for 0;
Space-time self-training. We first generate pseudo labels of anchor assignments for self self-training by acquiring the dominant anchor index for each feature from the regularising branch,
i arg max Dz.j
jEN())
where (1) is the index set of the anchors that stem from the same video clip the feature vector with index i. Observe that v and â are in spatial correspondence via a similarity transformation T. The former represents soft assignments of the feature vectors from the original view to the anchors, while the latter contains hard assignments of the features from the regularising branch. Our primary self-supervised loss minimises the distance of the features extracted from the other temporally close frames to the anchors.
LST Sg'
log ((v,n.))
(3)
where R the index set of the features extracted from the reference frames; T(-) spatially aligns the soft affinities e of the main branch with the pseudo labels obtained with the help of the regularising branch. As we analyse in Sec. 3.4 this loss term relates to the spatial and temporal properties of
the learned representation within the same view (since we do not propagate the gradient in Eq. (2) following our self-training approach), hence we refer to this objective as space-time self-training. However, implementing Assumptions 1-3, this objective only implicitly fulfils Assumption 4 on equivariance by means of seeking a consistent anchor assignment between the two views.
Cross-view consistency. To generate the pseudo labels in Eq. 2 we assume that the dominant assignment of & to the anchors q is meaningful, despite the two representations originating from different synthetic views (related by (-)). ). By following our space-time self-training in Eq. (3). we already find this assumption largely to hold for datasets of modest size (cf. Sec. 4.3 However, we observed additional accuracy benefits and improved training stability from explicitly facilitating the equivariance, our Assumption 4, especially when training larger datasets. We impose -iew- consistency term on the reference features only. Note that it complements the super-uuee with the pseudo labels in Eq. B. which omits the reference features. Re-using our grid sampling operator (M((.) parametrised by a cross-view grid size M, we subsample the features from the main and the regularising branches as (Tk))()) and f = GMr (k) such that r á E BBBM Similarly to Eq. (T), we compute pairwise affinities between these features:
h2 E Ezg exp(r:=fi/7)
ex/()
(4)
This affinity contrasts the cosine similarity between the corresponding features (since and 2 are spatially aligned) w. x. t. non-corresponding pairs. We define the cross-view loss term to uphold this correspondence in the embedding space by minimising
Lov = = -
iER log hy.
(5)
A hyperparameter weights its contribution to the total loss used for training our framework:
== LsT + XLev.
(6)
5

---

3,4 Analysis
We now take detailed look how our framework implements the assumptions from Sec. 3.1
Non-local feature diversity (Assumption 1). Our loss disambiguates feature representations by clustering, with the anchors acting attractors. Recall that our formulation of the feature distance to the anchors in Eq. ① is contrastive, i. e. it is measured relative to all other anchors. As a result, minimising such distance would not only encourage an increased cosine similarity of the features to the anchors, but also a decreased cosine similarity between the anchors themselves. Recall that the anchors are a subset of features sampled spatially from randomly selected video frame uniform grid. Consequently, our loss learns to discriminate between spatially distinct features.
Temporal coherence (Assumption 2). In Eq. ① we compute the affinity of the features to the anchors extracted from multiple videos in the training batch. Note that we select the dominant anchors in Eq. ( for self-training only from the same video sequence the feature itself. This implies that (i) the features will be attracted only to the anchors originating from the same video, and (ii) the distance between the anchors and the features from different video sequences will increase by virtue of our contrastive formulation of the affinity. Note that the latter also implements inter-video discrimination, analogous to instance-specific learning from image sets, which was shown to result in class-specific representations [e. g..
Temporal persistence of semantic content (Assumption 3). Our anchor sampling strategy ensures that all anchors stem from single video frame, the reference frame. By design of the pseudo labels (ef. Eq. ( our self-training loss in Eq. (3) aligns the representation of the other frames only with the reference originating from the same video.
Equivariance (Assumption 4). We generate a random similarity transformation to synthesise the input the regularising branch. The task of video object segmentation (VOS), studied here, is equivariant under this family of transformations: flipping or scaling the image should result in a corresponding change the segmentation output. Our cross-view consistency loss in Eq. (5) explicitly facilitates this property.
3.5 Discussion
Preventing degenerate solutions. In our preliminary experiments without the regularising branch and Assumption 4, we observed that the model would rapidly converge to a trivial solution, previously also reported by Jabri et al. [17]. Our investigation suggested that the network was encoding positional cues into a degenerate feature representation. We hypothesise that such solutions may be consequence of the limited spatial invariance in contemporary CNN implementations (2) and the widely adopted padding. The padding type is, in fact, irrelevant, since any deterministic padding strategy provides stable and predictable pattern. We also found it necessary to spatially jitter the grid for sampling the anchors, rather than to maintain a fixed offset, to prevent further shortcut solutions.
Computational scalability. Using grid sampling strategy to extract anchors allows for considerable computational savings. For example, if hxw = 32 x 32, and we use a grid of size N X = 8 x 8, this will have computational and memory saving factor of 16. This saving becomes even more valuable we scale up the storage costs of the affinities v and 0, which have the size of B T h- w X B N W. r.t. B and T. Improving the computational footprint with such subsampling is non-trivial in MAST 19 due to the use of the dense reconstruction loss, and in CRW 17] as it extracts image patches with 50% overlap, hence propagates the same pixels through the network up to three times.
Practical considerations. The order of the frames from a sequence in the batch is irrelevant and can be randomly selected, since do not require any assumptions about motion continuity. Further, do not use any momentum network or queue buffers, e, g. such as in [4]. Our training is surprisingly stable without any of them. Further, we only use similarity transformations to augment the training data, but no appearance-bart on learning natural changes in the appearance directly from video data.
augmentations such as photometric noise 8 11 14. Instead, we rely
Comparison to Caron et al. (5). contrast to [5], the representation of our clusters (anchors) is non-parametric. Specifically, we do not learn global set of cluster features shared by the complete video dataset. The advantage is that we neither need to set the number of clusters a priori, nor to specify prior the cluster assignment.

---

Table 1: Video object segmentation quality on DAVIS-2017 validation in terms of the mask (J) and boundary (F) accuracy (IoU). The subscript [-lr denotes the recall of the metric, while [.mm signifies the mean. x2 denotes an output stride of instead of 8, e. the spatial feature resolution is twice as large. # details dataset characteristics: the number of videos the total dataset duration in hours.
Method
Train Data
# #/L
Im
J.
Fm
F, J&J
Baseline (random initialisation)
43.1
CorrFlow 18
48.4
53.2
52.2
56.0
50.3
MAST
OxUvA
366/ / 14
61.2
73.2
66.3
78.3
63.7
Ours
63.4
76.1
67.2
79.7
65.3
MAST 19
63.3
73.2
67.6
77.7
65.5
YT-VOS
4.5K 5
Ours
67.1
81.2
71.6
84.9
69.3
ContrastCor
60.5
65.55
63.0
TrackingNet
30K 140
Ours
67.1
80.9
71.7
84.8
69.4
CRW
17
64.8
76.1
70.2
82.1
67.6
Kinetics-400 300K 833
Ours
66.7
81.4
70.7
84.1
68.7
Comparison to CRW [17]. The implementation of our approach simpler, since the frame ordering within the batch is irrelevant, and more computationally efficient, as we compute feature affinities in parallel rather than sequentially (see Table 3 for detail). We also use a weaker assumption on semantic persistence (Assumption 3), which can be advantageous in video sequences with occlusions. Additionally, we learn to discriminate features between different videos, which is not part of CRW.
Experiments
To evaluate the learned feature representations, we conduct experiments in the setting of semi- supervised VOS. The task provides a set of segmentation masks for the first frame in a video sequence and requires the evaluated algorithm densely track the demarcated objects in the remaining frames. We largely follow the VOS setup of Jabri et al. [17] and evaluate our method on DAVIS-2017 (35). Following Lai et al. [1). we additionally test our approach on the KouTube-VOS val by submitting our results to an evaluation server [42).
Implementation details. Similarly to Jabri et al. [1]. we use ResNet- 18 as the backbone network for our feature extractor. We evaluate the correspondences using the output of the fourth (the last) residual block. To obtain the output stride of 8 in our feature extractor, we remove the strides of the last two residual blocks, as previous work 17 38 At training time, we first scale the video frames such that the lowest side is between 256 and 320 pixels, and extract random crops of size 256 x 256. We train our network with Adam and the learning rate 10 4 on the smaller YouTube- VOS and OxUvA (35). whereas found SGD with the learning rate 10-3 to work better the larger Kinetics-400 and TrackingNet datasets. We set the temperature r = 0.05 throughout our experiments; observed its influence the accuracy to not be significant. The hyperparameter X, trading off the influence of the -ress-view consistency, equals 0.1 by default, and we empirically study its role in Sec. 4.3 We train models one A100 GPU, although training our most accurate configuration of the framework requires only 12GB of memory, hence a single Titan X GPU is actually sufficient.
Label propagation. To propagate the semantic labels from the initial ground-truth annotation, rely on the label propagation algorithm implemented by Jabri et [17, detailed in Appendix C In particular, for every feature embedding in the current frame, compute its cosine similarity w. r. the features in the previous Nr = 20 context frames and the first reference frame. We select NK 10 feature embeddings with the highest similarity, divide the similarity by and compute softmax to obtain normalised affinity values. The mask prediction for this feature embedding is a convex combination of the mask predictions of its neighbours, weighted by this affinity.
4.1 Evaluation on DAVIS-2017
To evaluate on DAVIS-2017 [29 val, we independently train our feature extractor on 4 datasets. The OxUvA dataset 35 spans 366 video sequences with total duration of 14 hours. The second
7

---

dataset is YouTube- VOS 42. which by comparison to OxUvA contains more videos (around 4.5K sequences), but has shorter overall duration of 5.6 hours. In addition, we train on larger datasets, TrackingNet (26) and Kinetics-400 77. These datasets contain significantly more video sequences, albeit of lower resolution and quality (e. 8., due to compression artefacts). While running the inference on DAVIS-200 2017 [29]. we process the frames at a resolution of 480p for fair comparison.
Table reports the segmentation accuracy in terms of two types of metrics: The Jaccard's index measures the iitersection-over-mniioo (IoU) of the object mask respectively the contour; we report the mean of mask and contour IoU, Im and Fm. as well as the recall J, and F, (with an IoU threshold of 0.5) [29]: J&& J&Fm m is the average of Im and Fm. In a comparable setting where all methods use the same dataset for training, our approach clearly improves over the state-of-the-ar accuracy. When trained on OxUvA, our approach outperforms MAST [9] by 1.6% J&Fm m- This improvement is even more pronounced when trained on YouTuue- VOS, where we outperform MAST 19 by 3.8% in J&Fm m score. This improvement is especially notable, since MAST [19] produces its feature embeddings at twice the resolution of our method, hence has a larger memory footprint. These results are consistent when training our approach NrackingNet, we improve over the previous result of Wang et al. [38] by 6.4% J&Fm. Compared to CRW 117. our approach reaches higher J J&Fm, m score by 1.1%. Remarkably, using the smaller YouTube- VOS dataset for training, our method already achieves comparable or higher VOS accuracy compared to previous work [17 38) that required significantly larger datasets for training. We also observe that the diversity and the quality of the dataset, i. e. the number of the videos and their resolution, tends to improve the quality of our learned features in terms of the segmentation accuracy.
larger but less carefully curated datasets. Using
4.2 Evaluation on YouTube-VOS
Following Lai et al. [19], we additionally evaluate our features on the YouTube- VOS 2018 valid split. With 474 video sequences containing 91 unique object classes, its diversity represents a significantly more challenging and comprehensive test bed for VOS than DAVIS-2017 We train our model on three datasets as before: YouTube- VOS,, TrackingNet, and Kinetics-400. The train set of YouTube- TOS contains only some of the object classes present in valid and the benchmark distinguishes between these as "seen" and "unseen" categories. We submit our results to the official evaluation server to obtain the quantitative metries In and F... YouTube-VOS valid has additional specifies compared to the DAVIS benchmark and previous work tends to use a different label propagation strategy to address them (e. g., the initial object masks may appear in different intermediate frames, rather than only in the first frame at once). Since we use the label propagation from CRW 117, we also evaluate this model trained on Kinetics-400 for a fair comparison. Note that the label propagation of MAST [19 benefits from a larger feature resolution and a more advanced two-stage inference process: first detecting a ROI and then computing feature correlations bounded by the ROI.
The results in Table show that our approach sets a new state of the art, improving over CRW 17 by 0.8% mean score. In comparable setting when method and CRW use the same training data, Kinetics-400, we improve by 0.7%. With larger training datasets, Kinetics and TrackingNet, we reach higher VOS accuracy compared to training on the smaller YouTTbeeVOOS Such scalability has not been shown in previous work [17 (38) where the training dataset was limited to a single instance, or two instances of comparable (small) scale 19.. Notably, the accuracy from training on Tube-VOS already matches that of CRW when trained on much more data. Finally, we also outperform fully supervised methods, OSVOS 4 and PreMVOS 23 especially on unseen categories.
4.3 Further analysis
We study (i) the interplay between the number of video sequences B per batch and the number of frames T' per video; (ii) the influence of the cross-view loss term via the trade-off hyperparameter X; (iti) the framework's 's sensitivity the size of the sampling grid of the anchors N and (iv) cross-view features M; (v) increasing the temporal window of the video clips in training. We conduct all experiments by training the corresponding model on YouTube- VOS. Fig. 3 visualises the results in terms of the &&F, score on AAVIS-2017 (val). Our baseline configuration. depicted as the centre node corresponds to the default setting of B = 16, T = 5, > = 0.1, N = 8, M = 4. Every edge of the same style and colour represents a change of selected hyperparameters, while keeping the default values for the rest. To put the significance of the differences into perspective, we also compute the standard deviation of the J FFF score on DAVIS-2017 across 5 training runs, amounting to +0.42.
8

---

Table 2: Results on YouTube-VOS 2018 (val). We compare our models pre-trained on YouTube- VOS, TrackingNet, and Kinetics to previous work. CRW [7 and Colorize [37 were trained on Kinetics-400; CorrFlow [18] and MAST [19] used OxUvA and YouTube- VOS, respectively.
T
60.0
B
80
69.0
16x 16
T
2
A
B
8x8
67.1
68.5
67.8
CRW 17 uses the same label propagation as ours.
69.3
= U
Method
Seen
Unseen
Mean
68.0
Im
Fm
Im
F
67.0
68.7
T= 10
4x4
Unsupervised
10
Colorize (37
43.1
38.6
36.6
37.4
38.9
61.1
66.2
69.3
T=
CorrFlow 18
50.6
46.6
43.8
45.6
46.6
MAST 19
63.9
64.9
60.3
67.7
64.2
Varying T
Varying
CRW 17
68.7
70.2
65.4
75.2
69.9
NxN
I.N.IE Tx B 80
M M
Murs-YoTTubeVOR
69.6
71.3
65.0
73.5
69.9
Ours Kinetics
69.9
71.3
66.5
74.8
70.6
Figure 3: Ablation study. We report J&Fm
Ours- Net
70.2
71.9
66.3
74.5
70.7
DAVIS-2017 (val) by varying the hyperpa- rameters of our baseline configuration (centre);
Supervised
16 (no. of videos in batch), T (no. frames per video), À = 0.1 (cross-view trade- off), N 8 (anchor grid size), M = (cross-
B
OSVOS 4
59.8
60.5
54.2
60.7
58.8
PreMVOS 23
71.4
75.9
56.5
63.7
66.9
STM 27
79.7
84.2
72.8
80.9
79.4
view grid size). See Sec. 4.3 for details.
Balance between the number of videos and their size. The effective batch size of our method is BxT, hence depends on the number of videos and their duration. Here, we investigate the balance between the size the video set T and the number of distinct video sequences B used in a single training batch. We fix B T to 80 as the invariant, and evaluate three additional configurations of and T. Our main observation in Fig. iis that using multiple frames, i. e. ensuring T > 1, is crucial: training with only a single frame per video (T [1=1) 1) reduces the quality of learned feature representations significantly, by 9.3% J&F. In light of Assumption 3, this is expected, as only in multi frami-frame setting our model can learn shared representation of the video set.
The influence of the cross-view consistency % By setting 7 = O, we train our approach without the cross-view consistency (cf. Eq. (5)) We find that this results in an accuracy drop of 2.3% J&F m- confirming its accuracy benefits. On the other extreme, we increase its values from 0. to 1.0 and find it dominating over the space-time self-training term, hence inhibiting learning of temporal coherence: J FFF. decreases by 0.8%. Overall, we confirm that the benefits of data augmentation used via cross-view consistency is only complementary to self- traiitrain This is in contrast the pivotal role of data augmentation in prior work [e. g., 5 8 Our main hypothesis, claborated in Appendix B.2 is that artificial image transformations do not reflect the transformations occurring in time in video sequences, which is important to establish accurate temporal correspondences.
Grid resolution. We vary the size of the sampling grid for the anchors N and the cross-view consistency M (Eq. (5)). Since a high grid density (i. e. large N and M) implies an increased computational footprint (cf. Sec. 3.5). we seek to reduce it without compromising VOS accuracy. A low grid resolution may be too coarse in terms of discriminating sufficient level of detail. Conversely, high grid resolution may focus the learning more on discriminating between object parts, which may be detrimental to object-level disambiguation. We find that both of these extremes lead to a tangibly lower VOS accuracy, g. using N 4 leads to drop in J & score by 8.2%. Varying M leads to a more moderate decrease in J J&Fm. by at most 1.5% for M = 8. Regarding the discrepancy between the optimal N 8 and A = 4, we hypothesise that while grid size of N = 4 may be sufficient to satisfy our Assumption 1, it may undermine Assumption 3, since a sparser grid may miss feature vectors representing the object of interest, visible in the temporally close frames.
Sampling more frames T per video. We now fix B 16 and increase the number of frames per video T. Higher T have the appeal of providing larger degrees of natural transformations. However, larger videos also contain more occlusion, which our method can handle, but also disocelusions, which violate our Assumption 3 temporal persistence, hence currently problematic. The results

---

80
MAST 19
10
30
30
MAST 1M
YouTube VOS
CRW 17
Figure 4: Qualitative examples DAVIS-2017 and CorTube-OOS The representation learned by our method is robust to occlusions, naturally occurring non-rigid transformations and deals well with object-background disambiguation.
confirm this expectation: using more frames does not further improve the VOS accuracy. For example, increasing to 15, we match F&Fm of the baseline, yet at additional computational costs.
Qualitative examples. Fig. shows examples of mask propagation, leveraging our learned feature representa- tion. We observe temporal correspondences to remain stable and accurate despite complex self-occlusions (e. g-, torso of dancer), non-rigid deformations, and appearance changes due shadows (e. 8., giraffes). Our method does not exhibit "bleeding" artefacts of MAST [19] and
Table 3: Computational efficiency.
Method
#
At
Mem
ContrastCor 38
0.4M
1d
44GB
Ours- Luns-TrackingNe
0.2M
1d
12GB
CRW 17
2M
22GB
Ours-K Kinetics-400
0.3M
2d
12GB
produces more complete segmentation masks than [17] despite using less data for training (YouTube-Vool VOS for
MAST 19
2M
22GB
the DAVIS-2017 example, TrackingNet for the YouTube- VOS result). See supplemental for more visual results.
Ours Ours-YouTulse YouTube- VOS
0.1M
6h
12GB
Computational efficiency. Table 3 summarises the computational benefits of our approach w. I. previous work. Regardless of the training data, our approach requires less time for convergence, both in terms of the number of training iterations (#) and wall clock duration At. This improvement comes with modest memory requirements: to train our framework we require only 12GB of GPU memory, twofold decrease compared prior work.
5 Conclusion
We presented a simple and computationally efficient unsupervised method for learning dense space- time representations from unlabelled videos. Our approach exhibits fast training convergence and compelling data efficiency. We achieve VOS accuracy surpassing previous results despite using only fractional amount of the training data required in prior work. We recognise the possibility of our research results to be used with malicious intents, such as for unauthorised surveillance. However, as a product fundamental research with low technology readiness levels of to this is highly unlikely. Our method is yet unable to handle disocclusions. We are excited to explore such capability improve learning wider spectrum of invariances by leveraging larger temporal windows in videos containing complex (egoo)motion where disocclusions are more likely occur.
10

---

Acknowledgments and disclosure of funding
This project has received funding from the European Research Council (ERC) under the European Union's Horizon2020 research and innovation programme (grant agreement No. 866008). This work has also been co-funded by the LOEWE initiative (Hesse, Germany) within the emergenCITY center.
References
[1] N. Araslanov and S. Roth. Self-supervised augmentation consistency for adapting semantic segmentation. In CVPR. pages 15384-15344 2021.
[2] A. Azulay and Y. Weiss. Why do deep convolutional networks generalize so poorly to small image
transformations? J. Mach. Learn. Res., 20:184:1 10:184:1-184:25, 2019.
[3] Y. Bai, H. Fan, I. Misra, G. Venkatesh, Y. Lu, Y. Zhou, Q. Yu, V. Chandra, and A. L. Yuille. Can temporal information help with contrastive self-supervised learning? arXiv:2011.13046 [cx.CV] 2020.
[4] S. Caclles, K. Maninis, Puset, Laixe D. Cremers, and L. V. Gool. One-shot video object segmentation. In CVPR, pages 5320 5329, 2017.
[5] M. Caron, Misra, Mairal, P. Goyal, Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NewrIPS*0200, pages 9912-9924.
[6] Caron, Bojanowski, A. Joulin, and Douze. Deep clustering for unsupervised learning visual features. In ECCV, volume 14, pages 139-156 2018.
[7] Carreira and A. Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In CVPR, pages 4724-4733, 2017.
[8] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. In ICML., pages 1597-1607, 2020.
[9] I. Dave, R. Gupta, N. Rizve, and M. Shah. TCLR: Temporal contrastive learning for video representation. arXiv:2101.07974 fes. CV), 2021.
[10] C. Feichtenhofer, H. Fan, B. Xiong, R. B. Ginshick, and K. He. A large-scale study unsupervised spatiotemporal representation learning. In CVPR, pages 3299-3309, 2021.
[11] Grill, Struh, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, Docrsch, B. Á. Pires, Z. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent A new approach to self-supervised learning. In NeurlPS#2020, pages 21271-21284.
[12] M. Gutmann and A. Hyvärinen. contracont estimation: A new estimation principle for unnormalized statistical models. In AISTATS, volume 9, pages 297-304, 2010.
[13] T. Han, W. Xie, and A. Zisserman. Self-supervised co-training for video representation learning. In 2000 pages 5679-5690.
[14] K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, pages 9726 970569973,2007 2020.
[15] O. Hénaff. Data-efficient image recognition with contrastive predictive coding. 4182-4192, 2020.
ICML, pages
[16] Hung, Jampani, S. Liu, P. Molchanov, Yang, and J. Kautz. SCOPS: Self-supervised co-part segmentation. In CVPR, pages 869-88 (19-878,77 2019.
[17] A. Jabri, A. Owens, and A. A. Efros. Space-time correspondence as a contrastive random walk. In 2000 pages 99545-19564
[18] 7. Lai and W. Xie. Self-supervised video representation learning for correspondence flow. In BMVC, 2019.
[19] Z. Lai, E. Lu, and W. Xie. MAST: A memory-augmented self-supervised tracker. In CVPR, pages 6478-6487, 2020.
[20] D.-H. Lee. Pseudo-label: The simple and efficient semi-supereised learning method for deep neural networks. In ICML Workshops, volume 2013.

---

[21] X. Li, S. Liu, S. D. Mello. X. Wang. J. Kautz, and M. Yang. Joint-task self-supervised learning for temporal correspondence. In NeurlPS#2019, pages 317-327.
[22] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, pages 3431-3440, 2015.
[23] J. Luiten, P. Voigtlaender, and B. Leibe. PReMVOS: Proposal-generation refinement and merging for video object segmentation. In ACCV, volume I 11364, pages 565-580, 2018.
[24] S. Meister, J. Hur, and S. Roth. UnFlow: Unsupervised learning of optical flow with bidirectional census loss. In AAAI, pages 7251-7259, 2018.
[25] I. Misra and L. van der Maaten. Self-supervised learning of pretext-invariant representations. In CVPR, pages 6706-6716 66, 2020.
[26] M. Müller, A. Bibi, S. Giancola, S. Al-Subaihi, and B. Ghanem. TrackingNet: A large-scale datasct and benchmark for object tracking the wild. In ECCV, volume I, pages 3 310-327, 2018.
[27] S. W. Oh, J. Lee, N. Xu, and S. J. Kim. Video object segmentation using space-time memory networks. In ICCV, pages 9225-9234, 2019.
[28] D. Pathak, R. B. Girshick, P. Dollár, T. Darrell, and B. Hariharan. Learning features by watching objects move. In CVPR, pages 6024-6033, 2017.
[29] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. V. Gool, M. H. Gross, and A. Sorkine-Horuung. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, pages 724-732, 2016.
[30] L. Perez and J. Wang. The effectiveness of data augmentation in image classification using deep learning. arXiv:1712.04621 [Vs. 2017.
[31] P. O. Pinheiro, A. Almahairi, R. Y. Benmalek, F. Golemo, and A. C. Courville. Unsupervised learning of dense visual representations. 2000,PS+2020, pages 4489 4500.
[32] R. Qian. T. Meng. B. Gong, M.-H. Yang. H. Wang. S. Belongie, and Y. Cui. Spatiotemporal contrastive video representation learning. In CVPR, pages 6964-6974, 2021.
[33] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, and S. Levine. Time-contrastive networks: Self supervised learning from video. /CRA, pages 1134-1141, 2018.
[34] Z. Teed and J. Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In ECCV, volume 2, pages 402-419, 2020.
[35] J. Valmadre, L. Bertinetto, J. F. Henriques, R. Tao, A. Vedaldi, A. W. M. Smeulders. P. H. S. Torr, and E. Gavves. Long- term tracking in wild: A benchmark. ECCV, volume 3, pages 692-707, 2018.
[36] A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv: 8880000778 [cs.LG), 2018.
[37] C. Vondrick. A. Shrivastava. A. Fathi, S. Guadarrama, and K. Murphy. Tracking emerges by colorizing videos. ECCV, volume 13, pages 402-419, 2018.
[38] N. Wang, W. Zhou, and H. Li. Contrastive transformation for self- self-supervised correspondence learning. In AAAL, pages 10174 10182, 2021.
[39] X. Wang. A. Jabri. and A. A. Efros. Learning correspondence from the cycle-consistency of time. In CVPR, pages 2566-2576, 2019.
[40] Y. Wang, J. Zhang, M. Kan, S. Shan, and X. Chen. Supervuee equivariant attention mechanism for weakly supervised semantic segmentation. CVPR, pages 12272-1222 122211 2020.
[41] Y. Xiong, M. Ren, W. Zeng. and R. Urtasun. Self-supervised repressntttoo learning from flow equivariance. In ICCV, pages 10191-10200, 2021.
[42] N. Xu, L. Yang. Y. Fan, Yang, D. Yue, Y. Liang. B. L. Price. S. Cohen, and T. S. Huang. YouTube-VOR sequence-to-sequene video object segmentation. In ECCV, volume 5, pages 619, 2018.
12


Reviews:
[{"review_id": "kh09arpU1yF", "reviewer": "Reviewer_zP7t", "summary": "The paper proposes a self-supervised representation learning method to learn dense spatial features for videos. The goal for the representation is to be able to perform correspondences and label propagation across frames. The proposed method involves an objective that encourages features to be distinguishable on inter-frame and intra-frame levels. Through experiments on two datasets, it is shown that these features are effective for label propagation, outperforming previous methods trained with the same data and demonstrating comparable performance to methods leveraging significantly larger amounts of training data. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n- The paper is extremely well written and is very easy to follow. The introduction concisely motivates the problem and presents an overview of the paper. The related work section does a thorough job of contrasting with existing work in this domain. The approach clearly presents the assumptions that are required to be satisfied for the objective to make sense. \n\n- The presented approach involves spatially sampling features to be used as the \"anchor\" features. Similarity of the features from a second branch to the anchor features is used as the targets. This core idea is simple to follow and is novel (to the best of my knowledge). \n\n- The paper does a thorough job of analyzing the implications of the presented objective (Sec 3.5) and presents motivation for all design choices (see below for some questions). \n\n- The proposed approach is extremely efficient in training time, especially compared to related literature in self-supervised learning. \n\n- The experimental evaluation is thorough and demonstrates superior results on two datasets compared to methods trained on the same datasets. However, the proposed representation underperforms compared to methods trained on larger datasets. Through extensive ablations, the effect of different hyperparameters are also analyzed. \n\nOverall I think this paper makes an interesting contribution to self-supervised learning (in the context of features for label propagation), that is significantly more efficient compared to existing methods and demonstrates impressive results. \n\n## Questions/Concerns\nThere are a few things that were not immediately clear and I think the paper would benefit from additional discussion on these aspects.\n\n### Anchor Sampling\nThe text presents the anchor sampling as merely a method to improve efficiency. Is this really true? If all the features were indeed used, wouldn't the model become untrainable (even in the presence of enough compute)? \nBecause in this case the anchor affinities $v \\in \\mathbb{R}^{BTHW \\times BTHW}$ would also have $v_{i,j} = 1$ if $i,j$ point to the same location. \n\n### Comparison to MAST[15]\nIn Sec 4.1, you explain that MAST generates features at twice the resolution of your method. Does this mean that they are solving a harder correspondences/label propagation problem since they have to perform 4x the number of correspondences? If so, this doesn't entirely justify the gain in performance on DAVIS-2017 (as discussed in Line 264-266).  \n\n### DAVIS-2017 Results\nIn Table 1, do all the methods use the same propagation method? If not, does your method have an unfair advantage because of the new propagation method? \n\n### YouTube-VOS Results\nSimilarly here, since the text claims that the MAST propagation method leverages additional information (like ROIs), is it possible to present results of MAST with your propagation method (using features before the ROI Align operator). \n\n### Scaling up to larger datasets\nCRW[13] seems to outperform your method, but uses significantly larger amount of training data. Since your proposed method is computationally efficient, is there a reason for not scaling up training to Kinetics or TrackingNet? \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have discussed limitations and societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours "}, "extra_text": {}}, {"review_id": "SwseCRn173K", "reviewer": "Reviewer_RkEz", "summary": "The paper presents a method for unsupervised video object segmentation. The method is based on enforcing equivariance between video batch and the same but transformed batch. The paper utilizes  anchor sampling strategy in order to reduce computational requirements, and compute the affinity between features and anchors.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive:\n- The paper is fairly well written, easy to read and to understand.\n- The paper contains thoughtful ablation studies which justify the choices made.\n- The proposed method is very simple, well-thought and easy to understand.\n\n\nNegative:\n- The paper lacks rigor comparison with the baselines. On the Davis-2017 dataset the paper is compared with CorrFlow and MAST using the same pretraining, however the label propagation mechanism is different. On VOS it is compared with CRW, but the training datased is different. So overall there is no single clear comparison with methods from the literature on the same dataset with the same propagation strategy.  Thus it is not possible to judge clearly if the method outperforms the competitors.\n\n\nSome minor staff:\n- Line 17: ample -> sample\n\n- The paper did not discuss some relevant image-based segmentation methods that also utilize equivariance, for example: SCOPS: Self-Supervised Co-Part Segmentation Hung et al.\n\n- Why is only random scaling and horizontal flipping considered as the transformation? Did the authors experiment with other kinds of transformations: rotations, sheering, color jittering?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no specific limitations or societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "There is no ethical concerns regrading this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5h"}, "extra_text": {}}, {"review_id": "RMni2J8N2bT", "reviewer": "Reviewer_cHxN", "summary": "This paper proposes a method for self-supervised pretraining on video:\n\nGiven a batch of video clips, dense per-frame features are extracted from a backbone architecture like ResNet. The objective is to apply a standard contrastive penalty where a softmax loss picks out a positive pair of feature embeddings amidst a large set of negatives. Rather than compare every feature vector to every other vector in the batch (across samples, time, and space) a subset of representative vectors are chosen for comparison. Starting from BxTxHxW features, we choose from each sample a single frame and a subsampled grid of values to get BxNxN \"anchors\" to compare to. The full set of BxTxHxW features will each be compared to all of the sampled BxNxN anchors.\n\nTo apply the loss, for each feature we need one of the comparison anchors to serve as a positive target. This is done by finding the closest match to features from an augmented version of the same clip. That is, another inference pass of the backbone takes in a randomly resized/cropped/flipped version of the clip. Those features are compared to the anchors to find the argmax closest match. This creates a pseudo-ground truth label used to supervise the original features. Importantly, the gradient is stopped so nothing is backpropped through the pseudo-label generation.\n\nPretraining is done with the proposed set up and evaluated on the downstream setting of unsupervised mask propagation on DAVIS and YT-VOS.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The good:\n\n- I think the idea itself is interesting and sufficiently innovative over the large body of emerging contrastive learning work. The idea of using the regularizing branch to choose pseudolabels makes sense to me, we've seen other work that shows a simple strategy of stopping gradients from the augmented branch will get you far [6]. Thanks to using video + dense spatial features, all the comparisons in the contrastive loss are between features from a single branch which is different from most existing contrastive methods though offers some similarity to [3] (acknowledged by the authors).\n- Overall the implementation and replication of this idea seems like it wouldn't be too much trouble, there's enough detail in the paper to pull it all together, and there's a degree to which it is simpler compared to prior video methods that rely more directly on correspondence/flow.\n- The ablations inform how to approach some of the critical design decisions of the method (how many frames should be used, how much to subsample the grid)\n\nConcerns:\n\n- It took me quite a number of read throughs to finally understand what the method was. While it may be a reflection of my own reading comprehension, I think that the paper would benefit from a clearer presentation. Perhaps some reworking of Figure 2 might help.\n- The most direct comparison is to [13] which offers similar motivation + set up + downstream evaluation, but the proposed method underperforms on both DAVIS and YT-VOS. Understandably, there are resource constraints that make it difficult to train on Kinetics to provide the apples-to-apples comparison. Still, the quantitative benchmark results in the paper could be stronger.\n\nOverall, I lean positive on this paper. I like the idea itself, though the presentation and results are not as strong as they could be. I think that this method would likely scale up well with more resources.\n\nOther comments:\n\n- It's good to see the results in Table 2b, though I'm curious, given the down and up in performance from T=5->10->15 how much variance is there in the results across training runs?\n- Why not include other data augmentation techniques? (L209) They are helpful signals to learn invariance to in other self-supervised work, seems like they would be of use here too. \n\n**Final comments:**\n\nI appreciate the effort the authors put in their rebuttal to clarify concerns around apples-to-apples comparison to prior work. I also sympathize with the headache in trying to download and use Kinetics. Overall, I'm still a fan of the proposed idea. I think the presentation and explanation of the method could be improved, but I stand by my recommendation to accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Promising to not provide advice for anyone attempting to use the proposed research in a malicious way is not exactly the strongest point, but overall the authors are correct that this is pretty basic early-stage research with few specific, harmful downstream applications.", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Ji8CHTCkvAC", "reviewer": "Reviewer_ckf7", "summary": "This paper aims at unsupervised feature learning from videos. The learned representation is tested in a video segmentation/tracking setting on the DAVIS and the Youtube-VOS datasets. Although the paper describes many principles for the learning strategy, I could not find any of them to be new and also the results do not indicate that something fundamental was added. ", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In 3.1. the paper introduces 4 assumptions to derive constraints for feature learning. However, all of them are well-known. The first one is spatial smoothness (which is, btw, wrong at object boundaries). The second and third assumptions look identical to me and introduce temporal smoothness. Also this has been commonly used for contrastive video representation learning. The last one is equivariance to scaling and flipping implemented by an augmentation procedure. Also this is pretty standard. Another highlighted concept is subsampling (motivated by spatial and temporal smoothness). I cannot see anything spectacularly new here either. \n\nIn 3.6. there is a discussion stating some questionable advantages of the proposed approach: Why is it good to ignore temporal causality? Common sense tells us that features generalize better if they model a causal process. Why is it good if clusters are instance-specific and do not span across instances? Establishing links across instances should improve generalization (though admittedly this is not relevant for video tracking). Scalability is bound to subsampling. What prevents other methods from subsampling as well to improve scalability? There is a price to pay, of course: as soon as the grid gets to coarse, there is relevant information loss (see ablation in Table 2b). \n\nThe efficiency argument is not sufficiently quantified by experiments. \n\nIn conclusion, I cannot say what this paper contributes. It is a relatively complicated description of well-known facts and procedures. Thus, I cannot recommend accepting the paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The conclusions include some superficial statements, which is sufficient for this type of paper. ", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper discusses a method for unsupervised learning of correspondences across video frames. Different from prior work like CRW [13] the proposed method offers additional flexibility (e.g., an anchor can be matched to various points). To establish the use of this additional flexibility the reviewers asked for additional baselines which the authors provided in the rebuttal. In a discussion one reviewer remained concerned about the novelty and its use (the authors admit that the improvement compared to [13] is "slight"). Moreover, the authors also state that "testing on Kinetics or TrackingNet would be useful. In particular, we recognise that training on Kinetics-400 would enable a direct comparison to CRW [13]." Up until writing of this meta-review the authors have not provided this result. AC thinks a careful comparison to very related work is desirable and shouldn't be omitted. Further, as suggested by the reviewer that remained more concerned, AC concurs that the writing and organization of this paper should be improved significantly and AC can understand why the reviewer remained concerned. AC strongly encourages the authors to improve the paper by adding additional experimental evidence and by assessing organization.
Decision: Accept (Poster)