Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a self-supervised representation learning method to learn dense spatial features for videos. The goal for the representation is to be able to perform correspondences and label propagation across frames. The proposed method involves an objective that encourages features to be distinguishable on inter-frame and intra-frame levels. Through experiments on two datasets, it is shown that these features are effective for label propagation, outperforming previous methods trained with the same data and demonstrating comparable performance to methods leveraging significantly larger amounts of training data. 
main_review: 
- The paper is extremely well written and is very easy to follow. The introduction concisely motivates the problem and presents an overview of the paper. The related work section does a thorough job of contrasting with existing work in this domain. The approach clearly presents the assumptions that are required to be satisfied for the objective to make sense. 

- The presented approach involves spatially sampling features to be used as the "anchor" features. Similarity of the features from a second branch to the anchor features is used as the targets. This core idea is simple to follow and is novel (to the best of my knowledge). 

- The paper does a thorough job of analyzing the implications of the presented objective (Sec 3.5) and presents motivation for all design choices (see below for some questions). 

- The proposed approach is extremely efficient in training time, especially compared to related literature in self-supervised learning. 

- The experimental evaluation is thorough and demonstrates superior results on two datasets compared to methods trained on the same datasets. However, the proposed representation underperforms compared to methods trained on larger datasets. Through extensive ablations, the effect of different hyperparameters are also analyzed. 

Overall I think this paper makes an interesting contribution to self-supervised learning (in the context of features for label propagation), that is significantly more efficient compared to existing methods and demonstrates impressive results. 

## Questions/Concerns
There are a few things that were not immediately clear and I think the paper would benefit from additional discussion on these aspects.

### Anchor Sampling
The text presents the anchor sampling as merely a method to improve efficiency. Is this really true? If all the features were indeed used, wouldn't the model become untrainable (even in the presence of enough compute)? 
Because in this case the anchor affinities $v \in \mathbb{R}^{BTHW \times BTHW}$ would also have $v_{i,j} = 1$ if $i,j$ point to the same location. 

### Comparison to MAST[15]
In Sec 4.1, you explain that MAST generates features at twice the resolution of your method. Does this mean that they are solving a harder correspondences/label propagation problem since they have to perform 4x the number of correspondences? If so, this doesn't entirely justify the gain in performance on DAVIS-2017 (as discussed in Line 264-266).  

### DAVIS-2017 Results
In Table 1, do all the methods use the same propagation method? If not, does your method have an unfair advantage because of the new propagation method? 

### YouTube-VOS Results
Similarly here, since the text claims that the MAST propagation method leverages additional information (like ROIs), is it possible to present results of MAST with your propagation method (using features before the ROI Align operator). 

### Scaling up to larger datasets
CRW[13] seems to outperform your method, but uses significantly larger amount of training data. Since your proposed method is computationally efficient, is there a reason for not scaling up training to Kinetics or TrackingNet? 


limitations_and_societal_impact: Yes, the authors have discussed limitations and societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a method for unsupervised video object segmentation. The method is based on enforcing equivariance between video batch and the same but transformed batch. The paper utilizes  anchor sampling strategy in order to reduce computational requirements, and compute the affinity between features and anchors.
main_review: Positive:
- The paper is fairly well written, easy to read and to understand.
- The paper contains thoughtful ablation studies which justify the choices made.
- The proposed method is very simple, well-thought and easy to understand.


Negative:
- The paper lacks rigor comparison with the baselines. On the Davis-2017 dataset the paper is compared with CorrFlow and MAST using the same pretraining, however the label propagation mechanism is different. On VOS it is compared with CRW, but the training datased is different. So overall there is no single clear comparison with methods from the literature on the same dataset with the same propagation strategy.  Thus it is not possible to judge clearly if the method outperforms the competitors.


Some minor staff:
- Line 17: ample -> sample

- The paper did not discuss some relevant image-based segmentation methods that also utilize equivariance, for example: SCOPS: Self-Supervised Co-Part Segmentation Hung et al.

- Why is only random scaling and horizontal flipping considered as the transformation? Did the authors experiment with other kinds of transformations: rotations, sheering, color jittering?


limitations_and_societal_impact: There is no specific limitations or societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method for self-supervised pretraining on video:

Given a batch of video clips, dense per-frame features are extracted from a backbone architecture like ResNet. The objective is to apply a standard contrastive penalty where a softmax loss picks out a positive pair of feature embeddings amidst a large set of negatives. Rather than compare every feature vector to every other vector in the batch (across samples, time, and space) a subset of representative vectors are chosen for comparison. Starting from BxTxHxW features, we choose from each sample a single frame and a subsampled grid of values to get BxNxN "anchors" to compare to. The full set of BxTxHxW features will each be compared to all of the sampled BxNxN anchors.

To apply the loss, for each feature we need one of the comparison anchors to serve as a positive target. This is done by finding the closest match to features from an augmented version of the same clip. That is, another inference pass of the backbone takes in a randomly resized/cropped/flipped version of the clip. Those features are compared to the anchors to find the argmax closest match. This creates a pseudo-ground truth label used to supervise the original features. Importantly, the gradient is stopped so nothing is backpropped through the pseudo-label generation.

Pretraining is done with the proposed set up and evaluated on the downstream setting of unsupervised mask propagation on DAVIS and YT-VOS.
main_review: The good:

- I think the idea itself is interesting and sufficiently innovative over the large body of emerging contrastive learning work. The idea of using the regularizing branch to choose pseudolabels makes sense to me, we've seen other work that shows a simple strategy of stopping gradients from the augmented branch will get you far [6]. Thanks to using video + dense spatial features, all the comparisons in the contrastive loss are between features from a single branch which is different from most existing contrastive methods though offers some similarity to [3] (acknowledged by the authors).
- Overall the implementation and replication of this idea seems like it wouldn't be too much trouble, there's enough detail in the paper to pull it all together, and there's a degree to which it is simpler compared to prior video methods that rely more directly on correspondence/flow.
- The ablations inform how to approach some of the critical design decisions of the method (how many frames should be used, how much to subsample the grid)

Concerns:

- It took me quite a number of read throughs to finally understand what the method was. While it may be a reflection of my own reading comprehension, I think that the paper would benefit from a clearer presentation. Perhaps some reworking of Figure 2 might help.
- The most direct comparison is to [13] which offers similar motivation + set up + downstream evaluation, but the proposed method underperforms on both DAVIS and YT-VOS. Understandably, there are resource constraints that make it difficult to train on Kinetics to provide the apples-to-apples comparison. Still, the quantitative benchmark results in the paper could be stronger.

Overall, I lean positive on this paper. I like the idea itself, though the presentation and results are not as strong as they could be. I think that this method would likely scale up well with more resources.

Other comments:

- It's good to see the results in Table 2b, though I'm curious, given the down and up in performance from T=5->10->15 how much variance is there in the results across training runs?
- Why not include other data augmentation techniques? (L209) They are helpful signals to learn invariance to in other self-supervised work, seems like they would be of use here too. 

**Final comments:**

I appreciate the effort the authors put in their rebuttal to clarify concerns around apples-to-apples comparison to prior work. I also sympathize with the headache in trying to download and use Kinetics. Overall, I'm still a fan of the proposed idea. I think the presentation and explanation of the method could be improved, but I stand by my recommendation to accept.
limitations_and_societal_impact: Promising to not provide advice for anyone attempting to use the proposed research in a malicious way is not exactly the strongest point, but overall the authors are correct that this is pretty basic early-stage research with few specific, harmful downstream applications.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: This paper aims at unsupervised feature learning from videos. The learned representation is tested in a video segmentation/tracking setting on the DAVIS and the Youtube-VOS datasets. Although the paper describes many principles for the learning strategy, I could not find any of them to be new and also the results do not indicate that something fundamental was added. 
main_review: In 3.1. the paper introduces 4 assumptions to derive constraints for feature learning. However, all of them are well-known. The first one is spatial smoothness (which is, btw, wrong at object boundaries). The second and third assumptions look identical to me and introduce temporal smoothness. Also this has been commonly used for contrastive video representation learning. The last one is equivariance to scaling and flipping implemented by an augmentation procedure. Also this is pretty standard. Another highlighted concept is subsampling (motivated by spatial and temporal smoothness). I cannot see anything spectacularly new here either. 

In 3.6. there is a discussion stating some questionable advantages of the proposed approach: Why is it good to ignore temporal causality? Common sense tells us that features generalize better if they model a causal process. Why is it good if clusters are instance-specific and do not span across instances? Establishing links across instances should improve generalization (though admittedly this is not relevant for video tracking). Scalability is bound to subsampling. What prevents other methods from subsampling as well to improve scalability? There is a price to pay, of course: as soon as the grid gets to coarse, there is relevant information loss (see ablation in Table 2b). 

The efficiency argument is not sufficiently quantified by experiments. 

In conclusion, I cannot say what this paper contributes. It is a relatively complicated description of well-known facts and procedures. Thus, I cannot recommend accepting the paper. 
limitations_and_societal_impact: The conclusions include some superficial statements, which is sufficient for this type of paper.
review_title: null

Meta-review: The paper discusses a method for unsupervised learning of correspondences across video frames. Different from prior work like CRW [13] the proposed method offers additional flexibility (e.g., an anchor can be matched to various points). To establish the use of this additional flexibility the reviewers asked for additional baselines which the authors provided in the rebuttal. In a discussion one reviewer remained concerned about the novelty and its use (the authors admit that the improvement compared to [13] is "slight"). Moreover, the authors also state that "testing on Kinetics or TrackingNet would be useful. In particular, we recognise that training on Kinetics-400 would enable a direct comparison to CRW [13]." Up until writing of this meta-review the authors have not provided this result. AC thinks a careful comparison to very related work is desirable and shouldn't be omitted. Further, as suggested by the reviewer that remained more concerned, AC concurs that the writing and organization of this paper should be improved significantly and AC can understand why the reviewer remained concerned. AC strongly encourages the authors to improve the paper by adding additional experimental evidence and by assessing organization.
Decision: Accept (Poster)