Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes VICRegL, a method that extends the existing (global) VICReg objective with local contrastive objectives where the loss is applied at the level of feature pixels (between spatial locations in the feature map before global pooling in CNNs).
Two approaches to building pairs of feature pixels for contrastive learning are proposed: 1) based on the l2 distance in the learned embedding space, and 2) based on the known spatial location between the two generated views.
In experiments, the method is evaluated on (global) image classification tasks on ImageNet and (local) semantic segmentation tasks on Pascal VOC, Citiscapes, and ADEK20K. 
Ablations demonstrate a trade-off between global and local performance. A good balance between global and local SSL objectives can obtain competitive image classification performance while achieving SotA performance on segmentation tasks.
strengths_and_weaknesses: Strengths:
- Study of SSL pre-training that performs well on various downstream tasks (local and global) is valuable
- Good performance in transfer to both classification and segmentation
- Technical presentation is mostly clear
- A good set of ablation experiments is provided, demonstrating:
	- the trade-off between local and global feature learning
	- The two ways to construct local pairs (l2 in embedding space and spatial distance)
	- The influence of the number of local pairs to use in the loss


Weaknesses:
- The approach to building positive local pairs via spatial distance is very similar to [N1], which is neither discussed in the paper nor compared to in experiments. 
- The method is demonstrated only with VICReg as the contrastive criterion. It is unclear if the approach to incorporate local objectives with the global one would also work with other "contrastive" objectives. In principle, it should also work, and demonstrating it with more standard contrastive objectives (e.g., SimCLR) would strengthen the paper.  
- Inconsistencies with results reported in prior work: DenseCL by Wang et al. reports 69.4 mIoU on PascalVOC with fine-tuning. It is unclear how the difference from the reported 66.8 in Table 1 comes about. 
- Several related works on learning both global and local representations are summarized in Section 2, but it is unclear how the proposed method relates to and differs from these works.
- The idea of using nearest-neighbors in feature space for contrastive learning has been explored in prior works (e.g., [N2, N3]). While it is used in a different context here, these works should be discussed. 
- The paper argues for learning "hierarchical features" (L39), however, it is unclear how the proposed method should learn more hierarchical features than existing approaches based on CNN architectures.
- The number of local pairs is fixed in the method. What happens, however, if the two views don't or barely overlap?
- The paper mentions that multi-crop was used with ConvNeXt architectures. Therefore, providing an ablation with and without multi-crop in this setting would be good. Also, it would be good to investigate more the reason for multi-crop failing with ResNet-50, e.g., test smaller batch sizes as hypothesized in Section 3.2.  
- Minor: Error in L104 - It should not be C in both input and output


[N1] Xiao et al.: "Region Similarity Representation Learning", ICCV2021

[N2] Dwibedi et al.: "With a little help from my friends: Nearest-neighbor contrastive learning of visual representations."

[N3] Koohpayegani et al.: "Mean Shift for Self-Supervised Learning"
questions: I would appreciate it if the authors could address the weakness listed above, especially the first four points, i.e., 
- How does the method compare to [N1]?
- Is the VICReg necessary, or would the approach also work with other objectives?
- How are the results for prior work obtained in Table 1, and what explains the inconsistency with DenseCL? 
- How does the work relate to and differ from the various related prior works (including [N1-N3])?
limitations: Limitations were addressed well in the paper. The point about whether feature pixels learn local information is interesting and could be explored through experiments, e.g., by clustering the feature pixels of an image and visualizing these clusters.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces locality constraints building upon the existing VICReg method. It does so by not destroying the features maps and adding VICReg loss terms on the features maps. Two simple methods of correspondence generation are proposed: a) Nearest neighbour in spatial location, and b) embedding similarity of local features. The paper claims to learn visual representation that improves performances on localized tasks such as detection and segmentation while maintaining the consistently good performance for global classification task. The paper uses two backbones for evaluations and shown to perform better than existing methods especially on linear evaluations. The paper has a good contribution, but a few evaluations and analysis seem lacking. 
strengths_and_weaknesses: Strength:
1. The idea is elegant, uses the existing loss but additionally on local features. 
2. It is shown to significantly improve segmentation performance (though difference in finetuning performances not huge.) which is convincing.
3. Ablations studies show insights to design choices, visualization on matches.
4. Overall, the paper is easy to follow, and the paper has a good contribution however there are a few concerns as follows


Weakness:
1. The main concern is that although the paper claims to benefit detection and segmentation tasks, only segmentation is evaluated.  Evaluations on detection are studied in most of the baseline papers but this is lacking in the paper. It may require a more comprehensive experiment to validate the effectiveness.
2. The authors use ‘multicrop’ technique but do not ablate this factor in the experiments. Also, it is not clear if other baselines (especially VICReg) also enjoy the benefit of multicrop. If they do not then, the comparison may not be fair. The authors may want to clarify this.
3. As the method uses features maps and losses on based on them, it would be interesting to see its impact on training speed and memory.

Please read below on other queries and comments below.


questions: 1. The method chooses location representation correspondences based on nearest location and embedding similarity. Any thought or analysis on noisy or spurious correspondences? For example, although nearest neighbours the patch may be coming from different object/semantics/texture. Similarly, based on embedding similarity, the network may choose unreliable pairs especially in the early stage of training.  

2. In the ablation of selected pairs, when all possible matches are selected, I believe both spatial and embedding distance losses will be the same, and network would try to match all local features though coming from different objects. Though not optimal, it seems the performance is improved compared to the baselines. Please could you provide intuition on this scenario and related improvement. 
3.  Is the network trained from scratch with global and local losses or, a pretraining with global loss is done?
4. How does the method compared to [1]

[1] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

The authors have given satisfactory answers to the rebuttal. I would like to increase my rating to borderline accept given the authors promise to include all discussions and a complete result for detection on PASCAL and MS-COCO with standard evaluation protocol.
limitations: 1.  [Comment/Suggestion] It is natural to fine-tune the network when supervisions are available, however, the gap in the performance boosts narrows after finetuning. May be a semi-supervised task and evaluation would highlight the importance of the method and a more meaningful scenario. 
2. It would be interesting to see any sort of analytical study to support the hypothesis of relation of batch size and dimension of the network’s output. This would be an important contribution on itself.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper describes a learning method to learn jointly global and local descriptors in a self-supervised way.

The global descriptor are learned following VICREG [1], a learning method that tackles the issue of mode collapse during representation learning. Two versions of the same images are derived using random transformations (crop, color jittering) and fed to a siamese network. Each branch produces a 1D feature vector representing the image. The branches are optimized separately with the VICREG constraints: the feature vector must be close in the embedding space (invariance loss), the standard deviation of the embedding dimensions must be close to one (variance loss), and the dimensions of the embeddings should not be correlated (covariance loss).

The local descriptor learning is the main contribution of the paper. The feature maps provide local descriptors (a feature map of dimension H x W x C can be seen as a set of HW local features of dimension C). These descriptors are optimized so that: 1) descriptors from the same location in the image are pushed together (spatial matching loss) 2) descriptors close in the embedding space are also pushed close together (even when they do not relate to the same image location i.e. they are not ground truth match).

The several components of the final loss are balanced with metaparameters investigated in the paper. The main finding are: 1) that the bigger weight given to the global loss, the worse local features are for object detection and semantic segmentation tasks; 2) the global criterion is relevant to improve the local features suggesting that they benefit from a global understanding of the image.

The paper evaluates the global descriptors on classification and local descriptors on object detection and semantic segmentation. It compares against relevant instances of the state of the art and experiments show that the introduced optimization is indeed relevant to learn local features in a self-supervised way.

[1] Vicreg: Variance-invariance-covariance regularization for self-supervised learning
strengths_and_weaknesses: Strengths:
- The problem tackled is relevant to the community: the joint learning of global and local features
- The originality of the l2 distance loss: it pushes feature vector that are close in the embedding space even though they are not positive matches.
- Relevant state-of-the art
- Relevant experiments (comparisons, analysis and ablation)
- Very well-written paper: the organisation and presentation of the ideas is easy to follow, and the description and analysis of the experimental results are clear and valuable to the reader.

Weaknesses:
- The main contribution seems incremental with respect to VICREG.
questions: L49: "contrasting feature vectors corresponding to far away locations can have a negative effect, as these vectors could have been pooled from locations that represent the same object in the image."
This is not clear. Does contrasting here mean pushing the feature vector together? In this case, wouldn't one want to push features of the same object together? i.e. this would be a positive outcome?

The l2-distance (Eq 3) is not intuitive at first. It pushes together local features close in the embedding space even though they might not be ground-truth matches. Is there any intuition on why this is beneficial (as supported by the experiments)?
One intuition is that it might act as regularizer or an implicit clustering.
Another is that the global criterion already pushes positive matches coarsely together and this loss refines the vector. Is there any observations or intuition on that interesting phenomenon?
limitations: - L107: It would be nice to introduce the notation of $\lambda, \mu, \nu$ in Eq 1 for completeness.

- L118: $h_{\phi}^l$ is called a 'local projector' whereas it is the equivalent of the VICREG expander but at the scale of the feature map. Since 120 uses the terminology 'global expander', it would seem more consistent to use the word 'local expander' rather than projector. This would also be more consistent with VICREG on which the paper relies on.

- L127, L135, 181: 'only the top-$\gamma$ pairs are kept'.  It is not clear whether for each feature position, only the top-$\gamma$ are kept to derive the loss, or if the top-$\gamma$ among all pairs in the batch are kept (assuming top-$\gamma$ means the pairs with the smallest distance). This would help the reader if the definition were introduced L127.

- L127, L135, 181: Is there a motivation / intuition on why it is better to keep only the top-$\gamma$ pairs?
Together with the l2-distance loss, this could reinforce the intuition that the l2-distance loss act as a regularizer/booster as it would be applied only to feature vectors that are already close to each other, which one can assume to be ground-truth positives.

- L306: a feature maps -> a feature map
review_title: null

Meta-review: metareview: This paper proposes to extend the existing VICReg objective to the local features for obtaining good performances on both image-level and dense prediction tasks. In specific, while the global features are obtained by an average pooling on the output feature maps, the local pairs are determined by both of the feature distance and spatial location distance. The technical novelty seems to be somewhat incremental due to a little bit simple modification of the existing global objective to the local objective for dense representation learning.  However, extensive experiments on several benchmarks including ablations and visualization clearly demonstrate the effectiveness of the proposed self-supervised representation learning for both classification and segmentation (+detection) tasks. Especially, the authors faithfully addressed most concerns and questions raised by the reviewers, and the overall quality of the paper seems to be significantly improved. Therefore, I would recommend to accept this paper.
confidence: Less certain
Decision: Accept