VICRegL: Self-Supervised Learning of Local Visual Features
Authors: Adrien Bardes
Jean Ponce
Yann LeCun
Venue: NeurIPS 2022 Accept (2022)
Keywords: self-supervised learning
representation learning
computer vision

Abstract: Most recent self-supervised methods for learning image representations focus on either producing a global feature with invariance properties, or producing a set of local features. The former works best for classification tasks while the latter is best for detection and segmentation tasks. This paper explores the fundamental trade-off between learning local and global features. A new method called VICRegL is proposed that learns good global and local features simultaneously, yielding excellent performance on detection and segmentation tasks while maintaining good performance on classification tasks. Concretely, two identical branches of a standard convolutional net architecture are fed two differently distorted versions of the same image. The VICReg criterion is applied to pairs of global feature vectors. Simultaneously, the VICReg criterion is applied to pairs of local feature vectors occurring before the last pooling layer. Two local feature vectors are attracted to each other if their l2-distance is below a threshold or if their relative locations are consistent with a known geometric transformation between the two input images. We demonstrate strong performance on linear classification and segmentation transfer tasks. Code and pretrained models are publicly available at: https://github.com/facebookresearch/VICRegL

VICRegL: Self-Supervised Learning of Local Visual Features
Adrien Bardes 1.2
Jean Ponce 2.4
Yann LeCun
1,,4 13,4
1 MMeta, FAIR
Inria, École normale supérieure, CNRS, PSL Research University
3 SCourant Institute, New York University
*Center for Data Science, New York University
Abstract
Most recent self-supervised methods for learning image representations focus on either producing a global feature with invariance properties, or producing a set local features. The former works best for classification tasks while the latter is best for detection and segmentation tasks. This paper explores the fundamental trade- off between learning local and global features. A new method called VICRegL is proposed that learns good global and local features simultaneously, yielding excellent performance detection and segmentation tasks while maintaining good performance on classification tasks. Concretely, two identical branches of a standard convolutional net architecture are fed two differently distorted versions of the same image. The VICReg criterion is applied to pairs of global feature vectors. Simultaneously, the VICReg criterion is applied to pairs of local feature vectors occurring before the last pooling layer. Two local feature vectors are attracted each other if their (2-distance is below threshold or if their relative locations are consistent with known geometric transformation between the two input images. We demonstrate strong performance on linear classification and segmentation transfer tasks. Code and pretrained models are publicly available at: https: //gi thub. com/ facebookresearceeecooooooo CVICLL
Introduction
Recent advances in sulf-supervised learning for computer vision have been largely driven by down- stream proxy tasks such as image categorization, with convolutional backbones [Chen et al., 2020a,b, Grill et al., 2020, Lee et al., 2021, Caron et al., 2020, Zbontar et al., 2021, Bardes et al., 2022, Tomasev et al., 2022], or vision transformers [Caron et al., 2021, Chen et al., 2021, Li et al., 2022, Zhou al., 2022a]. Current approaches rely on a joint embedding architecture and a loss function that forces the learned features to be invariant to a sampling process selecting pairs of different views of the same image, obtained by transformation such as cropping, rescaling, or color jittering [Misra and Maaten, 2020, Chen et al., 2020a, He et al., 2020, Grill et al., 2020]. These methods learn to eliminate the irrelevant part of position and color information, in order to satisfy the invariance criterion, and perform well on image classification benchmarks. Some recent approaches go beyond learning global features: to tackle tasks such as semantic segmentation where spatial information plays key role, [Yang et al., 2021, Xie et al., 2021, Hénaff et al., 2021, Yang et al., 2022, Hénaff et al., 2022, El-Nouby et al., 2022] also learn image models with more emphasis on local image structure. In the end most recent approaches to self-supervised learning of visual features learn the corresponding image model using either a (possibly quite sophisticated) global criterion, or a (necessarily) different exploiting local image characteristics and spatial information. The best performing local methods require non- non-paramttric pre- pre-processing step that compute unsupervised
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

segmentation masks [Hénaff et al., 2021], which can be done online [Hénaff et al., 2022], but with an additional computational burden.
We argue that more complex reasoning systems should be structured in a hierarchical way, by learning at several scales. To this end, we propose VICRegI., method that learn features at a global scale, and that additionally uses spatial information, and matches feature vectors that are either pooled from close-by regions in the original input image, or close in the embedding space, therefore learning features at a local scale. In practice, the global VICReg criterion [Bardes et al., 2022] is applied to pairs of feature vectors, before and after the final pooling layer of convolutional network, thus learning local and global features at the same time, When a segmentation mask is available as in [Hénaff al.. 2021], feature vectors that correspond to the same region in the mask can be pooled together, and compared using a contrastive loss function, which allows spatial vectors far away in the original image to be pooled together if they belong to the same object. In our case, segmentation masks are not available, and we therefore face two challenges: (1) there is no a priori information on how to pool vectors from the same object together, thus long-range matching should be learned in self-supervised manner, and (2) contrasting negatively feature vectors corresponding to far away locations can have a negative effect, as these vectors could have been pooled from locations that represent the same object in the image. In order to address these issues, VICRegL (1) matches feature vectors according to a 12 nearest-neighte and image locations, properly weighted, and (2) uses the VICReg criterion between matched feature vectors. We use VICReg for its simplicity and its non-conme con-conraatevee nature, which alleviates the need for negative samples and therefore does not have an explicit negative contrasting effect between feature vectors that could have been potential matches.
criterion exploiting both the distances between features
We demonstrate the effectiveness of VICRegL. by evaluating the learned representations on vision tasks such as image classification on ImageNet and semantic segmentation on various datasets. Our evaluation is (mostly) done in the setting where the backbone learned by VICRegL is frozen, with only a linear classification or segmentation head tuned to the task at hand. We believe that this setting is a much better evaluation metric than the commonly used fine-tuning benchmarks, as the performance can not be attributed to the use of a complex head, or to the availability of the ground truth masks. Our results show that learning local features, in addition to global features, does not hurt the classification performance, but significantly improves segmentation accuracy. On the Pascal VOC linear frozen semantic segmentation task, VICRegl. achieves 55.9 mIoU with a ResNet-50 backbone, which is a +8.1 mIoU improvement over VICReg, and 67.5 mloU with a ConvNeXt-S backbone, which is +6.6 mloU improvement.
2 Related work
Global features. Most recent methods for global feature learning are based on a joint embedding architecture that learns representations that are invariant to various views. These methods differ in the way collapsing solutions are avoided. Contrastive methods [Hjelm et al., 2019, Chen et al., 2020a, He et al., 2020, Chen et al., 2020b, Mitrovic et al., 2021, Dwibedi et al., 2021, Chen et al., 2021, Tomasev et al.. 2022] uses negative samples to push dissimilar samples apart from each other. Clustering methods [Caron al., 2018, 2020, 2021] ensure balanced partition of the samples within a set of clusters. Non- con-contrastive methods, which are dual to contrastive ones [Garrido et al., 2022], rely on maintaining the informational content of the representations by either explicit regularization [Zbontar et al., 2021, Bardes et al., 2022] or architectural design [Chen and He, 2020, Grill et al., 2020, Richemond et al., 2020, Lee et al., 2021J. Finally, the best performing methods today are based on vision transformers [Caron et al., 2021, Chen et al., 2021, Li et al., 2022, Zhou et al., 2022a,b) and deliver strong results in both downstream classification and segmentation tasks.
Local features. In opposition to global methods, local one focus on explicitly learning a set of local features that describe small parts of the image, which global methods do implicitly [Chen al., 2022], and are therefore better suited for segmentation tasks. Indeed these methods commonly only evaluate on segmentation benchmarks. A contrastive loss function can be applied directly: (1) the pixel level [Xie et al., 2021], which forces consistency between pixels at similar locations; (2) at the feature map level [Wang et al., 2021], which forces consistency between groups of pixels: (3) at the image region level [Xiao et al., 2021], which forces consistency between large regions that overlap in different views of an image. Similar to [Wang et al., 2021], our method VICRegI. operates at the feature map level but with a more advanced matching criterion that takes into account the distance in

---

pixel space between the objects. Copy pasting a patch on random background [Yang et al., 2021, Wang al., 2022] has also shown be effective for learning to localize an object without relying on spurious correlations with the background. Aggregating multiple images corresponding to several object instances into a single image can also help the localization task [Yang et al., 2022]. These approaches rely on carefully and handcrafted constructions of new images with modified background or with aggregation of semantic content from several other images, which is not satisfactory, while our method simply rely on the classical augmentations commonly used in self-supervised learning. The best current approaches consist in using the information from unsupervised segmentation masks, which can be computed
pre-processing step [Hénafff et al., 2021] computed online [Hénafff et al., 2022]. The feature vectors coming from the same region in the mask are pooled together and the resulting vectors are contrasted between each other with contrastive loss function. These approaches explicitly construct semantic segmentation masks using k-means for every input image, which is computationally not efficient, and is strong inductive bias in the architecture. Our method does not rely on these masks and therefore learns less specialized features.
3 Method
Background. VICReg was introduced as a self-supervised method for learning image representations that avoid the collapse problem by design. Its loss function is composed of three terms: variance term, that preserves the variance of the embeddings, and consists in a hinge loss function on the standard deviation, on each component of the vectors individually and along the batch dimension; an invariance term, which is simply an 12 distance between the embeddings from the two branches of a siamese architecture; and finally covariance term, that decorrelates the different dimensions of the embeddings, by bringing to 0 the off-diagonal coefficients of the empirical covariance matrix of the embeddings.
For completeness, we describe how the VICReg framework works [Bardes et al., 2022]. A seed image is first sampled in the unlabelled training dataset. Two views and x' are obtained by a rectangular crop at random locations in I, rescaling them to a fixed size (R, and applying various color jitters with random parameters. The views are fed to encoder fo RC Rx5 R producing their representations
fo(x) and y' = Sa(x') E C which are mapped by an expander ho RC RDD onto the embedddnggs z haly) and z ho(v') RR. The VICReg loss function is defined as follows:
((2,2') = A8(2,2²) + plu(z) + v(2') + vlc(2) + c(2'),,
(1)
where s, v and e are the invariance, variance and covariance terms as described in [Bardes et al., 2022], and X, y and are scalar coefficients weighting the terms.
3.1 VICRegL: feature vectors matching
When the encoder So is a convolutional neural network, the final representations are obtained by performing an average pooling operation @ RCCHH R.C the output feature maps, with C' the number of channels and (H, W) the spatial dimensions. We now denote the pooled representations Yep and v/g E RC and the unpooled representations y and y' E CCCHxW We denote Vi.j and vl.5 E RC the feature vectors at position (i,j) in their corresponding feature maps. The main idea to apply the VICReg criterion between pairs of feature vectors from y and //, by matching an element of y to one from y/. using spatial and [2-distance based information. We introduce local projector network in. IR. x + R. Dx HxW that embed the feature maps y and y/ E TRCx xgC'Hxx H
onto feature maps embeddings = = 11,(v) and z' = 1a(('') Dx The standard expander of VICReg is now the global expander hgg RC RD which maps the pooled representations ye and V/g E C to the embeddings =e = n& (2a) and 2/e = A (2/g) E RDD. We describe now how we perform the matching, and introduce our loss functions.
Location-based matching. In order to take into account the transformation that occurs between the views x and x' of an image and thus matching features from similar locations, we compute the absolute position in / that corresponds to the coordinate of each feature vector in its feature map. Each feature vector zp at position p in the feature map is matched to spatial nearest neighbor according the the absolute position in I, and among the H x W resulting pairs, only the top-y pairs
3

---

MODEL
Enboeding: D
LocalFasmaneeCC C: I
Convolational Encnder
Prilact lovel Staahei
Tec
Image
loal
Dupiicae
Convolational
Enceder
Posling
Vie
GLOBAL CRITERION
Figure 1: Overview of VICRegL: Learning local and global features with VICReg. Given a seed image, views are produced and fed to encoder that produces local features. The features are further processed by a local projector that embed them into a smaller space, without destroying the localization information. Two matchings, one based the spatial information provided by the transformation between the views, the other one based on the -i-dittance in the embedding space are computed, and the VICReg criterion is then applied between matched spatial embeddings. Additionally, the local features from the encoder are pooled together, and the pooled features are fed global expander. The VICReg criterion finally applied between the two resulting embeddings.
are kept. The location-based matching loss function is defined as follows:
2=(z,2') Σ Uzp: (Nn())
where the sum is over coordinates p P {(h, w) (h, w) E [1...., H) x [I..., W] the set of all coordinates in the feature map, and NN(p) denotes the (spatially) closest coordinate p' to according to the actual distance the seed image.
Feature-based matching, In addition to matching features that are close in terms of location in the original image, we match features that close in the embedding space. Each feature vector Ep position p is matched to its nearest neighbor in 2' according the (2 distance in the embedding space, and among the H x W resulting pairs, only the top-y pairs are kept. The feature-based matching loss function is defined follows:
La(z,2)) = Elzpp NN' NN,(2,)),
(3)
PEP
where the sum is coordinates in P and (z,) denotes the closest feature vector to 2p in the feature maps z', in terms of the -i-ditance. Similar to the location- based loss function, the feature- based loss function enforces invariance on a local scale, but between vectors that are close in the embedding space, and not necessarily pooled from the same location in the seed image. The purpose of this loss function is mainly to capture long-range interactions not captured by the location-based matching.
The general idea of top-y filtering climinate the mismatched pairs feature vectors that are too far away in the image for the location-based matching, and that therefore probably do not represent the same objects, but most importantly that are probably mismatched for the feature-based matching, especially at the beginning of the training when the network matches feature vectors representing different objects or textures. Sometime, two views don't 't or barely overlap, for the feature-bart based matching this is not an issue, as the purpose of this matching is to capture long-range interactions not captured by -rastio--baeed matching. For the location- based matching, given the parameters we use to generate the views (each view covers between 8% and 100% of the image, chosen uniformly). the probability for the views to not overlap is small, and even in that case matching the closest points between the views does not degrade the final performance. Indeed, we have tried to use variable

---

number of matches and threshold value used to compute the matches. which did not improve the performance compared using fixed number of matches. In practice. there are with high probability always good local features to match as the views have a low probability of not overlapping, and this explains why always matching the top-y pairs, compared introducing a threshold value which the pair is considered match, does not degrade the performance.
Our final loss function combination of the location-based and feature-based loss functions, which form the local criterion, with in addition a standard VICReg loss function applied on the pooled representations, which is the global criterion. Both location and reature-based loss functions are symmetrized, because for both, the search for the best match is not symmetric operation. Our final loss function is described as follows:
(4) where is an hyper- papee--ar controlling the importance one wants to put on learning global rather than local features. We study later in Section 4.2 the influence of a on the downstream performance, and show that there exists a trade-off such that the learned representations contain local and global information at the same time, and therefore transfer well on both image classification and segmentation tasks.
((2, 2') = al 22 240 (1 (C,((() + C.(e(,)) + La(z,2²) + 2))
3.2 VICRegL with the ConvNeXt backhone
The feature matching procedure is designed to work with any kind of convolutional neural network. In the experimental section, we provide results with a standard ResNet-50 backbone. However, one can considerably improve the performance on downstream tasks by using more sophisticated backbone. We propose to use the recently introduced ConvNeXt architecture [Liu et al., 2022], that is very similar the ResNet one, but with many simple modifications to the original architecture, which make it work as well as modern vision transformers. To the best of our knowledge, this is the first time ConvNeXts have been used in self-super learning, and our work shows that convolutional neural networks are still able to deliver state-of- ofte--of-thh-ar performances in most standard self-supervised learning benchmarks. In order to make the performance competitive with recent approaches. we use the multi-crop strategy introduced in [Caron et al., 2020]. Surprisingly, we found that the combination of multi-crop with encoders from the ResNet family and the VICReg criterion is extremely difficult to optimize, and haven't 't been able to make it work properly. However, multi-crop shows very good results when combined with CenvNeXts. Our intuitive explanation is based on study of the optimal batch size. The VICReg criterion regularizes the empirical covariance matrix of the embeddings, computed using the current batch, and we hypothesize that there is a link between the size of the batch, and the dimensionality of the representations. VICReg combined with ResNet-50 has shown to have an optimal batch size of 2048, which is exactly the dimensionality of the representations of a ResNet-50. We found that the optimal batch size when working with ConvNeXts is 512 which is much smaller, and correlates with the fact that Cen also have smaller representations (768 for ConvNeXt-S and 1024 for Con-NeXt-B). The optimal batch size might therefore be close to the dimensionality of the representations. Now, the multi-crop strategy artificially increases the size of the batch, which is much easier handle when the size of the batch before multi-crop is small. Indeed the effective batch size otherwise becomes too large, which causes optimization issues.
We now describe how the matching loss functions are adapted in order to work with multi-crop. Instead of generating only two views of the seed image, views (2 large, and N small), resized to two different resolutions are generated, and further encoded into the feature maps embeddings 21 and z² for large views, and {x") "1 defined as follows:
In=3 for small views. The spatial matching loss function is then
N
C
Σ 1== v.
UEp 1 (NN))) +
((2p El(pp m "N
(5)
PEP
PEP
where only the top-7/1 and top-72 pairs of feature vectors of large and small views respectively are kept in the computation of the loss. The feature-based loss function, and the global criterion are adapted in a very similar way, one large view is matched to the other large views and to the small views, and the final loss function is defined as follows:
[[(2") a(((=)))))) + (1 (H22 ( + La([z"]N (N)))))))))
(6)
5

---

Table 1: Comparison of various global and local self-supervised learning methods on different linear evaluation benchmarks. Evaluation the features learned from a ResNet-50 backbone trained with different methods (1) linear classification accuracy (%) (frozen) the validation set of ImageNet (2) Linear segmentation (mloU) (frozen and fine-tumin fine-tuning) on Pascal VOC. (3) Linear segmentation (mloU) (frozen) on Cityscapes. a is the weight of Eq. (4) balancing the importance given to the global criterion, compared to the local criterion. The best result for each benchmark is bold font. VICRegL. consistently improves the linear segmentation mloU over the VICReg baseline, which shows that introducing a local criterion is beneficial for localized understanding of the image.
Linear Cls. (%)
Linear Seg. (mloU)
ImageNet Frozen
Pascal VOC
Cityscapes Frozen
Method
Epochs
Frozen
TuneeTuned
Global feattre features
MoCo v2 [Chen et al., 2020b]
200
67.5
35,6
64.8
14.3
SimCLR [Chen et al., 2020a]
400
68.2
45.9
65.4
17.9
BYOL [Grill
[Grll et al., 2020]
300
72.3
47.1
65.7
22.6
VICReg [Bardes al., 2022]
300
71.5
47.8
65.5
23.5
Local features
PixPro [Xie et al., 2021]
400
60.6
52.8
67.5
22.6
DenseCL [Wang ct al., 2021]
200
65.0
45.3
66.8
11.2
DetCon [Hénaff et al., 2021]
1000
66.3
53,6
67.4
16.2
InsLoc [Yang al., 2022]
400
45.0
24.1
64.4
7.0
CP² [Wang et al., 2022]
820
53.1
21.7
65.2
8.4
ReSim [Xiao et al., 2021]
400
59.5
51.9
67.3
12.3
Ours
VICRegL 0.9
300
71.2
54.0
66.6
25.1
VICRegL 0.75
300
70.4
55.9
67.6
25.2
3.3
Implementation details
We provide here the implementation details necessary to reproduce the results obtained with our best ResNet-50 and ConvNeXts models. All the models are pretrained on the 1000-class unlabelled ImageNet dataset. Most papameprrreeters are kept unchanged compared to the implementation provided by [Bardes et al., 2022], the VICReg loss variance, invariance and covariance coefficients set to 25, 25 and 1. The global expander is 3-layers cully-coneected network with dimensions (2048-8192) 013-8192-81122 dimensions (1-8--12-512-512) With the ResNet-50 backbone,
The local projector is much smaller, due to memory limitations, and has
train models on 32 Nvidia Tesla V100-32Gb GPUs, with the LARS optimizer [You et al., 2017, Goyal et al., 2017], a weight decay of 10-6, batch size of 2048 and learning rate of 0.1. The learning rate follows cosine decay schedule [Loshchilov and Hutter, 2017], starting from with 10 warmup epochs and with final value of 0.002. The number of selected best matches 7 of Eq. (2) and (3) is set 20. With ConvNeXts backbones, we noticed that much smaller batch sizes actually improve the performance, therefore train ConvNeXt-5 models 8 Nvidia Tesla V100-32GB GPUs, with the AdamW optimizer [Loshchilov and Hutter, 2019], weight decay of 10-6 a batch size of 384 and learning rate 0.001, and our ConvNeXt-B models 16 Nvidia Tesla V100-32Gb GPUs with a batch size of 572 and the same other hyper-parameters. The learning rate follows a cosine decay schedule, starting from with 10 warmup epochs and with final value of 0.00001. The number of selected best matches and f22 of Eq. (5) are set to 20 for feature maps from large views and 4 for feature maps from small views.
4 Experimental Results
In this section, we evaluate the representations obtained after pretraining VICRegL with a ResNet-50, and NeX backbones [Liu al., 2022] various size, on linear classification on ImageNet- 1k [Deng et al., 2009], and linear semantic segmentation Cityscapes [Cordts al., 2016] and ADE20k [7hou et al., 2019]. We demonstrate that VICRegI. strongly improves segmentation results over VICReg while preserving the classification perfor-
Pacal VOC [Everingham et al., 2010],

---

Table 2: Comparison of various global and local self-supervised learning methods on differ- ent linear evaluation benchmarks. Evaluation of the features learned from ConvNeXt and ViT backbones trained with different methods on: (1) linear classification accuracy (%) (frozen) on the validation set of ImageNet (2) Linear segmentation (mloU) (frozen and fine-tuning) on Pascal VOC, (3) Linear segmentation (mloU) (frozen) on ADE20k. a is the weight of Eq. (4) balancing the importance given to the global criterion, compared to the local criterion. The best result for each benchmark bold font. denotes pretraining on ImageNet-22k.
Linear Cls. (%)
Linear Seg. (mloU)
ImageNet Frozen
Pascal VOC
ADE20k
Method
Backbone Params Epochs
Frozen
FT
Frozen
Global feanures
MoCo v3 [Chen et al., 2021]
ViT-$
21M
300
73.2
57.1
75.9
23.7
DINO [Caron et al., 2021]
VIT-S
21M
400
77.0
65.2
79.5
30.5
IBOT [Zhou et al., 2022a]
VIT-S
21M
400
77.9
68.2
79.9
33.2
VICReg [Bardes al., 2022]
CNX-S
50M
400
76.2
60.1
77.8
28.6
MoCo v3
VIT-B
85M
300
76.7
64.8
78.9
28.7
DINO
VIT-B
85M
400
78.2
70.1
82.0
34.5
IBOT [Zhou et al., 2022a]
VIT-B
85M
400
79.5
73.0
82.4
38.3
MAE [He al., 2022]
VIT-B
85M
400
68.0
59.6
82.4
27.0
VICReg
CNX-B
85M
400
77.6
67.2
81.1
32.7
Local features
CP² [Wang et al., 2022]
VIT-S
21M
320
62.8
63.5
79.6
25.3
Ours
VICRegL a = 0.9
CNX-S
50M
400
75.9
66.7
80.0
30.8
VICRegL = 0.75
CNX-S
50M
400
74.6
67.5
80.6
31.2
VICRegL =0.9
CNX-B
85M
400
77.1
69.3
81.2
33.5
VICRegL a =0.75
CNX-B
85M
76.3
82.5
35.3
VICRegL = -0.75"
CNX-XL
350M
150
79.4
78.7
84.1
43.2
mance, and is competitive with other local and global self- coff-supervised learning methods. We choose the linear evaluation with frozen weights our main evaluation metrics, we believe it is much better way of evaluating the learned representations. Indeed, the performance can be attributed to the use of complex segmentation head, or to the availability of the ground truth masks, and contrary to the frozen setting, the fine-tuning setting measures whether the relevant information is present in the representation, but does not measure if the information is easily extractable from it. We perform the linear evaluation using the protocol introduced by [Zhou et al., 2022al, where the learned feature maps fed linear classifier that outputs vector with the same size the number of target classes in the dataset, and then upsampled to the resolution the image to produce the predicted mask. The results are averaged over runs with randomly initialized parameters and found that the difference in performance between worse and best runs is always lower than 0.2%.
4.1 Comparison with prior work
ResNet-50 backbone. Table 1 presents our results against several other global and local self- supervised learning methods, all pretrained with a ResNet-50 backbone [He et al., 2016]. The main observation make the improvement of VICRegL over VICReg linear segmentation. On Pascal VOC, when the weights of the backhone are frozen, VICRegL a = 0.9 improves by +6,2 mloU while only loosing 0.3% classification accuracy, and VICRegL On fine-tuning the improvement is less significative, which we attribute to the non-informative nature of fine-tuning benchmarks. Indeed, some methods like InsLoc and CP2 that seem competitive on fine-tuning significantly underperform in the frozen regime, which shows that the actual performance of these methods can be attributed to the fact that the weights of the backbone benefit form the availability of the labels during the fine-tuning phase. On Cityscapes, which is much harder, most methods do not perform very well in the linear frozen regime, which sets new challenge supervised learning of local features. VICRegI. outperforms the VICReg baseline by +1.7 mloU, as well as every other local features methods by a significant margin. The second observation we make is the robustness of VICRegL in classification, which indicates that it learns both local and
0.75 improves by +8.1 mloU,
self-

---

74
89
64-
50
1.0 0.25
0.0
Figure 2: Study of the trade-off between local and global criteria. Evaluation on linear classifi- cation on ImageNet and on linear Segmentation on Pascal VOC of VICRegL pretrained with vari- ous a coefficients of Eq. (4), controlling the im- portance of the global criterion against the local criterion.
Figure 3: Selected matches: visualization of the locations of the best local matches selected by VICRegL. Left image is the seed image, with in red and blue the crop locations for the two views. Left column are the feature- based matches. Right column are the location-bases matches. Only 10 matches visualized for better clarity, but the actual number of selected matches is 20. We dis- play the matches according to the location of the feature vectors in the feature maps. Note that the receptive field of these feature vectors is much larger than only the patch represented by one square of the grid in the figure. Best viewed in color with zoom.
Table 3: Ablation: matching criterion. Compar- ison between using the feature-based matching loss (La), the location-based matching loss (C.), none of the two (Baseline), or both at the same time.
Method (%)
Seg. (mloU)
Baseline
73.8
56.0
C. 73.5
58.9
La 73.6
57.7
C.-La
73.6
60.3
global features at the same time. The performance of most local methods is greatly impacted on classification where they all perform around 10 to 20% below global methods. Global methods on the contrary are efficient for classification but underperform in segmentation compared to VICRegL.
ConvNeXt backbone. Table 2 presents our results when pretraining with ConvNeXts backbones against several other global and local self-supervised learning methods pretrained with vision trans- formers [Dosovitskiy al., 2021]. 1]. Similar to experiments with ResNet-50 backbone, the main observation we make is the improvement on segmentation tasks provided by the introduction of the local criterion. With ConyNeXt-S backbone, in the linear frozen regime, VICRegI a 0.9 improves over VICReg by +6.6 mloU on the Pascal VOC, and by +2.2 mloU on the ADE20K, while preserving most of the classification accuracy. VICRegL and +3.6 over VICReg on these two benchmarks respectively. With ConvNeXt-B backbone, the performance improvement remain consistent over VICReg, and VICRegL with other strong methods such as DINO and IBOT. The improvement also remain consistent in linear fine-tuning where VICRegL also achieves strong performance. Finally, we report the performance of a much larger ConvNeXt-XL Xt backbone, pretrained in ImageNet-22k,, which is significantly improved on segmentation tasks and set new state-of-the art in linear segmentation. Our results highlight the trade-off between classification and segmentation performance, which can be controlled by the weight given to the local criterion.
0.75 further improves by +7.4 mloU
0.75 is competitive
4.2 Ablations
For all the experiments done in this section, unless specified otherwise, we pretrain ConvNeXt-S on ImageNet over 100 epochs, with the hyper-parameters described in Section 3.3, and report both the linear classification accuracy ImageNet, and the linear frozen segmentation mloU on Pascal VOC.

---

Table 4: Ablation: SSL criterion. Introducing our local criterion with VICReg gives a stronger improvement compared to SimCLR, which is con- trastive. (ResNet-50, 300 epochs)
Table 5: Ablation: impact of multi-crop. In- troducing our local criterion yields an improved performance with or without the usage of multi- crop.
Method
Cls. (9)
Seg. (mloU)
Method
Multi crup
Cls. (8)
Seg. (mloU)
VICRog
71.1
47.8
VICReg
VICRegL
70.4
55.9
VICRegL.
69.9
57.8
SimCLR
67.5
45,9
VICReg
73.9
54.4
SimCLR-L
66.6
51.3
VICRegL
73.6
60.3
Table 6: Ablation:
number of selected matches.
Table 7: Ablation: VICReg local criterion. The collapse problem is automatically prevented with the global criterion. We study here how regulariz- ing the local feature vectors influence the perfor- mance. V: variance criterion is used, I: invariance criterion is used, C: covariance criterion is used.
The large feature maps are of size 7 x 7 and the
small ones are of size 3 x 3. There are therefore
a total number of 49 large and 9 small feature vectors and as many possible matches, and only
the top-71 large and top-72 small are kept.
71
Ta
Cls. (%)
Seg. (mloU)
Criterion Cls. (%)
Seg. (mloU)
10
73.4
73.4
20
4
73.6
60.3
VI
73.3
58.2
49
9
73.5
59.6
VIC
73.6
60.3
Trade-off between the local and global criterion. The parameter a of Eq. (4) controls the importance that is given to the global criterion, compared to the local criterion. Figure shows that there exists a fundamental trade-off between the ability of model to learn global visual features, as opposed to learning local features. In the case a 1.0, which is simply VICReg, the model very efficient at producing global representations of the image, demonstrated by the performance of 73.9% in classification accuracy. When a < which introduces the local criterion, the performance in segmentation is greatly increased, by +3.4 mloU when a 0.9, +4.3 mloU when 0.75 and +4.6 mloU when the local and global criteria are weighted equally. At the same time, the classification accuracy only drops by respectively 0.2%, 1.2% and 2.9%. This highlights the existence of sweet spot, where the model is strongly performing at both classification and segmentation, which indicates that it has learned both meaningful local and global features. When decreases too much, the model starts to lose its performance in both tasks, which shows that having a global understanding of the image is necessary, even for localized tasks.
Study of the importance between feature- and location- local criteria. VICRegL matches feature vectors according to location-based criterion L, of Eq. (2) and a feature- criterion La of Eq. (3). Table study the importance of these criterion. Baseline in the table means that no local criterion is used, and is simply VICReg. The location-based criterion gives the best improvement by +2.9 mloU over the baseline, compared to only +1.7 mIoU for the ceature-based criterion, but it is the combination of the two that significantly improves over the bascline by +4.3 mloU, which shows that using both the learned distance in the embedding space in combination with the actual distance in the pixel space produces local features with the best quality. In all cases, the classification accuracy is not affected, which is expected as the local criterion has little effect on the quality of the global features, and therefore the downstream classification accuracy.
Study of the number of matches. We study here the influence of changing the number of selected best matches 7 and 7/2 of Eq. (5), to keep for the computation of the local losses. For our experiments with multi-crop. the size of the feature maps is (2048 x 7 x 7) for the large crops and (2048 x x 3) for the small crops. There are therefore in one branch of the siamese architecture 49 feature maps for large crops, and 9 for small crops. Tables 6 shows that there is a trade-off between keeping all the matches (71 = 49 and 722 = 9). and keeping a small number of matches (71 10 and 72 2). and that the best segmentation performance is obtained with an in- in-between number of matches, (?1 = 20 and 7/2 4), which improves by +0.7 mloU compared to keeping all the matches. Similar to the study on the influence of the local losses, the classification accuracy is not affected, as the local criterion does not improve degrade the global features.
Study of VICReg components for the local criterion. The global criterion sufficient for the vectors to not collapse to trivial solutions. We therefore investigate if introducing the variance (V) and

---

covariance (C) criterion, in addition to the invariance (I) criterion, applied by the local loss functions on the feature vectors, is useful or not. Table shows that these regularization criteria are actually helping the performance, introducing the variance criterion improves on segmentation by +0.8 mloU, and additionally adding the covariance criterion further improves the performance by +1.3 mloU over the baseline. Similar to other ablations on the local loss functions, the classification accuracy is not significantly impacted.
Study of a different collapse prevention method. Our collapse-prevention mechanism is the variance and covariance regularization of VICReg, which is a non-contrastive criterion that therefore does not contrast negatively on potential positive matchings. We study however the incorporation of our local criterion with contrastive criterion, SimCLR [Chen et al., 2020a]. We simply replace VICReg by SimCLR in Eq. (2) and Eq. (3) and refer this new method as SimCLR-L. Table 4 reports the performance, and we observe that although there a gap in performance between regular SimCLR and VICReg, the additional benefit provided by the local criterion is much stronger with VICReg.
Impact of multi-crop. We study the impact of using the multi-crop strategy for the data augmentation. by comparing VICReg to VICRegL. Table 5 reports our results. Whether multi-crop is used or not, introducing the local criterion always improve significantly the segmentation results, while preserving again most of the classification performance.
4.3 Visualization
We provide in Figure 3, a visualization of the pairs of matched feature vectors selected by VICRegI.. Right to the seed image, the left column shows the feature-based matches, and the right column shows the location-based matches. Each case in the the grid represents a position in the feature map, and match between two feature vectors is represented by a yellow line. The receptive field of these feature vectors is larger than single case in the grid, and actually spans the entire image, but we observe that the embedding space is shaped such that the feature-based matching is coherent regarding the semantic content at a position in the image where feature vector pooled. A feature vector that is located at position corresponding to a texture representing "sky" or "grass" in one view is matched to another one on the other view located at position corresponding to a similar "sky" or "grass" texture. Additional visualizations are available in Appendix ??.
5 Conclusion
In this work, we introduced VICRegL, a method for learning both local and global visual features at the same time, by matching feature vectors with respect to their distance in the pixel space and in the embedding space. We show that introducing a local criterion significantly improves the performance segmentation tasks, while preserving the classification accuracy. We also demonstrate that convolutional networks are competitive to vision transformers in sule-vupe learning, by using the ConvNeXt backbone.
L Limitations and Future work. Convolutional neural networks by design produce feature maps that have a receptive field that covers the entire image. It is not clear to which extent a feature vector at given position in the feature maps actually contains mainly information about the objects located at the corresponding location in the input image. The learned tokens of vision transformers are also good candidates for local features, and a detailed study of the actual local nature of both the feature vectors of convolutional network and the tokens of vision transformer, would provide useful insights for future directions of supervisd learning of local features. Future work will also tackle the problem of learning hierarchical features, by applying criterion not only at local and a global scale, but also at multiple levels in the encoder network.
Acknowledgement. Jean Ponce was supported in part by the French government under management of Agence Nationale de Recherche as part of the "Investissements avenir" program, reference ANR-19-P31A-00N (PRAIRIE 31A Institute), the Louis Vuitton/ENS Chair in Artificial Intelligence and the Inria/NYU collaboration. Adrien Bardes was supported in part by a FAIR/Prairic CIFRE PhD Fellowship. The authors wish to thank Li Jing, Randall Balestriero, Amir Bar, Grégoire Mialon, Jiachen Zhu, Quentin Garrido, Florian Bordes, Bobak Kiani, Surya Ganguli, Megi Dervichi, Yubei Chen, Mido Assran, Nicolas Ballas and Pascal Vincent for useful discussions.
10

---

References
A. Bardes, J. Ponce, and Y. LeCun. Vicreg: Variance-imaarinnc-covaaaaa regularization for self-supervised learning. In ICLR, 2022. 3, 6,
M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep clustering for unsupervised learning. In ECCV, 2018. 2
M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 2020. 1,2,5
M. Caron, H. Touvron, I. Misra, Jegou, and J. M. P. B. A. Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 2,
Chen, S. Korbliihh, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020a. 1, 2, 6, 10
X. Chen and K. He. Exploring simple siamese representation learning. In CVPR, 2020. 2
X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv preprint wxXiv2003304229, 2020b. 1, 2,
X. Chen, S. Xie, and K. He. An empirical study of training self-supervised vision transformers. In ICCV, 2021. 2.
Y. Chen, A. Bardes, Z. Li, and Y. LeCun, Intra-instance vicreg: Bag of self-supervised image patch embedding. arXiv preprint arXiv:2206.08954 2022.
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and
B. Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 6
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, X. Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x 16 words: Transformers image recognition scale. In ICLR, 2021. 8
D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In ICCV, 2021. 2
A. El-Nouby, G. Izacard, H. Touvron, I. Laptev, H. Jegou, and E. Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2112. 2022.
M. Everingham, I.. V. Gool, J. W. Christopher K. I. Williams, and A. Zisserman. The pascal visual object classes (voe) challenge. IJCV, 2010.
Q. Garrido, Chen, A. Bardes, L. Najman, and Y. Lecun. On the duality between contrastive and non-contrastive self-supervised learning. arXiv preprint arXiv:2206.02574, 2022. 2
Goyal, Dollár, R. Girshick, P. Noordhuis, Wesolowski, A. Kyrola, A. Tulloch, Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:1706 02677. 2017. 6
J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS, 2020. 6
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020. 2
K. He, Chen, S. Xie, Y. Li, P. Doll, and R. Girshick. Masked autoencoders are scalable vision learners In CVPR, 2022.1 7

---

R. D. Hjelm, A. Fedorov, S. deep representations by mutual information estimation and maximization. In ICLR, 2019. 2
K. Grewal, A. Trischler, and Y. Bengio. Learning
J.J. Hénaff, S. Koppula, J.-B. Alayrac, A. van den Oord, O. Vinyals, and J. Carreira. Efficient visual pretraining with contrastive detection. In ICCV, 2021. 1, 3,
O. J. Hénaff, S. Koppula, E. Shelhamer, D. Zoran, A. Jaegle, A. Zisserman, J. Carreira, and R. Arand- jelović. Object discovery and representation networks. arXiv preprint arXiv:2103.10957, 2022. 1, 2.3
K.-H. Lee, A. Arnab, S. Guadarrama, J. Canny, and I. Fischer. Compressive visual representations. In NeurIPS, 2021. 1,2
C. Li, J. Yang, Zhang, M. Gao, B. Xiao, X. Dai, L. Yuan, and J. Gao. Efficient self-supprvised vision transformers for representation learning. In ICLR, 2022. 1,2
Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T.. Darrell, and S. Xie. convnet for the 2020s. In CVPR, 2022. 5, 6
I. Loshchilov and F. Hutter. Sgdr: stochastic gradient descent with warm restarts. In ICLR, 2017.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
I. Misra and L. v. d. Maaten. Self-supervised learning of pretext-invarian representations. In CVPR, 2020. 1
J. Mitrovic, B. McWilliams, J. Walker, L. Buesing, and C. Blundell. Representation learning via invariant causal mechanisms. In ICLR, 2021. 2
P. H. Richemond, J.-B. Grill, F. Altché, C. Tallec, F. Strub, A. Brock, S. Smith, S. De, R. Pascanu, B. Piot, and M. Valko. Byol works even without batch statistics. arXiv preprint arX1v:2010.10241, 2020. 2
N. Tomasev, I. Bica, B. McWilliams, L. Buesing, R. Pascanu, C. Blundell, and J. Mitrovic. Pushing the limits of self- resnets: Can we outperform supervised learning without labels on imagenet? arXiv preprint arXiv:2201.05119 2022. 1,2
F. Wang, H. Wang, C. Wei, A. Yuille, and W. Shen. Cp2: Copy-paste contrastive pretraining for semantic segmentation. arXiv preprint arXiv:2203. 1709, 2022. 3, 6,
X. Wang, R. Zhang, C. Shen, T. Kong, and L. Li. Dense contrastive learning for self-supervised visual pre-training In CVPR, 2021. 2,
T. Xiao, C. J. Reed, X. Wang, K. Keutzer, and T. Darrell. Region similarity representation learning. In ICCV, 2021. 2,
Z. Xie, Y. Lin, Z. Zhang. Y. Cao, S. Lin, and H. Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In CVPR, 2021. 1,2,
C. Yang, Z. Wu, B. Zhou, and S. Lin. Instance localization for self-supervised detection pretraining. In CVPR, 2021. 1,
J. Yang, K. Zhang. 7. Cui, Su, J. Luo, and X. Wei. Inscon: Instance consistency feature representa- tion via self-supervised learning. arXiv preprint arXiv:2203.07666 2022. 1, 3, 6
Y. You, Gitman, and Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03880 2017. 6
Zbontar, L. Jing, Misra. Y. LeCun, and S. Deny. Barlow twins: Self-supervised learning via
redundancy reduction. arXiv preprint 0mxiv22103032230 2021. 1,2
B. Zhou, H. Zhao, X. Puig. T. Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019. 6
Zhou, C. Wei, H. Wang, W. Shen, C. Xie, Yuille, and T. Kong, ibot: Image bert pre-training with online tokenizer. In ICLR, 2022a. 2,
Zhou, Y. Zhou, C. Si, W. Yu, K. Ng, and S. Yan. Mugs: A multi-granular self-supervised learning framework. arxiv preprint arXiv:2203.14415. 2022b. 2
12


Reviews:
[{"review_id": "t0aTYzLHKA", "reviewer": "Reviewer_yFUm", "summary": "The paper proposes VICRegL, a method that extends the existing (global) VICReg objective with local contrastive objectives where the loss is applied at the level of feature pixels (between spatial locations in the feature map before global pooling in CNNs).\nTwo approaches to building pairs of feature pixels for contrastive learning are proposed: 1) based on the l2 distance in the learned embedding space, and 2) based on the known spatial location between the two generated views.\nIn experiments, the method is evaluated on (global) image classification tasks on ImageNet and (local) semantic segmentation tasks on Pascal VOC, Citiscapes, and ADEK20K. \nAblations demonstrate a trade-off between global and local performance. A good balance between global and local SSL objectives can obtain competitive image classification performance while achieving SotA performance on segmentation tasks.", "questions": "I would appreciate it if the authors could address the weakness listed above, especially the first four points, i.e., \n- How does the method compare to [N1]?\n- Is the VICReg necessary, or would the approach also work with other objectives?\n- How are the results for prior work obtained in Table 1, and what explains the inconsistency with DenseCL? \n- How does the work relate to and differ from the various related prior works (including [N1-N3])?", "limitations": "Limitations were addressed well in the paper. The point about whether feature pixels learn local information is interesting and could be explored through experiments, e.g., by clustering the feature pixels of an image and visualizing these clusters. \n", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Study of SSL pre-training that performs well on various downstream tasks (local and global) is valuable\n- Good performance in transfer to both classification and segmentation\n- Technical presentation is mostly clear\n- A good set of ablation experiments is provided, demonstrating:\n\t- the trade-off between local and global feature learning\n\t- The two ways to construct local pairs (l2 in embedding space and spatial distance)\n\t- The influence of the number of local pairs to use in the loss\n\n\nWeaknesses:\n- The approach to building positive local pairs via spatial distance is very similar to [N1], which is neither discussed in the paper nor compared to in experiments. \n- The method is demonstrated only with VICReg as the contrastive criterion. It is unclear if the approach to incorporate local objectives with the global one would also work with other \"contrastive\" objectives. In principle, it should also work, and demonstrating it with more standard contrastive objectives (e.g., SimCLR) would strengthen the paper.  \n- Inconsistencies with results reported in prior work: DenseCL by Wang et al. reports 69.4 mIoU on PascalVOC with fine-tuning. It is unclear how the difference from the reported 66.8 in Table 1 comes about. \n- Several related works on learning both global and local representations are summarized in Section 2, but it is unclear how the proposed method relates to and differs from these works.\n- The idea of using nearest-neighbors in feature space for contrastive learning has been explored in prior works (e.g., [N2, N3]). While it is used in a different context here, these works should be discussed. \n- The paper argues for learning \"hierarchical features\" (L39), however, it is unclear how the proposed method should learn more hierarchical features than existing approaches based on CNN architectures.\n- The number of local pairs is fixed in the method. What happens, however, if the two views don't or barely overlap?\n- The paper mentions that multi-crop was used with ConvNeXt architectures. Therefore, providing an ablation with and without multi-crop in this setting would be good. Also, it would be good to investigate more the reason for multi-crop failing with ResNet-50, e.g., test smaller batch sizes as hypothesized in Section 3.2.  \n- Minor: Error in L104 - It should not be C in both input and output\n\n\n[N1] Xiao et al.: \"Region Similarity Representation Learning\", ICCV2021\n\n[N2] Dwibedi et al.: \"With a little help from my friends: Nearest-neighbor contrastive learning of visual representations.\"\n\n[N3] Koohpayegani et al.: \"Mean Shift for Self-Supervised Learning\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Tfdd3NDhUgU", "reviewer": "Reviewer_hQQN", "summary": "This paper introduces locality constraints building upon the existing VICReg method. It does so by not destroying the features maps and adding VICReg loss terms on the features maps. Two simple methods of correspondence generation are proposed: a) Nearest neighbour in spatial location, and b) embedding similarity of local features. The paper claims to learn visual representation that improves performances on localized tasks such as detection and segmentation while maintaining the consistently good performance for global classification task. The paper uses two backbones for evaluations and shown to perform better than existing methods especially on linear evaluations. The paper has a good contribution, but a few evaluations and analysis seem lacking. ", "questions": "1. The method chooses location representation correspondences based on nearest location and embedding similarity. Any thought or analysis on noisy or spurious correspondences? For example, although nearest neighbours the patch may be coming from different object/semantics/texture. Similarly, based on embedding similarity, the network may choose unreliable pairs especially in the early stage of training.  \n\n2. In the ablation of selected pairs, when all possible matches are selected, I believe both spatial and embedding distance losses will be the same, and network would try to match all local features though coming from different objects. Though not optimal, it seems the performance is improved compared to the baselines. Please could you provide intuition on this scenario and related improvement. \n3.  Is the network trained from scratch with global and local losses or, a pretraining with global loss is done?\n4. How does the method compared to [1]\n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\nThe authors have given satisfactory answers to the rebuttal. I would like to increase my rating to borderline accept given the authors promise to include all discussions and a complete result for detection on PASCAL and MS-COCO with standard evaluation protocol.", "limitations": "1.  [Comment/Suggestion] It is natural to fine-tune the network when supervisions are available, however, the gap in the performance boosts narrows after finetuning. May be a semi-supervised task and evaluation would highlight the importance of the method and a more meaningful scenario. \n2. It would be interesting to see any sort of analytical study to support the hypothesis of relation of batch size and dimension of the network’s output. This would be an important contribution on itself.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The idea is elegant, uses the existing loss but additionally on local features. \n2. It is shown to significantly improve segmentation performance (though difference in finetuning performances not huge.) which is convincing.\n3. Ablations studies show insights to design choices, visualization on matches.\n4. Overall, the paper is easy to follow, and the paper has a good contribution however there are a few concerns as follows\n\n\nWeakness:\n1. The main concern is that although the paper claims to benefit detection and segmentation tasks, only segmentation is evaluated.  Evaluations on detection are studied in most of the baseline papers but this is lacking in the paper. It may require a more comprehensive experiment to validate the effectiveness.\n2. The authors use ‘multicrop’ technique but do not ablate this factor in the experiments. Also, it is not clear if other baselines (especially VICReg) also enjoy the benefit of multicrop. If they do not then, the comparison may not be fair. The authors may want to clarify this.\n3. As the method uses features maps and losses on based on them, it would be interesting to see its impact on training speed and memory.\n\nPlease read below on other queries and comments below.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4kP0SFotBe6", "reviewer": "Reviewer_Pb2x", "summary": "The paper describes a learning method to learn jointly global and local descriptors in a self-supervised way.\n\nThe global descriptor are learned following VICREG [1], a learning method that tackles the issue of mode collapse during representation learning. Two versions of the same images are derived using random transformations (crop, color jittering) and fed to a siamese network. Each branch produces a 1D feature vector representing the image. The branches are optimized separately with the VICREG constraints: the feature vector must be close in the embedding space (invariance loss), the standard deviation of the embedding dimensions must be close to one (variance loss), and the dimensions of the embeddings should not be correlated (covariance loss).\n\nThe local descriptor learning is the main contribution of the paper. The feature maps provide local descriptors (a feature map of dimension H x W x C can be seen as a set of HW local features of dimension C). These descriptors are optimized so that: 1) descriptors from the same location in the image are pushed together (spatial matching loss) 2) descriptors close in the embedding space are also pushed close together (even when they do not relate to the same image location i.e. they are not ground truth match).\n\nThe several components of the final loss are balanced with metaparameters investigated in the paper. The main finding are: 1) that the bigger weight given to the global loss, the worse local features are for object detection and semantic segmentation tasks; 2) the global criterion is relevant to improve the local features suggesting that they benefit from a global understanding of the image.\n\nThe paper evaluates the global descriptors on classification and local descriptors on object detection and semantic segmentation. It compares against relevant instances of the state of the art and experiments show that the introduced optimization is indeed relevant to learn local features in a self-supervised way.\n\n[1] Vicreg: Variance-invariance-covariance regularization for self-supervised learning", "questions": "L49: \"contrasting feature vectors corresponding to far away locations can have a negative effect, as these vectors could have been pooled from locations that represent the same object in the image.\"\nThis is not clear. Does contrasting here mean pushing the feature vector together? In this case, wouldn't one want to push features of the same object together? i.e. this would be a positive outcome?\n\nThe l2-distance (Eq 3) is not intuitive at first. It pushes together local features close in the embedding space even though they might not be ground-truth matches. Is there any intuition on why this is beneficial (as supported by the experiments)?\nOne intuition is that it might act as regularizer or an implicit clustering.\nAnother is that the global criterion already pushes positive matches coarsely together and this loss refines the vector. Is there any observations or intuition on that interesting phenomenon?", "limitations": "- L107: It would be nice to introduce the notation of $\\lambda, \\mu, \\nu$ in Eq 1 for completeness.\n\n- L118: $h_{\\phi}^l$ is called a 'local projector' whereas it is the equivalent of the VICREG expander but at the scale of the feature map. Since 120 uses the terminology 'global expander', it would seem more consistent to use the word 'local expander' rather than projector. This would also be more consistent with VICREG on which the paper relies on.\n\n- L127, L135, 181: 'only the top-$\\gamma$ pairs are kept'.  It is not clear whether for each feature position, only the top-$\\gamma$ are kept to derive the loss, or if the top-$\\gamma$ among all pairs in the batch are kept (assuming top-$\\gamma$ means the pairs with the smallest distance). This would help the reader if the definition were introduced L127.\n\n- L127, L135, 181: Is there a motivation / intuition on why it is better to keep only the top-$\\gamma$ pairs?\nTogether with the l2-distance loss, this could reinforce the intuition that the l2-distance loss act as a regularizer/booster as it would be applied only to feature vectors that are already close to each other, which one can assume to be ground-truth positives.\n\n- L306: a feature maps -> a feature map", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem tackled is relevant to the community: the joint learning of global and local features\n- The originality of the l2 distance loss: it pushes feature vector that are close in the embedding space even though they are not positive matches.\n- Relevant state-of-the art\n- Relevant experiments (comparisons, analysis and ablation)\n- Very well-written paper: the organisation and presentation of the ideas is easy to follow, and the description and analysis of the experimental results are clear and valuable to the reader.\n\nWeaknesses:\n- The main contribution seems incremental with respect to VICREG.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept