FLSL: Feature-level Self-supervised Learning
Authors: Qing Su
Anton Netchaev
Hai Li
Shihao Ji
Venue: NeurIPS 2023 poster (2023)
Keywords: Transformer
ViT
Dense Prediction
Self-supervised Learning
Mean Shift
Self-attention
Representation learning
TL;DR: Proposed a novel mean-shift inspired feature-level self-supervised learning method that significantly improved the transferability on object detection and segmentation tasks.

Abstract: Current self-supervised learning (SSL) methods (e.g., SimCLR, DINO, VICReg, MOCOv3) target primarily on representations at instance level and do not generalize well to dense prediction tasks, such as object detection and segmentation. Towards aligning SSL with dense predictions, this paper demonstrates for the first time the underlying mean-shift clustering process of Vision Transformers (ViT), which aligns well with natural image semantics (e.g., a world of objects and stuffs). By employing transformer for joint embedding and clustering, we propose a bi-level feature clustering SSL method, coined Feature-Level Self-supervised Learning (FLSL). We present the formal definition of the FLSL problem and construct the objectives from the mean-shift and k-means perspectives. We show that FLSL promotes remarkable semantic cluster representations and learns an embedding scheme amenable to intra-view and inter-view feature clustering. Experiments show that FLSL yields significant improvements in dense prediction tasks, achieving 44.9 (+2.8)% AP and 46.5% AP in object detection, as well as 40.8 (+2.3)% AP and 42.1% AP in instance segmentation on MS-COCO, using Mask R-CNN with ViT-S/16 and ViT-S/8 as backbone, respectively. FLSL consistently outperforms existing SSL methods across additional benchmarks, including UAV object detection on UAVDT, and video instance segmentation on DAVIS 2017. We conclude by presenting visualization and various ablation studies to better understand the success of FLSL. The source code is available at https://github.com/ISL-CV/FLSL.

FLSL: Feature-level Self-supervised Learning
Qing Su'; Anton Netchaev?, Hai Li and Shihao Ji' Georgia State University, U.S. Army ERDC, 'Duke University
Abstract
Current self-supervised learning (SSL) methods (e-g., SimCLR, DINO, VICReg, MOCOv3) target primarily representations at instance level and do not gener- alize well to dense prediction tasks, such as object detection and segmentation. Towards aligning SSL with dense predictions, this paper demonstrates for the first time the underlying mean-shift clustering process of Vision Transformers (ViT), which aligns well with natural image semantics (e.g., world of objects and stuffs). By employing transformer for joint embedding and clustering, we propose bi-level feature clustering SSL method, coined Feature-Level Self-supervised Learning (FLSL). We present the formal definition of the FLSL problem and con- struct the objectives from the mean-shift and k-means perspectives. We show that FLSL promotes remarkable semantic cluster representations and learns an encoding scheme amenable to intra-view and inter-view feature clustering. Experiments show that FLSL yields significant improvements in dense prediction tasks, achieving 44.9 (+2.8)% AP and 46.5% AP in object detection, as well as 40.8 (+2.3)% AP and 42.1% 1% AP in instance segmentation on MS-COCO, using Mask R-CNN with ViT-S/16 and ViT-S/8 as backbone, respectively. FLSL consistently outperforms ex- isting SSL methods across additional benchmarks, including UAV object detection on UAVDT, and video instance segmentation on DAVIS 2017. We conclude by pre- senting visualization and various ablation studies to better understand the success of FL. The source code is available at htttt::///// http://gittb.... CIIe/ISSS-V/FFF.
Introduction
Following its success in natural language processing (NLP)
47 5
20 self-supervised learning
(SSL.) with transformer 58 22 has emerged as highly effective strategy and popular model choice over the CNN-based counterparts in vision tasks. The remarkable performance achieved by SSL has been demonstrated by SimCLR [E MOCOv3 16.. DINO 10. VICReg (3) SwAV (}] BYOL [27]. and among others. Without relying on manual supervision, a successful paradigm of SSL promotes semantic representations conducive to the downstream tasks, e.g., classification, detection and segmentation. However, most existing SSL methods operate at the instance-leved where an encoder is trained to maximize the agreement of the representations of multiple augmented views of an image. Though demonstrating strong performance on the classification tasks [14] 29], the instance-level SSL is inherently misaligned with the dense prediction tasks, such object detection, where the lower level semantic information plays bigger role than the instance-level semantic information. This leads to inferior transferability to those dense prediction tasks.
Recent attempts to bridge the semantic gap are mainly based on region 550 patch 69 or pixel (i.e., dense feature) matching tasks [63 rk 3X
with optional instance-level objectives. However, learning of distinct representation for each image patch or region still mismatches the natural semantics within an image (referred to as local semantics), where features of the same semantics should be highly correlated other than being distinct. Semantics can range from features of high similarity, features of the same object, to more complex semantic structures. In light of this, methods such as SoCo 65 ORL 770 and DetCon
32) leverage the off-the-shelf algorithms, e.g., selective
To whom correspondence should be addressed: qau3dgsu. edu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

w x+
Hplant
Semb
person
intervations
bbook
e+ plant
Eplant
claser
separttion
local
* eqprssettativee of chister of local Festurce.
* ripresetrstive of chater of positive eeprosentalivee
Figure I:
The bi-level clustering of FLSL. An object or stuff in an image is essentially cluster of features. Hence, their representations can be extracted cluster representatives, e.g. modes. In FLSL, we aim to make these representations both locally and globally semantic via bi-level clustering process. On the first level, the locally semantic representations are fostered by driving features of various concepts (book, person, plant, etc.) closer to their cluster modes 2 and away from features of other concepts within an image(intra-viet clustering). On the second level, cluster modes serving as representations & are pushed closer to their positive samples & in X*, which is augmented via random transformation f T (inter-view clustering). In such way, those representations encode the category information and become globally semantic.
search [S5] and wtwp:ssswwwbb-uuuuenoocooe algorithm
impose the semantic constraint the contrastive learning pipeline. Nonetheless, the inclusion of a non-trainable region proposal module in those methods restricts the model's ability learn the distinct representations for those RoIs from the rest of an image. This ability is vital in representation learning for object detection.
Existing SSL methods targeting dense prediction primarily focus the learning of globally semantic representations of image sub-regions, such as Rols, patches, or pixels. However, these methods fall short with limited consideration for the alignment of those representations with local semantics. This observation leads us to ask the following question: Can we learn a representation that is both locally and globally semantic for group of features (e.g., representing end-to-end trainable SSL approach? To this end, we propose the Feature Level Self-super Learning (FLSL). It leverages the mean-shiff clustering process inherent in transformer extract modes as representations and incorporates k-means-base representations are semantically coherent both locally and globally. Figure. of FLSL with details to be discussed in Sec. !
object stuff) in an
SSL approach to ensure that the extracted
F iillutrattes an overview
Contributions This paper takes step forward bridge the gap between the current SSL methods and downstream dense prediction tasks. Our contributions are summarized as follows:
1. We demonstrate for the first time the connection between the attention mechanism and mean-shift clustering, and reinterpret vision transformer from the perspective of mean-shift.
2. By employing transformer for joint embedding and feature clustering, propose FLSL, an end- to-end trainable SSL method that promotes the representations of feature clusters to be semantic at two levels: (i) intra-view: within an image, and (ii) inter-view: over entire dataset.
The derivation and construction of the FLSL objectves is rooted in mean-shift and the non-empty k-means clustering, Semantic representations on the first level are encouraged by optimizing the intra-cluster affinity with self-attention layer, while the second-level semantic representations are fostered via non-empty k-means clustering with positive samples retrieved through cross- attention layer.
We validate the synergy between FLSL and VIT, and show significant improvement in transfer- ability of the learned features to dense prediction tasks, including object detection and semantic segmentation. FLSL-pretrained VIT on ImageNet- 1k (INIK) demonstrates superior performance compared to the state-of-the-an ADCLR-INIA (76 and MAE 400 pretrained counterparts. More- over, consistently outperforms existing SSL methods across additional benchmarks, including UAV object detection UAVDT, and video instance segmentation DAVIS 2017.
2 Related work
SSL for dense prediction Recent attempts to bridge the gap between common SSL and dense prediction tasks focus primarily on sub-region matching tricks. For example. DenseCL 5% applies contrastive learning on pairs of patches with highest similarity. However, the patch-matching trick leads to distinct representations with low correlation among patches, which is not well-suited for the semantics of natural image. On top of the instance-level objective, PixPro [Z3 and IC-losss [55]
factor in agreement between positive pixel pairs which are assigned through thresholded-distameee in

---

maximizes the agreement between sliding- window-pooler representations in the overlapped region of two augmented views. DetCo [69] further incorporates instance-pan -ptcc level contrastive losses along with instance level and patch level losses. To learn representations at object level, SoCo 655 and ORL 77C employ selective search to crop out Rols. ORL further enables inter-object representation learning via BYOL 127 using top-ranked Rol pair. In contrast, SCRL (50] relaxes the semantic constraint using random crops within the intersection area of augmented views Rols. As discussed in Sec. [ all of these methods focus on learning globally semantic representations for image sub-regions, and do not touch on local semantics that are necessary for dense prediction.
PixPro and position projection in LC-loss. ReSim 68
Self-supervised vision transformer In pioneering works, self-supervised training of transformer for vision tasks generally follow the paradigm of masked autoencoder in NLP 47 200 For instance, iGPT [ features reconstruction of masked pixels as one of its objectives. In general, SSL for
ViT can be classified into two categories: the joint-embedding strategy epitomized by DINO [TC]
and MoCov3 [d. and the generative approaches represented by MAE (2 The crossover of the two strategies demonstrated by iBOT 78 Regarding dense prediction EVVIT
B8 designed for Swin Transformer 43), follows the oggion-machinn strategy and applies the DINO loss to the
probabilities of positive pairs determined by highest similarity. Instead of finding the best-matching patch, SelfPatch (75] considers the direct neighbors as its positive patches. However, with limited semantics contained in a fixed small area (e.g.. 8-connected neighbors), the method still suffers from semantic misalignment. To address the sub-region mismatch issue of DINO, ADCLR 76 constructs query tokens from random sub-regions and treats them as extra class tokens in the DINO objective. This promotes region-aware semantic representations that better aligned with the local semantics, and leads to substantial improvement in dense prediction.
3 Intuition: the connection between mean-shift and attention
As discussed in Sec. the misalignment between the current SSL methods and dense prediction tasks lies in the clustering bias at the semantic level. Instead of setting a fixed granularity, such as instance-level or fix-sized patch-level, a desired semantic representation scheme should be able to represent from single patch to a cluster of patches or even an entire image. The representation space of an image can be considered as an empirical probability density function of features, and the modes (local maxima) therefore can be regarded as the representatives of clusters [ I7 [8]. These modes can then be readily retrieved via clustering algorithms, particularly, non-parametric kernel density estimation (KDE) methods 32) when the image composition (e.g., number of objects and stuffs) is unknown. One typical Kas-b method is the mean-shif clustering B3 In the following, we first give an overview of self-attention (SA) mechanism of transformer and the mean-shift algorithm. We then show that the mean-shift update rule conforms to the SA mechanism of transformer.
Attention mechanism First introduced recurrent neural networks context extractor for machine translation (2) attention has premised major breakthroughs in NLP with the emergence of transformer that relies solely on the scaled dot- proudcctttt attention mechanism (58) given by
(I) where Q, K and V denote query, key and value matrices which pack together sets of query, key and value vectors, respectively. Det denotes the dimension of query and key vectors, and softmax (Z), exp(Z,,) / exp( Z.x). As a special case of attention, SA matches a sequence Z with itself to extract the semantic dependencies among its components, ie.,, D= K= W Z, where the projections W_'s 's are the parameter matrices.
attention (Q, K, V)= softmax Q K/VDxt)
Mean-shift clustering and attention Given N data points {zi C IR" the kemnl density estimate of p(z) with kernel K(1) can be defined as
1 Ti T,
p(z)= Σ p(zz)(z12z) Σ
K K(d(z, 26; E,)),
(2)
EN where p(z,) = n; is the mixing proportion of point 2;, s.1. 1=1 1, T; denotes the normalization term dependent only on the covariance matrix Σ, e.g., for a Gaussian kernel T: | 1/2 and
d(z, z; Ex) = (z 2x) Σ.¹ (z 21) is the Mahalanobis distance. Finding the modes of p(z) is
3

---

to seek stationary points by equating the gradient of p(z) to zero, ap(=) = 0, which arrives at
mi K (d( z;; E.)
2 f Σ p(zilz)z,, with p(zi
(3)
Σラ 1 K 2d(
where KK=dK/on dt. The above fixed-point iterative scheme is the mean-shift algorithm. Practically, on fz-normalized vectors, for a homoscedastic Gaussian kernel with constant mixing proportion and isotropic covariances (e.g., n. 1/N, 1/0² =T), Eq. furthe simplifies to
N
exp Tz)
Σ mcanshift (2, T)
j=1 exp (rzT2,) = 2 = Z softmax (TZ Z)
(4)
which conforms to the attention function (Eq. D with identity projection matrices, i.e., Wº = WK Wr I, and 7 = 1l//////pp Conversely, the conventional SA mechanism can be viewed as a generalized mean-shift:
(5) with learnable distance measure Z T (W wx)2 Z and projection Wv. Unlike GMM and k-means, mean-shifi is capable of modeling clusters of complex non-convex shape with cluster number automatically determined by local scale (proscribed by covariance) 33 Hence, it is well-aligned
S = Wy Z softmax (V/D,, (WT Z)
with the semantics of natural images.
VIT from the perspective of mean-shift In ViT images are initially tokenized and then processed through sequence of transformer layers. Each transformer layer comprised of a skip- connected multi-head SA (MHSA) and skip-connected MLP. MHSA can be constructed from Eq. 5 with m projections in parallel, i.e., [W W K. W h 1...., m. The m returned modes are then concatenated along channel dimension and reprojected to a single return through
= MHSA(Z) = W Woconcal
2Tm (122))....
+ bo.
(6)
Note that the & normalization assumed in Eq. 4 is moderately relaxed via layer normalization (LN) to incorporate the extra degree of freedom in the vector magnitude. With skip connection and the one-step mean-shift update described in Eqs. 5 a transformer layer essentially finds the local centroid for each query x and drives them closer to the re- re-projected centroids through = = x + 2, followed by an MLP processing step with skip connection. ViT iterates the process multiple times (e.g., 12 or 24 layers) to capture the contextual and semantic information of an image.
The clustering process above concords with one inductive bias of the attention mechanism represented by the sparse variable creation [24). i.e., an SA head learns a sparse function that only depends on small subset of input coordinates. In the context of clustering, the subset of input corresponds to the modes of density p(z). As the high-level semantic information is typically spatially sparse (e.g., the representaion for Rol in object detection, a single label for a region in segmentation, or scene-graph etc.), it is natural to leverage transformer for joint embedding and clustering to learn semantically meaningful representations.
4 Methodology
FLSL features a bi-level clustering process (Figure D.. which is formally described as follows.
Given dataset X' (e.g.. set of images), FLSL learns an encoding scheme fo: + Z, VX E EX, Z= fo(X). can be formulated as = UN where ge is a subset of Z forming a cluster, N₂ the number of clusters determined by clustering scheme, e.g., mean-shift, and N. S [Z]. FL.SI. aims to encourage the following properties:
(i) Intra-view: encodings corresponding to semantic concept (as a cluster), cluster representative (e.g., mode) & and far away from the encodings of other clusters; (ii) Inter-view: the cluster representatives és of the positive regions in Xs over X' are pushed closer to each other.
E are close to the
The FLSL-extracted features should be well-aligned with dense prediction tasks, such as object detection, where the representation an object or stuff (i.e., cluster of features) are desired to be (i) well-separated from others in an image (locally semantic), and (ii) close to its positive samples in the dataset (globally semantic). In this section, we present the objectives for both levels of clustering, which are then combined to form the final objective.

---

11 ~
ge())
X
2= (=/1
CA
2
[pr 11-1"
ngreement
Maximize
EMA
EMA
X
[=
SA
IN T
1.
g.(-)
Maximize agreementt
Figure 2: Overview of FLSL framework. Similar to DINO ITN. FL.ST. comprised teacher network and a student network, which have the same architecture a VIT encoder f and a projection head g but with different parameters. Two mean-shift operations: a non- parametric self-attention (SA) and a non parametric cross-attention (CA) are applied to the last layer of ft, before 9t. 9.. respectively, and the CA takes output of fa as queries. The two networks trained maximize the agreement between the probability distributions pis and p, s and the agreement between features 2,8 and their cluster representatives &,8.
4.1 Intra-view clustering with mean-shift
As discussed in Sec. local semantics of an image can be captured by non non-parametric clustering such as mean-shift. Hence, with mean-sile shift update rule Eq. El can be proved that the probability of zj given point p(= x2i = [softmax (TZ, Z] should satisfy:
p(2z1z)) Σ elal5z2- ((=5s==aTesh)) +(N- ) aile AuT)). V.j. E
kEc;
where N = 1Z) q is the set of indices of points in the same cluster including 2;, and Dij is the degree of separability defined as A,==2|z - (a x/, such that larger A., = EEje, As indicates better separation. For locally semantic encodings, we desire the in-cluster points to be close to each other, equivalently, to be close to its cluster representative, and stay far away from the out-cluster points, which indicates a large value. As becomes sufficiently large, the RHS of Eq can be approximated as 1/ k/Ee cxp ((z, 2 2, 2,2,)r), and for out-cluster points, the probability p(zjge, (z,) approaches to O. This results in semanties-al alligedd cluster representative via mean-shift a weighted sum of only in-cluster points. This can be realized by contrasting among points using attention map soft cluster mask to drive the query point 2; closer the returned mode 2, It leads to the intra-view clustering objective:
N
min f0
z
2/1.
(8)
Proof of Eq. 7 and detailed explanation is provided in Appendix
4.2 Inter-view clustering with k-means
To learn globally semantic representations, similar to the existing SSL methods, we formulate the problem as a variant of k-means clustering. For és extracted from an entire dataset, the k-means objective with generalized non-empty cluster constraint [4] can be expressed as
K
min M
0k((E)l( - DKI. (p) T)]
N
#E2k=1
where M is set of K centroids [m1.-. ... "...,"",,,...... E is a set of cluster representatives over the entire dataset, N' = 12... k( 2) arg min - 2 2112, d.j is the Kronecker delta, with d;₃ = 1 iff i i= and otherwise, [p]ia = 1/n' Ex dAsE)) and π the prior, e,g., a vector of the preset proportion for each cluster. With positive pairs ($+, created via data augmentation, the objective can then be constructed k-means clustering with extra separation margin for št:
min M
5KMEE) MM
ōEce )a(()) || (40()) 12 + DKL (p) TT) (10)
N7
A common approach to tackle the optimization problem above is to relax the hard cluster assignment constraint 6, E {0, 1} to [o, 1] via classification head to & with a small temperature (< 1). This relaxes Eq. to more general Gaussian Mixture Model (GMM) formulation (cf. Appendix B
By rewriting 5x( Mx()) in Eq. 10 as Σk= ( SeK(a + ( and with the relaxed hard cluster assignment via classification head, the objective for the inter-view clustering can be
5

---

expressed by
min M
1
H(p(2*),p(3) + DKL (fl(T), $
Σ
(11)
N'
where p(a:)= softmax (7 W a) centroids, and H(z, - log y (cf. Appendix
1 with We defined matrix of K orderly concatenated
Positive sample retrieval Unlike the common instance-level SSL, the positive samples FLSL are amorphous clusters of features, (x 2). corresponding to the same semantic concept in two views. In contrast to previous works assigning the best-matching patch [8 63) thresholded vicinity 73, we leverage the cluster assignment mechanism inherent in mean-shift, where a query automatically assigned to a cluster represented by the return For query from another view, the mean-shift naturally manifests as cross-attention (CA),
(12) With representations semantically coherent on local and global levels, the returned 2 from the augmented view Z by query z should agree with the returned from the original view. To help establish this semantic constraint, representations at the projected positions from the augmented view can be used as positive samples at the early stage of training. This process can be viewed as data retrieval in dense associative memory recognized in 48
*+ = Z softmax (T= 2-).
4.3 FLSL Objective
By combining the objectives from the two clustering levels, we arrive at the objective FLSL:
min 1
ΣΣ xEZ
u1| -122
Σ H(p(2*) p(2)) + DKL (fl(TT),
(13)
N'
=EZ
with 8 = SA 2, - = CA(z, ,z)
where v, 17 and the hyperparameters controlling the importance of each term, and the SA and CA above are non-parametric.
Figure illustraate the FLSL framework. We follow the common joint-embedding strategy of SSL, except that we simultaneously maximize the agreement between positive cluster representatives (p(2+) p()) and the agreement between
in-cluster point and its cluster representative (z. 2). The KL-divergence term in Eq. [3 serves as a volume maximization regularizer. Experiments show that the FLSL objective effectively promote locally and globally semantic representations, resulting in significantly improved transferability of learned features to object detection and segmentation. Note that FL.I. does not involve a class token in its objective (Eq. [3]
Experiments
In this section, we evaluate the performance of FLSL by conducting extensive experiments. Specifi- cally, we compare FLSL to existing SSL approaches on multiple dense prediction benchmarks: (i) MS-COCO 42 object detection and instance segmentation, (ii) UAVDT
object detection from UAV platforms, and (iii) DAVIS video instance segmentation 46 Moreover, we investigate the properties of FLSL features in terms of semantic alignment and feature separability in the embedding space. Detailed experimental setups are provided in the respective subsections and supplementary performed on Nvidia RTX A6000.
materials. All our experiments
Implementation details The implementation of VIT in our experiments mostly follows DeiT [54]
excluding the [class] token. The configuration of the ViT variants utilized in this paper summa- rized in Appendix E.3 The coefficients of Eq. 13 in our experiments are v = ,03,
and y 5 unless stated otherwise. We assume a uniform prior, i.e.. TT 1/K, Vk. Models are pretrained 45 with batch size of 512. We follow the data augmentation from BYOL (e.g., color jittering of brightness, contrast, saturation and hue, Gaussian blur and solarization) with preceding random crops and resizing (to 224 224) and make them asymmetric. Computation among dense features can expensive. Therefore, we apply grid random sampling to the queries. All VIT models are pretrained for 300 epochs as in most baselines for a fair comparison. Pseudo-code, training details, and settings of augmentation pipeline provided in Appendix E
on ImageNet-I1 1k 522 dataset using Adam W optimizer

---

Pretrain
Backhone
Epoch
#Params
Ap
AP
Ap
Apm
APM
AP
MoCa-v2
RN3O
200
23M
38.9
592
42.3
35.3
56.2
37.8
DetCo
RN50
200
21M
40.1
61.0
43.9
36.4
58.0
38.9
DenseL
BYOI.
RNS0
1000
23M
40.4
61.6
44.1
37.2
58.8
39.8
SCRL
RN50
1000
23M
41.3
62.4
45.0
37.7
59.6
40.7
MOCO-+3
2IM
39.8
59.6
39.2
MoBY
VIT-S/16
300
21M
41.1
63.7
44.8
37.6
60.3
39.8
DINO
VIT-S/16
300
21M
40.8
63.4
37.3
59.9
39.5
DINO+SelIPacch
VIT-S/16
200
21M
42.1
64.9
46.1
38.5
61.3
40.8
ADCLR
S/16
300
2IM
44.3
65.4
47.6
39.7
62.1
41.5
VIT-S/16
FL.S..
300
21M
46.5
69.0
51.3
42.1
65.3
45.0
Table 1: MASK R R-CNN ON COCO
Pretrain
Ap
Ap
AP
AP
Apm
Preirain
Backbome
APyoc
None
IN-Ik
VIT-5/16
IN-Ik Supv.
47.6
42.4
IN IK DINO
VVT
49.1
IN-21k Supy.
47.8
42.6
IN-IK DINO
VT--88
51.1
IN-1K DINO
48.9
32.9
52.2
62.4
43.7
IN-1K FLSL
VIE-S/1
IN-IK MAE
51.2
34.9
54.7
66.0
45.5
IN-IK FLSL
VIT-B/16
53.5
53.1
IN-IK
VIE-S/8
Table 2: VITDET-B/16 WITH MASK R-CNN ON COCO
Table 3: FASTER R-CNN FPN ON UAVDT
Baselines We compare FLSL with various existing SSL approaches that are based on the ResNet
31
and VIT [22] architectures: (a) self sule-suuee ResNet: MoCo-v2 [[5], DetCo [69 DenseCL
63
BYOL 2 and SCRL [50]; and (b) self-supervised ViT: MoCo-v3 [6]. MoBY 722. DINO
MAE 28 SelfPatch
7 and ADCLR 76
Protocol for hyperparameter tuning Standard instance-level SSL evaluation protocols typically utilize one of the two approaches: employing a k-NN classifier or training a linear classifier on fixed features. Since FLSL learns dense semantic representations rather than single instance-level representation, both standard evaluation protocols are not suitable for evaluating FLSL in training. Moreover, fine-tuning downstream dense prediction tasks can be computationally expensive due to complex prediction heads, and may introduce lask-specific biases during hyperparameter tuning. Therefore, we design bbox-aligned k-NN classifier modified from [67] to evaluate the feature quality directly without additional network tuning. Here is an overview of the method. Features of the training data are first extracted with a fixed model. These features are then aligned with their corresponding bounding boxes provided by ILSVRC [51]. For each image, a certain number of representative features (e.g., 9) are selected by a partition criterion and stored memory. The k-NN classifier matches each selected features to its k-nearest stored features, which collectively vote for its label. A feature is considered successfully classified if any the representative features match its class. This protocol is employed for hyperparameter tuning and ablation study of the FLSL pipeline. Appendix A provides further details on the choice of k; implementation specifics and evaluation results.
5.1 MS-COCO Object Detection Segmentation
We adopt Mask R-CNN detection framework by incorporating three variants of VIT: (i) VIT-S/16 with FPN T... (ii) ViT-S/8 with FPN, and (iii) VIT-B/16 with simple feature pyramid (ViTDet) A0 Models of (i) and (ii) are fine-tuned following the multi-scale training 666 X under the standard
Ix schedule for a fair comparison. For the model of (iii), we follow the training recipe of 40 and fine-tune the model for 100 epochs.
Results. Table reports the detection and segmentation performance of ViT-S/16 and ViT-S/8 with Mask R-CNN [30 on COCO. Specifically, FLSL with ViT-S/16 outperforms ADCLR 76 by +0.6% and +1.1% %, and substantially outperforms DINO+SelfPatch (5) by +2.8% and +2.4% on detection (Aphhox) and segmentation (AP"*), respectively. Both baseline methods feature patch-level contrastive learning. Unlike SelfPatch contrasting between patches within the adjacent neighborhood and ADCLR contrasting via learned queries of random crops, FL.SI. contrasts the representatives (modes) of feature clusters, which aligns closer with the downstream tasks and thus leads to superior performance. Notably, FLSL with ViT-S/8 further improves the performance by a large margin of +4.4% in Aph and +3.6% APnk over SelfPatch. Table 2 summarizes the results of ViTDet. FLSL shows large performance gains over the DINO baseline by +4.2% Apbbox and +3.3% Apmk FL.SL. also outperforms the SOTA generative approach, MAF, by +1.7% and +1 1.4% the two tasks, respectively.

---

Pretrain
Arch.
(J&rF)) mn
Jm
F.
supe.
VIT-S8
66.0
68.1
VLOG CT
RN50
48.7
46.4
50.0
YLS MAST
RNI8
65.5
67.6
IN-IK DINO
VIT-S/16
VII-B/16
IN-Ik DINO
VIT-S/8
69.9
66.6
73.1
ILSI
VIT-S/16
65.6
62.4
69.4
FLSI.
VIT-B/IÉ
66.1
62.9
70.0
Input [=12
1=10
1=
FL.T.
VTT-S8
73.5
69.9
78.1
Figure 3: Visualization of the maps of the aggregated attention score (ASS) from different layers of ViT-S/16. I = 0 denotes the projection layer. As layer goes deeper, the map becomes more partitioned with brightness aligned with the area of the underlying semantic region, e.g., objects stuff.
Table 4: DAVIS 2017 VIDEO INSTANCE SEGMENTATION. We evaluate the quality frozen features on vidco instance track- ing. We report mean region similarity Im and mean contour- based accuracy Fm.
5.2 Small Object Detection: UAVDT
To assess the transferability of FLSL beyond the datasets of common images like COCO, we further investigate its performance on a UAV benchmark, UAVDT [23]. which exhibits significant domain shifts from common images (i.e., images captured by ground-level cameras). We utilize Faster R-CNN framework 49 with the same ViT variants used in the COCO experiments and follow the training settings outlined in ClusDet (74). All ViT-backboned models are trained with X schedule.
Result Table 3 presents the performance of ViT-S/16, ViT-S/8, and ViT-B/16 with Faster R-CNN for detection tasks on UAVDT under different pretrain schemes. We utilize the official evaluation method in 2 which calculates the class-agnostic VOC AP exclusive of the predictions that falls in the ignored areas. FLSL consistently outperforms DINO (a typical instance-level SSL for ViT) across all three ViT variants by significant margin. With smaller objects and an imbalanced foreground- background ratio, the significance of local semantics becomes evident. Models require local context to discover small objects and make accurate predictions rather than relying solely the global semantics of the entire image. This situation aligns well with the strengths of FI.SI..
5.3 DAVIS Segmentation
To further assess the quality of frozen features learned by FLSL, we evaluate FLSL--prerriined ViT models on DAVIS2017 46), following the evaluation proto- col in R6 [0] that requires fixed representations with no extra training.
380
Results Table 4 shows that FLSL consistently out- performs DINO across all VIT variants in our exper- iments. The protocol evaluates the quality of learned dense features via segmenting scenes with k-nearest neighbors (k = 5) within a fixed window (12 x 12) between consecutive frames. This requires dense fea- tures to be locally semantic, i.e., features correspond- ing to the same semantics should be more correlated. Therefore, the improved performance confirms that FLSL encourages model extract locally semantic representations.
(4)
Input
[=12
1=8
1=4 [=0
(b)
Figure 4: (a) visualization of attention probing by query patches (marked out in green circle in the top row) from the last layer of ViT-S/16 pretrained with FLSL and with DINO. FLSL encourages the model to learn semantic correlations among patches; (b) visualization of separability of the dense represen- tations throughout the transformer (VIT-S/16).
5.4 Alignment with Image Semantics
To qualitatively show that FLSL is better aligned with the semantic layout of an image than the com- mon SSL methods, Figure 4 a) compares the self- attention probing maps for features learned via FLSL and DINO. Features from the last layer are used for evaluation. The visualizations are obtained with 2242 images. Positions of the query tokens are marked out in green circle in the top row. As shown in the middle and bottom rows of the figure, DINO promotes more correlated attention (i.e., less separation between tokens of query-relatee area and that of the rest image), while FLSL encourages
8

---

Sinkhorn
7/
7.
0=
0.00
U==
.01
VE
0/2
0
003
K
1024
2048
4096
8192
16384
1.0
1.0
0.1
68.7
70.7
71.2
65.1
R-NN top-1
68.1
72.1
72.4
72.5
72.1
1.0
66.6
1.0
5.0
72.4
Table 6:
IMPACT OF NUMBER OF CEN-
TROIDS K
Table 5: IMPACT OF COEFFICIENTS IN THE FLSL OBJECTIVE.
attention to the regions of high semantic relevance with the query tokens and results in clearer maps consistent with the underlying objects/stuff.
5.5 Feature Distribution and Separability
We demonstrate the qualitative results by visualizing the Aggregated Similarity Score (ASS) and the feature distribution in the embedding space using t-sne [57 in Figure S and Figure respectively. To generate the map of ASS, we sum up the cosine-similarity maps of all tokens, normalize the resulting map with its maximum score and visualize it thermal image, i.c., the brighter the pixel, the higher the score. For a semantically well-separated image, each patch only attends to the patches of its own semantic region, e.g., patch of an object has high similarity scores only with the patches of that object and low scores with the rest. This results in an image with partitions of different brightness proportional to the area of that region, Le., ideally the larger the size of object/stuff, the brighter the color. As shown in Figure S as the layer goes deeper, the brightmess partition of the ASS more consistent with the underlying objects and stuff in the images (e.g., person, vehicles, horse, switches, wall, and ground, etc.), which indicates the desired separation of the learned features. This also reflected in the t-sne visualization of the embeddings in Figure (b) where the representations become more clustered and separated as the attention layer goes deeper.
5.6 Ablation Study
Due to limited space, we present two major ablation studies in this section to help understand the effectiveness of FLSL. The model considered for this entire study is ViT-S trained with 100 epochs. We refer the reader to Appendix ! for the complete work.
Impact of coefficients in the FLSL objective The FLSL objective (Eq. 13 contains three compo- nents: (1) similarity between fz-normalized (features) and (modes), (2) cross-entropy of the probabilities of an augmented pair H (p(E+), p(2)). and (3) the volume maximization regularizor computationally expensive to optimally determine the values of more than two coefficients by performing grid search, especially when the ratios among them are large. We tackle this problem by first fixing 7 = 1 and setting 7 = 1 along with Sinkhorn normalization 19 to perform grid search on the value of with the empirical base condition < 1 and 7 With the fixed v. we then perform another grid search on y without Sinkhorn normalization. We implement Sinkhorn normalization as the softmax operation along the batch dimension. Table summerizes the score of bbox-aliss aligned j--N evaluation using different coefficient settings.
DKI. (p](T).
77 M.
Impact of number of centroids K FL.ST FL. is formulated as an explicit clustering problem, with the output dimension of the last fully-connected layer equal to the number of centroids K. Compared to its instance-level counterpart DINO IC, FLSL enjoys smaller output dimension (shown in Table 6. This because images have higher feature variance compared to feature clusters. For example, image in ImageNet may contain diverse content from different categories, requiring large number of centroids to cover the distribution. In contrast, a semantic cluster contains highly correlated features, such as similar textures objects from the same category, thus requiring fewer centroids. Experimentally, we find that a large number of centroids benefits performance, but is detrimental and costly when being too large. We pick K = 4, 096 for all our experiments strikes a good balance between performance and cost-effectivennss..
More experiment results on semantic segmentation and ablations including the impact of batch size and random pooling window size are relegated to Appendix.
Conclusions
This paper proposes FLSL, a feature-level supervised learning method that bridges the gap between the current SSL methods and downstream dense prediction tasks. We demonstrate for the first time the underlying mean-shifi clustering process of ViT, which aligns well with natural image semantics. Facilitated by ViT for joint embedding and feature clustering, FLSL performs a bi-level clustering: (i) intra-view clustering to extract the representatives for clusters of features within an image, and (ii) inter-view clustering to encourage the representatives to be globally semantic over

---

the entire dataset. FLSL achieves significant improvement over the SOTAs in the dense prediction tasks, including object detection and instance segmentation.
Limitations and broader impacts FLSL does not have any significant limitations other than the method is more complex (due to its bi-level clustering) than other SSL methods, and it currently only fits for ViT-based models dense prediction tasks. Exploring ways extend FLSL for tasks that necessitate a global representation while retaining its existing properties could be a potential future work. As far as we foresee, there is negative societal impact.
Acknowledgment
This research was sponsored by the Army Research Laboratory under Cooperative Agreement #W91 NF--2-22-0025. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
References
[1] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In European Conference on Computer Vision, pages 456-473. Springer, 2022.
[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: arXiv:1409.044
(e0..040944477 04939,04333222 2014.
[3] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: iifiance-invariacee-coo ization for self-supervise learning. arXiv preprint arXiv:2105. 2021.
regular-
[4] Paul S Bradley, Kristin P Bennett, and Ayhan Demiriz. Constrained k-means clustering. Microsoft Research, Redmond, 20(0):0, 2000.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pages 213-229. Springer, 2020.
[7] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018
[8] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. In ICCV, 2019.
[9] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 13:9912-9922 2020.
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[11] Miguel Carreira-Perpiiann. Reconstruction of sequential data with probabilistic models and continuity constraints. Advances in neural information processing systems, 12, 1999.
[12] Frédéric Chazal, Leonidas Guibas, Steve Oudot, and Primoz Skraba. Persistence-baset clustering in riemannian manifolds. Journal of the ACM (JACM), 60(6):1-38, 2013.
10

---

[13] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International conference on machine learning. pages 1691-1703. PMLR, 2020.
[14] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[15] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint Xiiv:2003.0429) 2020.
[16] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[17] Yizong Cheng. Mean shift, mode seeking, and clustering. IEEE transactions on pattern analysis and machine intelligence, 17(8):990-799 1995.
[18) Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis.
IEEE Transactions on pattern analysis and machine intelligence, 24(5):603-619, 2002.
[19] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[20] Jacob Devlin, Ming-Wai Meig-Wei&& Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 04005, 2018.
[21] Jian Ding, Enze Xie, Hang Xu, Chenhan Jiang, Zhenguo Li, Ping Luo, and Gui-Song Xia. Deeply unsupervised patch re-identification for pre-training object detectors. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2022.
[22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint Stti::0000.11222 2020.
[23] Dawei Du, Yuankai Qi. Hongyang Yu, Yifan Yang, Kaiwen Duan, Guorong Li, Weigang Zhang, Qingming Huang, and Qi Tian. The unmanned aerial vehicle benchmark: Object detection and tracking. In Proceedings of the European conference on computer vision (ECCV), pages 370-386, 2018.
[24] Benjamin L Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. Inductive biases and vari- able creation in self-attention mechanisms. In International Conference on Machine Learning, pages 5793-5831. PMLR, 2022.
[25] Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph- image segmentation. International journal of computer vision, 59:167-181, 2004.
[26] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint 170.007700. 2017.
[27] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:2 1212222222444. 2020.
[28] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[29] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 9729-9738, 2020.

---

[30] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[32] Olivier J Hénaff, Skanda Koppula, Jean-Baptise Alayrac, Aaron Van den Oord, Oriol Vinyals, and Joao Carreira. Efficient visual pretraining with contrastive detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10086-10096, 2021.
[33] Christian Hennig, Marina Meila, Fionn Murtagh, and Roberto Rocci. Handbook of cluster analyvis. CRC Press, 2015.
[34] Jyh-Jing Hwang, Stella X. Yu, Jianbo Shi, Maxwell Collins, Tien-Ju Yang, Xiao Zhang, and Blang-Chieh Chen. Segsort: Segmentation by discriminative sorting of segments. In ICCV, 2019.
[35] Ashraful Islam, Benjamin Lundell, Harpreet Sawhney, Sudipta N Sinha, Peter Morales, and Richard J Radke. Self-supervised learning with local contrastive loss for detection and semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5624-5633, 2023.
[36] Allan Jabri, Andrew Owens, and Alexei Efros. Space-tim time correspondence as a contrastive random walk. Advances in neural information processing systems, 33:19545-19560 2020.
[37] Dmitry Krotov and John Hopfield. Large associative memory problem in neurobiology and machine learning. arXiv preprini arXiv:2008.0665 069996 2020.
[38] Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. arXiv preprint 2007:22006.09785, 2021.
[39] Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966 2020.
[40] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. arXiv preprint arXiv:2203.16527 2022.
[41] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.
[42] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[43] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[44] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
[45] Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
[46] Jordi Pont-Tuset, Federico Perazzi, Sergi Caclles, Pablo Arbeláez, Alex Sorkine Hornung, Sorkine-Homung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arX4v: 00675, 2017.
[47] Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, et al. Improving language understanding by generative pre-training, OpenAl, 2018.
12

---

[48] Hubert Ramsauer, Bernhard Schäfl. Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprini wwxii:2008.022117, 2020.
[49] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[50] Byungseok Roh, Wuhyun Shin, Ildoo Kim, and Sungwoong Kim. Spatially consistent represen- tation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1144-1153, 2021.
[51] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC)), 115(3)211122555 2015.
[52] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115:211-2533 2015.
[53] Oriane Siméoni, Gilles Puy. Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. arXiv preprint arXiv:2109.142:
2021.
[54] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training tata-efficieet image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[55] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. International journal of computer vision, 104(2):154-177 2013.
[56] Laurens Van Der Maaten. Learning a parametric embedding by preserving local structure. In Artificial intelligence and statistics, pages 384-391. PMLR, 2009.
[57] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. Journal of machine learning research, 9(1 ()11, 2008.
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[59] Huy V Vo, Francis Bach, Minsu Cho, Kai Han, Yann LeCun, Patrick Pérez, and Jean Ponce. Unsupervised image matching and object discovery as optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8287-8296, 2019.
[60] Huy V Vo, Patrick Pérez, and Jean Ponce. Toward unsupervised, multi-object discovery in large-scale image collections. In European Conference on Computer Vision, pages 779-795. Springer, 2020.
[61] Van Huy Vo, Elena Sizikova, Cordelia Schmid, Patrick Pérez, and Jean Ponce. Large-scale unsupervised object discovery. Advances Neural Information Processing Systems, 34: 16744 16778, 2021.
[62] Matt P Wand and M Chris Jones. Kernel smoothing. CRC press, 1994.
[63] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 3024-3033, 2021.
[64] Xudong Wang, Ziwei Liu, and Stella X Yu. Unsupervised feature learning by cross-level instance-group discrimination. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 12586-12595 2021.
13

---

[65] Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin. Aligning pretraining for detection via object-level contrastive learning. Advances in Neural Information Processing Systems, 34:22682-22694 2021.
[66] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Lo, and Ross Girshick. Detectron2. https:/ /ttps://giie wtt.t//cww..accooooeeeeaa//aaaaa
2019.
[67] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non non-parametric instance discrimination. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3733-3742, 2018.
[68] Tete Xiao, Colorado J Reed. Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10539-10544 2021.
[69] Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8392-8401, 2021.
[70] Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Unsupervised object-level representation learning from scene images. Advances in Neural Information Processing Systems, 34:28864-28876, 2021.
[71] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pages 478-487. PMLR, 2016.
[72] Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao. and Han Hu. Self- supervised learning with swin transformers. arXiv preprint arXiv:2105. 04553, 2021.
[73] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16684-16699 2021.
[74] Fan Yang, Heng Fan, Peng Chu, Erik Blasch, and Haibin Ling. Clustered object detection in aerial images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8311-8320, 2019.
[75] Sukmin Yun, Hankook Lee, Jachyung Kim, and Jinwoo Shin. Patch-level representation learning for self-supervised vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8354 8363, 2022.
[76] Shaofeng Zhang, Feng Zhu, Rui Zhao, and Junchi Yan. Patch-level contrasting without patch cor-respondence for accurate and dense con-trastive representation learning. 2023.
[77] Daquan Zhou, Zhiding Yu, Enze Xie. Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference Machine Learning, pages
PMLR, 2022.
[78] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprini arXiv:2111.0788. 2021.
[79] Chengxu Zhuang, Alex Lin Zhai, Daniel Yamins, et al. Local aggregation for unsupervised learning of visual embeddings. ICCV, 2019.
14


Reviews:
[{"review_id": "7Jxa4T9wlf", "reviewer": "Reviewer_ggdQ", "summary": "This paper presents FLSL a self-supervised learning model designed to give good performances of the pre-trained model on downstream dense prediction tasks. The proposed method can be summarised as a two-level clustering problem to achieve this objective :\n\n- Intra-view clustering: which aims to cluster together semantically related embeddings and pull apart those which are not. This is achieved using cluster representative which are obtained using mean-shift clustering. The embeddings are then drawn to their representatives. The paper demonstrated the relation between mean-shift and self-attention which is used of that purpose\n\n- Inter-view clustering: the goal is facilitate the representations of semantically similar features across the dataset to be close. This is done by applying a soft-assignment of the representatives obtained previously to a set centroids.\n\n", "questions": "1. what is the computational cost of the proposed method compared to other SSL frameworks?\n2. Did you evaluate the importance of the different levels of clustering? like how much each participate to the actual benefits on downstream performances. I see some results in Table 5., but it shows the performance with k-NN and always considers the inter-view clustering. What happens when we do not consider the inter-view clustering?\n3. Also noticed that using smaller patches on the backbone in FLSL (e.g. Vit-s/8) seems to be better compared to backbone with larger patches(16), do you know the reasons behind this behaviour? (in table 1 and 3)", "limitations": "no particular limitations which need to be addressed. The main limitations of this work can be the complexity of the method which the authors spoke about and the potentially computational cost of the framework.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The demonstration of the relation between mean-shift and self-attention is a strong contribution of this paper and it plays a key role in the intra-view clustering objective\n2. The proposed method outperforms other SSL frameworks on standard dense benchmarks(ms-coco,  uavdt, davis)\n3. The paper is well written and the authors provided proofs for their claims as well as code and implementation details for reproducibility.", "weaknesses": "1. This paper uses a bbox-aligned k-NN classifier to perform the ablation studies of their work, but we don't know how the performances on this classifier correlates with the results on downstream tasks segmentation or detection.\n2. It would have been good to have further evaluations on semantic segmentation on other datasets such as ADE20k, PascalVOC, cityscapes to get a broader view of the performances of the model", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\\\n__Weaknesses:__\n\n__1.__ _How the performance on bbox-aligned k-NN classifier correlates with dense prediction tasks?_\n\nThanks for raising this question. FLSL learns dense semantic representations rather than a single instance-level representation. Therefore, we design a bbox-aligned k-NN classifier to evaluate the feature quality for hyperparameter tuning and ablation study.  As detailed in Appendix 5, to make this classifier correlates with the results on downstream tasks of detection or segmentation, the bounding box information provided by ImageNet-1K is leveraged during validation. Specifically, to have a higher bbox-aligned k-NN accuracy, the FLSL-learned representations in the region of ground-truth bounding-boxes should be both locally and globally semantic, such that \nthe extracted local features are indeed from the same object in an image while being close to the bbox-aligned features (of the same category) extracted by FLSL in training images for k-NN classification. Our empirical study also shows the effectiveness of this bbox-aligned k-NN classifier for downstream tasks of detection and segmentation. We will clarify this further in Appendix 5.\n\n\\\n__2.__ _It would have been good to have further evaluations on semantic segmentation on other datasets such as ADE20k, PascalVOC, cityscapes to get a broader view of the performances of the model._\n\nThanks for the suggestion. Please see the general rebuttal pdf file, where we provided ADE20k semantic segmentation results of FLSL compared with baseline methods. In line with SelfPatch, all models are fine-tuned with Semantic FPN under the standard 40k iteration schedule. Similar to the results on COCO object segmentation and DAVIS instance segmentation, FLSL consistently outperforms all the baseline methods on ADE20k. We will include this result in the appendix. \n\n\\\n__Questions:__\n\n__1.__ _What is the computational cost of the proposed method compared to other SSL frameworks?_\n\nThe per-epoch training time of FLSL on ViT-S/16 is 1.19x longer than DINO and is on par with SelfPatch, which is 1.21x longer than DINO, under the same model and hardware configuration. We will include this discussion in the main paper.\n\n\\\n__2.__ _Did you evaluate the importance of the different levels of clustering? like how much each participate to the actual benefits on downstream performances. I see some results in Table 5., but it shows the performance with k-NN and always considers the inter-view clustering. What happens when we do not consider the inter-view clustering?_\n\nYes, Table 5 provides the ablation study of the importance of the different levels of clustering. We always consider inter-view clustering (e.g., $\\eta=1$) because it governs the global meaningfulness of the representations. Without considering the inter-view clustering, the 2nd half of the FLSL pipeline in Figure 2 is discarded and the intra-view clustering alone leads to collapse in training and cannot learn semantically meaningful representations. We will clarify this in the caption of Table 5.\n\n\\\n__3.__ _Also noticed that using smaller patches on the backbone in FLSL (e.g. Vit-s/8) seems to be better compared to backbone with larger patches (16), do you know the reasons behind this behaviour? (in Table 1 and 3)_\n\nThis is because smaller patches results in higher resolution of feature maps, which benefits not only dense prediction tasks but also classification (see e.g., [1]).\n\n[1] Caron et al., Emerging properties in self-supervised vision transformers, ICCV 2021."}, {"review_id": "R8je1lUXWY", "reviewer": "Reviewer_rt9u", "summary": "The authors point out the limitations of previous SSL methods on dense prediction tasks, because of tis instance-level objectives. On the other hand, recent studies focusing on  dense prediction based on region, patch, and pixel to learn globally semantic representations on these sub-regions. To this end, the paper introduces a novel method that learns representations with both local and global semantics by leveraging the mean-shift clustering. The proposed method consists of intra-view clustering with mean-shift and inter-view clustering with k-means. The extensive experimental results shows its effectiveness on various dense prediction tasks.\n", "questions": "* Is the proposed method also competitive in terms of throughput?\n* The scales of random resized crop for each networks seem to have high lower bound compared to previous SSL methods. Is there any reason that the lower bound of scales should be relatively high?\n* Does the FLSL also show superiority in the fine-tuning task on ImageNet-1K?", "limitations": "The potential limitations are addressed well.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Well-written paper with superior performances\n* Conducted experiments on various dense prediction tasks\n", "weaknesses": "* Some SSL methods that are strong in dense prediction tasks are omitted from the tables. (e.g., iBOT and RC-MAE)\n    * For example, iBOT with ViT-S/16 outperform the methods in Table 1 in terms of AP^bbox and AP^mask. RC-MAE is also comparable to iBOT.\n* The algorithm table in the appendix seems to have a gap with the equations of the proposed method. Is there some omitted explanation in the algorithm table?\n* It would be better to compare the AAS of the method with other state-of-the-art methods so that the qualitative results can visualize distinctive behaviors.\n\n* Minor\n    * It would be better for the highest values in the tables to be bold.\n\n[1] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. \"ibot: Image bert pre-training with online tokenizer,\" ICLR'22\n\n[2] Youngwan Lee, Jeffrey Willette, Jonghee Kim, Juho Lee, Sung Ju Hwang, \"Exploring The Role of Mean Teachers in Self-supervised Masked Auto-Encoders,\" ICLR'23.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\\\n__Weaknesses:__\n\n__1.__ _Some SSL methods that are strong in dense prediction tasks are omitted from the tables. (e.g., iBOT and RC-MAE). For example, iBOT with ViT-S/16 outperform the methods in Table 1 in terms of $AP^{bbox}$ and $AP^{mask}$. RC-MAE is also comparable to iBOT._\n\nThanks for pointing out these related works. We discussed iBOT in the related work section. We choose ViT-S/* as backbone of FLSL and Mask R-CNN as detector because (1) this benchmark has a lower computational cost, and (2) there are more baselines to compare with. We did not include iBOT results in Table 1 because iBOT employs the **Cascade** Mask R-CNN as detector, which is more complex and expensive than Mask R-CNN, leading to an unfair comparison. RC-MAE only provides the results of ViT-B/16 with Mask R-CNN, and we will include its results in Table 2.\n\n\\\n__2.__ _The algorithm table in the appendix seems to have a gap with the equations of the proposed method. Is there some omitted explanation in the algorithm table?_\n\nA constant $\\log K$ (as a result of a uniform prior $\\pi=1/K$) is omitted in the algorithm table. Specifically, with a uniform prior $\\pi=1/K$, the KL divergence term in the objective function (13) reduces to the entropy of the student prediction plus a constant $\\log K$, the latter of which is omitted in the algorithm table.\n\n\\\n__3.__ _It would be better to compare the AAS of the method with other state-of-the-art methods so that the qualitative results can visualize distinctive behaviors._\n\nThis is an excellent suggestion. We provided a side-by-side comparison of AAS visualization between FLSL and DINO in the general rebuttal pdf file. As we can see, FLSL leads to AAS better aligned with underlying objects and stuff, and captures more objects alongside the label-related object in an image, while DINO tends to single out the label-related tokens and drives the tokens in the rest of an image to be highly correlated. We will add this visualization and discussion to the appendix. \n\n\\\n__4.__ _It would be better for the highest values in the tables to be bold._\n\nWe highlighted the FLSL results in light blue in the tables. As suggested, we will highlight the highest values with boldface for better visibility.\n\n\\\n__Questions:__\n\n__1.__ _Is the proposed method also competitive in terms of throughput?_\n\nThe per-epoch training time of FLSL on ViT-S/16 is 1.19x longer than DINO and is on par with SelfPatch, which is 1.21x longer than DINO, under the same model and hardware configuration. We will include this discussion in the main paper.\n\n\\\n__2.__ _The scales of random resized crop for each network seem to have high lower bound compared to previous SSL methods. Is there any reason that the lower bound of scales should be relatively high?_\n\nWe set a higher lower-bound to include more contextual information in each view to help the model to learn representations at higher semantic level. Specifically, in FLSL, a positive cluster representation is retrieved via mean-shift cluster assignment in the form of cross-attention. Consider a query token from a source local crop that is very small and only contains a shade of a single color and no structured features, the cross-view cluster assignment of that query would result in a cluster of all patches of the similar color in the target view. This restricts the semantic level to colors and hinders the model from learning meaningful representations at higher semantic level that better aligns with objects or stuff. Therefore, a higher lower-bound can provide more contextual information to facilitate FLSL to learn meaningful representations. We will clarify this further in Appendix 4.\n\n\\\n__3.__ _Does the FLSL also show superiority in the fine-tuning task on ImageNet-1K?_\n\nWe did not consider fine-tuning on classification task because FLSL is designed mainly for dense prediction tasks and there is no [class] token involved. As a future work, we will explore ways to extend FLSL for tasks that necessities a global representation while retaining its existing properties."}, {"review_id": "e6KtpdDXjm", "reviewer": "Reviewer_asnD", "summary": "Current self-supervised learning (SSL) methods, including SimCLR, DINO, VICReg, and MOCOv3, focus mainly on instance-level representations, limiting their use in tasks like object detection and segmentation. To overcome this, a new two-level feature clustering SSL method named Feature-Level Self-supervised Learning (FLSL) has been introduced, which uses the mean-shift clustering process of Vision Transformers to improve semantic cluster representations. Experimentally, FLSL outperforms existing SSL methods in dense prediction tasks, delivering impressive results on multiple benchmarks, notably MS-COCO, UAVDT, and DAVIS 2017 datasets.", "questions": "The authors claimed that they demonstrate for the first time the connection between the attention mechanism mean-shift clustering. However, there are already some works that demonstrate the connection between attention and clustering, e.g. [1]. Does this paper bring any new insights?\n\n[1] Zhou D, Yu Z, Xie E, et al. Understanding the robustness in vision transformers[C]//International Conference on Machine Learning. PMLR, 2022: 27378-27394.", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.It introduces the mean-shift and k-means for the pre-training of dense predictions.\n2. The performance is improved with the proposed method.", "weaknesses": "1. The idea of considering similar pixels or patches as positive pairs is not new. Many previous works have explored this idea already, e.g. [1].\n2. Many related works are not compared, e.g. [2, 3]. There are also some other works that I do not mention.\n\n\n\n\n\n\n\n[1] Hénaff O J, Koppula S, Alayrac J B, et al. Efficient visual pretraining with contrastive detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 10086-10096.\n[2] Xie Z, Lin Y, Zhang Z, et al. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 16684-16693.\n[3]  Xiao T, Reed C J, Wang X, et al. Region similarity representation learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 10539-10548.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\\\n__Weaknesses:__\n\n__1.__ _The idea of considering similar pixels or patches as positive pairs is not new. Many previous works have explored this idea already, e.g. [1]._\n\nThanks for bringing [1] to our attention. Yes, FLSL belongs to the family of SSL methods that consider similar pixels or patches as positive pairs. We have discussed in the related work section the representative SSL methods in this area, i.e., SoCo, ORL, PixPro, LC-loss, SelfPatch+DINO, and ADCLR, etc.\n\nThe work in [1] relies on dedicated algorithms (e.g., FH or MCG) to determine the mask of similar pixels/patches, which is analogous to SoCo and ORL that leverage non-trainable _selective search_ algorithms to find the RoIs as positive pairs. In contrast, FLSL is end-to-end trainable. It does not relies on any dedicated non-trainable algorithm for cluster determination. In contrast, FLSL leverages mean-shift clustering as self- and cross-attention to automatically find a positive pair of soft clusters. This accounts for one of our main contributions. \n\nWe will incorporate [1] to the related work section, and clarify our contributions further.\n\n\\\n__2.__ _Many related works are not compared, e.g. [2, 3]. There are also some other works that I do not mention._\n\nThanks for pointing out these related works. Indeed, there are many related works on this topic, and we incorporated the most representative ones in the paper, including SoCo, ORL, PixPro, LC-loss, SelfPatch+DINO, and ADCLR, etc. Paper [2], PixPro, has been discussed in related work. Paper [3] takes a contrastive learning strategy to maximize the similarity of global representations and the sliding-window-pooled representations in the overlapped region of two augmented views. There are several existing works considering this joint consistency on views of image and local patches (e.g., DetCo, which has been discussed in our paper). We will include paper [3] in related work too.\n\n\\\n__Questions:__\n\n__1.__  _The authors claimed that they demonstrate for the 1st time the connection between the attention mechanism and mean-shift clustering. However, there are already some works that demonstrate the connection between attention and clustering, e.g. [1]. Does this paper bring any new insights?_\n\nThanks for pointing out this related work. Paper [1] tries to interpret the attention mechanism from the perspective of information bottleneck (IB), which is further related to clustering. From this perspective, paper [1] is relevant to ours. \n\nHowever, the formulation of IB in paper [1] is essentially an EM-fitting of a GMM with soft assignment under certain major assumptions (i.e., Gaussian approximation in KL minimization and small smoothing scale as shown in their proof provided in the appendix). In contrast with soft GMM, mean-shift clustering is non-parametric (KDE) and poses no priori assumption of the underlying clusters (e.g., unknown number of clusters, no KL minimization involved). This makes mean-shift clustering closer in line with the attention mechanism that does not make much assumption on the input. We will cite paper [1] and include this discussion in related work.\n"}, {"review_id": "0v3BHbHY5D", "reviewer": "Reviewer_Hfeu", "summary": "This paper tackles self-supervised representation learning and primarily forces on learning representations for dense downstream prediction tasks, such as object detection and instance segmentation. To improve upon prior work, the main idea of the paper consists of two parts: (1) the paper leverages the underlying mean shift clustering process via the attention mechanisms of ViTs. In particular, the presented method relies on self-attention and cross-attention layers to achieve intra-level feature clustering. (2) In addition, the presented method takes a k-means perspective to achieve inter-level feature clustering.. The paper demonstrates the effectiveness of the approach via extensive experiments for dense prediction tasks, outperforming prior art for object detection and instance segmentation on MS-COCO.\n", "questions": "How well does pretraining work on non-curated datasets, like the COCO or OpenImages datasets? \nWhat is the additional computational cost compared to DINO?\n", "limitations": "The paper briefly mentions the limitations near the end of the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper makes an interesting observation by connecting the attention mechanism (self-attention and cross-attention layers in particular) to mean shift clustering. This results in a relatively clean implementation (see figure 2 for an overview). \n\n- The overall idea is intuitive:\n1. The feature representations that belong to a certain cluster are close to the cluster representative (center) and pushed away from the representatives of other clusters.\n2. The cluster representatives of the positive areas are pulled together (positives)\nThis is also reflected in the final loss function and can be implemented via a clean implementation. The pseudo-code in the supplementary materials is very insightful.\n\n- The presented approach demonstrates strong performance on multiple downstream tasks and datasets.  MS-COCO is used for object detection and instance segmentation, UAVDT is used for object detection from UAV platforms, and DAVIS is used for video instance segmentation. The proposed method consistently outperforms DINO (a strong baseline) in these cases.\n\n- The paper contains ablations about the loss function and impact of the number of clusters K. The supplementary also includes additional details about the training setups.\n\n- Overall, the paper is well-written\n", "weaknesses": "- A few comparisons or discussions with related works are missing. For instance, DeepCluster [a, b], PCL [c], and CDL [d] show similarities with the presented approach as they also leverage clustering. A comparison with CDL would be  the most interesting as it relies on instance level and group level losses.\n\n- The presented approach is slightly more complex than prior work (DINO). In particular, the presented approach requires additional attention layers.\nAs the architecture differs from conventional methods due to the introduction of the self-attention and cross-attention layers, what is the additional computational cost? This information is currently not present in the paper.\n\n- It’s also not clear how robust the method is towards certain dataset biases (e.g., object-centric datasets or imbalanced datasets). The approach relies on a uniform prior over the clusters, which is valid for the ImageNet dataset. However, prior works have been able to pretrain on uncurated datasets (e.g., SEER [e]). It’s currently not clear if this is also possible with the proposed method. It would be valuable to include experiments for COCO pretraining.\n\n[a] Caron et al., Deep clustering for unsupervised learning of visual features, ECCV 2018.\n\n[b] Caron et al., Unsupervised pre-training of image features on non-curated data, ICCV 2019.\n \n[c] Li et al., Prototypical contrastive learning of unsupervised representations, ICLR 2021.\n\n[d] Wang et al., Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination, CVPR 2021.\n\n[e] Goyal et al., Self-supervised Pretraining of Visual Features in the Wild, 2021.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\\\n__Weaknesses:__\n\n__1.__  _A few comparisons or discussions with related works are missing. For instance, DeepCluster [a, b], PCL [c], and CDL [d] show similarities with the presented approach as they also leverage clustering. A comparison with CDL would be the most interesting as it relies on instance level and group level losses._\n\nThanks for the suggestion. In the early version of this paper, we had a subsection reviewing the related works in deep clustering, including the above-mentioned ones [a,b,c,d]. However, due to page limit and the scope of the paper, we had to omit them from the submitted version. We believe the connection between FLSL and deep clustering is essential, and we will add the deep clustering subsection back to related work.\n\n\\\n__2.__ _The presented approach is slightly more complex than prior work (DINO). In particular, the presented approach requires additional attention layers. As the architecture differs from conventional methods due to the introduction of the self-attention and cross-attention layers, what is the additional computational cost? This information is currently not present in the paper._\n\nThe per-epoch training time of FLSL on ViT-S/16 is 1.19x longer than DINO and is on par with SelfPatch, which is 1.21x longer than DINO, under the same model and hardware configuration. We will include this discussion in the main paper.\n\n\\\n__3.__ _It’s also not clear how robust the method is towards certain dataset biases (e.g., object-centric datasets or imbalanced datasets). The approach relies on a uniform prior over the clusters, which is valid for the ImageNet dataset. However, prior works have been able to pretrain on uncurated datasets (e.g., SEER [e]). It’s currently not clear if this is also possible with the proposed method. It would be valuable to include experiments for COCO pretraining._\n\nThanks for the suggestion. We have the following observations regarding the robustness of FLSL towards dataset biases. \n\n* First, ImageNet at the feature/cluster-level can be viewed as uncurated. ImageNet is curated at the instance-level, i.e., balanced for each class. However, ImageNet is single-labelled, and we do not know what the distribution of \"class\" is when it is on feature/cluster-level or sub-image level, and the number of classes might be way more than 1K. For example, for an image labeled as “hen”, it may contain many objects and stuff alongside a small hen, and some of the objects and stuff in the image may be out of class vocabulary. In this sense, ImageNet can be viewed as an uncurated dataset at the feature/cluster-level (similar to COCO). As shown in the AAS visualization in the general rebuttal pdf file, the inputs are all COCO-like images from ImageNet-1K, FLSL captures non-label-related objects/stuff in the images, which results in more content-aligned AAS, while DINO mostly singles out label-related features and renders the tokens from the rest of the image correlated to each other.\n\n* Second, a uniform prior is a relatively safe choice. For imbalanced dataset, paper [1] shows that a uniform prior is a safer choice as the performance drop as a result of a uniform prior on an imbalanced dataset is much smaller than that of a non-uniform prior (-1.0 vs. -7.2). Thus, for a dataset with agnostic class distribution as in our case, we adopt a uniform prior.\n\nWe will include the discussion above in the appendix. Due to restricted time and limited computational resource, we are not able to provide COCO pretrained results for the moment. We will provide these results in the appendix if time permits.\n\n[1] Assran et al., The hidden uniform cluster prior in self-supervised learning, arXiv preprint, 2022."}, {"review_id": "6R52xArRSx", "reviewer": "Reviewer_taUM", "summary": "This paper proposes Feature-level Self-supervised Learning (FLSL) to handle dense prediction downstream tasks. Specifically, the authors employ the transformer for joint embedding and clustering and construct the objectives from the mean-shift and k-means perspectives. Experiments show that FLSL yields significant improvements in dense prediction tasks. ", "questions": "See weakness.", "limitations": "No clearly visible limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-motivated and the proposed method seems to work well on dense prediction tasks.\n\n2. The analysis of the connection between mean-shift clustering and SA sounds reasonable. The authors analyze the relationship between ViT and clustering from a new perspective.\n\n3. Experimental results on detection and segmentation show significant improvements and demonstrate the effectiveness of FLSL.", "weaknesses": "1. The relation between ADCLR and FLSL is not clear. I see both ADCLR and FLSL use cross-attention to learn patch-level information. So, what's the main difference between ADCLR and the inter-view objective?\n\n2. ''feature-level'' looks somewhat misleading. For me, we typically divide the SSL into ''feature-wise'' (Barlow Twins [1], ZeroCL [2], ARB [3], VICReg [4]) and ''instance-wise'' methods (SimCLR [5], Moco [6]) by the objectives on the different dimension. The proposed method is more like a cluster-level method (SwAV [7]).\n\n[1] Zbontar J, Jing L, Misra I, et al. Barlow twins: Self-supervised learning via redundancy reduction[C]//International Conference on Machine Learning. PMLR, 2021: 12310-12320.\n\n[2] Zhang S, Zhu F, Yan J, et al. Zero-cl: Instance and feature decorrelation for negative-free symmetric contrastive learning[C]//International Conference on Learning Representations. 2021.\n\n[3] Zhang S, Qiu L, Zhu F, et al. Align representations with base: A new approach to self-supervised learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 16600-16609.\n\n[4] Bardes A, Ponce J, LeCun Y. Vicreg: Variance-invariance-covariance regularization for self-supervised learning[J]. arXiv preprint arXiv:2105.04906, 2021.\n\n[5] Chen T, Kornblith S, Norouzi M, et al. A simple framework for contrastive learning of visual representations[C]//International conference on machine learning. PMLR, 2020: 1597-1607.\n\n[6] He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 9729-9738.\n\n[7]  Caron M, Misra I, Mairal J, et al. Unsupervised learning of visual features by contrasting cluster assignments[J]. Advances in neural information processing systems, 2020, 33: 9912-9924.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\\\n__Weaknesses:__\n\n__1.__  _The relation between ADCLR and FLSL is not clear. I see both ADCLR and FLSL use cross-attention to learn patch-level information. So, what's the main difference between ADCLR and the inter-view objective?_\n\nThanks for raising this question. The main difference between ADCLR and the FLSL inter-view objective is that ADCLR leverages a specially designed cross-attention (CA) between the pseudo CLS tokens (constructed from local crops) and patch tokens to retain the local information throughout the transformer, while FLSL inter-view objective encourages the consistent representations of positive clusters automatically determined via CA between tokens from student and teacher views. More details of the usage of CA in FLSL and ADCLR are provided below.\n\n* The CA in ADCLR occurs between \"pseudo\" CLS (pCLS) tokens and patch tokens. The \"pseudo\" CLS tokens are constructed with the instance-level representations of several small local crops via a dedicated projector, while the CA in FLSL occurs between the output tokens of student ViT and teacher ViT.\n\n*  The CA in ADCLR is uni-directional, i.e., from one pCLS token to patch tokens plus the pCLS itself, while the attention function of patch tokens does not consider pCLS tokens at all. In addition, there is no interaction among different pCLS tokens, whereas the CA in FLSL follows the common CA definition.\n\n*  The CA in ADCLR occurs at every layer in ViT, while the CA in FLSL only occurs at the end of the two ViTs.\n\nWe will include this discussion in the appendix.\n\n\\\n__2.__ _\"feature-level\" looks somewhat misleading. For me, we typically divide the SSL into \"feature-wise\" (Barlow Twins [1], ZeroCL [2], ARB [3], VICReg [4]) and \"instance-wise\" methods (SimCLR [5], Moco [6]) by the objectives on the different dimension. The proposed method is more like a cluster-level method (SwAV [7])._\n\nThanks for raising this question. The \"level\" in our paper refers to the semantic level, at which SSL operates. In this sense, all the above-mentioned methods can be categorized as instance-level since those methods learn meaningful representation for a whole image. FLSL, on the other hand, learns a semantically meaningful representation of a cluster of features, which is determined via mean-shift clustering at the patch/feature-level. Hence, we coined FLSL for our proposed method. We will clarify this further in the introduction section.\n"}]

Author rebuttal:
Please find the attached pdf file that includes (Figure 1) AAS visual comparison between FLSL and DINO, and (Table 1) ADE20K semantic segmentation results.

Meta-review: This paper proposes a novel self-supervised learning algorithm geared towards dense prediction vision tasks. It does that by introducing a mechanism to learn representations with both local and global semantics.
The paper received generally positive scores, with all reviewers recommending acceptance and praising the clarity of the presentation, the clear motivation, the innovative aspect of the proposed method, and the extensive experimental validation, which demonstrate significant improvement on detection and segmentation tasks over competing baselines.
Decision: Accept (poster)