Mind Your Augmentation: The Key to Decoupling Dense Self-Supervised Learning
Authors: Congpei Qiu
Tong Zhang
Yanhao Wu
Wei Ke
Mathieu Salzmann
Sabine Süsstrunk
Venue: ICLR 2024 poster (2024)
Keywords: Dense Self-supervised Learning
representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: we are providing novel and practical solution for enhancing Dense SSL, offering valuable insights into disentangled feature representation within the realm of self-supervised learning.

Abstract: Dense Self-Supervised Learning (SSL) creates positive pairs by building positive paired regions or points, thereby aiming to preserve local features, for example of individual objects. However, existing approaches tend to couple objects by leaking information from the neighboring contextual regions when the pairs have a limited overlap. In this paper, we first quantitatively identify and confirm the existence of such a coupling phenomenon. We then address it by developing a remarkably simple yet highly effective solution comprising a novel augmentation method, Region Collaborative Cutout (RCC), and a corresponding decoupling branch. Importantly, our design is versatile and can be seamlessly integrated into existing SSL frameworks, whether based on Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs). We conduct extensive experiments, incorporating our solution into two CNN-based and two ViT-based methods, with results confirming the effectiveness of our approach. Moreover, we provide empirical evidence that our method significantly contributes to the disentanglement of feature representations among objects, both in quantitative and qualitative terms.

Published as a conference paper at ICLR 2024
MIND YOUR AUGMENTATION: THE KEY TO DECOU- PLING DENSE SELF-SUPERVISED LEARNING
Congpei Qiu Tong Zhang" Yanhao Wu' Wei Kelf Mathieu Salzmann" Sabine Süsstrunk
"School Software Engineering, Xi'an an Jiaotong University, China School of Computer and Communication Sciences, EPFL. Switzerland
ABSTRACT
Dense Suff-Superviee Learning (SSL) creates positive pairs by building positive paired regions points, thereby aiming preserve local features, for example of individual objects. However, existing approaches tend to couple objects by leaking information from the neighboring contextual regions when the pairs have limited overlap. In this paper, we first quantitatively identify and confirm the existence of such a coupling phenomenon. We then address it by developing remarkably simple yet highly effective solution comprising novel augmentation method, Re- gion Collaborative Cutout (RCC), and a corresponding decoupling branch. Impor- tantly, our design is versatile and can be seamlessly integrated into existing SSL frameworks, whether based on Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs). We conduct extensive experiments, incorporating tion into two CNN-based and two ViT-based methods, with results confirming the effectiveness of our approach. Moreover, we provide empirical evidence that method significantly contributes to the disentanglement feature representations among objects, both in quantitative and qualitative terms.
solu-
INTRODUCTION
Self-supervised learning (SSL) has achieved significant progress, now surpassing traditional super- vised pre-training when applied to diverse domains and tasks (Chaitanya et al., 2020; Ericsson al., 2021; Caron et al., 2021; Wu et al., 2023). SSL methods aim to maximize the similarity between the feature representations of positive pairs. However, positive pairs are built with strong augmentation and their features are obtained by average pooling for each view, leading to inconsistent semantic alignment (Zhang et al., 2022) and a lack of localiztiion information. Consequently, this approach is most suitable for well-curated datasets and becomes problematic when applied to images containing multiple objects or tasks that necessitate precise location information. To alleviate the constraints imposed by SSL, recent literature has witnessed substantial efforts in the domain of dense-level SSL. At the core of these efforts is the idea of defining positive pairs by identifying matching locations, thereby preserving distinct semantic cues.
In the context of dense SSL,, cutout and masking are pivotal components for both CNNs and ViTs, as they contribute to the creation of meaningful positive pairs. Point-level CNN-based meth- ods (Wang et al., 2021; Xie et al., 2021) construct positive pairs by selecting matched points in feature space, while others, including both region-level CNN-based (Wei al., 2021) and ViT-based methods (Zhou et al., 2022; Xue et al., 2023), rely on image coordinates oracle and employ vari- ous augmentations such as cutout and masks to generate positive pairs. These pairs are subsequently utilized in teacher-student networks for feature alignment. However, a challenge arises when there is limited overlap between the views, often due to mismatches or over-cuts. In such cases, the features extracted from the query view tend to take shorteuts by leveraging information from the background to minimize the self-supervised learning (SSL) loss. This unintentional mechanism leads to closer proximity of features from unrelated regions, ultimately resulting in less discriminative regions be- ing dominated by their neighbors. We term this phenomenon coupling, as depicted in Fig. 1.
In this paper, we present simple yet highly effective strategy that seamlessly integrates into recent dense SSL methods to address the object coupling issue. We observe that existing augmentation
These authors contributed equally to this work. $ Corresponding author Code available at hutp://gihuh..co//tt wtt.://////1112///ee//55523 0244/enseSSS.

---

Published as a conference paper at ICLR 2024
PLRC
Augmentation
Cutout Region
Danse SSL loss 5%
Coupling area
Features from p
Object area
Information Leakage
CNN-based
ViT-based
(a) Illustration of coupling
(b) Point affinity visualizations
Figure 1: Visualization of Coupling Effects. (a) When positive pairs are mismatched or share small overlapping unmasked regions, the features in the query z" are pushed towards the key features z", causing the key feature's 's semantics to propagate
wider query semantic area (less discriminative). (b) The visualization depicts affinity maps generated by different methods, with each map showing feature similarity between a single point (marked by red dot) and the rest of the image. '-D' denotes model combined with our decoupling strategy.
techniques, such as blockwise masks (Zhou et al., 2022) and cutout (DeVries & Taylor, 2017), ap- ply masks or cutouts independently to the input view, without taking into account the presence of other masked or cut regions. This prevents the student views from providing sufficient semantic cues to align with the teacher encoders, especially when substantial occlusions occur, e.g., due to the overlap of multiple masked or cut regions. To address this challenge, we introduce Region Col- laborative Cutout (RCC), a method designed to avoid creating excessively large masked areas that could lead to information leakage from distant regions. Our approach involves creating regions uni- formly and implementing cutouts within these regions, starting with larger ones and progressively moving to smaller areas. The cutout operations among regions work collaboratively to ensure that no large masks exceed a specified threshold. More importantly, we introduce a de-coupling branch to eliminate incorrect dependencies between each region and its surroundings by adding contextual noise the surroundings of sampled regions within images. Specifically, we construct de-coupled views by replacing the background regions of key views defined by RCC masks with randomly sampled background images. We then directly apply the self-supervised learning loss to the features extracted from the same foreground region in both the decoupled and key views. This design allows the network to learn independently from each local region, promoting the capture robust repre- sentations less influenced by irrelevant context. Importantly, our de-coupling branch is compatible with existing methods and requires only modification the augmentation pipeline.
In nutshell, our contributions can be outlined follows:
1. We tackle the challenge of coupling in dense SSL, offering quantitative empirical evidence to validate its presence across different methods with both CNNs and ViTs.
2. We introduce novel augmentation method tailored for dense SSL, named Region Col- laborative Cutout (RCC). This method plays pivotal role in redueing object coupling by effectively reducing the difficulty of aligning features in masked regions.
3. Building upon the foundation of our new augmentation method, we further enhance feature robustness to noise by introducing a decoupling branch. This branch ensures that features are learned independently from their surroundings.
We showcase our contributions through comprehensive experiments and ablation studies, providing empirical validation for the effectiveness of our entire method as well as its individual components, namely RCC and the decoupling strategy.
RELATED WORK
Self-supervised Learning. With the introduction of InfoNCE (Oord et al., 2018), inspired by the triplet loss in metric learning (Oh Song et al., 2016; Sohn, 2016), contrastive learning has become
2

---

Published as a conference paper at ICLR 2024
the dominant SSL strategy. SimCLR (Chen et al., 2020a) constructs positive pairs by applying random data augmentations twice to the same image and treats other images as negatives. MoCo (He et al., 2020) explores simpler architecture, and BYOL (Grill et al., 2020) starts the trend of simply minimizing positive pairs' distance. Other explorations include clustering- seaeedd (Asano et al., 2019; Caron al., 2018; 2020; Ji et al., 2017; Zhang et al., 201 9b;a) and feature decorrelation (Zhontar et al., 2021). These methods primarily focus on capturing semantic invariance through image-level augmentation. Consequently, they are less suitable for images with multiple objects (He al., 2019; Zhang et al., 2022) and dense prediction tasks, where local context becomes crucial.
Dense Self-supervised Learning. Dense SSL methods aim to enhance the representation of regions by constructing positive pairs with finer granularity. Notably, the point-level methods, including DenseCL. (Wang et al., 2021), PixPro (Xie et al., 2021), and VADeR (O Pinheiro et al., 2020), establish positive point pairs by evaluating pairwise cosine distances among all points from two augmented views and selecting those with higher similarity. However, these methods focus solely on point similarity and do not consider the relationships with their neighbors, leading to inaccurate pairings. response, SetSim (Wang et al., 2022b) and Point-level RCL (Bai et al., 2022) seek to refine the selection of positive pairs by taking neighboring points into account. Nonetheless, their strategies heavily rely on image-level SSL, resulting in mismatches in the selection of positive pairs.
On the other hand, region-level methods focus on identifying matching regions and applying SSL loss to features extracted from these regions. Self-EMI (Liu et al., 2020) minimizes the Earth mover's distance between two paired regions; InsLoc (Yang et al., 2021) utilizes pretext tasks to facilitate region-level contrastive learning; DetCon (Hénafff et al., 2021) and SoCo (Wei et al., 2021) employ unsupervised masks and bounding boxes, respectively, as external labor-free supervision to build positive pairs. These region-based networks share a similarity with image-level SSL, where only the most discriminative parts can be learned due to average pooling. In the realm of ViT structures, recent works use masked image modeling (MIM) as augmentations to reconstruct masked patches in color space (Xie et al., 2022; He et al., 2022) or the features of masked patches (Bao et al., 2022; Xue et al., 2023). Among these approaches, iBOT (Zhou et al., 2022) serves as the core component of the vision foundation model DINOv2 (Oquab et al., 2023). Despite their impressive results, prior Dense SSL methods have overlooked the issue of feature coupling.
Augmentations. Most of the augmentations, e.g., cutout (DeVries & Taylor, 2017), gaussian blur, and random crops, are designed for supervised image classification tasks. Some alternatives, in- cluding mixup (Zhang et al., 2017), cutmix (Yun et al., 2019), and TokenMix(Liu et al., 2022), are also effective for supervised semantic segmentation and object detection. SimCLR (Chen et al., 2020b) was the first to adopt augmentation in SSL,, evidencing that SSL benefits from stronger aug- mentation. Recently, SDMP (Ren al., 2022) integrates the mixup method into SSL. frameworks. Additionally, patch-based augmentations (Qin et al., 2022) (rotations, infill, and shuffle) have been proposed for ViT's pre-training robustness against noise. However, these augmentation methods are tailored to enhance image-level discriminative features for classification tasks, often overlooking the essential localization information due to its inherently complex nature.
While dense SSL has emerged as a promising direction, particularly replacing image-level ap- proaches, many challenges associated with feature coupling in Dense SSL. remain unaddressed, particularly in scenarios involving images with multiple objects. We aim to tackle this problem by introducing novel augmentations and network modules specifically designed for Dense SSL.
MOTIVATION: EMPIRICAL STUDY
We initiate our work with an empirical analysis that aims to shed quantitative light on the is- sue of coupling in dense SSL. Let X E ##xxx3 denote sample image from a pre-training dataset D, P is the region where the feature is extracted from, and pC is the surrounding region providing contextual information. Note that each region may contain either a single or multiple patches. Thus, the set of image patches corresponding to P is represented by Xp. With an encoder E : R x3 - h the feature representation of P is computed as f(X, P) = g(E().,P) where g denotes feature extraction from regions. For an augmented query view X and key view x we formulate dense SSL as
Diess (27,, =bx) LESL [X( P.): ,P,)].
3

---

Published as a conference paper at ICLR 2024
Z
cos()
Coupling Radio
iBot-D
Leopart
1.5
Leopart-D
MaskAlign
MaskAlign-D
25%
X
XB
X
0.9
0%
25%
50%
75%
Information dropping pipaline
Mask Radio
(a) Inter-object coupling measuring pipeline
(b) ViTs- methods
Figure 2: Measuring inter-object coupling in viTs. (a) Random masking is applied to image X with varying ratios, combined with an image X B to create image XAB Yellow boxes highlight regions from which features are extracted. (b) We report the CR across different mask ratios for ViT-based models. The -D indicates the method integrated with our decoupling strategy.
where fa denote the teacher and student branch. CSst, is any existing SSL loss such InfoNCE and fg. The primary objective of this framework is to facilitate the learning of semantically invariant features, ensuring that the features extracted from region Pa closely correspond those from Pk- However, this objective also necessitates the retrieval of semantic information from the context PC when region Pg lacks sufficient semantic cues to match region Pk.
Our intuition for measuring coupling is to examine the level of independence exhibited by local features when exposed to varying contexts. To better investigate the coupling issue, we showcase coupling in both inter-object and intra-object scenarios. Note that image-level SSL is obtained by replacing the region extractor with average pooling over the entire input view. In this case, there is risk of information leakage through spatial shifts.
3.1 INTER-OJJECT COUPLING
We first investigate the inter-object case as depicted in Fig. 2(a), where we consider pairs of objects (A and B) belonging to different categories. Specifically, we construct three views: AB con- taining both A and B, and X and X containing individual object. To evaluate the extent of entanglement between two objects, we randomly remove patches of object A with different ratios in the image XAB and extract their features through backbones pre-trained with different methods:
ZA f( PA.), zA, f(X-AB PAz), zB F(XBPPE
where PA, As: PB the regions containing A,, Ag and B. We then define the coupling ratio as maxx 0(2Az zB),c)
CR
(3)
max
0(ZA,, =B),e)
where -(.,)) is the angle between two feature vectors, and is scalar introduced for numerical stability. CR indicates information leakage from neighbor X to XAB Our findings are presented in Figure 2(b), which illustrates the results obtained from different pre-trained models. Specifically, when region A loses more semantic information, the feature AA, tends to converge towards zB, providing clear evidence of the existence of inter-object shortcuts. As expected, in the case image-level methods, CR tends to approach 1, as no context regions are involved during the pre-training stage. The same trends be found in CNNs, and more details can be found in Appendix B.
3.2 INTRA-OBJECT COUPLING
Apart from the entanglement between objects, coupling within individual objects is also prevalent. We begin by sampling ground-truth object mask denoted as M along with its image X. From this mask, we select a quarter to represent the object part, denoted as Mop. We then remove the remaining portions of the object mask, resulting in modified image X XOP and construct X by replacing the rest parts with a randomly sampled background, as shown in Fig 3(a). To extract region-level features of the object part, we perform masked average pooling using Mop the
4

---

Published as a conference paper at ICLR 2024
OP XMapp
CR
M op
iBOT
1BOT-D
Masked Avg-pool
M op
Coupling Ratio
0.2
DenseCL-O DenseCL
XOP
X
0.04
0.59
0.34
0.40
0.05
0.80
Coupling
(a) Intra Intra-object coupling measuring pipeline
(b) Histogram of coupling ratio
Figure 3: Measuring intra-object coupling. (a) We select quarter of the object Mop as object part. The views X _ are constructed by masking out the remaining object and replacing with sampled backgroud. The CR is calculated between region-level features extracted using masked average pooling. (b) We draw the CR histograms of the pre-trained models.
feature map obtained by forwarding the constructed view through the backbone network E. We define the coupling ratio within objects as:
OP CR cos (2 M
A higher value of CR indicates that the features of the object part are more inclined to the influence of surroundings. Figure 3(b) displays histograms of the computed coupling ratios for the sampled object parts, underscoring how irrelevant background significantly affects representations of object parts. The mean intra-object CR clearly decreases with our strategy.
As demonstrated by the above experiments, the region in lack of semantics due to stronger mask- ing tends more to retrieve information from the context. Consequently, we further verify that the coupling will gradually increase the augmented views become more challenging in Appendix C.I.
DE-COUPLING FROM DENSE-LEVEL SHORTCUTS
Our empirical studies have brought to light the existence of dense-level shorteuts that impede the models from effectively capturing local visual semantics. this section, we introduce our de- coupling strategy solution mitigate these challenges. Importantly, our de-coupling strategy is designed for general applicability, and we will show how to integrate our method into existing dense-level pre-training methods.
4.1 DE-COUPLING STRATEGY
As discussed in Section 3, strong augmentations with higher masking ratios force the student net- works search for shortcuts to align the features with those from the teacher branch. To counter this, our strategy starts with proposing new augmentation method, namely Region Collaborative Cutout (RCC), to generate views with reduced difficulty levels to block the shortcut to contextual in- formation. Moreover, introduce mixture of cropped views with various backgrounds to mitigate the bias from the contextual information single image.
4.1.1 DECOUPLED VIEWS GENERATION
To create the decoupled view, we leverage RCC to generate masks for extracting foreground in- formation from the key view while filling the unmasked region with randomly sampled image as background context information. Without loss of generality, we first divide the input into N N grids and randomly create single bounding box with a pre-determined range of scale and aspect ratio within each grid as regions, yielding (b; ) N regions with diverse scales and positioned uni- formly across the image. Each box is then used as anchor collaboratively perform cutout to obtain the mask M Unlike blockwise mask (Zhou et al., 2022) and cutout proposal (Wei al., 2021), 1), where undesirable large masks or cuts can be attained, RCC initiates mask generation in a descending order based on the bounding box region size, simultaneously addressing overeut regions caused by larger ones. For the i-th step, we calculate the actual cutout ratio r.i incurred by the pre- vious steps. If it is higher than pre-defined r., we will restore the region and perform new random
5

---

Published as a conference paper at ICLR 2024
RCC-based Augmentation Pipeline
Mask based Dense SSL
x grids
RCC
(3
Masked Patches
Region
XM
(EMA)
X
XD
De-coupling
X
RCC
Background
RCC
Encoder
Feature Extraction
= Dense Loss
De-coupling Loss
Figure 4: Structure of RCC and decoupling strategy. Our RCC is applied in the augmentation stage to replace previous masking methods. Given input image X with generated proposal gions, we replace the original mask strategy with our RCC, leading to masked view M for student encoder. We construct the de-coupled view X with X and randomly sampled X student branch. {P:} denotes the regions that we used to constrcut positive pairs, which can be ei- ther single patches regions. Note that, the decoupling does not reconstruct the background in de-coupling branch.
additional
cutour with ratio re- Otherwise, another cutout region with ratio r. Tei will be applied. We defer the pseudo-code Alg.
The MRC masks provide effective augmentation for dense SSL when pre-trained on multi-object datasets. Fig. 5 shows the difference between the Cutout and our RCC. Our method prevents masks of different regions from being con- nected in the same place, which blocks the semantic infor- mation for the student encoder. Comparison with blockwise masking and additional results can be found in Appendix D.
(a) Cutout
(b) RCC
Figure 5: Cutout vs. RCC. Masks
We then generate the de-coupled view X of an image X with a randomly sampled background image X and the binary matrix MRCC of M
are generated for each bounding box.
X
X (I MaCc) + X @ Macc.
(5)
4.1.2 DE-COUPLING Loss
In the de-coupled view X", each region is surrounded by the remaining unmasked context and con- text noise controlled by MRcc. For a feature extraction region that has overlap with unmasked region M we directly align the dense-level feature defined by Pr PrMece RCC to the corresponding one on original view X. This can be expressed as
=DC= Lss with =D = (CIIP,PPMMCO). Nf = f((,P))
(6)
Note that P can be bounding box, single patch token in ViTs, point in CNNs. Then, we apply the SSL. loss to each feature =I is tasked with distinguishing between incomplete yet relevant context and introduced noise while predicting 2 with a global context. This both de-couples the model from entangled irrelevant contexts and enhances the out-of-context learning capacity.
achieve de-coupling. With the above objective, the dense-level
4.2 COMBINATION WITH DENSE-LEVEL SSL
Our decoupling strategy seamlessly integrates into existing dense-level frameworks, as illustrated in Fig. 4. In addition the query view X and key view X we generate a decoupled view X using the key view X and masks Macc generated by RCC. It's important to note that the background image B is randomly selected from each batch. Given the set of feature extraction positions P, as positive pairs, the dense SSL objective with decoupling is expressed
DDense-Dl D ( x xD) =
1 Ane [(P.)
Σ
Dense 200
A =xe))
Apc Pol
Σ
Loolspy, x/p)), (7)
PfEPD

---

Published as a conference paper at ICLR 2024
where ADC represents the de-coupling weight, Pp is defined as the set of feature extraction positions that are covered by Pj and intersect with the foreground Mcce The notation denotes the cardinality of the set, which can correspond to either the number of patches (in the context of point- level methods) or regions (for region-level methods). LD applies the same type of loss as De and thus requires further modification. More details and examples of combining our strategy with other dense SSL frameworks are provided in Appendix E.I.
5
EXPERIMENTS
5.1 IMPLEMENTATION DETAILS
De-coupling Setting. To demonstrate the effectiveness and generalization capability of our de- coupling strategy, we apply our module DenseCL. (Wang al., 2021), SoCo (Wei et al., 2021), Leopart (Ziegler & Asano, 2022), iBOT (Zhou et al., 2022) and MaskAlign (Xue et al., 2023), which include both CNN- and ViT-based methods. With the exception of SoCo, which utilizes Selee- tive Search to generate proposals as region prior, we apply 3 3 grids to randomly generate regions for the other methods. For the backbone, we employ ResNet-50 (He et al., 2016) on CNN-based methods and ViT-S/16 (Dosovitskiy et al., 2021) on Vit-based methods. We defer the evaluation of aims for unsupervised segmentation and has different evaluation metrics.
Leopart Appendix
Pre-training Setting, In the pre-training stage, we sample only 25% images from batch to con- struet the de-coupling branch. Note that the de-coupled views will only forwarded to the student encoder, with acceptable additional computational costs (reported in Appendix F). As we target SSL on the mojti- datasets, following the protocol of (Bai et al., 2022; Wang et al., 2021), we pre-train each model on COCO train2017 for 800 epochs. For fair comparison, adopt the same rrper-parameter for every method with or without the de-coupling strategy.
5.2 DENSE PREDICTION
Evaluation Protocols. For CNN-based methods, we follow the dense prediction protocols of MoCo-v2. On VOC (Everingham et al., 2010) detection tasks, we fine-tune Faster R-CNN struc- ture Ren et al. (2015), whereas on COCO (Lin et al., 2014) detection and instance segmentation, we adopt Mask R-CNN (He et al., 2017) structure. For ViT-based methods, we follow the col as iBOT, where we apply Cascade Mask R-CNN (Cai & Vasconcelos, 2019) for object detection and instance segmentation on COCO. For semantic segmentation ADE20K (Zhou et al., 2017), we employ the task layer in UperNet (Xiao et al., 2018) 8) and fine-tune the whole network.
proto-
Table 1: Main results pre-training with CNN-based methods. '-D' denotes combining with de- coupling strategy. We pre-train models marked by with their official codes and setting on COCO.
VOC Det.
COCO Det.
COCO ISeg.
Method
AP
AP50
AP75
AP
AP
AP75
AP
AP
AP75
MoCo v2t
54.6
81.0
60.4
37.8
57.4
41.0
32.9
54.1
35.2
ReSimt
56.6
81.7
63.5
38.3
57.8
41.4
33.5
54.4
35.6
DenseCL.
56.7
81.7
63.0
38.5
58.1
41.5
33.6
54.8
35.7
DenseCL-D
57.2
82.2
63.7
39.3
58.7
42.6
34.2
55.7
36.5
PLRC
57.1
82.1
63.8
39.8
59.6
43.7
35.9
56.9
38.6
SoCot
56.8
81.7
63.5
38.5
57.9
41.5
33.4
54.6
35.4
SoCo-D
57.8
82.5
64.4
40.3
60.1
44.0
35.1
56.9
37.6
Results of CNN-based Methods. Tab. 1 reports the performance on CNN-based methods. We com- pare three types of methods according to feature extraction strategy: 1) Image-level, i.e., directly perform average pooling on the whole feature map, like MoCo v2 (Chen al., 2020b); 2) Point- level, i.e., every feature vector as positive pairs are used for SSL, including DenseCL (Wang et al., 2021) and PLRC (Bai et al., 2022); 3) Region-level, which uses set of features from the feature map followed by average pooling, such as ReSim (Xiao et al., 2021) and SoCo (Wei et al., 2021). Considering the substantial memory requirements of PLRC, demonstrate the integration of

---

Published as a conference paper at ICLR 2024
method into DenseCL representative point-level approach. Additionally, we select SoCo representative region-level method for evaluation. It's important to note that all dense SSL methods consistently outperform image-leve SSL methods in downstream tasks. Particularly noteworthy is the impressive performance improvement achieved by both DenseCI. and SoCo when incorporating de-coupling strategy, surpassing their original results.
Results of ViT-based Methods, Tab.2, we present a comparison between dense SSL approaches utilizing ViTs, both with and without our de-coupling strategy. The results show significant im- provements when employing our strategy. Specifically, iBOT with our de-coupling achieves signif- icant performance gains, surpassing its original baseline by large margin of 2.8 AP, 2.1 AP, and 1.7 mloU across three tasks. Similarly, MaskAlign-D also demonstrates remarkable performance enhancements compared its original version. These comparisons underscore the efficacy of our de-coupling strategy in aiding dense SSL methods to more effectively capture dense semantics while mitigating the issues associated with coupling. Importantly, our approach exhibits versatility, as can be readily applied to various SSL strategies and backbone architectures.
Table 2: Main results pre-training with ViT-based methods. 'D'' denotes combining with de- coupling strategy.
COCO Det.
COCO ISeg.
ADE Seg.
Method
AP
Ao
50
AP75
AP
AP 50
AP75
mloU
iBOT
42.3
61.2
45.6
37.0
58.3
39.4
39.9
iBOT-D
45.1
64.3
48.7
39.1
61.2
41.7
41.6
MaskAlign
45.6
65.2
49.7
39.6
62.0
42.4
43.7
MaskAlign-D
46.7
66.4
50.5
40.5
63.2
43.5
44.3
5.3 OBJECT-LEVEL. KNN
We appreciate the simplicity and effectiveness of k-NN, which directly reflect the quality of learned image-level representations, so we extend it to dense-level tasks and obtain object-level k-NN (OKNN). Concretely, we extract object-level features from the final feature map with ground-truth bounding boxes using RolAlign and average pooling. We extract N object features image from the training set along with corresponding labels. Similar to image-level A-NN, we predict the label for each object in the evaluation set by finding the k-nearest object-level features in the training set.
Disturbed OKNN. To quantify the influence of cou- pling for multi-object images, we further conduct dis- turbed OKNN (OKNN-D) by replacing the region of each image uncovered by N selected bounding boxes with randomly sampled background images. The feature extraction and prediction protocols are kept the same.
Table 3: Main Results of OKNN and OKNN-D. We evaluate models on COCO and report the top-1, top-5 accu- racy.
OKNN
OKNN-D
Method
Top.1
Top.5
Top.1
Top.5
Comparison. Tab. 3 shows the O-KNN and OKNN-
DenseCL
72.5
89.6
41.5
59.9
D accuracy on COCO using Irain2017 for feature ex- traction and val2017 for evaluation. The proposed de-
DenseCL-D
74.5
91.2
45.1
63,9
SoCo
89.7
43.7
coupling strategy boosts the performance of both OKNN and OKNN-D when combined with other dense SSL methods. The improvement on OKNN-D is more ob- vious, demonstrating the effectiveness of our approach to improving model robustness for irrelevant contexts.
SoCo-D
75.8
90.8
47.8
65.0
iBOT
75.2
90.8
45.8
68.3
iBOT-D
93.3
52.1
71.9
MaskAlign
77.4
91.6
53.8
74.0
MaskAlign-D
79.0
92.7
55.2
76.1
ABLATION STUDY
We dissect our de-coupling strategy and study the impact of each to reveal the strengths of our designs here. In ablation studies, we pre- pre-trained iBOT and DenseCL on COCO for 200 epochs. More details and analysis of our RCCand ablation results of DensCL are in Appendix D and G.
6.1 MASK-TYPE FOR DE-COUPLING
8

---

Published as a conference paper at ICLR 2024
To demonstrate the effectiveness of our RCC, we conducted experiments utilizing various types of masking for lnn-oopplinggicluudnnn: a) Rec: Uti- lizing a random rectangular region as the fore- ground; b) Da Randomly mask out patches; c) Bie Employing blockwise masks like BEiT (Bao et al., 2022); d) DCCutouti Replacing our RCC with Cutout. In Tab. 4, we report the re- sults, including the Top-1 accuracy of OKNN, AP in COCO detection, and the average coupling ra- tio for both inter-object and intra-object levels, as introduced in Section 3. Our method, when combined with all of these masking techniques, demon- strates substantial improvements in de-coupling, as evidenced by the enhanced detection and OKNN accuracy. This provides strong validation for the effectiveness of our loss design. Notably, the orig- inal Cutout method, which tends to overcut, leads to a degradation in recognition and classification performance. Among all the augmentation techniques tested, DC consistently achieves the best performance across all metrics, further affirming the overall effectiveness of our strategy.
Table 4: Comparison of mask types for de-
coupling. iBOT is used bascline, we only change the masking strategy for de-coupling.
Method
coco Det.
OKNN
CRinter
CRiatr
38.7
69.9
0.17
+DC
39.3
69.3
1.32
0.15
+DC
39.9
70.9
1.25
0.10
DCClle
41.1
72.9
1.28
0.09
DCCuute
39.2
67.7
1.26
0.10
DCCRC
42.0
74.1
1.16
0.04
6.2 Loss COMPARISON
In Table 5, we conduct a comparative analysis of de-coupling loss with the objectives employed in InsLoc (Yang et al., 2021) and CP2 (Wang et al., 2022a). These methods also utilize the copy-paste strategy with rectangular mask to define the fore- ground region. The primary distinction among these approaches lies in their strategies for aligning positive pairs during the pre-training stage. Specif-
Table 5: Comparison with other copy-paste objectives. iBOT is used as baseline.
Method
coco Det.
OKNN
C'rntte
CR
iBOT
38.7
69.9
1.80
0.17
+DCAu
38.0
69.0
1.51
0.36
+DG,
37.2
66.7
1.48
+DC,-p
39.3
69.3
1.32
0.15
ically, InsLoc employs average pooling on the foreground region, denoted as DCaeg. CP² matches each point feature from the query view to all the foreground point features from the key view, termed as DCp-o (point-to-all). In contrast, our approach simply matches features with the same position, denoted as DCp-p (point-to-point) It's important to note that we use rectangular masks for all types of loss for fairnsss in this comparison. Both DCavg and DCp-o result in inferior intra-object coupling and a decrease in performance on downstream tasks. We attribute this to the observation that dense-level SSL. benefits from fine-grained and accurate positive pair matching, rather than em- ploying a global or coarse-graines matching strategy. Our point-to-poing approach aligns better with the nature of dense SSL, leading improved coupling and performance.
6.3 AUGMENTATION APPLYING RCC TO MASK-BASER SSL
To demonstrate the effectiveness of RCC as an augmentation for both CNNs and ViTs, we se- lect SoCo which utilizes proposal-level Cutout, and iBOT which utilizes Blockwise mask, and re- place their original mask strategy with RCC. Each model is pre-trained on COCO for 800 epochs without the de-coupling branch. As shown Tab. 6, all models using RCC outperform those
Table 6: Comparison of mask types for dense SSL. We replace existing mask strategies of SoCo and iBOT with RCC, denoting as '-RC'..
Method
COCO Det.
COCO ISog.
OKNN
CRiiter
SoCo
38.5
33.4
73.5
1.15
SoCo-RCC
38.9
33.9
74.4
1.10
iBOT
42.3
37.0
75.2
1.43
IBOT-RCC
43.1
37.6
76.8
1.35
employing the original strategy, with a lower coupling ratio, verifying the strength of RCC.
7 CONCLUSION
In this paper, we have dedicately designed experiments that have unveiled the dual impact of aug- mentation, both as a tool for enhancing feature representations and as a cause of feature coupling. To tackle this challenge, we introduced RCC, novel augmentation technique, alongside dedicated de-coupling branch for dense SSL pre- mreirrani Our extensive experiments and comprehensive ab- lation studies have confirmed the superiority of RCC and our de-coupling strategy within dense SSL frameworks, and we have shown that our de-coupling strategy seamlessly integrates into various frameworks without imposing excessive computational overhead.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant No. 62376209 and the Swiss National Science Foundation via the Sinergia grant CRSI15-180359,
REFERENCES
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. arXiv preprint 19/ 2019.
Yutong Bai, Xinlei Chen, Alexander Kirillov, Alan Yuille, and Alexander C Berg. Point-level re- gion contrast for object detection pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16061-16070, 2022.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image trans- formers. In International Conference on Learning Representations, 2022. URL https: / //openreview met/forum?id-gp Baannndddd-BB2225eee Ba---2h22225
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance segmentation. IEEE transactions on pattern analysis and machine intelligence, 43(5)::483-149;, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882 098822 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Krishna Chaitanya, Ertune Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems, 33: 13:12266-12555 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04223 042997 2020b.
Terrance DeVries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 170800450 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. ICLR, 2021.
Linus Eriesson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 5414-5423, 2021.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88 (2):303-338, 2010.
10

---

Published as a conference paper at ICLR 2024
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings the IEEE International Conference on Computer Vision, pp. 2961-2969, 2017.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre- pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 0918-9977, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16000-16009, 2022.
Olivier Hénaff, Skanda Koppula, Sean-Baptiste Alayrac, Aaron Van den Oord, Oriol Vinyals, and João Carreira. Efficient visual pretraining with contrastive detection. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 10086-10096, 2021.
Pan Ji, Tong Zhang, Hongdong Li, Mathieu Salzmann, and Ian Reid. Deep subspace clustering networks. Advances in Neural Information Processing Systems, 30, 2017.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics
quarterly, 2(1-2):83-97, 1955.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Jihao Liu, Boxiao Liu, Hang Zhou, Hongsheng Li, and Yu Liu. Tokenmix: Rethinking image mixing for data augmentation in vision transformers. In European Conference on Computer Vision, pp- 455-471. Springer, 2022.
Songtao Liu, Zeming Li, and Jian Sun. Self-emd: Self-supervised object detection without imagenet. arXiv preprint arXiv:201 33677, 2020.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings the IEEE conference on computer vision and pattern recognition, pp.
2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 10000 2017.
Pedro O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville. Unsupervised learning of dense visual representations. Advances in Neural Information Process- ing Systems, 33:4489-4500, 2020.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 4004-0015 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint
2018

---

Published as a conference paper at ICLR 2024
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Yao Huang, Hu Xu, Vasu Sharma, Shang- Dhang-Wen Li, Wojeiech Galuba. Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Yao Qin, Chiyuan Zhang. Ting Chen, Balaji Lakshminarayanan, Alex Beutel, and Xuezhi Wang. Understanding and improving robustness of vision transformers through patch-based negative augmentation. Advances in Neural Information P Processing Systems, 35: 16276-16289, 2022.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28:91-99, 2015.
Sucheng Ren, Huiyu Wang. Zhengqi Gao, Shengfeng He, Alan Yuille, Yuyin Zhou, and Cihang Xie. A simple data mixing prior for improving self-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14595-14604, 2022.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in Neural Information Processing Systems, 29, 2016.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsu- pervised semantic segmentation by contrasting object mask proposals. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10052-10062. 2021.
Feng Wang. Huiyu Wang, Chen Wei, Alan Yuille, and Wei Shen. Cp 2: Copy-paste contrastive pretraining for semantic segmentation. In European Conference on Computer Vision, pp. 499- 515. Springer, 2022a.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for supe-supe visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0033,3033, 2021.
Zhaoqing Wang, Qiang Li, Guoxin Zhang, Pengfei Wan, Wen Zheng, Nannan Wang, Mingming Gong, and Tongliang Liu. Exploring set similarity for dense self-supervised representation learn- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16590-16599, 2022b.
Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin. Aligning pretraining for detection via object-level contrastive learning. Advances in Neural Information Processing Systems, 34: 22682-22694, 2021.
Yanhao WU, Tong Zhang, Wei Ke, Sabine Süisstrunk, and Mathieu Salzmann. Spatiotemporal self- supervised learning for point clouds in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5251-5260, 2023.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pp- 418-434, 2018.
Tete Xiao, Colorado Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 10539-10548, 2021.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring lixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pp. 16684-16693, 2021.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9153-96533 2022.
12

---

Published as a conference paper at ICLR 2024
Hongwei Xue, Peng Gao, Hongyang Li, Yu Qiao, Hao Sun, Houqiang Li, and Jiebo Luo. Stare what you see: Masked image modeling without reconstruction. Proceedings of the IEEE/CVF Conference Computer Vision and Pattem Recognition, pp.
2023.
Ceyuan Yang, Zhirong Wu, Bolei Zhou, and Stephen Lin. Instance localization for self-supervised detection pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3987-3996, 2021.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. arXiv preprint arXiv: 770..0388..0 6(12):6, 2017.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceed- ings of the IEEE/CVF international conference computer vision, pp. 6023 6032,032,200 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny, Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.032.00 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.0941 2017.
Tong Zhang, Pan Ji, Mehrtash Harandi, Richard Hartley, and Ian Reid. Scalable deep k-subspace clustering. In Computer Vision-ACCV 2018: 14th Asian Conference Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part V 14, pp. 466-481. Springer, 2019a.
Tong Zhang, Pan Ji, Mehrtash Harandi, Wenbing Huang, and Hongdong Li. Neural collaborative subspace clustering. In International Conference on Machine Learning, pp. 7384-7393 PMLR, 2019b.
Tong Zhang, Congpei Qiu, Wei Ke, Sabine Süsstrunk, and Mathieu Salzmann. Leverage your local and global representations: A new sule-supe learning strategy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16580-16589, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattem recognition, pp. 633-641, 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representa- tions (ICLR), 2022.
Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14502-14511, 2022.
13

---

Published as a conference paper at ICLR 2024
A ALGORITHM
A.1 ORIGINAL CUTOUT FOR DENSE SSL
Algorithm Proposal-level Cutout (DeVries Taylor, 2017)
Input:
Proposals {b; (æm yat, 222, 222, w2) cutout ratio Output: Masked positions M
M +-+ {
for i < K do
t レ Rand( d.z2): <- Rand(yn d, y22)
M + ( {(x : + a). E [1, 1 + b)}
b Performing cutout
end for
A.2 REGION COLLOABORATIVE CUTOUT
Algorithm 2 Region Collaborative Cutout
Input: Proposals (b, (I11, M/1, Z.a, ya) cutout ratio re.
Output: Masked positions M
M < 0
B < (b:})
Region set of proposals sorted from large to small
for i < K do
Tas B0)- (MNB()
Actual ratio of B(i) being covered
B(]
rei > To then
M - MM{(I,)) v) : 22), E [ykt, 912)
> Recover overcut proposal
t
end if
- (x:² Za) (ra - b (y12 - 911) (rc rra)
I + Rand( (,2 Rand(ya - b(2
M +- M (x, : 1 a), E [1, b)}
b Performing cutout
end for
B DETAILS FOR MEASURING COUPLING
Inter-object coupling. As illustrated in Fig 6, we calculate the CR for the backbone pre-trained with various methods. For CNN-based models, we reduce the scale of object A with different reduction ratios. For ViT-based models, we apply random masking different mask ratios. To extract the region-level features. we apply RolAlign on the final feature map output by the pre-trained backbone with given bounding boxes. All the features used for cal- culating C'R values are extracted from images that are sampled from COCO train2017 with single object.
patches of object A with
Intra-object coupling. The CR histograms of iBOT (Zhou et al., 2022) and DenseCL (Wang al., 2021) are reported in Fig 3(b), and we also report the results SoCo(Wei et al., 2021) and MaskAlign(Xue et al., 2023) in Fig. 7. The object part Mop is the intersection of a randomly selected quarter region on the bounding box and the segmentation mask the sampled object. For feature extraction, perform average pooling with mask Mop on the final feature map. The CR values are calculated with images sampled from COCO train2017 with ground- -ruuh annotations.
C THE CAUSES OF COUPLING
A challenging pretext benefits image-level pre-training for learning more generalizable semantics for curated datasets and images containing one object. However, the same rule does not apply to dense SSI., and we demonstrate that challenging pretext tasks can lead coupling in dense SSL..
14

---

Published as a conference paper at ICLR 2024
Zg
cos())
Coupling Radio
1.8
PLRC
DensetL
DenseCL-D
7,56
SoCo
SoCo-D
1.2
MoCo-V2
25
A
XB
X-4
0%
25%
50%
75%
Information dropping pipaline
Reduction Radio
(a) Inter-object coupling measuring pipeline (CNNs)
coulping
(b) CNNs-based methods
Figure 6: Measuring inter-object coupling CNNs. (a) For CNN-based models, reduce the scale of object A with different ratios to calculate the CR. (h) We report the CR of CNN-based models. "DD' denotes method combined with our decoupling strategy.
SoCo-D
MaskAlign D MaskAlign
SoCo
0.04
0.02
0.05
0.06
0.07
0.11
0.15
0.19
0.28
Coupling Ratio
Coupling Ratio
(a) CR of SoCo (Wei et al., 2021)
(b) CRR. of MaskAlign (Xue al., 2023)
Figure 7: Intra-object coupling of SoCo and MaskAlign. We draw the histogram in terms of the coupling ratio. 'DD' denotes those methods combined with our de-coupling strategy.
C.1 STRONGER AUGMENTATION.
In image-level SSL, data augmentation helps to drop redundancy and keep the most discriminative information for an image. Nevertheless, it is necessary to keep essential semantic information for aligning 24,, with z As an illustrative example, we employ masking with different sizes to verify that the coupling is caused by strong augmentation. Specifically, we conduct experiments with two mask-involving methods: SoCo which utilizes Cutout, and iBOT with Blockwise Mask. We pre- train each model on COCO for 200 epochs with varying mask ratios and use the inter-object CR as an indicator to assess the extent of coupling. For iBOT, we calculate C'R on masked patches. As shown in Fig. strong correlation exists between the masking ratio and extent coupling, which validates our assumption.
C.2 ALIGNING STRATEGY.
Apart from masking complexity, we investigate the influence of alignment accuracy on coupling. We pair features with varying alignment accuracies to demonstrate how misalignment affects coupling in DenseCL and iBOT.
DenseCL. For a pair of views X X and corresponding point-level features ( Jimi {at Jimi
DenseCL aligns each 2! with the nearest one in the key view:
LDenseCu
L(2), 24),
arg max cos(2. (2(22={,))) =
To establish an imprecise alignment, we align each 2 to its n-th nearest key, denoted as S-T/@n (S-T/@1 represents the original strategy). We also provide the result with S-T/Rand that randomly aligns each query to arbitrary key.
iBOT aligns the feature 2 of each masked patch with the teacher output 2t at the same position as the reconstruction loss, which we denote as T@ 1. For a less precise aligning strategy denoted as T @,, align the z! with z/, with 2f being the n-nearest neighbor of z.
15

---

Published as a conference paper at ICLR 2024
Coupling 1.5
Coupling Ratio
1.8
0.2
03
0.4
05
Mask Ratio
Mask Ratio
(a) SoCo
(b) iBOT
Figure 8: Coupling ratio with different Mask Ratio. Both SoCo and iBOT are pre-trained
COCO for 200 epochs with varying mask ratios." "Coupling Ratio' denotes the inter-object C.R.
24
Coupling Ratio
Coupling
1.4
1.6
12
12
185
162
1806
N6W
16st8
Aligning Strategy
Aligning Strategy
(a) DenseCL
(b) iBOT
Figure 9: Coupling ratio with different aligning strategy. 'Coupling Ratio' denotes the inter-
object CR. The horizontal axis each method represents the increasingly imprecise aligning strate- gies from left to right.
In Fig. 9, we present the inter-object CR of each model pre-trained on COCO for 200 epochs, illustrating that less accurate alignment of positive pairs tends to lead more significant coupling.
D MORE ANALYSIS OF MASKS
As one of the core designs of our de-coupling strategy. the proposed RCC designed specifically for dense SSL. We thus further explore the effects of RCC compared to other masking strategies.
D.1 RCC vs. CUTOUT
RandomRessizeCrpp plays crucial role in image-level SSL and enhances the network's ability capture robust semantic representations, however, there has been a lack of region-specific masking strategies for dense SSL.. This naturally leads the idea of Cutout(De & Taylor, 2017), which randomly cuts out square regions of the image for each region. However, there are typically over- lapping areas of cutout between adjacent regions, which results in over-cutout, i.e., regions that largely covered by the cutout from other regions. To verify the effectiveness of our method in solv- ing this issue, we provide an additional experiment. Specifically, we generate N RCC and Cutout masks with a cutout ratio of 0.4 and calculate the average proportion of the actual cutout area in each proposal region. As shown in Fig. 11, when the number of proposals in each image increases, Cutout masks suffer from severe over-cutout issues, and the actual cutout ratio is much greater than 0.4. Our RCC stabilizes at the preset cutout ratio.
D.2 COMPARSION OF DIFFERENT MASKING STRATEGIES
To illustrate the distinctions between masking strategies, we employ four different methods for gen- erating masks, depicted in Fig. 12. We partition image into 2 x 2 regions and generate one bounding box for each region with predefined cut ratio. It's important to note that the center each bounding box is randomly selected within its respective region, potentially causing overlap between
16

---

Published as a conference paper at ICLR 2024
Box#
Box##
Box+2
Box=22
Mask Generation
Overcut Recovery
Mask Generation
Mask Generation
Mask Generation
(a) RCC in overcut condition
(b) RCC without overcut
Figure 10: Illustration of RCC algorithm. RCC iteratively recovers the overcut region caused by We display the example with boxes.
the larger bounding boxes.
PRC
Mean Curouut Ratio
0.8
0.2
10
Number of proposals per image
Figure 11: Actual cutout ratio RCC and Cutout in different numbers of regions. We the cutout ratio for each region to 0.4 for both methods. The X-axis is the number of proposed regions and Y-axis is the average cutout ratio. All the cutouts are applied each region, which is generated randomly with scale [0.2, 1.0] and aspect ratio [0.5, 21, and mask size is set to 224 x 224. The experiment repeated 500 times.
masks from different regions. comparison to the other three methods, our approach generates masks with fewer overlaps and more extensive coverage across the entire image.
(a) Random Mask
(b) Blockwise Mask
(c) Cutout
(d) RCC
Figure 12: Examples of different masking strategies. We generate masks utilizing different strate- gies with mask ratio of 0.4. For Cutout and RCC, we generate regions on x 2 grids.
E IMPLEMENTATION DETAILS
E. DENSE-LEVEL OBJECTIVE WITH DE-COUPLING
We will explain how incorporate the de-coupling strategy with different methods.
Point-level/aatch-lee methods. Point-level methods directly perform SSL on each point-level feature from the output feature map. For de-couping, first generate de-coupled view X with
17

---

Pi), Published as a conference paper at ICLR 2024
the background mask Macc and the key view X. We align the point-level features locating the foreground region with the output of teacher model with corresponding coordinate. The implemen- DenseCL(Wang al., 2021), iBhu et al., 2022) and MaskAlign (uue et al., 2023) follow the above setting. For brevity, we define the mask as the index set of patches covers in the case of point-level methods. As the details:
tations of de-coupling
a) DenseCL implements infoNCE loss on pair-wise point-level features extracted from view X" and X which leads objective:
A DensecCL = (1 A) k, 7 Hi
Encelld' k (k"( 1r),
(9)
where LNCE is the infoNCE loss with temperature т. Denote point-level features from view X as k the de-coupling version of DenseCL then defined as:
DDneeeCL-D = (1 2 + ADC
Σ jEM
( k², (1
(10)
M
b) iBOT adopts the same image-level distillation loss as DINO (Caron et al., 2021), but introduces novel pixel-level distillation loss on masked patches, as reconstruction loss. In this paper, we apply iBOT without multi-crop. This leads to the objective:
LiBOT =- X log f CLS (X)
Σ ((x*, log (x iEM
(11)
IM
where M represents the mask for reconstruction. The masked view, denoted as X is created using X and P. indicates the position of each patch. We construct the de-coupled view X with X* which leads to iBOT-D as:
ADC Rccl
LiBOT-D (1- ADD)
Σ
(LX*,P))" log fx(X Pp)).
(12)
JEMA
c) MaskAlign applics pre-trained encoder as a frozen teacher and directly aligns the visible fea- tures with a smooth LI loss to the corresponding features output by the teacher model. This leads to the objective:
{ || ø2ll2
|| - $
L1.1211
(13)
11|21 z2ll1
otherwise
2
LMaskAlign
Σ CuI [fa(X P.),
(14)
[MC
ig.
We then generate de-coupled view X from X and combine de-coupling strategy as
LaaakAAignnD = --DDD A
ADc
Σ Lu [x [I.
P)),
MMccl
jEMPiC
(15)
Region-level methods. Region-level methods define regions the collections of point-level fea- tures, serving as object priors. To incorporate with such setting, we perform our de-coupling loss on the intersection of the selected region and foreground M The implementations of SoCo(Wa et al., 2021) and Leopart(Ziegler Asano, 2022) follow the above setting. Specifically:
SoCo apply two types of framework for extracting region- features: R50-C4 structure that leverages conv5 block with RoiAlign on feature map output by cone4; an R50-FPN structure that extracts multi-scale region-level features with an extra FPN. We adopt R50-C4 for its simplicity.
18

---

Published as a conference paper at ICLR 2024
Note that SoCo applies selective search to generate K proposals for each view object prior, thus use the same proposals to obtain background mask MRCC. For two augmented view X and corresponding proposal region (PT) (PA Ti=1 the objective of SoCo-C4 is defined as:
LsoCo Σ cos (J&(X", PI), Pt)).
(16)
We extract region-level de-coupled features on the intersection of M RCo and each proposal, i.e.:
+ AnC Σ cos (J,(X,, Pj n IX
LsoCo- = (1
(17)
K
b) Leopart applies SwAV-like prototype- prodotype-based based loss(Caron al., 2020) on region level features. It generates V views with by global views and aligns region-level features masked by the global view's aftention, which leads the objective with 2D) cross-entropy loss I as:
Leopart
Lişj Mary (I(I,(X,
(18)
j=0 1=0
where Ma;; is the binary mask of global view's attention to the intersection with view X,. Pa, P, the corresponding pairwise intersection between X,, x-- Because local views typically provide incomplete semantics, We only perform de-coupling on global views to avoid difficult pretext that leads to coupling. With the de-coupled view x d generated by the global view Xj, we define the de-coupling objective as:
LLeoport-D = (1- (1-ADD)er Leopart + Ape Σ (Ma o 0l (),( (19)
1=0
where the binary mask M Mp,,, Maj the foreground defined by RCC and attention mask corre- sponding to X, respectively.
E.2 PRE-TRAINING DETAILS
De-coupling. For region generation and RCC mask, we divide the input view into 3 x 3 grids and create a single bounding box in each grid with the scale in (0.15, 4) and aspect ratio in the range (0.5.2). The RCC cutout ratio is selected from the range 00.3, 0.5). The de-coupled views are generated with the augmented key views, which are processed with the standard augmentation pipeline such as crop, jittering and gaussian filtering. We set the de-coupling weight Ape to 0.3.
Optimization hyper-parameters. For DenseCL, we follow the same setting to adopt an SGD optimizer with the base learning rate Irbase of 0.3. For SoCo, we adopt LARS (You et al., 2017) with Iraaee as 2.0 and batch size of 1024. For Leopart, we keep all the same. For iBOT, we utilize AdamW (Loshchilov & Hutter, 2017) and set the Irbaee
x 10-3 and the batch size of 512. For MaskAlign, we use Adaw with Irbase = 3 x 10-4 and the batch size of 1024.
Pre-training schedule. For models with ResNet-50, to compare with the existing baseline, follow the common schedule of 800-epoch pre-training adopt in (Wang et al., 2021; Bai et al., 2022). For models with ViT-S/16, since we mainly focus on the comparisons between same models with and without de-coupling, we adopt the 800-epoch pre-training schedule for both iBOT and MaskAlign.
E.3 EVALUATION PROTOCOLS.
Here, will explain the implementation details of the evaluation protocols in paper.
19

---

Published as a conference paper at ICLR 2024
Table Semantic segmentation results of model pre-trained with Leopart. "-D' denotes com- bining with de-coupling strategy, 'LC" denotes finetuning linear classifier, 'FCN' denotes fine- tuning an FCN model, and 'K=500' denotes over-clustering with K set as 500.
VOC12
COCO-Stuff
Method
LC
FCN
K=500
LC
K=500
Leopart
65.0
67.2
45.2
52.5
44.4
Leopart-D
66.5
68.9
48.2
53.2
45.3
E.3.1 FINETUNING FOR DOWNSTREAM TASKS
CNN-based evaluation We compare models using a ResNet-50 backbone on three dense-level benchmarks: Pascal VOC detection, COCO detection, and instance segmentation. For VOC de- tection, we fine-tune Faster R-CNN detector (C4-backbone) on the combined set of trainval2007 and Irainva/2012. As the variation on Pascal VOC is large, following (He et al., 2020), we run 5 trials and compute means. We then report AP, AP50, and AP75 on the test2007 set. For COCO de- tection and instance segmentation, we fine-tune Mask R-CNN detector (C4-backbone) on COCO train2017 with x schedule. The evaluation is performed on the COCO val2017 split.
ViT-based evaluation We compare models using ViT-S/16 backbone on three benchmarks: COCO detection and instance segmentation, and ADE20K semantic segmentation following (Zhou et al., 2022). For COCO detection and segmennattinn, with a 1x schedule on the train2017 7 split. For ADE20K segmentation, we adopt the task layer in UPerNet and fine-tune the entire network for 160k iterations the training split.
fine-tune Cascade Mask R-CNN framework
E.3.2 DETAILS OF OKNN AND OKNN-D
OKNN. We obtain object-level features from the output feature map by using ground bounding boxes. We achieve this using Rol Align with size of 7 x 7 and average pooling. Specifically, we resize each input image to 224 x 224. For ResNet50, we upsample the output of cono4 to 28 28 and obtain a 14 x 14 feature map output by conv5. Meanwhile, output of ViT-S/16. For the training set, we extract N object-level features per image to store the features and labels in memory bank because choosing all the objects in the dataset will lead to imbalanced numbers across the categories in the training set, which will lead to the results being biased. Similarly to image-level KNN, we predict the label for each object in the evaluation set by finding the k-nearest object-level features in the training set. For evaluation on COCO, we set N = and A 20. The above process does not need further training and is much faster. More importantly, compared to the evaluation with a fine-tuning stage, O-KNN directly reflects how well the features from the same class are preserved instead of favoring the mask or detection prediction.
directly utilize the 14 x 14
OKNN-D. D. OKNN-D Follows the same pipeline for feature extraction and label prediction as OKNN. The difference is that for every imput image, we directly replace the regions uncovered by the selected bounding boxes with arbitrary background images. When a pre-trained model encounters severe coupling issues, tends to intertwine each selected object with irrelevant background bias, resulting in more significant degradation of recognition.
F ADDITIONAL RESULTS
Evaluation of Leopart. The evaluations of Leopart(Ziegler Asano, 2022) follow its original pro- tocols. Specifically, we utilize three techniques for evaluation: a) linear classifier (LC): we fine-tune a 1 xx convolutional layer on top of the spatial output of the frozen backbone following (Van Gans- beke et al., 2021); b) over-clustering, we evaluate the frozen feature using spatially dense clustering and Hungarian matching (Kuhn, 1955); e) FCN, we fine-tune a FCN model (Long et al., 2015) fol- lowing same setting (Wang al., 2021). We report results Pascal VOC and COCO-Stufi in Tab. 7. Our model advances original Leopart significantly in transfer learning on VOC. On COCO- Stuff, Leopart-D also achieves improvement of 0.7 on LCC and 0.9 over-clustering. Note that
20

---

Published as a conference paper at ICLR 2024
Table Training time of different models. We detail the actual training time of different models. Each model is pre-trained on COCO for 800 epochs with 8-GPU 3090 machine. B denotes the batch size, *DD' denotes models with de-coupling. For de-coupling, we sample 25% key views to construct the de-coupled views which are only forwarded to the student encoder.
Method
Total Crops Number
Backbone
Training Time
SoCo
Bx2x2242
ResNet-55
27h
SoCo-D
Bx2x2249+Bx0025522
ResNet-55
30h
DenseCI.
Bx2x2242
ResNet-50
29h
DenseCL-D
Bx22x224++x0025x2222
ResNet-500
32h
iBOT
Bx2x2242
ViT-S/16
24h
iBOT-D
Bx2XX224*+x0022xx224
ViT-S/16
27h
Alsk
BxIx2242
ViT-S/16
10h
MaskAlign-D
Bx1x224"=Bx0.25x222
VIT-S/16
13h
Table Different pre-training schedules. We report COCO detection and instance segmentation performance. '-MC' denotes iBOT with multi-crop stratigy.
COCO Det.
COCO Iseg.
Method
Epoch
AP
AP50
AP75
AP
AP
AP7s
iBOT
800
42.3
61.2
45.6
58.3
39.4
iBOT
900
42.7
61.9
45.8
37.2
58.6
39.6
iBOT-MC
800
43.7
62.8
47.0
38.1
59.6
40.7
iBOT-D
800
45.1
64.3
48.7
39.1
61.2
41.7
Leopart only pre-trained for 50 epochs, following the original setting. It demonstrates that our de-coupling strategy can also work with short pre-training schedule.
Training time and learning efficiency. In Tab. 8, we provide the training times for each method when pre-trained on COCO for 800 epochs. To maintain a balance between the de-coupling effect and additional computational costs, we employ a strategy where we sample 25% key views to create the de-coupled views without introducing extra augmentations. These de-coupled views are exclusively forwarded to the student encoder. Across most dense SSL methods, the inclusion of de-coupling branch results in approximately 10% increase in training time.
Learning efficiency. To investigate whether the observed benefits of our de-coupling strategy are solely due to increase in the number of views, we conducted comparison using different pre- training schedules for iBOT. a) In the first schedule, we used 25% of query views to create de- coupled views and pre-trained iBOT for 900 epochs COCO, maintaining the same total number of views. b) In the second schedule, we employed multi-crop strategy for iBOT's pre-training, which included global views sized 224 and four local views sized at 962, denoted as iBOT-MC. As shown in Tab. 9, our iBOT-D outperformed all other models by significant margin, demonstrating that the advantages of our approach extend beyond the increase in the number of views.
G ADDITIONAL ABLATIONS
G.1 ABLATION ON CNNS
Mask types. In Section 6, we conducted ablation experiments to assess the impact of different components of the de-coupling strategy using iBOT. Following the same experimental setup, we pre- trained DenseCL on COCO for 200 epochs and examined the effects our RCC and de-coupling loss. The results are presented in Table 10. For the details of the masking strategy: a) DCRes: We generated rectangular masks within the scale range of [0.2, 0.6] and aspect ratio range of f2/3, 3/2]. b) DCRan: We randomly masked out patches background with mask ratio sampled from the range (0.4, 0.6]. c) For other methods, we sampled the mask/cutout ratio from the range |0.3, 0.5]. Similar our findings with iBOT, observed that the presence of copy-paste branch aids in
21

---

Published as a conference paper at ICLR 2024
Comparison of mask types. DenseCL used as a baseline, we only change the masking strategy for de-coupling.
Table 10:
Table 11: Comparison with copy-paste meth- ods. DenseCL. used as baseline.
Method
VOC Det.
OKNN
CRinter
CRintra
Method
Det.
OKNN
Citenter
CRjatro
DenseCL
55.0
67.5
1.64
0.31
DenseCL
55.0
67,5
1.64
0.31
+DCRsc Ruc
66.7
1.25
0.24
+DC
54.4
65.2
1.33
0.33
+DCRon
55.5
69.3
1.22
0.17
+DC,
54.1
1.41
0.40
+DCostout
54.6
65.1
1.30
0.25
+DC +DC,-r
55.2
66.7
1.25
0.24
DiC
55.3
69.0
1.16
0.20
+DCRcC
55.9
70.0
1.18
0.12
de-coupling at the inter-object level. However, only masks that introduce context noise at finer granularity can effectively mitigate intra-object coupling. Masks generated with RCC outperformed all other strategies in both downstream performance and de-coupling effects
Loss types. Table 1I presents comparison of different types of de-coupling loss using rectangular mask. The results demonstrate that aligning each decoupled dense-level feature directly to the key with the corresponding position is effective approach without performance degradation.
G.2 ABLATION HYPER-PARAMETERS
Number of grids. The input view is divided into Ngrid X N Ngrid grids to generate a single bounding box in each grid. We use varying values of Nyrid and pre-train iBOT-D on COCO for 200 epochs to investigate its influence on RCC for decoupling. Results in Tab. 12 indicate a low number of grids which prevents RCC from effectively generating cutout regions in various positions and will weaken the effectiveness of de-coupling, resulting in lower performance on downstream tasks. In practice, set N grid to 3 for computational efficiency.
Table 12: Ablation on the number of grids.
We report AP of COCO detection and cou- pling ratio with models pre-trained with dif- ferent numbers of grids.
x grids
x ggrids
N grid
2
4
COCO Det.
39.4
41.2
42.0
42.1
C.
1.42
1.10
1.15
1.14
3x33gids
x grids
CRIntta
0.16
0.08
0.04
0.03
Figure 13:
Background mask with different numbers of grids.
de-coupling weight. We pre-train the iBOT-D with different de-coupling weights Ape with results shown in Tab. 13. A high λnс will disturb the original pre-training objective, leading to worse performance on downstream tasks. We thus set the Xpc to 0.3 for the balance of de-coupling and dense SSL objectives.
Table 13: Ablation on ADC- We report AP of COCO detection and coupling ratio with models pre-trained with different do-coupling weights Apc-
ADe
0.1
0.2
0.3
0.4
0.6
COCO Det.
40.5
41.6
42,0
41.2
38.4
C Clinter
1.25
1.18
1.15
1.12
1.16
CITaaaaa
0.07
0.03
0.04
0.02
0.02
22

---

Published as a conference paper at ICLR 2024
X
Feature Map
Cij
context
Encoder
Bijj
LSC(L,)
Xeu
local Cij
Figure 14: Pipeline of calculating LSC. We sample small crop Xe., from the origninal view X with the center lying on the patch of i-th row and j-th column. Xc., and X are respectively forwarded to the encoder. We calculate the cosine similarity between the image-level feature of Xc., and the corresponding region-level feature of X as the value of LSSC(i,)..
H VISUALIZATION
H.1 POINT AFFINITIES
Previous work Bai al. (2022) provides simple yet intuitive visualization technique to show how well the SSL model learns from image. We adopt similar approach to visualize the feature similarity maps generated by different pre-training methods, which intuitively demonstrates the ef- fectiveness of our de-coupling strategy. We sample images from COCO for visualization and resize them to the size of 448 x x448. The feature maps are obtained by the interpolation of the output fea- ture map from the final layer 56x For each image, a specific feature was selected (marked as a red dot), and its cosine similarity to the remaining features in other locations from the feature map was calculated. The models used for visualization are pre-trained on COCO for 800 epochs. The results of CNN-based and ViT-based models are shown in Fig. 15 and 16.
H.2 LOCAL SEMANTICS CONSISTENCY
As depicted Fig. 14, we argue that a network effectively capturing local semantics should be able to maintain consistency with or without neighboring context. Based this assumption, the effectiveness of the do-coupling paradigm through new type of visualization. For given image
verify
X of size 512 512, we first patchify it with a patch size of 16, yielding 32 x 32 patches Pej. We define crop region Cij. whose center lies on Pij. containing x patches. The local semantics consistency map detined
context Local F((c,,,), ZCig CGs
local content). Ci,, LSC((..) cos (z ; =Cas
(20)
We directly pass a local crop. denoted as Xesj. through the encoder to obtain the local feature at Cij- Simultaneously, send the entire image X to the backbone network E and extract region-level features corresponding to the area Cij. resulting in =C;; =gontezt higher value the LSSC' map shows that the representation of the local region is resilient to the absence of contextual information, rather than being oved by its neighboring context, indicating a robust local feature. The results for both CNN-based and ViT-based models are illustrated in Figure 17 and Figure 18.
H.3 FURTHER DISCUSSION WITH DINO v2
DINO v2(Oquab et al., 2023) applies iBOT loss for dense-level pre-training. To further investigate the model's behavior of capturing local semantics when pre-trained on large-scale curated datasets, we visualize DINO v2 following the same pipeline adopted Appendix H.1 with the officially released ViT-S/14 and ViT-B/14 distilled models. We show the failure cases in Fig. 19 where the model entangles the features of sub-diseriminative objects with their surroundings. We hypothesize that even the foundation model pre-trained dense level without the de-coupling constraint suffer from the coupling issue. We will leave further discussion on this issue for future work.
23

---

Published as a conference paper at ICLR 2024
SoCo
SoCo-D D
PLRC
DenseCL
DenseCL-D
Figure 15: Point affinity visualization CNN-based models. The selected point-level feature is marked as a red dot.
24

---

Published as a conference paper at ICLR 2024
iBOT
iBOT-D
MaskAlign
MaskAlign-D
Figure 16: Point affinity visualization ViT-based models. The selected point-level feature is marked as a red dot.
25

---

Published as a conference paper at ICLR 2024
SoCo
SoCo-D
DenseCL
DenseCL-D
BBG
Figure 17: Visualization of LSC map on CNN-based models. Brighter colors denote higher LSC.
26

---

Published as a conference paper at ICLR 2024
iBOT
iBOT-D
MaskAlign
MaskAlign-D
BBC
Figure 18: Visualization of LSC map on ViT-based models. Brighter colors denote higher LSC.
27

---

Published as a conference paper at ICLR 2024
DINO v2((s DINO v2( iBOT-D(S)
DINO v2(S) DINO v2( iBOT-D(S)
Figure 19: Point affinity visualization on DINO v2 and iBOT-D. The selected point-level feature
marked as a red dot. '(S)' denotes the backbone of ViT-S and 'B' denotes ViT-B.
28


Reviews:
[{"review_id": "2K5IeP2wz1", "reviewer": "Reviewer_wwZs", "summary": "This paper considers the problem of information leakage from the neighboring contextual regions in dense self-supervised learning. The contributions of this paper include the identification and confirmation of the coupling phenomenon when pairs have a limited overlap, the design of a decoupling branch, a novel region collaborative cutout augmentation, and the effectiveness of the proposed approach in the dense self-supervised learning frameworks. This approach can be applied to both CNNs and ViTs as the approach is only related to the augmentation.", "questions": "Context within the region. The shape of each object varies and is irregular. And the region is defined by a bounding box (at least in object detection task), There would be context information in the bounding box. How to measure or address the coupling or leakage for this part of information?\n\nThis method would introduce another hyperparameter, the threshold of cutout ratio. How to set this across different datasets?\n\nAblation study. The numbers about COCO Det. in Table 6 do not match the numbers reported in Table 2 for iBOT. This is because the training epochs are different. If we view the results in Table 2 as the converged one, the comparison made in Table 6 may not lead to a convincing conclusion. While three experiments are presented in this section, a more interesting ablation study would be the effectiveness of the decoupling branch. \n\nWhile the authors claim that the proposed method can be combined with existing SSL methods, there is a concern that the proposed method may not work well for the method with contrastive loss within each batch. The reason for this is that the decoupling branch serves a similar purpose. However, there may be additional contribution as the losses are computed at different levels with different masks.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper is well written and easy to follow.\n\nThe proposed method is simple and can be directly combined with existing dense self-supervised learning without introducing additional loss types.\n\nThe proposed method, RCC and decoupling branch, are demonstrated to be effective with several existing self-supervised learning methods in the experiments.", "weaknesses": "While the augmentation in Fig. 5 is easy to understand, the random masking in Fig. 2(a) is vague in terms of the illustration purpose.\n\nThe main text is expected to be self-contained. However, there are several cases where content is put in the appendix. For example, the figure 7(a) is useful to understand the definition of variables in equation 4, however, is put in the appendix. One possible way to address this problem is to merge Fig. 7(a) with part of Fig. 3. Again, the Alg 2 is another example for the understanding of the proposed RCC (region collaborative cutout).\n\nThe original contribution of the approach may be limited as the proposed RCC could be viewed as the combination of thresholding the cutout ratio within a region and filling the cutout region with background images. This is different from the Cutout, but more like combining existing strategies.\n\nThe order of Tables 4-6 does not match the appearance in the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "50Dx1WjK2T", "reviewer": "Reviewer_GFda", "summary": "This paper presents a new method called Region Collaborative Cutout for self-supervised learning to alleviate the object coupling issue. This simple and straightforward method achieves evident gains over previous methods.", "questions": "No further questions.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written, and it is easy to catch the main motivation and solution. And I think the proposed method is well motivated.\n\n2. Multiple previous methods are used as baselines to build the proposed Region Collaborative Cutout upon, and non-trivial improvements are observed. Besides, the method is proved effective for both CNNs and ViTs.\n\n3. The ablation studies are comprehensive and convincing.", "weaknesses": "I am not very familiar with the research line of SSL. So I have no further suggestions for this paper. Generally, I like this simple yet effective method. It further highlights that constructing appropriate positive pairs by delicately designed strong augmentations is important.\n\nHowever, one of my slight concerns is about the whole area of SSL since DINOv2 was released. It is pre-trained on extremely large-scale and curated data with several practical SSL optimization targets. It is very strong in many applications, such as retrieval, segmentation, and detection. Even with a frozen DINOv2 backbone, we can achieve state-of-the-art performance in some challenging tasks. Therefore, could the authors discuss the position of this submission by taking the recent SSL trend into consideration?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KvFU9J8AwO", "reviewer": "Reviewer_VesV", "summary": "Submission 945 presents a visual self-supervision method suitable for dense (/local) tasks such as detection/segmentation, etc. It motivates itself by presenting demonstrations that show that attention maps (/point affinities) do not localize well to object-parts and this is due to the false positives generated by current self-supervision methods.\n\nIt presents a variant of CutMix for dense self-supervision that mixes tokens from the input image with tokens from an external image. The mixing strategy is developed such that a token from the input image is largely surrounded by tokens from the output image so that the input token is positionally “out of context”. It then presents a regularizer for any visual self-supervision loss such that features extracted from the “in context” and “out of context” tokens (taken from the input image) are similar.", "questions": "### Suggestions:\n- Please clarify the differences between TokenMix and similar token mixing works and this paper.\n- Please revise and improve the writing and presentation of the first four sections of this paper to make it more immediately accessible.\n- Please briefly discuss the motivation for the decoupling regularizer in the context of existing methods such as DinoV2 achieving high-quality part-level attention maps without considering the “coupling” phenomenon.\n- Figure 5 on page 7 is what finally made the method click for me – please move it to earlier in the paper.\n\n### Minor questions:\n- Experimental clarifications would be beneficial: Why is 800 epochs of pretraining on COCO specifically chosen for all methods? How were the hyperparameters tuned for the baselines? Also, most of the experiments do not mention splits.\n- Apart from the proposed masking, there appears to be no other augmentations mentioned and there is no code. Were standard augmentations such as jitters, flips, blur, etc. (https://github.com/bytedance/ibot/blob/main/main_ibot.py#L574) also used? Did the augmentation strategy match the existing methods on which decoupling was applied?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The presented work is reasonably thorough in its experiments and generality. In particular, I like that it presents a rationale for both ViTs and CNNs.\n- While confusingly presented, the high-level idea of asking token-wise features to be invariant to some of their surrounding tokens is a simple idea and appears to lead to better attention maps and downstream performance.", "weaknesses": "(in no particular order)\n\n### Poor presentation\nUnfortunately, the unclear writing and figures significantly dampened any enthusiasm for this paper and it took multiple repeated readings to get a high-level sense of what is proposed. As a few examples,\n- Paragraph 3 of the Introduction only makes sense if you already know the entire method.\n- Figures 1(a) and 2(a) are really hard to understand and are used to motivate the work. For example, what do the red dashed lines, the arrows, and the “ViTs-based measuring pipeline” indicate in 2(a)?\n- I don’t understand the written presentation of the mask generation (Sec. 4.1.1, par. 1) at all and its associated algorithm in the appendix uses undefined notation that is hard to follow (if it is defined elsewhere, please also add it to the caption). As a result, figure 4 does not immediately follow either.\n\nIMO the paper requires a significant revision for clarity.\n\n### Engineered token mixing has been done previously:\nThe proposed method has two main contributions: a cutmix style augmentation at the token level and a self-supervised loss leveraging that augmentation. However, while presented as new here, mixing tokens from different images in a carefully engineered way has been done before in TokenMix (ECCV’22, https://arxiv.org/abs/2207.08409 ) in the context of supervised classification and some other papers that follow up on it. Please discuss these works and clarify any differences, so as to better contextualize the key novel contribution of the self-supervised loss function that takes advantage of token mixing here.\n\n### Unclear motivation and relationship to current work:\nIn my reading, the main motivation of this work is achieving higher quality part-level attention maps by reducing the dependence between a token’s features and its surroundings. It is then unclear to me if this strategy can then capture long-range nonlocal dependence – could you please comment on this point and clarify if I misunderstood?\n\nMoreover, in recent work, DinoV2 (https://arxiv.org/abs/2304.07193) demonstrated that high-quality part-level representations can be learned by simply scaling up model and dataset sizes without considering the “coupling” between objects and background. I am not asking for comparisons, but I would like the response to briefly clarify the motivation of the proposed strategy in this context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)