Understanding Self-Supervised Pretraining with Part-Aware Representation Learning
Authors: Jiyang Qi
Jie Zhu
Mingyu Ding
Xiaokang Chen
Ping Luo
Leye Wang
Xinggang Wang
Wenyu Liu
Jingdong Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Part-aware representation
Self-supervised learning
Masked image modeling
Contrastive learning

Abstract: In this paper, we are interested in understanding self-supervised pretraining through studying the capability that self-supervised representation pretraining methods learn part-aware representations. The study is mainly motivated by that random views, used in contrastive learning, and random masked (visible) patches, used in masked image modeling, are often about object parts.

We explain that masked image modeling is a part-to-part task: the masked patches of the object are hallucinated from the visible patches, and that contrastive learning is a part-to-whole task: the projection layer hallucinates the whole object representation from the object part representation learned from the encoder. The explanation suggests that the self-supervised pretrained encoder is required to understand the object part. We empirically compare the off-the-shelf encoders pretrained with several representative methods on object-level recognition and part-level recognition. The results show that the fully-supervised model outperforms self-supervised models for object-level recognition, and most self-supervised contrastive learning and masked image modeling methods outperform the fully-supervised method for part-level recognition. It is observed that the combination of contrastive learning and masked image modeling further improves the performance.

Under review as a conference paper at ICLR 2023
UNDERSTANDING SELF-SUPERVISED PRETRAINING WITH AWARE REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, interested in understanding self-supervised pretraining through studying the capability that self-superrised representation pretraining methods learn part-aware representations. The study is mainly motivated by that random views, used in contrastive learning, and random masked (visible) patches, used in masked image modeling, often about object parts.
We explain that masked image modeling is a part-to-part task: the masked patches of the object hallucinated from the visible patches, and that contrastive learning is a part-to-whole task: the projection layer hallucinates the whole object represen- tation from the object part representation learned from the encoder. The explanation suggests that the self-supervised pretrained encoder is required to understand the object part. We empirically compare the aff-the-shelf encoders pretrained with several representative methods on object-level recognition and part-level recogni- tion. The results show that the fully-supervised model outperforms self-supervised models for object-level recognition, and most self-supervised contrastive learning and masked image modeling methods outperform the fully-supervised method for part-level recognition. It is observed that the combination of contrastive learning and masked image modeling further improves the performance.
INTRODUCTION
Self-supervised representation pretraining has been attracting lot of research efforts recently. The goal is to train an encoder that maps an image to representation from visual contents without the necessity of human annotation, expecting that the encoder benefits the downstream tasks, e.g.. segmentation and detection.
There are two main frameworks: contrastive learning' and masked image modeling. Contrastive learning aims to maximize the agreement of the embeddings of random augmented views from the same image. Masked image modeling partitions an image into masked patches and visible patches, and makes predictions for masked patches from visible patches. Figure/ gives examples of random views for contrastive learning and masked and visible patches for masked image modeling.
We observe that a random view and a set of masked (visible) patches usually contain portion of an object. It is also reported in self-supervised learning methods, e.g.. DINO (Caron et al., 2021) and iBOT (Zhou et al., 2021), ) that different attention heads in VITs can attend to different semantic regions or parts of an object. In light of this, we attempt to understand self-super pretraining by studying the capability that the pretrained encoder learns part representations.
We present a parl-to-whole explanation for typical contrastive learning methods (e.g.. SimCLR (Chen al., 2020), MoCo (Chen et al., 2021), and BYO (Grill et al., 2020)): the embedding of the whole object is hallucinated from the embedding of the part of the object contained in the random crop through a projection layer. In this way, embeddings of random crops from the same image naturally agrees with each other. Masked image modeling is a part-to-part process: the embeddings of the masked patches of the object (a part of the object), are hallucinated from the visible patches (the other part of the object).
In this paper, we use contrastive learning to refer to the methods that compare random views, e.g., SimCL.R, MoCo, and BYO..

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
(d)
(e)
Figure 1: (a) original image, (b-c) two random crops, and (d-e) masked and visible patches.
DeiT
MoCo v3
CAE
Figure 2: Top-24 patch retrieval results with three frozen encoders of DeiT, MoCo v3, and CAE, by taking the patch in the red box as the query. It can be seen that the retrieved results from CAE and MoCo v3 are about the object part (wing and dog mouth) and more precise than DeiT (about the whole object) implying that self-supervised pretraining methods, CAE and MoCo v3 are stronger at learning part-aware representations than the super-sup method DeiT.
We empirically compare the supervised model DeiT (Touvron et al., 2020) and typical supe-supe representation pretraining methods, including MoCo v3 (Chen et al., 2021), DINO (Caron et al., 2021), CAE (Chen et al., 2022a), MAE (He et al., 2021), BEiT (Bao et al., 2021), and iBOT (Zhou et al., 2021), on object-level recognition (image classification and object segmentation) and part-level recognition (patch retrieval, patch classification, and part segmentation).. Figure 2 presents patch retrieval results using the encoders learned through CAE, MoCo v3, and DeiT. implying that the encoders pretrained by CAE and MoCo v3 are able learn part-aware representations.
Through extensive studies and comparisons, we make the following observations. 1) DeiT outper- forms contrastive learning and MIM methods except iBOT in object-level recognition tasks, which may benefit from its explicit object-level supervision. 2) In contrast, self-supervised methods learn better part-aware representations than DeiT. For example, while DeiT superior to DINO and CAE by 0.4% and 2.3% on ADE20K object segmentation, DINO and CAE outperform DeiT by 1.6% and 1.1% on ADE20K part segmentation, respectively. 3) contrastive learning. the encoder can learn part-aware information, while the projected representation tends to be more about the whole object. The evidence could be found part retrieval experiments on MoCo v3, DINO, and iBOT. 4) The MIM method CAE shows good potential in part-aware representation learning. Interestingly, the method combines contrastive learning and MIM is promising, e.g., iBOT learns better representations both object and part levels.
To summarize, this paper presents the following contributions:
We study the capability of learning part-aware representations as way of understanding self- self-supervised representation pretraining.
We explain masked image modeling part-to-part task and contrastive learning part- to-whole task, and speculate that self-supervvsed pretraining has the potential for learning part-aware representations.
2

---

Under review as a conference paper at ICLR 2023
We empirically compare several pretrained models on object-level and part-level recognition tasks, showing interesting findings with supporting evidence of the capability of part-award representation learning for self- self-supervised learning.
RELATED WORK
Contrastive learning. Contrastive pretraining has been an intense academic field in the CNN era. In this work, we use it to refer to methods for comparing random views (Caron et al., 2020; Chen et al., 2020; Zbontar et al., 2021; Xie et al., 202 Chen et al., 2021: Caron et al., 2021), including some instance discrimination work such as (Grill et al., 2020; Chen & He, 2021; Bardes et al., 2021). As one of the representative works, SimCLR (Chen et al., 2020) learns representations through maximizing agreement between different views of the same image in the latent space. YOL (Grill al., 2020) uses two asymmetrical networks to bootstrap latent representation without negative samples involved during the interaction. As vision transformer (ViT) (Dosovitskiy et al., 2021) shows excellent performance via supervised learning, it is adopted subsequently in contrastive pertaining. and numerous outstanding works are proposed. For example, MoCo v3 (Chen et al., 2021) observes the hidden instability while training self-supervised ViT and solves it by using a fixed random patch projection. DINO (Caron et al., 2021) explores new properties derived from self-supervised VIT and accordingly designs a learning strategy interpreted as a form of self-distillation with no labels.
Masked image modeling (MIM). Masked image modeling is another self-supervised pretraining paradigm that attracts much attention recently. BEiT (Bao et al., 2021) follows masked language modeling in the natural language process (NLP) area and predicts tokens via mapping image patches by V-E (Ramesh et al., 2021). PeCo (Dong et al., 2021) boosts BEiT by taking into considcration more semantic information in visual tokens. MAE (He et al., 2021) learns rich hidden information by directly performing masked image reconstruction in RGB color space using VIT while SimMIM (Xie et al., b0 uses twin-transformas (Liu et al., 2021). CAE (Chen al., 2022a) adds a regressor between encoder and decoder, which is designed to align unmasked patches with masked ones, leading to a pure context encoder. Recently, trend that combines MIM with siamese frameworks has surfaced and showed encouraging results including MST (Li et al., 2021), SplitMask (El-Nouby et al., 2021), iBOT (Zhou et al., 2021). dBOT (Liu al., 2022), and SIM (Tao et al., 2022).
Understanding self-supervised contrastive pretraining. The studies on understanding contrastive pretraining (Saunshi et al., 2022; Chen et al., 2022b; Zhong et al., 2022; Wei et al., 2022) mainly focus on random augmentations (views), contrastive loss function and its variants under the assumption that: the augmentations of inputs from the same class have significant overlap in the representation space, but there is little overlap for inputs from different classes. Our work is complementary to these studies. Inspired by the observation that random views usually contain a portion of an object, and methods (Caron et al., 2021; Zhou et al., 2021) show that different attention heads in ViTs can attend to different semantic regions of an object, we investigate what the encoder and the projector do in typical self-supervised contrastive pretraining. We speculate that the pretraining task is part-to-whole problem, predicting the representation of the whole object through the projector from the representation (obtained from the encoder) of the part of an object. We use empirical results to verify our analysis.
Understanding seff-supervised masked image modeling. The comparison of attention in different layers between the pretrained models from MIM and the supervised approach is conducted: MIM pretraining brings locality to the trained model with sufficient diversity on the attention heads (Xie et al., 2022a). Consistent with the analysis in NLP, empirical studies are conducted in Xie et al. (2022b) to verify that MIM benefits from larger models, more data, and longer training. CAE (Chen et al., 2022a) gives the comparison between contrastive and MIM and shows MIM cares about all patches and thus achieves better results for fine-tuning. Cao et al. (2022) provides a mathematical understanding of MIM. Kong& & Zhang (2022) points out that the learned occlusion invariant feature contributes to the success of MIM. In this work, we speculate that masked image modeling is a part- to-part process: the embeddings of the masked part of the object are hallucinated from the visible part using the position information of the masked patches, leading to better part-aware representation than the supervised model DeiT' (Touvron et al., 2020).
3

---

Under review as a conference paper at ICLR 2023
Encoder
Projector
2g
Predictor
Enroder
X,
Projector
3
Aligmment
Figure 3: The pipeline of a typical contrastive learning approach. Two augmented views, red box and blue box, generated from the original image. The augmented view in red is fed into the encoder and the projector, and then the predictor (which does not appear in earlier works like MoCo (Chen al., 2021) and SimCLR (Chen et al., 2020)), and the view in blue is fed into the encoder and the projector. The two outputs are expected to be aligned. The gradient is stopped for the bottom stream.
UNDERSTANDING CONTRASTIVE LEARNING AND MASK IMAGE MODELING
3.1 CONTRASTIVE LEARNING
Contrastive learning aims to learn the encoder through maximizing the agreement between differently augmented views of the same image in the representation space. An example pipeline is depicted in Figure 3. Given an image I, the augmentations, c.g., random cropping, random color distortion, and random Gaussian blur, are applied to generate set of N augmented views, {V1, V2, Vv3. An augmented view V, is fed into an encoder Encoder, generating the encoded representation x., and followed by projector, generating the projection Z,, The basic goal is maximize the agreement between the projections {Z1, " ,ZN]; i.e., minimize the loss
N CCPT E,
(rrojector(Bncoder//)
(rojector(Encodee((..))
(I)
In the formulation with contrastive loss, the agreement between the projections of random augmen- tations from different images is minimized.
Part-to-whole prediction explanation. Let us consider two crops randomly sampled from the original image (see the examples given in Figure 1(b-c)). The encoded representation of the first crop is expected describe a part of the object dog; the encoded representation of the second crop is expected to describe another part of the object dog". The two representations are related but different. Contrastive learning methods project the two encoded representations into two projected representations that are expected to agree. We hypothesize that the projection process maps the encoded part representation to the representation of the whole object'. Through this way, the projected representations will agree to different views from the same image. It is assumed that the part-to-whole projection is more reliable if the encoded representation is semantically richer and is able to describe the part information. The part-to-whole whole process suggests that the encoder pretrained by contrastive learning methods is potentially capable of learning part-aware representations.
Figure 4 provides patch search results of representative contrastive learning method MoCo v3 (Chen et al., 2021) based on the encoded representations before and after the projections. One can see that the results through the encoded representations are mainly about the local part, and the results through the projections tend to include the other parts of the same object. In other words, the projections tend be about the whole object. Similar observations are also shown in Chen al. (2022b). The search hypothesis.
results verify the
3.2 MASKED IMAGE MODELING
Mask image modeling is the task of predicting some parts of an image from the remaining parts.
An augmented view of an image is partitioned into patches, R (Ri, RR Rar The task is
II this paper, we mainly study the capability of learning representations about objects and parts, and leave the hackground as the future work.
the study of representations It is said that different parts have common causes in the external world (Becker & Hinton, 1992). Our hypothesis that the common cause the whole object.
4

---

Under review as a conference paper at ICLR 2023
Encoded representation
Projected representation
Figure 4: Illustration of patch search results using encoded representations and projections (pretrained with MoCo v3 as). Left: patch search results with encoded representations. Right: patch search results with projections. In each result, the small patch encircled by the red box taken as the query. It can be seen that for encoded representations, the returned patches are about the same part, and for projections, the result patches are about the same object, verifying the part-to-whole hypothesis.
to predict subset of patches Rm, named masked patches, from the remaining patches R., named visible patches. Considering contrastive learning that explicitly compares representations random views, we take context autoencoder (CAE) (Chen et al., 2022a) an example that explicitly predicts the encoded representations of the masked patches from the encoded representations of the visible patches*.
One goal of CAE (illustrated in Figure 5), which we call masked representation modeling (MRM), is to maximize the agreement between the predictions of the representations of masked patches (through a regressor) and the representation of masked patches computed from the encoder by minimizing the loss
MRR (EncooseeoEEd Encoder(Pm))
(2)
Here, we do not include the positional embeddings of masked and visible patches for clarity. It is noted that MRM differs from contrastive learning: MRM does not compare multiple random views, but compares the regressed representations for masked patches and the encoded representations of masked patches. In addition, there is another loss for target prediction (reconstruction) for the masked patches, which is commonly used in masked image modeling (MIM) methods:
MIM (Deecodrr(eegressnn
(Encoder(P..)). (Regreso
(Regressor
TPrg)l
where Target (P...) is function to map the masked patches to the targets, e.g., d-VAE (Ramesh et al., 2021) token used CAE and BeiT (Bao et al., 2021), or normalized RGB values used MAE (He et al., 2021).
Part-to-part prediction explanation. The masked image modeling approaches, including CAE, MAE, and BEÎT, make use of the positions of masked patches for making predictions for masked patches from visible patches. The visible patches and masked patches often contain different parts of an object. In other words, MIM aims to predict the masked part of an object from the visible part. We name this a part-to-part process. There are two part-to-part tasks: one is to reconstruct the part targets from the visible part representations (MAE and CAE) from the visible part raw pixels (BEiT), and the other one is to regress the masked part representations (CAE). The part-to-pan torp process suggests that the encoder pretrained by MIM methods is potentially capable of learning part-aware representations. Figure 2 illustrates the capability with the patch retrieval results.
'Some MIM methods. such as BEiT (Bao et al., 2021) and MAE (Masked Autoencoder) (He et al., 2021) do not have an explicit process to predict the encoded representations of masked patches, instead, directly reconstruct the targets.
5

---

Under review as a conference paper at ICLR 2023
(4)
(b)
Lateat contextual regressor
Encoder
Decoder
X
Encoder
Aligument
Visible patch
Masked patch
Mask
Target
The pipeline of an MIM approach, context autoencoder (CAE). An augmented view (in blue) of the image is partitioned into visible and masked patches. The CAE approach feeds visible patches into the encoder and extracts their representations Zy and then completes the pretext task by predicting the representations Zm of the masked patches from the visible patches in the encoded representation space with latent contextual regressor and alignment constraint, and mapping predicted representations Zn of masked patches to the targets. The pretrained encoder in (a) is applied to downstream tasks by simply replacing the pretext task part (b, with the downstream task completion part.
Figure 5:
Table Top-I accuracy with linear probing, and attentive probing (Chen et al., 2022a), the ImageNet classification benchmark (Deng et al., 2009).
Table 2: Linear evaluation ADE20K (Zhou
et al., 2019) object-level semantic segmentation (150 classes) using 4x upsampling and a single x convolutional layer frozen backhones.
Method
Linear
Attentive
Method
mloU
mAcc
aAcc
Supervised Model:
Supervised Model:
DeiT
81.8
81.8
DeiT
34.9
44.2
75.4
Contrastive Learning:
Contrastive Learning:
MoCo v3
77.0
MoCo
34.7
43.9
75.9
DINO
77.3
77.8
DINO
34.5
43.5
76.1
Masked Image Modeling (MIM):
Masked Image Modeling (MIM):
BEIT 41.8
51.9
BEiT
17.8
23.7
64.9
MAE 67.8
74.2
MAE
27.1
34.8
71.6
CAE
70.4
77.1
CAE
32.6
42.2
75.2
Contrastive Learning + MIM:
Contrastive Learning MIM:
iBOT
79.5
79.8
iBOT
38.3
47.4
78.1
4 EXPERIMENTS
We study seven representative methods with the same ViT-B encoder, including a supervised method DeiT (Touvron et al., 2020); contrastive learning methods MoCo v3 (Chen et al., 2021), DINO (Caron et al., 2021); );; masked image modeling (MIM) methods BEiT (Bao et al., 2021), MAE (He et al., 202 I), and CAE (Chen et al., 2022a); and iBOT (Zhou et al., 2021) that combines contrastive learning and MIM. We take the training epochs specified in each work to ensure that all compared models are properly trained: 300 for DeiT, 300 (600²) for MoCo v3, 400 (1600*) ) for DINO and iBOT, 800 for BEÏT, and 1600 for MAE and CAE. Frozen encoders used in all experiments understand what these different representation pretraining methods learn. More details can be found in Appendix A.
4.1 OBJECT-LEVEL RECOGNITION
We benchmark two widely-studied object-level recognition, i.c., image classification and semantic segmentation to show the capability that the pretrained encoder learns object-level representations.
TThe number of effective training epochs introduced in Zhou al. (2021).

---

Under review as a conference paper at ICLR 2023
Table 3: Part retrieval (AP, %) and classification (accuracy, %) results on the cropped part patches of CUB-200-2011 and COCO. The "Encoded" and "Projected" refer to the encoded and projected representations. "Linear" and "Attentive" columns denote the linear probing and attentive probing accuracy, respectively.
Part Retricval
Part Classification
Methods CUB-2000 200-2000
COCO
CUB-200-2011
COCO
Encoded Projected
Encoded Projected
Linear
Attentive
Linear
Attentive
Supervised Model:
DeiT 35.0
44.1
90.9
92.9
88.5
91.4
Contrastive Learning:
MoCo v3
50.8
28.4
52.3
36.8
93.8
96.0
92.4
95.3
DINO
48.9
31.7
51.8
41.2
93.2
95.2
91.7
94.5
Masked Image Modeling (MIM):
BETT 27.9
35.3
55.4
86.5
69.3
86.5
MAE 28.5
37.1
86.9
92.8
88.0
93.9
CAE 58.0
57.0
89.5
95.8
91.1
95.5
Contrastive Learning MIM:
iBOT 49,3
31.2
59.2
41.5
93.8
95.8
92.1
95.1
Image classification. We report the linear probing, and attentive probing results of the selected models on ImageNet (Deng al., 2009). For attentive probing, we follow the protocol CAE (Chen et al., 2022a) that append cross-attention layer together with batch normalization layer and linear classifier.
We have the following observations from Table 1. 1) The supervised model, DeiT performs better than self-supervised models at object-level recognition. 2) The models that leverage contrastive learning, i.e., MoCo, DINO, and iBOT, show superior linear probing performance than MIM-based models, demonstrating they contain more object-awar abjret-wware high-level semantics. 3) MIM-based models, e.g., CAE, show inferior results in linear probing while competitive results with contrastive-basee methods in attentive probing. The reason might be that MIM is capable attending to all the regions, including non-object regions in image, thus needs spatial feature selection step attend the object part, which is pointed out in Chen et al. (2022a). BEiT and MAE perform inferior, implying that the two methods are less capable of learning semanties.
Object-level semantic segmentation. We perform linear evaluation on ADE20K (Zhou et al., 2019) show the object-level semantic capabilities of the pretrained models. A 4x bilinear interpolation and single X 1 convolutional layer for pixel labeling are attached to the frozen encoder.
We can see from Table that the supervised model DeiT outperforms all self-supervised models except iBOT, including contrastive learning and MIM methods on ADE20K object-level segmentation. This implies that in general the superviedd models are not strong at object-level understanding. which is consistent with the observations for image classification. iBOT (Zhou et al., 2021), 1), as a combination of contrastive learning and MIM, shows surprisingly better performance than the supervised model DeiT on ADE20K, implying the power of combining contrastive learning and masked image modeling for downstream tasks.
4.2 PART-LEVEL RECOGNITION
Self-supervised methods like iBOT (Zhou et al., 2021) and DINO (Caron et al., 2021) qualitatively show that different attention heads in ViT' can attend to different semantic regions of an object. We conduct the quantitative evaluation for part-aware representation obtained by pretrained models that is not well explored before, through three part-level recognition tasks, part retricval, part classification, and part segmentation.
Part retrieval. We conduct part retrieval experiments on two datasets, 2011) and COCO (Lin et al., 2014). We build the part patch databases by cropping the patches centered at the keypoint. We consider four and three keypoints from the two datasets, respectively. For each keypoint, find the minimum 1.2 distance (d) from the distances between it and all the
(Wah et al.,

---

Under review as a conference paper at ICLR 2023
Table 4: Part-level linear semantic segmentation results on ADE20K-Parr, Pascal-Part and LIP datasets.
ADE20K-Par
Pascal-Part
LIP
Methods
209 Part Classes
193 Part Classes
19 Part Classes
mloU
mAcc
aAce
mloU
mAcc
aAcc
mloU
mAce
aAcc
Supervised Model:
DeiT
27.3
34.7
69.2
27.4
36.1
65.8
41.4
52.6
73.5
Contrastive Learning:
MoCo v3
27.1
34.7
70.1
27.1
35.8
66.0
41.9
53.0
74.5
DINO
28.9
36.8
70.3
27.8
36.5
66.4
41.0
51.9
74.0
Masked Image Modeling (MIM):
BETT
18.6
25.8
58.2
14.8
21.4
47.0
27.2
36.5
60.1
26.3
35.0
67.3
24.3
32.9
61.5
38.2
48.7
71.3
CAE
28.4
36.9
71.1
27.8
37.0
66.3
43.7
75.9
Contrastive Learning MIM:
IBOT
32.2
40.0
73.4
30.7
40.0
69.7
44.6
55.7
76.6
other keypoints in the same image, then crop a d x d patch centered at this keypoint and resize it to 224 x 224. We use the cosine distance the patch distance and evaluate the retrieval performance using average precision (AP) as the retrieval metric.
The results provided in Table 3. We have the following observations. 1) Self-supervised models except BEiT and MAE outperform the supervised model DeiT, indicating the capability that contrastive learning and CAE learn part-aware representations. BEiT and MAE perform inferior, consistent to the observations in ImageNet classification in Table 1. 2) iBOT performs the best, and the reason might be that the capability of learning part-aware representations is boosted by making use of both contrastive learning and masked image modeling.
We also report the part retrieval performance of the projected representations of contrastive learning methods in Table 3. The performance is much lower than the encoded representations. This provides an extra evidence for the part-to-whole hypothesis of contrastive learning: the projected representations are more about the whole object.
Part classification. We further conduct part classification experiments on the datasets used for part retrieval. We consider two kinds of extra learnable layers, linear probing and attentive probing, for classification. The results in Table 3 show that: 1) While DeiT performs the best in the image classification task (see Table 1), for part classification, contrastive-basel methods like MoCo v3, DINO, and iBOT outperform DeiT by more than 2% under both linear and attentive probing settings. 2) Though MIM-based models CAE and MAE inferior to DeiT in object-level classification (e.g.. more than 10% and 4% lower in linear and attentive probing), they show competitive performance in linear probing and higher results than DeiT in attentive probing, demonstrating they learn better part- aware representations. 3) BEiT is inferior to other works, and iBOT has good performance, implying that the probing quality of pretrained encoders is good indicator for downstream performance.
Part segmentation. We perform part-level linear semantic segmentation to study the finer-grained part representation modeling capability of different pretraining paradigms on three widely used datasets: ADE20K-Part (Zhou et al., 2019) containing 209 parts from the ADE20K dataset (Zhou al., 2019). Pascal-Paas Part (Chen et al., 2014) including 193 part categories, and LIP (Gong al., 2017) consisting of 19 semantic human part labels. Similar to the object-level semantic segmentation experiments, linear evaluation is employed here. We maintain the same training protocols for all methods for fair comparisons. See Appendix A.2 and A.4 for dataset and training details.
The results are reported in Table with the following observations. 1) Contrastive learning models, ie.. MoCo v3 and DINO, achieve competitive performance with the supervised model DeiT: DINO outperforms DeiT on ADE20K-PPI
and Pascal-Part Part, and MoCo v3 outperforms DeiT on LIP 2) The MIM model CAE, outperforms DeiT by large margins all three datasets, e.g., 1.1% ADE20K Part and 2.3% on LIP, indicating CAE learns good part-aware representations. Similar to part retrieval, possibly due to pretraining quality in representation encoding, BEiT and MAE perform
8

---

Under review as a conference paper at ICLR 2023
ADEeoK-Object
ADE20K-Part
Pascal-Object
Pascal-Part
30
30
mloU (%)
27
27
24
24
21
21
DefT
MoCo v3
DINO
MAE
CAE
iBOT
Defl
MoCo v.
DINO
MAE
CAE
iBOT
Deff
Mc v3
DINO
MAE
CAE
iBOT
DefT
MoCo vd
DINO MAE
CAE
iBOT
Figure 6: Comparisons between object-level and part-level semantic segmentation ADE20K and Pascal-Part datasets. Though the supervised DeiT's is superior over self-supervised models (i.e.. MoCo v3, DINO, MAE, CAE) object-level segmentation, is generally inferior self-supervised models on part segmentation, demonstrating self-supervised methods learn good part-aware representations. iBOT enjoys the benefits of contrastive learning and MIM. See Appendix A.3 for detailed results.
inferior. 3) Compared with object-level segmentation results in Table DeiT learns better object-level semantics by explicit supervision than both contrastive learning and MIM, however, it is generally inferior to self-supervised models on part segmentation. 4) The model iBOT, which leverages both contrastive learning and MIM, outperforms all other works three datasets, demonstrating its powerful capability in learning finer part-level semantics. Combining the two self-supervised learning techniques is thus promising direction.
In summary, we show that self-supervised methods are potentially capable of learning part-aware representations. Among them, CAE is representative MIM work, showing good performance by explicitly predicting the encoded representations of the masked patches the encoding space; contrastive learning methods MoCo v3 and DINO outperform BEiT and MAE; and iBOT performs the best by combining contrastive learning and MIM. The observations are evidenced by three part-based segmentation benchmarks consistently.
4.3 OBSERVATION SUMMARY BETWEEN OBJECT-LEVEL AND PART-LEVEL. SEGMENTATION
We conduct both object-level and part-level linear semantic segmentation on different hierarchies of the same dataset. Considering that the 209 classes in ADE20K-Part are basically chosen from 59 object classes, we denote the 59-object dataset as ADE20K-Object Similarly, Pascal-Object consists of 16 object categories, corresponding to the 193 part categories in Pascal-Part.
The results in Figure show that: although the supervised DeiT is superior over contrastive learning and masked image modeling methods generally inferior to self-supervised models on ADE20K-Par and Pascal-Part, demonstrating self- supervised methods can learn good part-aware representations. Similar observations could be found from the object classification in Table and part classification in Table 3.
ADE20K-Objer and Pascal-Object except iBOT, it is
In comparison to contrastive learning, CAE shows stronger capability of learning part-aware representations, and weaker capability of learning object-level semantics. The superiority of iBOT, a combination of contrastive learning and masked image modeling, demonstrates that enjoys the benefits of contrastive learning and masked image modeling.
5 CONCLUSION
We attempt to study the capability of learning part-aware representations of self-supervieed repre- sentation pretraining methods. We provide speculations for contrastive learning and masked image modeling: -hht- and part-to-part with empirical results justifying the speculations. Our study presents aspect to understand what self-supervised representation pretraining methods learn.
Future work. The strong capability of part-aware representation learning is one of the properties of self-supervised pretraining. There should be other characteristics that are leaved the future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hangbo Bao, Li Dong, and Furu Wei. arXiv:2 0825444 2021.
BEÎT: BERT pre-training of image transformers.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-iivaraanc--vvaaiac owg.oooaaaariineeeggggaiaiiiiooi for self--uue learning. arXiv preprint 2100::20000049066 2021.
Suzanna Becker and Geoffrey F. Hinton. A self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355(6356):111-163,, 1992.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, pp. 1209-1218, 2018.
Shuhao Cao, Peng Xu, and David Clifton. How to understand masked autoencoders. arXiv preprint arXiv:2202.03667, 2022.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09988 098822 2020.
Mathilde Caron, Hugo Touvron. Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. ICCV, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ICML, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020.
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, pp. 1971-1978, 2014.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprint arXiv:2202.03022 2022a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, pp. 15750-15758, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, pp. 9640-9649, 2021.
Yubei Chen, Adrien Bardes, Zengyi Li, and Yann LeCun. Intra-instance vicreg: Bag of self-supervised image patch embedding. arXiv preprint arXiv:2206.08955 2022b.
MMSegmennation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https: / com/ caamm//penmmmmt ab/ www.llaammmeeeeenttaaa ion, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255. Icee, 2009.
Xiaoyi Dong, Jiammin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. arXiv preprini arXiv:2111.
2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR. OpenReview.net, 2021.
Alaaeldin El-Nouby. Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2112. 10740, 2021.
10

---

Under review as a conference paper at ICLR 2023
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results. http://www.pascal. 2010.
Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin. Look into person: Self- supervised structure-sensitive learning and a new benchmark for human parsing. In CVPR, July 2017.
Jean-Bastien Grill, Florian Strub, Florent Aitché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.00 07733, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.
Xiangwen Kong and Xiangyu Zhang. Understanding masked image modeling via learning occlusion invariant feature. arXiv preprint arXiv:2208. 04164, 2022.
Zhaowen Li, Zhiyang Chen, Fan Yang. Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. Mst: Masked self-supervised transformer for visual representation. Advances in Neural Information Processing Systems, 34:13165-13176, 2021.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755. Springer, 2014.
Xingbin Liu, Jinghao Zhou, Tao Kong, Xianming Lin, and Rongrong Ji. Exploring target representa- tions for masked autoencoders. arXiv preprint arXiv:2209.039/1 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 10012-10022, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ICML. pp. 8821-8831. PMLR, 2021.
Nikunj Saunshi, Jordan T. Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham M. Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. CoRR, Abb/2200..14037, 2022. URL https: /aats:///rxivii aariiviiii. org 2220 14037.
Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision representation learning. arXiv preprint wrXiv2200.001244, 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. arXiv preprint arXiv:2012.12877, 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive lcarning rivals masked image modeling in fine-tuning via feature distillation. 2022.
arXiv preprint
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai. Yue Cao, and Han Hu. Self-supervised learning with swin transformers, arXiv preprint arXiv:2105.0455 202
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint arXiv: 20t::2111.990 1909886. 2021b.
11

---

Under review as a conference paper at ICLR 2023
Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang. Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. CoRR, 0ttp/22055135453 2022a. doi: 10.48550//rX1v.22055 13543. URL https: / /oi..ii..////////../ wttpo....g........ arrivv 2205. 13543..
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Yixuan Wei, Qi Dai, and Han Hu. On data scaling in masked image modeling. CoRR, abs/2206.04666 2022b. doi: 10.485500a2Xxv.2206..0k URL https: /ww./i..oooo owg/0...g....... 480.44555//// wtt.s//aaxi...o/ 2206. 04664.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny, Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2 2021.
Yuanyi Zhong, Haoran Tang, Junkun Chen, Jian Peng, and Yu-Xiong Wang. Is self- learning more robust than supervised learning? arXiv preprint arXiv:2206.00255, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 127(3):02--21, 2019.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:21 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 MODEL. DESCRIPTION
For all the models involved in the experiments including DeiTT (Touvron et al., 2020), MoCo v3 (Chen al., 2021), DINO (Caron al., 2021), BEiT (Bao et al., 2021), MAE (He et al., 2021), CAE (Chen et al., 2022a), and iBOT (Zhou et al., 2021), we use their official code to implement the encoders. It worth noticing that for DINO and iBOT, we choose the checkpoint of the teacher models as they have been reported to perform better than the student models in their papers (Caron et al., 2021; Zhou al., 2021).
A.2 DATASETS
ADE20K (Zhou et al., 2019) is one of the most challenging benchmarks, containing 150 fine-grained semantic concepts and variety of scenes with 1,038 image-level labels. There 20,210 images in the training set and 2,000 images in the validation set. We choose 59 out of total 150 semantic concepts that are concrete objects containing parts (Zhou et al., 2019), termed ADE20K-Objes We also select 209 part categories that emerge both in the training set and the validation set, called ADE20K-Paas
Pascal-Part (Chen et al.. 2014) is a set of additional annotations for PASCAL VOC 2010 (Everingham al., 2010), thereby holding the same statisties as those of PASCAL VOC 2010. It provides segmentation masks for each part of objects. Concretely, the dataset includes 20 object-level categories and 193 part-level categories. our experiments, we remove object categories that do not contain parts including boat, table, chair, and sofa.
LIP (Gong al., 2017) large-scale benchmark for human parsing research, which includes 50,462 images with pixel-wise annotations on 19 semantic part labels. In detail, it includes 19,081 full-body images, 13,672 upper-body images, 403 lower-body images, 3,386 head-missed images, 2,778 back-view images and 21,028 images with occlusions. There 30,462 images in the training set and 10,000 images in the validation set. The rest 10,000 images are served as the test set with missing labels for competition evaluation.
CUB-200-2011 (Wah et al., 2011) is a popular benchmark for fine-grained image classification, and also provides bounding box and part location annotations. It contains 11,788 images of 200 bird species and 15 part keypoint annotations per bird. In this work, we mainly leverage its part keypoint annotations. And only 4 part categories (right eye, right leg, left wing, and tail) are chosen to be considered in our experiments, to make sure that the selected keypoints are far enough away from each other and enough context information can be contained in the cropped patches. (We also tried using all keypoints and the conclusion is consistent.)
COCO (Caesar et al., 2018), as one of the most widely-used human pose estimation datasets, contains more than 200,000 images and 250,000 labeled person instances. Similar to CUB-200-201 mentioned above, only (nose, right wrist, and left ankle) of its 17 keypoint categories are considered in our experiments.
A.3 DETAILED RESULTS FOR ORJECT-LEVEL. AND PART-LEVEL SEGMENTATION
In this section, provide detailed comparisons between object-level and part-level semantic seg- mentation in Table and Table 6. Similar observations as in Figure 6 in the main paper are found: although the supervised DeiT is superior over self-supervised methods on ADE20K-Objec and Pascal-Object except iBOT, it is generally inferior to self-supervised models on ADE20K-PA Part and Pascal-Part demonstrating self supervseed methods can learn good part- aware representations. BEÎT and MAE perform inferior, perhaps because the two methods do not have an explicit process to predict the encoded representations of masked patches, instead, directly reconstruct the targets.
A.4 EXPERIMENT DETAILS
Part retrieval. In our part retrieval experiments, we directly use the pretrained encoders to extract features, without additional training process. For each method, take the hetter from the class
13

---

Under review as a conference paper at ICLR 2023
Table 5: Linear semantic segmentation results on ADE20K-Objec and ADE20K-Part.
Object Seg on ADE20K-Objec
Part Seg on ADE20K-
Methods
Object Classes
Part Classes
mloU
mAce
aAcc
mloU
mAce
aAcc
Supervised Model:
DeiT
52.6
62.9
83.8
27.3
34.7
69.2
Contrustive Learning:
MoCo v3
50.2
60.4
83.6
27.1
34.7
70.1
DINO
50.8
60.8
83.9
28.9
36.8
70.3
Masked Image Modeling (MIM):
BEiT
28.6
37.2
73.4
18.6
25.8
58.2
MAE
41.0
50.6
79.9
26.3
35.0
67.3
CAE
47.4
58.4
82.9
28.4
36.9
71.1
Contrastive Learning + MIM:
iBOT
55.2
65.1
85.6
32.2
40.0
73.4
Table 6: Linear semantic segmentation results on Pascal-Object and Pascal-Part
Object Seg on Pascal-Object
Part Seg on Pascal-Part
Methods
16 Object Classes
193 Part Classes
mloU
mAce
aAcc
mloU
mAce
aAcc
Supervised Model:
DeiT
92.2
95.3
96.8
27.4
36.2
65.8
Contrastive Learning:
MoCo
89
93.7
95.7
27.1
35.8
66.0
DINO
88.0
92.7
95.3
27.8
36.5
66.4
Masked Image Modeling (MIM):
BETT
56.4
69.0
76.8
14.8
21.4
47.0
MAE
76.1
84.6
89.5
24.3
32.9
61.5
CAE
83.3
89.7
93.2
27.8
37.0
66.3
Contrastive Learning MIM:
iBOT
92.1
95.3
97.1
30.7
40.0
69.7
token or the average embedding of all patch tokens as the extracted representation. With each patch the query patch, calculate the cosine similarity between its representation and all the other patches' in the dataset and utilize the average precision (AP) as the retrieval metric. Finally, we average all the obtained AP scores (with all patches respectively taken the query patch for retrieval) as the final retrieval score of the method.
Apart from the part retrieval experiments shown in Table the visualized patch retrieval results in Figures/ and are obtained based on ImageNet (Deng et al., 2009) validation set. Concretely, from each pre-processed 224x224 validation image in ImageNet, we uniformly crop 49 patches sized 56x56 using stride of 28. With all the cropped patches from the validation set, we select one patch as query and find top 24 patches with the highest cosine similarity with it.
Part classification. For linear probing, we learn supervised linear classification layer on the extracted class token of the frozen encoders. While for attentive probing, following Chen al. (2022a), cross attention module and batch normalization layer without affine transformation are additionally inserted between the encoder and the linear classifier. And a new learnable class token is taken the query of the cross attention module, to replace the original class token extracted by the frozen encoder. We use SGD optimizer with a learning rate 0.4 and 0.04 for linear probing and attentive probing, respectively. For both linear probing and attentive probing, the models trained for 90 epochs. And the momentum of SGD is set to 0.9, the weight decay is set to 0, and the batch size is set to 1024.
14

---

Under review as a conference paper at ICLR 2023
DeiT
MoCo
DINO
MAE
CAE
iBOT
DeiT
MoCo v3
DINO
MAE
CAB
iBOT
Figure 7: Patch retrieval comparisons of encoded representations on cropped patches from ImageNet.
Segmentation. We use the same model structure that contains a parameter- fixed pretrained encoder (e.g., MAE and DeiT) and a simple learnable 1 x 1 convolutional layer for object-level and part-level segmentation tasks. Note that the learning rate (4e - 4), training iterations (160k), and batch size (16) among all the experiments maintain the same during training for fair comparisons. For ADE20K, the input size is set to 512 x 512 following previous works (Bao et al., 2021: He et al., 2021; Chen al., 2022a; Zhou et al., 2021). For Pascal-Part, adopt 480 x 480 image input resolution following Contributors (2020). As for LIP, we use the same input size (320 x 320) proposed in LIP (Gong et al., 2017).
A.S IMAGENET PATCH RETRIEVAL VISUALIZATION
We visualize more patch retrieval results of the encoded representations on the ImageNet validation set Figures and 8. It is observed that the retrieved patches of supe-sueee methods are generally more about the semantics of the query part than that of DeiT. The results demonstrate that the encoded representations of DeiT focus more on object-level semantics, while the encoded representations of these self-supervised methods more about pert- semanties. Among these methods, the retrieved patches of MAE have less semantic correlation but often share similar hues.
15

---

Under review as a conference paper at ICLR 2023
DeiT
MoCo
DINO
MAE
CAE
iBOT
DeiT
MoCo v3
DINO
MAE
CAF
iBOT
DeiT
MoCo
DINO
MAE
CAE
iBOT
DeiT
MoCo v3
DINO
MAE
CAE
iBOT
Figure 8: Patch retrieval comparisons of encoded representations on cropped patches from ImageNet.
16


Reviews:
[{"review_id": "hFOjYM2lg5", "reviewer": "Reviewer_AQH3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to understand the two mainstream self-supervised methods. It speculates that masked image modeling is a part-to-part process and contrastive learning is a part-to-whole process. Both empirical results and qualitative analysis are provided to support the claims.", "summary_of_the_review": "This paper speculates that contrastive learning is a part-to-whole process and MIM is a part-to-part process and self-supervised learning could learn part-aware representations. It provides an innovative perspective to understand self-supervised learning, supported by comprehensive empirical results and qualitative analyses. The paper is well written but some claims are not well supported. I'm also wondering to what extent the conclusions in this paper can help the future model since the different objectives matter more than the training paradigm (contrastive/MIM). I think this paper is a bit above the borderline.", "strength_and_weaknesses": "Strengths:\n1. This paper makes an innovative claim on the current self-supervised learning methods, its conclusion leads people to a more insightful understanding of self-supervised learning.\n2. Authors conduct a comprehensive empirical study on both object-level and part-level tasks to support their claims.\n3. Many qualitative analyses are made, which give a clear and intuitive explanation.\n\nWeaknesses:\n1. For the part segmentation part, the authors' explanation, 'possibly due to pretraining quality in representation encoding, BEiT and MAE perform inferior', is not convincing. I do not think it is safe to draw the conclusion based on only one MIM method CAE. Therefore, whether MIM method can be simply described as a part-to-part process is questionable from the experimental results. \n2. I do not see a clear reason why iBOT performs the best from the authors' explanation. It's unclear why combining part-to-part and part-to-whole processes could generate a better model. I'm afraid that the differences among the models/objectives would matter more than the differences in the training paradigm (contrastive/MIM). This could make the claims in this paper less useful for guidance on future SSL models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_WH78AGc4_a", "reviewer": "Reviewer_nE5Q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an analysis of existing self-supervised learning methods with vision transformers. Extensive experiments are performed on the off-the-shelf pretrained model with the linear and attentive probes to verify the speculation that contrastive learning is a part-to-whole task and masked image modeling is a part-to-part task. ", "summary_of_the_review": "Overall I think this paper goes one step further than CAE and performs an extensive comparison on various benchmarks, which is appreciated. However, some of the experiment results do not align with the authors' claim and the conclusion is somewhat unsurprising and not enlightening. Thus my rating is 5.", "strength_and_weaknesses": "Pros:\n\nThe motivation of this paper is clear, the paper is well-written which makes it easy to follow. The experiment setting makes sense to me.\n\nCons:\n\nThe part-to-part and the part-to-whole difference between masked image modeling and contrastive learning is already mentioned in the CAE paper, which makes the finding and conclusion less interesting and surprising.\nSome experiment results do not align with the claim well, for example, in the part-retrieval and part-segmentation experiments, BEiT and MAE which are MIM models do not perform better than contrastive learning or supervised learning.\nBased on that, no more instructive conclusion about how to better use the property is given (combining two of them like iBOT shouldn't count as a contribution in this paper), which makes the novelty part weak.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DY_Aoqx6Ga", "reviewer": "Reviewer_SGtU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is a study on what is the different when learning with random crops and contrastive learning (CL) in a supervised learning (SL) or self-supervised learning (SSL) way, versus random patch masking via self-supervised learning (SSL) as in the Masked image modeling (MIM) variants (MAE/BeiT/CAE). The authors claim that the former is learning a \"part-to-whole\" task while the latter a \"part-to-part\" task; both can lead to strong part-aware representations, while it seems that SL excels more on object-level tasks, while SSL methods like CAE or MoCo-v3 are able to learn part-aware representations.", "summary_of_the_review": "This paper is at its core an empirical study on part-aware representations; the usefulness of the analysis in section 3 is unclear to me, looking forward for some clarifications from the authors on this. There is no technical novelty, while the insights from the study are also not strong, basically verifying the superiority of iBoT over other SSL methods tested on both object and part tasks. I am looking forward to the author's responses.\n\n--- Post rebuttal update:\n\nI want to thank the authors for constructive discussion. I think iff the authors make the edit they promise to the text  and clarify their contributions, I think I can raise my score to borderline accept. Although limited in novelty, the study in this paper can be valuable to the community, if the contributions are not over-claimed. ", "strength_and_weaknesses": "### Strengths\n\nS1: the paper is studying part-level recognition, a very interesging and important task\n\nS2: The paper features an extensive empirical study on many object-level recognition and part-level recognition tasks. \n\n\n### Weaknesses\n\nW1: It is unclear to me what the part-to-whole vs  part-to-part analysis offers, especially with respect to part-aware representations: Sections 3.1 and 3.2 seem to suggest the same thing: Both CL and MIM, \"[are] potentially capable of learning part-aware representations.\". Are there any insights or interesting experiments that are derived from that analysis and distinction? Cause what i see in Sec 4 is an empirical analysis on which losses can learn part-aware representation, that can stand without section 3, really.\n\nW2: It is unclear to me if the \"part-to-whole\" effect is only a function of the input (random crops) and the contrastive aspect of the loss; I think that what really matters is what is the contrastive loss appled on: If the loss is on aggregated features from the whole crop, I see this making sense. But what is a contrastive loss is applied at the token level?  How would for example DenseCL (Wang, Xinlong, et al. \"Dense contrastive learning for self-supervised visual pre-training.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.) fits in this framework? \n\nW3: The authors are focusing on random crops, which is only one of the augmentations used for CL - yet there are other augmentations at play. A more fair comparison would use an identical augmentation setup apart from either adding random crops or masking. Are all the other augmentations (color jittering, etc) shared across compared methods?\n\nW4: it is unclear to me, eg from the abstract what is the focus of the study, as I see two  axes that are convoluted: MIM vs CL and SL versus SSL methods. The abstract highlights the first axis, so does Sec 3, but then Figure 2 shows that both CAE (MIM) and MoCov3 (Contrastive) can give part-aware representations, while DeiT (SL) cannot. I think that leaving SL out of the study would make it clearer.\n\nW5: the insights from the empirical study are not analysed enough and can in places seems exargerated:, eg \"This implies that in general the self-supervised models are not strong at object-level understanding,\" this is a very generic statement to make from the results in Table 2. \n\nQ1: Are you retraining all the models under some otherwise identical setup, or using the publicly available models out of the box? Overall, three are many differences between the models compared beyond the loss (from batchsizes, to augmentations, to optimizers to other ViT-related tricks used in different papers). \n\nQ2: Although figure 1 and 3 is mostly figurative it seems to me that it exaggerates the effects of the random crops used in practice. Apart from methods that use local crops as in multi-crop, the scale parameter for the random crops is usually not as large as the one used in this figure. What is the crop scale used in all methods, ie what is the percentage of the max width/height that each readnom crop uses?\n\nQ3: Where are the patches seen in Fig 2 and 4 from? Imagenet?\n\nN1: I wonder how things would change for SL if a more recent variant of the DeiT series was used, e.g. Deit III from \nTouvron, Hugo, Matthieu Cord, and Hervé Jégou. \"Deit iii: Revenge of the vit.\" arXiv preprint arXiv:2204.07118 (2022).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject