Understanding Self-Supervised Pretraining with Part-Aware Representation Learning
Authors: Jie Zhu
Jiyang Qi
Mingyu Ding
Xiaokang Chen
Ping Luo
Xinggang Wang
Wenyu Liu
Leye Wang
Jingdong Wang
Venue: Accepted by TMLR ()

Abstract: In this paper, we are interested in understanding self-supervised pretraining through studying the capability that self-supervised methods learn part-aware representations. The study is mainly motivated by that random views, used in contrastive learning, and random masked (visible) patches, used in masked image modeling, are often about object parts. 

We explain that contrastive learning is a part-to-whole task: the projection layer hallucinates the whole object representation from the object part representation learned from the encoder, and that masked image modeling is a part-to-part task: the masked patches of the object are hallucinated from the visible patches. The explanation suggests that the self-supervised pretrained encoder leans toward understanding the object part. We empirically compare the off-the-shelf encoders pretrained with several representative methods on object-level recognition and part-level recognition. The results show that the fully-supervised model outperforms self-supervised models for object-level recognition, and most self-supervised contrastive learning and masked image modeling methods outperform the fully-supervised method for part-level recognition. It is observed that the combination of contrastive learning and masked image modeling further improves the performance.

Published in Transactions on Machine Learning Research (07/2023)
Understanding Self-Supervised Pretraining with Part-Aware Representation Learning
Jie Zh Jiyang Qi Mingyu Ding1.6r, Xiaokang Chen'; Ping Luo Xinggang Wang", Wenyu Liu', Leye Wang Jingdong Wang Shujie@stuu plu. edu.cn, (jyanggi, zgweng, liwwy) @hust. edu, en, myding@berkeley.eee (pkuesk, leyewang) (agewwnng@pp....oom edu.en, en, aaoo@cs.hku.kee wenggingtong@outlook com
@p...edu.cn,
Key Lab of High Confidence Software Technologies (Peking University). Ministry of Education, China School of Computer Science, Peking University, 3 School of AI, Peking University. UC Berkeley School of EIC, Huazhong University of Science & Technology, " University of Hong Kong, Baadu
Reviewed on OpenReview: https: // openrevieww forum? id= HP7Qpui15
Abstract
In this paper, we are interested in understanding self-supervised pretraining through studying the capability that self-supervised methods learn part-aware representations. The study is mainly motivated by that random viows, used in contrastive lcarning, and random masked (visible) patches, used in masked image modeling, are often about object parts.
We explain that contrastive learning part-lo-whole task: the projection layer hallucinates the whole object representation from the object part representation learned from the encoder, and that masked image modeling is part-to-part task: the masked patches of the object are hallucinated from the visible patches. The explanation suggests that the self-supervised pretrained encoder leans toward understanding the object part. We empirically compare the off-the-shelf encoders pretrained with several representative methods on object-level recognition and part-level recognition. The results show that the sully-supervised model outperforms sule-vupe models for object-level recognition, and most self-supervised contrastive learning and masked image modeling methods outperform the fully-supervised method for part-level recognition. It is observed that the combination of contrastive learning and masked image modeling further improves the performance.
1 Introduction
Self-supervised representation pretraining has been attracting a lot of research efforts recently. The goal is train an encoder that maps an image representation from visual contents without the necessity of human annotation. Through this, it is expected that the encoder benefits the downstream tasks, e.g., segmentation and detection.
There are two main frameworks: contrastive learning' and masked image modeling. Contrastive learning aims to maximize the agreement of the embeddings of random augmented views from the same image. Masked image modeling partitions an image into masked patches and visible patches, and makes predictions for masked patches from visible patches. Figure gives examples of random views for contrastive learning and masked and visible patches for masked image modeling.
We observe that random view and set of masked (visible) patches usually contain a portion of an object. It is also reported in self-supervised learning methods, e.g., DINO (Caron et al., 2021) and iBOT (Zhou et al.,
"Equal contribution.
!Corresponding author.
In this paper, use contrastive learning (CT.) to refer random-view based methods, e.g., SimCLR, MoCo, and BYOL.

---

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
(a)
(b)
(c)
(d)
(c)
Figure 1: (a) original image, (b-c) two random crops, and (d-e) masked and visible patches. DeiT
MoCo v3
CAE
Figure 2: Top-24 patch retrieval results with three frozen encoders of DeîT, MoCo v3, and CAE, by taking the patch in the red box as the query. It be seen that the retrieved results from CAE and MoCo v3 are about the object part (wing and dog mouth) and more precise than DeiT (about the whole object) implying that self-supervised pretraining methods, CAE and MoCo v3 are stronger at learning part-aware representations than the fully-supervised method DeïT. Details could be found in Sec. 3.
2021), that different attention heads in ViIs can attend different semantic regions or parts of an object. In light of this, attempt to understand self-supervised pretraining by studying the capability that the pretrained encoder learns part representations.
We present a port-to-whole explanation for typical contrastive learning methods (e.g., SimCLR (Chen et al., 2020), MoCo (Chen et al., 2021), and BYOL (Grill et al., 2020)): the embedding of the whole object is hallucinated from the embedding of the part the object contained in the random crop through a projection layer. Tn this way, embeddings of random crops from the same image naturally agrees with each other. Masked image modeling is a part-to-part process: the embeddings of the masked patches of the object (a part of the object), are hallucinated from the visible patches (the other part of the object).
We empirically compare the supervised model DeiT (Touvron et al., 2020), which serves an important baseline for analyzing SSL models, and typical self-supervised representation pretraining methods, including MoCo v3 (Chen et al., 2021), DINO (Caron et al., 2021), CAE (Chen et al., 2022a), MAE (He et al., 2021), BEIT (Bao et al., 2021), and iBOT (Zhou et al., 2021), on object-level recognition (image classification and object segmentation) and part-level recognition (patch retricval, patch classification, and part segmentation). Figure 2 presents patch retrieval results using the encoders learned through CAE, MoCo v3, and DeiT,, implying that the encoders pretrained by CAE and MoCo v3 able to learn part-aware representations.
Through extensive studics and comparisons, we make the following obscrvations. 1) DeiT outperforms contrastive learning and MIM methods except iBOT in object-level recognition lasks, which may benefit. from its explicit object-level supervision. 2) In contrast, self-supervised methods learn better part-aware representations than DeiT. For example, while DeiT superior DINO and CAE by 0.4% and 2.3% on ADE20K object segmentation, DINO and CAE outperform DeiT by 1.6% and 1.1% on ADE20K part segmentation, respectively. 3) In contrastive learning, the encoder learn part-aware information, while
2

---

Published in Transactions on Machine Learning Research (07/2023)
Encoder
Projector
2g
Predictor
Encoder
X,
Projector
3
Alignment
Figure 3: The pipeline of a lypical contrastive learning approach. Two augmented views, red box and blue box, generated from the original image. The augmented view in red is fed into the encoder and the projector, and then the predictor (which does not appear in carlier works like MoCo (Chen et al., 2021) and SimCLR (Chen et al., 2020)), and the view in blue is fed into the encoder and the projector. The two outputs are expected to be aligned. The gradient is stopped for the bottom stream.
the projected representation tends to be more about the whole object. The evidence could be found in part retrieval experiments on MoCo v3, DINO, and iBOT. 4) The MIM method CAE shows good potential in part-aware representation learning. Interestingly, the method combines contrastive learning and MIM is promising, e-g-, iBOT learns better representations at both object and part levels.
This paper presents the following contributions:
We study the capability of learning part-aware representations way of understanding self- supervised representation pertaining based on both qualitative analysis and empirical results.
We explain masked image modeling as part-to-pant task and contrastive learning as part-to- whole task, and speculate that self-supervised pretraining has the potential for learning part-aware representations.
We empirically compare several pretrained models on object-level and part-level recognition tasks, showing interesting findings with supporting evidence of the capability of part-aware representation learning for self-supervise learning.
2 Related Work
Contrastive learning (CL). Contrastive pretraining has been intense academic field in the CNN era. In this work, use it to refer to methods for comparing random views (Caron et al., 2020; Chen et al., 2020; Zbontar et al., 2021; Xie et al., 2021b; Chen et al., 2021; Caron et al., 2021), including some instance discrimination work such as (Grill et al., 2020; Chen & Нe, 2021; Bardes et al., 2021). A representative work, SimCLR (Chen et al., 2020) learns representations through maximizing agreement between different views of the same image the latent space. BYOL (Grill et al., 2020) uses two asymmetrical networks bootstrap latent representation without negative samples involved during the interaction. As vision transformer (ViT) (Dosovitskiy et al., 2021) shows excellent performance via supervised lcarning, it is adopted subsequently in contrastive pertaining, and numerous outstanding works are proposed. For example, MoCo v3 (Chen et al., 2021) observes the hidden instability while training self-supervised VIT and solves it by using a fixed random patch projection. DINO (Caron et al., 2021) explores new properties derived from self-supervised ViT and accordingly designs a learning strategy interpreted as a form of self-distillation with no labels.
Masked image modeling (MIM). Masked image modeling is another self-supervised pretraining paradigm that attracts much attention recently. BETT (Bao et al., 2021) follows masked language modeling in the natural language process (NLP) area and predicts tokens via mapping image patches by d-VAE (Ramesh et al., 2021). PcCo (Dong et al., 2021) ) boosts BETT by taking into consideration more semantic information in visual lokens. MAE (He et al., 2021) learns rich hidden information by directly performing masked image reconstruction in RGB color space using ViT while SimMIM (Xie et al., 2021c) uses twin-trransfomer (Liu al., 2021). CAE (Chen et al., 2022a) adds a regressor between encoder and docoder, which designed align unmasked patches with masked ones, leading to a pure context encoder. Recently, a trend that combines MTM with siamese frameworks has surfaced and showed encouraging results including MST (Li
3

---

Published in Transactions on Machine Learning Research (07/2023)
et al., 2021), SalitMask (El-Nouby et al,, 2021), iBOT (Zhou et al,, 2021), dBOT (Liu et al, 2022), and SIM (Tao et al., 2022).
Understanding self-supervised contrastive pretraining. The studies on understanding contrastive pretraining (Saunshi et al., 2022; Chen et al., 2022b; Zhong et al., 2022; Wei et al., 2022) mainly focus on random augmentations (views), contrastive loss function and its variants under the assumption that: the augmentations of inputs from the same class have significant overlap in the representation space, but there is little overlap for imputs from different classes. Our work is complementary to these studies. Inspired by the observation that random views usually contain portion of an object, and methods (Caron et al., 2021; Zhou et al., 2021) show that different attention heads in ViTs can attend to different semantic regions of an object, we investigate what the encoder and the projector do in typical self-supervised contrastive pretraining. We speculate that the pretraining task is a part-to-wiie whole problem, predicting the representation of the whole object through the projector from the representation (obtained from the encoder) of the part of an object. We use empirical results to verify our analysis.
Understanding self-supervised masked image modeling. The comparison of attention in different layers between the pretrained models from MIM and the supervised approach is conducted: MIM pretraining brings locality to the trained model with sufficient diversity on the attention heads (Xie et al., 2022a).. Consistent with the analysis in NLP, empirical studies are conducted in Xie et al. (2022b) to verify that MTM benefits from larger models, more data, and longer training. CAE (Chen et al., 2022a) gives the comparison between contrastive and MIM and shows MIM cares about all patches and thus achieves better results for fine-tuning. Cao et al. (2022) provides a mathematical understanding of MIM. Kong & Zhang (2022) points out that the learned occlusion invariant feature contributes to the success of MTM. In this work, we speculate that masked image modeling is pert-to-part process: the embeddings of the masked part of the object are hallucinated from the visible part using the position information of the masked patches, leading to better part-aware representation than the supervised model DeiT (Touvron et al., 2020). More discussions can be found in Appendix A.8.
3
Understanding Contrastive Learning and Mask Image Modeling
In this section, we briefly review representative formulations of CL and MIM, and provide qualitative analysis of their part-aware explanations.
3.1 Contrastive Learning
Contrastive learning aims to learn the encoder through maximizing the agreement between differently augmented views of the same image in the representation space. An example pipcline is depicted in Figure 3. Given an image I, the augmentations, e.g., random cropping, random color distortion, and random Gaussian blur, are applied to generate a set of N augmented views, (V1, V2,".. ,. An augmented view V, is fed into an encoder Encoder, generating the encoded representation Xn, and followed by a projector, generating the projection zn. The basic goal is to maximize the agreement between the projections (z1, Ze, ..,,, i.e., minimize the loss
N E ((rroeedo((Bncoder())) Projector((nccoee(()))) LCPT ΣM
(1)
In the formulation with contrastive loss, the agreement between the projections of random augmentations from different images is minimized.
Part-to-whole prediction explanation. Let us consider two crops randomly sampled from the original image (see the examples given in Figure ((bc)). The encoded representation of the first crop is expected to describe a part of the object dog; the encoded representation of the second crop is expected to describe another part of the object dog?. The two representations are related but different. Contrastive lcarning methods project the two encoded representations into two projected representations that are expected to
IIn this paper, we mainly study the capability of learning representations about objects and parts, and leave the study of representations of the background the future work.
4

---

Published in Transactions on Machine Learning Research (07/2023)
Encoded representation
Projected representation
Figure 4: Illustration of patch search results using encoded representations and projections (pretrained with MoCo v3 as). Left: patch search results with encoded representations. Right: patch search results with projections. each result, the small patch encireled by the red box taken the query. It can be seen that for encoded representations, the returned patches are about the same part, and for projections, the result. patches about the same object, verifying the trr-tt--whoee hypothesis.
agree. We hypothesize that the projection process maps the encoded part representation to the representation of the whole objectt. Through this way, the projected representations will agree to different views from the same image* 4, It is assumed that the part-to-whole projection is more reliable the encoded representation is semantically richer and is able to describe the part information. The part-to-whole process suggests that the encoder pretrained by contrastive learning methods is potentially capable of learning part-aware representations.
Figure 4 provides patch search results of a representative contrastive learning method MoCo v3 (Chen et al., 2021) based on the encoded representations before and after the projections. The visualized patch retrieval results in Figure
well as those Figure 2 are obtained based on ImageNet (Deng et al., 2009) validation set. Concretely, we uniformly crop 49 patches sized 56x56 using stride of 28 from each pre-processed 224x224 validation image in ImageNet. With all the cropped patches from the validation set, select one patch as query and find the top 24 patches with the highest cosine similarity with it.
One can see Figure 4 that the results through the encoded representations are mainly about the local part, and the results through the projections tend to include the other parts of the same object. In other words, the projections tend to be about the whole object. The search results verify the part-to-whole hypothesis. Similar observations are also shown in Chen et al. (2022b) where the embedding space tends preserve more equivariance and locality, while the projection has more invariance.
3.2 Masked Image Modeling
Mask image modeling the task of predicting some parts of an image from the remaining parts. An augmented view of an image partitioned into patches, R (R1, R2,..., Rar. The task is to predict subset of patches Rm, named masked patches, from the remaining patches R,, named visible patches. Considering contrastive learning that explicitly compares representations of random views, we take context autoencoder
"It said that different parts have common in the external world (Bocker & Hinton, 1992). Our hypothesis is that the common cause is the whole object.
Foor methods that use local-gion globall crops, e.g., DINO, contains range of 0.05 0.4 for local crops and 0.4 for global crops, which our hypothesis are expected describe smaller and larger parts the object, respectively.
5

---

Published in Transactions on Machine Learning Research (07/2023)
(b)
X
Latent contextual regresor
Xm
Encoder
Decoder
X
Encoder
Aligument
Visible patch
Masked patch
LI
Mask query
Targot
Figure 5: The pipeline of an MIM approach, context autoencoder (CAE). An augmented view (in blue) of the image is partitioned into visible and masked patches. The CAE approach feeds visible patches into the encoder and extracts their representations X, and then completes the pretext task by predicting the representations Xm the masked patches from the visible patches in the encoded representation space with latent contextual regressor and alignment constraint, and mapping predicted representations Xmm of masked patches the targets. The pretrained encoder in (a) is applicd downstrcam tasks by simply replacing the pretext lask part (b, c) with the downstream task completion part.
(CAE) (Chen et al., 2022a) as an example that explicitly predicts the encoded representations of the masked patches from the encoded representations of the visible patches".
One goal of CAF (illustrated in Figure 5), which we call masked representation modeling (MRM), is to maximize the agreement between the predictions of the representations of masked patches (through a regressor) and the representation of masked patches computed from the encoder by minimizing the loss
fMRM (Regressor (Ene:ssoor((noco)((R))))) (nn))
Here, we do not include the positional embeddings of masked and visible patches for clarity. It is noted that MRM differs from contrastive learning: MRM does not compare multiple random views, but compares the regressed representations for masked patches and the encoded representations of masked patches. In addition, there is another loss for larget prediction (reconstruuction) for the masked patches, which is commonly used in masked image modeling (MIM) methods:
fam (Decoder (Regressor (Enooosro)E(c))r)(R))))))).. Target
(3)
where (am) is a function map the masked patches the targets, e.g., d-VAE (Ramesh et al., 2021) token used in CAE and BeiT (Bao et al., 2021), or normalized RGB values used in MAE (He et al., 2021).
Part-to-part prediction explanation. The masked image modeling approaches, including CAE, MAF, and BETT, make use of the positions of masked patches for making predictions for masked patches from visible patches. The visible patches and masked patches often contain different parts of an object. In other words, MTM aims to predict the masked part an object from the visible part. We name this a part-ta-port process. There are two part-to-part tasks: one to reconstruct the part targets from the visible part represcntations (MAE and CAE) or from the visible part raw pixels (BEIT), and the other is regress the masked part representations (CAE). The part-to-pate process suggests that the encoder pretrained by MIM methods is potentially capable of learning part-aware representations. Figure 2 illustrates the capability with the patch retrieval results. More visualizations can be found in Appendix A.9.
4 Experiments
We study seven representative methods with the same VIT-B encoder, including supervised method DeiT (Touvron et al., 2020) that serves as an important baseline for analyzing SSL models, contrastive learning methods MoCo v3 (Chen et al., 2021), DINO (Caron et al., 2021); masked image modeling (MIM) methods BEiT (Bao et al., 2021), MAE (He et al., 2021), and CAE (Chen et al., 2022a); and iBOT (Zhou
Some MIM methods, such BEIT (Bao al., 2021) and MAE (Masked Autoencoder) (He al., 2021) do have explicit process predict the encoded representations of masked patches, instead, directly reconstruct. the targets.

---

Published in Transactions on Machine Learning Research (07 2003))
Table 1: Top-1 accuracy with linear probing, and attentive probing (Chen et al., 2022a), on the ImageNet classification benchmark (Deng et al., 2009).
Table 2: Object retrieval results on CIFAR10 and the cropped object of COCO. The "Encoded" and "Pro- jected refer to the encoded and projected representa- tions.
Method
Linear
Attentive
Object Retrieval (AP)
Methods
CIFARIO
COCO
Supervised Model:
Encoded
Projected
Encoded
Projected
Deif
81.8
81.8
Superoised Model:
Contrastive Learning:
Dei'T
71.3
71.2
MoCo v3
76.2
77.0
Contrastive Learning:
DINO
77.3
77.8
MoCo v3
40.6
56.0
47.7
55.9
Masked Image Modeling (MIM):
DINO
31.3
61.5
38.0
57.3
BEIT
41.8
51.9
Masked Image Modeling (MIM):
MAE
67.8
74.2
BEIT
14.6
11.7
CAE
70.4
77.1
MAE 15.9
Contrastive Learning + MIM:
CAE
39.1
46.0
IBOT
79.5
79.8
Controstive Learning + MIM:
iBOT
40.2
60.0
53.2
55.8
et al., 2021) that combines contrastive learning and MIM. We take the training epochs specified in each work to ensure that all comparod models are properly trained and use publicly available checkpoints 6 300 for DeiT, 300 (600") for MoCo v3, 400 (1600") for DINO and iBOT, 800 for BEÏT, and 1600 for MAE and CAE. Frozen encoders are used in all experiments to understand what these different representation pretraining methods learn. More details about models can be found in Appendix A.1.
4.1 Object-Level Recognition
We benchmark three widely-studied object-level recognition, i.e., image classification, object retrieval, and semantic segmentation to show the capability that the pretrained encoder learns object-level representations.
Image classification. We report the linear probing, and attentive probing results of the selected models on ImageNet (Deng et al., 2009). For attentive probing, we follow the protocol in CAE (Chen et al., 2022a) that append a pross-attention layer together with a batch normalization layer and a linear classifier.
We have the following observations from Table self-supervised models object-level recognition. The models that leverage contrastive learning, i.e., MoCo, DINO, and iBOT, show superior linear probing performance than MIM-based models, demonstrating they contain more object-aware high-level semantics. @ MIM-based models, e.g., CAE, show inferior results in linear probing while competitive results with contrastive-based methods in attentive probing. The reason might be that MIM is capable of altending to all the regions, including non-object regions in an image, thus needs a spatial feature selection step to attend to the object part, which pointed out in Chen et al. (2022a). BEIT and MAE perform inferior in lincar and attentive probes, implying that the two methods are less capable of learning semantics.
The supervised model, DeiT performs better than
Object-level retrieval. We evaluate the object-level retrieval performance on two datasets, i.e., Cl- FAR10 (Krizhevsky et al., 2009) and COCO (Lin et al., 2014). For CIFAR10, we directly use each image (resized to 224 x 224) to perform the retrieval task. For COCO, we build the retrieval database by leveraging the bounding box (that is approximately square) in the annotation file to crop the objects and resizing to 224 X 224. The retrieval results are reported Table 2. One can see that DeiT' obtains the best performance over sule-supervieed methods in both datasets, showing stronger object-aware capability. Besides, we also notice that for part-to-whole SSL methods, e.g., MoCo v3, the projected feature has higher accuracy than the
"Considering that the training protocols of different models vary widely in most respects (c.g., batch size, augmentation, and ViT-rclated tricks to prevent training instability and crashes), it is unrensonable to use the identical sctup. For example, DINO centering and sharpening strategy avoid collapse.
The number effective training epochs introduced in Zhou (2021).
7

---

Published in Transactions on Machine Learning Research (07/2023)
encoded one, possibly because the projection boosts the object-aware representations, which further supports our part-to-whole hesothesis.
Object-level semantic segmentation. We perform lin- car evaluation on ADE20K (Zhou et al., 2019) to show the object-level semantic capabilities of the pretrained models. A 4x bilinear interpolation and single 1x] convolutional layer for pixel labeling are attached to the frozen encoder. The learning rate (4e - 4), training iterations (160k), and batch size (16) among all the experiments maintain the same during training for fair comparisons. The input size is set to 512 x 512 following previous works (Bao et al., 2021; He et al., 2021; Chen al., 2022a; Zhou al., 2021).
Table Linear evaluation of ADE20K (Zhou et al., 2019) object-level semantic segmentation (150 classes) using 4x upsampling and a single 1 convolutional layer on frozen backbones.
Method
mloU
mAcc
aAcc
Supervised Model:
DeiT
34.9
44.2
75.4
Contrastive Learning:
MoCo v3
34.7
43.9
75.9
DINO
34.5
43.5
76.1
We see from Table that the supervised model DeiT outperforms all self-supervised models except iBOT, including contrastive learning and MIM methods on ADE20K object-level segmentation. This implics that these self-supervised models are not strong at object-level understanding, which is consistent with the observations for image classification. iBOT (Zhou et al., 2021), as a combination of contrastive learning and MIM, shows surprisingly better performance than the supervised model DeiT on ADE20K, implying the power of combining contrastive learning and masked image modeling for downstream tasks.
Masked Image Modeling (MIM):
BETT
17.8
23.7
64.9
MAE
27.1
34.8
71.6
CAE
32.6
42.2
75.2
Contrastive Learning + MIM:
iBOT
38.3
47.4
78.1
4.2 Part-Level Recognition
Self-supervised methods like iBOT (Zhou et al., 2021) and DINO (Caron et al., 2021) qualitatively show that different attention heads in ViT's can attend to different semantic regions of an object. We conduct the quantitative ovaluation for part-aware representation obtained by pretrained models that is not well explored before, through three part-level recognition tasks, part retricval, part classification, and part segmentation.
Table 4: Part retricval and classification results on the cropped part patches of CUB-200-2011 and COCO. The "Encoded' and "Projected' refer to the encoded and projected representations. "Linear' and "Attentive' columns denote the linear probing and attentive probing accuracy, respectively.
Part Retrieval (AP)
Part Classification (Acc)
Methods
CUB-200-2011
COCO
CUB-200-2011
COCO
Encoded
Projected
Encoded
Projected
Linear
Attentive
Linear
Attentive
Supervised Model:
DeiT
35.0
44.1
90.9
92.9
88.5
91.4
Contrastive Learning:
MoCo v3
28.4
52.3
36.8
93.8
96.0
92.4
95.3
DINO
48.9
31.7
51.8
41.2
93.2
95.2
91.7
94.5
Masked Image Modeling (MTM):
BETT
27.9
35.3
55.4
86.5
69.3
86.5
MAE
28.5
37.1
86.9
92.8
88.0
93.9
CAE
58.0
57.0
$9.5
95.8
91.1
95.5
Contrastive Learning MIM:
IROT
31.2
59.2
41.5
92.1
Part retrieval. We conduct part retrieval experiments on two datasets, CUB-200-2011 (Wah et al., 2011) and COCO (Lin et al., 2014), which provide both the positions and corresponding categories of the keypoints. We build the part patch databascs by cropping the patches where these keypoints are located at the center. We consider four and three keypoints from the two datasets, respectively. For each keypoint, we find the minimum L.2 distance (d) from the distances between it and all the other keypoints in the same image, then
8

---

Published in Transactions on Machine Learning Research (07/2023)
crop d x d patch centered at this keypoint and resize it to 224 x 224 8. For each method, we directly use the pretrained encoders to extract features, without additional training processes, and take the better one from the class token or the average embedding of all patch tokens as the extracted representation. With each patch as the query patch, we attempt to retrieve patches belonging to the same category by calculating the cosine similarity between its representation and all the other patches' in the dataset and utilize the average precision (AP) as the retricval metric. Finally, we average all the obtained AP scores (with all patches respectively taken as the query patch for retrieval) as the final retrieval score to evaluate the retrieval performance of the method.
The results are provided in Table 4. We have the following observations. 1 Self-supervised models except BETT and MAE outperform the supervised model DeiT, indicating the capability that contrastive learning and CAE learn part-aware representations. BETT and MAE perform inferior, consistent to the observations in ImageNet classification Table 1. iBOT performs the best, and the reason might be that the capability of learning part-aware representations is boosted by making use of both contrastive learning and masked image modeling.
We also report the part retrieval performance of the projected representations of contrastive learning methods in Table 4. The performance is much lower than the encoded representations. This provides an extra evidence for the part-to-whole hypothesis of contrastive learning: the projected representations are more about the whole object.
Finally, we visualize some patch retrieval results of the encoded representations on the ImageNet validation set in Figures 6 (More visualization is presented in Appendix A.9). Tt is observed that the retrieved patches of self-supervised methods are generally more about the semantics of the query part than that of DeiT. The results demonstrate that the encoded representations of DeiT focus more on object-level semantics, while the encoded representations of these self-supervised methods are more about part-level semantics. Among these methods, the retrieved patches of MAE have less semantic correlation but often share similar hues.
Part classification. We further conduct part classification experiments on the datasets used for part retrieval, requiring pretrained models to assign the patches into the corresponding category label. We consider two kinds of extra learnable layers, lincar probing and attentive probing, for classification. For lincar probing, we learn a supervised linear classification layer on the extracted class token of the frozen encoders. While for attentive probing, following Chen et al. (2022a), cross attention module and a batch normalization layer without affine transformation are additionally inserted between the encoder and the linear classifier. And a new learnable class token is taken as the query of the cross attention module, to replace the original class token extracted by the frozen encoder. We use SGD optimizer with a learning rate of 0.4 and 0.04 for linear probing and attentive probing, respectively. For both linear probing and attentive probing, the models are trained for 90 epochs. And the momentum of SGD is set 0.9, the weight decay is set to 0, and the batch size is set to 1024.
The results in Table 4 show that: ! While DeiT performs the best in the image classification task (sce Table 1), for part classification, contrastive- methods like MoCo v3, DINO, and iBOT outperform DeiT by more than 2% under both linear and attentive probing settings. @ Though MIM-based models CAE and MAE are inferior to DeiT in object-level classification (e.g., more than 10% and 4% lower in linear and attentive probing), they show competitive performance lincar probing and higher results than DeiT in atlentive probing, demonstrating they learn better part-aware representations. @ BETT is inferior to other works, and iBOT has good performance, implying that the probing quality of pretrained encoders is good indicator for downstrcam performance.
Part segmentation. We perform part-level linear semantic segmentation ion to study the finer-grained part representation modeling capability of different pretraining paradigms on three widely used datasets: ADE20K- Part (Zhou et al., 2019) containing 209 parts from the ADE20K dataset (Zhou et al., 2019; Zhu et al., 2023), Pascal-Part (Chen et al., 2014) including 193 part categories, and LIP (Gong et al., 2017) consisting of 19 semantic human part labels. See Appendix A.2 for more datasct details. Similar to the object-level semantic segmentation experiments, linear evaluation is employed here. We maintain the same training
We explain the approach of resizing patches to full image resolution instead of other methods extract part representations in Appendix A.3
9

---

Published in Transactions on Machine Learning Research (07/2023)
DefT
MoCo
DINO
MAE
CAF
IBOT
DeiT
MoCo
DINO
MAE
CAE
iBOT
Figure 6: Patch retrieval comparisons of encoded representations on cropped patches from ImageNet.
Table 5: Part-level linear semantic segmentation results (%) on the ADE20K-Part, Pascal-Part, and LIP datasets.
DE20K-Part
Pascal-Part
LIP 19 Part Classes
Methods
209 Part Classes
193 Part Classes
mloU
mAcc
aAce
mloU
mAce
aAcc
mloU
mAce
aAcc
Supervised Model:
DelT
27.3
34.7
69.2
27.4
36.1
65.8
41.4
52.6
73.5
Controstive Learning:
MoCo v3
27.1
34.7
70.1
27.1
35.8
66.0
41.9
53.0
74.5
DINO
28.9
36.8
70.3
27.8
36.5
66.4
41.0
51.9
74.0
Masked Image Modeling (MIM):
BETT
18.6
25.8
58.2
14.8
21.4
47.0
27.2
36.5
60.1
MAE
26.3
35.0
67.3
24.3
32.9
61.5
38.2
48.7
71.3
CAE
28.4
36.9
71.1
27.8
37.0
66.3
43.7
55.1
75.9
Contrastive Learning MIM:
IBOT
32.2
40.0
73.4
30.7
40.0
69.7
44.6
55.7
76.6
protocols including learning rate (4e 4), training iterations (160k), and batch size (16) for all methods for fair comparisons For ADE20K, the inpul size is set to 512 x 512 following previous works (Bao el. al., 2021; He et al., 2021; Chen et al., 2022a; Zhou et al., 2021). For Pascal-Part, we adopt 480 x 480 as image input
We also consider do hyperparameter tuning per method more fair comparisons. See Appendix A.5
10

---

Published in Transactions on Machine Learning Research (07/ 2203)
ADE20K-Objec
ADE20K-Part
Pascal-Object
Pascal-Par Para
54
30
90
30
mloU (%)
27
27
48
80
24
24
42
70
21
21
36
18-
60
18
30
15
50
15
DeiT
MoCo v3 DINO
MAE
CAE
iBOT
DeiT
MoCo v3
DINO
MAE
CAE
iBOT
DeiT
MoCo v.3
DINO MAE
CAE
iBOT
DeiT
MoCo v3
DINO MAE
CAE iBOT
Figure 7: Comparisons between object-level and part-level semantic segmentation ADE20K and Pascal-Part datasets. Though the supervised DeiT is superior over self-supervised models (i.e., MoCo v3, DINO, MAE, CAE) on object-level segmentation, it is generally inferior to self-supervised models on part segmentation, demonstrating self-supervised methods learn good part-aware representations. iBOT enjoys the benefits of contrastive learning and MIM. See Appendix A.4 for detailed results.
resolution following Contributors (2020). As for LIP, we use the same input size (320 x 320) proposed in LIP (Gong et al., 2017).
The results are reported in Table 5 with the following observations. Contrastive learning models, ie., MoCo v3 and DINO, achieve competitive performance with the supervised model DeiT: DINO outperforms DeiT on ADE20K-Part and Pascal-Part, and MoCo v3 outperforms DeiT on LIP. @ The MIM model CAB, outperforms DeiT by large margins on all three datasets, e.g., 1.1% on ADE20K- Part and 2.3% on LIP, indicating CAE learns good part-aware representations. Similar to part retrieval, BEiT and MAE perform inferior in absolute metrics possibly due to pretraining quality in representation encoding 10, Also, it is worth mentioning that the performance gap (1.0%) between DeiT and MAF on part-level segmentation is significantly narrowed compared to that (11.6%) of object-level segmentation as shown in Figure 7. This indicates that although MAE does not outperform DeiT on part-level tasks, tends to learn more about parts than DeiT. Similar observations on BETT can also be found in Appendix A.4. Compared with object-level segmentation results in Table 3, DeiT learns better object-level semantics by explicit supervision than both contrastive learning and MIM, however, it is generally inferior to self-supervised models on part segmentation ! The model iBOT, which leverages both contrastive learning and MIM, outperforms all other works on three datasets, demonstrating its powerful capability in learning finer part-level semantics. Combining the two self-supervised learning techniques is thus a promising direction.
In summary, we show that supesuuee methods are potentially capable of learning part-oware representations. Among them, CAÉ is a representative MIM work, showing good performance by explicitly predicting the encoded representations of the masked patches in the encoding space; contrastive learning methods MoCo v3 and DINO out perform BEIT and MAE; and iBOT performs the best 11, The observations are evidenced by three past-based segmentation benchmarks consistently.
4.3 Observation Summary between Object-Level and Lert-Level Segmentation
We conduct both object-level and part-level linear semantic segmentation on different hierarchies of the same datasct. Considering that the 209 classes in ADE20K-Part are basically chosen from 59 object classos, we denote the 59-object dataset as ADE20K-Object. Similarly, Pascal-Object consists of 16 object categories, corresponding to the 193 part categories in Pascal-Part.
The results in Figure 7 show that: although self-supervised models (except iBOT) are inferior to the supervised DeiT on SDE20K-Objeet and Pascal-Object, they significantly narrow the performance gap on ADE20K-Part
We discuss the reasons in detail and add human pose estimation experiments in Appendix A.6
The iBOT authors verified that introducing MIM brings richer semantics the patch tokens in Soe. 4.3.1 of their paper. This could be the reason for the superior performance object-level and part-level tasks observed in IBOT where MTM and CL. are combined.
11

---

Published in Transactions on Machine Learning Research (07/2023)
and Pascal-Part and even outperform DciT, demonstrating self-supervised methods tend to learn more about parts and potentially produce good part-aware representations. Similar observations could be found from the object classification in Table 1 and part classification in Table 4.
In comparison to contrastive learning, CAE shows a stronger capability of learning part-aware representations, and a weaker capability of learning object-level semantics. The superiority of iBOT, a combination of contrastive learning and masked image modeling, demonstrates that it enjoys the benefits of contrastive learning and masked image modeling. We have outlined several key takeaways in Appendix A.7 derived from our experiments and analysis, which we hope could inspire further works in sele-sud learning.
Additionally, although our main focus is not on in-the-wild datasets like COCO and Openlmages, we conduct a preliminary exploration on them. Following the same setting as in our paper, we conducted a quick experiment with a transformer-basee experiments show that its part retrieval (32.3% AP) on CUB-200 and LIP segmentation results (45.89% mIoU) outperform MAE (28.5% AP and 38.2% mloU) pretrained on ImageNetlkk. It also narrows the gap wetween DeiT (35.0% AP) in part retrieval and outperforms DeiT (41.4% mloU) in part segmentation. These results indicate that pretraining on more in-the-wild datasets has the potential to learn better part representations as cropping can produce more parts of multiple objects in one image. We leave it as an interesting future direction.
Mo (Hu et al., 2022) that is pretrained on COCO. Our
5 Conclusion
We attempt to study the capability of learning part-aware representations of self-supervised representation pretraining methods. We provide speculations for contrastive learning and masked image modeling: part-to- whole and part-to-part, with empirical results justifying the speculations. Our study presents an aspect to understand what self-supervised representation pretraining methods learn.
Limitation and future work. The part-aware representation learning capability one of the properties of self-super pretraining. There should be other characteristies that are worth exploring. The proposed approach has no ethical or societal issues on its own, except those inherited from computer vision.
Acknowledgments
This work is supported by National Science Foundation of China (NSFC) Grant No. 61972008.
References
Hangbo Bao, Li Dong, and Furu Wei. BETT: BERT pre-training of image transformers. arXiv:2106.08254, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varianc-iaaarraccccovvrnnnn regularization for self-supervised learning. arxiv preprint arXim:2105. 04906, 2021.
Suzanna Becker and Geoffrey E. Hinton. A self-organizing neural network that discovers surfaces in random-dot. stereograms, Nature, 355 16566) 1111111--66,,,-111 1992.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, pp. 1209-1218, 2018.
Shuhao Cao, Peng Xu, and David A Clifton. How to understand masked autoencoders. arXiv preprint arXiv:2202.03670, 2022.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXix:2006. 09882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
12

---

Published in Transactions on Machine Learning Research (07/2023)
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, volume 119 of Proceedings of Machine Learning Research, pp- 1597 1607. PMLR, 2020.
Xianjic Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistie models and body parts. In CVPR, pp. 1971 1978, 2014.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXic preprint Xrx:22200. 03006, 2022a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, pp. 15750 15758, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, pp. 9640-9644 2021.
Yubei Chen, Adrien Bardes, Zengyi Li, and Yann LeCun. Intra-instance vicreg: Bag of self-supervised image patch embedding. arXiv preprint arXiv:2206.0895, 08954, 2022b.
MMSegmeenation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and bench- mark. https://github. com/ open apen--mlaab/mmsgeentattt 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largo-scale hicrarchical image database. In CVPR, pp. 248-254 Icce, 2009.
Mingyu Ding, Xiaochen Lian, Linjic Yang, Peng Wang, Xiaojie Jin, Zhiwu Lu, and Ping Luo. Hr-nas: Searching efficient high-resolution neural architectures with lightweight transformers. In Proceedings of the IEFE/C CCF Conference on Computer Vision and Pattern Recognition, pp. 2982-2992 2021.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pro-training of vision transformers. arXiv preprint arxiv:2111 12710, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worh 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
Alaaeldin El-Nouby, Gautier lzacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2112, 10740, 2021.
M. Everingham, T.. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results. http://www.pascal- 2010.
www.eeallgaaalleeee/eeeaaa CVoeeenggee/////oooo/oe VVCC/vo20100 wwwo:////wwwkaoooo///o/oooooo/// workshoppindex..uue
Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaolui Shen, and Liang Lin. Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing. In CVPR, July 2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXie preprint arXiv:2006. 07733, 2020.
Kaiming He, Xinlci Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autocncoders are scalable vision learners. arXio preprinl ur urXiv2211.006377 2021.
Ronghang Hu, Shoubhik Debnath, Saining Xie, and Xinlei Chen. Exploring songsequenne masked autoen- coders. ar Xiu preprint ar arXiv:2210.0722 07.0724, 2022.
13

---

Published in Transactions on Machine Learning Research (07/2023)
Xiangwen Kong and Xiangyu Zhang. Understanding masked image modeling via learning occlusion invariant feature. arxiv preprint or arXim:208..00 0444, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. Mst: Masked self-supervised transformer for visual representation. Advonces in Neural Information Processing Systems, 34:13165 131776, 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp- 740-755. Springer, 2014.
Xingbin Liu, Jinghao Zhou, Tao Kong, Xianming Lin, and Rongrong Ji. Exploring target representations for masked auloencoders. Xiv preprinl ar Xivi:220090088 03917, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp- 10012-10022, 2021.
Shlok Mishra, Anshul Shah, Ankan Bansal, Abhyuday Jagannatha, Abhishek Sharma, David Jacobs, and Dilip Krishnan. Object-aware cropping for self-supervised learning. arXiv preprint 1xXi::212.003111 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, pp. 8821 8831. PMLR, 2021.
Nikunj Saunshi, Jordan T. Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham M. Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. CoRR, abs/2202. 14037, 2022. URL https: org. 2202. 14037.
Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision representation learning. arXiv preprint 2Xim:2206.0120) 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. Xiviv preprint arXiv22112.1887 2020.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue V Gool. Revisiting contrastive methods for unsupervised learning of visual representations. Advances in Neural Information Processing Systems, 14:16238-16256 2021.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for sclf-supervised visual pre-training. In CVPR, pp. 3024-3033 2021.
Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXie preprint. arXiv:2205.141 14141, 2022.
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In Proceedings of the TEEE/CVP International Conference on Computer Vision, pp.
2021a.
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self-supervised learning with swin transformers. arXiv preprint arXiv:2105 03.04455, 2021b.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint or Xin:21 211. 11.11..9988 11.09886, 2021c
14

---

Published in Transactions on Machine Learning Research (07/2023)
Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. CoRR, abs/2205, 13543, 2022a. doi: 10.48550//rXiv222000 22t.kiv22222.1544.0 URL https://doi.org 10. 11550//rXiv.2200 13543.
Zhenda. Xie, Zheng Zhang, Yue Cao, Yutong Lin, Yixuan Wei, Qi Dai, and Han Hu. On data scaling in masked image modeling. CoRR, abs/2206.0466 044644, 2022b. doi: 0005500/2Xiv.2200.444000 URL
https: //doi: orgoo///// 10. 48550/arXiv..... www.a..www................. 04664.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphanc Deny. Barlow twins: Self-supervised learning via redundancy reduction. ar preprinl ar arXiv:2103. 03230000000 2021.
Yuanyi Zhong, Haoran Tang, Junkun Chen, Jian Peng, and Yu-Xiong Wang. Is self-supervised learning more robust than supervised learning? arXiv preprint ar0iv: 05259, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 127(3):302 2212,,221111 2019.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Ibot: Image bert pro-training with online tokenizer. arXiv preprint arXiv:2111.07822 2021.
Jie Zhu, Huabin Huang, Banghuai Li, and Leye Wang. E-crf: Embedded conditional random field for boundary- caused class weights confusion in semantic segmentation. In The Eleventh International Conference on Learning Representations, 2023.
15

---

Published in Transactions on Machine Learning Research (07/ 2202)
A Appendix
A.1 Model Description
For all the models involved in the experiments including DeïT (Touvron et al., 2020), MoCo v3 (Chen et al., 2021), DINO (Caron et al., 2021), BETT (Bao et al., 2021), MAE (He et al., 2021), CAE (Chen et al., 2022a), and iBOT (Zhou et al., 2021), we use their official code to implement the encoders. It is worth noticing that for DINO and iBOT, we choose the checkpoint of the teacher models as they have been reported to perform better than the student models in their papers (Caron el al., 2021; Zhou al., 2021).
A.2 Datasets
ADE20K (Zhou et al., 2019) is one of the most challenging benchmarks, containing 150 fine-grained semantic concepts and a variety of scenes with 1,038 image-level labels. There are 20,210 images in the training set and 2,000 images in the validation set. We choose 59 out of total 150 semantic concepts that are concrete objects containing parts (Zhou et al., 2019), termed ADEeOK-Object. We also select 209 part categories that, emerge both in the training set and the validation set, called ADE20K-Part.
Pascal-Part (Chen et al., 2014) is set of additional annotations for PASCAL VOC 2010 (Everingham et al., 2010), thereby holding the same statisties those of PASCAL VOC 2010. It provides segmentation masks for cach part of objects. Concrotely, the dataset includes 20 object-level categories and 193 part-level categories. In our experiments, we remove 4 object categories that do not contain parts including boat, table, chair, and sofa.
LIP (Gong et al., 2017) is a largo-scale benchmark for human parsing rescarch, which includes 50,462 images with pixel-wise annotations on 19 semantic part labels. In detail, it includes 19,081 full-body images, 13,672 upper-body images, 403 lower-body images, 3,386 head-missed images, 2,778 back-view images and 21,028 images with occlusions. There are 30,462 images in the training set and 10,000 images in the validation set. The rest 10,000 images are served as the test set with missing labels for competition evaluation.
CUB-200-2011 (Wah et al., 2011) is a popular benchmark for fine-grained image classification, and also provides bounding box and part location annotations. It contains 11,788 images of 200 bird species and 15 part keypoint annotations per bird. In this work, we mainly leverage its part keypoint annotations. And only part categories (right eye, right leg, left wing, and tail) are chosen to be considered in our experiments, to make sure that the selected keypoints are far enough away from cach other and enough context information can be contained in the cropped patches. (We also tried using all keypoints and the conclusion is consistent.)
COCO (Caesar et al., 2018), as one of the most widely-used human pose estimation datasets, contains more than 200,000 images and 250,000 labeled person instances. Similar to CUB-200-2011 mentioned above, only 3 (nose, right wrist, and left ankle) of its 17 keypoint categories are considered in our experiments.
CIFAR10 (Krizhevsky al., 2009) is a widely-used dataset that has 50, 000 training images and 10, 000 test images. CIFAR10 has 10 categorics. The dimension for CIFAR10 images is 32 x 32 x 3.
A.3 Part Resizing Approach
We explain the idea behind our part resizing approach below. There are three main reasons for using this approach:
To maintain the shape of the position embedding the same during pretraining, is essential for the transformer to distinguish patches located at different positions. Rashly resizing position embedding, e.g., through interpolation, may introduce some noise and cause mismatch with the parameters of other pretrained transformer components.
Is there a domain gap between training data and the resized part? No may be the answer. As shown in Table 6, the crop operation during self-supervised pretraining, mostly adopts scale range of (0.08 - 1), followed by image resizing. Hence, the pretrained models may see these 'upscaled' parts during training potentially. Therefore, there is no domain gap using our part resizing approach.
16

---

Published in Transactions on Machine Learning Research (07/2023)
Table 6: The crop scale of different pretrained models.
Method DefTT
MoCo v3
DINO
BETT
MAE
CAE
iBOT
Crop Scale
0.08-1
0.08-1
0.4-1, 0.05-0.4
0.08-1
0.2-1
0.08-1
0.14-1, 0.05-0.4
We also considered another manner: processing the whole image and then taking the individual patch representations. However, due to the presence of the attention module, different patch representations would interact globally. Therefore, the part representation would contain information from other parts and may lead to information leaks hence the confusion about part representations.
A.4 Detailed Results for Object-Level and Part-Level Segmentation
In this section, we provide detailed comparisons between object-level and part-level semantic segmentation Table 7 and Table Similar observations as in Figure 7 in the main paper are found: although the supervised DeiT superior over self-supervisod methods on ADE20K-Objer and Pascal-Object except iBOT, is generally inferior lo self-supervised models on ADE20K methods can learn good part-aware representations. BETT and MAE perform inferior, perhaps because the two methods do not have an explicit process to predict the encoded representations of masked patches, instead, directly reconstruct the targets.
and Pascal-Part, demonstrating self-supervised
Table 7: Linear semantic segmentation results on ADE20K-Object and ADE20K-Part.
Object Seg ADE20K-Object
Part Seg ADE20K-Parr
Methods
Object Classes
209 Part Classes
mloU
mAce
aAce
mloU
mAce
aAcc
Supervised Model:
Dei'I
52.6
62.9
83.8
27.3
34.7
69.2
Contrastive Learning::
MoCo v3
50.2
60.4
83.6
34.7
DINO
50.8
60.8
83.9
28.9
36.8
70.3
Masked Image Modeling (MTM):
BEIT
28.6
37.2
73.4
18.6
25.8
58.2
MAE
41.0
50.6
79.9
26.3
35.0
67.3
CAE
47.4
58.4
82.9
28. 4
36.9
71.1
Contrastive Learning + MIM:
IROT
55.2
65.1
85.6
40.0
73.4
Table 8: Linear semantic segmentation results on Pascal-Object and Pascal-Part Part.
Object Seg on Pascal-Object
Part Seg on Pascal-Part
Methods
16 Object Classes
Part Classes
mloU
mAcc
aAce
mloU
mAcc
aAcc
Supervised Model:
DeiT
92.2
95.3
96.8
27.4
36.2
65.8
Contrastive Learning:
MoCo v3
89.4
93.7
95.7
27.1
35.8
66.0
DINO
88.0
92.7
95.3
27.8
36.5
66.4
Masked Image Modcling (MIM):
BETT
56.4
69.0
76.8
14.8
21.4
47.0
MAE
76.1
84.6
89.5
24.3
32.9
61.5
CAE
83.3
89.7
93.2
27.8
37.0
66.3
Contrastive Learning + MIM:
iBOT
92.1
95.3
97.1
30.7
40.0
69.7
17

---

Published in Transactions on Machine Learning Research (07 2022)
Table 9: Different learning rates in LIP part segmenta- tion.
Table 10: Linear human pose estimation results on the COCO dataset.
Method 1c
5
20
4
3c
4c
4
5c
lc
3
Method
AP50
AR
AR50
Superoised Model:
Supervised Model:
DefT
40.5
41.3
41.4
41.4
41.3
Def'I
6.8
27.5
14.7
44.3
Contrastive Learning:
Contrastive Learning:
MoCo v3
40.0
41.7
41.7
41.9
41.7
41.8
MoCo v3
12.6
42.1
21.61
56.3
DINO
38.1
40.8
40.9
41.0
40.9
40.9
DINO
16.6
49.0
25.9
62.1
Masked Image Modeling (MIbd):
Masked Image: Modeling (MIM):
BETT
23.1
27.2
27.4 .
27.2
27.1
27.5
MAE
23.6
59.8
33.3
69.6
MAE
34.7
38.5
38.6
38.2
38.7
38.8
CAE
17.5
50.1
28.9
64.0
CAE
42.8
44.1
44.2
43.7
44.3
Contrastive Learning
MIM:
Contrustive Learning + MIM:
IBOT
15.5
48.1
26.0
62.8
BOT
41.5
44.4
44.5
44.6
44.5
44.8
A.5 Ablation on Learning Rate for Part Semantic Segmentation
We use batch size 16 and 160k iterations following previous methods He et al. (2021); Bao al. (2021); Zhou et al. (2021). For optimizer,
adopt AdamW which is widely used for Vision Transformer (Dosovitskiy et al., 2021). As for learning rate, we conduct experiments on LIP part segmentation with different learning rates including le -, 5, 2e - 4, 3e 4, 4e 4, 5e - 4, and le 3 for all pretrained models. The results are presented in Table 9. The best results are bold. One can see that except for le - 5, other learning rates slightly influence the final performance. In terms of the overall performance of each model, MoCo v3, CAE, and iBOT show superior performance over supervisod model DeiT, validating our findings in the main paper.
A.6 Why BEiT and MAE Perform Inferior?
In lincar evaluation, the quality of output representations always depends on the pretraining quality of the encoder. A well-trained encoder usually produces represenlations of high quality. In our experiments, we find that BETT only obtains 56.7 linear probe on ImageNet while other models achieve about 70 linear probe on TmtgeNet. The large gap indicates that BEiT does not learn high-quality representations as other models, i.e., its pretraining quality is not good enough. This could be the main reason that it performs worst downstream lasks (both object-level and part-level tasks).
For MAE, we find that the colors of the retrieved patches in Figure 6 and Figure 8 in Appendix are quite similar. Based on it, we speculate that its low-level color reconstruction target helps MAE learn better middle- or low-level features while providing limited high-level semantics. To validate this point, we further conduct experiments on the human pose estimation task, which has been proven by previous work Ding et al. (2021) be a task requiring middle-level features. Similar to lincar probing, we train one linear layer over the features extracted by the frozen backbones to predict human keypoints, The results are shown in Table 10 and MAE outperforms all other counterparts by large margins. So believe MAE tends to learn lower-level features and pay relatively less attention to the high-level semantics, which leads to its inferior performance on classification and segmentation tasks.
A.7 Takeaways
We have outlined several key takeaways derived from our experiments and analysis, which we believe could benefit further works in the self-supervised learning field.
1) Supervised models care more about the whole objects, while self-supervised models have a stronger capability of learning part-aware representations. understanding self-supervised representation pretraining.
18

---

Published in Transactions on Machine Learning Research (07/2023)
2) Contrastive learning may learn higher-level semantics and be more semantically abundant than MTM. Combining MIM and CL is potentially capable of learning multi-level semantics.
3) MIM methods, e.g., CAE, learn slightly better part-level features than contrastive learning methods. While the learned features of MIM methods, to some extent, depend on the reconstruction largets, e-g., the RGB (color) target used MAE leads it learn lower-level features, hence only sub-optimal results on high-level tasks.
Those takeaways evidenced correspondingly as follows:
1) In Figure 7, Table 7, and Table 8, from object-level to part-level, considerable performance improvements (relative to DeiT) are observed for all self-supervised models. Among these methods, DINO, CAE, and iBOT show larger improvements, demonstrating they can learn better feat for part-aware segmentation. For example, DeiT (52.6% in mloU) outperforms DINO (50.8%) and CAE (47.4%) by 1.8% and 5.2% in ADE20K object-level experiment. However, in ADE20K part-level experiment, the situation is completely opposite. DINO (28.9% in mIoU) outperforms DeiT (27.3%) by 1.6% and CAE (28.4%) outperforms DeïT by 1.1%. Similar results can be seen in segmentation experiments on the Pascal dataset, as well as classification results on COCO and CUBB-200.
2) As shown in the second row of Figure 2, the retrieved patches of MoCo v3 are all about dog mouths. While the retrieved patches of CAE contain some patches about the mouths cats or foxes. Similarly, in Figure 8, the retrieved patches of CAE for the watch also include patches about the dial of the dial phone. Contrastive learning methods are more likely to retrieve patches with the same category while CAE sometimes retrieves some patches with similar textures. This observation will be more frequently found from the top-100 retrieved patches, based on which we show contrastive learning methods tend to learn better high-level semantics than MIM. And as shown in our paper, iBOT outperforms other methods by large margins in almost all tasks, including object-level and part-level tasks.
On part retrieval, CAE outperforms contrastive learning methods by large margin (about 8% and 5% on CUB-200 and COCO, respectively). On object-level linear probing or segmentation tasks, CAE performs clearly worse than contrastive learning methods. While this performance gap is significantly narrowed even closed on corresponding part-level tasks. @ the retrieved patches of MAE in Figure and Figure 8 tend lo share similar hues, and MAE performs much better than other methods on the human pose estimation task which requires middle-level features.
A.8 More Discussion about Related Work
Image crop is important in self-supervised learning and brings promising properties to self-supervieed models: Van Gansbeke et al. (2021) shows that MoCo learns spatially structured representations when trained with a multi-crop strategy; By leveraging object-aware cropping, Mishra et al. (2021) encourages the sulf-vuper model to learn both object and scene level semantic representations from real-world uncurated datascts. Similarly, in our part-to-whot whole explanation, random crop on ImageNet generates views that describe parts of object and are encoded lo part representations and projected to whole for agreement. DenseCL, (Wang et al., 2021) designs dense self-supervised learning method that directly works at the level of pixels (or local features) by taking into account the correspondence between local features. DetCo (Xic et al., 2021a) achieves better performance trade-off on both classification and detection through multi-level supervision to intermediate representations and contrastive learning between global image and local patches.
A.9
More Examples for ImageNet Patch Retrieval Visualization
We visualize more patch retrieval results of the encoded representations on the ImageNet validation set in
Figures 8.
19

---

Published in Transactions on Machine Learning Research (07 (07/2023)
DefT
MoCo v3
DINO
MAE
CAE
IOOT
DeiT
MoCo
DINO
MAE
CAF
BOOT
DeiT
MoCo v3
DINO
MAE
CAE
IOTT
Figure 8: Patch retrieval comparisons of encoded representations on cropped patches from ImageNet.
20


Reviews:
[{"review_id": "Qr2CH6SOji", "reviewer": "Reviewer_fpt9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I enjoyed reading the paper. Its findings, although not surprising, have not been adequately established in prior work. Thus, the empirical contributions of this paper will be of interest to the TMLR audience.\n\nThe main strengths of the paper are:\n\n(+) Very well written. The contributions of the paper are well described, technical descriptions are clear, experimental settings are well specified, and conclusions are well founded on the experimental results.\n\n(+) Seeks to study the ability of different representation learning methods to learn representations of object parts from object-centric images. To the best of my knowledge, part representations have not been the topic of analysis in prior work. The paper provides the first thorough analysis on this topic, evaluating two branches of SSL: contrastive learning and MIM.\n\n(+) The paper provides concrete evidence of the interpretation of CL as a part-to-whole task. For example, the qualitative results of Figure 4 and part retrieval and classification of table 3.\n\n(+) The paper conducts experiments with various SSL models, and on various downstream tasks that require part-aware representations, making the conclusions very convincing. \n\n(-) On the negative side, one of the claimed paper contributions is the interpretation of MIM as a part-to-part task and CL as a part-to-whole task. This interpretation is intuitive but not surprising. While I have not seen such an interpretation on any specific paper, I don’t think it is new to field experts. I certainly have known this interpretation before reading the paper. So, while I appreciate the author's effort in trying to make this intuition crisp and provide concrete evidence for such interpretations, claiming it as a contribution is, in my opinion, exaggerated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper analyses different representation learning procedures (supervised, contrastive and MIM) with respect to their abilities to learn object part representations. It presents a part-to-whole explanation for typical contrastive learning methods and a part-to-part explanation for masked image modeling. It also conducts an empirical study with the following key observations.\n\n1) DeiT (supervised) outperforms contrastive learning and MIM methods except for iBOT in object-level recognition tasks 2) Self-supervised methods learn better part-aware representations than DeiT. 3) In contrastive learning, the encoder can learn part-aware information, while the projected representation tends to be more about the whole object. 4) The MIM method CAE shows good potential in part-aware representation learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The only change I would request is an adjustment of the tone concerning the part-to-part/part-to-whole interpretation. Please see the comment in the section above.", "broader_impact_concerns": "I have no ethical concerns regarding this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3CiH6OlPkH", "reviewer": "Reviewer_xNEG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and easy to follow. \n- Paper presents interesting observations. The patch retrieval visualizations are especially interesting. \n- A fairly thorough study is presented. \n\nWeaknesses:\n- The paper should discuss approaches which make use of local-global crops (SwAV, DINO) in context of part-to-whole explanation presented in the paper. \n- The augmentation - random resize crop used as an augmentation in typical SSL works is closely related to tthe part-to-whole explanation presented in the paper. I suggest that the authors include some discussion wrt [a,b] to strengthen the work. \n- In Section 3.1, the authors mention that similar observations were made in Chen et al 2022b. The authors should make it clear the new contributions made by their analyses.  \n- Missing discussion on approaches [b,c,d] which develop SSL-tasks for non-object centric datasets, like COCO and OpenImages. How would the analyses look like for those (more in-the-wild datasets) ? \n\n[a] Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations\n\n[b] Object-Aware Cropping for Self-Supervised Learning\n\n[c] Dense Contrastive Learning for Self-Supervised Visual Pre-Training\n\n[d] DetCo: Unsupervised Contrastive Learning for Object Detection\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The paper presents a study of self-supervised and supervised models from the lens of object and part aware learning. \n- It is shown that SSL models learn part aware representations and perform well on related tasks like part-level recognition and segmentation. On the other hand, supervised models show superior performance on object-level recognition tasks including semantic segmentation and object detection.\n- The paper also shows the the benefits of contrastive learning based approaches vs ones using MIM-based pretext tasks. Overall, method combining both seem to be the most beneficial. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The weaknesses presented in the previous section are critical for me. Following are some minor suggestions. \n- Abstract : \"Self-supervised pretrained encoder is required..\" - unclear. \n- The paper has a couple of long sentences. Authors can try to split them into two. Eg. Introduction : \"The goal of. .. \". \n- Figure 5 :  X/Z mismatch ? \n- Easier to parse the tables if best results are boldfaced. ", "broader_impact_concerns": "The authors discuss ethical implications in the paper. I do not have any additional concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rlEJCPjYpj", "reviewer": "Reviewer_PsXF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper is quite clearly written and easy to read\n2. Investigations of properties of representations is potentially interesting and useful\n\nWeaknesses\n1. Limited contribution. The results on whole-image tasks were already known (btw, how do the results reported here compare to those?) The results on part-level tasks seem new, but it is not clear what exactly they teach the reader. \n2. The approach with resizing patches to full image resolution and classifying/retrieving those seems somewhat impractical (why would one do it?) It seems processing the whole image and then taking the individual patch representations can be more meaningful/practical? Especially for masked modeling approaches, which have never seen these “upscaled” parts at training time?\n3. I am not sure if the experimental evaluation is controlled enough: it seems the authors use the same hyperparameters for training linear eval on top of all methods, without much explanation. What was the process for selecting these, Is this setting fair? Could it be that these hyperparameters favor some of the methods? A fairer approach would be to do some hyperparameter tuning per method.\n4. Presentation issues/questions  \na. Among the three stated contributions at the end of introduction, it seems only one is a real contribution: (1) seems basically subsumed by (3), and (2) does not have much substance to it.  \nb. BYOL and Dino are referred to as contrastive, but I believe they are not? (in that, there are no positive and negative samples, just self-distillation)  \nc. I do not understand what the following means, please explain. “We build the part patch databases by cropping the patches centered at the keypoint. We consider four and three keypoints from the two datasets, respectively.”  \nd. Paper length seems somewhat excessive: feels like the introduction and section 3 could be shortened a lot and some of the qualitative results could be moved to the appendix.  \ne. First sentence of the abstract reads weird.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper analyzes the properties of visual representations learned with supervised learning and various self-supervised learning methods (both contrastive-type, masked modeling, and combination of both) - in particular, compares their performance on “object-level” recognition - usual image classification and semantic segmentation - as well as on “part-level” recognition - classification and retrieval of image patches, as well as semantic segmentation of object parts. The paper argues that self-supervised learning approaches are relatively stronger on part-level tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the weaknesses above:\n1. Explain the significance and usefulness of the results\n2. Explain the thinking behind the part resizing approach\n3. Demonstrate that the evaluation protocol is fair\n4. Fix the presentation issues", "broader_impact_concerns": "No concerns, the paper just does some analysis of representation learning", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FEmQbissck", "reviewer": "Reviewer_Jg9r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-------------------\n- The paper is well-motivated and the extent of works covered and considered is wide and range from standard (MoCo) to SOTA (iBOT). \n- The observations made in this paper are novel, and provide a novel perspective to the understanding the working of SSL methods.\n- Overall, the results have been well demonstrated using extensive experiments and might inspire design of newer SSL methods.\n\nWeaknesses\n-----------------------\n- It is not clear to me what exactly is the experimental evaluation setup for part retrieval and classification? For retrieval, it is mentioned that the inputs are created by cropping based on keypoints as the centers. So what is the retrival task aimed to do? Is it aimed to retrieve similar patches from different images, or different patch from the same underlying image? Also, how are the ground truths for these generated? Similar questions for part classification - how are the labels to the cropped patches obtained?\n\n- The training in DeiT also performs random cropping of the images. Does this or does this not effect the capability to learn part masks using fully-supervised methods?\n\n- Can an experiment be also added to evaluate the retrieval accuracy on object-level tasks (without fine-tuning - similar to Table 3 Part retrieval, but for objects). This would further help in evaluating the part-whole representations of SSL methods like MoCo (in case that projected has a higher accuracy than encoded).\n\n\nMinor issues\n-------------------------\n- Fig 5 - Captions uses $Z$ but figure uses $X$.\n- Table 3 - CAE performs better than iBOT, but it is given that iBOT performs best.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper performs a retrospective study of self-supervised methods from the perspective of part vs object semantics learnt. They compare fully-supervised models trained on object labels (DeiT), contrastive learning based self-supervised methods (MoCo, DINO), MIM based SSL methods (MAE, CAE, BEiT) as well as hybrid appproaches (iBOT). \n\nOverall they conclude that while fully supervised methods learn more object-aware representations, SSL methods learn part-aware representations. Further, contrastive based methods learn part-level information in the encoder feature and object-level info in the projected feature, while MIM models learn predominantly part-based features. As an evidence, it is shown that: while DeIT outperforms SSL based methods on object-level tasks, this gap is reduced (or often outperformed) while considering part-level tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I request the authors to clarify the experimental setting of part-mask retrieval and classification, as well as add the experiments of object retrieval.", "broader_impact_concerns": "This work uses existing datasets, models and pretrained checkpoints, and as such does not introduce any new ethical concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission compares the performance of representations learned through supervised learning, contrastive learning and masked image modeling, showing that the supervised representations are generally better for object-level tasks, whereas self-supervised representations perform better at part-level tasks. iBOT, which combines contrastive learning and masked image modeling, is generally competitive with the best approach across all tasks.

Most concerns raised by reviewers were minor, and reviewers were satisfied with the authors' responses. After the rebuttal period, all reviewers lean toward accepting the paper. I support acceptance. Before the camera-ready submission, the authors should revise the paper to clarify points of confusion that arose among the reviewers and incorporate the new experiments conducted during the rebuttal period.
Decision: Accept (poster)