Unsupervised Object-Level Representation Learning from Scene Images
Authors: Jiahao Xie
Xiaohang Zhan
Ziwei Liu
Yew-Soon Ong
Chen Change Loy
Venue: NeurIPS 2021 Poster (2021)
Keywords: unsupervised learning
self-supervised learning
representation learning

Abstract: Contrastive self-supervised learning has largely narrowed the gap to supervised pre-training on ImageNet. However, its success highly relies on the object-centric priors of ImageNet, i.e., different augmented views of the same image correspond to the same object. Such a heavily curated constraint becomes immediately infeasible when pre-trained on more complex scene images with many objects. To overcome this limitation, we introduce Object-level Representation Learning (ORL), a new self-supervised learning framework towards scene images. Our key insight is to leverage image-level self-supervised pre-training as the prior to discover object-level semantic correspondence, thus realizing object-level representation learning from scene images. Extensive experiments on COCO show that ORL significantly improves the performance of self-supervised learning on scene images, even surpassing supervised ImageNet pre-training on several downstream tasks. Furthermore, ORL improves the downstream performance when more unlabeled scene images are available, demonstrating its great potential of harnessing unlabeled data in the wild. We hope our approach can motivate future research on more general-purpose unsupervised representation learning from scene data.

Unsupervised Object-Level Representation Learning from Scene Images
Jiahao Xie
Xiaohang Zhan
Ziwei Liu'
Yew Soon Ong!
Chen Change Loy!
Nanyang Technological University
"The Chinese University of Hong Kong
***TAR, Singapore
(jiahao003, ziwei liu, asysong, ccloy)antu.
xlachangzhandouttlok.. com
Abstract
Contrastive self- learning has largely narrowed the gap supervised pre-training on ImageNet. However, its success highly relies on the object-centric priors of ImageNet, i.e., different augmented views of the same image correspond to the same object. Such a heavily curated constraint becomes immediately infeasible when pre-trained on more complex scene images with many objects. To over- come this limitation, we introduce Object-level Representation Learning (ORL), a self- supervieed learning framework towards scene images. Our key insight is to leverage image-level self-suue pre-training as the prior to discover object-level semantic correspondence, thus realizing object-level representation learning from scene images. Extensive experiments on COCO show that ORL significantly improves the performance of self-supervised learning scene images, even surpassing supervised ImageNet pre-training on several downstream tasks. Furthermore, ORL improves the downstream performance when more unlabeled scene images are available, demonstrating its great potential of harnessing unla- beled data in the wild. We hope our approach can motivate future research on more general-purpose purpose unsupervised representation learning from scene data.'
Introduction
Unsupervised visual representation learning aims obtaining transferable features with abundant unlabeled data. Recent self-supervised learning (SSL) methods based on contrastive learning [60, 22, 37, 19, 4, 7] have largely narrowed the gap and even surpassed the supervised counterpart on a number of downstream tasks [30, 49, 15, 47, 35, 23]. These methods build upon the instance discrimination task that maximizes the agreement between different autta-u views of the same image. Despite their success, current SSL methods are primarily pre-trained on the unlabeled ImageNet [8] dataset that contains iconic images with single object as shown Figure ((a). The underlying object-centric constraint of ImageNet makes it hard to be applied in real world scenarios where more complex scene images with multiple objects are available. Meanwhile, naïvely adopting the off-the-shelf contrastive learning methods on scene images introduces inconsistent learning signals since random crops of the same image may correspond to different objects as shown in Figure 1(b). Indeed, it has been shown that current contrastive learning methods tend to struggle on more complex scene datasets [19, 50, 34, 58] like COCO [33] or Places365 [72]. Therefore, is imperative to design an effective object-level representation learning paradigm illustrated in Figure I(c) to harness massive unlabeled scene images in the wild.
Project page: https://wwr //www. mmlab ntu. wtt.ccooo/oooo/oo.....
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Intra-image
Intr-image
Intra-object
Inter-image
Inter-image
Inter-abject
(8) Image-level contrastive learning (b) Image-level contrastive learning (c) Object-lovel contrastive leamnig
on object-centric images
on scene images
on scene images
(d) Example region correspondence discovered across images
Figure I: (a) Current image-level contrastive learning methods heavily rely on the object-centric bias of ImageNet, i.e., different crops correspond the same object. Prior works use either the different views of the same image [60, 22, 37, 5, 19, 7] (i.e., intra-image) or similar images [74, 63, 1, 11] (i.e., inter-image) form positive pairs. (b) Directly adopting image-level contrastive learning methods on scene images can cause inconsistent learning signals since different crops may correspond to different objects. (c) Object-level contrastive learning can overcome the limitation in (b) by enforcing object-level consistency. (d) We find that image-level contrastive learning encodes priors for region correspondence discovery across images, and high-response regions are usually objects or object parts (we show one discovered object- object-instance pair per image pair for clarity), which is useful for object-level representation learning.
In this work, we are interested going beyond ImageNet to obtain better representations on non- iconic images. Apparently, it is challenging to Iearn representations from scene-level images since they are entangled with many concepts including structures, objects, backgrounds and relationships. remains open question how to take advantages of spatial information of multiple objects naturally residing in the scene images when no object annotations are available, let alone further deriving object-level correspondence construct positive object- instetttintttneeee pairs.
To tackle these challenges, we introduce novel object-level unsupervised representation learning framework tailored for scene images. Our framework is based a key insight of the current contrastive learning methods: they can implicitly group different images with similar visual concepts together though they explicitly optimized to group different views of the same image. This phenomenon reveals that image-level contrastive learning has already induced a latent space with rich visual concepts. Though the latent space usually entangles other scene concepts like structures, backgrounds and relationships, it will be useful for object discovery if appropriately deployed. Through computing the similarity of sampled regions between eenearest-niighhee (KNN) images, we conclude two observations: 1) image-level contrastive learning encodes priors for region correspondence discovery across images; 2) high-response regions usually objects object parts.
Based on the observation above, we propose a multi mulli-sttaee framework for unsupervised object-level representation learning. Specifically, we first extract potential object-based regions in scene images using the unsupervised region proposal algorithms (e.g., selective search [56]). We then propose region correspondence generation scheme to leverage the off-the-shelf image-level contrastive learning pre-trained model to discover corresponding object-instance pairs for the proposed regions in the embedding space. Finally, use the obtained object-instance pairs to construct positive sample pairs for object-level representation learning. Figure ((d) shows several cross-image object-instance pairs discovered by our framework COCO dataset using the latent prior of BYOL [19], the saute-of-the-aa image-level contrastive learning method. The discovered nnter-corresponding pairs substantially provide diverse intra-class variances at the object-instance level aid object-level representation learning.
Overall, our main contributions summarized follows:

---

1) We observe that existing image-level contrastive learning methods have priors to discover object- level correspondence across images. We leverage this prior for the first time for unsupervised cross-image object-level correspondence discovery.
2) With the obtained correspondence, we introduce novel multi-stage self- self-supervised learning pipeline, termed as ORL, for object-level representation learning from scene images, going beyond object-centric ImageNet.
3) We contribute the first study for object-level SSL. ORL substantially outperforms image-level contrastive learning approaches pre-trained on COCO dataset (~1 images with labels discarded), setting a new state of the art on this challenging dataset that contains diverse scenes in the wild. The COCO pre- trainrrieed ORL even surpasses supervised ImageNet pre- pre-training several considered downstream tasks. When SSL is conducted on a larger "COCO+" dataset (COCO train2017 set plus COCO
set, ~241 images in total), ORL further improves the performance, demonstrating its potential to benefit from more unlabeled scene data.
2
Related work
Self-supervised learning. Self-supervised learning builds unsupervised representations by exploiting the internal priors or structures of data in the form a pretext task. A wide range of pretext tasks have been proposed in the past few years. Examples include patch context prediction [10], jigsaw puzzles [39], inpainting [43], colorization [31, 70]. cross-channel prediction [71]. visual primitive counting [40], and rotation prediction [14]. Although good representations emerge with these pretext tasks, they are prone to lose generality due to their hand-crafted nature.
Recently, contrastive learning [20] that performs instance discrimination [60, 22, 37, 19,4, 7] has shown great potential in this field, largely narrowing the gap to fully supervised learning. The core idea of contrastive learning is to gather positive pairs and separate negative pairs in the embedding space. A positive pair is usually formed with two transformed views of the same image while the negative pairs are formed with different images. Typically, contrastive learning methods require large number of negative samples to avoid mode collapse. These samples can be maintained within baini-bach// [42, 27, 66, 26, 2, 5J. memory bank [60, 53, 74, 37] queue [22, 6]. BYOL [19] and SWAV [4] further remove the necessity of involving negative pairs. BYOL directly predicts the features of one view from another view, while SwAV predicts the cluster assignments between multiple views of the same image. Despite their improved performance, the existing image-level contrastive learning methods are largely confined to the underlying object-centric bias of ImageNet.
More recently, a group of works that perform pixel-level [45, 58, 64, 50, 34, 25] or region-level [48, 65, 61, 62, 9] representation learning have emerged. Our work is more related to region-level representation learning but substantially different from this line of research in the following aspects: 1) they still largely pre-train object-centric ImageNet while pre-train non-iconic scene images, 2) they align pre- pre-training specifically for dense prediction downstream tasks while we target at more cnneral-purpose representation learning that improves performance in both dense prediction and classification tasks, 3) their randomly cropped local regions do not contain the explicit object notion ours, and 4) they only rely to construct corresponding positive pairs from the same image while we leverage the discovered high-level semantic correspondence to construct positive pairs across images.
intra-image transformations (e.g., random cropping)
There are also a few prior attempts [18, 3, 16] for self-supervised learning on non-curated scene images. As opposed to our work, most of them consider larger models and datasets to explore the limit of current self-supervised learning methods without further considering the object-level information residing in scene images.
Visual correspondence. Visual correspondence aims at finding pairwise pixels or regions across images that result from the same scene [67], which can be regarded as similarity learning of visual descriptors among matched points or patches. While early efforts learn dense correspondence with labeled data [21, 68, 29, 55, 41], some recent works learn the similarity between the parts or landmarks of the data in an unsupervised manner [52, 51]. Our work substantially differs from this line of research from original intention. Previous works aim at accurately detecting all correspondence given two images, whereas our work focuses on retrieving high-quality correspondence to improve representation learning.
3

---

Imsge-leve pre-training
Stage 2: Correspondence discovery
Stage Object-leve pre-training
simiantt
amilanty
prolkte
cncoder
munnarii croode:
Rol pair renseval
warpal
Rol gencation
KNN image nerirva!
11J0r inter-Rol poir
Figure 2: Overview of our three-stage pipeline. In Stage I, we pre-train an image-level contrastive learning model, e.g. BYOL. In Stage 2, we first use the pre-trained model to retrieve KNNs for each image in the embedding space obtain image-level visually similar pairs. We then use unsupervised region proposal algorithms (e.g., selective search) to generate rough Rols for each image pair. Afterwards, we reuse the pre-trained model to retrieve the top-ranked Rol pairs, i.e., correspondence. We find these pairs of Rols are almost objects or object parts. In Stage 3, with corresponding Rol pairs discovered across images, we finally perform object-level contrastive learning using the same architecture as Stage 1.
3 Methodology
We propose new multi-stage self-supervised learning framework, i.e., ORL, for object-level rep- resentation learning from scene images. ORL extends the existing image-level contrastive learning framework object level by leveraging priors from image-level instance discrimination. The over- all pipeline of ORL is illustrated in Figure 2. It contains three stages: image-level pre- pre-training, correspondence discovery, and object-level pre-training. We detail each stage as follows.
3.1 ORL pipeline
Contrastive learning, Our pipeline contains several contrastive learning modules in Stage and Without loss of generality, we consider BYOL [19] as our basic contrastive learning module. BYOL uses two neural networks: the online network fo(r) and the target network (r()). The target network provides the regression target train the online network while its weights g are updated by an exponential moving average of the online parameters 0 with decay rate t E [0, following B BOL. Given two input images F and 22, the loss function is defined as:
Preliminary:
(z), x2) A II (x1) - gs (12))l2
We name it an "intra-" version of BYOL if and 22 are two augmented views from the same image, otherwise an "inter-" one.
Stage 1: Image-level pre-training. The foremost stage is to obtain an unsupervised pre-trained model from image-level tasks. As shown in Figure 2 Stage 1, given from the same input image v. we pre-train the network following the loss function Limage = L (v, v²), constituting standard image-level BYOL pre-training. This stage can be freely replaced with other lmage-level contrastive learning methods. We adopt BYOL here for its simplicity and effectiveness.
augmented views and o'
Stage 2: Correspondence discovery. We employ the pre-trained image-level contrastive learning model in Stage to mine object-level correspondence for the whole dataset. As shown in Figure Stage the overall discovery process comprises three steps.
(i) Image-level nearest-neighhor retrieval. Specifically, for each query image the training set D, we first retrieve its top K nearest neighbors Ng. 1 1,,.., K, by cosine distance in the embedding space using the features learned from the first stage to form image-level pairs that contain similar visual context.

---

(ii) Region- of of-interest (Rol) generation. To generate object-based RoIs, we apply unsupervised region proposal algorithms, e.g., selective search [56]. for each image in the pair. Considering the redundancy of generated proposals (each image can have thousands of proposals), we filter certain number of them with some pre-defined thresholds including the minimal scale, the range of aspect petereectionovvr--mieee
ratio, and the maximal operation, we select the top 100 proposals ranked with objectiveness as the candidate Rol set for subsequent RoI pair retrieval. To extract features with the equally-sized input that is compatible with the backbone, we crop and resize each Rol to 224 x 224. Note that even top Rols ranked with objectiveness are still very noisy, containing large proportion of non-object regions.
(IoU) among the filtered boxes. After the filtering
(iii) Top-ranked Rol pair retrieval. For each query Rol from x, we compute its cosine similarity in the embedding space with all Rols from its nearest-neighhor image N₂. using the features learned from Stage 1 again. Within the calculated cosine similarity matrix M, E TR. 12:10x1000 we retrieve top-ranked N Rol pairs to construct the set of object-level corresponding pairs { where n {1,.... N). These high-response corresponding regions are almost objects or object parts. Finally, we save the nearest-neighbor image id and bounding box coordinate information of each corresponding pair.
Stage 3: Object-level pre-training. With the corresponding inter-image Rol (inter-Rol) pairs obtained in Stage 2. we perform object-level representation learning following the BYOL framework as shown in Figure Stage Specifically, given an input image Ir we first randomly select one nearest-neighbor image N& to obtain the corresponding set of inter-Rol pairs {B We then randomly select one inter-RoI pair BE as a positive pair. With the bounding box coordinate stored in By. we crop the corresponding inter-Rols from and Mrs respectively, and resize each patch to 96x96, constituting two patches P1 and p2- We feed the two patches to the online network and target network separately to compute the loss Linter = L P2)-
To make full use of discovered objects, we introduce the intra-Rol contrastive learning via augmenting object patches. Specifically, we randomly select one filtered bounding box from x obtained in Stage 2. and spatially jitter the box around its original location with the following operations': : (i) random box center shifting within 50% of its width and height, (ii) random area scaling between 50% and 200% of the original box, and (iii) a random aspect ratio between 1/2 and 2/1. Similarly, we crop the two mutra-Rols and p'. and resize each patch to 96 for forward propagation to compute the loss Limia Rot = L (p. p'), ). The diverse spatial jittering of the bounding box encourages the network to preserve common object information and disregard the background, thus further improving the localization ability.
We keep the two original global views in BYOL as well since they preserve the global image-level information compared with the local patches. The final loss for our ORL can thus be formulated as:
CORL- = Linuge Xe Rol + AiLinteeRRol
(2)
where Å1, A2 As are the loss weights to balance each term. We set all loss weights to 1 by default. Following BYOL, we also compute the symmetric loss CORL by separately feeding v', p', P2 to the online network and v. p. P1 to the target network.
3.2 Implementation details
Dataset. We pre-train our models on the COCO train2017 set that contains ~ 118k images without using labels. Compared with the heavily curated object-centric ImageNet dataset, COCO contains more natural and diverse scenes in the wild, which is closer to real-world scenarios. We also perform sepf-supe learning on a larger "COCO+" dataset (COCO train2017 set plus COCO unlabeled2017 set) to verify whether our method can benefit from more unlabeled scene data.
Image augmentations. The global image augmentation setting is the same as BYOL [19]: 224x224-pixel random resized crop with a random horizontal flip, followed by a random color distortion, random grayscale conversion, random Gaussian blur and solarization. For the local patch augmentation, we directly crop the corresponding morr--Rol and inter-Rot on the input images, and
'In practice, we filter bounding boxes with the minimal scale 96 pixels, the aspect ratio between 1/3 and 3/1, and the maximal IoU as 0.5.
'Inspired by the anchor technique used in object detection, we adopt the similar operations for our bounding box augmentations but in continuous fashion.
5

---

resize each cropped patch to 96 to take place of the random resized cropping. The subsequent augmentations exactly follow the global ones.
Network architecture. We adopt ResNer-50 [24] as the default backbone. We use the same MI.P projector and predictor in BYOL: linear layer with output size 4096 followed by batch normalization (BN) [28], rectified linear units (ReLU) [38], and a final linear layer with output dimension 256. We share the backbone and projector weights among the global and local branches while the weights of predictor are not shared.
Optimization. For pre- pre-traine te--ttaiiing in Stage 1 and Stage Specifically, we use the SGD optimizer with a weight decay of 0.0001 and momentum of 0.9. We adopt the cosine learning rate decay schedule [36] with a base learning rate of 0.2, linearly sc7led[17] with the batch size (Ir = 0.2 x BatchSize/ /256). The batch size is set to 512 by default, which is friendly to typical 8-GPU implementations. To keep the training iterations comparable with the ImageNet supervised pre-training, we train our models for 800 epochs with a warm-up period of 4 epochs. The exponential moving average parameter starts from 0.99 and is increased to during training, following [19]. For correspondence generation in Stage 2, we retrieve top A = 10 nearest neighbors for each image and select top-ranked N 10% Rol pairs for each image-level nearest-neighbor pair.
the training hyper- syper-parameters.
4 Experiments
4.1 Transferring to downstream tasks
We evaluate the quality of learned representations by transferring them to multiple downstream tasks. Following common protocol [18, 37], we use two evaluation setups: (i) the pre-trained network frozen feature extractor, and (ii) the network parameters fully fine-tuned weight initialization. We provide more experimental details the supplementary material.
Pre-train data
VOCO7 mAP
ImageNet Top-1
Places205 Top-1
iNat.
Table 1: Image classifi-
Method
Top-1
cation with linear models. All unsupervised methods are based on 800-epoch pre- training COCO(+) with ResNet-50. We report mAP on the VOCO7 dataset and top-1 center-crop accuracy on all other datasets. Num- bers for all other methods are reproduced by us.
Random [18]
9.6
13.7
16.6
4.8
Supervised [37]
ImageNet
87.5
75.9
51.5
45.4
SimCLR [5]
COCO
78.1
50.9
48.0
22.7
MoCo v2 [6]
COCO
82.2
55.1
48.8
27.8
BYOL [19]
COCO
84.5
57.8
50.5
29.5
ORL (ours)
COCO
86.7
59.0
52.7
31.8
BYOL [19]
COCO+
87.0
59.6
52.7
30.9
ORL (ours)
COCO+
88.6
60.7
54.1
32.0
Pre-train data
VOCO7 low-shot (mAP)
Table Low-shot image classification on VOC07 using linear
Method
2
16
32
64
96
Random Supervied
9.2
9.4
11.1
12.3
14.3
17.4
21.3
23.8
ImageNet
53.0
63.6
73.7
81.8
83.8
85.2
86.0
SVMs trained the
fixed representations.
SimCLR [5]
COCO
33.3
43.5
52.5
61.1
66.7
70.5
73.7
75.0
All unsupervised meth-
MoCo v2 [6]
COCO
39.5
49.3
60.4
69.3
74.1
76.8
79.1
80.1
BYOL [19]
COCO
39.4
50.9
62.2
71.7
76.6
79.2
81.3
82.2
ods are pre-trained
ORL (ours)
COCO
39.6
51.2
63.4
72.6
78.2
81.3
83.6
84.7
on COCO(+) for 800 epochs with ResNet-50. 85,4
BYOL [19]
COCO+
41.1
54.3
66.6
75.2
80.1
82.6
84.6
We report mAP for each case across five runs.
ORL (ours)
COCO+
42.1
54.9
67.4
75.7
81.3
83.7
85.8
86.7
Image classification with linear models. Following [18, 37], we assess the quality of features by training linear classifiers on top of the fixed representations extracted from different depths of network for four datasets: VOCO7 [12], ImageNet [8], Places205 [73], and iNaturalist18 [57]. These
datasets involve diverse classification tasks ranging from object classification, scene recognition to finc- fine-grained recognition. For VOCO7, train linear SVMs using LIBLINEAR package [13]

---

Pre-train data
tection and instance segmentation fine-tuned 1% labels
10% labels
Method
Top-1
Top-5
Top-1
Top-S
Table 3: Semi-supervised learn- ing on ImageNet. All unsuper- vised methods are pre-tained COCO(+) for 800 epochs with ResNet-50. We fine-tune all mod- els with 1% and 10% ImageNet labels, and report both top- and top-5 center-crop accuracy on the ImageNet validation set.
Random
1.6
5.0
21.8
44.2
Supervised [69]
ImageNet
25.4
48.4
56.4
80.4
SimCLR [5]
COCO
23.4
46.4
52.2
77.4
MoCo v2 [6]
COCO
28.2
54.7
57.1
81.7
BYOL [19]
COCO
28.4
55.9
58.4
82.7
ORL (ours)
COCO
31.0
58.9
60.5
84.2
BYOL [19]
COCO+
28.3
56.0
59.4
83.6
ORL (ours)
COCO+
31.8
60.1
60.9
84.4
Pre-train data
COCO detection
COCO instance seg.
Table 4: Object de-
Method
Ap
AP
AP
Apmk
APgu
AP
Random [54]
32.8
50.9
35,3
29.9
47.9
32.0
Supervised [54]
ImageNet
39.7
59.5
43.3
35.9
56.6
38.6
COCO. All unsuper- vised methods are based
SimCLR [5]
COCO
37.0
56.8
40.3
33.7
53.8
36.1
800-epoch pre-training on COCO(+). We use
MoCo v2
COCO
38.5
58.1
42.1
34.8
55.3
37.3
Self-EMD [34]
COCO
39.3
60.1
42.8
Mask R-CNN R50-FPN
DenseCL [58]
COCO
39.6
59.3
43.3
35.7
56.5
38.4
(Ix schedule), and report
BYOL [19]
COCO
39.5
59.3
43.2
35.6
56.5
38.2
bounding-box AP (Apb))
ORI (ours)
COCO
40.3
60.2
44.4
36.3
57.3
38.9
and mask AP (APm)). Numbers for MoCo
BYOL [19]
COCO+
40.0
60.1
44.0
36.2
57.1
39.0
ORL (ours)
COCO+
40.6
60.8
44.5
36.7
57.9
39.3
are adopted from [58].
following the setup in [18, 37J. We train on trainval split of VOCO7 and evaluate mAp on test split. For ImageNet, Places205 and iNaturalist18, we follow [71, 18, 37] and train a 1000-way, 205-way and 81 8142-way linear classifier, respectively. We train train split of each dataset, and report top-1 center-crop accuracy on the respective val split. Table 1 reports the results for the best-performing layer of each method. ORL substantially outperforms the BYOL baseline all four datasets. We also observe that the COCO pre-trained ORL surpasses the supervised ImageNet pre-trained counterpart Places205 by 1.2% in top-1 accuracy. This is the first time that self- supervised learner outperforms the ImageNet pre-training using only ~~110 images compared with ImageNet. When pre-trained a larger COCO+ dataset, ORL again outperforms BYOL. Note that apart from Places205 (2.6% gains), ORL also surpasses the supervised ImageNet counterpart on VOCO7 by 1.1 % mAP, using merely 155 images.
Low-shot image classification. We perform low-shot image classification with few training examples per class VOCO7 dataset following the same setup in [18]. We vary the number of labeled examples per category used to train linear SVMs on train split of VOCO7 and report the average mAp across five independent samples for each low-shot case evaluated on test split. Table 2 provides the results. ORL shows consistent performance improvement over BYOL for each low-shot value, with larger gains achieved the number of labeled examples per class is inereasing. ORL also gradually bridges the gap to the supervised ImageNet pre-training under this scenario. We observe consistent performance boost when pre-training the COCO+ dataset. Note that the COCO+ pre-trained ORL again outperforms the supervised ImageNet pre-training when the low-shot samples are 64 and 96.
Semi-supervised learning. We perform semi-superviser learning ImageNet following the proto- col previous studies [60, 26, 37, 5, 19]. Specifically, we first randomly select 1 % and 10% labeled data from ImageNet train split. We then fine-tune models these two training subsets and report both top- and top-5 accuracy on the official val split of ImageNet in Table 3. Again, ORL outperforms BYOL as well as the supervised ImageNet counterpart by large margins.
Object detection and segmentation. We train Mask R-CNN model [23] with R50-FPN back- bone [32] implemented in Detectron2 [59]. We fine-tune all layers end-to-end on COCO train2017 split with the standard 1 x schedule and evaluate on COCO va12017 split. We follow the same setup in [54], with batch normalization layers synchronized across GPUs [44]. As shown in Table 4,
ORL yields 0.8% AP and 0.7% AP improvements over BYOL for object detection and instance segmentation, respectively. The improvements are consistent over all evaluation metrics. When
7

---

Table Ablations for ORL. (a) Effect of intra-Rol and inter-Rol losses. (b) Effect of NNs and Rol pairs. (c) Comparison with multi-crop BYOL. (d) Comparison with ground truth bounding boxes. (e) Pre-trainig schedules. We report mAP of linear SVMs on VOCO7 classífication benchmark.
(h)
pre-train intra-Rol inter-Rol
VOC07
of NNs
# of Rol pairs
BYOL
84.5
10
20
5%
10%
20%
85.7
ORL
85.9
VOCO7
84.7
86.7
87.0
86.1
86.7
86.4
86.7
(c)
(d)
(c)
pre-train
input views
VOCO7
boxes
VOCO7
pre-train
100
200
400
800
1600
BYOL 2 224 + 4 96
84.0
GT
85.4
BYOL
77.1
81.8
83.7
84.5
84.9
ORL
2 224 + 96
86.7
SS
86.7
ORL
83.5
85.2
86.3
86.7
87.1
pre-trained the COCO+ dataset, ORL again outperforms BYOL. It should be well noted that ORL even outperforms the most recent Self-EMD and DenseCL that are specifically designed for dense prediction downstream tasks. More importantly, either COCO or COCO+ pre-trained ORL can surpass the supervised ImageNet pre- pre-training on all metrics. This further demonstrates the superiority of learning unsupervised representations the object level.
4.2 Ablation study
In this subsection, we conduct extensive ablation experiments to examine the effect of each component that contributes ORL. We pre-train our models COCO and observe the downstream performance of all ablations on VOCO7 SVM classification benchmark as introduced in Section 4.1.
Effect of Intra-Rol and inter-Rol losses. Table 5a ablates the effect of our introduced Limua-Rol and Cinter-Rol losses in Equation 2. Adding either Linaa-Rol or LinterRRol can improve the performance, with the best results obtained by adding both terms.
Effect of nearest neighbors and Rol pairs. Table 5b ablates the effect of the number of nearest neighbors K and Rol pairs N used for generating inter-Rol pairs in Stage of ORL. We set N 10% when ablating K, and set K 10 when ablating N. We observe that retrieving more nearest neighbors leads to better performance since more nearest neighbors provide more diverse image-level pairs to the subsequent generation of inter-Rol pairs. Although setting K = 20 produces slightly better performance, we choose K = 10 by default as trade-off considering the tendency of the saturated performance. Our method more robust to the number of retrieved top-ranked Rol pairs after image-level nearest-neighbor retrieval, with N 10% performing slightly better.
Comparison with multi-crop BYOL. Prior work [4] has indicated that cropping multiple views of the same image can improve the performance of self-supervised learning methods pre-trained on comparison or simply more number of mixed views, we randomly crop four additional smaller views for B BOL to ensure the number and size of the input patches are equal ORL (i.e., 2 x 224 x 96). As shown in Table 5c, different from the observation on ImageNet, simply adding more low-resolution crops tends to hurt the performance since will further intensify the inconsistent noise on scene images. In contrast, ORI substantially outperforms this multi-crop variant, validating that the gains are truly due to our object-level representation learning mechanism.
ImageNet. To investigate whether our gains are due to more accurate object-instanee
Comparison with ground truth bounding boxes. In Stage 2, ORL requires an unsupervised region proposal algorithm to extract approximate object-based regions, which inaccurate some extent. We further investigate whether the performance can be improved when more accurate object regions are available, i.e., with bounding box annotations. To this end, we replace our ddeectivessarrh generated object proposals with ground truth bounding boxes provided from COCO train2017 set, while keeping all other procedures unchanged. As shown Table 5d, adopting ground truth (GT) bounding boxes performs inferior to selective search (SS). This is mainly due to that although the ground truth bounding boxes can provide more accurate object location, their numbers are too scarce compared with large amount of region proposals generated by selective search. The more diverse
8

---

Figure 3: Top-ranked region correspondence discovered by ORL in Stage 2. We show pair of discovered object-instance per image pair for clarity. More discovered correspondence pairs are provided in the supplementary material.
Image
BYOL
ORL
Image
BYOL
ORL
Figure 4: Attention maps generated by BYOL and ORL. ORL can activate more object regions and produce more accurate object boundary in the heatmap than B BYOL. We provide more attention maps in the supplementary material. Best viewed with zoom in.
region proposals can potentially induce more unknown object or object-put parctt-prtttoo discovery beyond the manually annotated objects. In this case, the diversity make up for the inaccuracy.
Pre-training schedules. Table 5e shows the results with different pre-training schedules, from 100 epochs to 1600 epochs. The performance of both ORL and BYOL improves when pre-trained for longer epochs, while ORL consistently outperforms BYOL by at least 2.2% mAP. Note that our 200-epoch ORL has already surpassed the 1600-epoch B BOL (85.2) 25.2% mAP vs. 84.9% mAP), demonstrating that the performance efficiency of ORL least 8x than BYOL below 1600 epochs.
4.3 Visualization
Correspondence pairs. Figure visualizes some top-ranked region correspondence discovered in Stage 2 of ORI. We observe that each generated inter-Rol pair largely correspond to the regions with similar visual concepts (i.e., objects or object parts) across images. contrast to typical contrastive learning methods that perform aggressive intra-image augmentations to simulate intra-class variances, our discovered inter-Rol pairs can substantially provide more natural and diverse intra-class variances at the object-instance level.
Attention maps. Figure visualizes the attention maps generated by BYOL and ORL. We observe that both BYOL and ORL can produce relatively high- attention maps that focus on the foreground objects. This reflects from the side that current image-level contrastive learning methods have already induced a latent space with rich visual concepts. Nevertheless, ORL can activate more object regions and produce more accurate object boundary than BYOL in the generated attention maps. It is mainly due to introducing object-level similarity learning into ORL, which can minimize

---

the inconsistent noise caused by image-level contrastive learning. In contrast, BYOL only uses the whole image to extract features, thus activating the most discriminative region.
5 Conclusion
In this work, we have presented a new self-supervised learning framework, ORL, for object-level representation learning from scene images. We leverage the latent prior of image-level self-supervised pre-training for discovering object-based region correspondence across images. The generated object- instance correspondence enables us perform pairwise contrastive learning at the object level. ORL significantly improves the performance of self-supervised learning from scene images in a variety of downstream tasks. We expect that our method can be applied to larger-scale unlabeled data in the wild to fully realize its potential, and hope that our study can attract the community's ' attention to more general-purpose purpose unsupervised representation learning from scene images.
Limitations
In this paper, we mainly perform pre-training experiments with ResNet-50 on COCO dataset, and further scale them up on COCO+ dataset. However, the promise of self-supervised learning is to harness massive unlabeled data by scaling up to ever-larger datasets. Some prior works [18, 3, 16] have attempted to leverage larger models and datasets explore the limit of current self-supervised learning methods. For instance, a recent representative work SEER [16] performs billion-scale self- supervised pre-training on internet images using the RegNet architectures [46] with 700M parameters over 512 GPUs. Training at scale requires huge computational resources that are inaccessible to many researchers. which not the core of our paper. We wish to highlight that our general- purpose ORL has yielded better performance than concurrent works [58, 34] that are tailored for dense prediction downstream tasks when pre-trained on COCO (Table 4), even surpassing the supervised ImageNet pre-training on several downstream tasks (Table 1-4). We expect that scaling ORL with larger architectures and datasets can further unleash potential. Besides, ORL may not handle well on images with cluttered backgrounds since they will deviate the generated proposals to focus on these background regions. A possible remedy is to use some heuristic algorithms like saliency estimation to avoid the background regions. Another limitation is that ORL is multi-stage framework. We expect an end-to-end framework to further improve the efficiency. We leave these explorations to future work.
Broader impact
We present a more effective approach for learning unsupervised visual representations. Compared to supervised learning, it can liberate humans from expensive annotations as well as take advantages of rapidly growing real-world data. Like other learning algorithms, self-supervised learning should be applied with cautions when deployed in the real-world scenario. First, it is susceptible to biased learning if the algorithm is given with biased data. The exposure to unlabeled data may amplify such biases. Thus, debiasing measures have to be taken. Second, it remains non-trivial to dissect what learned by supersuper models. Similar concerns about the calibration, robustness, and interpretability of supervised models are equally applicable to the unsupervised counterpart. Our work is limited to the improvement of self-supervised learning within our scope. However, we acknowledge the importance of providing more transparent explanations for classification decisions, well as the credibility of each prediction. Finally, our method still relies on the traditional regime of centralized learning. Privacy can be compromised if the method is applied on unsaved platform. Federated learning can be a solution. How to scale self self-supervised learning to the regime of decentralized learning will be an interesting research question to answer.
Acknowledgements
This study is supported under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). The project is also supported by Singapore MOE AcRF Tier 2 (725020020-0005), the Data Science and Artificial Intelligence Research Center NTU.
10

---

References
[1] Mehdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu, Chi-Heng Lin, Erik C Johnson, Kiran Bhaskaran- Nair. Max Dabagia, Keith B Hengen, William Stay-Roncal Michal Valko, al. Mine your own view: Self-supervised learning through across samole prediction. arXiv preprint arXiv:2102.10100 2021.
[2] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NewrlPS, pages 15509-15519, 2019. 3
[3] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre- pre-training of image
features on non-curated data. In ICCV, pages 2959-29688, 2019. 3. 10
[4] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In NewrIPS, 2020. 1, 8
[5] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 1, 2, 6, 7
[6] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved basclines with momentum contrastive
learning. arXiv preprint arXiv:2003.0 244,09000,,4,7,, 2020. 3, 6.
[7] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, pages 15750-15758 2021. 2,
[8] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li. Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical
image database. In CVPR. pages 248-255, 2009. 1, 6
[9] Jian Ding, Enze Xie, Hang Xu, Chenhan Jiang, Zhenguo Li, Ping Luo, and Gui-Song Xia. Unsupervised pretraining for object detection by patch reidentification. arXiv preprint arXiv: 203::2/03300814 2021. 3
[10] Carl Doersch, Abhinay Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015. 3
[11] Debidatta Dwihedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest- searest-neighbor contrastive learning of visual representations. In ICCV, 2021. 2 [12] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88(2)30 2010. 6
[13] Rong- En Fan, Kai-Wei Chang. Cho-Jui Hsich, Xiang- Rui Wang, and Chih-Jen Lin. Liblinear: A library for large linear classification. JMLR, 9:1871- 2008.
[14] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting
image rotations. In ICLR, 2018. 3
[15] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hicrarchies for accurate object detection and semantic segmentation. In CVPR, pages 580-587, 2014. 1
[16] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervieed pretraining of visual features in the wild. arXiv preprint arXiv:2103 201.0013..900 2021. 3. 10
[17] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv
preprint /706 02677, 2017. 6
[18] Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self-supervised
visual representation learning. In ICCV, pages 6391 6391-6400, 2019. 3, 6, 7, 10
[19] Jean- Bonn-Battinn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap
your own latent: A new approach to self-supervised learning. In NeurlPS, 2020. 1, 2, 3, 4, 5, 6,
[20] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping.
In CVPR, 2006. 3
[21] Xufeng Han. Thomas Leung, Yangging Jia. Rahul Sukthankar, and Alexander C Berg. Matchnet: Unifying
feature and metric learning for patch-based matching. In CVPR, 2015. 3
[22] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised
visual representation learning. In CVPR, pages 9729-9738. 2020. 1. 2,
[23] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-can. In ICCV, pages 2961-2969.
2017. 1,7
[24] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, 2016. 6
[25] Olivier J H Skanda Koppula, Jean-Baptiss Alayrac, Aaron van den Oord, Oriol Vinyals, and João Carreira. Efficient visual pretraining with contrastive detection. In ICCV, 2021.
[26] Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv: / 1905. 000509972, 2019. 3.7
[27] R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler. and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019. 3
11

---

[28] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing
internal covariate shift. In ICML, 2015.
[29] Angjoo Kanazawa, David W Jacobs, and Manmohan Chandraker. Warpnet: Weakly supervised matching
for singlc-view reconstruction. In CVPR, 2016. 3
[30] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional
neural networks. In NeurlPS, 2012. 1
[31] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic
colorization, In ECCV, pages 577-593, 2016. 3
[32] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, pages 2117-2125, 2017. 7
[33] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 1
[34] Songtao Liu, Zeming Li, and Jian Sun. Self-emd: Self-supervised object detection without imagenet. arXiv
preprint arXiv:201 1. 13677, 2020. 3. 7, 10
[35] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta-
tion. In CVPR, 2015.
[36] Nyy Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint
arXiv:1608 03983, 2016, 6
[37] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarian representations. In
CVPR. pages 6707-6717, 2020. 1, 6, 7
[38] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICMI.,
2010. 6
[39] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016. 3
[40] Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count. In ICCV, pages 5898-5906, 2017. 3
[41] Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi. Lf-net: Learning local features from images. arXiv preprint
/805,088000966 2018. 3
[42] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint a8Xiv: 087.03388 2018. 3
[43] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders:
Feature learning by inpainting. In CVPR, 2016.
[44] Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun. Megdet:
A large mini-batch object detector. In CVPR, pages 6181-6189 2018. 7
[45] Pedro O Pinhciro, Amjad Almahairi, Ryan Y Benmaleck, Florian Golemo, and Aaron Courville. Unsuper-
vised learning of dense visual representations. In NeurlPS, 2020. 3
[46] Iliaa Radosavovic, Raj Prateek Kosaraju. Ross Girshick. Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, pages 10428- 10436, 2020. 10
[47] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection
with region proposal networks. In NeurlPS, pages 91-99, 2015.
[48] Byungseok Roh, Wuhyun Shin, I1doo Kim, and Sungwoong Kim. Spatially consistent representation learning. In CVPR, pages
2021. 2151,2021. 3
[49] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge.
JJV, 115(3)2211-2523 2015.
[50]
Ramprasaath R Selvaraju, Karan Desai, Justin Johnson, and Nikhil Naik. Casting your model: Leamingg to
localize improves self-supervised representations. In CVPR, pages 11058 11058-11067, 11067, 2021. 1,
[51] James Thewlis, Samuel Albanic, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of landmarks by descriptor vector exchange. In /CCV, 2019. 3
[52] James Thewlis, Andrea Vedaldi, and Hakan Bilen. Unsupervised object learning from dense equivariant
image labelling. In NewrIPS, 2017.
[53] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: weeiii9060.00 05849, 2019. 3
[54] Yonglong Tian. Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. In NeurlPS, 2020.
[55] Nikolai Ufer and Bjorn Ommer. Deep semantic feature matching. In CVPR, 2017. 3
[56] Jasper RR Uijlings, Koen FA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. IJCV, 104(2):154-171, 2013. 2,5
[57] Grant Van Horn, Oisin Mac Aodha, Yang Song. Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, pages 8769-877 8778, 2018. 6
12

---

[58] Xinlong Wang, Rufeng Zhang, Chunhua Shen. Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, pages 3024-3033, 2021. 3, 10
[59]
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019. 7 Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric
[60]
instance discrimination. In CVPR. pages 3733-3742, 2018. 1, 3.7
[61] Tete Xiao, Colorado Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. ICCV, 2021.
[62]
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo.
Detco: Unsupervised contrastive learning for object detection. In ICCV, pages 8392-8401, 2021. 3
[63] Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Delving into inter-image
invariance for unsupervised visual representations. arXiv preprint arXiv:2008. //022, 2020. 2
[64] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring
pixel-level consistency for unsupervised visual representation learning. CVPR, pages 16684-16693,
2021. 3
[65] Ceyuan Yang. Zhirong Wu, Bolei Zhou, and Stephen Lin. Instance localization for self-supervised detection pretraining. CVPR, pages 3987-3996, 2021.
[66] Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant
and spreading instance feature. In CVPR, pages 6210-6219, 2019. 3
[67]
Ramin Zabih and John Woodfill, Non-parametric local transforms for computing visual correspondence.
In ECCV, 1994. 3
[68] Sergey Zagoruyko and Nikos Komodakis. Learning to compare image patches via convolutional neural
networks. In CVPR, 2015. 3
[69] Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi-supervised
learning. /CCV, pages 1476-1485, 2019.
[70] Richard Zhang. Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
[71] Richard Zhang. Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by
cross coos--channel prediction. In CVPR, 2017. 3.7
[72] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: 10 million image database for scene recognition. TPAMI, 40(6):1452 11552-1444, 1464, 2017.
[73] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features scene recognition using places database. In NeurlPS, pages 487-495, 2014.
[74] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In /CCV, pages 6002-6012, 2019. 2,
13


Reviews:
[{"review_id": "bbscr-6UA9", "reviewer": "Reviewer_fNXx", "summary": "This paper aims to learn object-centric self-supervised representation from non-iconic scene images. In addition to the standard self-supervised objective (e.g., BYOL), the paper suggests attracting the object patch with the (i) nearby patch of the chosen object and (ii) similar objects in different images. For (ii), the paper uses the similarity score of the standard image-level self-supervised model. This additional training signal from similar object patches improves the transfer performance of the learned representation on classification and detection/segmentation tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Learning representations from non-iconic images is an important direction of self-supervised learning. There have been a large number of concurrent works on this topic (including DenseCL and Self-EMD discussed in the paper), but most of them reported the trade-off between image-level (e.g., classification) and pixel/object-level (e.g., detection/segmentation) tasks. Unlike the prior work that leverages the spatial features before the final pooling layer to improve detection performance, this paper simply matches the pooled features of related object patches, improving both classification and detection performance. Furthermore, the paper demonstrates that the model can find the visual correspondence of objects and produces a better attention map.\n\nWhile the paper is well-written and easy-to-follow, I have two questions in detail:\n\nI. How does the three-stage pipeline is exactly implemented?\n\n- How long does the image-level model in Stage 1 pretrained? Is it a strict two-step approach of training Stage 1 and Stage 3 for 800 epochs, respectively? Does Stage 3 reinitialize the model or fine-tune the one from Stage 1? If latter, the other baseline models also should be fine-tuned again using the same computational budget.\n- How often does Stage 2 perform? Are the correspondence maps only computed once? Do the correspondence maps become better when using the model trained from Stage 3? If it gives a better correspondence, would additional repetitions of Stage 2 and 3 give further improvement?\n\nII. How does the attention maps are produced?\n\n- The results seem promising, but I could not find the implementation detail. Also, it would be informative to provide quantitative results.\n\np.s. Typo: Missing punctuation after textbf in line 203.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Both limitations and societal impact are well discussed in Appendix, though they could be moved to the main text following the intention of extending the page limits to nine pages.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "bB66Pri-41h", "reviewer": "Reviewer_5LqK", "summary": "This paper presents a new self-supervised learning framework that leverages scene images to enhance object-level representation learning that only relies on object images. The basic idea is to apply unsupervised region proposal and k-NN based object similarity to diversify intra- and inter-image pairs with more variances to aid object-level representation learning. The improvements on various downstream tasks, the ablations and visualizations all demonstrate the advantages of the presented framework. The main contribution is to utilize scene images for contrastive self-supervised learning.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main novelty of the paper is to enhance the contrastive self-supervised learning by including not only object images from ImageNet, but also scene images from Coco. The paper is written well in general and the experiments are extensive.\n\nI did not give a higher rating due to the following:\n- The technical contribution of the paper is not very significant as the multi-stage framework largely depends on previous contrastive learning work as BYOL. The improvement is basically due to the data augmentation from Coco which is not technically difficult.\n- The generality of the framework could have been better demonstrated by not just on BYOL. Please discuss other possibilities in more detail.\n- The trade-off between the performance gain and the additional cost due to multiple stages needs to be discussed more. \n\nSome minor things of writing:\n- Please cite the published paper other than its preprints on arXiv (e.g., BYOL)\n- L169 the symmetric loss is not clear here.\n- L104 ‘detect’ -> ‘detecting’\n- L129 ‘is comprised of’ -> ‘comprises’ \n- L203 ‘models Following’ -> ‘models. Following’\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and broader impact of the work are discussed in the supplementary material and look good to me.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "IbkPXUyBQM6", "reviewer": "Reviewer_k49a", "summary": "This work proposes a technique to adapt self-supervised contrastive learning to scene images where multiple objects are present in each image. Specifically, the proposed technique has three stages: 1. image-level pre-training with standard contrastive learning; 2. Finding object pairs images (crops) using selective search proposals and pre-trained features from stage-1; and 3. Doing contrastive learning with object image pairs. Several experimental comparisons demonstrate consistent improvements w.r.t. baselines that does only image-level contrastive learning.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Using self-supervised image features and selective search for correspondence object discovery is simple yet interesting technique.\n- Experimental results on wide range of datasets and on several tasks.\n- Consistent performance improvements w.r.t. image-level self-supervised learning techniques.\n\nWeaknesses:\n- The proposed stage-wise technique is more like an engineering solution to mine more/cleaner positive and negative pairs for contrastive learning. As such, the technical innovation is incremental.\n- There is no discussion or comparisons with several works that extend image-level self-supervised learning to more local pixel features. Couple of works that I can find: [a,b].\n- L65 says that this is the first study to extent self-supervised learning beyond image-level contrastive learning. This is an over-claim and not correct in my opinions as there are several existing works that propose pixel-level contrastive learning techniques.\n\nClarifications:\n- Line 51 says that the proposed technique uses 'off-the-shelf contrastive learning pre-trained model'. Does this mean the stage-1 is already pre-trained on ImageNet data? Or, is it trained from scratch on COCO dataset? This point is important to clarify in the rebuttal.\n\n[a] Pinheiro et al. Unsupervised learning of dense visual representations. NeurIPS 2020.\n[b] Chaitanya et al. Contrastive learning of global and local features for medical image segmentation with limited annotations. NeurIPS 2020.\n\n**Post Rebuttal**:\n\nIt seems reviewers generally agree that the proposed approach is more like an engineered system rather than providing any sound technical innovations. Like other positive reviewers, I do not find any clear issues in this work and experiments are reasonably convincing. It would be better if authors had provided experimental comparisons to more dense pixel-level contrastive learning works. After carefully considering author response and reviewer discussion, I tend to keep my original rating (6).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and broader impact are added in the supplementary. It is better to make the discussion of limitations more specific to this work (like, when the proposed algorithm fails and possible remedies) instead of mentioning general limitations of learning on large-scale datasets.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "DVkmpe_CAYc", "reviewer": "Reviewer_JkCU", "summary": "This paper proposes an object-level self-supervised pre-training approach which extends and tries to improve existing work on image-level self-supervised pre-training. The authors motivate the problem by mentioning that image-level self-supervised pre-training methods are infeasible for images with many objects due to the hard constraints imposed by such methods. The paper proposes to apply self-supervised pre-training to similar objects in and across images. The authors use pre-trained image-level model to select the best candidate object pairs across images and then pre-train self-supervised models using the selected object regions. The authors report some performance improvements on the COCO dataset. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a slightly interesting approach for self-supervised pre-training. The proposed approach achieves promising initial results - improving over existing image-level pre-training approaches and even fully-supervised baselines. The motivation behind this work is sound. and the authors have explained the limitations of existing works well.\n\nHowever, a large number of aspects have not been considered in the paper. Such aspects need to be dealt with to ensure completeness of the paper. I list a few major limitations of the paper here. \n\n1. The most important limitation of the paper is a missing study on how the proposed approach scales with the availability of more data. The authors should try to train their model on larger datasets like OpenImages. This will show whether the proposed approach has any potential for larger-scale training.\n\n2. The use of a image-level model to first find the nearest images/objects and then training the same model on the retrieved objects seems circular. It's not clear why the model should change/learn anything new if the similar objects are already retrieved using the same model. What additional information does this framework gain?\n\n3. The authors should also provide a baseline by training the ORL on the ImageNet dataset to highlight the problem with using iconic images for pre-training. \n\n4.  In table 5d, the authors compare the performance when using selective search vs the performance when using ground-truth boxes. The authors show that using the ground-truth boxes leads to a lower performance than when using selective search. The authors hypothesize that this might be due to more diverse boxes returned by selective search. Did the authors try using jittered ground-truth boxes to induce more diversity in the ground-truth boxes? That might lead to better performance than selective search.\n\n5. Finally, I think the paper can be made much stronger if the authors conduct downstream experiments on datasets other than COCO. Does the proposed approach lead to similar performance improvements when the pre-training dataset is different from the downstream dataset? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately discussed the limitations and societal impact, but only in the supplementary material and not as part of the text of the main paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: - The proposed method is tackling an important problem. The reviewers found the approach is reasonable but technically not very novel. 
- The many concerns from the reviewers are addressed well enough by the rebuttal. 
- The strength is mainly the experiment results which are wide, consistent and reasonably convincing.
- The clarity is good enough but can be improved. 
Decision: Accept (Poster)