Object-aware Cropping for Self-Supervised Learning
Authors: Shlok Kumar Mishra
Anshul Shah
Ankan Bansal
Janit K Anjaria
Abhyuday Narayan Jagannatha
Abhishek Sharma
David Jacobs
Dilip Krishnan
Venue: Accepted by TMLR ()

Abstract: A core component of the recent success of self-supervised learning is cropping data augmentation, which selects sub-regions of an image to be used as positive views in the self-supervised
loss. The underlying assumption is that randomly cropped and resized regions of a given
image share information about the objects of interest, which is captured by the learned
representation. This assumption is mostly satisfied in datasets such as ImageNet where
there is a large, centered object, which is highly likely to be present in random crops of
the full image. However, in other datasets such as OpenImages or COCO, which are more
representative of real world uncurated data, there are typically multiple small objects in
an image. In this work, we show that self-supervised learning based on the usual random
cropping performs poorly on such datasets (measured by the difference from fully-supervised
learning). Instead of using pairs of random crops, we propose to leverage an unsupervised
object proposal technique; the first view is a crop obtained from this algorithm, and the
second view is a dilated version of the first view. This encourages the self-supervised model
to learn both object and scene level semantic representations. Using this approach, which we
call object-aware cropping, results in significant improvements over random scene cropping on
classification and object detection benchmarks. For example, for pre-training on OpenImages,
our approach achieves an improvement of 8.8% mAP over random scene cropping (both meth-
ods using MoCo-v2). We also show significant improvements on COCO and PASCAL-VOC
object detection and segmentation tasks over the state-of-the-art self-supervised learning
approaches. Our approach is efficient, simple and general, and can be used in most existing
contrastive and non-contrastive self-supervised learning frameworks.

Published in Transactions on Machine Learning Research (12/2022)
Object-aware Cropping for Self-Supervised Learning
Shlok Mishra', Anshul Shah", Ankan Bansal', Janit Anjaria',
Abhyuday Jagannatha', Abhishek Sharma, David Jacobs', Dilip Krishnan University of Maryland, College Park,
"Johns Hopkins University, *University of Massachusetts Amherst
Google Research
{shlokm, dwj]0umd edu, diliphayOgoogle.com
Reviewed on OpenReview: httppr//ppaaaeeesss...e////aaaaa
Abstract
A core component of the recent success of self-supervised learning is cropping data augmenta- tion, which selects sub-regions of an image to be used as positive views in the self-supervised loss. The underlying assumption is that randomly cropped and resized regions of a given image share information about the objects of interest, which is captured by the learned representation. This assumption is mostly satisfied in datasets such as ImageNet where there is a large, centered object, which is highly likely to be present in random crops of the full image. However, in other datasets such as Openlmages or COCO, which are more representative of real world uncurated data, there are typically multiple small objects in an image. In this work, we show that self-supervised learning based the usual random cropping performs poorly on such datasets (measured by the difference from fully-supervised learning). Instead of using pairs of random crops, we propose to leverage an unsupervised object proposal technique; the first view is a crop obtained from this algorithm, and the second view is a dilated version of the first view. This encourages the self-supervised model to learn both object and scene level semantic representations. Using this approach, which we call object-owar cropping, results in significant improvements over random scene cropping on classification and object detection benchmarks. For example, for pre-training OpenImages, our approach achieves an improvement of 8.8% mAP over random scene cropping (both meth- ods using MoCo-v2). We also show significant improvements on COCO and PASCAL-VOC object detection and segmentation tasks over the state-of-tho-art sclf-supervised lcarning approaches. Our approach efficient, simple and general, and can be used in most existing contrastive and non-contrastive self-supervised learning frameworks.
1 Introduction
In recent works on self-supervised learning (SSL) of image representations, the most successful approaches have used data augmentation as a crucial tool (Chen et al.] 2020a He et al. 2019 Grill et al. 2020 Tian
et al. 2019 Caron et al. 2020b). Given a randomly chosen image sample, augmentations of the image are generated using common image transformations such cropping and resizing a smaller region of the image, color transformations (hue, saturation, contrast), rotations etc. (Chen et al. 2020a. Gidaris et al. 2018). Of these augmentations, the use of cropping is clearly the most powerful (see Chen et al. (2020a), Fig. 5). This makes intuitive sense: cropping followed by resizing forces the representation to focus on different parts of an object with varying aspect ratios. This makes the representation robust to such natural transformations as scale and occlusion. The implicit assumption in this scheme is that the object of interest (classification or detection target) occupics most of the image and is fairly centered in the image, so that random crops of the image usually result in (most of) the object still being present in the cropped image. Such an assumption holds for "iconic" datasets such ImageNet Krizhevsky et al. (001)). Forcing the resulting representations

---

Published in Transactions on Machine Learning Research (12/2022)
X
First Random Crop
Second Random Crap
Object Crop
Dilated Objact Crop
Figure 1: Illustration of object aware cropping. Top-Left: We show the original image with random crops overlaid. Bottom (red panel): Overlap between random crops tend to miss the object of interest. Top-Right: We show crops generated from the LOD Vo et al. (2021) algorithm and also the dilated object crop. Bottom- Right (green panel): We use LOD-based object-aware crops. These incorporates both object and scene information into the MoCo-v2 (or other SSL frameworks).
to closer together maximizes the mutual information between the crops (also called views) van den Oord
ct al. 20118: Tian et al. (2019).
However, in the case of "non-iconic" datasets such as OpenImages Kuznetsova al. 2020 and COCO Lin
ct al. (2014), the objects of interest small relative to the image size and rarcly centered, Fig, 1 These datasets are more representative of real-world uncurated data. We find that the default random cropping approach (which we call scene cropping) leads to a significant reduction in performance for self-supervised contrastive learning approaches. For example, using the default pipeline of MoCo-v2 Chen et al. (2020b), we find that there is a gap of 16.5% mean average precision (mAP) compared to fully supervised learning. Other state of the art methods such as BYOT. Grill et al. (2020), SwAV Caron et al. (202bb and CMC TTia et al. (2019) perform poorly as well (see Table 2). As show, the core problem here is that random scene crops do not contain enough information about (small) objects, causing degraded representation quality.
However, merely switching from scene-level crops to purely object-level crops does not exploit the correlations that exist between scenes and objects in most natural images. These correlations are helpful for downstream tasks Xiao et al. (2020). Keeping this mind, we introduce Object-A ware Cropping(OAA) (OAC), which applies a simple pre-processing step using the unsupervised object proposal method LOD Vo et al. (2021). LOD outputs multiple object proposal rectangles, one of which we pick at random as the first candidate region. We then expand (dilate) this rectangle to create second candidate region. We finally employ random cropping within each of these rectangles to create final pair of "positive' views that used for the SSL loss. The use of random cropping reduces mutual information between the views, thereby making the pretext task harder for SSL losses, and improving final representation quality. We call this cropping approach "obj-obj-dilate" in the rest of the paper.
In addition to an SSL loss that uses multiple views, we introduce two additional unsupervised losses which leverage the LOD proposal introduced above. The first loss, which we call object localization, encourages the network's representations to carry information about objectness by predicting which patch features
2

---

Published in Transactions on Machine Learning Research (12/2022)
Method
Dataset
mAP
Supervised
53.26
InsDis Wu et al.
(2018b)
ImageNel Net
48.82
MoCo
He et al.
2020
ImageNet
50.51
PCL-v1
2021
ImageNet
53.93
PCL-v2
al.
2021
IetgeNet
53.92
MoCo-v2 (Chen & He| (2020)
ImageNet
50.67
MoCHI Kalantidis et al. (2020)
ImageNet
52.61
MMCL(Shah et al. (2021
ImageNet
50.73
MoCo-v2 + OAC [Ours)
ImageNet
53.96
MoCo-v2 (Baseline)
OHMS
54.74
MoCo-v2 + OAC (Ours)
OHMS
57.13
Table 1: We achieve superior performance on VOC detection when pre-training on our proposed OpenImages Hard Multi-object Subset OHMS dataset as compared to ImageNet trained models by 6.52mAP. Our proposed OAC also helps ImageNet (2nd last row), but it helps much more on the proposed OHMS multi- object dataset. ImageNet basclines has been trained for 100 epochs Shah al. (2021) and OpenImages model have been trained for same 100 ImageNet equivalent epochs.
contain the object (labels are determined by the extent of the LOD proposal). The second loss we add is the rotation prediction task where given rotated object and dilated-object, we predict the rotation of the object. Rotation loss helps is learning better object level represenlations. Use of these losses leads to further improvements to downstream performance.
We conduct a number of experiments incorporating object-aware cropping, finding consistent improvements state of the art self-supervised methods such as MoCo-v2 Chen et al. (2020b), BYOL Grill et al. (2020) and Dense-Ci CL Wang et al. (2021) | across varied datasets and tasks. We also propose Openlmages Hard
Multi-object Subset OHMS, which is a balanced subset of OpenImages and has images with atleast two different classes. We show by pre-training on the OHMS dataset using our OAC can give superior performance
on object detection task IAA.1 Table as compared to pre-training on ImageNet which has been used
extensively in the literature He et al. (2019); Chen et al. (2020a); Caron et al. (2020a).
2 Related Work
Recent progress in self-supervised learning, basod on contrastive and non-contrastive approaches, has achicved excellent performance on various domains, datasets and lasks He el al. (2019); Chen al. (2020b a van den Oord al. (2018); Tian al. (2019); Gidaris et al. (2020): Misra van der Maaten (2019); Tian al. [2020]] Wu et al. 2018a Grill et al. 22000 Gidaris al. (2018) Larsson et al. (2017) : Noroozi et al. (2088) ) Pathak et al. (2016]. The top-performing methods have all used related ideas of pulling together "views' of sample in representation space. Some of these approaches, in addition, use negative samples to add "push"
factor, and this is termed contrastive self-supervised learning. Theoretical and empirical studies have been
published to better understand the behavior and limitations of these approaches Arora et al. (2019): Xiao
et al. (2021 ;: Purushwalkam Guptal (2020); Tosh et al. (2021); Wang & Isola (0000): Yang
(0200); Chuang et al. (2020): Liu et al. (2021b): Kalantidis et al. (2000 Newell & Deng 2020 Cai (020
A number of papers have observed that the default SSL approaches above (whether contrastive not)
perform poorly uncurated datasets such Openlmages Kuznetsova et al. (2020). To address this, recent
works have used different workarounds such as knowledge distillation Tian et al. (2021). clustering Goyal et al. (2021), localization Selvaraju el al. (2020), unsupervised semantic segmentation masks Hénalf et al. (2021). pixel-level pretext tasks Xie et al. (2021), Instance localization Yang et al. (2021) and local contrastive learning Liu et al. (2021a). The common element among top-performing imago-based SSL approachos, regardless of dataset, task or architecture, is their reliance on strong data augmentations such as random cropping, gaussian blurring, color jittering rotations. These augmentations create meaningful positive views, and
3

---

Published in Transactions on Machine Learning Research (12/2022)
other randomly sampled images in the dataset used to create negative views in the case of contrastive
SST. methods. SST, data augmentation pipelines are adapted from the supervised learning literature Cubuk
lt al. (2018): Zoph et al. (2020): Krizhevsky et al. (2012); Simard et al. (2003); DeVries & Taylor (2017):
Cubuk et al. 2018|: Zhang et al. (0018): Cubuk et al. [2019); Wu et al. (2019); Yun et al. (2019); Lim et al.
(2019): Halaya el al. (2019). :. Chuang et al. (2022): Peng et al. (2022) also deal with issues in using random croping, but they don't 't report results by pre-training on multi-object datasets like COCO and OpenImages,
which is our main focus in this paper. Secondly, their improvement on Detection Segmentation are only +0.3map while our improvement is +1.5 map by just changing the cropping strategy. Also, it's not very clear how to extend Chuang et al. (0022) Peng et al. (2022), since they rely upon bootstrapped models to generate better positives. And as we saw on OpenImages, bootstrapped models which use random cropping don't ' really perform well.
The closest work to our object cropping work is Selvaraju et al. (020); which introduces a technique to choose crops around objects basod on saliency maps Selvaraju et al. (2016), showing good improvements over the baseline of random crops for COCO datasets (see Table 5) As shown in our results, Obj-Obj+DDilate crop consistently performs better than Selvaraju et al. (2020) (Table 5). Our approach is also significantly simpler to incorporate into existing pipelines, requiring no change to the training, architecture or loss. [aansook]] ct al. ) show that constrained multi-cropping improves performance of SSL methods: our approach can be incorporated into their pipeline to further improve performance.
3 Analysis of Self- Sepervupe Learning methods on the Openlmages Dataset
In this section, we first identify some limitations of state-of-the-art SSL methods such as MoCo-v2 He et al, (2019); Chen et al. (2020b), SwAV Caron et al. (000b) and BYOL Grill et al. (2020) when pretraining on the Openlmages dataset. These methods have nearly closed the performance gap with supervised learning methods when pre-traine pre-trained and linear probed on ImageNet Deng et al. (2009). However, the performance of these methods on OpenImages dataset (where images contain multiple small objects) has not been extensively studied. OpenImages Kuznetsova et al. (2020) encompasscs images of complex scenes and several objects (containing, on average, 8 annotated objects per image). It consists of total of 9.1 million images. To perform controlled experiments on the effectiveness of cropping SSL method performance, we construct subset of the OpenImages dataset called OHMS Openimages Hard Multi-object Subset. We construct the dataset as follows: We sample images that have labelled bounding boxes to enable comparisons with fully supervised learning. Secondly, we sample images with objects from at least 2 distinct classes to create a dataset that better reflects real-world uncurated data. Finally, we only consider class categories with at least 900 images to mitigate effects of imbalanced class distribution. After this processing, we have 212, 753 images present across 208 classes and approximately 12 objects per image on average.
We provide further details in the Appendix SecA.
3.1 Performance of SSL methods
We pretrain several SSL methods MoCo-v2, CMC Tian et al. (2019), SwAV Caron et al. (2020b) and BYOL Grill al.] (2020) on OHMS dataset. MoCo-v2, BYOL, and other recent state of the art SSL approaches all relying on scene-scene cropping of the same image to generate positive samples.
Table 2 shows our results. We see significant difference in performance between fully supervised training and SSL approaches on the OHMS dataset, with a gap of 16.3 mAP points on average across the 4 SSL approaches considered. On ImageNet, the top-1 accuracy gap is considerably smaller with an average gap of only 8.5, nearly half that of OHMS. The last row shows the significant boost obtained by using our object-aware cropping approach, which we describe in the next section, with MOCO-v2. The gap between SSL and supervised training on OHMS is now the same as ImageNet.
4

---

Published in Transactions on Machine Learning Research (12/2022)
Model
OHMS (mAP)
ImageNet. (Top-1 %)
Supervised Performance
66.3
76.2
CMC
Tian et al.
2019
48.7 (-17.6)
60.0 (-16.2)
BYOL Grill et al. 2020
50.2 (-16.1)
70.7 (-5.5)
SwAV Caron et al.
2020b
51.3 (-15.0)
72.7 (-3.5)
MoCo-v2 (Scene-Scene crop)
49.8 (-16.5)
67.5 (-8.7)
MoCo-v2 DDijeet-Oljeet+Dilaaee crop) (Ours)
58.6 (-7.7)
68.0 (-8.2)
Table 2: Classification results on OHMS and Imagenet. For each SSL method, we show in parentheses the gap to fully supervised training (same number of epochs). The last row shows that our proposed approach using obj-obj+ cropping reduces the gap on OHMS by nearly half compared to the basclines, improving over the scene-scene cropping based SSL methods by between 8.8 mAP points. We also observe improvements on ImageNet as well.
3.2 Analysis and Motivation
We conduct further experiments to better analyze the results seen in Table 2 and to motivate our proposed approach. Our experiments help to narrow down scene cropping as one main cause the poor performance of SSL on OHMS, rather than other differences with ImageNet, such as object size, class distributions or image resolution.
Object Size: We compare MoCo-v2 performance to that of fully supervised learning, with both methods using scene-based cropping. Fig. 2 (left) shows that the performance gap between supervised learning and SSL methods does not vary significantly for objects of different sizes in OHMS. This suggests that once object sizes are below a threshold where scene cropping tends to ignore object information, MoCo-v2 performance is mostly independent of object scale.
Long-tail Distríbution: Even after selecting least 900 images per class, our OHMS subset has significant variation in the number of images per class (from around 1000 to 60000). Fig. 2 (right) plots the performance function of the number of instances in cach class. We do not see significant change in ivlative performance the number of instances in a class changes. This rules out long tails of the distribution as cause for the poor absolute performance of MoCo-v2 on OHMS.
of MoCo-v2 and supervised training
Can ImageNet pre-training help? We pre-trained a supervised model on ImageNet and then fine-tuned the final fully-connected layer on the OHMS dataset. We can see from the second column in Table 3 that this pre-training does not help to close the performance gap. One of the reasons that ImageNet pre-training does not help is that OHMS and ImageNet have significantly different class distributions (c.g. see Li et al. (019 for detailed analysis).
Can resizing the images help? We also experimented with resizing the images in OHMS to the same approximate size (384 x 384) and aspect ratio as ImageNet. The result is shown in the third column of Table confirming that controlling for image size does not help to close the gap.
Can cropping on ground truth objects help? We see from column of Table 3 that using random cropping on ground truth object boxes does not help reduce the performance gap either. As show later in Sectior that learning from both object and context is important for learning semantic information on multi-object datasots.
Varying the lower scale of random resized crop: MoCo-v2 (Chen et al. (020b)) used scene crops whose size was chosen from a uniform distribution ranging from 20% to 100% of the ImageNet image size (384 x 384). Since OHMS images are bigger and objects generally occupy a smaller fraction compared ImageNet, we vary the lower bound for scene crops to measure the impact. The last six columns of Table 3 shows that varying the range of scene crop is insufficient to close the performance gap.
5

---

Published in Transactions on Machine Learning Research (12/2022)
Log of of ap for the class
Log of sum of ap for the class
McCo-v2 Supervised
MoCo-v2
-2
Supervised
Log of mean of #pixels for each class
Log of number of instances for each class
Figure 2: Analysis of OHMS data distribution. Left: Performance of supervised and MoCo-v2 pre-training function of the scale of the objects; plot the log of the average of pixels against the sum of AP for each class. We see no discernible pattern of performance of MoCo-v2 or supervised lcarning as a function of object scale. Right: Performance of supervised learning and MoCo-v2 as function of the number of instances in a class; we plot the log number of instances in a class against the sum of AP for that class. We do not see any discernible pattern performance difference as a function of class size.
IN
Resize to
Scene-Scene Crop
Supervised
Pretraining
(384 384) GT-Cru CT-Crrp
0.8-1.0
0.6-1.0
0.4-1.0
0.2-1.0
0.1-1.0
0.05-1.0
66.3
28.3
45.9
45.3
26.5
37.6
45.6
49.8
46.1
43.1
Table 3: Linear evaluation on our OHMS dataset with different pre-training strategies with MoCo-v2 (see Section 3 for details). Column uses fully supervised learning on OHMS. We see that for self-supervised pre-training, specific range of scene crops helps to close the large gap between SSL and supervised training. However, there is a sweet spot of scene crop range where MoCo-v2 performance is highest.
We conclude that the performance gap between supervised training and SSL training is likely due to the data. augmentation, rather than characteristics of the image distribution. Further analysis experiments provided in the appendix (Sec D).
4 Proposed Approach
In this section first discuss how add object-awareness to the loss, followed by two new loss functions which help in learning better object aware representations.
4.1 Object Proposals
To enable object-swareness, consider unsupervised object proposal model LOD Vo et al. 2021 LOD
is a large scale unsupervised object proposal method Vo et al. (2021). The authors suggest formulation of unsupervised object discovery as a ranking problem using distributed methods. In our experiments we use LOD to generate up to 10 proposals per image and select one object randomly among these proposals. The details of other faster semi supervised object proposals (which use object labels from VOC datasct for training) are present in appendix Section C. Since, our method is completely Self-Supervised we focus on using T.OD object proposal generation method.

---

Published in Transactions on Machine Learning Research (12/2022)
Dilated object proposals (Obj-Obj+Dilate Crop): Scene pixels spatially close to the object are more likely to have a positive correlation with the object. With this intuition, we generate the second view by dilating the the randomly selected LOD proposal. We dilate the box by 10% or 20% of the image size, followed by a random crop. Changing dilation gives us control over how much scene information is incorporated (a value of 10% works well in most cases). Note that the original and dilated boxes are both followed by a random crop, ensuring that the first view is not trivially included the second view. The choice of which crop to use as query or key is arbitrary and either object or dilated object crops can be used as key and query.
Different projection heads for Object and Dilated Object crops: The projection head, introduced in CCen et al. (2020a) is an important component of most SSL methods. This is an MLP that maps representations from the encoder backbone to a lower-dimensional space where the loss function is applied. SimCLR Chen et al. (2020a) show that projection heads can remove information that may be useful for the downstream task, such as the color or orientation of objects. They show that by using this MLP, the last layer of network (i.e layer before mlp) can maintain more information. However Chen & He (2020) use single projection head is used for both views, since both the views on ImageNet are object crops. We hypothesize that Object and Scene crops often contain different object orientation and color information; hence we propose to use different projection heads for scene and objects.
4.2 Loss Objectives
Following
He et al. 2019), we use momentum queue and optimize the model using the InfoNCE loss
explg-k*/ /T)
Lmnen - log
(1)
exply.k+/o) /T) + Σ explyyk-//T)
Here q is a query representation, 1+ a representation of the positive (similar) key sample, and {k-} representations of the negative (dissimilar) key samples. T is the temperature hyper-parameter. We augment the standard contrastive loss with two additional losses to learn richer features for both objects and scenes. Next we describe both these losses in detail.
} are
4.2.1 Rotation task Crot
In the standard rotation prediction pretext task Gidaris et al. (2018), an image crop is randomly rotated with angle in set p. An MLP is then tasked at correctly predicting the rotation of a patch given its features extracted using the feature extractor f. Our object cropping strategy generates an object and scene crop from a given image. We modify the standard rotation task to estimate rotation of the object crop wrt the scene crop. Specifically, we randomly rotate the object crop and extract features Zobj- The scene crop is kept as is and its features Zscene are obtained by feeding the scene crop through feature extractor f. Note that rotation augmentations are applied to object crop in addition to the standard MoCo-style augmentations. The rotation prediction MLP takes as input the concatenated features and estimates the relative rotation. Our rotation loss is Lrot is a standard cross entropy loss with rolation labels as the largets. Different from the standard rotation prediction task which estimates absolute rotation, our approach estimates rotation of the object relative to the scene and complements the contrastive loss. Note that since we deal with in-the-wild datasets, estimating absolute rotation of a crop is ill-posed since a particular object might occur in a variety of poses thus leading to incorrect gradients.
4.2.2 Object localization task Lioc
Since we are working with images coming from non-iconic datascts, the object could potentially occupy a small region of the image. We propose to add a pretexl task of localizing an object inside the scene using features alone. Specifically in this task, given an image, we predict the spatial location of the object in the image. Here we take original image and divide the image into p + p patches. We then take an object crop and mark all the patches where the object is present as 1 and other patches as 0 which we use as our label Y. Similar to rotation task we first extract features for the object proposals Zabj- For the original image we
7

---

Published in Transactions on Machine Learning Research (12/2022)
Model
Crops
Obj-Obj+Diilae
Scene-Scene
mAP
Supervised
66.3
MoCo-v2
49.8
BYOL
50.2
MoCo-v2
Ground Truth boxes Ground Truth boxes
58.9
MoCo-v2
60.2
MoCo-v2
LOD crops
59.1
MoCo-v2 + Rotation + Localization
LOD crops
red59.7
BYOL
LOD crops
59.5
Table 4: Crop approaches on OHMS: using LOD crops to generate one view, and a dilated crop for the other positive, we are able to reduce the difference between SSL and Supervised Learning by close to 50% (compare the last two rows to the second row). Using ground-trun woound-ruuhh boxes to generate crops from OHMS improves the pre-training performance marginally compared LOD crops. Our Obj-Obj+ Dilate outperforms the Scene-Scene baseline by significant margin on the OHMS dataset.
apply all the same augmentations except random crop (since we want the full image and not the cropped version of the image). Then we obtain the features for the original image Zori- We then pass the features through MLP layer. Finally the loss fot is two-class classification loss on concatenation of object features, original image features and labels Y.
Why don't we see degenerate solution for object localization? A natural question is why doesn't our model cheat and remember all the object location for all the object proposals. The answer to the question is the random cropping on the object proposals. Since label Y will change every time according the random crop paramcters. Hence the network cannot learn the object patch location and has to focus on the semantics of objects and scenes to figure out the spatial location of object in the scene. Exact implementation detail of the object localization task is shown in the appendix.
using random crops the object proposals, the
Final loss function: Our final loss function
Loac Lmoco + Crot + Lol
(2)
We simply combine all the losses and call it Loae (Object-Aware Cropping) and backpropagate on all of them.
5 Results
We created subset of the OpenImages dataset with 212k images, as described in Section 3 We also experiment with the complete OpenImages (~1.9 million images.) In addition, perform pre-training TmageNet, Deng et al. (2009) and MS-COCO Lin et al. (2014). ImageNet (with 1.2M training images) has been extensively used and is the standard dataset used for benchmarking of SSL methods. MS-COCO has ~ 118k training images and 896k labelled objects which is approximately 7 objects per image. For pre- traininnigg on MoCo-v2, we closely follow the standard protocol described in Chen et al. (2020bb. We randomly select. from 10 object proposals provided by LOD. All our training and evaluation is performed on ResNet-500 He et al. (2015). For our baseline use standard scene-scene crop, where we take two random crops in image and treat them positive views. This is the default approach used in MoCo-v2 and other SSL approaches. From our analysis in Section this approach performs poorly on datascts such as OHMS.
As discussed, the Obj-Obj+Dilale uses a random crop on the object proposal or dilated version, to generate the final views. Since the object proposal itself is a small fraction of the image (e.g. COCO, an object crop typically covers about 39% of the image), using the usual default lower value for the random crop range (usually 0.2) works poorly as results in extremely small crops from the image. Therefore, we set the lower limit such
that it matches the minimum sized crop in case of the usual scene crop (Smin average Object proposal size
0.2
8

---

Published in Transactions on Machine Learning Research (12/2022)
Description
AP
A AP30
AP75
AP,
AP
APm
Supervised (Random Initialization)
32.8
50.9
35.3
29.9
47.9
32.0
Supervised (InageNet Pre-trained)
39.7
59.5
43.3
35.9
56.6
38.6
MoCo-v2 Chen et al.
(2020b)
38.2
58.9
41.6
34.8
55.3
37.8
BYOL Henaff al. 2021
38.8
58.5
42.2
35.0
55.9
38.1
Dense- CL Wang et al.
(2021
39.6
59.3
43.3
35.7
56.5
38.4
CAST Selvaraju et al.
2020
(180K steps) (Uses BYOL)
39.4
60.0
42.8
35.8
57.1
37.6
Self-EMD Liu et al.
[2021a.
39.8
60.0
43.4
MoCo-v2 + OAC (Ours)
39.7
60.1
43.4
36.0
57.3
38.8
Dense- Canse-CL-i + OAC (Ours)
40.4
60.4
44.0
36.6
57.9
39.5
MoCo-v2 + OAC (Using all losses)(Ouss)
40.7
60.9
43.9
36.9
58.3
39.6
BYOL + OAC (Using all (osses)(Ours)
41.1
61.4
44.2
37.1
59.2
40.1
Dense-CI CL + OAC (Using all (osses)(Oure)
41.4
61.5
44.7
37.5
59.5
40.4
Table 5: Object detection (first 3 columns) and Semantic Segmentation (last columns) results on COCO dataset. All SSL models have been pre-trained on COCO dataset and then finetuned on COCO. All other methods run for 90K, finetuning iterations. For any SSL method, we compare (BYOL, Moco-v2, Dense- CL) adding Object-A Aeeett-waree wwre-Crropiin (OAC) cropping losses improves performance. We see further improvement by adding the proposed rotation and localization losses(last 3 rows).
Model
AP
APso
AP15
Supervised (Imaag-Net-preeraininn])
56.8
83.2
63.7
MoCo-v2 Scene-Scene crop (Chen et al. 2020b
51.5
79.4
56.4
MoCo-v2 Dijatejiiilatte crop (6 0.1) (Ours)
53.4
80.1
59.1
MoCo-v2 Obj-Obj+ crop + Rotation (8 = 0.1) (Ours)
53.8
79.6
60.1
MoCo-v2 ObjlObj+Diiltte crop Object Localization (6 0.1) (Ours)
54.1
80.6
60.2
MoCo-v2 Combined (Ó = 0.1) OAC (Ours)
54.6
81.0
60.6
Table 6: Object detection results on VOC dataset (OHMS pre-training, using LOD proposals). All models
have been pre-trained on OHMS and then fine-Luned on VOC. We can see that all of our proposed components improve upon the baseline scene scene crop and combining all of them improves upon the baseline by +3.1 mAP.
We evaluate the pre-trained models on classification (linear evaluation), object detection and semantic segmentation. For VOC object detection, COCO object detection and COCO semantic segmentation, we closely follow the common protocols listed in Detectron2 Wu et al. (2019). For VOC object detection, we evaluate the Fostee
Ren et al. (2015) detector on VOC trainval07+12 dataset. using the standard 1 x standard protocol. For COCO-OP O0-00----jeett- detection and semantic segmentation, we fine tune on the MaskRCNN detector (FPN-backbone) He et al. (2018) on COCO train2017 split (118k images) with the standard
schedule, evaluating on the COCO 5k va12017 split. We compare to the state of the art SSL methods, including Self-EMD Liu et al. (2021a). DetCon Hénaff ct al. (2021) BYOL Richemond et al. 2020), DenseCL Wang et al. (2021) and the default MoCo-v2 Chen al. (2020b).
Table and Table 4 shows results on OHMS dataset. We can that Obj-Obj-Dilate crops outperform the bascling by 8.2 mAP, closing the gap between supervised learning and MoCo-v2 baseline by almost 50%. In Obj-Obj+Dilate crops, a dilated object crop would potentially contain the entire object and more scene information; therefore, the representation from the dilated object-crop contains complementary information from both the object and the scene. We also show in Table g an ablation with ground truth bounding boxes being used guide the object cropping. This performs marginally better than the use of LOD crop, suggesting that tight fit. around the object is not necessary for improved representations.

---

Published in Transactions on Machine Learning Research (12/2022)
Description
AP
A0
A AP75
APs
AP;
A APm
COCO: MoCo-v2 (Scene-Soene crop)
38.2
58.9
41.6
34.8
55.3
37.8
COCO: MoCo-v2 + OAC (Ours)
41.3
61.8
44.7
37.3
58.5
40.1
VOC: MoCo-v2 IScene-Scene crop)
56.1
81.3
61.3
VOC: MoCo-v2 + OAC (Ours)
58.8
83.6
64.9
Table Object detection (first 3 columns) and semantic segmentation (last 3 columns) results on COCO (first rows) and VOC (last rows). All SSL models have been pre-trained on complete OpenImages datasct million images) for 75 epochs and then finetuned COCO and VOC dataset.
Model
Dataset
Crops
Obj-Obj+Dillas
Scene-Scene
mAP
Supervised
Full Openlmages
74.0
MoCo-v2
Full OpenImages
50.5
MoCo-v2 + OAC
Full OpenImages
62.1
Supervised
Random Subset OpenImages
74.0
MoCo-v2
Random Subset Openlmages
50.5
MoCo-v2 + OAC
Random Subset Openlmages
62.1
Table 8: Results on full OpenImages and random subset of OpenImages. On full Openlmages dataset the difference between supervised learning and MoCo-v2 is still large, showing that random cropping is issue not just OHMS but also on the full dataset.
Table 5) shows results on object detection and semantic sogmentation for COCO (by pre-training on COCO trainval2017 datasets and finetuning on COCO). We train MoCo-v2, BYOL and Dense-CI, models. Our MoCo-v2 Obj-Obj+ Dilate cropping outperforms MoCo-v2 Scene-Scene baseline. Our proposed cropping is agnostic to the pre-training SSL method; we show results by adding our approach to Dense-CL Wang et al. (2021). We also outperform the CAST model [Selvaraju al. (2020) which also uses localized crops based on saliency maps: our approach is simpler and performs better by around 1.4 mAP. Table (appendix) shows results of object detection PASCAL-VOC. We pro-train cropping oulperforms the MoCo-v2 baseline by 3.2 mAP and the BYOL baseline by 2.5 mAP. Improved results on iconic datasets like Aircraft, Birds and Cars can be found in appendix (Table 4). Additional results on varying number of proposals used can be found in appondix (Table 6).
COCO and then fine-tune VOC. OAC
In addition to small OHMS dataset we also show results on full OpenImages dataset. Table 7 shows results on object detection and semantic segmentation for COCO and object-detection on VOC by pre-training full OpenImages dataset Kuznetsova et al. (2020) (all 1.9 million images) for 75 epochs. We show improved performance over the baseline both object detection and semantic segmentation tasks by using Obj-Obj+ Dilate crops. Our proposed dilation method works not only for small multi-object datasets like COCO but also for datasets like OpenImages and performs well under a transfer learning setup.
We also show results on ADE20k following Chen et al. (017)) The baseline MoCo-v2(COCO pre-training) gets 37.5 mIoU, while we get 39.2 mloU. Similar to VOC and COCO, we see consistent performance improvement on ADE20k well.
Results on ADE20K:
Results Full OpenImages: We also report classification results full OpenImages dataset i.e 1.7 million images and on 212k random subset in Table X For the full datasct pre-training has been done for 100 epochs and for the random subset the pre-training has been done for 200 epochs. We can see that on the full dataset, the gap between supervised learning baseline and MoCo-v2 is still large, while the gap on random subset is not as big.
Results on ImageNet: We also show improved results on ImageNet pre-training using object-aware cropping (Table appendix) and MoCo-v2. For object detection on VOC2007, we see improvement of 1.0
10

---

Published in Transactions on Machine Learning Research (12/2022)
mAP; and 0.5 mAP improvements for object detection COCO. Our approach is thus adaptable to the pre-training dataset and SST. algorithm.
Use of Multiple Projection Heads: classification gives us a boost of 1.1 mAP on Obj-Obj+Dilate crop. Pre-training on COCO and finetuning on VOC dataset for object-detection task gives boost of 0.4 mAP. Hence using multiple projection heads results in consistent improvement.
The use of different projection heads for each view on Openlmages
Varying Dilation Parameter: Table (appendix) shows the effect of varying the dilation parameter. A sweet spot exists at moderate dilation value of 8 = 0.1 for COCO object detection.
6 Conclusion
We have introduced object-aware cropping, simple, fast and highly effective data angmentation alternative to random scene cropping. We conducted numerous experiments to show that object cropping significantly improves performance over scene cropping for self-supervised pre-training for classification, object detection and semantic segmentation on a number of datasets. The approach can be incorporated into most self- supervised learning pipelines in a seamless manner.
7 Acknowledgments
This work is supported[, in part.] by the US Defense Advanced Research Projects Agency (DARPA) Semantic Forensics (SemaFor) Program under [award / grant / contract number] HR001120C0124, Any opinions, findings, and conclusions recommendations expressed in this material are those of the author and do not necessarily reflect the views of the DARPA. This research is also supported by the National Science Foundation under grant no. 1910191.
References
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning, 2019.
Tiffany Tianhui Cai, Jonathan Frankle, David J. Schwab, and Ari S. Morcos. Are all negatives created equal in contrastive instance discrimination?, 2020.
Mathilde Caron, Ishan Misra, J. Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. ArXiv, /ttp//2000008888 2020a.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual fcatures by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9912 9924. Curran Associates, Inc., 2020b. URL httpps//procceedingg. iet.reeeddiinnnnenii..r...ee cttccc/////c/...cccccccccccccc// cc/paaper//020//iiee pdff
Siang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. ArXiv, abs/ abs/1706.05587, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, 2020.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning, 2020b.
Ching- Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanic Jegelka. Debiased contrastive learning. In H. Tarochelle, M. Ranzato, R. Hadsell, M. F. Balcan,
11

---

Published in Transactions on Machine Learning Research (12/2022)
and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 8765- 8775. Curran Associates, Inc., 2020. URL http://pcce.iiaaa.r.oooooa.aappp Sccdac//222aaaeee22c116664/////3
newr
pdf.
Ching- Yao Chin Chuang, R. Devon Hjelm, Xin Wang, Vibhav Vineet, Neel Joshi, Antonio Torralba, Stefanic Jegelka, and Ya heng Song. Robust contrastive learning against noisy views. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16649-16660, 2022.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with a reduced search space, 2019.
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoe V. Le. Autoaugment: Learning angmentation policies from data. ArXiy, abs/1805.0956 2018.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Terrance DeVries and Graham W. Taylor. Dataset augmentation in feature space, 2017.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Revisiting contrastive methods for unsupervised learning of visual representations, 2021.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting 2018.
image rotations. ArXiv,
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthicu Cord. Learning representations by predicting bags of visual words, 2020.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski. Self-supervised pretraining of visual features the wild, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukeuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach self-supervised learning, 2020.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster autoangment: Learning augmentation strategies using backpropagation, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2015.
arXiv preprint
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn, 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Procoedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Olivier J. Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron van den Oord, Oriol Vinyals, and João Carrcira. Efficient visual protraining with contrastive detection, 2021.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning, 2020.
12

---

Published in Transactions on Machine Learning Research (12/2022)
Alex Krizhevsky, Iya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L.. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL attpp://pproceedings Papeee
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, al. The open images dataset v4. International Journal of Computer Vision, pp. 1-26, 2020.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as proxy task for visual understanding. 2017 IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pp. 840 849, 2017.
Hengduo Li, Bharat Singh, Mahyar Najibi, Zuxuan Wu, and Larry S. Davis. An analysis of pre-training on object detection, 2019.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsuper-
vised representations. In International Conference Learning Representations, 2021. URL https;
//openreview //openrerise..nt///orrmm// wtt.://oooorrr//dd/aa/sssss//e/
Sungbin Lim, Iddoo Kim, Tacsup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment, 2019.
Tsung- Yi Lin, M. Maire, Serge J. Belongie, James Hays, P. Perona, D. Ramanan, Piotr Dollár, and C, L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Songlao Liu, Zeming Li, and Jian Sun. Self-emd: Self-supervised object detection without imagenet, 2021a.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Zhaoyu Wang, Li Mian, Jing Zhang, and Jie Tang. Self-supervised learning: Generative contrastive, 2021b.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarien representations, 2019.
Alejandro Newell and Jia Deng. How useful is self-supervised protraining for visual tasks? 2020 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2020. doi:
2020.00737. URL http:///xx: doi org/10. 1199/CVPR42600. 2020. 00733
Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, and Hamed Pirsiavash. Boosting self-supervised learning via knowledge transfer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Deepak Pathak, Philipp Krähenbühl, Jelf Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. 2016 TEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2536 2544, 2016.
Xiang Peng, Kai Wang, Zheng Hua Zhu, and Yang You. Crafting better contrastive views for siamese repro- sentation learning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16010 10009,, 2022.
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases, 2020.
Shaoqing Ren, Kaiming Hc, Ross B. Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39: 1137-1149, 2015.
Pierre H Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallee, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, al. Byol works even without batch statisties. arXiv preprint axXiv:2010.10241, 2020.
13

---

Published in Transactions on Machine Learning Research (12/2022)
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618 626, 2016.
Rampraseath R. Selvaraju, Karan Desai, Justin Johnson, and Nikhil Naik. Casting your model: Learning localize improves self-supervised representations. arXiv preprint (xXi:2202.00633) 2020.
Anshul Shah, Suvrit Sra, Rama Chellappa, and Anoop Cherian. Max-margin contrastive learning, 2021.
Patrice Simard, David Steinkraus, and John Platt. Best practices for convolutional ncural networks applied to visual document analysis. pp. 958-962, 01 2003. doi: 10.110//ICDAR.200.... 1227800.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. MrXiv:1906.05849, 2019.
Contrastive multiview coding. arXiv preprint
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In H. Larocholle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systemss, volume 33, pp. 6827 6839, Curran Associates, Inc., 2020. URL http:://prcceddng...uuu... neurnps.....r......emm www.eeew....ppppeeeeeeee httprppps.cc/ppppp/////11.. 62225aee112/7793111477550000
.pdf
Yonglong Tian, Olivier J. Henaff, and Aaron van den Oord. Divide and contrast: Sclf-supervised learning from uncurated data, 2021.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy,
and linear models, 2021.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv, abs/1807.03748, 2018.
Huy V. Vo, Elena Sizikova, Cordelia Schmid, Patrick P'erez, and Jean Ponce. Largo-scale unsupervised object discovery. In NeurIPS, 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere, 2020.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training, 2021.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. 2019.
https
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature lcarning via non-parametric instance discrimination. In 2018 TEEE/CV /CVF Conference on Computer Vision and Pattern Recognition, pp. 3733 3742, 2018a. doi: D0.109/CVVR.200S.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non -a--aaametric instance discrimination. In Proccedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-33 3742, 2018b
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. ArXiv preprint Ttl::2006.0999// 2020.
Tete Xiao, Xiaolong Wang, Alexei A. Efros, and Trevor Darrell. What should not be contrastive in contrastive learning, 2021.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp-
2021.
14

---

Published in Transactions on Machine Learning Research (12/2022)
Ceyuan Yang, Zhirong Wu, Bolci Zhou, and Stephen Lin. Instance localization for self-supervised detection pretraining. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 3986 3995, 2021.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, 2018.
Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin D. Cubuk, and Quoe V. Le. Rethinking pre-training and self-training, 2020.
15


Reviews:
[{"review_id": "XXxBkKQG1O", "reviewer": "Reviewer_moVN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall the paper is well written, addresses an important problem in self-supervised representation learning (that is learning representations from less curated and more naturally occurring data distributions), and achieves strong results. The main strengths were further described in the section above (\"summary of contributions\"). \n\nIn my view, the main weakness of the paper is the following. The method uses 3 different self-supervised objectives. However, throughout the paper, the authors keep referring to only a single term (the contrastive between the object-aware crops) as the main contributor to the improved performance when training from complex images (eg on OpenImages). However, the authors do not report the results of MoCo/BYOL with the additional 2 loss terms. The only related result to this is in Table 6, which shows that adding the new crops, rotation, and localization (in this order) improves results on VOC, but there are no results showing the performance of a standard contrastive method with the additional rotation and localization pretext tasks (while foregoing the new crops). Thus, the claim that object-aware cropping is responsible for improved performance is still unsubstantiated. Even if the results of the above ablation show that the new crops lead to bigger improvements than rotation and localization by themselves, clearly, the other terms also play an important role in achieving state-of-the-art performance. So, I would still argue that the overall emphasis given to these components is lacking.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a self-supervised representation learning method to learn from complex (multi-object) datasets, like OpenImages and COCO, which are more representative of real-world uncurated data. The proposed method optimizes three terms simultaneously: 1) a contrastive term, 2) a rotation prediction term, and a 3) localization term. Regarding the first term, the main technical novelty is the new image sampling procedure which, on one hand, focuses on objects in the scene (by leveraging unsupervised object proposals), and on the other hand, obtains the two views by using crops that are spatially close. The third loss term (localization) also seems to be novel, and seeks to train a model where object-centric crops (derived from the unsupervised proposals) can be localized in the full image. The proposed procedure resulted in significant improvements when compared to traditional methods like MoCo which learn from random crops. The paper shows convincing improvements on downstream tasks like detection and segmentations on COCO and VOC.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please, see the changes requested in the weaknesses section. Namely, the ablation study and the emphasis on the rotation and localization terms.", "broader_impact_concerns": "I have no ethical concerns regarding this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MlNEObjcEB", "reviewer": "Reviewer_ANCp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The overall improvement look good as Table 5 shows. It is also interesting to see BYOL fails to learn from noisy datasets. This is actually a very interesting observation that highlights the importance of negative samples. This direct is worth further explored in the future.\n\n2. It is good to see dataset such as OHMS that mimics the real world distribution. The proposed dataset could potentially lead to several new contrastive learning methods.\n\nWeaknesses:\n\n1. The main motivation is that noisy cropping could degenerate the solutions of contrastive learning. It is unclear why one need rotation task and object localization task, as they seem to be orthogonal directions compared to contrastive learning. The only ablation study lie in Table 6, which adopts VOC dataset as the downstream task. It would be great to see how the proposed cropping approach performs without the additional two tasks on the COCO dataset.\n\n2. The problem seems to be explored by several previous works that are not discussed here. For instance, [1] also proposes an object localization strategy to perform better cropping for contrastive learning. In [2], the same problem is addressed, where the authors propose a robust loss function to overcome the noisy cropping problem. Works with similar ideas can also be found in the other conferences. The existence of these previous works decrease the novelty of the proposed idea. In particular, the only baseline considered here is the Scene-Scene crop. More empirical comparisons to the previous works should be included to consolidate the experiments.\n\n3. It is a bit unclear why we need two projection heads here. This makes it hard to tell how the proposed cropping method improves over the baselines. For instance, one can also augment MoCo-v2 with the rotation task and object localization loss to (potentially) improve the performance. Ideally, there should be more results that exclude all the additional losses and modifications, but simply changes vanilla cropping method to LOD.\n\n4. In additional to OHMS, it would be great to see experiments with the standard evaluation setting, e.g., with pretraining on the ImageNet. It is tricky to evaluate the proposed approach on a self-constructed dataset, as there could be tunable factors during the process of dataset construction that makes the experiments\n\n5. It is a bit odd to see the related works at the end of the paper. Similar to the second point, there are several related works that need to be discussed and compared. It makes sense to present an overview before the method section to give readers a glimpse about how similar problems are addressed.\n\n6. In the appendix, what is the role of new p’(x) in (5), as p’ is not showed in the equation. If one use p’ to replace a part of p in (5), it is no longer a variational lower bound of the MI between X and C. Does this replacement create a new random variable? The theoretical analysis does not make the motivation clear.\n\nOverall, the paper addresses an important problem in contrastive learning. The contribution could be potentially very useful to the field. Nevertheless, the ablation studies and comparison to previous works can be improved.\n\n[1] Peng et al., Crafting Better Contrastive Views for Siamese Representation Learning, CVPR 2022\n\n[2] Chuang et al., Robust Contrastive Learning against Noisy Views, CVPR 2022\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a robust way to perform data augmentation, specifically, cropping, in contrastive learning. Along with the rotation loss and localization loss, the proposed loss function (OAC) achieves several empirical improvement across different benchmarks. Overall, the paper is well-written and the contribution is stated clearly.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Overall, the paper is in a good shape. Having the following results would significantly strengthen the contribution.\n\n\n1. Empirical experiments as stated in weakness sections. For instance, (1) Mocov2 + rotation + object localization (without LOD) (2) Empirical comparison with baselines such as [1] or [2] (3) Results on standard benchmarks such as ImageNet. By consolidating the experiments, I believe the contribution of the paper would be more convincing.\n\n2. More discussion w.r.t. related works. As there are several similar works are addressing the same problem, it would be great to provide a paragraph to briefly go over the previous works.\n\n\n", "broader_impact_concerns": "The paper addresses a fundamental problem in self-supervised problem. I am not aware of any ethical issue in this work.\n\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BNiwg1kNx3", "reviewer": "Reviewer_Rh3t", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\n\nS1) the paper is very well written and structured and easy to read and understand\n\nS2) OHMS is in general a very nice testbed for understanding performance of SSL algorithms in multii-object scenarios, and the analysis in Sec 2 clear and interesting.\n\nS3) The method seems to give gains for different SSL methods, including Dense-CL which is a method targetting detection.\n\n__Weaknesses__\n\nW1) Results are limited to COCO and VOC. ADE20K would be a nice third dataset to test to.\n\nW2) Although OHMS is in general a very nice testbed for understanding performance of SSL algorithms in multii-object scenarios, it is also kind of a \"wish list\" for the proposed approach, ie it contains only images that should be better represented with the proposed approach. Therefore Table 2 / Table 6 results should be taken with a pinch of salt. It would be really interesting to also compare performance of the proposed approach and other SSL methods when trained on a _random_ 200k-sized subset from all of Openimages in Table 2. \n\nW3) Added computational cost of running LOD. How high is this cost? are the exact same representations used for this step? I assume this is a step that is run offline; the authors can report how much time it takes for each of the datasets they use (ImNet, COCO, VOC, OHMS, OpenImages)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript presents an approach for contrastive self-supervised learning (SSL) targetting object detection tasks. \n\nThe main contribution is a novel way of selecting the positive views based on the LOD object discovery algorithm of Vo et al. by first selecting one of those proposals as the first positive view, and then creating a dilated version of that as  the second view. They complement this with two SSL losses, a secondary localization-oriented loss that leverages LOD scores and a rotation prediction loss. \n\nThey study how a number of common SSL algorithms perform when trained on a dataset that selected to be more complex than imagenet, and introduce OHMS, a subset of OpenImages that contains images with objects from at least two classes. It is a 200k image dataset with 208 classes and 12 objects per image. The \n\n\nThe authors show that when pretraining on OHMS with their method, they can get better results for downstream object detection datasets.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "R1) See weaknesses above: Results on ADE20K, another comparison in Tab2 and discussion on the computational cost of LOD.\n\nR2) Table 1 is currently misleading should be split based on pretraining dataset. It is unfair to compare methods pretrained on Imagenet with ones trained on OHMS. At least one row for MoCo-v2 without OAC trained on the same OHMS dataset is needed as a fair baseline. This exists in other tables for other metrics, but should also be here for mAP on VOC to paint the whole picture\n\nR3) The claims that Imagenet has a \"a large, centered object\" versus OpenImages being a dataset where \"the objects of interest are small relative to the image size and rarely centered\" are generally exargerated. There has been a number of works that show that ImageNet is not as iconic as generally though [Relabeling-Imagenet] while Figure 5 in [LVIS] shows that objects of interest in (the full) OpenImages dataset are indeed mostly centered. Of course the authors are right and OpenImages is more diverse than ImageNet, however unlike the section title,the authors do not use OpenImages for all the analysis, but they define a subset that is constructed to make the proposed algorithm shine. It would be great to tone down such claims and be explicit when the conclusions come from OHMS vs full OpenImages.\nThe two papers mentioned above are missing related works that should be cited and discussed at least in related work. \n\nR4) It would be nice to report more results for the \" Obj-Obj+Dilate crop\" case without the added losses. Eg two more rows for Table 7, to see clearly what gain comes from that versus the added loss terms. I understand that the two are related to the LOD algorithm, but still it would be great if one can get gains by simply changing the cropping strategy and nothing else.\n\n__Some clarifying questions__\n\nQ1) What is the overlap between the classes of VOC and COCO and the 208 classes the OHMS dataset? How about Imagenet and COCO/VOC? \n\nQ2) Sorry if I missed it but is there a train/val split for OHMS or are you performing evaluations and object detection on exactly the training images?\n\nQ3) is the last row of Table 2 using the added losses (rotation and localization tasks)? If yes, what are the equivalent results without these added losses?\n\nQ4) Can you clarify what you mean by \"different projection heads for scene and objects\" (end of Sec 3.1)  for MoCo-v2 which is generally the base SSL algorithm for the proposed approach? isnt the scene MLP (which i assume is the teacher) side also an EMA of the student side?\n\n__References__\n\n[LVIS] Gupta, A., Dollar, P. and Girshick, R., 2019. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5356-5364).\n\n[Relabeling-Imagenet] Yun, Sangdoo, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. \"Re-labeling imagenet: from single to multi-labels, from global to localized labels.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2340-2350. 2021.\n", "broader_impact_concerns": "Added biases for this approach could be inherited via the proposals for the LOD algorithm.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers recommend acceptance and I have no objection. The idea is clearly described and experiments are convincing. There were some initial concerns by the reviewers, and the revision addressed them adequately. I looked carefully through the reviews and the authors' responses, and have no remaining concern. Recommend accept as is.
Decision: Accept (poster)