Consistent Instance Classification for Unsupervised Representation Learning
Authors: Depu Meng
Zigang Geng
Zhirong Wu
Bin Xiao
Houqiang Li
Jingdong Wang
Venue:  (2021)

Abstract: In this paper, we address the problem of learning the representations from images without human annotations. We study the instance classification solution, which regards each instance as a category, and improve the optimization and feature quality. The proposed consistent instance classification (ConIC) approach simultaneously optimizes the classification loss and an additional consistency loss explicitly penalizing the feature dissimilarity between the augmented views from the same instance. The benefit of optimizing the consistency loss is that the learned features for augmented views from the same instance are more compact and accordingly the classification loss optimization becomes easier, thus boosting the quality of the learned representations. This differs from InstDisc and MoCo that use an estimated prototype as the classifier weight to ease the optimization. Different from SimCLR that directly compares different instances, our approach does not require large batch size. Experimental results demonstrate competitive performance for linear evaluation and better performance than InstDisc, MoCo and SimCLR at downstream tasks, such as detection and segmentation, as well as competitive or superior performance compared to other methods with stronger training setting.

Under review as a conference paper at ICLR 2021
CONSISTENT INSTANCE CLASSIFICATION FOR UNSU- PERVISED REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we address the problem of learning the representations from images without human annotations. We study the instance classification solution, which regards each instance as a category, and improve the optimization and feature quality. The proposed consistent instance classification (ConIC) approach simul- tancously optimizes the classification loss and an additional consistency loss ex- plicitly penalizing the feature dissimilarity between the augmented views from the same instance. The benefit of optimizing the consistency loss is that the learned features for augmented views from the same instance are more compact and ac- cordingly the classification loss optimization becomes easier, thus boosting the quality of the learned representations. This differs from InstDisc (Wu et al. 2018) and MoCo (He et al. 2019 Chen al.) 2020c) that use estimated prototype as the classifier weight to ease the optimization. Different from SimCLR (Chen et. 2020b) that directly compares different instances, our approach does re- quire large batch size. Experimental results demonstrate competitive performance for linear evaluation and better performance than InstDisc, MoCo and SimCLR downstream tasks, such as detection and segmentation, as well as competitive or superior performance compared other methods with stronger training setting.
INTRODUCTION
Learning good representations from unlabeled images is a land-standing and challenging problem. The mainstream methods include: generative modeling (Hinton al.) 2006; Kingma & Welling
2014). colorization (Zhang et al.
2016), transformation or spatial relation prediction (Doersch et al.,
2015
Noroozi & Favaro 2016 Gidans et al., 2018). and discriminative methods, such instance
classification (Dosovitskiy et al.
2016 He et 2211)). and contrastive learning (Chen et al. 2020b).
The instance discrimination methods show promising performance for downstream tasks. There are two basic objectives that are optimized (He et al.) 2019 Chen et al.] 2020b; Yu et al. 2020; Wang & Isola 2020); contraction and separation. Contraction means that the features of the augmented views from the same instance should be close possible. Separation means that the features of the augmented views from one instance should lie in a region different from other instances.
The instance classification framework. such InstDisc (Wu et al. 2018), and MoCo (He et al. 2019 Chen et al. 2020c). adopts a prototype-based classifier, where the prototype is estimated as the moving average of the corresponding features of previous epoches Ww et al. (2018) as the output of an moving-average network He et al.] (2019); Chen et al. (2020c). The prototype-based schemes ease the optimization of the classification loss in the challenging case that there is over one million categories. BYOL (Grill et al. (2020) computes the prototype in way similar to MoCo, and only aligns the feature of augmented views with its prototype leaving the separation objective implicitly optimized. The prototype, computed from a single view rather than many views and from networks with different parameters, might not be reliable enough, making the contraction and separation optimization quality not guaranteed.
The contrastive learning framework []] such as SimCLR (Chen et al. 2020b) and Ye al. 2019 simultaneously maximizes the similarities between each view pair from the same instance and min-
InstDise (w et al. 2018 and MoCo (He al. 2019 Chen al. 2020c) are also closely related to contrastive learning and regarded as contrastive learning methods by some researchers.

---

Under review as a conference paper at ICLR 2021
(a)
(b)
(c)
Figure 1: Visualizing the activation maps. (a) input image, (b) activation maps from our approach,
(c) activation maps from only optimizing the classification loss. One can see that our approach (b) tends more focus on the textured region.
imizes the similarities between the view pair from different instances. This framework directly compares the feature of one view to different view other than to prototype, avoiding the unreli- ability of the prototype estimation. It, however, requires large batch size for each SGD iteration to compare enough number of negative instances for imposing the separation constrain increasing the difficulty in large batch training.
We propose simple unsupervised representation learning approach, consistent instance classifica- tion (ConIC), to improve the optimization and feature quality. Our approach jointly minimizes two losses: instance classification loss and consistency loss. The instance classification loss is formu- lated by regarding each instance as a category. Its optimization encourages that different instances lie in different regions. The consistency loss is formulated to directly compares the features of the augmented views from the same instance and encourages high similarity between them.
One benefit from the consistency optimization is to directly and explicitly make the features of the same instances compact and thus to accelerate the optimization the classification loss. This is different from Wu al. (2018), He et al. (201)), heuristically estimating the classifier weights using the prototypes and does not suffer from the prototype estimation reliability issue. On the other hand, our approach does not rely on large batch training, that is essential for SimCLR (Chen et al. 2020b), because the whole loss in our formulation can be decomposed as sum of components each of which only depends on one instance.
Furthermore, we observed that jointly optimizing the consistency and classification losses leads to that the representation more focused on the textured region, as shown Figure T This implies that the learned representation is more capable of characterizing the objects, and thus potentially more helpful for downstream tasks like object detection and segmentation.
We demonstrate the effectiveness of our approach in unsupervised representation learning on Ima- geNet. Our approach achieves competitive performance under the linear evaluation protocol. When finetuned downstream tasks, such as object detection VOC, object detection and instance segmentation on COCO, instance segmentation on Cityscapes and LVIS, as well as semantic seg- mentation Citeyscapes, COCO Stuff, ADE and VOC, our approach performs better than InstDise, MoCo and SimCLR, and competitively or superior compared to other methods with stronger training setting (e.g., InfoMin and SwAV).
RELATED WORK
Generative approaches. Generative models, such as auto-encoders (Hinton et al.) 2006 Kingma
& Welling 2014 Vincent et al. 2008), context encoders (Pathak et al., 2016), GANs (Donahue &
Simonyan 2019 and GPTs (Chen et al. 2020a), learn an unsupervised representation by faithfully
reconstructing the pixels. Later self-supervised models, such as colorization (Zhang al.) 2016) and brain encoders (Zhang et al. 2017), improve generative models by withholding some part of the data and predicting
We will show one possible reason that requires large hatch.
2

---

way that the features Under review as a conference paper at ICLR 2021
Spatial relation prediction. The representation is learned by solving pretext tasks related to image patch spatial relation prediction, such as predicting the spatial relation between two patches sampled from an image, e.g., patch is on the left of another patch. (Doersch et al. 2013); solving Jigsaw Puzzles and determining the spatial configuration for the shuffled (typically 9) patches (Noroozi & Favaro
2016); and predicting the rotation (Gidaris al. 2018).
Exemplar- CNN Dosovviskiyy et al.] 2016) regards the views formed by augmenting each instance a class, and formulates an instance classification problem. InstDise (Wul et al. 2018), MoCo (He et al., 2019), CMC (Tian et al. 2019) and PIRL (Misra & van der Maaten, 2019 generalize exemplar-CMy CNN by heuristically estimating the classifier weights using prototypes for easing the optimization. Our proposed approach follows the instance classification approach, and exploit an additional consistency loss to help optimization.
Instance classification.
Instance clustering. Rather than regarding each instance as category, the instance clustering frameworks (Caron et a 2018:
2019) 2019 2020; Asano et al. 2020 [uang al.] 2019) Xie al. 2016 Yan et al.] 2020] Yang et al. (0166) learn representations in which the instances are well optimized. DeepCluster (Caron et al. 2018) simply adopt the k-means clustering method by simultaneously optimizing the network parameters, and uses k-means assignments as pseudo-labels to learn repre- sentations. SWAV (Caron et al. 2020 simultancously clusters the data while enforcing consistency between cluster assignments for different views of the same instance.
Contrastive learning. Contrastive predictive coding (van den Oord et al. 2018 Hénaff et al. 2019) predicts the representations of patches below certain position from those above it by optimizing contrastive loss. DIM (Hjelm et al.] 2019) and ANDIM (Bachman et al.) 2019) achieves global-to- pecal/loclltto-neghheee patch representation prediction (overlapping) across augmented views using the contrastive loss.
The contrastive learning framework (Ye et al., 2019 Chen al. 2020b) formulates contrastive loss encouraging the high similarity between the augmented views from the same instance, and low similarity between the instance and other instances. WWang & Isola (2020) presents novel formulation based on two measures: alignment and uniformity, and shows that it is an alternative of contrastive loss. Yu al. (2020) connects contractive and contrastive learning, cross-entropy, and so on, and provides theoretical guarantees for learning diverse and discriminative features.
Consistency in semi-supervised learning. Consistency regularization, enforcing the similarity be- tween the predictions or features of different views for the same unlabeled instance, has been widely applied semi-supervised learning, such as IT Model (Laine & Aila (017). Temporal Ensem- ble (Laine & Aila 2017). and Mcan Teacher (Tarvainen & Valpola 2017). We exploit the consis- tency loss to help optimize the classification loss for unsupervised representation learning.
APPROACH
...,. IN). the goal is learn a fea- ture extractor (a neural network) x = f()). The discrimination approach expands each image I, to set of augmented views {I,,, 12 .2....... ... IK), and formulates the problem of the augmented views of each instance are similar (contraction) and the features of different in- stances are distributed separately (separation). the following, we first review three related instance classification methods, then we introduce our approach and present the analysis.
Given a set of image instances without any labels, I I2,
3.1 INSTANCE CLASSIFICATION
Exemplar CNN. Exemplar ENeNplar-CNNN CNN (Dosovitskiy et al. 2016) formulates unsupervised representation learning as instance classification problem. The augmented views from one instance regarded as one category, and the augmented views from different instances are regarded as different cate- gories. The softmax loss is used and written for the kth view of the nihh instance:
/
log ΣØ
8//7
where T is the temperature. Exemplar-CMe CNN uses the standard backpropagation algorithm to learn the network f()) and the classification weights (W1, W2,..., Wx).
3

---

Under review as a conference paper at ICLR 2021
5.50
0.040
05-
0.5
0.038
5.48
0.036
5.46
1.0
1..
0.034
(b)
-1.0
-0.
0.
05
15
Figure 2: Visualizing learned feature distributions for 2D toy examples. Each color corresponds to augmented views of the same instance. (a) jointly optimize the sistency and classification losses. (b) only optimize the classification loss.
Figure 3: Illustration of the fi- nal classification loss and consis- tency loss values in training.
The InstDisc approach (Wu al., 2018) optimizes the network parameters, and heuris- tically estimates the classifier weights /w, W...... ] in each epoch using feature moving average scheme, i.e., compute the exponential averages of the features of the corresponding stances (stored in memory bank) in the previous epochs. The heuristic weight estimation scheme eases the network optimization.
InstDisc.
MoCo. MoCo (He et al. 2019) instead adopts a network moving average scheme. In each SGD iteration, MoCo updates a momentum network whose parameters are moving average of the previous network parameters. It computes the features from the momentum network the classifier weights, which are further maintained by a queue. This leads to better classifier weight estimates.
3.2 CONSISTENT INSTANCE CLASSIFICATION
We introduce consistency loss explicitly penalize the dissimilarity between augmented views from the same instance. Let sim (u, v) u v |/|||||lv|| denote the inner product between lo nor- malized u and v, i.e. cosine similarity. The consistency loss for two views x², and x from the image I is formed as
(x x(,)
sim(x), x
Tx
(2)
Here, we normalize the feature vector x = x/||x||2 as done in InstDisc and MoCo. The consistency loss for the N images each with K augmented views is written
L. > n=1 E
L x,(x),,
X x2)2
(3)
The classification loss for the images each with K augmented views is written as
w T x
C. - Σ (x(x -
log
(4)
n=1
1=1
Σ/= e 8//7
where let the classifier weight be fy-normalized vector: normalizing the prototype vector as done in InstDise and MoCo. We combine the two losses together,
||w||2 1. which is similar to
C.
(5)
where a is weight for the consistency loss to balance the two losses, avoiding over-opti the consistency loss merely optimizing the classification loss.
general, when the features for each class are more compact, different classes are more easily separated and the softmax classification are more efficiently optimized. Our approach has the benefits: the feature distribution for the same instance is compact and the distributions for different instances
Consistency helps optimizing the classification loss.
well separable, because of maximizing the consistency. Figure 2 (a) illustrates the benefit from simultancously optimizing the consistency loss and the classification loss. Figure 2 shows the insufficiency of only optimizing the classification loss. One can see that the distributions of different instances in Figure 2 (a) are better separated and the distribution for each instance more compact.
4

---

Under review as a conference paper at ICLR 2021
55.0
62.2
35.0
25.6
64.0
61.8
25.4
63.5
61.6
62.5
81.2
34.8
(a)
butlch
1.044
(b)
batch 222
(c)
batch
Figure 4: Performance with different batch sizes: (a) Linear evaluation on ImageNet, (b) VOC detection, and (c) COCO-stuff segmentation. The performances with batch sizes 256, 512 and 1024
are similar.
Let's see how the consistency term makes the gradient of the classifier weight more effective. We have the gradient for the classification loss (x(x)) with respect to the classifier weight wn
(e2)) (Pir own
1)x where Pk
The gradient from two views x and x (x.))) - m #577 gw
t *///
+ (l.(x%)
(P 1)x, + (P - 1)x
(6)
Own
Own
According to the law of cosines, ||X; 112 and || = 1, we have
1)2 + (Pin 1)2 + 2( nn D)((PH 1)|,; xx.
(7)
When the consistency term is included, x and X are very close, implying that x T Txg is larger. In the case Pi, and P are not changed, the magnitude ||gw is larger. and accordingly the classifier weight w, is updated effectively and quickly. In contrast, when the consistency term is not included, x1, and &e, might be very diverse as discussed (see the discussion in "Optimizing the classification loss is not direct to optimize the consistency loss.") This results in that |/gwll2 is smaller, and thus the classifier weight w,, is updated less effectively and less quickly.
Figure 3 shows the final classification loss L. and consistency loss Le value with different consis- tency loss weights. We can see that increasing the consistency weight when smaller than 2.5 helps optimizing the classification loss, and when larger than 2.5 harms the classification loss optimiza- tion. Appendix B we discuss the reason: over- over-weights weighting the consistency loss could lead to a trivial solution. In our experiments, set a to 2.5 in which case the training classification loss is minimum.
Optimizing the classification loss is not direct to optimize the consistency loss. Optimizing the classification loss intuitively expects that each instance lies in a different region. We expect that the augmented views of an instance x., are assigned the nith region and compactly distributed. We find that merely optimizing the classification loss L. is not easy to make the features of the augmented views of the same instance contractive, consequently the features are not compactly distributed.
The reason is that larger similarity between augmented views is not explicitly encouraged, and is im- plicitly imposed through the classifier weight. The angle between x and x ((x;, x(, (reflecting the similarity between X and X 0(x), St, )), is upbounded:
0 x2) I 0 wn) 0(w.. R.))
Minimizing the classifier loss L, if given Wn. is possible that the numerators (e.g. w,, x and w,. X)), are larger and accordingly the upbound 0(8 ww)+0(w) W., X2, is smaller. However, we find that there exist many transformations
so that the upbound is the same: w (R xx) w n & and
0({,, wx) ( n2 w,.). In this case, 0(x x3. is likely to be very different from (Rx n: X )- This implics that there is still a gap between optimizing the upbound ((x', wn) + 0 (w,, x()) and
directly optimizing 0 (x x)).. As result, merely optimizing the classification loss is not easy to make the features for one instance compactly distributed in the corresponding region.
Batch size. We present rough analysis showing that instance classification, including our approach, MoCo, and InstDisc, does not require large batch (see He et al.) (2019) and the empirical validation in Figure 4 for our approach). We rewrite the loss function in Equation 5 aa
e x//T
L= Σ n=1
[a
D (1 sim(x;, xx,))
log
(9)
1=1
1=z
5

---

Under review as a conference paper at ICLR 2021
The reformulation indicates that the loss can be decomposed the sum of components, where each component depends on a different instance. The separation between instances is got through the classifier weights each of which encodes the information of the corresponding instance. The deconposability property leads to that the optimization of & using SGD behaves similarly to the standard classification problem with SGD: large batch size is not necessary. Figure shows that the performances with batch sizes 256, 512 and 1024 are similar.
In contrast, the contrastive loss over all the N instances in SimCLR, is
L =
(log
log
(10)
==1
We can see that this can not be decomposed as a sum of components each of which depends on dif- ferent instance, which is general requirement for SGD. Each instance depends on other instances. We believe that this is the reason why SimCLR needs large batch size (Chen et al. 2020b
4 IMPLEMENTATION DETAILS
Data augmentation. We adopt the augmentation scheme similar to SimCLR (Chen et al. 2020b). We randomly crop the input image with the crop scale (0. and resize it to 224 x 224. Then apply random horizontal flipping, color jittering, grayscale, and Gaussian blur.
Network architecture. We use ResNet-50 (He et al.. 2016) extract features. Following SimCLR we adopt the same projection head consisting of a two- two-layer natch-normalized MLP (Linear->BN- BeIJU>Linearr>Be and reduce the feature dimension from 2048 128 in pretraining.
Training. We use the SGD algorithm with momentum optimizer. We set the momentum parameter to 0.9, the weight decay parameter to 1e - 4, the batch size to 512, and the epoch number 200. We adopt the cosine learning rate schedule, with the initial learning rate 0.06. Each instance in the current mini-batch is augmented into two views during training. The temperature
set to be 0.1. We use SyncBN. For ablation study, we train all the models for 100 epochs. The training is performed on NVIDIA V100 GPUs We use the PyTorch 1.3 platform (Paszke et al., 2019).
Sampling classifier weight update. The analysis is based on the standard SGD algorithm. For clarity, we assume each iteration samples instance with two augmented views. The analysis can be easily extended to sampling more instances with more augmented views. The loss function becomes
x//7
L = 2(1 - sim x²))
log
(11)
1=
8//T
It can be seen that the denominator in the second term on the right-hand side, 2j= e / is summation of N elements, and thus the complexity is O(N). We propose to approximate it by summing fewer (N' = 65536 is the same to the queue size in MoCo He et al. (201)) elements:
/
(12)
where we let the sampling compensation weight N-1 for better approximation (See Ap- pendix A.I. This approximation reduces the forward loss computation complexity to O(N').
The normal iteration process needs update all the N classifier weights in each iteration, imply- ing the complexity is still ((V). Fortunately, through derivation (see Appendix A.2), we find that we do not need compute the gradients and update the classifier weights corresponding to the stances that are not sampled. We can delay the update to the iteration that the instances are sampled. In other words, at each iteration we only compute the gradients and update the classifier weights corresponding to the instances that are sampled. Consequently, the gradient computation and the classifier weight update, and accordingly each iteration takes ((N') time.
In our implementation, rather than sampling all the N' classifier weights at each iteration, we only use the classifier weights, which correspond to the (e.g.. 512) instances in a mini- batch, to replace the classifier weights that the earliest sampled. The potential benefit is to reduce the 10 cost if we store the weights in the disk or the CPU memory and only store the sampled weights in the GPU memory, which is practically valuable for very large scale cases (e.g., B or more images).

---

Under review as a conference paper at ICLR 2021
Table Illustrating how the consistency weight a influences the performance The observations are
consistent to the one about the classification loss shown in Figure
VOC Det.
COCO Det.
COCO Keypoint
DensePose
Instance Seg.
Semantic
Seg.
VOC07+12
Mask-R
Mask-R
2X
DP-R Apdz
City
LVIS
City. mlnt)
Stuff
ADE
Context
ApM
3P
Aphh
Apm
Apłp
AP
Apmk
Apm*
mloU
mlot
mlot
mlot
63.6
54.9
80.7
38.4
34.9
65.8
87.2
63.7
31.6
24.2
76.7
33.4
41.5
75.2
47.2
56.1
81.5
39.0
66.2
87.3
64.5
32.4
24.7
76.9
33.6
41.0
33.6
65.1
56.7
82.0
39.3
35.5
66.3
87.1
64.4
32.3
24.4
77.3
34.3
41.7
77.2
48.7
2.5
65.1
56.4
81.9
39.2
35.5
66.3
87.1
64.4
33.1
25.4
77.9
35.3
41.7
77.1
49.4
3.5
64.6
56.6
81.6
39.3
35.5
66.3
87.2
65.0
33.2
24.9
77.9
34.5
42.0
76.9
48.8
56.1
39.2
66.0
87.2
77.0
21.5
21.0
$ 53.0
61.0
5 243
41.3
635
240
01.
62
1450
31.0
NYY LITD IN COM
63.0
C000 CHBE D00A
AIL AFASRD
215
1600
60.
5350
ADEIOK
OUTOO AT cOMp
Figure 5: Illstrating the effect of sampling classifier weight update. Three results, the baseline w/o sampling, sampling update w/o sampling compensation, and sampling update w/ sampling compen- sation, are reported. The results show that sampling update w/ sampling compensation performs better than w/o sampling compensation and similar to the baseline w/o sampling.
EXPERIMENTS
We conduct the evaluation by training the models on ImageNet (Deng et al. 2009) w/o using the labels. We follow He al. (2019) and adopt two protocols, linear evaluation ImageNet, and downstream task performance with fine-tuning.
Ablation study: consistency. Figure 3 shows how the consistency weight parameter influences the classification loss. The results are as our expectation and suggest that the classification loss decreases when the parameter increases certain value 2.5 that choose and then the classification loss increases. The results on the downstream tasks and linear evaluation shown in Table indicate consistent observations: the overall performance when 2.5 performs satisfactorily, and better than the performance w/o consistency (a = 0).
in experiments,
A Ablation study: sampling classifier weight update. We evaluate how sampling classifier weight update and sampling compensation affect the performance. Figure iSjdicttes that sampling com- pensation makes the results w/ the sampling scheme overall similar the results w/o the sampling scheme and better than w/o sampling compensation (3 1).
Comparison with state-of-the-arts We compare our approach, consistent instance classification (ConIC) to recent state-of of AlignUniform, SwAV, and Info The pretrained models of MoCo v1/v2, AlignUniform, SwAV, and InfoMín are obtained from GitHuu provided by the corresponding authors. The PIRL pretraine model obtained from PyContrast" We implement Exemplar- CNN and InstDisc using the same setup with ours, including & normalization and data augmentation. The comparison to these meth- ods is fair as the models are pretrained with almost the same setting, c.g., #epochs is all 200, data augmentation is almost the same, the backbone the same, and each instance is augmented two views. We fine-tune all the models using the same setting for the downstream tasks.
-fiee-of-hhearrrr the-art solutions: Exemplar- CNN, InstDisc, PIRL, MoCov1, MoCov2,
The results for downstream tasks given in Table The overall performance of approach (ConIC) is the best, and the overall performance of our approach w/ sampling classifier weight update (ConIC w/sampling) is the second best. In contrast, the best among the previous methods, AlignUng Uniform (Wang & Isola 2020) performs satisfactorily for most tasks and unsatisfactorily for the segmentation tasks on Cityscapes, ADE20K, and Pascal- -VCC. The superiority of our approach shows that minimizing the consistency loss improves the capability of characterizing the objects and the feature transferability.
'The results of other methods for downstrcam tasks are given in Appendix https:// thub. com/ Hobbi http:/oooooiiiloooo////otrtrre

---

Under review as a conference paper at ICLR 2021
Table Comparison of our approach ConIC with recent state-of-the-art solutions. We highlight the
best and second-bes beanddbesttto scores among the approaches w/o strong setup in red and blue, respectively.
Method
Instance Seg.
Semantic Segmeniation
COCO Keypoint
City.
LVIS
City.
Stuff
ADE
VOC
Context
AP
AP
Exemplar CNN (Dosovissky
25.3
34.0
11.5
18.6
6b.1
57.2
Iis:Digg (Wu et ai. 2013
31.9
24.6
76.8
33.9
41.3
76.9
49.0
65.8
86.9
Missa
der Manten
25.0
34.1
40.2
47.6
66.2
$6.9
MoCo VI (He 2019
32.7
25.2
77.5
34.3
41.4
76.2
17.3
06.3
87.0
MoCo v2 (Clica al 2020g
33.0
25.6
77.6
33.4
11.6
78.3
50.3
66.5
87.5
AlignUnitom Wang
2020
33.5
25.6
76.7
35.9
40.7
74.2
50.8
66.4
87.3
ConfC.
33.6
25.66
78.5
36.0
42.1
75.3
50099
06.5
N7.3
ConIC w/ sampling
33.5
25.0
35.5
41.9
50.8
06.2
87.2
Approuches with stronger selup
SwAV (Caron al. 2020
3336
25.77
74.8
33.1
42.4
77.3
47.2
65.7
86.7
InfoMin
2020
33.7
25.6
78.2
36.2
42.3
78.7
51.1
66.5
87.5
SimCLR Chen of 20200
31.5
26.1
59.4
11.1
37.8
32.0
20.2
65.3
86.8
VOC Detection
COCO Detection
DensePose
Method
VOC,7
VOC07=12
Mask R Tx
Mask 2x
DP DERNNN
AP
AP
AP
AP
AP"
AP
AP
APmi
AP
Exemplar- Dosovisski et al. 2016
47.1
75.0
53.9
80.1
38.7
35.0
41.4
37.4
64.4
InstDisc (W ar
16.9
75.1
56.0
11.4
37.1
61.0
PIRL (Mira & van der Moaten 2019
45.9
73.9
55.4
81.0
38.7
35.1
41.4
37.4
64.3
Mo HH al 2019
16.6
74.9
55.9
81.5
4
35.6
11.7
37.5
64.3
MoCo v2 (Chen el 2020c
48.2
76.3
37.0
82.4
39.7
36.0
11.9
37.8
65.1
AlignUniform Waan & Isola 2020
48.6
77.0
57.2
82.4
39.7
35.9
11.9
37.8
64.6
ConIC
76.88
57.5
39.9
36.0
41.9
37.9
ConIC w/ sampling
48.8
76.5
57.3
82.7
39.7
36.0
42.0
37.9
64.8
Approaches with strouger setup
SwAV Caron et al. 2020
42.5
75.0
54.9
81.9
40.9
37.0
42.7
38.5
62.6
InfoMin Aug.
2020
77.0
57.6
36.
42.5
38,4
65.6
SimCLR Chen et a. 20.00
25.5
50.8
39.1
72.2
39.7
36.1
42.2
38.2
62.7
Table 3: Comparison for linear evaluation ImageNet. Our approach gets comparable results to MoCov2, A-U, and others that train models using similar setup. See Appendix F for more discus-
sions.
Local Agg.
Method
ConfC
ConfC-S
E-CNN
InsDise
PIRL
CMC
CPC 12
MoCo v1
McCo v2
SimCLR
A-U
PIC
PCL.I
PCL
BowNer
Sal.a
InfoMin
SWAY
SinCLR
YOL
#Epochs
200
200
200
200
200
200
200
200
200
200
200
200
200
200
200
280
280
800
800
1000
1000
Top-1
acc
67.6
67.4
60.2
61.4
6
61.7
66.2
63.8
60.6
67.
66.6
67.7
67.6
61.5
67.6
62.1
1
68.8
73.0
75.3
69.3
71.3
In addition, we report the results of three approaches w/ stronger setup, InfoMin, SwAV, and Sim- CLR. We got the pretrained models provided by the authors. (1) InfoMin performs similarly to our approach, but it adopts stronger augmentation, RandemAugment (Cubuk et al. (020) that is learned from supervised learning. (2) SimCLR (1000 epochs) performs inconsistently and surpris- ingly 5 5 (3) SwAV performs much better than our ConIC for COCO detection, and much worse for VOC detection, DensePose, semantic segmentation on Cityscapes, COCO stuff, Pascal- VOC, and Pascal-Contex.
Linear evaluation results on ImageNet are in Table Our approach performs competitively in comparison to MoCo v2, PIC, and PCL v2 whose training setup is similar to our approach. Other approaches, e.g, InfoMin, SwAV, BYOL, training the models using stronger augmentation, more views, more epochs, respectively, get higher performance. See more analysis in Appendix
CONCLUSION
We exploit the consistency loss minimization to help the optimization of the instance classification loss. The benefits include: the representations of different views of the same instance pact; the representations of different distances are more separable; the representations characterize more about the textured region in an image. These lead high capability downstream tasks like object detection and segmentation.
com-
We contacted the authors to see if we use the models correctly for some downstream tasks, and the feedback they did not check the performance for those downstream tasks.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. In ICLR, 2020.
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximiz- ing mutual information across views. In NeurlPS, 2019.
Holger Caesar, Jasper R. R. Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. CVPR, 2018.
Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin, Zheng Zhang, and Han Hu. Parametric instance classi- fication for unsupervised visual feature learning. CoRR, abs/2006. 14618, 2020.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features, In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), ECCV, 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-tra pre-training of image features on non-curated data. In ICCV, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. CoRR, abs/2006.09882, 2020.
Liang- Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CoRR, abs/ 706.0/7776.5587, 2017.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. ICML, 2020a.
Ting Chen, Simon Kormblith, Mohammad Norouzi, and Geoffrey E. Hinton. simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020b.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. CoRR, abs/2003.04297, 2020c.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshop, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In NeurlPS, pp- 10542-10552, 2019.
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin A. Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. TPAMI, 38(9):1734-13 1747, 2016.
Mark Everingham, Luc Van Gool, Christopher I. Williams, John M. Winn, and Andrew Zisser- man. The pascal visual object classes (VOC) challenge. IJCV, 88(2):303-338 2010.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations predicting bags of visual words. In CVPR, 2020.

---

Under review as a conference paper at ICLR 2021
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your latent: A new approach to self-supervised learning. CoRR, abs/2006.07733, 2020.
Riza Alp Güler, Natalia Neverova, and Iasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In CVPR, 2018.
Agrim Gupta, Piotr Dollár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, 2019.
Bharath Hariharan, Pablo Arbelaez, Lubomir D. Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In Dimitris N. Metaxas, Long Quan, Alberto Sanfeliu, and Luc Van Gool (eds.), ICCV, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. CoRR, abs/1911 1.05222, 2019.
Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding. CoRR, abs/1905.09273 2019.
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 1877):152-11544 2006.
R. Devon Hjelm, Alex Fedorov, Samuel Maroii--ar Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Jiabo Huang, Qi Dong, Shaogang Gong, and Xiatian Zhu. Unsupervised deep learning by neigh- bourhood discovery. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), ICML, 2019.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Be- longie. Feature pyramid networks for object detection. In CVPR, 2017.
Ishan Mísra and Laurens van der Maaten. Self-supervised learning of pretext-invarian representa- tions. CoRR, ftss/9912.01991 2019.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan L. Yuille. The role of context for object detection and semantic seg- mentation in the wild. In CVPR, 2014.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. CORR, abs/1 000.099469 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang. Zach Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. CORR, 90200193,0 2019.
Deepak Pathak, Philipp Krähenbühl. Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
10

---

Under review as a conference paper at ICLR 2021
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. TPAMI, 396))1137-11149 2017.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding.
CoRR,
abs/1906.05849, 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. CoRR, abs/2005.10243, 2020.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1 807 207.03748, 2018.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In William W. Cohen, Andrew McCal- lum, and Sam Roweis (eds.), ICML, 2008
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. CoRR, abs/2005.10242, 2020.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
Zhirong Wu, Yuanjun Xiong. Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-
parametric instance discrimination. CVPR, 2018.
Junyuan Xie, Ross B. Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), ICML, 2016.
Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan. Clusterfit: Improving generalization of visual representations. In CVPR, 2020.
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In CVPR, 2016.
Mang Ye, Xu Zhang, Pong C. Yuen, and Shih-Fu Chang. invariant and spreading instance feature. In CVPR, 2019.
Unsupervised embedding learning via
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. CoRR, abs/2006.08558, 2020.
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In ECCV, 2016.
Richard Zhang, Phillip Isola, and Alexei Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In CVPR, pp. 1058-1067, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, 2017.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. ICCV, 2019.
11

---

Under review as a conference paper at ICLR 2021
A SAMPLING CLASSIFIER WEIGHT UPDATE
A.1 THE CHOICE OF /
We approximate the part (corresponding the negative instances) of the denominator in the classi-
fication loss L, = log
e R//7
by sampling subset of classifier weights. That is
Σg RE/T"
N
Σ
Σ
e
/
(13)
Assume that the classifier weights corresponding to the (N - 1) negative instances, (Wi....: Wn-1, Wn-1..... W are ii.d., we want that the expectations of the term on the left hand side and the right hand side are the same:
BiB
Σ
Σ 5 j=1ij
e 2
(14)
The term on the left hand side is
" x BN Ele 2t/T].
E
(15)
where w has the same distribution with Wy (j * n). Similarly, the term on the right hand side becomes
El Σ w x 2 (N 1) Ele" x3/7].
(16)
Thus, we have 3 = N-1 During the SGD iterations, the i.i.d. assumption does not hold. But our experiments show that the choice, B N-1 improves the performance, better than B 1. We think that tuning 3 manually might lead to superior performance.
NO.
A.2 DELAY UPDATE OF THE UNSAMPLED CLASSIFIER WEIGHTS
Consider classifier weight w that sampled at the (s)th iteration and the (sth iteration, and is not sampled from the (s + 1)th iteration to the (s + k 1)th iteration. We have (1) the gradient of the loss L with respect to w is zero, 8w O, (2) the gradient of the la regularizer R | |
is 2R Aw, and thus the gradient becomes: gw( Aw.
The update equation of SGD with momentum becomes
vw (o+1)7 w(++1)
vil (w(s)
m
17 (1 (((+1)))))
(17)
from which we get:
(a+k) (++++))
Vw
m (1 (1- (1-1)1++1)) -n(
2)
A (1 (1- ( x)
(w)s))
(x)
(18)
This means that we do not need to really compute w at the iterations in which it is not sampled, and only need to update the iteration in which is sampled again,
In addition, we observe that w (unsampled) is updated independently and does not influence the update of other classifier weights Consequently, we are safe delay the update of the classifier weights that are not sampled to the iteration in which the weight is sampled again.
B MORE ANALYSIS
Trivial solution for merely optimizing the consistency luss. Let us look at the consistency loss in Equation It is obvious that L. > 0. We see that the minimum L. 0 holds, if the features
12

---

Under review as a conference paper at ICLR 2021
of all the augmented views for image the same: x, x It also holds in theory when that different images can have different representations: x, * x However, we empirically observe that merely optimizing the consistency loss always leads to the trivial solution: the representations of all the augmented views for all the images are the same, X 7 X
Hard sample mining. It is known that the softmax loss has benefit: hard samples contribute more to the gradient and thus the parameter update. We show that the consistence term has a similar property. The gradient of the consistence term l in Equation 2 with respect to x is
Ol Ox;
2(1 - X
(19)
In the hard sample case, the similarity x, T &/ is smaller and far from (1 X x2)) is larger, implying the gradient magnitude is larger. This means more contribution to the gradient. In the easy sample case, the contribution would be smaller.
C EVALUATION SETUP
C.I EVALUATION DOWNSTREAM TASKS
We perform object detection, COCO keypoint detection, COCO DensePose estimation and Instance segmentation experiments on Detectron2 (u et al. 2019) framework.
Object detection. We perform object detection on Pascal VOC (Everingham et al. 2010) and
COCO (Lin et al. (0144) datasets. For Pascal VOC, we use Faster-RCNV (Ren et al.
2017) with
R50-C4 backbone as the detector. Following He et al. (2019), extra BNs are added in newly initial- ized layers. We fine-tune all layers (including BN layers) in object detection experiments. Initial learning rate is 0.02. Two training schemes adopted: (i) the model is trained train2007 set for 9k iterations, with learning rate decay at 6k and 8k iteration. (ii) the model is trained on trainval0 07+12 set for 24k iterations, with learning rate decay at 18k and 22k iterations. We report APA 50 and standard COCO-style Apbb, For COCO object detection, we use Mask-RCNN (He et al. 2017) with R50-FPN (Lin et al. 2017) backbone the detector. SyncBN is adopted in back- bone, FPN and ROI Heads. The model is fine-tuned on t train2017 set and evaluated on va ral22 1211111 set. We use standard 1x and 2 fine-tune schedule. Standard COCO-style bounding box Ap and mask Apm are reported.
COCO keypoint detection. We perform human pose estimation on COCO keypoint (Lin et al. 2014) dataset. We use Mask-RCNN (He et al. (017)) (keypoint version) with R50-FPN backbone as the detector. SyncBN is adopted in backbone, FPN and ROI Head. The model is fine-tuned on train2011 set and evaluated on val2017 set. Standard 2x fine-tune schedule is applied. We report APpp and APAp 50'
COCO DensePose estimation For DensePose Güüler al., 2018 estimation, We use Dense- Pose R-CNN with R50-FPN backbone. SyneBN is adopted in backbone, FPN and ROI Box Head. The model trained on train2014 4 + valmi nusminival 2214 and evaluated on minival201- 4. We use "six" fine-tune schedule (improved baseline TR-50_FNNsl1 in De- tectron2). We report APdp DensePose GPS metric.
Instance segmentation. We perform instance segmentation on COCO (Lin et al. 2014), Cityscapes (Cordts et al.] 2016), and LVIS (Gupta et al.] 2019) datasets. COCO instance seg- mentation is jointly-trained with COCO object detection with Mask-RCNN model. We use Mask- RCNN with R50-FPN for fine-tuning. SyncBN is adopted backbone, FPN and ROI Heads. For Cityscapes, the model is trained eityscapes. fine-iin mee.innnnneeee. see.ttaiinnnnneeeeeee..ia.aaiiit and evaluated on cityscapes fine instance seeeesse_vve val for 24k iterations. For LVIS, the model trained on lvis v0. 5. train and evaluated on lvis v0. 5 val with 2x schedule. Standard Apmk is reported.
Semantic segmentation. We perform semantic segmentation Cityscapes (Cordts et al.) 2016 COCO-stuff (Caesar et al.] [2018), ADE20k (Zhou et al.] 2017). Pascal- VOC (Everingham et al. 2010), and Pascal-Context (Mottaghi al. 2014) datasets. We Deeplab (Chen et al. 2017
13

---

Under review as a conference paper at ICLR 2021
with R50-dilated8 backbone. We use SGD with momentum optimizer and lambda poly learning rate schedule for semantic segmentation experiments. We employ cross entropy loss on both the final output of DeeplabV3 and the intermediate feature map output from Stage3, where the weight over the final loss is and the auxiliary loss is 0.4. Single-scale testing adopted for all experiments. For Cityscapes experiments,
train the model for 40k iterations with batch size 8, initial learning rate 0.01, input size 1024 x 512. For COCO-stuff experiments, we train the model for 60k iterations with batch size 16, initial learning rate 0.01, imput size 520 x 520, For ADE20K experiments, the model is trained for 150k iterations with batch size 16, initial learning rate 0.02, input size 520 x 520. For Pascal- VOC experiments, we use train_aus train_aug2012 set (augmented by Haarhaarn al. (20IT) as training set. The model is trained for 60k iterations with batch size 16, initial learning rate 0.001 and input size 513 513. For Pascal-Context experiments, the model is trained for 30k iterations with batch size 16, initial learning rate 0.001 and input size 520 x 520. Standard mloU metric is reported.
C.2 LINEAR EVALUATION
We freeze the pretrained backbone and train linear classifier on the frozen feature. The classifier trained for 100 epochs with initial learning rate and cosine learning rate schedule. We set weight decay to O. The data augmentation is the same as supervised ImageNet classification.
D IMPLEMENTATION DETAILS OF THE TOY EXAMPLE
Figure shows the learned feature distributions for 2D toy examples. We train the models (with ResNet-50 encoder)
toy dataset, containing ImageNet images. We apply RandomCrop (0.7,1) on the images to generate augmented views. The models are trained for 200 epochs with cosine schedule and initial learning rate 0.0001. We use batch size 8 and weight decay le 6. The dimension of features output from projection head is 2. For the classification only experiment, set a 0. For the jointly optimization of the consistency loss and classification loss, we = 0.1. After training, the learned features of 20 random augmented views of each image are recorded. We apply kernel density estimation with a Gaussian kernel of std 0.04 the recorded features for visualization. Each color represents the learned feature distribution of augmented views from an image.
E DATA AUGMENTATION
We provide the PyTorch pseudo code of the data augmentation adopted, follows:
augmentation =
transforms. RandonRessizeddrrrpp22 (224, scale= (0.15,, 1.)),
transforms Lansoonffrciicootllllll..
transforms han.eoormms..nnooooplplly((
zransforms.colorjitte (0. 0.8, 0.8, 0.2)
1, p=0.8), transforms. hww.::mmssnnnnnoennassssaeeeeeee (p=0.
transforms. RandomApply ([GaussianBlur (1.1, 2.1)), p=0.5),
transforms. ToTensor
10
normalize]
F EXPERIMENT RESULTS OF OTHER METHODS
The abbreviations Table are explained the following: ConlC-S ConIC w/ sampling, Local
Agg. Local Aggregation (Zhuang al. 2019). E-CNN Exemplar -NNN
CNN (Dosovitskiy et al.
2016 A-U = AlignUniform (Wang & Isola 2020). BowNet Gidaris et al. (0020).
Table 4 shows the results of on VOC object detection form some other methods that not included in Table 2 The results are got from the corresponding papers. BYOL, PCL, BowNet. SeLa adopted different evaluation setups and thus their results are not reported. Because of time limitation,
14

---

Under review as a conference paper at ICLR 2021
rently not able re-implement these algorithms use their provided pretrained models and evaluate them on other downstream tasks.
Table 4: VOC object detection results for other methods that are not included in Table
VOC Detection
Method
VOCO7
VOC07+12
Apá
AP
Apés
AP
Local Ageregation (Zhuang 2019)
69.1
PIC (Cao et al. 2020)
57.1
82.4
CPC v2 (Henalt et 2019) (RsNect1161)
76.6
ConIC
48.8
76.8
57.5
82.4
ConIC w/ sampling
48.8
76.5
57.3
82.7
15


Reviews:
[{"review_id": "j4u0GkOiGvb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the instance classification solution for an unsupervised representation learning problem. Particularly, this paper proposes an additional consistency loss that is simultaneously optimized with classification loss, in order to penalize feature dissimilarity between augmented views of the same instance. Such consistency loss makes classification loss optimization easier and avoids large batch size. Extensive experiments on downstream tasks, e.g., segmentation and detection, show the effectiveness of the proposed method.\n\nCompared with previous instance classification solutions, the major contribution is that this paper introduces a consistency loss on augmented views from the same instance. The idea that compacts the augmented views is not new. And, the theoretical contribution of the proposed consistency loss is limited, which just penalizes the cosine dissimilarity. Thus, it is far from a \"decent\" ICLR paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SufIunNG2s", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a loss function for unsupervised representation learning. It has two terms: an instance classification loss and a consistency loss. The novel part seems to be the consistency loss. It explicitly penalizes the dissimilarity between different views of the same instance.\n\nGood paper, accept\n\n+The proposed method is supported with extensive experiments. The proposed consistency loss which is basically cos similarity between different views of the same sample seems to be novel.\n\n-The proposed term L_C is kind of redundant. The proposed term L_C explicitly pushes positive samples to come closer based on cos similarity.\nBut, the other term, L_S, already bring positive samples closer and repels the negative samples. Here by positive, I mean samples that are different views of the same instance. Likewise, by negative I mean samples that are from different instances. That being said, explicitly adding L_c term with hyperparameter \\alpha provided more fixability in balancing the push/repel phenomenon by tuning the hyper-parameter \\alpha. \n\n-Compared to the state-of-the-art methods reported in Table 2, the improvements are really really minor. In fact, in most columns, there is no meaningful improvement. \n\nminor comment:\nw/ stronger setup --> \"with\" stronger setup\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HknMklB9oN6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes adding an additional loss term to instance classification, within the context of self-supervised pre-training.  Specifically, in addition to the standard classification loss term that views each image (and its augmentation) as a separate category, a second loss term is added, which is 1 minus the inner product of the representations for two augmentation views of the same image.  In a sense, this is incorporating an aspect of the contrasting learning approach, as two views of the same image are being explicitly pulled toward one another due to the consistency loss.  \n\nPositives:\n+ sensitivity analysis with respect to the balancing coefficient $\\alpha$ between the classification loss and consistency loss demonstrates that there is an optimal $\\alpha$ such that the consistency loss leads to a minimum of the classification loss, even compared with setting $\\alpha=0$.  In other words, the consistency loss is helping to optimize the classification loss.\n+ related to above, empirical results and ablation study demonstrates the benefit of adding in consistency loss for subsequent downstream tasks\n+ qualitatively, adding the consistency loss seems to help the network focus on relevant/textured regions of the images\n\nNegatives:\n- Novelty is arguably somewhat limited, as the proposed method boils down to adding a term that encourages similarity between two views of the same image, very similar to contrastive learning.\n- Empirical results are mixed - in general on-par with existing methods.  Performance is generally consistent across the explored tasks, but does not appear to be significant better than existing approaches.  One data point that would be of interest is whether the proposed approach does better when training for more epochs, similar to methods such as SwAV or MoCo v2 on ImageNet linear evaluation.\n\nOverall summary: \n\nGiven the negatives mentioned above, I currently see the paper as being borderline/marginally below threshold.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a consistency loss for instance discrimination by adding a term to maximize the squared dot product between two views of the same image. The impact of the proposed approach is evaluated on a variety of settings with mixed improvements. While reviewers generally found the proposed method to be interesting, there were concerns regarding the novelty of the approach, the size of the performance improvement, and the choice to focus on instance discrimination vs. more recent approaches based on contrastive instance discrimination. 

While I do not share the reviewer concerns regarding novelty, I am sympathetic to the concerns regarding the size of the improvement and the focus on instance discrimination. As such, I recommend that the paper be rejected in its current form. I would encourage the authors to apply their analysis and method to more recent contrastive instance discrimination approaches such as SimCLR and SWaV as well as non-explicitly contrastive, but high performing methods like BYOL. I would also encourage the authors to provide quantitative empirical analyses demonstrating the impact of the consistency term on large models rather than just toy models to demonstrate the impact of the consistency term in representational space. 
Decision: Reject