Contrastive Multiview Coding
Authors: Yonglong Tian
Dilip Krishnan
Phillip Isola
Venue:  (2020)
Keywords: Representation Learning
Unsupervised Learning
Self-supervsied Learning
Multiview Learning

Abstract: Humans view the world through many sensory channels, e.g., the long-wavelength light channel, viewed by the left eye, or the high-frequency vibrations channel, heard by the right ear. Each view is noisy and incomplete, but important factors, such as physics, geometry, and semantics, tend to be shared between all views (e.g., a "dog" can be seen, heard, and felt). We hypothesize that a powerful representation is one that models view-invariant factors. Based on this hypothesis, we investigate a contrastive coding scheme, in which a representation is learned that aims to maximize mutual information between different views but is otherwise compact. Our approach scales to any number of views, and is view-agnostic. The resulting learned representations perform above the state of the art for downstream tasks such as object classification, compared to formulations based on predictive learning or single view reconstruction, and improve as more views are added. On the Imagenet linear readoff benchmark, we achieve 68.4% top-1 accuracy. 

Under review as a conference paper at ICLR 2020
CONTRASTIVE MULTIVIEW CODING
Anonymous authors
Paper under double-blind review
ABSTRACT
Humans view the world through many sensory channels, e.g., the long-wavelength light channel, viewed by the left eye, or the high-frequency vibrations channel, heard by the right ear. Each view is noisy and incomplete, but important factors, such as physics, geometry, and semantics, tend to be shared between all views (e.g.. a "dog" can be seen, heard, and felt). We hypothesize that powerful representation is one that models cov-riiari factors. Based on this hypothesis, a contrastive coding scheme, in which a representation is learned that aims to maximize mutual information between different views but is otherwise compact. Our approach scales to any number of views, and is view-agnostic. The resulting learned representations perform above the state of the art for downstream tasks such as object classification, compared to formulations based on predictive learning or single view reconstruction, and improve as more views are added. On the Imagenet linear readoff benchmark, we achieve 68.4% 68.4% top-1 accuracy.
investigate
INTRODUCTION
foundational idea in coding theory is to learn compressed representations that nonetheless can be used to reconstruct the raw data. This idea shows up in contemporary representation learning in the form of autoencoders (Salakhutdinov & Hinton, 2009) and generative models (Kingma & Welling, 2013; Goodfellow et al., 2014), which try to represent data point or distribution as losslessly as possible. Yet lossless representation might not be what we really want, and indeed it is trivial to achieve - the raw data itself is lossless representation. What we might instead prefer is to keep the "good" information (signal) and throw away the rest (noise). How can we identify what information signal and what is noise?
To an autoencoder, or a max likelihood generative model, a bit is a bit. No one bit better than any other. Our conjecture in this paper is that some bits are in fact better than others. Some bits code important properties like semantics, physics, and geometry, while others code attributes that we might consider less important, like incidental lighting conditions thermal noise in a camera's 's sensor.
We hypothesize that the good bits are the ones that are shared between multiple views of the world, for example between multiple sensory modalities like vision, sound, and touch. Under this perspective "presence of dog" is good information, since dogs can be seen, heard, and felt, but "camera pose" is bad information, since camera's pose has little or effect on the acoustic and tactile properties of the imaged scene. There is significant evidence in the cognitive science and neuroscience literature that such cross-view representations are encoded by the brain (c.g., Smith & Gasser (2005); Den Ouden et al. (2012); Hohwy (2013)).
Our goal is therefore to learn representations that capture information shared between multiple sensory views but that are otherwise compact (i.e. throw away the bad information). To do so, we employ contrastive learning, where we learn a feature embedding such that views of the same scene map to nearby points while views of different scenes map to far apart points. In particular, we adapt the recently proposed method of Contrastive Predictive Coding (CPC) (Oord et al., 2018), except we simplify it - removing the recurrent network and generalize showing how to apply it to arbitrary collections of views, rather than just to temporal predictions. In reference to CPC, we term our method Contrastive Multiview Coding (CMC). The contrastive objective in our formulation, in CPC, can be understood as attempting to maximize the mutual information between the representations of each view.
We intentionally leave "good bits" only loosely defined and treat its definition as an empirical question. Ultimately, the proof is in the pudding: we consider representation to be good if it makes subsequent

---

Under review as a conference paper at ICLR 2020
Learned representation
21
22
2/
24
fize
fia
fe
fos
foa
vpj E 1/2
vi Vi Anchor
1ij E 1/2
Negative
Depth
Segmentation
Positive
(a) Two view contrastive learning
(b) Multiview representations
Figure 1: (a) Given pair of sensory views, deep representation is learnt bringing views of the some scene together in embedding space, while pushing views of different scenes apart. Here show an example learning from luminance channel (L) an image and the ab-color channel. The strawberry's L and ab channels embed to nearby points whereas the ab channel of different image (a photo of blueberries) embeds to far away point. (b) Example of 4-view dataset (NYU RGBD (Nathan Silberman & Fergus, 2012)) and its learned representation. Dotted lines represent the contrastive objective. The encodings for each view may be concatenated form the full representation of scene.
problem solving easy, on tasks of human interest. For example, useful representation of images might be feature space in which it is easy to learn to recognize objects. We therefore evaluate our method by testing if the learned representations transfer well standard semantic recognition tasks. On several benchmark tasks, our method achieves state of the art results, compared to other methods for self-supervised representation learning. We additionally find that the quality of the representation improves as a function of the number of views used for training. Finally, compare the contrastive formulation multiview learning to the recently popular approach of cross-view prediction, and find that in head-to-head comparisons, the contrastive approach learns stronger representations.
The core ideas that we build on: contrastive learning, mutual information maximization, and deep representation learning, not and have been explored in the literature representation and multiview learning (Li et al., 2018; Xu et al., 2013; Arora et al., 2019). Our main contribution is to up framework extend these ideas any number of views, and empirically study the factors that lead to success in this framework. A review of the related literature is given at the end of the paper, in Section Fig. gives pictorial overview of our framework for the different learning tasks we consider in this paper, to learn representations across datasets with different sets of views. Our main contributions are:
We apply contrastive learning the multiview setting, attemping to maximize mutual information between representations of different views of the same scene (c.g., between different image channels, or different modalities).
Our approach yields representations that outperform the state-of-the-eee art in self self-uupervised learning sead-to-hedd comparisons. For example, in the ImageNet linear readoff evalua- tion, we achieve 68.4% top-1 accuracy, which is slightly above the state of the art concurrent work Bachman al. (2019).
We show that the contrastive objective is superior to cross-view prediction.
We extend the framework to learn from more than two views, and show that the quality of
the learned representation improves number of views increase.
We conduct controlled experiments to measure the effect of mutual information on represen- tation quality.
METHOD
Our goal to learn representations that capture information shared between multiple sensory views without human supervision. We start by reviewing previous predictive learning (or reconstruction-
2

---

Under review as a conference paper at ICLR 2020
P1
1/2
Figure 2: Predictive Learning vs Contrastive Learn- ing. Cross-view prediction (Top) leams latent repre- sentations that predict view from another, with loss measured the oulpui space. Common pre- diction losses, such as the Li and L2 norms, are unstructured, in the sense that they penalize cach output dimension independently, perhaps leading representations that do not capture all the shared information between the views. In contrastive learn- ing (Bottom), representations are learnt by contrast- ing congruent and incongruent views, with loss mea- sured in representation space. The red dotted out- lines show where the loss function is applied.
9
(a) Predictive learning
U/
0
0o
(b) Contrastive learning
based learning) methods, and then elaborate on contrastive learning within two views. We show connections to mutual information maximization and extend scenarios including more than two views. We consider collection of M views of the data, denoted as Vi...., VM- For each view V, we denote U/ as a random variable representing samples following 8/ ~ P(V.).
2.1 PREDICTIVE LEARNING
Let V12 and V22 represent two views of a dataset. For instance, V1 might be the luminance a particular image and 222 the chrominance. We define the predictive learning setup deep nonlinear transformation from 2) to 2/2 through latent variables 2, as shown in Fig. 2. Formally, di2 = g=)), where / and g represent the encoder and decoder respectively and vi2 is the prediction of v2 given v1- The parameters of the encoder and decoder models are then trained using an objective function that tries to bring 1h2 "close to" U2. Simple examples of such an objective include the L1 or C2 loss functions. Note that these objectives assume independence between each pixel or element of 1/2 given V1, i.e., p(v2 v1) = IT,((22/11) thereby reducing their ability model correlations or complex structure. The predictive approach has been extensively used in representation learning, for example, colorization (Zhang al., 2016; 2017) and predicting sound from vision (Owens al., 2016).
f(u)) ) and
2.2 CONTRASTIVE LEARNING WITH TWo VIEWS
The idea behind contrastive learning is to learn an embedding that separates (contrasts) samples from two different distributions. Given a dataset of Vi and 1/2 that consists of a collection of samples consider contrasting congruent and incongruent pairs, i.e. samples from the joint distribution I ~ p(U1, v2) or x = [vi, ni).. which we call positives, versus samples from the product of marginals, y ~ p(u1 (p2)
{uf, ug], which we call negatives.
We learn a "critic" (-)-) that is high for positives and low for negatives. Similar to recent setups for contrastive learning (Oord et al., 2018; Gutmann & Hyvärinen, 2010; Mnih & Kavukcuoglu, 2013), we train this function to correctly select a single positive sample xr out of set {xr, y1, 1/2 y/s that contains A negative samples:
ha(
Leontrast
E log
he(z) + Di ha(y:)
To construct S, we simply fix one view and enumerate positives and negatives from the other view, allowing us to rewrite the objective as:
ho(i 2f === ho ho(tul, (5
L
Vi.V controst
E (v) m 1.....
log
where k is the number of negative samples v for given sample 2t. In practice, it can be extremely large, and so directly minimizing Eq. 2 is infeasible. In Section 2.4, we show an approximation based on Noise Contrastive Estimation (Gutmann & Hyvärinen, 2010) that allows for tractable computation.
3

---

Under review as a conference paper at ICLR 2020
Figure 3: Graphical models and infor- mation diagrams (inf) associated with the core view and full graph paradigms, for the of views, which gives total learning objectives.
num- bers within the regions show how much "weight" the total loss places each par- tition of information (i.e. how many of the objectives that partition contributes to). region with no number corre- sponds to 0 weight. For example, in the full graph case, the mutual information between all 4 views is considered in all 6 objectives, and hence marked with the number 6.
1/2
1/3
V2
V/
Vi
Vi
2
6
(a) Core View
(b) Full Graph
Implementing the critic We implement the critic ho(-) as a neural network. To extract compact latent representations of U1 and U2, we employ two encoders fo, (-) and fez with parameters 0 and 02 respectively. The latent representions are extracted as 21 Sa, (v1 ). z2 Sag (v2). On top of these features, the score computed the exponential of a bivariate function of 21 and 22. e.g., bilinear function parameterized by W12:
he( v2}} = eo (Wraooo (e2)og(es)
VVV.V: Loss contrastt in Eq. treats view V as anchor and enumerates over V2. Symmetrically, get contraast by anchoring V2. We add them up our two-view loss:
C(V,, V2)
V1.12 contrast +
V2.V1 contrast
(4)
After the contrastive learning phase, we use the representation 21, 22, or the concatenation of both, [21, 22], depending paradigm. This process visualized in Fig.
Connecting to mutual information The optimal critic hi is proportional to the density ratio between the joint distribution p(=1, 22) and the product of marginals p(21 (2)2)) (proof provided in Sec. C:
p(21, 22)
p (22)
h5(fe1, o2)) x
p(21)p(22) x
p(21)
(5)
This quantity is the pointwise mutual information, and its expectation, in Eq. 2, yields an estimator related mutual information. A formal proof given by Oord al. (2018); Poole et al. (2019), which we recapitulate in Section C, showing that:
T(2: 2z) 2 log(k) Lcontrast
(6)
where, as above, & the number of negative pairs in sample set S. Hence minimizing the objective maximizes the lower bound on the mutual information I(24; 22), which is bounded above by [(v;; By) by the data processing inequality. The dependency on l also suggests that using more negative samples lead to improved representation; show that this is indeed the case in Section A.1.2. We note that recent work (McAllester Statos, 2018) shows that the bound in Eq. can be very weak; and finding better estimators of mutual information an important open problem.
2.3 CONTRASTIVE LEARNING WITH MORE THAN TWO VIEWS
We present more general formulations of Eq. that can handle any number of views. We call them the "core view" and "full graph" paradigms, which offer different tradeoffs between efficiency and effectiveness. These formulations visualized in Fig. 3.
Suppose we have a collection of M views Vi...., VMr The "core view" formulation sets apart one view that want optimize over, say Vi, and builds pair-wise representations between Vi and each
4

---

Under review as a conference paper at ICLR 2020
other view V > by optimizing the sum of set of pair-wist objectives:
M
Lc cC= (V,
V,)
second, more general formulation the "full graph" where we consider all pairs (i. + j. and build (?) relationships in all. By involving all pairs, the objective function that we optimize is:
CF Σ C(V.
EVV.V Vy)
ISKJISM
Both these formulations have the effect that information is prioritized in proportion the number of views that share that information. This can be seen in the information diagrams visualized in Fig. 3. The number in each partition of the diagram indicates how many of the pairwise objectives, L(V, Vj), that partition contributes to. Under both the core view and full graph objectives, a factor, like "presence of dog", that common to all views will be preferred over factor that affects fewer views, such as "depth sensor noise".
The computational cost of the bivariate score function in the full graph formulation is combinatorial in the number of views. However, it is clear from Fig. 3 that this enables the full graph formulation to capture more information between different views, which may prove useful for downstream tasks. For example, the mutual information between 1/2 and V/3 or 1/2 and V44 is completely ignored in the core view paradigm (as shown by a count in the information diagram).
2.4 APPROXIMATING THE SOFTMAX DISTRIBUTION WITH
ESTIMATION
Better representations using CV in Eq. 2 are learnt by using many negative samples. However, computing the full softmax loss is prohibitively expensive for large N. We alleviate computational load by using Noise-Contrasti the full softmax in Eqn. 2, as has also been used in Mnih & Kavukcuoglu (2013)².
Estimation (NCE) (Gutmann & Hyvärinen, 2010) to approximate
Given an anchor of from Vi, the probablity that an atom U2 E {ugli j = 1, 2, ... N} from V/ is the best match of of. using the score he given by:
p(u2lo1) ho(tuf, v2})
Σ/=1 ho ho((ui,
where the normalization factor Z = En ho ((vi,, n²)) is expensive to compute for large N. Here we use he((r), v2}) = exp(fo, fo, (e1) Wiefoo, (o2)),(02)/t) where T modulates the distribution.
Crise-Contrastive Estimation (Gutmann & Hyvärinen, 2010) (NCE) is an effective way to estimate unnormalized statistical models. NCE fits density model p to data distributed as (unknown) distribution Pd, by using binary classifier to distinguish it from noise samples distributed Pa. To learn p(02 (.). we use binary classifier, which treats 2/2 as the data sample when given vf- The noise distribution P.. (-lf)) we choose here is a uniform distribution over all atoms from V/2 i... Pr (-lof) = 1/N. If we sample m noise samples to pair with each data sample, the posterior probability that given atom 12 comes from the data distribution is:
pa(vzlof) P(D 1/2 of)) Pa(ez/uf) + (v2le)))))
(10)
and we estimate this probability by replacing Pa(1s luf) with our unnormalized model distribution ho(uf, 02). Minimizing the negative pogpportrrior probability of correct labels D over data and noise samples yields our final objective, which is the NCE-based approximation of Eq. 2 (6 is the empirical data distribution):
LNCE
E of pfp)())) (a
E
[log(P(D 1/1g; + E
[lo = 0/ v } (11)
'Confusingly, the literature has previously referred to Eqn. 2 as "InfoNCE" (Oord et al., 2018). Our NCE approximation does not refer to the allusion to NCE in the name "InfoNCE". Rather we are here describing an NCB approximation the "InfoNCE" softmax objective.
5

---

Under review as a conference paper at ICLR 2020
Memory bank. Following Wu al. (2018), we maintain memory bank to store latent features for each training sample. Therefore, we can efficiently retrieve m noise samples from the memory bank to pair with each positive sample without recomputing their features. The memory bank is dynamically updated with features computed on the fly.
An alternative to the NCE based approximation above, is to simply do m++1 way softmax classification with m noise samples retrieved from the memory bank. We note that CPC (Oord et al., 2018) and Deep InfoMax (Hjelm al., 2019) use this + way softmax classification as their ultimate contrastive loss rather than the NCE-based contrastive loss in Eq. 11 (but note that CPC refers to the m + approximation as also "based on NCE"). Empirically we have found that the m 1-way softmax classification approach performed worse than our NCE-base ---daadd approximation, given the same number of noise samples.
3 EXPERIMENTS
We extensively evaluate Contrastive Multiview Coding (CMC) on a number of datasets and tasks. We evaluate on two established image representation learning benchmarks: Imagenet and STL-10 (See A.1. We further validate our framework on video representation learning tasks (See A.2). where we use image and optical flow modalities, as the two views that are jointly learned. The last set of experiments extends our CMC framework to more than two views and provides empirical evidence of it's effectiveness.
3.1 BENCHMARKING CMC ON IMAGENET
Following Zhang et al. (2016), we evaluate task generalization of the learned representation by training 1000-way linear classifiers on top of different layers.
Setup. Given a dataset of RGB images, we convert them to the Lab image color space, and split each image into I. and ab channels, as originally proposed in SplitBrain autoencoders (Zhang et al., 2017). During contrastive learning, I. and ab from the same image are treated as the positive pair, and ab channels from other randomly selected images are treated as a negative pair (for given L). Each split represents a view of the orginal image and is passed through a separate encoder. As in SplitBrain, we design these two encoders by evenly splitting given deep network, such as AlexNet (Krizhevsky et al., 2012), into sub-networks across the channel dimension. By concatenating representations layer-wise from these two encoders, achieve the final representation of an input image. As proposed by previous literature (Oord et al., 2018; Hjelm et al., 2019; Arora et al., 2019), the quality of such representation is evaluated by freezing the weights of encoder and training linear or non- non-linear classifiers on top of each layer.
To compare with other methods, adopt standard AlexNet and split it into two encoders. Because of splitting, each layer only connects to half of the neurons in the previous layer, and therefore the number of parameters in our model halves. We remove local response layer and add batch normalization to each layer. For the memory-based CMC model, we adopt ideas from Wu et al. (2018) for computing and storing memory. We retrieve 4096 negative pairs from the memory bank to contrast each positive pair. The training details are present in Sec. D.2.
Table 1 shows the results of comparing the CMC against other models, both predictive and contrastive. Our CMC is the best among all these methods; futhermore CMC tends to perform better at higher convolutional layers, similar to another contrasting- based model Inst-Dis (Wu al., 2018).
CMC with ResNets. We verify the scalability of CMC with larger networks such ResNets He et al. (2016). Here we do not split ResNets, rather we use ResNet-50, ResNet-101 or ResNet-50 x2 to encoder each of the two views (L and ab). The results are shown in Table 2, where ResNet-50, ResNet-101, and ResNet-50 x2 achieve 64.1%, 65.0%, and 68.4% top-1 accuracies, respectively. To our best knowledge, 68.4% on ImageNet is the highest published accuracy ever achieved by insperssied/usuuprvvisee methods (We note that a concurrent work AMDIM Bachman et al. (011) achieves similar results).

---

Under review as a conference paper at ICLR 2020
ImageNet Classification Accuracy
Method
convl
conv2
conv3
conv4
conv5
ImageNet-LLaelh
19.3
36.3
44.2
48.3
50.5
Random
11.6
17.1
16.9
16.3
14.1
Data-Ini Init (Krähenbühl et al., 2015)
17.5
23.0
24.5
23.2
20.6
Context (Doersch et al., 2015)
16.2
23.3
30.2
31.7
29.6
Colorization (Zhang et al., 2016)
13.1
24.8
31.0
32.6
31.8
Jigsaw (Noroozi & Favaro, 2016)
19.2
30.1
34.7
33.9
28.3
BiGAN (Donahue et al., 2017)
17.7
24.5
31.0
29.9
28.0
SplitBrain (Zhang et al., 2017)
17.7
29.3
35.4
35.2
32,8
Counting (Noroozi al., 2017)
18.0
30.6
34.3
32.5
25.7
Inst-Dis (Wu et al., 2018)
16.8
26.5
31.8
34.1
35,6
RotNet (Gidaris et al., 2018)
18.8
31.7
38.7
38.2
36.5
DeepCluster (Caron et al., 2018)
12.9
29.2
38.2
39.8
36.1
AET (Zhang al., 2019)
19.3
32.8
40.6
39.7
37.7
CMC
18.4
33.5
38.1
40.4
42.6
Table I: Top-1 classification accuracy 1000 classes of ImageNet Deng et al. (2009) with single crop. We compare CMC method with other unsupervised representation learning approaches by training 1000-way logistic regression classifiers on top of the feature maps of each layer, as proposed by Zhang al. (2016). Methods marked with only have half the number of parameters compared to others, because of splitting.
Accuracy (%) Top-1
ResNet-50
ResNet-101
ResNet-550
64.1
65.0
68.4
Table 2: Single crop top-1 classification accuracy ImageNet. We evaluate CMC with ResNet-50,, ResNet- 1011 ResNet-50 x2 encoder for each of two views (L and ab).
3.2 DOES REPRESENTATION QUALITY IMPROVE AS NUMBER OF VIEWS INCREASES?
We further extend our CMC learning framework multiview scenarios. We experiment the NYU-Depth-V2 (Nathan Silberman & Fergus, 2012) dataset which consists of 1449 labeled images. We focus more on understanding the behavior and effectiveness of CMC rather than competing with the current state-of-the-aris. The views we consider are: luminance (L channel), chrominance (ab channel), depth, surface normal (Eigen Fergus, 2015), and semantic labels.
Setup. To extract features from each view, and I fully connected layer. As the size the dataset is relatively small, we adopt the sub-patch based contrastive objective (see B) to increase the number of negative pairs. Patches with size of 128 x 128 are randomly cropped from the original images for contrastive learning (from images of size 480 x 640). For downstream tasks, we discard the fully connected layers and evaluate using the convolutional layers as representation.
neural network with 5 convolutional layers,
To measure the quality of the learned representation, we consider the task of predicting semantic labels from the representation of L. We follow the core view paradigm and use L are the core view, thus learning set of representations on L by contrasting different views with A UNet style architecture (Ronneberger et al., 2015) is utilized to perform the segmentation task. Contrastive training is performed on the above architecture that is equivalent of the UNet's encoder. After contrastive training is completed, we initialize the encoder weights of the UNet from the L encoder (which are equivalent architectures) and keep them frozen. Only the decoder trained during this finetuning stage.
Since we use the patch-based contrastive loss, in the view setting case, CMC coincides with DIM (Hjelm et al., 2019). The 2-4 view cases contrast L with ab, and then sequentially add depth and surface normals. The semantic labeling results are measured by mean loU over all classes and pixel accuracy, shown in Fig. 4. We that the performance steadily improves We have tested different orders of adding the views, and they all follow similar pattern.
views added.
We also compare CMC with two baselines. First, randomly initialize and freeze the encoder, and we call this the Random baseline; serves as a lower bound on the quality since the representation is just random projection. Rather than freezing the randomly initialized encoder, could train it
7

---

Under review as a conference paper at ICLR 2020
36
58
56
32
Pixel Accuracy
54
8
30
CMC
CMC
Supervised
52
Supervised
0
Random
50
Random
24
48
46
Number of Views
Number Views
Figure 4: We show the Intersection over Union (IoU) (left) and Pixel Accuracy (right) for the NYU-Depth-V dataset, CMC is trained with increasingly more views from to 4. As views added, both these metrics steadily increase. The views are (in order inclusion): L, ab, depth and surface normals.
Table 3: Results on the task predicting semantic labels from L channel representa- tion which is learnt using the patch-based contrastive loss and all 4 views. We compare CMC with Random and Supervised baselines, which serve as lower and upper bounds re- spectively. Th core-view paradigm refers to Fig. 3(a), and full-view Fig. 3(b).
Pixel
Accuracy (%)
mloU (%)
Random
45.5
21.4
CMC (core-view)
57.1
34.1
CMC (full-graph)
57.0
34.4
Supervised
57.8
35.9
jointly with the decoder. This end-to-end Supervised baseline serves as an upper bound. The results are presented in Table 3, which shows our CMC produces high quality feature maps even though it's unaware the downstream task.
3.3 PREDICTIVE LEARNING VS. CONTRASTIVE LEARNING
While experiments in section 3.1 show that contrastive learning outperforms predictive learn- ing (Zhang et al., 2017) in the context of Lab color space, it's unclear whether such an advantage is due to the natural inductive bias of the task itself. To further understand this, go beyond chrominance (ab), and try to answer this question when geometry or semantic labels are present.
We consider three view pairs the NYU-Depth dataset: (1) L and depth, (2) L and surface normals, and (3) and segmentation map. For each of them, we train two identical encoders for L., one using contrastive learning and the other with predictive learning. We then evaluate the representation quality by training a linear classifier on top of these encoders on the STL-10 dataset.
Accuracy on STL-10 (%)
Views
Predictive
Contrastive
Table 4: We compare predictive learning with contrastive learning by evaluating learned encoder on unseen dataset and task.
I,, Depth
55.5
58.3
Normal
58.4
60.1
The contrastive learning framework consis- tently outperforms predictive learning.
L. Seg. Map
57.7
59.2
Random
25.2
Supervised
65.1
The comparison results are shown in Table which shows that contrastive learning consistently outperforms predictive learning in this scenario where both the task and the dataset are unknown. We also include "random" and "supervised" baselines similar to that in previous sections. Though the unsupervised stage we only use 1.3K images from dataset much different from the target dataset STL-10, the object recognition accuracy close to the supervised method, which uses an end-to-end deep network directly trained STL-10.
Given two views V1 and V2 of the data, the predictive learning approach approximately models p(U2/01) Furthermore, losses used typically for predictive learning, such pixel-wise reconstruction losses usually impose an independence assumption on the modeling: p(2)/1)) x I1,((22 (1)). On the other hand, the contrastive learning approach by construction does not assume conditional
8

---

Under review as a conference paper at ICLR 2020
Classification Accuracy (%)
Y.10 Y.chdr
L-UV
Classification Accuracy
160
85
L.00
102
96
HED
51
KY
288
250
80
50
XY2
49
384
75
xz
HSV
48
80
90
100
110
2
Mutual Information Estimated by MINE (Belghazi 2018) (nat)
Mutual Information Estimated by MINE
(Belghazi 2018) (nat)
Figure 5: How does mutual information between views relate representation quality? (Left) Classification accuracy against estimated between channels of different color spaces; (Right) Classification accuracy vs estimated MI between patches at different distances (distance in pixels is denoted next to each data point). MI estimated using MINE (Belghazi et al., 2018).
independence across dimensions of 02. In addition, the use of random jittering and cropping between views allows the contrastive learning approach benefit from spatial co-occurrence (contrasting in space) in addition to contrasting across views. We conjecture that these are two reasons for the superior performance of contrastive learning approaches over predictive learning.
3.4 How DOES MUTUAL INFORMATION AFFECT REPRESENTATION QUALITY?
Given fixed set of views, CMC aims maximize the mutual information between representations of these views. We have found that maximizing information in this way indeed results in strong representations, but it would be incorrect to infer that information maximization (infomax) is the key to good representation learning. In fact, this paper argues for precisely the opposite idea: that cross-view representation learning effective because it results in kind of information minimization, discarding nuisance factors that are not shared between the views.
The resolution to this apparent dilemma that we want maximize the "good" information the signal - in our representations, while minimizing the "bad" information the noise. The idea behind CMC that this can be achieved by doing infomax learning on two views that share signal but have independent noise. This suggests "Goldilocks principle": good collection of views is one that shares some information but not too much. Here we test this hypothesis on two domains: learning representations on images with different colorspaces forming the two views; and learning representations on pairs of patches extracted from an image, separated by varying spatial distance.
In patch experiments randomly crop two RGB patches of size 64x64 from the same image two views. Their relative position fixed. Namely, the two patches always starts at position (z, y) and (xr + d. y d) with (ar, y) being randomly sampled. While varying the distance d, start from 64 to avoid overlapping. There is a possible bias that with an image of relatively small size (e.g., 512x512), large d (e.g.. 384) will always push these two patches around boundary. To minimize this bias, we use high resolution images (e.g. 2k) from DIV2K (Agustsson & Timofte, 2017) dataset.
Fig. 5 shows the results of these experiments. The left plot shows the result of learning representations on different colorspaces (splitting each colorspace into two views, such as (L. ab), (R, GB) etc). We then use the MINE estimator Belghazi et (2018) estimate the mutual information between the views. We measure representation quality by training a linear classifier on the learned representations on the STL-10 dataset Coates al. (2011). The plots clearly show that using colorspaces with minimal mutual information give the best downstream accuracy (For the outlier HSV in this plot, HED
we conjecture the representation quality is harmed by the periodicity of H. Note that the not periodic.). On the other hand, the story is more nuanced for representations learned between patches at different offsets from each other (Fig. 5, right). Here we see that views with too little or too much MI perform worse; sweet spot in the middle exists which gives the best representation. That there exists such a sweet spot should be expected. If two views share no information, then, in principle, there is incentive for CMC to learn anything. If two views share all their information,

---

Under review as a conference paper at ICLR 2020
nuisances discarded and arrive back at something akin to an autoencoder or generative model, that simply tries to represent all the bits in the multiview data.
These experiments demonstrate that the relationship between mutual information and representation quality is meaningful but not direct. Selecting optimal views, which just share relevant signal, may be a fruitful direction for future research.
RELATED WORK
Unsupervised representation learning is about learning transformations of the data that make sub- sequent problem solving easier (Bengio al., 2013). This field has long history, starting with classical methods with well established algorithms, such as principal components analysis (PCA (Jolliffe, 2)1 and independent components analysis (ICA (Hyvärinen et al., 2004)). These methods tend to learn representations that focus low-level variations in the data, which are not very useful from the perspective of downstream tasks such as object recognition.
Representations better suited to such tasks have been learnt using deep neural networks, starting with seminal techniques such as Boltzmann machines (Smolensky, 1986; Salakhutdinov & Hinton, 2009), autoencoders (Hinton & Salakhutdinov, 2006), variational autoencoders (Kingma & Welling, 2013), generative adversarial networks (Goodfellow et al., 2014) and autoregressive models (Oord al., 20. Numerous other works exist, for review see (Bengio et al., 2013). A powerful family of models for unsupervised representations are collected under the umbrella of "self-supperised" learning (Sa, 2004; Zhang et al., 2017; 2016; Isola et al., 2015; Wang & Gupta, 2015; Pathak et al., 2016; Zhang al., 2019). In these models, an input to the model is transformed into an output X, which is supposed to be close to another signal Y. which itself is related to X in some meaningful way. Examples of such X/Y pairs are: luminance and chrominance color channels of an image (Zhang et al., 2017), patches from a single image (Oord et al., 2018), modalities such as vision and sound (Owens et al., 2016) or the frames of a video (Wang & Gupta, 2015). Clearly, such examples numerous in the world, and provides us with nearly infinite amounts of training data: this is one of the appeals of this paradigm. Time contrastive networks (Sermanet et al., 2017) use a triplet loss framework to learn representations from aligned video sequences of the same scene, taken by different video cameras. Closely related to sulf-vuuee learning is the idea of multi-view learning, which is general term involving many different approaches such co-training (Blum & Mitchell, 1998), multi-kernel learning (Cortes et al., 2009) and metric learning (Bellet et al., 2012; Zhuang et al., 2019); for comprehensive surveys please see (Xu et al., 2013; Li et al., 2018). Nearly all existing works have dealt with one or two views such as video or image/sound. However, in many situations, many more views are available to provide training signals for any representation.
The objective functions used to train deep learning based representations in many of the above methods are either reconstruction- baaed loss functions such as Euclidean losses in different norms e.g. (Isola et al., 2017), adversarial loss functions (Goodfellow et al., 2014) that learn the loss in addition to the representation, or contrastive losses e.g. (Gutmann & Hyvärinen, 2010; Hjelm et al., 2019; Oord et al., 2018; Arora et al., 2019; Hénaff et al., 2019) that take advantage of the co-occurence of multiple views. Another recently introduced novel objective function is instance discrimination (Wu et al., 2018). In this work, we compare the two most commonly used objectives: predictive and contrastive. The prior works most similar to our own (and inspirational to us) are Contrastive Predictive Coding (CPC) (Oord et al., 2018) and Deep InfoMax (Hénaff et al., 2019). These two methods, like ours, learn representations by contrasting between congruent and incongruent representations of scene, and are motivated as forms of infomax learning. CPC learns from two views the past and future - and is applicable to sequential data. Deep Infomax (Hjelm et al., 2019) considers the two views to be the input to a neural network and its output. These two methods share the same mathematical objective, but differ in the definition of the views. Our technical method is also highly related, but differs in the following ways: we extend the objective to the case of more than two views; and we use loss function which more closely follows the original method of noise contrastive estimation (Gutmann & Hyvärinen, 2010) (See details in Section 2.4). Although CPC, Deep InfoMax, and the present paper are all very similar at the mathematical level, they each explore a different set of view definitions, architectures, and application settings, and each contributes its own unique empirical investigation of this paradigm of representation learning.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Information Diagram- Wikipedia.
diagram. 4
Eirikur Agustsson and Radu Timofte. Nire 2017 challenge single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 135, 2017.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv: 2019. 2, 6, 10, 14
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint (.Ki:09900000100 2019. 2, 6
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062, 2018. 9
Aurélien Bellet, Amaury Habrard, and Marc Sebban. Similarity learning for provably accurate sparse linear classification. arXiv preprint urXiv:1206.6660 6476, 2012.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1822 2013. 10
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceed- ings of the eleventh annual conference Computational learning theory, pp. 92-100. ACM, 1998. 10
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In Proceedings of the 34th International Conference on Machine Learning-Van Volume 70, pp. 517-526. JMLR. org, 2017. 16
Uta Buchler, Biagio Brattoli, and Bjorn Ommer. Improving spatiotemporal self- supervuperviioon by deep reinforcement learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 770-786, 2018. 11
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), pp. 132-149, 2018. 7
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223, 2011. 15
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Learning non-linear combinations of kernels. In Advances in neural information processing systems, pp. 396 404, 2009. 10
Hanneke EM Den Ouden, Peter Kok, and Floris P De Lange. How prediction errors shape perception, attention, and motivation. Frontiers psychology, 3:548, 2012.
Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009. 7, 15
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1422-1430, 2015.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. International Conference Learning Representations, 2017.
16
11

---

Under review as a conference paper at ICLR 2020
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with common multi-scal convolutional architecture. In Proceedings of the IEEE international conference on computer vision, 2015. 7
Chuang Gan, Boqing Gong, Kun Liu, Hao Su, and Leonidas Guibas. Geometry guided convolutional neural networks for self-supervised video representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5589-5597, 2018.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations, arXiv preprint arXiv: 1803.07728, 2018. 7
Melvyn A Goodale and A David Milner. Separate visual pathways for perception and action. Trends in neurosciences, 1992. 16
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems, pp. 2672-2680, 2014. 1, 10
Michael Gutmann and Aapo Hyvärinen.
estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010. 3, 5, 10
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 770-778, 2016.
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint S0ti::/905.092111 2019, 10
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504--555 2006. 10
Devon Hjelm, Alex Fedorov, Samuel Mavoie-Marccildon. Karan Grewal, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019, 6, 7, 10, 14, 15, 16, 17, 18, 19
Jakob Hohwy. The predictive mind. Oxford University Press, 2013.
Aapo Hyvärinen, Juha Karhunen, and Erkki Oja. Independent component analysis, volume 46. John Wiley & Sons, 2004. 10
Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H Adelson. Learning visual groups from co-occurrences in space and time. arXiv preprint
2015. 10
Phillip Isola, Lun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017. 10
Ian Jolliffe. Principal component analysis. Springer, 2011.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.000 14, 2013. 1,
arXiv preprint
Philipp Krähenbühl, Carl Doersch, Jeff Donahue, and Trevor Darrell. Data-dependent initializations of convolutional neural networks. arXiv preprint arXiv:1511.06855 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012. 6, 14, 16, 20
Hsin- Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Hsuan Yang. Unsupervised representation learning by sorting sequences. In Proceedings of the IEEE International Conference on Computer Vision, pp. 667-676. 2017. 17, 21
12

---

Under review as a conference paper at ICLR 2020
Yingming Li, Ming Yang, and Zhongfei Mark Zhang. A survey of multi-view representation learning. IEEE Transactions on Knowledge and Data Engineering, 2018. 2, 10
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. arXiv
preprint arXiv: 00397,00077.. 2019. 20
Zelun Luo, Boya Peng, De-An Huang, Alexandre Alahi, and Li Fei-Fei. Unsupervised learning of long-term motion dynamics for videos. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2203-2212, 2017. 17
David McAllester and Karl Statos. Formal limitations on the measurement of mutual information. arXiv preprint arXiv:1811.04255 2018. 4
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European Conference Computer Vision, pp. 527-544. Springer, 2016. 17, 21
Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in neural information processing systems, pp. 2265-2273, 2013. 3,5
Hossein Mobahi, Ronan Collobert, and Jason Weston. Deep learning from temporal coherence in video. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 737-744. ACM, 2009. 17
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. ECCV, 2012. 2,7
Mehdi Noroozi and Paolo Favaro. Unsupervised learning visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016. 7
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5898-5906, 2017. 7
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint 1601 06759,, 2016. 10
Aaron van den Oord, Yazhe L.,, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 10..xvv1807007777 2018. 4, 5, 6, 10, 14, 15, 16, 18, 19
Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William Freeman. Visually indicated sounds. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2405-2413, 2016. 3, 10
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016. 10
Ben Poole. Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On variational bounds of mutual information. arXiv preprint 1905 06692,, 2019. 4
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015. 7
Virginia Sa. Sensory modality segregation. In Advances in neural information processing systems, pp. 913-920, 2004.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Artificial intelligence and statistics, pp. 448-455, 2009. 1,
Nawid Sayed, Biagio Brattoli, and Bjöm Ommer. Cross and learn: Cross-modal self-supervision. arXiv preprint arXiv:1811.03879, 2018. 17, 21
Gerald E Schneider. Two visual systems. Science, 1969. 16
13

---

Under review as a conference paper at ICLR 2020
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang. Stefan Schaal, and Sergey Levine. cone-contrastive networks: Self-supervised learning from pixels. 2017. 10
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems, pp. 0085568-55 (68-576, 2014. 16
Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial life, 11(1-2):13-29, 2005. 1
Paul Smolensky. Information processing in dynamical systems: Foundations of harmony theory. Technical report, Colorado Univ at Boulder Dept of Computer Science, 1986. 10
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: /2/.0402 2012. 16
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating vidcos with scene dynamics. In Advances In Neural Information Processing Systems, pp. 613-621, 2016. 17
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2794-2802, 2015. 10
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018. 6, 7, 10, 19
Chang Xu, Dacheng Tao, and Chao Xu. A survey multi-view learning. arXiv preprint
arXiv:1304.534, 2013. 2. 10
Christopher Zach, Thomas Pock, and Horst Bischof. A duality based approach for realtime tv-1 1 optical flow. In Joint pattern recognition symposium, pp. 214-223. Springer, 2007. 16
Liheng Zhang, Guo-Jun Qi, Liqiang Wang, and Jiebo Luo. Aet vs, aed: Unsupervised representation learning by auto-encoding transformations rather than data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2547-2555, 2019. 7, 10
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pp. 649-666. Springer, 2016. 3, 6, 7, 10
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1058-1067, 2017. 3. 6, 7, 8, 10, 14, 15, 16
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. arXiv preprint arXiv: 2019. 10
A ADDITIONAL EXPERIMENTS
A.1 CMC ON IMAGES
Given dataset of RGB images, we convert them to the Lab image color space, and split each image into L and ab channels, as originally proposed in SplitBrain autoencoders (Zhang et al., 2017). During contrastive learning. L and ab from the same image are treated as the positive pair, and ab channels from other randomly selected images are treated as a negative pair (for given L). Each split represents a view of the orginal image and is passed through a seprate encoder. This corresponds to the "full graph" model of Eq. 8 with I. and ab channels as the two views. As in SplitBrain, we design these two encoders by evenly splitting a given deep network, such as AlexNet (Krizhevsky et al., 2012), into sub-networks across the channel dimension. By concatenating representations layer-wise from these two encoders, we achieve the final representation of an input image. As proposed by previous literature (Oord al., 2018; Hjelm al., 2019; Arora et al., 2019), the quality of such a representation evaluated by freezing the weights of encoder and training linear or non con-linear classifiers top of each layer.
14

---

Under review as a conference paper at ICLR 2020
A.1.1
S0L-10
STL-10 (Coates et al., 2011) is an image recognition dataset designed for developing unsupervised or self-supervised learning algorithms. It consists of 100000 unlabeled training 96 96 RGB image samples and 500 labeled samples for each of the 10 classes
Setup. We adopt the same data augmentation strategy and network architecture as those in DIM (Hjelm al., 2019). A variant of AlexNet takes as input 64 x 64 images, which are ran- domly cropped and horizontally flipped from the original 96 x 96 size images. For a fair comparison with DIM. we also train model in a patch- based contrastive fashion during unsupervised pre- training. With the weights of the pre-trained encoder frozen, a two-layer fully connected network with 200 hidden units is trained top of different layers for 100 epochs to perform 10-way classification. We also investigated the strided crop strategy of CPC (Oord et al., 2018). 8). Fixed sized overlapping patches of size 16 16 with overlap of pixels are cropped and fed into the network separately. This ensures that features of one patch contain minimal information from neighbouring patches; and increases the available number of negative pairs for the contrastive loss. Additionally, we include NCE-based contrastive training and linear classifier evaluation.
Comparison. We compare CMC with the state of the art unsupervised methods in Table 5. Three columns are shown: the conv5 and fc7 columns use respectively these layers of AlexNet as the encoder (again remembering that we split across channels for L and ab views). For these two columns compare against the all methods except CPC, since CPC does not report these numbers in their paper (Hjelm et al., 2019) 9). In the Strided Crop setup, we only compare against the approaches that contrastive learning, DIM and CPC, since this method was only used by those works. We note that in Table 5 for all the methods except SplitBrain, we report numbers are shown in the original paper. For SplitBrain, we reimplemented their model faithfully and report numbers based on our reimplementation (we verified the accuracy of our SplitBrain code by the fact that we get very similar results with our reimpementation as in the original paper (Zhang et al., 2017) for ImageNet experiments, see below).
The family of contrastive learning methods. such as DIM, CPC, and CMC, achieve higher classifica- tion accuracy than other methods such as SplitBrain that use predictive learning: or BiGAN that use adversarial learning. CMC significantly outperforms DIM and CPC in all cases. We hypothesize that this outperformance results from the modeling of cross-view mutual information, where view-specific noisy details are discarded. Another head-to-head comparison happens between CMC and SplitBrain, both of which modeling images as seprated L and ab streams; we achieve a nearly 8% absolute improvement for conv5 and 17% improvement for fc5. Finally, we notice that the predictive learning methods suffer from a big drop in performance when the encoding layer is switched from conv5 to fc7. On the other hand, the contrastive learning approaches are much more stable across layers, suggesting that the mutual information maximization paradigm learns more semantically meaningful representations shared by the different views. From a practical perspective, this is a significant advantage as the selection of specific layers should ideally not change downstream performance by too much.
In this experiments we used AlexNet as backbone. Switching to more powerful networks such as ResNets is likely to further improve the representation quality.
A.1.2 IMAGENET
ImageNet (Deng et al., 2009) consists of 1000 image classes and is frequently considered as a testbed for unsupervised representation learning algorithms.
Effect of the number of negative samples. We investigate the relationship between the number of negative pairs m in NCE-based loss and the downstream classification accuracy on a randomly chosen subset of 100 classes of Imagenet (the same set of classes is used for any number of negative pairs). We train a 100-way linear classifier using CMC pre-trained features with varying number of negative pairs, starting from 64 pairs upto 8192 (in multiples of 2). Fig. 6 shows that the accuracy of the resulting classifier steadily increases but saturates at around 60.3% with m = 4096 samples. AlexNet is used in this study.
15

---

Under review as a conference paper at ICLR 2020
Method
classifier
conv5
tc7
Strided Crop
62.19
55.78
NAT (Bojanowski & Joulin, 2017)
64.32
61,43
MLP
BiGAN (Donahue et al., 2017)
71.53
67.18
SplitBrain' (Zhang al., 2017)
72.35
63.15
DIM (Hjelm et al., 2019)
72.57
70.00
78.21
MLP
CPC (Oord et al., 2018)
77.81
CMC (Patch)
Linear
76.65
79.25
82.58
CMC (Patch)
MLP
80.14
80.11
83.43
CMC (NCE)
Lincar
83.28
86.66
CMC (NCE)
MLP
84.64
86.88
Supervised
68.70
Table 5: Classification accuracies STL-10 using layer MLP classifier for evaluating the representations learned by small AlexNet. For all methods we compare against, include the numhers that are reported the DIM (Hjelm et al., 2019) paper, except for SplitBrain, which is our reimplementation. Methods marked with
have half the number of parameters because of splitting.
Accuracy on ImageNer-100 (%)
60
CMC
59
58
57
56-
55
54
53
128
256
512
1024
2048
4096
8192
Negatives
Figure 6: We plot the number of negative examples in NCE-hased contrastive loss against the accuracy for 100 randomly chosen classes of Imagenet 100. seen the accuracy steadily increases with m.
A.2 CMC ON VIDEOS
We apply CMC videos by drawing insight from the two-streams hypothesis (Schneider, 1969; Goodale & Milner, 1992), which posits that human visual cortex consists of two distinct processing streams: the ventral stream, which performs object recognition, and the dorsal stream, which processes motion. In our formulation, given an image i, that is frame centered at time f,, the ventral stream associates it with neighbouring frame ic+k, while the dorsal stream connects centered at t. Therefore, we extract i,, 4++ and ft from two modalities as three views of a video; for optical flow we use the TV-L1 algorithm (Zach al., 2007). Two separate contrastive learning objectives are built within the ventral stream (ig, de+k) and within the dorsal stream (ie, ft). For the ventral stream, the negative sample for i, is chosen as a random frame from another randomly chosen video; for the dorsal stream, the negative sample for i, is chosen as the flow corresponding to a random frame in another randomly chosen video.
optical flow fr
Pre-training. We train CMC UCF101 (Soomro et al., 2012) and two CaffeNets (Krizhevsky et al., 2012) for extracting features from images and optical flows, respectively. In our implementation, ft represents 10 continuous flow frames centered at t. We batch size of 128 and contrast each positive pair with 127 negative pairs. CMC is trained with Adam for 300 epochs, with an initial learning rate of 0.001 which decayed by factor of after 200 and 250 epochs.
Action recognition. We apply the learn representation to the task of action recognition. The spatial network from (Simonyan & Zisserman, 2014) is a well-established paradigm for evaluating pre-trained RGB network action recognition task. We follow the same spirit and evaluate the transferability of our RGB CaffeNet on UCF101 and HMDB51 datasets. We initialize the action recognition CaffeNet up to conv5 using the weights from the pre-trained RGB CaffeNet. The averaged accuracy three splits is present in Table 6. Unifying both ventral and dorsal streams during pre- pre-training produces higher accuracy for downstream recognition than using only single stream. Increasing the number
16

---

Under review as a conference paper at ICLR 2020
Method
# of Views
UCF-101
HMDB-51
Random
48.2
19.5
ImageNet
67.7
28.0
VGAN (Vondrick al., 2016)
52.1
LT-Motion* (Luo et al., 2017)
53.0
TompCoh (Mobahi et al., 2009)
45.4
15.9
Shuffle and Learn (Misra et al., 2016)
50.2
18.1
Geometry (Gan et al., 2018)
55.1
23.3
OPN (Lee et al., 2017)
56.3
22.1
ST Order (Buchler et al., 2018)
58.6
25.0
Cross and Learn (Sayed et al., 2018)
2
58.7
27.2
CMC (V)
55.3
CMC (D)
57.1
CMC (V+D)
59.1
26.7
Table 6: Test accuracy (%) on UCF-101 which evaluates task transferability and on HMDB-51 which evaluates kask and dataser transferability. Most methods either use single RGB view or additional optical flow view, while VGAN explores sound the second view. iindicates different network architecture.
Metric (%)
L
ab
Depth
Normal
mloU
21.4
15.6
30.1
29.5
Random
pix. acc.
45.5
37.7
51.1
50.5
mloU
34.4
26.1
39.2
37.8
CMC
pix. acc.
57.0
49.6
59.4
57.8
mloU
35.9
29.6
41.0
41.5
Supervised
pix. acc.
57.8
52.6
59.1
59.6
Table 7: Performance on the task of using single view to predict the semantic labels. where can be L, ab, depth surface normal. Our CMC framework improves the quality of unsupervised representations towards that of supervised ones, for all views investigated. This uses the full-graph paradigm Fig. ??(b).
of views of the data from 2 to 3 (using both streams instead of one) provides boost for UCF-101. Furthermore, on UCF-101, we outperform all other methods; and on HMDB-51, -1, CMC is second-best in performance.
A.3 IS CMC IMPROVING ALL VIEWS?
A desirable unsupervised representation learning algorithm operating on multiple views or modalities should improve the quality of representations for all views. We therefore investigate our CMC framwork beyond channel. To treat all views fairly, we train these encoders following the full graph paradigm, where each view is contrasted with all other views.
We evaluate the representation of each view by predicting the semantic labels from only the representation of e, where is L, ab, depth or surface normals. This uses the full-graph paradigm. As in the previous section, compare CMC with Random and Supervised baselines. As shown in Table 7, the performance of the representations learned by CMC using full-graph significantly outperforms that of randomly projected representations, and approaches the performance of the fully supervised representations. Furthermore, the full- -raph representation provides a good representation learnt for all views, showing the importance of capturing different types of mutual information across views.
B CONTRASTING SUB-PATCHES
Instead of contrasting features from the last layer, patch-based method (Hjelm et al., 2019) contrasts feature from the last layer with features from previous layers, hence increasing the number of negative pairs. For instance, use features from the last layer of fe, to contrast with feature points from feature maps produced by the first several conv layers of for- This is equivalent to contrast between global patch from view with local patches from the other view. In this fashion, directly
17

---

Under review as a conference paper at ICLR 2020
perform m 1 way softmax classification, the same (Oord al., 2018; Hjelm al., 2019) for fair comparison in Sec. A.1.1.
Such patch-based contrastive loss is computed within each mini-batch and does not require memory bank. Therefore, deploying it in parallel training schemes is easy and flexible. However, patch- contrastive loss usually yields suboptimal results compared NCE-based contrastive loss, according to our experiments.
C PROOFS
We prove that: (a) the optimal score function h5((11, 22]) is proportional to density ratio between the joint distribution p(v1, 122) and product of marginals p(v) (p(22) as shown in Eq. 5: (b) Minimizing the contrastive loss Leontrast maxmizes lower bound on the mutual information between two views, as shown in Eq.
We will use the most general formula of contrastive loss Leontrast shown in Eq. 1 for our derivation. But we note that replacing Leontrast with L V2.VV
contrast is straightforward, The overall proof follows similar derivation introduced in (Oord et al., 2018).
C.I SCORE FUNCTION AS DENSITY RATIO ESTIMATOR
We first show that the optimal score function h b2 that minimizes Eq. 1 is proportional to the density ratio between joint distribution and product of marginals, shown as Eq. 5. For notation convenience, we denote p(U1 02) as data distribution pa(-) and p(v))p(22) as noise distribution Pa (-). The loss in Eq. is indeed cross-entropy loss of classifying the correct positive pair out from the given set S. Without loss of generality, we assume the first pair (uf. vg) in S is positive or congruent and all others (wi, i i = 1, 1,2, 2..,... are negative or incongruent. The optimal probability for the loss, p(pos 0) S),, should depict the fact that (ef, v comes from the data distribution pa(-) while all other pairs come from the noise distribution pa (-). Therefore,
p(pos 0 = pa(lf, 02) 11, Pr (of, vb)
Σx Pa vg)) 1is Pn (V)
p(re (. Il. p (p(1))
Σ p Il p (w1))(1)))
plop
pten)p(es
plu Σラ PODD
where we plug the definition of pa(-) and pm (.(.), and divide H (p(()) for both the numerator and denominator. By comparing above equation with the loss function in Eq. 1, we can see that
the optimal score function (1 v2)) is proportional to the density ratio p() derivation is agnostic which layer the score function starts from, e.g., p= The above
be defined either
the raw input (V1, U2) or the latent representation (21, 22). As we care more about the property of the latent representation, for the following derivation we will use hip ((21, #2)), which proportional (() to ((2))
18

---

Under review as a conference paper at ICLR 2020
C.2 MAXIMIZING LOWER BOUND ON MI
Now we substitute the score function in Eq. 1 with the above density ratio, and the optimal loss objective Coontrost becomes:
Cope
hi, (13),
Copt contrast
-E -EE log
Σ i=0 WW (fz 22))
p(=5.=9)
EE log
p(aj.
1=0 p(=1)p(23)
+ p(21)
& p(zf, 24)
IE log 8
p(z9, =2)
p(z))p(z{)
p(=1)p(=9)
p(21122)
IE log g
k LE p(27 =
p(z1)
p(=1)p(22)
20
E log
1
k
5
p(2) 2
p(2 22 p(2 pp(p))(())
log(A)
E log
p 22)
log(k)
E (x1 (opp) (-)
log
( (p(2)
log(k) [221 22)
Therefore, for any two views V, and Vj. we have [(z;; 2 log(k) controst Vi V2). As the k increases, the approximation step becomes more accurate. Given any k. minimizing (V, Vj) maximizes the lower bound on the mutual information [(2;; 2y). ). We should note that increasing k to infinity does not always lead higher lower bound. While log(k) increases with larger k the optimization problem becomes harder and LA (V, Vς) also increases.
Copt
D IMPLEMENTATION DETAILS
D.1 STL-10
For fair comparison with DIM (Hjelm et al., 2019) and CPC (Oord al., 2018), adopt the same architecture as that used DIM and split it into two encoders, each shown as in Table For the implementation of the score function, we adopt similar "enconer-annd-dot-proodd strategy, which is tantamount to a bilinear model.
Adam optimizer with initial learning rate of 0.001, 3 = 0.5, 3g 0.999. We train for a total of 200 epochs with learning rate decayed by 0.2 after 120 and 160 epochs. In the non-linear classifier evaluation stage, setting. For the NCE-based contrastive learning stage, we train for 320 epochs with the learning rate initialized as 0.03 and further decayed by 10 for every 40 epochs after the first 200 epochs. The temperature is set as 0.1. In general, E [0.05, 0.2) works reasonably well.
In the patch- contrastive learning stage,
the same optimizer
D.2 IMAGENET
For patch-based contrastive loss, the same optimizer setting in Sec. D.I except that the learning rate initialized as 0.01.
For NCE-basd contrastive loss in both full ImageNet and ImageNet100 experiments present in Sec. A.1.2, the encoder architecture used for either L or ab channels shown in Table 9. In the unsupervised learning stage of AlexNet,
use SGD train the network for total of 400 epochs. The temperature is set as 0.07 by following previous work (Wu et al., 2018). The learning rate is initialized 0.03 with decay of 10 for every 50 epochs after the first 250 epochs. Weight decay is
19

---

Under review as a conference paper at ICLR 2020
Half of AlexNe((Krizhevvsy et al., 2012) for STL-10
Layer
data
64
convl
64
48
1
pooll
31
48
3
2
0
conv2
31
96
pool2
15
96
0
conv3
15
192
3
conv4
15
192
conv5
15
96
3
1
pool5
fc6
2048
7
0
fc7 2048
1
0
fc8
64
1
1
0
Table 8: The variant of AlexNet architecture used in our CMC for STL-10 (only half present here due to splitting). X spatial resolution of layer. C number of channels layer; K conv or pool kernel size; S computation stride; P padding; channel size is dependent on the input source, e.g. for I. channel and 2 for ab channel.
set as 10 and momentum is kept as 0.9. For the linear classification stage, we train for 160 epochs. The learning rate initialized as 0.1 and decayed by 0.2 every 20 epochs after the first 100 epochs. We set weight decay 0 and momentum 0.9.
For ResNets in CMC stage, there are three differences. First, we use larger learning rate, that is, set base learning rate of 0.03 for every 128 images and then roughly scale it up with the batch size. Specifically, we train: (1) ResNet-50 with bsz 280 and Ir = 0.08; (2) ResNet--10 with only train for 280 epochs with learning rate decayed at 160, 200, and 240 epochs. Third, we used Fast Autoaugment (Lim al., 2019) data augmentation. In the linear evaluation stage, we train for 100 epochs. The learning rate is initialized as 30 for ResNet-50 and 10sNet-101, and 50 for ResNet-50 x2. It is decayed by 0.2 every 15 epochs after the first 60 epochs. We set weight decay as 0 and momentum as 0.9.
bsz 200 and ir 0.05; (3) ResNet-50 x2 with bsz 156 and Ir 0.04. Second,
Half of AlexNe((Krizheessky et al., 2012) for ImageNet
Layer
X
C
K
S
P
data
224
convl
55
48
11
2
pool1
27
48
3
0
conv2
27
128
5
2
pool2
13
128
3
0
conv3
13
192
3
conv4
13
192
3
conv5
13
128
3
pool5
6
128
3
0
fe6
1
2048
fc7
1
2048
fe8
128
Table 9: AlexNet architecture used in CMC for ImageNet (only half is present here due to splitting). X spatial resolution of layer, C number of channels in layer; K conv or pool kernel size; S computation stride; P padding; * channel size is dependent on the input source, e.g. for channel and for ab channel.
20

---

Under review as a conference paper at ICLR 2020
D.3 UCF101 AND HMDB51
Following previous work (Misra et al., 2016; Lee et al., 2017; Sayed et al., 2018; Buchler et al., 2018), we use CaffeNet for the video experiments. We tailor the network and use features from the fc6 layer for contrastive learning. Dropout of 0.5 is used to alleviate overfitting.
D.4 NYU DEPTH-VE V2
While experimenting with different views on NYU Depth-V V2 dataset, we encode the features from patches with a size of 128 x 128. The detailed architecture is shown in Table 10. In the unsupervised
training stage, we use Adam optimizer with an initial learning rate of 0.001, 31 = 0.5, B2 = 0.999. We train for a total of 3000 epochs with learning rate decayed by 0.2 after 2000, 2400, and 2800 epochs. For the downstream semantic segmentation task, we use the same optimizer setting but train for fewer epochs. We only train 200 epochs for CMC mre-trained models, and train 1000 epochs for the Random and Supervised baselines until convergence. For the classification task evaluated on STL-10, 10, we use the same optimizer setting in Sec. D.I report numbers in Table 2.
Encoder Architecture on NYU
Layer
X
C
P
data
128
convl
64
64
3
pooll
32
64
2
2
0
conv2
16
128
1
conv3
8
256
1
conv4
8
256
3
conv5
4
512
2
fc6
1
512
4
0
fc7
256
0
Table 10: Encoder architecture used in our CMC for playing with different views NYU V2. X spatial resolution of layer, C number of channels layer; K conv or pool kernel size; S computation stride; P padding; channel size dependent the input source, e.g. 1 for I, for ab, for depth, 3 for surface normal, and for segmentation map.
21


Reviews:
[{"review_id": "r1lX1jDjtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presented a multi-view learning method that is based on negative sampling in contrastive learning. The core idea is to set an anchor view and the sample positive and negative data points from the other view and maximise the agreement between positive pairs in learning from two views. When more than two views are presented, the learning objective is a sum over all possible combinations of two views. The performance of the proposed model is good, and the ablation study is interesting. \n\nComments:\n\n1. The core concept, or at least one of the core concepts, in multi-view learning is the conditional independence.\n\nNormally, the underlying assumption in multi-view learning is that, given the class label, the samples from multiple views are conditionally independent from each other. Therefore, the goal is to learn distinctive representations from different data sources/disjoint populations, so then after learning, the ensemble of them is able to capture a set of diverse aspects of the data. A \"side-effect\" of learning from multiple views is that individual views indeed get improved by learning from others. Meanwhile, self-supervised learning is the case when the input data to the designed learning system is also the target of the system. \n\nThe paper presented an idea for self-supervised learning from multiple views, which is not exactly the same, but still in the same regime. This concept could be used to explain some empirical findings in this paper. Since it is expected, there is even no need in conducting experiments. \n\n\n\n\n2. My main concern of this paper is the novelty, however, the empirical results are strong.\n\nThe paper mainly presented a simple yet effective method for self-supervised learning from two views, and the generalisation is a sum over all possible combinations of two views. The method itself has already been proposed many years ago as mentioned in the related work section in the paper, and the generalisation was also described in prior work, which makes me doubt the novelty of the paper. \n\nThe earliest work to the best of my knowledge is [1], and later on there are a couple workshops [2,3] on multi-view learning which largely settled the field of learning from multiple views from neural networks', kernels', and bayesian perspectives. Many things mentioned in this paper have already been discovered at that time. \n\n3. The theoretical justification is not as strong as the generalised CCA.\n\nCCA has been applied in the field of multi-view learning and self-supervised learning for long, and it was initially proposed for comparing the correlation between two sets of samples of two random variables. A successful generalisation is the generalised CCA which is capable of learning from multiple views. The formula of GCCA as referred in [4] is simple and elegant, and then the extension of using neural networks is also straightforward. Since people has relatively clearer understanding of CCA itself, the generalised version or the kernel version of it is also well-understood. \n\nA nice theoretical understanding of contrastive unsupervised learning is provided in [5], and I recommend the authors to study.\n\n\n[1] de Sa, Virginia R. \"Learning classification with unlabeled data.\" Advances in neural information processing systems. 1994.\n[2] ICML Workshop, \"Learning With Multiple Views\". 2005\n[3] NIPS workshop, \"Learning from multiple sources\". 2008\n[4] Benton, Adrian, et al. \"Deep generalized canonical correlation analysis.\" ICLR workshop 2017.\n[5] Arora, Sanjeev, et al. \"A theoretical analysis of contrastive unsupervised representation learning.\" ICML 2019.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SkgXlgdptr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new self-supervised learning methods by utilizing contrastive predictive coding technique.  The proposed algorithm is more effective than existing self-supervised learning algorithm.  The presented results are encouraging.  \n1. In section 3.2,  the authors show that  a large number of views would improve the representation quality. However,  multi-views may provide redundancy information. What is the core information that affect  the representation quality?\n\nIn fact,  I am not an expert on self-supervised learning and  contrastive predictive coding,  so my reviewer confidence is low.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "ByxjhmgRYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This interesting paper on an important topic; however, its readability could be dramatically improved, especially for the reader less familiar with the problem. \n\nIn order to make the paper more accessible, the authors should reorganize the introduction by breaking it down into two parts:\n1) a more traditional introduction \n- one intuitive paragraph about multi-view coding\n- one intuitive paragraph with an illustrative example on how the proposed approach will help solve a problem; at the same intuitive level, compare-and-contrast it with existing approaches \n- one intuitive, in-detail paragraph on how the proposed approach works\n- one paragraph summarizing the main findings/results \n2) a second, new section, that will turn the current Figures 1 & 2 into a complete description of an illustrative example (the current, detailed \"captions\" are a good start, but they should be fleshed out into a full, detailed section of the paper)  \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use contrastive predictive coding for self-supervised learning.  The proposed approach is shown empirically to be  more effective than existing self-supervised learning algorithms.  While the reviewers found the experimental results encouraging, there were some questions about the contribution as a whole, in particular the lack of theoretical justification.
Decision: Reject