Title: CONTRASTIVE MULTIVIEW CODING

Abstract: Humans view the world through many sensory channels, e.g., the long-wavelength light channel, viewed by the left eye, or the high-frequency vibrations channel, heard by the right ear. Each view is noisy and incomplete, but important factors, such as physics, geometry, and semantics, tend to be shared between all views (e.g., a "dog" can be seen, heard, and felt). We hypothesize that a powerful representation is one that models view-invariant factors. Based on this hypothesis, we investigate a contrastive coding scheme, in which a representation is learned that aims to maximize mutual information between different views but is otherwise compact. Our approach scales to any number of views, and is view-agnostic. The resulting learned representations perform above the state of the art for downstream tasks such as object classification, compared to formulations based on predictive learning or single view reconstruction, and improve as more views are added. On the Imagenet linear readoff benchmark, we achieve 68.4% top-1 accuracy.

1. INTRODUCTION:

A foundational idea in coding theory is to learn compressed representations that nonetheless can be used to reconstruct the raw data. This idea shows up in contemporary representation learning in the form of autoencoders (Salakhutdinov & Hinton, 2009) and generative models (Kingma & Welling, 2013; Goodfellow et al., 2014) , which try to represent a data point or distribution as losslessly as possible. Yet lossless representation might not be what we really want, and indeed it is trivial to achieve -the raw data itself is a lossless representation. What we might instead prefer is to keep the "good" information (signal) and throw away the rest (noise). How can we identify what information is signal and what is noise?

To an autoencoder, or a max likelihood generative model, a bit is a bit. No one bit is better than any other. Our conjecture in this paper is that some bits are in fact better than others. Some bits code important properties like semantics, physics, and geometry, while others code attributes that we might consider less important, like incidental lighting conditions or thermal noise in a camera's sensor.

We hypothesize that the good bits are the ones that are shared between multiple views of the world, for example between multiple sensory modalities like vision, sound, and touch. Under this perspective "presence of dog" is good information, since dogs can be seen, heard, and felt, but "camera pose" is bad information, since a camera's pose has little or no effect on the acoustic and tactile properties of the imaged scene. There is significant evidence in the cognitive science and neuroscience literature that such cross-view representations are encoded by the brain (e.g., Smith & Gasser (2005) ; Den Ouden et al. (2012); Hohwy (2013)).

Our goal is therefore to learn representations that capture information shared between multiple sensory views but that are otherwise compact (i.e. throw away the bad information). To do so, we employ contrastive learning, where we learn a feature embedding such that views of the same scene map to nearby points while views of different scenes map to far apart points. In particular, we adapt the recently proposed method of Contrastive Predictive Coding (CPC) (Oord et al., 2018) , except we simplify it -removing the recurrent network -and generalize it -showing how to apply it to arbitrary collections of views, rather than just to temporal predictions. In reference to CPC, we term our method Contrastive Multiview Coding (CMC). The contrastive objective in our formulation, as in CPC, can be understood as attempting to maximize the mutual information between the representations of each view.

We intentionally leave "good bits" only loosely defined and treat its definition as an empirical question. Ultimately, the proof is in the pudding: we consider a representation to be good if it makes subsequent problem solving easy, on tasks of human interest. For example, a useful representation of images might be a feature space in which it is easy to learn to recognize objects. We therefore evaluate our method by testing if the learned representations transfer well to standard semantic recognition tasks. On several benchmark tasks, our method achieves state of the art results, compared to other methods for self-supervised representation learning. We additionally find that the quality of the representation improves as a function of the number of views used for training. Finally, we compare the contrastive formulation of multiview learning to the recently popular approach of cross-view prediction, and find that in head-to-head comparisons, the contrastive approach learns stronger representations.

The core ideas that we build on: contrastive learning, mutual information maximization, and deep representation learning, are not new and have been explored in the literature on representation and multiview learning (Li et al., 2018; Xu et al., 2013; Arora et al., 2019) . Our main contribution is to set up a framework to extend these ideas to any number of views, and to empirically study the factors that lead to success in this framework. A review of the related literature is given at the end of the paper, in Section 4. Fig. 1 gives a pictorial overview of our framework for the different learning tasks we consider in this paper, to learn representations across datasets with different sets of views. Our main contributions are:

• We apply contrastive learning to the multiview setting, attemping to maximize mutual information between representations of different views of the same scene (e.g., between different image channels, or different modalities). • Our approach yields representations that outperform the state-of-the-art in self-supervised learning in head-to-head comparisons. For example, in the ImageNet linear readoff evaluation, we achieve 68.4% top-1 accuracy, which is slightly above the state of the art concurrent work Bachman et al. (2019) . • We show that the contrastive objective is superior to cross-view prediction.

• We extend the framework to learn from more than two views, and show that the quality of the learned representation improves as number of views increase. • We conduct controlled experiments to measure the effect of mutual information on representation quality.

2. METHOD:

Our goal is to learn representations that capture information shared between multiple sensory views without human supervision. We start by reviewing previous predictive learning (or reconstruction- ing. Cross-view prediction (Top) learns latent representations that predict one view from another, with loss measured in the output space. Common prediction losses, such as the L1 and L2 norms, are unstructured, in the sense that they penalize each output dimension independently, perhaps leading to representations that do not capture all the shared information between the views. In contrastive learning (Bottom), representations are learnt by contrasting congruent and incongruent views, with loss measured in representation space. The red dotted outlines show where the loss function is applied.

based learning) methods, and then elaborate on contrastive learning within two views. We show connections to mutual information maximization and extend it to scenarios including more than two views. We consider a collection of M views of the data, denoted as V 1 , . . . , V M . For each view V i , we denote v i as a random variable representing samples following v i ∼ P(V i ).

2.1. PREDICTIVE LEARNING:

Let V 1 and V 2 represent two views of a dataset. For instance, V 1 might be the luminance of a particular image and V 2 the chrominance. We define the predictive learning setup as a deep nonlinear transformation from v 1 to v 2 through latent variables z, as shown in Fig. 2 . Formally, z = f (v 1 ) and v2 = g(z), where f and g represent the encoder and decoder respectively and v2 is the prediction of v 2 given v 1 . The parameters of the encoder and decoder models are then trained using an objective function that tries to bring v2 "close to" v 2 . Simple examples of such an objective include the L 1 or L 2 loss functions. Note that these objectives assume independence between each pixel or element of v 2 given v 1 , i.e., p(v 2 |v 1 ) = Π i p(v 2i |v 1 ), thereby reducing their ability to model correlations or complex structure. The predictive approach has been extensively used in representation learning, for example, colorization (Zhang et al., 2016; 2017) and predicting sound from vision (Owens et al., 2016) .

2.2. CONTRASTIVE LEARNING WITH TWO VIEWS:

The idea behind contrastive learning is to learn an embedding that separates (contrasts) samples from two different distributions. Given a dataset of V 1 and V 2 that consists of a collection of samples {v i 1 , v i 2 } N i=1 , we consider contrasting congruent and incongruent pairs, i.e. samples from the joint distribution x ∼ p(v 1 , v 2 ) or x = {v i 1 , v i 2 }, which we call positives, versus samples from the product of marginals, y ∼ p(v 1 )p(v 2 ) or y = {v i 1 , v j 2 }, which we call negatives. We learn a "critic" h θ (•) that is high for positives and low for negatives. Similar to recent setups for contrastive learning (Oord et al., 2018; Gutmann & Hyvärinen, 2010; Mnih & Kavukcuoglu, 2013), we train this function to correctly select a single positive sample x out of a set S = {x, y 1 , y 2 , ..., y k } that contains k negative samples:

EQUATION

To construct S, we simply fix one view and enumerate positives and negatives from the other view, allowing us to rewrite the objective as:

L V1,V2 contrast = - E {v 1 1 ,v 1 2 ,...,v k+1 2 } log h θ ({v 1 1 , v 1 2 }) k+1 j=1 h θ ({v 1 1 , v j 2 }) (2)

where k is the number of negative samples v j 2 for a given sample v 1 1 . In practice, k can be extremely large, and so directly minimizing Eq. 2 is infeasible. In Section 2.4, we show an approximation based on Noise Contrastive Estimation (Gutmann & Hyvärinen, 2010 ) that allows for tractable computation.

1 2 1 1 3 2 2 1 3 1 1 6 3 3 1 1 3 1 (a) Core View (b) Full Graph V 1

Figure 3 : Graphical models and information diagrams (inf) associated with the core view and full graph paradigms, for the case of 4 views, which gives a total of 6 learning objectives. The numbers within the regions show how much "weight" the total loss places on each partition of information (i.e. how many of the 6 objectives that partition contributes to). A region with no number corresponds to 0 weight. For example, in the full graph case, the mutual information between all 4 views is considered in all 6 objectives, and hence is marked with the number 6.

Implementing the critic We implement the critic h θ (•) as a neural network. To extract compact latent representations of v 1 and v 2 , we employ two encoders f θ1 (•) and f θ2 (•) with parameters θ 1 and θ 2 respectively. The latent representions are extracted as

z 1 = f θ1 (v 1 ), z 2 = f θ2 (v 2

). On top of these features, the score is computed as the exponential of a bivariate function of z 1 and z 2 , e.g., a bilinear function parameterized by W 12 :

EQUATION

Loss L V1,V2 contrast in Eq. 2 treats view V 1 as anchor and enumerates over V 2 . Symmetrically, we can get

L V2,V1

contrast by anchoring at V 2 . We add them up as our two-view loss:

EQUATION

After the contrastive learning phase, we use the representation z 1 , z 2 , or the concatenation of both, [z 1 , z 2 ], depending on our paradigm. This process is visualized in Fig. 1 .

Connecting to mutual information:

The optimal critic h * θ is proportional to the density ratio between the joint distribution p(z 1 , z 2 ) and the product of marginals p(z 1 )p(z 2 ) (proof provided in Sec. C.1):

EQUATION

This quantity is the pointwise mutual information, and its expectation, in Eq. 2, yields an estimator related to mutual information. A formal proof is given by Oord et al. (2018); Poole et al. ( 2019), which we recapitulate in Section C, showing that:

EQUATION

where, as above, k is the number of negative pairs in sample set S. Hence minimizing the objective L maximizes the lower bound on the mutual information I(z i ; z j ), which is bounded above by I(v i ; v j ) by the data processing inequality. The dependency on k also suggests that using more negative samples can lead to an improved representation; we show that this is indeed the case in Section A.1.2. We note that recent work (McAllester & Statos, 2018) shows that the bound in Eq. 6 can be very weak; and finding better estimators of mutual information is an important open problem.

2.3. CONTRASTIVE LEARNING WITH MORE THAN TWO VIEWS:

We present more general formulations of Eq. 2 that can handle any number of views. We call them the "core view" and "full graph" paradigms, which offer different tradeoffs between efficiency and effectiveness. These formulations are visualized in Fig. 3 .

Suppose we have a collection of M views V 1 , . . . , V M . The "core view" formulation sets apart one view that we want to optimize over, say V 1 , and builds pair-wise representations between V 1 and each other view V j , j > 1, by optimizing the sum of a set of pair-wise objectives:

EQUATION

A second, more general formulation is the "full graph" where we consider all pairs (i, j), i = j, and build n 2 relationships in all. By involving all pairs, the objective function that we optimize is:

EQUATION

Both these formulations have the effect that information is prioritized in proportion to the number of views that share that information. This can be seen in the information diagrams visualized in Fig. 3 . The number in each partition of the diagram indicates how many of the pairwise objectives, L(V i , V j ), that partition contributes to. Under both the core view and full graph objectives, a factor, like "presence of dog", that is common to all views will be preferred over a factor that affects fewer views, such as "depth sensor noise".

The computational cost of the bivariate score function in the full graph formulation is combinatorial in the number of views. However, it is clear from Fig. 3 that this enables the full graph formulation to capture more information between different views, which may prove useful for downstream tasks.

For example, the mutual information between V 2 and V 3 or V 2 and V 4 is completely ignored in the core view paradigm (as shown by a 0 count in the information diagram).

2.4. APPROXIMATING THE SOFTMAX DISTRIBUTION WITH NOISE-CONTRASTIVE ESTIMATION:

Better representations using L V1,V2 contrast in Eq. 2 are learnt by using many negative samples. However, computing the full softmax loss is prohibitively expensive for large N . We alleviate computational load by using Noise-Contrastive Estimation (NCE) (Gutmann & Hyvärinen, 2010) to approximate the full softmax in Eqn. 2, as has also been used in Mnih & Kavukcuoglu (2013) 1 .

Given an anchor v i:

1 from V 1 , the probablity that an atom v 2 ∈ {v j 2 |j = 1, 2, ..., N } from V 2 is the best match of v i 1 , using the score h θ is given by:

EQUATION

where the normalization factor Z = N j=1 h θ ({v i 1 , v j 2 }) is expensive to compute for large N . Here we use h θ ({v 1 , v 2 }) = exp(f θ1 (v 1 ) T W 12 f θ2 (v 2 )/τ ), where τ modulates the distribution.

Noise-Contrastive Estimation (Gutmann & Hyvärinen, 2010) (NCE) is an effective way to estimate unnormalized statistical models. NCE fits a density model p to data distributed as (unknown) distribution p d , by using a binary classifier to distinguish it from noise samples distributed as p n . To learn p(v 2 |v i 1 ), we use a binary classifier, which treats v 2 as the data sample when given v i 1 . The noise distribution p n (•|v i 1 ) we choose here is a uniform distribution over all atoms from V 2 , i.e., p n (•|v i 1 ) = 1/N . If we sample m noise samples to pair with each data sample, the posterior probability that a given atom v 2 comes from the data distribution is:

EQUATION

and we estimate this probability by replacing p d (v 2 |v i 1 ) with our unnormalized model distribution h θ (v i 1 , v 2 ). Minimizing the negative log-posterior probability of correct labels D over data and noise samples yields our final objective, which is the NCE-based approximation of Eq. 2 (p is the empirical data distribution):

L N CE = -E v i 1 ∼ p(v1) { E v2∼ p(•|v i 1 )

log(P (D = 1|v 2 ;

v i 1 )) + m E v2∼pn(•|v i 1 )

log(P (D = 0|v 2 ; v i 1 )) } (11) 

3.2. DOES REPRESENTATION QUALITY IMPROVE AS NUMBER OF VIEWS INCREASES?:

We further extend our CMC learning framework to multiview scenarios. We experiment on the NYU-Depth-V2 (Nathan Silberman & Fergus, 2012) dataset which consists of 1449 labeled images. We focus more on understanding the behavior and effectiveness of CMC rather than competing with the current state-of-the-arts. The views we consider are: luminance (L channel), chrominance (ab channel), depth, surface normal (Eigen & Fergus, 2015) , and semantic labels.

Setup. To extract features from each view, we use a neural network with 5 convolutional layers, and 1 fully connected layer. As the size of the dataset is relatively small, we adopt the sub-patch based contrastive objective (see B) to increase the number of negative pairs. Patches with a size of 128 × 128 are randomly cropped from the original images for contrastive learning (from images of size 480 × 640). For downstream tasks, we discard the fully connected layers and evaluate using the convolutional layers as a representation.

To measure the quality of the learned representation, we consider the task of predicting semantic labels from the representation of L. We follow the core view paradigm and use L are the core view, thus learning a set of representations on L by contrasting different views with L. A UNet style architecture (Ronneberger et al., 2015) is utilized to perform the segmentation task. Contrastive training is performed on the above architecture that is equivalent of the UNet's encoder. After contrastive training is completed, we initialize the encoder weights of the UNet from the L encoder (which are equivalent architectures) and keep them frozen. Only the decoder is trained during this finetuning stage.

Since we use the patch-based contrastive loss, in the 1 view setting case, CMC coincides with DIM (Hjelm et al., 2019) . The 2-4 view cases contrast L with ab, and then sequentially add depth and surface normals. The semantic labeling results are measured by mean IoU over all classes and pixel accuracy, shown in Fig. 4 . We see that the performance steadily improves as new views are added.

We have tested different orders of adding the views, and they all follow a similar pattern.

We also compare CMC with two baselines. First, we randomly initialize and freeze the encoder, and we call this the Random baseline; it serves as a lower bound on the quality since the representation is just a random projection. Rather than freezing the randomly initialized encoder, we could train it Table 3 : Results on the task of predicting semantic labels from L channel representation which is learnt using the patch-based contrastive loss and all 4 views. We compare CMC with Random and Supervised baselines, which serve as lower and upper bounds respectively. Th core-view paradigm refers to Fig. 3 (a), and full-view Fig. 3(b) .

jointly with the decoder. This end-to-end Supervised baseline serves as an upper bound. The results are presented in Table 3 , which shows our CMC produces high quality feature maps even though it's unaware of the downstream task.

3.3. PREDICTIVE LEARNING VS. CONTRASTIVE LEARNING:

While experiments in section 3.1 show that contrastive learning outperforms predictive learning (Zhang et al., 2017) in the context of Lab color space, it's unclear whether such an advantage is due to the natural inductive bias of the task itself. To further understand this, we go beyond chrominance (ab), and try to answer this question when geometry or semantic labels are present.

We consider three view pairs on the NYU-Depth dataset: (1) L and depth, (2) L and surface normals, and (3) L and segmentation map. For each of them, we train two identical encoders for L, one using contrastive learning and the other with predictive learning. We then evaluate the representation quality by training a linear classifier on top of these encoders on the STL-10 dataset. The comparison results are shown in Table 4 , which shows that contrastive learning consistently outperforms predictive learning in this scenario where both the task and the dataset are unknown. We also include "random" and "supervised" baselines similar to that in previous sections. Though in the unsupervised stage we only use 1.3K images from a dataset much different from the target dataset STL-10, the object recognition accuracy is close to the supervised method, which uses an end-to-end deep network directly trained on STL-10.

Given two views V 1 and V 2 of the data, the predictive learning approach approximately models p(v 2 |v 1 ). Furthermore, losses used typically for predictive learning, such as pixel-wise reconstruction losses usually impose an independence assumption on the modeling: p(v 2 |v 1 ) ≈ Π i p(v 2i |v 1 ). On the other hand, the contrastive learning approach by construction does not assume conditional independence across dimensions of v 2 . In addition, the use of random jittering and cropping between views allows the contrastive learning approach to benefit from spatial co-occurrence (contrasting in space) in addition to contrasting across views. We conjecture that these are two reasons for the superior performance of contrastive learning approaches over predictive learning.

3.4. HOW DOES MUTUAL INFORMATION AFFECT REPRESENTATION QUALITY?:

Given a fixed set of views, CMC aims to maximize the mutual information between representations of these views. We have found that maximizing information in this way indeed results in strong representations, but it would be incorrect to infer that information maximization (infomax) is the key to good representation learning. In fact, this paper argues for precisely the opposite idea: that cross-view representation learning is effective because it results in a kind of information minimization, discarding nuisance factors that are not shared between the views.

The resolution to this apparent dilemma is that we want to maximize the "good" informationthe signal -in our representations, while minimizing the "bad" information -the noise. The idea behind CMC is that this can be achieved by doing infomax learning on two views that share signal but have independent noise. This suggests a "Goldilocks principle": a good collection of views is one that shares some information but not too much. Here we test this hypothesis on two domains: learning representations on images with different colorspaces forming the two views; and learning representations on pairs of patches extracted from an image, separated by varying spatial distance.

In patch experiments we randomly crop two RGB patches of size 64x64 from the same image as two views. Their relative position is fixed. Namely, the two patches always starts at position (x, y) and (x + d, y + d) with (x, y) being randomly sampled. While varying the distance d, we start from 64 to avoid overlapping. There is a possible bias that with an image of relatively small size (e.g., 512x512), a large d (e.g., 384) will always push these two patches around boundary. To minimize this bias, we use high resolution images (e.g. 2k) from DIV2K (Agustsson & Timofte, 2017) dataset. 2011). The plots clearly show that using colorspaces with minimal mutual information give the best downstream accuracy (For the outlier HSV in this plot, we conjecture the representation quality is harmed by the periodicity of H. Note that the H in HED is not periodic.). On the other hand, the story is more nuanced for representations learned between patches at different offsets from each other (Fig. 5 , right). Here we see that views with too little or too much MI perform worse; a sweet spot in the middle exists which gives the best representation. That there exists such a sweet spot should be expected. If two views share no information, then, in principle, there is no incentive for CMC to learn anything. If two views share all their information, no nuisances are discarded and we arrive back at something akin to an autoencoder or generative model, that simply tries to represent all the bits in the multiview data.

These experiments demonstrate that the relationship between mutual information and representation quality is meaningful but not direct. Selecting optimal views, which just share relevant signal, may be a fruitful direction for future research.

4. RELATED WORK:

Unsupervised representation learning is about learning transformations of the data that make subsequent problem solving easier (Bengio et al., 2013) . This field has a long history, starting with classical methods with well established algorithms, such as principal components analysis (PCA (Jolliffe, 2011)) and independent components analysis (ICA (Hyvärinen et al., 2004) ). These methods tend to learn representations that focus on low-level variations in the data, which are not very useful from the perspective of downstream tasks such as object recognition.

Representations better suited to such tasks have been learnt using deep neural networks, starting with seminal techniques such as Boltzmann machines (Smolensky, 1986; Salakhutdinov & Hinton, 2009) , autoencoders (Hinton & Salakhutdinov, 2006) Zhang et al., 2019) . In these models, an input X to the model is transformed into an output X, which is supposed to be close to another signal Y , which itself is related to X in some meaningful way. Examples of such X/Y pairs are: luminance and chrominance color channels of an image (Zhang et al., 2017) , patches from a single image (Oord et al., 2018), modalities such as vision and sound (Owens et al., 2016) or the frames of a video (Wang & Gupta, 2015) . Clearly, such examples are numerous in the world, and provides us with nearly infinite amounts of training data: this is one of the appeals of this paradigm. Time contrastive networks (Sermanet et al., 2017) ). These two methods, like ours, learn representations by contrasting between congruent and incongruent representations of a scene, and are motivated as forms of infomax learning. CPC learns from two views -the past and future -and is applicable to sequential data. Deep Infomax (Hjelm et al., 2019) considers the two views to be the input to a neural network and its output. These two methods share the same mathematical objective, but differ in the definition of the views. Our technical method is also highly related, but differs in the following ways: we extend the objective to the case of more than two views; and we use a loss function which more closely follows the original method of noise contrastive estimation (Gutmann & Hyvärinen, 2010 ) (See details in Section 2.4). Although CPC, Deep InfoMax, and the present paper are all very similar at the mathematical level, they each explore a different set of view definitions, architectures, and application settings, and each contributes its own unique empirical investigation of this paradigm of representation learning.

:

Confusingly, the literature has previously referred to Eqn.

as "InfoNCE"(Oord et al., 2018). Our NCE approximation does not refer to the allusion to NCE in the name "InfoNCE". Rather we are here describing an NCE approximation to the "InfoNCE" softmax objective.

Figures and Tables:

Figure 1: (a) Given a pair of sensory views, a deep representation is learnt by bringing views of the same scene together in embedding space, while pushing views of different scenes apart. Here we show an example of learning from the luminance channel (L) of an image and the ab-color channel. The strawberry's L and ab channels embed to nearby points whereas the ab channel of a different image (a photo of blueberries) embeds to a far away point. (b) Example of a 4-view dataset (NYU RGBD (Nathan Silberman & Fergus, 2012)) and its learned representation. Dotted lines represent the contrastive objective. The encodings for each view may be concatenated to form the full representation of a scene.

Figure 2: Predictive Learning vs Contrastive Learn-

Figure 4: We show the Intersection over Union (IoU) (left) and Pixel Accuracy (right) for the NYU-Depth-V2dataset, as CMC is trained with increasingly more views from 1 to 4. As more views are added, both these metrics steadily increase. The views are (in order of inclusion): L, ab, depth and surface normals.

Figure 5: How does mutual information between views relate to representation quality? (Left) Classification accuracy against estimated MI between channels of different color spaces; (Right) Classification accuracy vs estimated MI between patches at different distances (distance in pixels is denoted next to each data point). MI estimated using MINE (Belghazi et al., 2018).

Fig. 5 shows the results of these experiments. The left plot shows the result of learning representations on different colorspaces (splitting each colorspace into two views, such as (L, ab), (R, GB) etc). We then use the MINE estimator Belghazi et al. (2018) to estimate the mutual information between the views. We measure representation quality by training a linear classifier on the learned representations on the STL-10 dataset Coates et al. (2011). The plots clearly show that using colorspaces with minimal mutual information give the best downstream accuracy (For the outlier HSV in this plot, we conjecture the representation quality is harmed by the periodicity of H. Note that the H in HED is not periodic.). On the other hand, the story is more nuanced for representations learned between patches at different offsets from each other (Fig.5, right). Here we see that views with too little or too much MI perform worse; a sweet spot in the middle exists which gives the best representation. That there exists such a sweet spot should be expected. If two views share no information, then, in principle, there is no incentive for CMC to learn anything. If two views share all their information, no

use a triplet loss framework to learn representations from aligned video sequences of the same scene, taken by different video cameras. Closely related to self-supervised learning is the idea of multi-view learning, which is a general term involving many different approaches such as co-training (Blum & Mitchell, 1998), multi-kernel learning (Cortes et al., 2009) and metric learning (Bellet et al., 2012; Zhuang et al., 2019); for comprehensive surveys please see (Xu et al., 2013; Li et al., 2018). Nearly all existing works have dealt with one or two views such as video or image/sound. However, in many situations, many more views are available to provide training signals for any representation. The objective functions used to train deep learning based representations in many of the above methods are either reconstruction-based loss functions such as Euclidean losses in different norms e.g. (Isola et al., 2017), adversarial loss functions (Goodfellow et al., 2014) that learn the loss in addition to the representation, or contrastive losses e.g. (Gutmann & Hyvärinen, 2010; Hjelm et al., 2019; Oord et al., 2018; Arora et al., 2019; Hénaff et al., 2019) that take advantage of the co-occurence of multiple views. Another recently introduced novel objective function is instance discrimination (Wu et al., 2018). In this work, we compare the two most commonly used objectives: predictive and contrastive. The prior works most similar to our own (and inspirational to us) are Contrastive Predictive Coding (CPC) (Oord et al., 2018) and Deep InfoMax (Hénaff et al., 2019

Memory bank. FollowingWu et al. (2018), we maintain a memory bank to store latent features for each training sample. Therefore, we can efficiently retrieve m noise samples from the memory bank to pair with each positive sample without recomputing their features. The memory bank is dynamically updated with features computed on the fly.An alternative to the NCE based approximation above, is to simply do m+1 way softmax classification with m noise samples retrieved from the memory bank. We note that CPC (Oord et al., 2018) and Deep InfoMax(Hjelm et al., 2019) use this m + 1 way softmax classification as their ultimate contrastive loss rather than the NCE-based contrastive loss in Eq. 11 (but note that CPC refers to the m + 1 approximation as also "based on NCE"). Empirically we have found that the m + 1-way softmax classification approach performed worse than our NCE-based approximation, given the same number of noise samples.3 EXPERIMENTSWe extensively evaluate Contrastive Multiview Coding (CMC) on a number of datasets and tasks. We evaluate on two established image representation learning benchmarks: Imagenet and STL-10 (See A.1.1). We further validate our framework on video representation learning tasks (See A.2), where we use image and optical flow modalities, as the two views that are jointly learned. The last set of experiments extends our CMC framework to more than two views and provides empirical evidence of it's effectiveness.To compare with other methods, we adopt standard AlexNet and split it into two encoders. Because of splitting, each layer only connects to half of the neurons in the previous layer, and therefore the number of parameters in our model halves. We remove local response layer and add batch normalization to each layer. For the memory-based CMC model, we adopt ideas from Wu et al. (2018) for computing and storing a memory. We retrieve 4096 negative pairs from the memory bank to contrast each positive pair. The training details are present in Sec. D.2. Top-1 classification accuracy on 1000 classes of ImageNet Deng et al. (2009) with single crop. We compare our CMC method with other unsupervised representation learning approaches by training 1000-way logistic regression classifiers on top of the feature maps of each layer, as proposed by Zhang et al. (2016). Methods marked with † only have half the number of parameters compared to others, because of splitting.

Table content: <table><tr><td>Table 1 shows the results of comparing the CMC against other models, both predictive and contrastive.</td></tr><tr><td>Our CMC is the best among all these methods; futhermore CMC tends to perform better at higher</td></tr><tr><td>convolutional layers, similar to another contrasting-based model Inst-Dis (Wu et al., 2018).</td></tr></table>

Single

Table content: <table/>

We compare predictive learning

Table content: <table><tr><td/><td>Accuracy on STL-10 (%)</td><td/></tr><tr><td>Views L, Depth L, Normal L, Seg. Map</td><td>Predictive Contrastive 55.5 58.3 58.4 60.1 57.7 59.2</td><td>with contrastive learning by evaluating the learned encoder on unseen dataset and task. The contrastive learning framework consis-tently outperforms predictive learning.</td></tr><tr><td>Random</td><td>25.2</td><td/></tr><tr><td>Supervised</td><td>65.1</td><td/></tr></table>

, variational autoencoders (Kingma & Welling, 2013), generative adversarial networks(Goodfellow et al., 2014) and autoregressive models(Oord et al., 2016). Numerous other works exist, for a review see(Bengio et al., 2013). A powerful family of models for unsupervised representations are collected under the umbrella of "self-supervised" learning(Sa, 2004; Zhang et al., 2017; 2016; Isola et al., 2015; Wang & Gupta, 2015; Pathak et al., 2016;

Table content: <table/>

Peer review (iclr2020):
The paper presented a multi-view learning method that is based on negative sampling in contrastive learning. The core idea is to set an anchor view and the sample positive and negative data points from the other view and maximise the agreement between positive pairs in learning from two views. When more than two views are presented, the learning objective is a sum over all possible combinations of two views. The performance of the proposed model is good, and the ablation study is interesting. Comments: 1. The core concept, or at least one of the core concepts, in multi-view learning is the conditional independence. Normally, the underlying assumption in multi-view learning is that, given the class label, the samples from multiple views are conditionally independent from each other. Therefore, the goal is to learn distinctive representations from different data sources/disjoint populations, so then after learning, the ensemble of them is able to capture a set of diverse aspects of the data. A *side-effect* of learning from multiple views is that individual views indeed get improved by learning from others. Meanwhile, self-supervised learning is the case when the input data to the designed learning system is also the target of the system. The paper presented an idea for self-supervised learning from multiple views, which is not exactly the same, but still in the same regime. This concept could be used to explain some empirical findings in this paper. Since it is expected, there is even no need in conducting experiments. 2. My main concern of this paper is the novelty, however, the empirical results are strong. The paper mainly presented a simple yet effective method for self-supervised learning from two views, and the generalisation is a sum over all possible combinations of two views. The method itself has already been proposed many years ago as mentioned in the related work section in the paper, and the generalisation was also described in prior work, which makes me doubt the novelty of the paper. The earliest work to the best of my knowledge is [1], and later on there are a couple workshops [2,3] on multi-view learning which largely settled the field of learning from multiple views from neural networks*, kernels*, and bayesian perspectives. Many things mentioned in this paper have already been discovered at that time. 3. The theoretical justification is not as strong as the generalised CCA. CCA has been applied in the field of multi-view learning and self-supervised learning for long, and it was initially proposed for comparing the correlation between two sets of samples of two random variables. A successful generalisation is the generalised CCA which is capable of learning from multiple views. The formula of GCCA as referred in [4] is simple and elegant, and then the extension of using neural networks is also straightforward. Since people has relatively clearer understanding of CCA itself, the generalised version or the kernel version of it is also well-understood. A nice theoretical understanding of contrastive unsupervised learning is provided in [5], and I recommend the authors to study. [1] de Sa, Virginia R. *Learning classification with unlabeled data.* Advances in neural information processing systems. 1994. [2] ICML Workshop, *Learning With Multiple Views*. 2005 [3] NIPS workshop, *Learning from multiple sources*. 2008 [4] Benton, Adrian, et al. *Deep generalized canonical correlation analysis.* ICLR workshop 2017. [5] Arora, Sanjeev, et al. *A theoretical analysis of contrastive unsupervised representation learning.* ICML 2019.