Year: 2019

Abstract:
Humans view the world through many sensory channels, e.g., the long-wavelength light channel, viewed by the left eye, or the high-frequency vibrations channel, heard by the right ear. Each view is noisy and incomplete, but important factors, such as physics, geometry, and semantics, tend to be shared between all views (e.g., a “dog” can be seen, heard, and felt). We hypothesize that a powerful representation is one that models view-invariant factors. Based on this hypothesis, we investigate a contrastive coding scheme, in which a representation is learned that aims to maximize mutual information between different views but is otherwise compact. Our approach scales to any number of views, and is view-agnostic. The resulting learned representations perform above the state of the art for downstream tasks such as object classification, compared to formulations based on predictive learning or single view reconstruction, and improve as more views are added. On the Imagenet linear readoff benchmark, we achieve 68.4% top-1 accuracy.

heading: 1 INTRODUCTION
text: A foundational idea in coding theory is to learn compressed representations that nonetheless can be used to reconstruct the raw data. This idea shows up in contemporary representation learning in the form of autoencoders (Salakhutdinov & Hinton, 2009) and generative models (Kingma & Welling, 2013; Goodfellow et al., 2014), which try to represent a data point or distribution as losslessly as possible. Yet lossless representation might not be what we really want, and indeed it is trivial to achieve – the raw data itself is a lossless representation. What we might instead prefer is to keep the “good” information (signal) and throw away the rest (noise). How can we identify what information is signal and what is noise?
To an autoencoder, or a max likelihood generative model, a bit is a bit. No one bit is better than any other. Our conjecture in this paper is that some bits are in fact better than others. Some bits code important properties like semantics, physics, and geometry, while others code attributes that we might consider less important, like incidental lighting conditions or thermal noise in a camera’s sensor.
We hypothesize that the good bits are the ones that are shared between multiple views of the world, for example between multiple sensory modalities like vision, sound, and touch. Under this perspective “presence of dog” is good information, since dogs can be seen, heard, and felt, but “camera pose” is bad information, since a camera’s pose has little or no effect on the acoustic and tactile properties of the imaged scene. There is significant evidence in the cognitive science and neuroscience literature that such cross-view representations are encoded by the brain (e.g., Smith & Gasser (2005); Den Ouden et al. (2012); Hohwy (2013)).
Our goal is therefore to learn representations that capture information shared between multiple sensory views but that are otherwise compact (i.e. throw away the bad information). To do so, we employ contrastive learning, where we learn a feature embedding such that views of the same scene map to nearby points while views of different scenes map to far apart points. In particular, we adapt the recently proposed method of Contrastive Predictive Coding (CPC) (Oord et al., 2018), except we simplify it – removing the recurrent network – and generalize it – showing how to apply it to arbitrary collections of views, rather than just to temporal predictions. In reference to CPC, we term our method Contrastive Multiview Coding (CMC). The contrastive objective in our formulation, as in CPC, can be understood as attempting to maximize the mutual information between the representations of each view.
We intentionally leave “good bits” only loosely defined and treat its definition as an empirical question. Ultimately, the proof is in the pudding: we consider a representation to be good if it makes subsequent
problem solving easy, on tasks of human interest. For example, a useful representation of images might be a feature space in which it is easy to learn to recognize objects. We therefore evaluate our method by testing if the learned representations transfer well to standard semantic recognition tasks. On several benchmark tasks, our method achieves state of the art results, compared to other methods for self-supervised representation learning. We additionally find that the quality of the representation improves as a function of the number of views used for training. Finally, we compare the contrastive formulation of multiview learning to the recently popular approach of cross-view prediction, and find that in head-to-head comparisons, the contrastive approach learns stronger representations.
The core ideas that we build on: contrastive learning, mutual information maximization, and deep representation learning, are not new and have been explored in the literature on representation and multiview learning (Li et al., 2018; Xu et al., 2013; Arora et al., 2019). Our main contribution is to set up a framework to extend these ideas to any number of views, and to empirically study the factors that lead to success in this framework. A review of the related literature is given at the end of the paper, in Section 4. Fig. 1 gives a pictorial overview of our framework for the different learning tasks we consider in this paper, to learn representations across datasets with different sets of views. Our main contributions are:
• We apply contrastive learning to the multiview setting, attemping to maximize mutual information between representations of different views of the same scene (e.g., between different image channels, or different modalities). • Our approach yields representations that outperform the state-of-the-art in self-supervised
learning in head-to-head comparisons. For example, in the ImageNet linear readoff evaluation, we achieve 68.4% top-1 accuracy, which is slightly above the state of the art concurrent work Bachman et al. (2019). • We show that the contrastive objective is superior to cross-view prediction. • We extend the framework to learn from more than two views, and show that the quality of
the learned representation improves as number of views increase. • We conduct controlled experiments to measure the effect of mutual information on represen-
tation quality.

heading: 2 METHOD
text: Our goal is to learn representations that capture information shared between multiple sensory views without human supervision. We start by reviewing previous predictive learning (or reconstruction-
based learning) methods, and then elaborate on contrastive learning within two views. We show connections to mutual information maximization and extend it to scenarios including more than two views. We consider a collection of M views of the data, denoted as V1, . . . , VM . For each view Vi, we denote vi as a random variable representing samples following vi ∼ P(Vi).

heading: 2.1 PREDICTIVE LEARNING
text: Let V1 and V2 represent two views of a dataset. For instance, V1 might be the luminance of a particular image and V2 the chrominance. We define the predictive learning setup as a deep nonlinear transformation from v1 to v2 through latent variables z, as shown in Fig. 2. Formally, z = f(v1) and v̂2 = g(z), where f and g represent the encoder and decoder respectively and v̂2 is the prediction of v2 given v1. The parameters of the encoder and decoder models are then trained using an objective function that tries to bring v̂2 “close to” v2. Simple examples of such an objective include the L1 or L2 loss functions. Note that these objectives assume independence between each pixel or element of v2 given v1, i.e., p(v2|v1) = Πip(v2i|v1), thereby reducing their ability to model correlations or complex structure. The predictive approach has been extensively used in representation learning, for example, colorization (Zhang et al., 2016; 2017) and predicting sound from vision (Owens et al., 2016).

heading: 2.2 CONTRASTIVE LEARNING WITH TWO VIEWS
text: The idea behind contrastive learning is to learn an embedding that separates (contrasts) samples from two different distributions. Given a dataset of V1 and V2 that consists of a collection of samples {vi1, vi2}Ni=1, we consider contrasting congruent and incongruent pairs, i.e. samples from the joint distribution x ∼ p(v1, v2) or x = {vi1, vi2}, which we call positives, versus samples from the product of marginals, y ∼ p(v1)p(v2) or y = {vi1, vj2}, which we call negatives. We learn a “critic” hθ(·) that is high for positives and low for negatives. Similar to recent setups for contrastive learning (Oord et al., 2018; Gutmann & Hyvärinen, 2010; Mnih & Kavukcuoglu, 2013), we train this function to correctly select a single positive sample x out of a set S = {x, y1, y2, ..., yk} that contains k negative samples:
Lcontrast = −E S
[ log
hθ(x)
hθ(x) + ∑k i=1 hθ(yi)
] (1)
To construct S, we simply fix one view and enumerate positives and negatives from the other view, allowing us to rewrite the objective as:
LV1,V2contrast = − E {v11 ,v12 ,...,v k+1 2 }
[ log
hθ({v11 , v12})∑k+1 j=1 hθ({v11 , v j 2})
] (2)
where k is the number of negative samples vj2 for a given sample v 1 1 . In practice, k can be extremely large, and so directly minimizing Eq. 2 is infeasible. In Section 2.4, we show an approximation based on Noise Contrastive Estimation (Gutmann & Hyvärinen, 2010) that allows for tractable computation.
1
2
1
1
3 2 2 1
3
1
1
6 3 3
1 1
3
1
(a) Core View (b) Full Graph
V1 <latexit sha1_base64="BTRfsG7MuUcz7Tb1mYDXTk3FGhE=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpodX3+27Vq3lzkFXiF6QKBRp996s3SFgWc4VMUmO6vpdikFONgkk+rfQyw1PKxnTIu5YqGnMT5PNTp+TMKgMSJdqWQjJXf0/kNDZmEoe2M6Y4MsveTPzP62YYXQe5UGmGXLHFoiiTBBMy+5sMhOYM5cQSyrSwtxI2opoytOlUbAj+8surpHVR872af39Zrd8UcZThBE7hHHy4gjrcQQOawGAIz/AKb450Xpx352PRWnKKmWP4A+fzB9fVjX4=</latexit><latexit sha1_base64="BTRfsG7MuUcz7Tb1mYDXTk3FGhE=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpodX3+27Vq3lzkFXiF6QKBRp996s3SFgWc4VMUmO6vpdikFONgkk+rfQyw1PKxnTIu5YqGnMT5PNTp+TMKgMSJdqWQjJXf0/kNDZmEoe2M6Y4MsveTPzP62YYXQe5UGmGXLHFoiiTBBMy+5sMhOYM5cQSyrSwtxI2opoytOlUbAj+8surpHVR872af39Zrd8UcZThBE7hHHy4gjrcQQOawGAIz/AKb450Xpx352PRWnKKmWP4A+fzB9fVjX4=</latexit><latexit sha1_base64="BTRfsG7MuUcz7Tb1mYDXTk3FGhE=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpodX3+27Vq3lzkFXiF6QKBRp996s3SFgWc4VMUmO6vpdikFONgkk+rfQyw1PKxnTIu5YqGnMT5PNTp+TMKgMSJdqWQjJXf0/kNDZmEoe2M6Y4MsveTPzP62YYXQe5UGmGXLHFoiiTBBMy+5sMhOYM5cQSyrSwtxI2opoytOlUbAj+8surpHVR872af39Zrd8UcZThBE7hHHy4gjrcQQOawGAIz/AKb450Xpx352PRWnKKmWP4A+fzB9fVjX4=</latexit><latexit sha1_base64="BTRfsG7MuUcz7Tb1mYDXTk3FGhE=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpodX3+27Vq3lzkFXiF6QKBRp996s3SFgWc4VMUmO6vpdikFONgkk+rfQyw1PKxnTIu5YqGnMT5PNTp+TMKgMSJdqWQjJXf0/kNDZmEoe2M6Y4MsveTPzP62YYXQe5UGmGXLHFoiiTBBMy+5sMhOYM5cQSyrSwtxI2opoytOlUbAj+8surpHVR872af39Zrd8UcZThBE7hHHy4gjrcQQOawGAIz/AKb450Xpx352PRWnKKmWP4A+fzB9fVjX4=</latexit>
V2 <latexit sha1_base64="NdtrmwDhkKHNNnS1oER0xSZxDQA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmnXqp5b9e6vKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDZWY1/</latexit><latexit sha1_base64="NdtrmwDhkKHNNnS1oER0xSZxDQA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmnXqp5b9e6vKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDZWY1/</latexit><latexit sha1_base64="NdtrmwDhkKHNNnS1oER0xSZxDQA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmnXqp5b9e6vKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDZWY1/</latexit><latexit sha1_base64="NdtrmwDhkKHNNnS1oER0xSZxDQA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmnXqp5b9e6vKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDZWY1/</latexit>
V3 <latexit sha1_base64="OZ7/eGjiaAd6iFmhE9KemdqLXmM=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6xEnC/YgOlQgFo2ilh1b/sl+uuFV3DrJKvJxUIEejX/7qDWKWRlwhk9SYrucm6GdUo2CST0u91PCEsjEd8q6likbc+Nn81Ck5s8qAhLG2pZDM1d8TGY2MmUSB7YwojsyyNxP/87ophtd+JlSSIldssShMJcGYzP4mA6E5QzmxhDIt7K2EjaimDG06JRuCt/zyKmldVD236t3XKvWbPI4inMApnIMHV1CHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gDa3Y2A</latexit><latexit sha1_base64="OZ7/eGjiaAd6iFmhE9KemdqLXmM=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6xEnC/YgOlQgFo2ilh1b/sl+uuFV3DrJKvJxUIEejX/7qDWKWRlwhk9SYrucm6GdUo2CST0u91PCEsjEd8q6likbc+Nn81Ck5s8qAhLG2pZDM1d8TGY2MmUSB7YwojsyyNxP/87ophtd+JlSSIldssShMJcGYzP4mA6E5QzmxhDIt7K2EjaimDG06JRuCt/zyKmldVD236t3XKvWbPI4inMApnIMHV1CHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gDa3Y2A</latexit><latexit sha1_base64="OZ7/eGjiaAd6iFmhE9KemdqLXmM=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6xEnC/YgOlQgFo2ilh1b/sl+uuFV3DrJKvJxUIEejX/7qDWKWRlwhk9SYrucm6GdUo2CST0u91PCEsjEd8q6likbc+Nn81Ck5s8qAhLG2pZDM1d8TGY2MmUSB7YwojsyyNxP/87ophtd+JlSSIldssShMJcGYzP4mA6E5QzmxhDIt7K2EjaimDG06JRuCt/zyKmldVD236t3XKvWbPI4inMApnIMHV1CHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gDa3Y2A</latexit><latexit sha1_base64="OZ7/eGjiaAd6iFmhE9KemdqLXmM=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6xEnC/YgOlQgFo2ilh1b/sl+uuFV3DrJKvJxUIEejX/7qDWKWRlwhk9SYrucm6GdUo2CST0u91PCEsjEd8q6likbc+Nn81Ck5s8qAhLG2pZDM1d8TGY2MmUSB7YwojsyyNxP/87ophtd+JlSSIldssShMJcGYzP4mA6E5QzmxhDIt7K2EjaimDG06JRuCt/zyKmldVD236t3XKvWbPI4inMApnIMHV1CHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gDa3Y2A</latexit>
V4 <latexit sha1_base64="RGbnccGESBHMfrtDKDZHJvcqDa8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmlfVT236t3XKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDcYY2B</latexit><latexit sha1_base64="RGbnccGESBHMfrtDKDZHJvcqDa8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmlfVT236t3XKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDcYY2B</latexit><latexit sha1_base64="RGbnccGESBHMfrtDKDZHJvcqDa8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmlfVT236t3XKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDcYY2B</latexit><latexit sha1_base64="RGbnccGESBHMfrtDKDZHJvcqDa8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmlfVT236t3XKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDcYY2B</latexit>
V1 <latexit sha1_base64="BTRfsG7MuUcz7Tb1mYDXTk3FGhE=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpodX3+27Vq3lzkFXiF6QKBRp996s3SFgWc4VMUmO6vpdikFONgkk+rfQyw1PKxnTIu5YqGnMT5PNTp+TMKgMSJdqWQjJXf0/kNDZmEoe2M6Y4MsveTPzP62YYXQe5UGmGXLHFoiiTBBMy+5sMhOYM5cQSyrSwtxI2opoytOlUbAj+8surpHVR872af39Zrd8UcZThBE7hHHy4gjrcQQOawGAIz/AKb450Xpx352PRWnKKmWP4A+fzB9fVjX4=</latexit><latexit sha1_base64="BTRfsG7MuUcz7Tb1mYDXTk3FGhE=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpodX3+27Vq3lzkFXiF6QKBRp996s3SFgWc4VMUmO6vpdikFONgkk+rfQyw1PKxnTIu5YqGnMT5PNTp+TMKgMSJdqWQjJXf0/kNDZmEoe2M6Y4MsveTPzP62YYXQe5UGmGXLHFoiiTBBMy+5sMhOYM5cQSyrSwtxI2opoytOlUbAj+8surpHVR872af39Zrd8UcZThBE7hHHy4gjrcQQOawGAIz/AKb450Xpx352PRWnKKmWP4A+fzB9fVjX4=</latexit><latexit sha1_base64="BTRfsG7MuUcz7Tb1mYDXTk3FGhE=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpodX3+27Vq3lzkFXiF6QKBRp996s3SFgWc4VMUmO6vpdikFONgkk+rfQyw1PKxnTIu5YqGnMT5PNTp+TMKgMSJdqWQjJXf0/kNDZmEoe2M6Y4MsveTPzP62YYXQe5UGmGXLHFoiiTBBMy+5sMhOYM5cQSyrSwtxI2opoytOlUbAj+8surpHVR872af39Zrd8UcZThBE7hHHy4gjrcQQOawGAIz/AKb450Xpx352PRWnKKmWP4A+fzB9fVjX4=</latexit><latexit sha1_base64="BTRfsG7MuUcz7Tb1mYDXTk3FGhE=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpodX3+27Vq3lzkFXiF6QKBRp996s3SFgWc4VMUmO6vpdikFONgkk+rfQyw1PKxnTIu5YqGnMT5PNTp+TMKgMSJdqWQjJXf0/kNDZmEoe2M6Y4MsveTPzP62YYXQe5UGmGXLHFoiiTBBMy+5sMhOYM5cQSyrSwtxI2opoytOlUbAj+8surpHVR872af39Zrd8UcZThBE7hHHy4gjrcQQOawGAIz/AKb450Xpx352PRWnKKmWP4A+fzB9fVjX4=</latexit>
V2 <latexit sha1_base64="NdtrmwDhkKHNNnS1oER0xSZxDQA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmnXqp5b9e6vKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDZWY1/</latexit><latexit sha1_base64="NdtrmwDhkKHNNnS1oER0xSZxDQA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmnXqp5b9e6vKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDZWY1/</latexit><latexit sha1_base64="NdtrmwDhkKHNNnS1oER0xSZxDQA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmnXqp5b9e6vKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDZWY1/</latexit><latexit sha1_base64="NdtrmwDhkKHNNnS1oER0xSZxDQA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmnXqp5b9e6vKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDZWY1/</latexit>
V3 <latexit sha1_base64="OZ7/eGjiaAd6iFmhE9KemdqLXmM=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6xEnC/YgOlQgFo2ilh1b/sl+uuFV3DrJKvJxUIEejX/7qDWKWRlwhk9SYrucm6GdUo2CST0u91PCEsjEd8q6likbc+Nn81Ck5s8qAhLG2pZDM1d8TGY2MmUSB7YwojsyyNxP/87ophtd+JlSSIldssShMJcGYzP4mA6E5QzmxhDIt7K2EjaimDG06JRuCt/zyKmldVD236t3XKvWbPI4inMApnIMHV1CHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gDa3Y2A</latexit><latexit sha1_base64="OZ7/eGjiaAd6iFmhE9KemdqLXmM=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6xEnC/YgOlQgFo2ilh1b/sl+uuFV3DrJKvJxUIEejX/7qDWKWRlwhk9SYrucm6GdUo2CST0u91PCEsjEd8q6likbc+Nn81Ck5s8qAhLG2pZDM1d8TGY2MmUSB7YwojsyyNxP/87ophtd+JlSSIldssShMJcGYzP4mA6E5QzmxhDIt7K2EjaimDG06JRuCt/zyKmldVD236t3XKvWbPI4inMApnIMHV1CHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gDa3Y2A</latexit><latexit sha1_base64="OZ7/eGjiaAd6iFmhE9KemdqLXmM=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6xEnC/YgOlQgFo2ilh1b/sl+uuFV3DrJKvJxUIEejX/7qDWKWRlwhk9SYrucm6GdUo2CST0u91PCEsjEd8q6likbc+Nn81Ck5s8qAhLG2pZDM1d8TGY2MmUSB7YwojsyyNxP/87ophtd+JlSSIldssShMJcGYzP4mA6E5QzmxhDIt7K2EjaimDG06JRuCt/zyKmldVD236t3XKvWbPI4inMApnIMHV1CHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gDa3Y2A</latexit><latexit sha1_base64="OZ7/eGjiaAd6iFmhE9KemdqLXmM=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6xEnC/YgOlQgFo2ilh1b/sl+uuFV3DrJKvJxUIEejX/7qDWKWRlwhk9SYrucm6GdUo2CST0u91PCEsjEd8q6likbc+Nn81Ck5s8qAhLG2pZDM1d8TGY2MmUSB7YwojsyyNxP/87ophtd+JlSSIldssShMJcGYzP4mA6E5QzmxhDIt7K2EjaimDG06JRuCt/zyKmldVD236t3XKvWbPI4inMApnIMHV1CHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gDa3Y2A</latexit>
V4 <latexit sha1_base64="RGbnccGESBHMfrtDKDZHJvcqDa8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmlfVT236t3XKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDcYY2B</latexit><latexit sha1_base64="RGbnccGESBHMfrtDKDZHJvcqDa8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmlfVT236t3XKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDcYY2B</latexit><latexit sha1_base64="RGbnccGESBHMfrtDKDZHJvcqDa8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmlfVT236t3XKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDcYY2B</latexit><latexit sha1_base64="RGbnccGESBHMfrtDKDZHJvcqDa8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmlfVT236t3XKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDcYY2B</latexit>
v2 <latexit sha1_base64="UQ8RRPt2NN02WZInLZUJLciw6fw=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbRqVc+teg9XlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwAKKI2f</latexit><latexit sha1_base64="UQ8RRPt2NN02WZInLZUJLciw6fw=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbRqVc+teg9XlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwAKKI2f</latexit><latexit sha1_base64="UQ8RRPt2NN02WZInLZUJLciw6fw=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbRqVc+teg9XlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwAKKI2f</latexit><latexit sha1_base64="UQ8RRPt2NN02WZInLZUJLciw6fw=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbRqVc+teg9XlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwAKKI2f</latexit>
v3 <latexit sha1_base64="fHYLD/bLdDeenuRAHuRzLEbLywI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8eK9gPaUDbbSbt0swm7m0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hobua3xqg0j+WTmSToR3QgecgZNVZ6HPcue+WKW3XnIKvEy0kFctR75a9uP2ZphNIwQbXueG5i/Iwqw5nAaambakwoG9EBdiyVNELtZ/NTp+TMKn0SxsqWNGSu/p7IaKT1JApsZ0TNUC97M/E/r5Oa8MbPuExSg5ItFoWpICYms79JnytkRkwsoUxxeythQ6ooMzadkg3BW355lTQvqp5b9R6uKrXbPI4inMApnIMH11CDe6hDAxgM4Ble4c0Rzovz7nwsWgtOPnMMf+B8/gALrI2g</latexit><latexit sha1_base64="fHYLD/bLdDeenuRAHuRzLEbLywI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8eK9gPaUDbbSbt0swm7m0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hobua3xqg0j+WTmSToR3QgecgZNVZ6HPcue+WKW3XnIKvEy0kFctR75a9uP2ZphNIwQbXueG5i/Iwqw5nAaambakwoG9EBdiyVNELtZ/NTp+TMKn0SxsqWNGSu/p7IaKT1JApsZ0TNUC97M/E/r5Oa8MbPuExSg5ItFoWpICYms79JnytkRkwsoUxxeythQ6ooMzadkg3BW355lTQvqp5b9R6uKrXbPI4inMApnIMH11CDe6hDAxgM4Ble4c0Rzovz7nwsWgtOPnMMf+B8/gALrI2g</latexit><latexit sha1_base64="fHYLD/bLdDeenuRAHuRzLEbLywI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8eK9gPaUDbbSbt0swm7m0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hobua3xqg0j+WTmSToR3QgecgZNVZ6HPcue+WKW3XnIKvEy0kFctR75a9uP2ZphNIwQbXueG5i/Iwqw5nAaambakwoG9EBdiyVNELtZ/NTp+TMKn0SxsqWNGSu/p7IaKT1JApsZ0TNUC97M/E/r5Oa8MbPuExSg5ItFoWpICYms79JnytkRkwsoUxxeythQ6ooMzadkg3BW355lTQvqp5b9R6uKrXbPI4inMApnIMH11CDe6hDAxgM4Ble4c0Rzovz7nwsWgtOPnMMf+B8/gALrI2g</latexit><latexit sha1_base64="fHYLD/bLdDeenuRAHuRzLEbLywI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8eK9gPaUDbbSbt0swm7m0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hobua3xqg0j+WTmSToR3QgecgZNVZ6HPcue+WKW3XnIKvEy0kFctR75a9uP2ZphNIwQbXueG5i/Iwqw5nAaambakwoG9EBdiyVNELtZ/NTp+TMKn0SxsqWNGSu/p7IaKT1JApsZ0TNUC97M/E/r5Oa8MbPuExSg5ItFoWpICYms79JnytkRkwsoUxxeythQ6ooMzadkg3BW355lTQvqp5b9R6uKrXbPI4inMApnIMH11CDe6hDAxgM4Ble4c0Rzovz7nwsWgtOPnMMf+B8/gALrI2g</latexit>
v4 <latexit sha1_base64="+c4psOKK6zsb24Fz44doEjLIVSY=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbSuqp5b9R5qlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwANMI2h</latexit><latexit sha1_base64="+c4psOKK6zsb24Fz44doEjLIVSY=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbSuqp5b9R5qlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwANMI2h</latexit><latexit sha1_base64="+c4psOKK6zsb24Fz44doEjLIVSY=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbSuqp5b9R5qlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwANMI2h</latexit><latexit sha1_base64="+c4psOKK6zsb24Fz44doEjLIVSY=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbSuqp5b9R5qlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwANMI2h</latexit>
v1 <latexit sha1_base64="iwb6UOVPBQMaW/UYkl1KcM9pxNQ=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0nf65crbtVdgKwTLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjks1IvNTyhbEyHvGupohE3frY4dUYurDIgYaxtKSQL9fdERiNjplFgOyOKI7PqzcX/vG6K4Y2fCZWkyBVbLgpTSTAm87/JQGjOUE4toUwLeythI6opQ5tOyYbgrb68TlpXVc+teg/XlfptHkcRzuAcLsGDGtThHhrQBAZDeIZXeHOk8+K8Ox/L1oKTz5zCHzifPwikjZ4=</latexit><latexit sha1_base64="iwb6UOVPBQMaW/UYkl1KcM9pxNQ=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0nf65crbtVdgKwTLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjks1IvNTyhbEyHvGupohE3frY4dUYurDIgYaxtKSQL9fdERiNjplFgOyOKI7PqzcX/vG6K4Y2fCZWkyBVbLgpTSTAm87/JQGjOUE4toUwLeythI6opQ5tOyYbgrb68TlpXVc+teg/XlfptHkcRzuAcLsGDGtThHhrQBAZDeIZXeHOk8+K8Ox/L1oKTz5zCHzifPwikjZ4=</latexit><latexit sha1_base64="iwb6UOVPBQMaW/UYkl1KcM9pxNQ=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0nf65crbtVdgKwTLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjks1IvNTyhbEyHvGupohE3frY4dUYurDIgYaxtKSQL9fdERiNjplFgOyOKI7PqzcX/vG6K4Y2fCZWkyBVbLgpTSTAm87/JQGjOUE4toUwLeythI6opQ5tOyYbgrb68TlpXVc+teg/XlfptHkcRzuAcLsGDGtThHhrQBAZDeIZXeHOk8+K8Ox/L1oKTz5zCHzifPwikjZ4=</latexit><latexit sha1_base64="iwb6UOVPBQMaW/UYkl1KcM9pxNQ=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0nf65crbtVdgKwTLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjks1IvNTyhbEyHvGupohE3frY4dUYurDIgYaxtKSQL9fdERiNjplFgOyOKI7PqzcX/vG6K4Y2fCZWkyBVbLgpTSTAm87/JQGjOUE4toUwLeythI6opQ5tOyYbgrb68TlpXVc+teg/XlfptHkcRzuAcLsGDGtThHhrQBAZDeIZXeHOk8+K8Ox/L1oKTz5zCHzifPwikjZ4=</latexit>
v3 <latexit sha1_base64="fHYLD/bLdDeenuRAHuRzLEbLywI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8eK9gPaUDbbSbt0swm7m0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hobua3xqg0j+WTmSToR3QgecgZNVZ6HPcue+WKW3XnIKvEy0kFctR75a9uP2ZphNIwQbXueG5i/Iwqw5nAaambakwoG9EBdiyVNELtZ/NTp+TMKn0SxsqWNGSu/p7IaKT1JApsZ0TNUC97M/E/r5Oa8MbPuExSg5ItFoWpICYms79JnytkRkwsoUxxeythQ6ooMzadkg3BW355lTQvqp5b9R6uKrXbPI4inMApnIMH11CDe6hDAxgM4Ble4c0Rzovz7nwsWgtOPnMMf+B8/gALrI2g</latexit><latexit sha1_base64="fHYLD/bLdDeenuRAHuRzLEbLywI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8eK9gPaUDbbSbt0swm7m0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hobua3xqg0j+WTmSToR3QgecgZNVZ6HPcue+WKW3XnIKvEy0kFctR75a9uP2ZphNIwQbXueG5i/Iwqw5nAaambakwoG9EBdiyVNELtZ/NTp+TMKn0SxsqWNGSu/p7IaKT1JApsZ0TNUC97M/E/r5Oa8MbPuExSg5ItFoWpICYms79JnytkRkwsoUxxeythQ6ooMzadkg3BW355lTQvqp5b9R6uKrXbPI4inMApnIMH11CDe6hDAxgM4Ble4c0Rzovz7nwsWgtOPnMMf+B8/gALrI2g</latexit><latexit sha1_base64="fHYLD/bLdDeenuRAHuRzLEbLywI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8eK9gPaUDbbSbt0swm7m0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hobua3xqg0j+WTmSToR3QgecgZNVZ6HPcue+WKW3XnIKvEy0kFctR75a9uP2ZphNIwQbXueG5i/Iwqw5nAaambakwoG9EBdiyVNELtZ/NTp+TMKn0SxsqWNGSu/p7IaKT1JApsZ0TNUC97M/E/r5Oa8MbPuExSg5ItFoWpICYms79JnytkRkwsoUxxeythQ6ooMzadkg3BW355lTQvqp5b9R6uKrXbPI4inMApnIMH11CDe6hDAxgM4Ble4c0Rzovz7nwsWgtOPnMMf+B8/gALrI2g</latexit><latexit sha1_base64="fHYLD/bLdDeenuRAHuRzLEbLywI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8eK9gPaUDbbSbt0swm7m0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hobua3xqg0j+WTmSToR3QgecgZNVZ6HPcue+WKW3XnIKvEy0kFctR75a9uP2ZphNIwQbXueG5i/Iwqw5nAaambakwoG9EBdiyVNELtZ/NTp+TMKn0SxsqWNGSu/p7IaKT1JApsZ0TNUC97M/E/r5Oa8MbPuExSg5ItFoWpICYms79JnytkRkwsoUxxeythQ6ooMzadkg3BW355lTQvqp5b9R6uKrXbPI4inMApnIMH11CDe6hDAxgM4Ble4c0Rzovz7nwsWgtOPnMMf+B8/gALrI2g</latexit>
v1 <latexit sha1_base64="iwb6UOVPBQMaW/UYkl1KcM9pxNQ=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0nf65crbtVdgKwTLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjks1IvNTyhbEyHvGupohE3frY4dUYurDIgYaxtKSQL9fdERiNjplFgOyOKI7PqzcX/vG6K4Y2fCZWkyBVbLgpTSTAm87/JQGjOUE4toUwLeythI6opQ5tOyYbgrb68TlpXVc+teg/XlfptHkcRzuAcLsGDGtThHhrQBAZDeIZXeHOk8+K8Ox/L1oKTz5zCHzifPwikjZ4=</latexit><latexit sha1_base64="iwb6UOVPBQMaW/UYkl1KcM9pxNQ=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0nf65crbtVdgKwTLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjks1IvNTyhbEyHvGupohE3frY4dUYurDIgYaxtKSQL9fdERiNjplFgOyOKI7PqzcX/vG6K4Y2fCZWkyBVbLgpTSTAm87/JQGjOUE4toUwLeythI6opQ5tOyYbgrb68TlpXVc+teg/XlfptHkcRzuAcLsGDGtThHhrQBAZDeIZXeHOk8+K8Ox/L1oKTz5zCHzifPwikjZ4=</latexit><latexit sha1_base64="iwb6UOVPBQMaW/UYkl1KcM9pxNQ=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0nf65crbtVdgKwTLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjks1IvNTyhbEyHvGupohE3frY4dUYurDIgYaxtKSQL9fdERiNjplFgOyOKI7PqzcX/vG6K4Y2fCZWkyBVbLgpTSTAm87/JQGjOUE4toUwLeythI6opQ5tOyYbgrb68TlpXVc+teg/XlfptHkcRzuAcLsGDGtThHhrQBAZDeIZXeHOk8+K8Ox/L1oKTz5zCHzifPwikjZ4=</latexit><latexit sha1_base64="iwb6UOVPBQMaW/UYkl1KcM9pxNQ=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0nf65crbtVdgKwTLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjks1IvNTyhbEyHvGupohE3frY4dUYurDIgYaxtKSQL9fdERiNjplFgOyOKI7PqzcX/vG6K4Y2fCZWkyBVbLgpTSTAm87/JQGjOUE4toUwLeythI6opQ5tOyYbgrb68TlpXVc+teg/XlfptHkcRzuAcLsGDGtThHhrQBAZDeIZXeHOk8+K8Ox/L1oKTz5zCHzifPwikjZ4=</latexit>
v4 <latexit sha1_base64="+c4psOKK6zsb24Fz44doEjLIVSY=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbSuqp5b9R5qlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwANMI2h</latexit><latexit sha1_base64="+c4psOKK6zsb24Fz44doEjLIVSY=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbSuqp5b9R5qlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwANMI2h</latexit><latexit sha1_base64="+c4psOKK6zsb24Fz44doEjLIVSY=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbSuqp5b9R5qlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwANMI2h</latexit><latexit sha1_base64="+c4psOKK6zsb24Fz44doEjLIVSY=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbSuqp5b9R5qlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwANMI2h</latexit>
v2 <latexit sha1_base64="UQ8RRPt2NN02WZInLZUJLciw6fw=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbRqVc+teg9XlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwAKKI2f</latexit><latexit sha1_base64="UQ8RRPt2NN02WZInLZUJLciw6fw=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbRqVc+teg9XlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwAKKI2f</latexit><latexit sha1_base64="UQ8RRPt2NN02WZInLZUJLciw6fw=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbRqVc+teg9XlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwAKKI2f</latexit><latexit sha1_base64="UQ8RRPt2NN02WZInLZUJLciw6fw=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbRqVc+teg9XlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwAKKI2f</latexit> Figure 3: Graphical models and infor-
mation diagrams (inf) associated with the core view and full graph paradigms, for the case of 4 views, which gives a total of 6 learning objectives. The numbers within the regions show how much “weight” the total loss places on each partition of information (i.e. how many of the 6 objectives that partition contributes to). A region with no number corresponds to 0 weight. For example, in the full graph case, the mutual information between all 4 views is considered in all 6 objectives, and hence is marked with the number 6.
Implementing the critic We implement the critic hθ(·) as a neural network. To extract compact latent representations of v1 and v2, we employ two encoders fθ1(·) and fθ2(·) with parameters θ1 and θ2 respectively. The latent representions are extracted as z1 = fθ1(v1), z2 = fθ2(v2). On top of these features, the score is computed as the exponential of a bivariate function of z1 and z2, e.g., a bilinear function parameterized by W12:
hθ({v1, v2}) = efθ1 (v1) TW12fθ2 (v2) (3)
Loss LV1,V2contrast in Eq. 2 treats view V1 as anchor and enumerates over V2. Symmetrically, we can get LV2,V1contrast by anchoring at V2. We add them up as our two-view loss:
L(V1, V2) = LV1,V2contrast + LV2,V1contrast (4)
After the contrastive learning phase, we use the representation z1, z2, or the concatenation of both, [z1, z2], depending on our paradigm. This process is visualized in Fig. 1.
Connecting to mutual information The optimal critic h∗θ is proportional to the density ratio between the joint distribution p(z1, z2) and the product of marginals p(z1)p(z2) (proof provided in Sec. C.1):
h∗θ({v1, v2}) ∝ p(z1, z2) p(z1)p(z2) ∝ p(z1|z2) p(z1) (5)
This quantity is the pointwise mutual information, and its expectation, in Eq. 2, yields an estimator related to mutual information. A formal proof is given by Oord et al. (2018); Poole et al. (2019), which we recapitulate in Section C, showing that:
I(zi; zj) ≥ log(k)− Lcontrast (6) where, as above, k is the number of negative pairs in sample set S. Hence minimizing the objective L maximizes the lower bound on the mutual information I(zi; zj), which is bounded above by I(vi; vj) by the data processing inequality. The dependency on k also suggests that using more negative samples can lead to an improved representation; we show that this is indeed the case in Section A.1.2. We note that recent work (McAllester & Statos, 2018) shows that the bound in Eq. 6 can be very weak; and finding better estimators of mutual information is an important open problem.

heading: 2.3 CONTRASTIVE LEARNING WITH MORE THAN TWO VIEWS
text: We present more general formulations of Eq. 2 that can handle any number of views. We call them the “core view” and “full graph” paradigms, which offer different tradeoffs between efficiency and effectiveness. These formulations are visualized in Fig. 3.
Suppose we have a collection of M views V1, . . . , VM . The “core view” formulation sets apart one view that we want to optimize over, say V1, and builds pair-wise representations between V1 and each
other view Vj , j > 1, by optimizing the sum of a set of pair-wise objectives:
LC = M∑
j=2
L(V1, Vj) (7)
A second, more general formulation is the “full graph” where we consider all pairs (i, j), i 6= j, and build ( n 2 ) relationships in all. By involving all pairs, the objective function that we optimize is:
LF = ∑
1≤i<j≤M
L(Vi, Vj) (8)
Both these formulations have the effect that information is prioritized in proportion to the number of views that share that information. This can be seen in the information diagrams visualized in
Fig. 3. The number in each partition of the diagram indicates how many of the pairwise objectives, L(Vi, Vj), that partition contributes to. Under both the core view and full graph objectives, a factor, like “presence of dog”, that is common to all views will be preferred over a factor that affects fewer views, such as “depth sensor noise”.
The computational cost of the bivariate score function in the full graph formulation is combinatorial in the number of views. However, it is clear from Fig. 3 that this enables the full graph formulation to capture more information between different views, which may prove useful for downstream tasks. For example, the mutual information between V2 and V3 or V2 and V4 is completely ignored in the core view paradigm (as shown by a 0 count in the information diagram).

heading: 2.4 APPROXIMATING THE SOFTMAX DISTRIBUTION WITH NOISE-CONTRASTIVE ESTIMATION
text: Better representations using LV1,V2contrast in Eq. 2 are learnt by using many negative samples. However, computing the full softmax loss is prohibitively expensive for large N . We alleviate computational load by using Noise-Contrastive Estimation (NCE) (Gutmann & Hyvärinen, 2010) to approximate the full softmax in Eqn. 2, as has also been used in Mnih & Kavukcuoglu (2013)1.
Given an anchor vi1 from V1, the probablity that an atom v2 ∈ {vj2|j = 1, 2, ..., N} from V2 is the best match of vi1, using the score hθ is given by:
p(v2|vi1) = hθ({vi1, v2})∑N j=1 hθ({vi1, v j 2})
(9)
where the normalization factor Z = ∑N j=1 hθ({vi1, v j 2}) is expensive to compute for large N . Here we use hθ({v1, v2}) = exp(fθ1(v1)TW12fθ2(v2)/τ), where τ modulates the distribution. Noise-Contrastive Estimation (Gutmann & Hyvärinen, 2010) (NCE) is an effective way to estimate unnormalized statistical models. NCE fits a density model p to data distributed as (unknown) distribution pd, by using a binary classifier to distinguish it from noise samples distributed as pn. To learn p(v2|vi1), we use a binary classifier, which treats v2 as the data sample when given vi1. The noise distribution pn(·|vi1) we choose here is a uniform distribution over all atoms from V2, i.e., pn(·|vi1) = 1/N . If we sample m noise samples to pair with each data sample, the posterior probability that a given atom v2 comes from the data distribution is:
P (D = 1|v2; vi1) = pd(v2|vi1)
pd(v2|vi1) +mpn(v2|vi1) (10)
and we estimate this probability by replacing pd(v2|vi1) with our unnormalized model distribution hθ(v i 1, v2). Minimizing the negative log-posterior probability of correct labels D over data and noise samples yields our final objective, which is the NCE-based approximation of Eq. 2 (p̂ is the empirical data distribution):
LNCE = − E vi1∼p̂(v1) { E v2∼p̂(·|vi1)
[ log(P (D = 1|v2; vi1)) ] +m E
v2∼pn(·|vi1)
[ log(P (D = 0|v2; vi1)) ] } (11)
1Confusingly, the literature has previously referred to Eqn. 2 as “InfoNCE” (Oord et al., 2018). Our NCE approximation does not refer to the allusion to NCE in the name “InfoNCE”. Rather we are here describing an NCE approximation to the “InfoNCE” softmax objective.
Memory bank. Following Wu et al. (2018), we maintain a memory bank to store latent features for each training sample. Therefore, we can efficiently retrieve m noise samples from the memory bank to pair with each positive sample without recomputing their features. The memory bank is dynamically updated with features computed on the fly.
An alternative to the NCE based approximation above, is to simply dom+1 way softmax classification with m noise samples retrieved from the memory bank. We note that CPC (Oord et al., 2018) and Deep InfoMax (Hjelm et al., 2019) use this m + 1 way softmax classification as their ultimate contrastive loss rather than the NCE-based contrastive loss in Eq. 11 (but note that CPC refers to the m+ 1 approximation as also “based on NCE”). Empirically we have found that the m+ 1-way softmax classification approach performed worse than our NCE-based approximation, given the same number of noise samples.

heading: 3 EXPERIMENTS
text: We extensively evaluate Contrastive Multiview Coding (CMC) on a number of datasets and tasks. We evaluate on two established image representation learning benchmarks: Imagenet and STL-10 (See A.1.1). We further validate our framework on video representation learning tasks (See A.2), where we use image and optical flow modalities, as the two views that are jointly learned. The last set of experiments extends our CMC framework to more than two views and provides empirical evidence of it’s effectiveness.

heading: 3.1 BENCHMARKING CMC ON IMAGENET
text: Following Zhang et al. (2016), we evaluate task generalization of the learned representation by training 1000-way linear classifiers on top of different layers.
Setup. Given a dataset of RGB images, we convert them to the Lab image color space, and split each image into L and ab channels, as originally proposed in SplitBrain autoencoders (Zhang et al., 2017). During contrastive learning, L and ab from the same image are treated as the positive pair, and ab channels from other randomly selected images are treated as a negative pair (for a given L). Each split represents a view of the orginal image and is passed through a separate encoder. As in SplitBrain, we design these two encoders by evenly splitting a given deep network, such as AlexNet (Krizhevsky et al., 2012), into sub-networks across the channel dimension. By concatenating representations layer-wise from these two encoders, we achieve the final representation of an input image. As proposed by previous literature (Oord et al., 2018; Hjelm et al., 2019; Arora et al., 2019), the quality of such a representation is evaluated by freezing the weights of encoder and training linear or non-linear classifiers on top of each layer.
To compare with other methods, we adopt standard AlexNet and split it into two encoders. Because of splitting, each layer only connects to half of the neurons in the previous layer, and therefore the number of parameters in our model halves. We remove local response layer and add batch normalization to each layer. For the memory-based CMC model, we adopt ideas from Wu et al. (2018) for computing and storing a memory. We retrieve 4096 negative pairs from the memory bank to contrast each positive pair. The training details are present in Sec. D.2.
Table 1 shows the results of comparing the CMC against other models, both predictive and contrastive. Our CMC is the best among all these methods; futhermore CMC tends to perform better at higher convolutional layers, similar to another contrasting-based model Inst-Dis (Wu et al., 2018).
CMC with ResNets. We verify the scalability of CMC with larger networks such as ResNets He et al. (2016). Here we do not split ResNets, rather we use ResNet-50, ResNet-101 or ResNet-50 x2 to encoder each of the two views (L and ab). The results are shown in Table 2, where ResNet-50, ResNet-101, and ResNet-50 x2 achieve 64.1%, 65.0%, and 68.4% top-1 accuracies, respectively. To our best knowledge, 68.4% on ImageNet is the highest published accuracy ever achieved by self-supervsied/unsupervised methods (We note that a concurrent work AMDIM Bachman et al. (2019) achieves similar results).

heading: 3.2 DOES REPRESENTATION QUALITY IMPROVE AS NUMBER OF VIEWS INCREASES?
text: We further extend our CMC learning framework to multiview scenarios. We experiment on the NYU-Depth-V2 (Nathan Silberman & Fergus, 2012) dataset which consists of 1449 labeled images. We focus more on understanding the behavior and effectiveness of CMC rather than competing with the current state-of-the-arts. The views we consider are: luminance (L channel), chrominance (ab channel), depth, surface normal (Eigen & Fergus, 2015), and semantic labels.
Setup. To extract features from each view, we use a neural network with 5 convolutional layers, and 1 fully connected layer. As the size of the dataset is relatively small, we adopt the sub-patch based contrastive objective (see B) to increase the number of negative pairs. Patches with a size of 128× 128 are randomly cropped from the original images for contrastive learning (from images of size 480× 640). For downstream tasks, we discard the fully connected layers and evaluate using the convolutional layers as a representation.
To measure the quality of the learned representation, we consider the task of predicting semantic labels from the representation of L. We follow the core view paradigm and use L are the core view, thus learning a set of representations on L by contrasting different views with L. A UNet style architecture (Ronneberger et al., 2015) is utilized to perform the segmentation task. Contrastive training is performed on the above architecture that is equivalent of the UNet’s encoder. After contrastive training is completed, we initialize the encoder weights of the UNet from the L encoder (which are equivalent architectures) and keep them frozen. Only the decoder is trained during this finetuning stage.
Since we use the patch-based contrastive loss, in the 1 view setting case, CMC coincides with DIM (Hjelm et al., 2019). The 2-4 view cases contrast L with ab, and then sequentially add depth and surface normals. The semantic labeling results are measured by mean IoU over all classes and pixel accuracy, shown in Fig. 4. We see that the performance steadily improves as new views are added. We have tested different orders of adding the views, and they all follow a similar pattern.
We also compare CMC with two baselines. First, we randomly initialize and freeze the encoder, and we call this the Random baseline; it serves as a lower bound on the quality since the representation is just a random projection. Rather than freezing the randomly initialized encoder, we could train it
Pixel Accuracy (%) mIoU (%)
Random 45.5 21.4 CMC (core-view) 57.1 34.1 CMC (full-graph) 57.0 34.4 Supervised 57.8 35.9
Table 3: Results on the task of predicting semantic labels from L channel representation which is learnt using the patch-based contrastive loss and all 4 views. We compare CMC with Random and Supervised baselines, which serve as lower and upper bounds respectively. Th core-view paradigm refers to
Fig. 3(a), and full-view Fig. 3(b).
jointly with the decoder. This end-to-end Supervised baseline serves as an upper bound. The results are presented in Table 3, which shows our CMC produces high quality feature maps even though it’s unaware of the downstream task.

heading: 3.3 PREDICTIVE LEARNING VS. CONTRASTIVE LEARNING
text: While experiments in section 3.1 show that contrastive learning outperforms predictive learning (Zhang et al., 2017) in the context of Lab color space, it’s unclear whether such an advantage is due to the natural inductive bias of the task itself. To further understand this, we go beyond chrominance (ab), and try to answer this question when geometry or semantic labels are present.
We consider three view pairs on the NYU-Depth dataset: (1) L and depth, (2) L and surface normals, and (3) L and segmentation map. For each of them, we train two identical encoders for L, one using contrastive learning and the other with predictive learning. We then evaluate the representation quality by training a linear classifier on top of these encoders on the STL-10 dataset.
The comparison results are shown in Table 4, which shows that contrastive learning consistently outperforms predictive learning in this scenario where both the task and the dataset are unknown. We also include “random” and “supervised” baselines similar to that in previous sections. Though in the unsupervised stage we only use 1.3K images from a dataset much different from the target dataset STL-10, the object recognition accuracy is close to the supervised method, which uses an end-to-end deep network directly trained on STL-10.
Given two views V1 and V2 of the data, the predictive learning approach approximately models p(v2|v1). Furthermore, losses used typically for predictive learning, such as pixel-wise reconstruction losses usually impose an independence assumption on the modeling: p(v2|v1) ≈ Πip(v2i|v1). On the other hand, the contrastive learning approach by construction does not assume conditional
independence across dimensions of v2. In addition, the use of random jittering and cropping between views allows the contrastive learning approach to benefit from spatial co-occurrence (contrasting in space) in addition to contrasting across views. We conjecture that these are two reasons for the superior performance of contrastive learning approaches over predictive learning.

heading: 3.4 HOW DOES MUTUAL INFORMATION AFFECT REPRESENTATION QUALITY?
text: Given a fixed set of views, CMC aims to maximize the mutual information between representations of these views. We have found that maximizing information in this way indeed results in strong representations, but it would be incorrect to infer that information maximization (infomax) is the key to good representation learning. In fact, this paper argues for precisely the opposite idea: that cross-view representation learning is effective because it results in a kind of information minimization, discarding nuisance factors that are not shared between the views.
The resolution to this apparent dilemma is that we want to maximize the “good” information – the signal – in our representations, while minimizing the “bad” information – the noise. The idea behind CMC is that this can be achieved by doing infomax learning on two views that share signal but have independent noise. This suggests a “Goldilocks principle”: a good collection of views is one that shares some information but not too much. Here we test this hypothesis on two domains: learning representations on images with different colorspaces forming the two views; and learning representations on pairs of patches extracted from an image, separated by varying spatial distance.
nuisances are discarded and we arrive back at something akin to an autoencoder or generative model, that simply tries to represent all the bits in the multiview data.
These experiments demonstrate that the relationship between mutual information and representation quality is meaningful but not direct. Selecting optimal views, which just share relevant signal, may be a fruitful direction for future research.

heading: 4 RELATED WORK
text: Unsupervised representation learning is about learning transformations of the data that make subsequent problem solving easier (Bengio et al., 2013). This field has a long history, starting with classical methods with well established algorithms, such as principal components analysis (PCA (Jolliffe, 2011)) and independent components analysis (ICA (Hyvärinen et al., 2004)). These methods tend to learn representations that focus on low-level variations in the data, which are not very useful from the perspective of downstream tasks such as object recognition.
Representations better suited to such tasks have been learnt using deep neural networks, starting with seminal techniques such as Boltzmann machines (Smolensky, 1986; Salakhutdinov & Hinton, 2009), autoencoders (Hinton & Salakhutdinov, 2006), variational autoencoders (Kingma & Welling, 2013), generative adversarial networks (Goodfellow et al., 2014) and autoregressive models (Oord et al., 2016). Numerous other works exist, for a review see (Bengio et al., 2013). A powerful family of models for unsupervised representations are collected under the umbrella of “self-supervised” learning (Sa, 2004; Zhang et al., 2017; 2016; Isola et al., 2015; Wang & Gupta, 2015; Pathak et al., 2016; Zhang et al., 2019). In these models, an input X to the model is transformed into an output X̂ , which is supposed to be close to another signal Y , which itself is related to X in some meaningful way. Examples of such X/Y pairs are: luminance and chrominance color channels of an image (Zhang et al., 2017), patches from a single image (Oord et al., 2018), modalities such as vision and sound (Owens et al., 2016) or the frames of a video (Wang & Gupta, 2015). Clearly, such examples are numerous in the world, and provides us with nearly infinite amounts of training data: this is one of the appeals of this paradigm. Time contrastive networks (Sermanet et al., 2017) use a triplet loss framework to learn representations from aligned video sequences of the same scene, taken by different video cameras. Closely related to self-supervised learning is the idea of multi-view learning, which is a general term involving many different approaches such as co-training (Blum & Mitchell, 1998), multi-kernel learning (Cortes et al., 2009) and metric learning (Bellet et al., 2012; Zhuang et al., 2019); for comprehensive surveys please see (Xu et al., 2013; Li et al., 2018). Nearly all existing works have dealt with one or two views such as video or image/sound. However, in many situations, many more views are available to provide training signals for any representation.
The objective functions used to train deep learning based representations in many of the above methods are either reconstruction-based loss functions such as Euclidean losses in different norms e.g. (Isola et al., 2017), adversarial loss functions (Goodfellow et al., 2014) that learn the loss in addition to the representation, or contrastive losses e.g. (Gutmann & Hyvärinen, 2010; Hjelm et al., 2019; Oord et al., 2018; Arora et al., 2019; Hénaff et al., 2019) that take advantage of the co-occurence of multiple views. Another recently introduced novel objective function is instance discrimination (Wu et al., 2018). In this work, we compare the two most commonly used objectives: predictive and contrastive. The prior works most similar to our own (and inspirational to us) are Contrastive Predictive Coding (CPC) (Oord et al., 2018) and Deep InfoMax (Hénaff et al., 2019). These two methods, like ours, learn representations by contrasting between congruent and incongruent representations of a scene, and are motivated as forms of infomax learning. CPC learns from two views – the past and future – and is applicable to sequential data. Deep Infomax (Hjelm et al., 2019) considers the two views to be the input to a neural network and its output. These two methods share the same mathematical objective, but differ in the definition of the views. Our technical method is also highly related, but differs in the following ways: we extend the objective to the case of more than two views; and we use a loss function which more closely follows the original method of noise contrastive estimation (Gutmann & Hyvärinen, 2010) (See details in Section 2.4). Although CPC, Deep InfoMax, and the present paper are all very similar at the mathematical level, they each explore a different set of view definitions, architectures, and application settings, and each contributes its own unique empirical investigation of this paradigm of representation learning.

heading: A ADDITIONAL EXPERIMENTS

heading: A.1 CMC ON IMAGES
text: Given a dataset of RGB images, we convert them to the Lab image color space, and split each image into L and ab channels, as originally proposed in SplitBrain autoencoders (Zhang et al., 2017). During contrastive learning, L and ab from the same image are treated as the positive pair, and ab channels from other randomly selected images are treated as a negative pair (for a given L). Each split represents a view of the orginal image and is passed through a seprate encoder. This corresponds to the “full graph” model of Eq. 8 with L and ab channels as the two views. As in SplitBrain, we design these two encoders by evenly splitting a given deep network, such as AlexNet (Krizhevsky et al., 2012), into sub-networks across the channel dimension. By concatenating representations layer-wise from these two encoders, we achieve the final representation of an input image. As proposed by previous literature (Oord et al., 2018; Hjelm et al., 2019; Arora et al., 2019), the quality of such a representation is evaluated by freezing the weights of encoder and training linear or non-linear classifiers on top of each layer.
A.1.1 STL-10
STL-10 (Coates et al., 2011) is an image recognition dataset designed for developing unsupervised or self-supervised learning algorithms. It consists of 100000 unlabeled training 96× 96 RGB image samples and 500 labeled samples for each of the 10 classes.
Setup. We adopt the same data augmentation strategy and network architecture as those in DIM (Hjelm et al., 2019). A variant of AlexNet takes as input 64 × 64 images, which are randomly cropped and horizontally flipped from the original 96× 96 size images. For a fair comparison with DIM, we also train our model in a patch-based contrastive fashion during unsupervised pretraining. With the weights of the pre-trained encoder frozen, a two-layer fully connected network with 200 hidden units is trained on top of different layers for 100 epochs to perform 10-way classification. We also investigated the strided crop strategy of CPC (Oord et al., 2018). Fixed sized overlapping patches of size 16× 16 with an overlap of 8 pixels are cropped and fed into the network separately. This ensures that features of one patch contain minimal information from neighbouring patches; and increases the available number of negative pairs for the contrastive loss. Additionally, we include NCE-based contrastive training and linear classifier evaluation.
Comparison. We compare CMC with the state of the art unsupervised methods in Table 5. Three columns are shown: the conv5 and fc7 columns use respectively these layers of AlexNet as the encoder (again remembering that we split across channels for L and ab views). For these two columns we can compare against the all methods except CPC, since CPC does not report these numbers in their paper (Hjelm et al., 2019). In the Strided Crop setup, we only compare against the approaches that use contrastive learning, DIM and CPC, since this method was only used by those works. We note that in Table 5 for all the methods except SplitBrain, we report numbers are shown in the original paper. For SplitBrain, we reimplemented their model faithfully and report numbers based on our reimplementation (we verified the accuracy of our SplitBrain code by the fact that we get very similar results with our reimpementation as in the original paper (Zhang et al., 2017) for ImageNet experiments, see below).
The family of contrastive learning methods, such as DIM, CPC, and CMC, achieve higher classification accuracy than other methods such as SplitBrain that use predictive learning; or BiGAN that use adversarial learning. CMC significantly outperforms DIM and CPC in all cases. We hypothesize that this outperformance results from the modeling of cross-view mutual information, where view-specific noisy details are discarded. Another head-to-head comparison happens between CMC and SplitBrain, both of which modeling images as seprated L and ab streams; we achieve a nearly 8% absolute improvement for conv5 and 17% improvement for fc5. Finally, we notice that the predictive learning methods suffer from a big drop in performance when the encoding layer is switched from conv5 to fc7. On the other hand, the contrastive learning approaches are much more stable across layers, suggesting that the mutual information maximization paradigm learns more semantically meaningful representations shared by the different views. From a practical perspective, this is a significant advantage as the selection of specific layers should ideally not change downstream performance by too much.
In this experiments we used AlexNet as backbone. Switching to more powerful networks such as ResNets is likely to further improve the representation quality.
A.1.2 IMAGENET
ImageNet (Deng et al., 2009) consists of 1000 image classes and is frequently considered as a testbed for unsupervised representation learning algorithms.
Effect of the number of negative samples. We investigate the relationship between the number of negative pairs m in NCE-based loss and the downstream classification accuracy on a randomly chosen subset of 100 classes of Imagenet (the same set of classes is used for any number of negative pairs). We train a 100-way linear classifier using CMC pre-trained features with varying number of negative pairs, starting from 64 pairs upto 8192 (in multiples of 2). Fig. 6 shows that the accuracy of the resulting classifier steadily increases but saturates at around 60.3% with m = 4096 samples. AlexNet is used in this study.

heading: A.2 CMC ON VIDEOS
text: We apply CMC on videos by drawing insight from the two-streams hypothesis (Schneider, 1969; Goodale & Milner, 1992), which posits that human visual cortex consists of two distinct processing streams: the ventral stream, which performs object recognition, and the dorsal stream, which processes motion. In our formulation, given an image it that is a frame centered at time t, the ventral stream associates it with a neighbouring frame it+k, while the dorsal stream connects it to optical flow ft centered at t. Therefore, we extract it, it+k and ft from two modalities as three views of a video; for optical flow we use the TV-L1 algorithm (Zach et al., 2007). Two separate contrastive learning objectives are built within the ventral stream (it, it+k) and within the dorsal stream (it, ft). For the ventral stream, the negative sample for it is chosen as a random frame from another randomly chosen video; for the dorsal stream, the negative sample for it is chosen as the flow corresponding to a random frame in another randomly chosen video.
Pre-training. We train CMC on UCF101 (Soomro et al., 2012) and use two CaffeNets (Krizhevsky et al., 2012) for extracting features from images and optical flows, respectively. In our implementation, ft represents 10 continuous flow frames centered at t. We use batch size of 128 and contrast each positive pair with 127 negative pairs. CMC is trained with Adam for 300 epochs, with an initial learning rate of 0.001 which is decayed by a factor of 5 after 200 and 250 epochs.
Action recognition. We apply the learn representation to the task of action recognition. The spatial network from (Simonyan & Zisserman, 2014) is a well-established paradigm for evaluating pre-trained RGB network on action recognition task. We follow the same spirit and evaluate the transferability of our RGB CaffeNet on UCF101 and HMDB51 datasets. We initialize the action recognition CaffeNet up to conv5 using the weights from the pre-trained RGB CaffeNet. The averaged accuracy over three splits is present in Table 6. Unifying both ventral and dorsal streams during pre-training produces higher accuracy for downstream recognition than using only single stream. Increasing the number
of views of the data from 2 to 3 (using both streams instead of one) provides a boost for UCF-101. Furthermore, on UCF-101, we outperform all other methods; and on HMDB-51, CMC is second-best in performance.
A.3 IS CMC IMPROVING ALL VIEWS?
A desirable unsupervised representation learning algorithm operating on multiple views or modalities should improve the quality of representations for all views. We therefore investigate our CMC framwork beyond L channel. To treat all views fairly, we train these encoders following the full graph paradigm, where each view is contrasted with all other views.
We evaluate the representation of each view v by predicting the semantic labels from only the representation of v, where v is L, ab, depth or surface normals. This uses the full-graph paradigm. As in the previous section, we compare CMC with Random and Supervised baselines. As shown in Table 7, the performance of the representations learned by CMC using full-graph significantly outperforms that of randomly projected representations, and approaches the performance of the fully supervised representations. Furthermore, the full-graph representation provides a good representation learnt for all views, showing the importance of capturing different types of mutual information across views.

heading: B CONTRASTING SUB-PATCHES
text: Instead of contrasting features from the last layer, patch-based method (Hjelm et al., 2019) contrasts feature from the last layer with features from previous layers, hence increasing the number of negative pairs. For instance, we use features from the last layer of fθ1 to contrast with feature points from feature maps produced by the first several conv layers of fθ2 . This is equivalent to contrast between global patch from one view with local patches from the other view. In this fashion, we directly
perform m+ 1 way softmax classification, the same as (Oord et al., 2018; Hjelm et al., 2019) for a fair comparison in Sec. A.1.1.
Such patch-based contrastive loss is computed within each mini-batch and does not require a memory bank. Therefore, deploying it in parallel training schemes is easy and flexible. However, patch-based contrastive loss usually yields suboptimal results compared to NCE-based contrastive loss, according to our experiments.

heading: C PROOFS
text: We prove that: (a) the optimal score function h∗θ({v1, v2}) is proportional to density ratio between the joint distribution p(v1, v2) and product of marginals p(v1)p(v2), as shown in Eq. 5; (b) Minimizing the contrastive loss Lcontrast maxmizes a lower bound on the mutual information between two views, as shown in Eq. 6
We will use the most general formula of contrastive loss Lcontrast shown in Eq. 1 for our derivation. But we note that replacing Lcontrast with LV1,V2contrast is straightforward. The overall proof follows a similar derivation introduced in (Oord et al., 2018).

heading: C.1 SCORE FUNCTION AS DENSITY RATIO ESTIMATOR
text: We first show that the optimal score function h∗θ({v1, v2}) that minimizes Eq. 1 is proportional to the density ratio between joint distribution and product of marginals, shown as Eq. 5. For notation convenience, we denote p(v1, v2) as data distribution pd(·) and p(v1)p(v2) as noise distribution pn(·). The loss in Eq. 1 is indeed a cross-entropy loss of classifying the correct positive pair out from the given set S. Without loss of generality, we assume the first pair (v01 , v 0 2) in S is positive or congruent and all others (vi1, v i 2), i = 1, 2, ..., k are negative or incongruent. The optimal probability for the loss, p(pos = 0|S), should depict the fact that (v01 , v02) comes from the data distribution pd(·) while all other pairs come from the noise distribution pn(·). Therefore,
p(pos = 0|S) = pd(v 0 1 , v
0 2) ∏k i=1 pn(v i 1, v
i 2)∑k
j=0 pd(v j 1, v j 2) ∏ i 6=j pn(v i 1, v i 2)
= p(v01 , v
0 2) ∏k i=1 p(v i 1)p(v
i 2)∑k
j=0 p(v j 1, v j 2) ∏ i 6=j p(v i 1)p(v i 2)
=
p(v01 ,v 0 2)
p(v01)p(v 0 2)∑k
j=0 p(vk1 ,v k 2 )
p(vk1 )p(v k 2 )
where we plug in the definition of pd(·) and pn(·), and divide ∏k i=0 p(v i 1)p(v 2 2) for both the numerator and denominator. By comparing above equation with the loss function in Eq. 1, we can see that the optimal score function h∗θ({v1, v2}) is proportional to the density ratio p(v1,v2)p(v1)p(v2) . The above derivation is agnostic to which layer the score function starts from, e.g., h can be defined on either the raw input (v1, v2) or the latent representation (z1, z2). As we care more about the property of the latent representation, for the following derivation we will use h∗W12({z1, z2}), which is proportional to p(z1,z2)p(z1)p(z2) .

heading: C.2 MAXIMIZING LOWER BOUND ON MI
text: Now we substitute the score function in Eq. 1 with the above density ratio, and the optimal loss objective Loptcontrast becomes:
Loptcontrast = −E S log [ h∗W12({z01 , z02})∑k i=0 h ∗ W12 ({zi1, zi2}) ]
= −E S log
 
p(z01 ,z 0 2)
p(z01)p(z 0 2)∑k
i=0 p(zi1,z i 2)
p(zi1)p(z i 2)
 
= E S log
[ 1 + p(z01)p(z 0 2)
p(z01 , z 0 2)
k∑
i=1
p(zi1, z i 2)
p(zi1)p(z i 2)
]
≈ E S log
[ 1 + p(z01)p(z 0 2)
p(z01 , z 0 2)
k E z1 [ p(z1|z2) p(z1) ]]
= E S log
[ 1 + p(z01)p(z 0 2)
p(z01 , z 0 2)
k
]
≥ log(k)− E S log
[ p(z01 , z 0 2)
p(z01)p(z 0 2)
]
= log(k)− E (z1,z2)∼pz1,z2 (·) log
[ p(z1, z2)
p(z1)p(z2)
]
= log(k)− I(z1; z2)
Therefore, for any two views Vi and Vj , we have I(zi; zj) ≥ log(k) − Loptcontrast(Vi, Vj). As the k increases, the approximation step becomes more accurate. Given any k, minimizing Lk(Vi, Vj) maximizes the lower bound on the mutual information I(zi; zj). We should note that increasing k to infinity does not always lead to a higher lower bound. While log(k) increases with a larger k, the optimization problem becomes harder and Lk(Vi, Vj) also increases.
D IMPLEMENTATION DETAILS
D.1 STL-10
For a fair comparison with DIM (Hjelm et al., 2019) and CPC (Oord et al., 2018), we adopt the same architecture as that used in DIM and split it into two encoders, each shown as in Table 8. For the implementation of the score function, we adopt similar “encoder-and-dot-product” strategy, which is tantamount to a bilinear model.
In the patch-based contrastive learning stage, we use Adam optimizer with an initial learning rate of 0.001, β1 = 0.5, β2 = 0.999. We train for a total of 200 epochs with learning rate decayed by 0.2 after 120 and 160 epochs. In the non-linear classifier evaluation stage, we use the same optimizer setting. For the NCE-based contrastive learning stage, we train for 320 epochs with the learning rate initialized as 0.03 and further decayed by 10 for every 40 epochs after the first 200 epochs. The temperature τ is set as 0.1. In general, τ ∈ [0.05, 0.2] works reasonably well.
D.2 IMAGENET
For patch-based contrastive loss, we use the same optimizer setting as in Sec. D.1 except that the learning rate is initialized as 0.01.
For NCE-basd contrastive loss in both full ImageNet and ImageNet100 experiments present in Sec. A.1.2, the encoder architecture used for either L or ab channels is shown in Table 9. In the unsupervised learning stage of AlexNet, we use SGD to train the network for a total of 400 epochs. The temperature τ is set as 0.07 by following previous work (Wu et al., 2018). The learning rate is initialized as 0.03 with a decay of 10 for every 50 epochs after the first 250 epochs. Weight decay is

heading: Half of AlexNet(Krizhevsky et al., 2012) for STL-10
text: set as 10−4 and momentum is kept as 0.9. For the linear classification stage, we train for 160 epochs. The learning rate is initialized as 0.1 and decayed by 0.2 every 20 epochs after the first 100 epochs. We set weight decay as 0 and momentum as 0.9.
For ResNets in CMC stage, there are three differences. First, we use larger learning rate, that is, we set a base learning rate of 0.03 for every 128 images and then roughly scale it up with the batch size. Specifically, we train: (1) ResNet-50 with bsz = 280 and lr = 0.08; (2) ResNet-101 with bsz = 200 and lr = 0.05; (3) ResNet-50 x2 with bsz = 156 and lr = 0.04. Second, we only train for 280 epochs with learning rate decayed at 160, 200, and 240 epochs. Third, we used Fast Autoaugment (Lim et al., 2019) as data augmentation. In the linear evaluation stage, we train for 100 epochs. The learning rate is initialized as 30 for ResNet-50 and ResNet-101, and 50 for ResNet-50 x2. It is decayed by 0.2 every 15 epochs after the first 60 epochs. We set weight decay as 0 and momentum as 0.9.

heading: Half of AlexNet(Krizhevsky et al., 2012) for ImageNet

heading: D.3 UCF101 AND HMDB51
text: Following previous work (Misra et al., 2016; Lee et al., 2017; Sayed et al., 2018; Buchler et al., 2018), we use CaffeNet for the video experiments. We tailor the network and use features from the fc6 layer for contrastive learning. Dropout of 0.5 is used to alleviate overfitting.

heading: D.4 NYU DEPTH-V2
text: While experimenting with different views on NYU Depth-V2 dataset, we encode the features from patches with a size of 128× 128. The detailed architecture is shown in Table 10. In the unsupervised training stage, we use Adam optimizer with an initial learning rate of 0.001, β1 = 0.5, β2 = 0.999. We train for a total of 3000 epochs with learning rate decayed by 0.2 after 2000, 2400, and 2800 epochs. For the downstream semantic segmentation task, we use the same optimizer setting but train for fewer epochs. We only train 200 epochs for CMC pre-trained models, and train 1000 epochs for the Random and Supervised baselines until convergence. For the classification task evaluated on STL-10, we use the same optimizer setting as in Sec. D.1 to report numbers in Table 2.

heading: Encoder Architecture on NYU

Review summaries:
The paper presented a multi-view learning method that is based on negative sampling in contrastive learning. The core idea is to set an anchor view and the sample positive and negative data points from the other view and maximise the agreement between positive pairs in learning from two views. When more than two views are presented, the learning objective is a sum over all possible combinations of two views. The performance of the proposed model is good, and the ablation study is interesting.