Self-supervised Representation Learning with Relative Predictive Coding
Authors: Yao-Hung Hubert Tsai
Martin Q. Ma
Muqiao Yang
Han Zhao
Louis-Philippe Morency
Ruslan Salakhutdinov
Venue: ICLR 2021 Poster (2021)
Keywords: self-supervised learning
contrastive learning
dependency based method

Abstract: This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives. We empirically verify the effectiveness of RPC on benchmark vision and speech self-supervised learning tasks. Lastly, we relate RPC with mutual information (MI) estimation, showing RPC can be used to estimate MI with low variance.

Published as a conference paper at ICLR 2021
SELF-SUPERVISED REPRESENTATION LEARNING WITH RELATIVE PREDICTIVE CODING
Yao-Hung Hubert Tsai', Martin Q. Ma', Muqiao Yang', Han Zhao Louis-Philippe Morency', Ruslan Salakhutdinov"
1 CCareggee Mellon University, 2D.E. Shaw Co., University of Illinois at Urbana-Champaign
ABSTRACT
This paper introduces Relative Predictive Coding (RPC), a new contrastive repre- sentation learning objective that maintains good balance among training stabil- ity, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters reg- ularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives. We empirically verify the effectiveness of RPC on benchmark vision and speech self-supervised learning tasks. Lastly, we relate RPC with mutual information (MI) estimation, showing RPC can be used to estimate MI with low variance
INTRODUCTION
Unsupervised learning has drawn tremendous attention recently because it can extract rich repre- sentations without label supervision. Self-supervised learning, subset of unsupervised learning. learns representations by allowing the data to provide supervision (Devlin et al., 2018). Among its mainstream strategies, self-supervised contrastive learning has been successful visual object recognition (He et al., 2020; Tian et al., 2019; Chen et al., 2020c), speech recognition (Oord et al., 2018: Rivière al., 2020), language modeling (Kong al., 2019), graph representation learning (Velickovic et al., 2019) and reinforcement learning (Kipf et al., 2019). The idea of self-supervised contrastive learning is to learn latent representations such that related instances (e-g., patches from the same image; defined positive pairs) will have representations within close distance, while unrelated instances (e.g., patches from two different images; defined as negative pairs) will have distant representations (Arora et al., 2019).
Prior work has formulated the contrastive learning objectives as maximizing the divergence between the distribution of related and unrelated instances. In this regard, different divergence measurement often leads to different loss function design. For example, variational mutual information (MI) esti- mation (Poole et al., 2019) inspires Contrastive Predictive Coding (CPC) (Oord et al., 2018). Note that MI is also the KL-divergence between the distributions of related and unrelated instances (Cover & Thomas, 2012). While the choices of the contrastive learning objectives are abundant (Hjelm et al., 2018; Poole et al., 2019; Ozair et al., 2019), we point out that there are three challenges faced by existing methods.
The first challenge is the training stability, where an unstable training process with high variance may be problematic. For example, Hjelm et (2018); Tschannen al. (2019); Tsai et al. (2020b) show that the contrastive objectives with large variance cause numerical issues and have poor downstream performance with their learned representations. The second challenge the sensitivity to minibatch size, where the objectives requiring huge minibatch size may restrict their practical usage. For instance, SimCLRv2 (Chen et al., 2020c) utilizes CPC as its contrastive objective and reaches state-of-the-eee performances multiple self-supervised and semi-supervised benchmarks. Nonetheless, the objective is trained with a minibatch size of 8, 192, and this scale of training re- quires enormous computational power. The third challenge is the downstream task performance, which the one that we would like to emphasize the most. For this reason, in most cases, CPC
Project page: httppe///ggttbb.cccc////mmmmmmma
com hwwp:///mmmmmaaatmmmmmmm////////
predd i ve_coding

---

Published as a conference paper at ICLR 2021
Table I: Different contrastive learning objectives, grouped by measurements of distribution divergence. Pxy and Pxx Py represents the distribution of un- [(, y) E F for F being any class of functions ʃ X x 3 R.
represents the distribution of related samples (positively-ppat related samples (negatively-paieed t: Compared to JCPC and JRPC, we empirically find JwPc performs worse on complex real-world image datasets spanning CIFAR-10/-100 (Krizhevsky et al., 2009) and ImageNet (Russakovsky ct al., 2015).
Objective
Good Training Stailitty
Lower Minibatch Senclivvity
Good Downstream Performance
reltting to KL divverreeee between Pay and Ps Py: Jow (Donsker & Variihin, Jow (Nigyyen ot al, 2010), and Jene (Dord al., 2018)
Joy Y) u Bey- (11)))) keg ou (olass)) JewestX i- supye Epgy ((x.)) Epgoy
Xx Y) oupyer Ryni Per Ao
log (e/t/////// to En (Foan))
relating to JS-divageence between Pay and PxPpy Jas (Nowozia et 2016)
Jas YJ - sup Rpay log(1 + [e Rryry Degi] 4 efftar))
X
reluting Wassentein divergeent between Pay PyPy: Jaoe (Onir all,, 2019) with To demoting space I-Lipschitz functions
JwpelX, Y) "Pperrs Fi Pal Ia Ie (nftewni & 2 eftea.))
x
relating d-divergeenee between Pay PxPyi Jape (ours)
Aio((X))) -- mayye Br,, ((x:))) eupe [i
is the objective that we would adopt for contrastive representation learning, due to its favorable performance in downstream tasks (T'channenn et al., 2019; Baevski et al., 2020).
This paper presents new contrastive representation learning objective: the Relative Predictive Cod- ing (RPC), which attempts to achieve a good balance among these three challenges: training stabil- ity, sensitivity to minibatch size, and downstream task performance. At the core of RPC is the relative parameters, which are used to regularize RPC for its boundedness and low variance. From a modeling perspective, the relative parameters act as a l2 regularization for RPC. From a statis- tical perspective, the relative parameters prevent RPC from growing to extreme values, as well as upper bound its variance. In addition to the relative parameters, RPC contains no logarithm and exponential, which are the main cause of the training instability for prior contrastive learning objec- tives (Song& & Ermon, 2019).
To empirically verify the effectiveness of RPC, we consider benchmark self-supervised represen- tation learning tasks, including visual object classification on CIFAR- (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), 1), and ImageNet (Russakovsky et al., 2015) and speech recogni- tion on LibriSpeech (Panayotov et al., 2015). Comparing RPC to prior contrastive learning objec- tives, we observe a lower variance during training, lower minibatch size sensitivity, and consistent performance improvement. Lastly, we also relate RPC with MI estimation, empirically showing that RPC can estimate MI with low variance.
2 PROPOSED METHOD
This paper presents new contrastive representation learning objective the Relative Predictive Coding (RPC). At a high level, RPC 1) introduces the relative parameters to regularize the objective for boundedness and low variance; and 2) achieves good balance among the three challenges in the contrastive representation learning objectives: training stability, sensitivity to minibatch size, and downstream task performance. We begin by describing prior contrastive objectives along with their limitations on the three challenges in Section 2.1. Then, we detail our presented objective and its modeling benefits in Section 2.2. An overview of different contrastive learning objectives is provided in Table 1. We defer all the proofs in Appendix.
Notation We use an uppercase letter to denote a random variable (e.g.. X), a lower case letter to denote the outcome of this random variable (e.g., xr), and a calligraphy letter to denote the sample space of this random variable (e.g., X). Next, the samples (Ir y) are related (or positively-paired we refer (ar, v) ~ Pxy with Pxy being the joint distribution of X x Y. If the samples (ar, y) are unrelated (negatively-paire
we refer (x, y) Px with Py being the product of marginal distributions over X x Y. Last, we define f E FF for F being any class of functions f X x y - R.
2.1 PRELIMINARY
Contrastive representation learning encourages the contrastiveness between the positive and the neg- ative pairs of the representations from the related data X and Y. Specifically, when sampling a pair
2

---

Published as a conference paper at ICLR 2021
of representations (z, 3) from their joint distribution ((2, PxY), this pair is defined as pos- itive pair; when sampling from the product of marginals ((z, y) ~ PyPy), ) this pair is defined as negative pair. Then, Tsai et al. (2020b) formalizes this idea such that the contrastiveness of the representations can be measured by the divergence between Pxy and Px Py, where higher diver- gence suggests better contrastiveness. To better understand prior contrastive learning objectives, we categorize them in terms of different divergence measurements between Pxy and Px with their detailed objectives presented in Table 1.
We instantiate the discussion using Contrastive Predictive Coding (Oord et al., 2018, JCPc), which with DKL referring to the KL-divergence:
a lower bound of (Pxy || Px)
ef(an)))
JCPC (X, == sup E JEF
log 1 E c( N
(I)
Pr
Then, Oord et al. (2018) presents to maximize (C Y). so that the learned representations X and Y have high contrastiveness. We note that JCPC has been commonly used in many recent self-supervised representation learning frameworks (He et al., 2020; Chen et al., 2020b). where they constrain the function to be fax y) cosine(3r, y) with cosine(-) being cosine similarity. Under this function design, maximizing Jcre leads the representations of related pairs to be close and representations of unrelated pairs to be distant.
The category of modeling DRL (Pxy || PxPr) also includes the Donsker- Varadhan objective (JDv (Donsker & Varadhan, 1975; Belghazi et al., 2018)) and the Nguyen-Wainrights Wainright-Jordan ob- jective (JNWJ (Nguyen et al., 2010; Belghazi et al., 2018)), where Belghazi et al. (2018); Tsai et al. (2020b) show that JDv Y) JNWJ(X Y) DKL (Pxx (Px
Py)). The other diver- gence measurements considered in prior work are Dis(Pxy PxPy) (with Das referring to the Jenson-Shannon divergence) and Daass (Pxy || Px Py) (with Dasss referring to the Wasserstein- divergence). The instance of modeling Dis(
Px is the Jensen-Shanno f-GAN objective (Jos (Nowozin et al., 2016; Hjelm et al., 2018)), where (X.(X.Y) = 2 2(Dus PxPy) log 2). The instance of modeling (JwPc (Ozair al., 2019)), where JwPo(X,X Y) modifies JCPo(X,)) Y) objective (equation 1) by searching the function from F to Fc. FC denotes any class of I-Lipschitz continuous functions from (X' x y) to R, and thus Fc F. Ozair et al. (2019) shows that (w Y) is the lower bound of both (Px || PxPr) and Dwase (Pxy Px See Table for all the equations. To conclude, the contrastive representation learning objectives are unsupervised representation learning methods that maximize the distribution divergence between Pxy and Py. The learned represen- tations cause high contrastiveness, and recent work (Arora et al., 2019; Tsai et al., 2020a) theoret- ically show that tasks.
(Pxy PxPr) is the Wasserstein Predictive Coding
representations could improve the performance on downstream
After discussing prior contrastive representation learning objectives, we point out three challenges in their practical deployments: training stability, sensitivity to minibatch training size, and downstream task performance. In particular, the three challenges can hardly be handled well at the same time, where we highlight the conclusions in Table 1. Training Stability: The training stability highly relates to the variance of the objectives, where Song & Ermon (2019) shows that JDV and JNwJ exhibit inevitable high variance due to their inclusion of exponential function. As pointed out by Tsai et al. (2020b), JCPC, JwPC, and JJs have better training stability because JCPC and JwPC can be realized as a multi-class classification task and JIs can be realized as a binary classification task. The cross-entropy loss adopted in JCPc, JwPC, and Jus is highly-optimized and stable in existing optimization package (Abadi et al., 2016; Paszke et al., 2019). Sensitivity to minibatch training size: Among all the prior contrastive representation learning methods, Jerc is known to be sensitive to the minibatch training size (Ozair et al., 2019). Taking closer look at equation 1, JCPC deploys an instance selection such that y1 should be selected from (y1- V2.... , UNN, ). with (I,, 91) Pxy. (zz 1)>1) Px with N being the minibatch size. Previous work (Poole et al., 2019; Song & Ermon, 2019; Chen et al., 2020b; Caron et al., 2020) showed that a large N results in more challenging instance selection and forces JCPc to have better contrastiveness of y1 (related instance for æ²) against {リ, 2 (unrelated instance for x). JDv. JNWJ, and JuS do not consider
JJss(X. Y) achieves its supreme value when f' y) log(p(z, v)/((x)p(v)) (Tsai et al., 2020b). Plug- in f" 2) into (X, Y), we can conclude Jas(X, X,Y, Y) 2(Das (xx Px 2og
3

---

Published as a conference paper at ICLR 2021
the instance selection, and JwPC reduces the minibatch training size sensitivity by enforcing 1- Lipschitz constraint. Downstream Task Performance: The downstream task performance is what care the most among all the three challenges. JcPc has been the most popular objective as manifests superior performance over the other alternatives (T'shannenn et al., 2019; Tsai et al., 2020b;a). We note that although Jwrc shows better performance Omniglot (Lake et al., 2015) and CelebA (Liu et al., 2015) datasets, we empirically find it not generalizing well to CIFAR-10/- 100 (Krizhevsky et al., 2009) and ImageNet (Russakovsky al., 2015).
2.2 RELATIVE PREDICTIVE CODING
In this paper, we present Relative Predictive Coding (RPC), which achieves a good balance among the three challenges mentioned above:
JRPC (X, Y) = sup Exxy [f v\I aBpx (f(x. " Exxy [f w) - [f FEF
v)
(2)
where > 0, 3 0, 7 0 are hyper-parameters and we define them as relative parameters. Intuitively, JRPC contains no logarithm or exponential, potentially preventing unstable training due to numerical issues. Now, we discuss the roles of a; 3, At a first glance, acts to discourage the scores of Pxx and Px Py from being close, and B/7 acts as a & regularization coefficient to stop from becoming large. For deeper analysis, the relative parameters act to regularize our objective for boundedness and low variance. To show this claim, we first present the following lemma:
Lemma 1 (Optimal Solution for JRPC) Let Letr(a, p(z)p(y) be the density ratio. JRPC has the r(z.v)-a optimal solution (x, v) B == Ta,B,1 v) with - S Ta.B.r S 3
p(z.y)
Lemma suggests that JRPC achieves its supreme value at the ratio Ta,9,-(2: y) indexed by the relative parameters a, 7 (i.e., we term Ta, y) as the relative density ratio). We note that (o,8,1 y) is an increasing function w.r.t. r(z, y) and is nicely bounded even when r(z, y) is large. We will now show that the bounded ra,3,7 suggests the empirical estimation of JRPG has boundeness and low variance. In particular, let { y/ } be samples drawn uniformly at random from Pxy and {a'j, 2/1 1 be m samples drawn uniformly at random from Py. Then, we use neural networks to empirically estimate JRPC as jmr:
(z.
Definition (J RPC' empirical estimation of JRPC) We parametrize f via a family of neural net- e S Rd where d E and θ is compact. Then, JRPC 8 Em 2 (222 (z) 3,
works Fe = {fe 0
E
SuppeeFe 1 ΣE 1= fo (z., E afo
Then, with probability at Proposition 1 (Boundedness of jm informal) 0 JRPC S 23
a²
24
least 1 - 8, |JRPC - | = 0(( /al/a-loxx (+16)) where = min (n, m).
Proposition 2 (Variance of jmn. informal) There exist universal constants e1 and o2 that depend only on o, 8,y, such that Var fm =0
From the two propositions, when and are large, i.e., the sample sizes are large, jmpe is bounded, and its variance vanishes to O. First, the boundedness of jm. suggests jm.n will not grow to ex- tremely large small values. Prior contrastive learning objectives with good training stability (e.g., JcPc/J/s/JwPc) also have the boundedness of their objective values. For instance, the empirical estimation of JCPC is less than log N (equation 1) (Poole et al., 2019). Nevertheless, JCPC often performs the best only when minibatch size is large, and empirical performances of Jıs and JwPC: not as competitive as JCPC. Second, the upper bound of the variance implies the training of JPC can be stable, and in practice observe much smaller value than the stated upper bound. On the contrary, Song & Ermon (2019) shows that the empirical estimations of JDV and JNwJ exhibit inevitable variances that grow exponentially with the true (Pxyll Py).
Lastly, similar to prior contrastive learning objective that are related to distribution diver- gence measurement, associate JRPC with the Chi-square divergence D₂a || PxPy)
4

---

Published as a conference paper at ICLR 2021
Epx Pr (x2(2x))) 1 (Nielsen & Nock, 2013). The derivations are provided in Appendix. By hav- 1 Pxr B.a Px Py the mixture distribution of Pxy and Px Py, we can rewrite gener-
ing P
B+7
E 2,, y)]. Hence, JRPC can be regarded alization of Dx² with the relative parameters a, 3, where Dx² can be recovered from JRPC by specializing a = 0, 3 = and 7 = 1 ge.g. Dx² = 20 - 1). Note that JRPC may not be a formal divergence measure with arbitrary a,
(RPc(X,Y) (x,
3 EXPERIMENTS
We provide an overview of the experimental section. First, we conduct benchmark self-supervised representation learning tasks spanning visual object classification and speech recognition. This set of experiments are designed to discuss the three challenges of the contrastive representation learning objectives: downstream task performance (Section 3.1), training stability (Section 3.2), and mini- batch size sensitivity (Section 3.3). We also provide an ablation study on the choices of the relative parameters in JRPC (Section 3.4). On these experiments we found that JRPC achieves a lower variance during training, a lower batch size insensitivity, and consistent performance improvement. Second, we relate JRPC with mutual information (MI) estimation (Section 3.5). The connection that MI is an average statistic of the density ratio, and we have shown that the optimal solution of JAPe is the relative density ratio (see Lemma 1). Thus we could estimate MI using the density ratio transformed from the optimal solution of JRPC. On these two sets of experiments, we fairly compare JRPC with other contrastive learning objectives. Particularly, across different objectives, we fix the network, learning rate, optimizer, and batch size (we use the default configurations sug- gested by the original implementations from Chen et al. (2020c), Rivière et al. (2020) and Tsai et al. (2020b).) The only difference will be the objective itself. In what follows, we perform the first set of experiments. We defer experimental details in the Appendix.
Datasets. For the visual objective classification, we consider CIFAR- (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and ImageNet (Russakovsky et al., 2015). CIFAR-10/-100 and ImageNet contain labeled images only, while STL-110 contains labeled and unlabeled images. For the speech recognition, we consider LibriSpeech- 100h (Panayotov al., 2015) dataset, which contains 100 hours of 16kHz English speech from 251 speakers with 41 types of phonemes.
Training and Evaluation Details. For the vision experiments, we follow the setup from Sim- CLRv2 (Chen et al., 2020c), which considers visual object recognition as its downstream task. For the speech experiments, we follow the setup from prior work (Oord et al., 2018; Rivière al., 2020), which consider phoneme classification and speaker identification as the downstream tasks. Then, we briefly discuss the training and evaluation details into three modules: 1) related and unrelated data construction, 2) pre- vre-training, and 3) fine-tuning and evaluation. For more details, please refer to Appendix or the original implementations.
Related und Unrelated Data Construction, In the vision experiment, we construct the related im- ages by applying different augmentations on the same image. Hence, when (ar, y) ~ Pxy, x and y the same image with different augmentations. The unrelated images are two randomly selected samples. In the speech experiment, we define the current latent feature (feature at time f) and the future samples (samples at time t) related data. In other words, the feature in the latent space should contain information that can be used to infer future time steps. A latent feature and randomly selected samples would be considered unrelated data.
> Pre-training, The pre-training stage refers to the self-supervised training by a contrastive learn- ing objective. Our training objective is defined in Definition where we use neural networks to parametrize the function using the constructed related and unrelated data. Convolutional neural net- works are used for vision experiments. Transformers (Vaswani et al., 2017) and LSTMs (Hochreiter & Schmidhuber, 1997) are used for speech experiments.
B Fine-tuming and Evaluation. After the pre-training stage, we fix the parameters in the pre-trained networks and add small fine-tuning network on top of them. Then, we fine-tune this small network with the downstream labels in the data's training split. For the fine-tuning network, both vision and speech experiments consider multi-layer perceptrons. Last, we evaluate the fine-tuned repre- sentations on the data's 's test split. We would like to point out that we do not normalize the hidden representations encoded by the pre-training neural network for loss calculation. This hidden nor-
5

---

Published as a conference paper at ICLR 2021
Table 2: Top-1 accuracy (%) for visual object recognition results. Jnv and Jyws are not reported on ImageNet due to numerical instability. ResNet depth, width and Selective Kernel (SK) configuration for each setting are provided in ResNet depth+widtt++ (2020c) because we only train for 100 epochs rather than 800 to the fact that running 800 epochs uninterruptedly on cloud TPU very expensive. Also, we did not employ a memory buffer (He et al., 2020) to store negative samples. We and we did not employ a memory buffer. We also provide the results from fully supervised models comparison (Chen ct al., 2020b;e). Fully supervised training performs worse STL-10 because it does employ the unlabeled samples in the dataset (Löwe et al., 2019).
column. A slight drop of JCpo performance compared Chen al.
Supf-supe
Dataset
ResNet Depth+WWidh--Se
Supervised
JDV
JNWI
Jıs
JwPC
JCPC
JRPC
CIFAR-10
18 No SK
91.10
90.54
83.55
80.02
91.12
91.46
93.12
CIFAR-I0
50 + No
92.23
92.67
87.34
85.93
93.42
93.57
95.70
CIFAR-100
18+ x + No SK
77.10
77.27
74.02
72.16
77.36
77.98
79.11
CIFAR-100
50++ x + No SK
79.02
78.52
75.31
73.23
79.31
79.89
81.20
STL-10
50++ 1x + No SK
82.25
81.17
79.07
76.50
83.40
84.10
71.40
ImageNet
50 x + SK
66.21
62. 10
73.48
74.43
78.50
ImageNet
152 + 2x + SK
71.12
69.51
77.80
78.40
80.40
Table 3: Accuracy (%) for LibriSpeech- 100h phoneme and speaker classification results. We also provide the results from fully supervised model comparison (Oord al., 2018).
Self-supervised
Task Name
Supervised
JCPC
JDV
JNwI
JAPC
Phoneme classification
64.6
61.27
62.09
69.39
74.6
Speaker classification
97.4
95.36
95.89
97.68
98.5
malization technique is widely applied (Tian et al., 2019; Chen et al., 2020b;c) to stabilize training and increase performance for prior objectives, but find it unnecessary in JRPC-
3.1 DOWNSTREAM TASK PERFORMANCES ON VISION AND SPEECH
For the downstream task performance in the vision domain, test the proposed JRPC and other contrastive learning objectives on CIFAR- (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and ImageNet ILSVVR--2012 (Russakovsky et al., 2015). Here report the best perfor- mances JRPC can get on each dataset (we include experimental details in A.7.) Table 2 shows that the proposed JRPC outperforms other objectives on all datasets. Using JRPC on the largest network (ResNet with depth of 152, channel width of 2 and selective kernels), the performance jumps from 77.80% of JCPc to 78.40% of JRPC-
Regarding speech representation learning, the downstream performance for phoneme and speaker classification are shown Table (we defer experimental details in Appendix A.9.) Compared to JCPC, JRPC improves the phoneme classification results with 4.8 percent and the speaker classifi- cation results with 0.3 percent, which is closer to the fully supervised model. Overall, the proposed JRPC performs better than other unsupervised learning objectives on both phoneme classification and speaker classification tasks.
3.2 TRAINING STABILITY
We provide empirical training stability comparisons on JDV, JNWJ, Jero and JRPC by plotting the values of the objectives as the training step increases. We apply the four objectives to the SimCLRv2 framework and train on the CIFAR- 10 dataset. All setups of training are exactly the same except the objectives. From our experiments, JDv and JNwJ soon explode to NaN and disrupt training (shown as early stopping in Figure la; extremely large values are not plotted due to scale constraints). On the other hand, JRPC and JCPC has low variance, and both enjoy stable training. As result, performances using the representation learned from unstable Jpv and JNwJ suffer in downstream task, while representation learned by Jurc and JerC work much better.

---

Published as a conference paper at ICLR 2021
CIFAR-10
LibriSpeech-100n
Value of objective functions during training
7.5
Classfiiattion accuracy (%)
Joc
68
Joc
67
2.5
66
Value
90.5
0.0
Jov
cassification
65
Jare
90.0
Joce
63
-5.0
62
Janc
5000 10000 15000
25000
61
Step
512
32
128
512
minibatch size
minibatch size
(a)
(b)
(c)
Figure (a) Empirical values of JDV, JNWJ, JCPC Jarc performing visual object recognition on
CIFAR-10. JDV and JNwJ soon explode to NaN values and stop the training (shown as early stopping in the figure), while JCPc and JRPC are more stable. Performance comparison of JCPc and JRPC on (b) CIFAR- 10
and (c) LibriSpeech- 100h with different minibatch sizes, showing that the performance of JRPC is less sensitive minibatch size change compared to JCPC.
3.3 MINIBATCH SIZE SENSITIVITY
We then provide the analysis on the effect of minibatch size on JRPC and JCPC, since JCPC known to be sensitive to minibatch size (Poole et al., 2019). We train SimCLRv2 (Chen et al., 2020c) on CIFAR-10 and the model from Rivière et al. (2020) on LibriSpeech- 100h using JRPC and JCPC with different minibatch sizes. The settings of relative parameters are the same as Section 3.2. From Figure 1b and Ic, we can observe that both JRPC and JCPC achieve their optimal performance at a large minibatch size. However, when the minibatch size decreases, the performance of JCPC shows higher sensitivity and suffers more when the number of minibatch samples is small. The result suggests that the proposed method might be less sensitive the change of minibatch size compared to JCPC given the same training settings.
3.4 EFFECT OF RELATIVE PARAMETERS
We study the effect of different combinations of relative parameters in Jure by comparing down- stream performances visual object recognition. We train SimCLRv2 CIFAR-10 with dif- ferent combinations of a, 3 and y in JRPC and fix all other experimental settings. We choose 0.00,, 1.0), B E 10,00001, 1.0), {0, 0.001, 1.0] and report the best performances under each combination of a, 8, and 7. From Figure 2, we first observe that a > 0 has better downstream performance than a when and y are fixed. This observation as expected, since > encourages representations of related and unrelated samples to be pushed away. Then, we find that a small but nonzero / (B 0.001) and large y (y 1.0) give the best performance compared to other combinations. Since 3 and y serve as the coefficients of E2 regularization, the results imply that the regularization mance. The results here are not competitive as Table 2 because the CIFAR-I0 result reported in Table 2 is using a set relative parameters (a 1.0, 3 = 0.005, 7 = 1.0) that is different from the combinations in this subsection. Also, details.) conclusion, we find empirically that a non-zert a, a small and a large y will lead to the optimal representation for the downstream task CIFAR-10.
strong and sensitive factor that will influence the perfor-
quite different ranges of ImageNet (see A.7 for
3.5 RELATION TO MUTUAL INFORMATION ESTIMATION
The presented approach also closely relates to mutual information estimation. For random variables X and Y with joint distribution Pxy and product marginals Px the mutual information is defined as I(X;Y) DR1. (Pxy P Pv). Lemma I states that given optimal solution f y) of
JRPC, we can get the density ratio r(z, y) == p(x, v)/p(x)p(g) r(z, y)
>/9+a 7 We (- 3.
can empirically estimate f(x,y) y) from the estimated f(x, 3) via this transformation, and use f(x, y) to estimate mutual information (Tsai et al., 2020b). Specifically, I(X;Y) s - Σ log f(z;, 1)) with (Ir; yi) p&n where p&n is the uniformly sampled empirical distribution of Px,

---

Published as a conference paper at ICLR 2021
Q =
a =0.001
a =1
0.0
41.8
71.95
10
50.67
72.23
10
61.22
73.27
Y
0.001
60.05
73.52
30.66
76.9
38.76
87.5
76.45
50
-25
1.0
85.97
90.76
83.29
86.33
91.2
84.02
81.49
91.3
85.23
0.0
0.001
1.0
0.0
0.001
1.0
0.0
0.001
1.0
B
Figure 2: Heatmaps of downstream task performance on CIFAR-10, using different a, 3 and 7 in the JarC- We conclude that nonzero a, small 8 (B = 0.001) and a large 7(> = 1.0) are crucial for better performance.
Figure 3: Mutual information estimation performed on 20-d correlated Gaussian distribution, with the corre- lation increasing each 4K steps. JRPC exhibits smaller variance than SMILE and DoE, and smaller bias than JCPC.
We follow prior work (Poole et al., 2019; Song & Ermon, 2019; Tsai et al., 2020b) for the experi- ments. We consider and Y as two 20-dimensional Gaussians with correlation p. and our goal is estimate the mutual information T Y): Then, perform cubic transformation on y so that ++ y². The first task is referred to as Gaussian task and the second is referred to as Cubic task, where both have the ground truth T(X; Y) = 10llo (1 p²). The models are trained 20, 000 steps with / ((X; Y) starting at 2 and increased by 2 per 4, 000 steps. Our method is compared with baseline methods JCPC (Oord et al., 2018), JNWJ (Nguyen et al., 2010), JIs (Nowozin al., 2016), SMILE (Song & Ermon, 2019) and Difference of Entropies (DoE) (McAllester & Stratos, 2020).
All approaches use the same network design, learning rate, optimizer and minibatch size for fair comparison. First, we observe Jerc (Oord et al., 2018) has the smallest variance, while it exhibits large bias (the estimated mutual information from Jerc has an upper bound log Oogtttcch size)). Sec- ond, JNWJ (Nguyen et al., 2010) and JJSD (Poole et al., 2019) have large variances, especially in the Cubic task. Song & Ermon (2019) pointed out the limitations of JCPC. JNWJ, and JJSD, and developed the SMILL method, which clips the value of the estimated density function to reduce the variance the estimators. DoE (McAllester & Stratos, 2020) neither a lower bound nor upper bound of mutual information, but achieve accurate estimates when underlying mutual informa- tion is large. JRPC exhibits comparable bias and lower variance compared to the SMILE method, and is more stable than the DoE method. We would like highlight method's low-variance property, where we neither clip the values of the estimated density ratio nor impose an upper bound of our estimated mutual information.
4 RELATED WORK
As subset of unsupervised representation learning, self-supervised representation learning (SSL) adopts self-defined signals as supervision and uses the learned representation for downstream tasks, such as object detection and image captioning (Liu al., 2020). We categorize SSL. work into two groups: when the signal is the input's hidden property or the corresponding view of the input. For the first group, for example, Jigsaw puzzle (Noroozi & Favaro, 2016) shuffles the image patches and defines the SSL task for predicting the shuffled positions of the image patches. Other instances are Predicting Rotations (Gidaris et al., 2018) 8) and Shuffle & Learn (Misra et al., 2016). For the second group, the SSL task aims at modeling the co-occurrence of multiple views of data. via the contrastive or the predictive learning objectives (Tsai et al., 2020a). The predictive objectives encourage recon- struction from one view of the data to the other, such predicting the lower part of an image from
8

---

Published as a conference paper at ICLR 2021
its upper part (ImageGPT by Chen et al. (2020a)). Comparing the contrastive with predictive learn- ing approaches, Tsai et al. (2020a) points out that the former requires less computational resources for good performance but suffers more from the over-fitting problem.
Theoretical analysis (Arora et al., 2019; Tsai et al., 2020a; Tosh et al., 2020) suggests the con- trastively learned representations Tian et al. (2020) shows what matters more for the performance are 1) the choice the contrastive learning objective; and the creation of the positive and negative data pairs in the contrastive objective. Recent work (Khosla et al., 2020) extends the usage of contrastive learning from the self- supervised setting to the supervised setting. The supervised setting defines the positive pairs as the data from the same class in the contrastive objective, while the self-supervised setting defines the positive pairs as the data with different augmentations.
lead to good downstream performance. Beyond the theory,
Our work also closely rates to the skewed divergence measurement between distributions (Lee, 1999; 2001; Nielsen, 2010; Yamada et al., 2013). Recall that the usage of the relative parameters plays crucial role to regularize our objective for its boundness and low variance. This idea is similar to the skewed divergence measurement, that when calculating the divergence between distributions P and Q, instead of considering D Q), these approaches consider D(P || aP + (1 a))) with D representing the divergence and < a < 1. natural example is that the Jensen-Shannen divergence is symmetric skewed KL divergence: Das Q) 0.5DkL (P)) 0.5P P + 0.5Q) + 0.5DxL (Q 0.5P + 0.5Q). Compared to the non-skewed counterpart, the skewed divergence has shown to have more robust estimation for its value (Lee, 1999; 2001; Yamada al., 2013). Dif- ferent from these works that focus on estimating the values of distribution divergence, we focus on learning self-supervised representations.
5 CONCLUSION
In this work, we present RPC, the Relative Predictive Coding, that achieves a good balance among the three challenges when modeling contrastive learning objective: training stability, sensitivity to minibatch size, and downstream task performance. We believe this work brings an appealing option for training self-supervised models and inspires future work to design objectives for balancing the aforementioned three challenges. In the future, we are interested in applying RPC in other application domains and developing more principled approaches for better representation learning.
ACKNOWLEDGEMENT
This work supported in part by the NSF IIS1763562, NSF Awards #1750439 #1 #1722822, Na- tional Institutes of Health, IARPA D17PC00340, ONR Grant N000141812861, and Facebook PhD) Fellowship. We would also like acknowledge NVIDIA's GPU support and Cloud TPU support from Google's TensorFlow Research Cloud (TFRC).
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large- scale machine learning. In 12th {USENIX) Symposium on Operating Systems Design and Imple- mentation ((OSDI) 16), pp. 265 2016.
Martin Anthony and Peter L. Bartlett. Neural network learning: Theoretical university press, 2009.
foundations. foooretical
cambridge
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229 2019.
Alexei Baevski, Henry Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. arXiv preprint arXiv:2006. 11477, 2020.

---

Published as a conference paper at ICLR 2021
Peter L Bartlett. The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network. IEEE transactions on Information Theory, 44(2):525-53) 1998.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint NoXY::8011.0062.. 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:20066.09882 2020.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proceedings of the 37th International Conference Machine Learning, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint urXiv:2002.05709 2020b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020c.
Ching-Yas Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. arXiv preprint arXiv:2007.00222 2020.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223, 2011.
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
Jacob Devlin, Ming-Wes Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /1t0:::v8/00008000 2018.
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, i. Communications on Pure and Applied Mathematics, 28(1):1-47, 1975.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1rXiv:1803.07722 2018
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in neural information processing systems, pp- 5767-5777, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Devon Hjelm, Alex Fedorov, Samuel Lavoie- Lavoie-Marchildon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint I 0667006670, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
K Hornik, M Stinchcombe, and H White. Multilayer feedforward networks universal approxi- mators. Neural Networks, 2(5):359-366, 1989.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1502.0300 2015.
10

---

Published as a conference paper at ICLR 2021
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint
1131: 2020.
Thomas Kipf, Elise der Pol, and Max Welling. Contrastive learning of structured world models. arXiv preprint arXiv:1911.1224 2019.
Lingpeng Kong, Cyprien de Masson d' Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yo- gatama. mutual information maximization perspective of language representation learning. arXiv preprint arXiv: 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum, Human-level concept learning through probabilistic program induction. Science, 350(6266): 1332-)):333333333 2015.
Lillian Lee. Measures of distributional similarity. Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pp. 25-32, College Park, Maryland, USA, June 1999. Association for Computational Linguistics. doi: 10.3115/1034678.1 1034693. URL https: / //www. a web. org/ anthologyyy.///oaaoooooooo P99-1004
Lillian Lee. On the effectiveness of the skew divergence for statistical language analysis. In AIS- TATS. Citeseer, 2001.
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 510-519, 2019.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Zhaoyu Wang, Li Mian, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. arXiv e-prints, pp. arXiv-2006, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Sindy Löwe, Peter O'Connor, and Bastiaan Veeling, Putting end to end-to-end: Gradient-isolated learning of representations. Advances in Neural Information Processing Systems, pp. 3039- 3051, 2019.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, pp. 875-884, 2020.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European Conference on Computer Vision, pp. 527-544.
Springer, 2016.
XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(1 (6(11))5847586) 2010.
Frank Nielsen. A family of statistical symmetric divergences based on jensen's inequality. arXiv preprint arXiv: 1004,4044 2010.
Frank Nielsen and Richard Nock. On the chi square and higher-order chi distances for approximating f-divergences. IEEE Signal Processing Letters, 21(1):10-15 2013.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 180.vv/007003777 2018.

---

Published as a conference paper at ICLR 2021
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Ser- manet. Wasserstein dependency measure for representation learning. Advances in Neural Information Processing Systems, pp. 15604-15614, 2019.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems, pp- 8024-8035, 2019.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On varia- tional bounds of mutual information. arXiv preprint arXiv:1905.0692. 2019.
Morgane Rivière, Armand Joulin, Eemannem Mazaré, and Emmanuel Dupoux. Unsupervised pretraining transfers well across languages. In ICASSP 2020-2020 JEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 7414-7418. IEEE, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. arXiv preprint arXiv:1910.0622., 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: 05849, 2019.
Yonglong Tian, Chen Sun, Ben Poole. Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005.10243, 2020.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. arXiv preprint arXiv:2008. 000020008.000000 2020.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. De- mystifying sele-supe learning: An informatiiontheorrtical framework. arXiv preprint arXiv:2006.05576. 2020a.
Yao-Hung Hubert Tsai, Han Zhao, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhut- dinov. Neural methods for point-wise dependency estimation. arXiv preprint arXiv:2006.0555. 2020b.
Michael T'ccannen, Josip Djolonga, Paul Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv: /907. 13625, 2019.
Aad W Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6908 2017.
Petar Velickovic, William Fedus, William Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR (Poster), 2019.
Makoto Yamada, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Masashi Sugiyama. Rel- ative density-ratio estimation for robust distribution comparison. Neural computation, 25(5): 1370, 2013.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: I 7088 03888, 2017.
12

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.1 PROOF OF LEMMA I IN THE MAIN TEXT
Lemma 2 (Optimal Solution for JRPC, restating Lemma in the main text) Let
JRPC (X, Y) == sup Exx [f x)l (f v) - EPr [f(a,v)] - Bx (f,
feF
and rxx, v) p(>)n(y)
pla.ul
be the density ratio. JRPC has the optimal solution
r(z,y)= (2, with 3r(x,y) +
1 S ra,3,y S ?
(x,
3
Proof: The second-order functional derivative of the objective is
-BdPx,y ydPxPy
which is always negative. The negative second-oree functional derivative implies the objective has a supreme value. Then, take the first-order functional derivative @JRPS and it to zero: Om
dPx.y - a d - B f(x,)) dPx,y - - f(ff(()))) dPxPy = O.
We then get
dPx. - a dPxPs
p(x ap(z)p(() r(z,y)
(xx,y)) 3 3d + 7. PP
By(æ, 1) + (x(x))()) Br(a,y) +7
Since 0 S r((x y) 0o, we have - 3)(,,9)+
00,
r(z.y)-a
z. Hence,
0 = (2, v) with a I ra, S B"
A.2 RELATION BETWEEN JRPG AND Dx²
In this subsection, we aim to show the following: 1) Dxx || PxPr) = Exxpy (z²(z.)) y)1 - 1;
and 2) JRPC(X,X)) Y) = 2
B+7
8+2Ep Ep [r2 (2, y)l by having P' = 3 8-7 Pxy + 87, Px as the mixture
distribution of Pxy and Px Py.
Lemma (Px || PxPr)
Epx Py (x,
Proof: By definition (Nielsen & Nock, 2013),
/ dPxx
/
(dPA (dPxPy
dPxx 2
(xxPxy 1l PxPx)=
dPxPy -
Py
/
p(a,,)
/
dPxPy -1= ((xxu))PPPP-
p(x)p(u))
Exx 1/2 ()) - 1.
Lemma Defining p' B Pxy + 2 P_Pxyy as a mixture distribution of Pxy and Py,y 547
B4g
JRPC(X,Y)) 2p 1r2.9. (I,
13

---

Published as a conference paper at ICLR 2021
Proof: Plug in the optimal solution (x
dPx,y-o-dPa Py (see Lemma 2) into JRPC:
JRPC Bf Pxp (x, (1.
3
Exx (02
2 NE (
2
/ (x,1)) (dPxy dPxPy)
2
1
f0
(g.dPxr ty dPxPy)
= / dPx. -a.PxPs -a -o-y---dPpp-
1/ dxx,y a dx ,
)²
B. +7. -dpxyy dPxPy
2 B dPPxxx +7 dPxty B.
2 B B.Px t -. dPxPy ) (3. +7 ( t.dPPPr)
1 . dPx.y -a.Pxxr /
3 + /
dpx, dlx xPxPy 2
(
B
7 7. ?_dPxPP dPPPr).
dPxx
3. xlxx + 7 dl Py 3347
3+7
Since we define Ta,8,y
y a Py p.PPxy dPP
and P' = 344 Px
Pxy + xxpPxPr
B+7 B
JRPC
Ep 1a, (x, v))
2
A.3 PROOF OF PROPOSITION I IN THE MAIN TEXT
The proof contains two parts: showing 0 S Jure S consistent estimator for JRPG (see Section A.3.2).
+ a² 28 2y
1
(see Section A.3.1) and Jmre is a
A.3.1 BOUNDNESS OF JRPC
Lemma 5 (Boundness of JRPC) 0 S JRPC S
+ a² 23 27
1
Proof:
Lemma 4 suggests JRPC = 2+2 2 Ira (z, v) with P' = 344 Pxx+
as the mixture distribution of Pxy and Px Hence, it is obvious (ire((XY))>) 20 We leverage the intermediate results in the proof of Lemma 4:
a- d ) (B.ddPx++ ddxxxt--dPxPp JRPC(X, - / / ( 3. 17.4--d00 - Py dPx,y dPxPy Py
/ dPx.y dpxxxxxx +7 ppdxPxyy
dPxx ( dPxy-a-dPxP,
2
d. 3. 1. dPxxx
Exxy (ra (2. v)1
By By (ra.
Ep
2
Since a S ra,B, S }, Jnre Jarc(( (X,Y) s 23 27
a²
A.3.2 CONSISTENCY
We first recall the definition of the estimation of JRPC:
(Jmon, empirical estimation of JRpC, restating Definition in the main text) We parametrize f via a family of neural networks Fe == {So 0 E C Rd4 where d N and θ is compact. Let {æt, y be samples drawn uniformly at random from Pxy and fal, v,) am be m samples drawn uniformly at random from Px Py. Then,
Definition
jm.n = sup
1
Σ i=1
fol 91)
Eafola
Σ
2 lo (r y:)
Σ
fa (z.).
foeFo
n
m j=1
2
j=1
14

---

Published as a conference paper at ICLR 2021
Our goal is to show that RPC is consistent estimator for JRPC. We begin with the following definition:
RPC,0 = Σ fo(
1 n
1 m m j=1
Σ
1
Σ
(zi Ui)
m Σ m 5=1
7 l (2/. (3)
n
n
and
(nrc.o] = Exx [Jo - (o( v)l - Erxy (x, 7 Exx ry (8 (4)
8
I.
Then, follow the steps:
The first part is about estimation. We show that, with high probability, SPC,, is close to
|RPC,,] for any given 0. E
The second part is about approximation. We will apply the universal approximation lemma of neural networks (Hornik et al., 1989) to show that there exists network 0" such that E jRPC.0.] is close to JRPC-
Part I Estimation: With high probability, RPC., is close to E [RPPC.))) for any given 0. Throughout the analysis on the uniform convergence, we need the assumptions on the boundness and smoothness of the function fo- Since show the optimal function f is bounded in JRPG, we can use the same bounded values for fa without losing too much precision. The smoothness of the function suggests that the output of the network should only change slightly when only slightly perturbing the parameters. Specifically, the two assumptions are as follows:
Assumption (boundness of fe) There exist universal constants such that f Fe, C₁, S fo S Cv. For notations simplicity, we let M Cv Cr. be the range of fa and U = max {|Col, Cr. be the maximal absolute value of fo. In the paper, we can choose to constrain that CL a and
Cu since the optimal function f" has <
Assumption (smoothness of fo) There exists constant p > 0 such that V(x v) (X x Y) and 0,, 02 E θ, Ife, (x. v) fo, (x. v) S p|0 0z)-
Now, we can bound the rate of uniform convergence of a function class in terms of covering num- ber (Bartlett, 1998);
Lemma 6 (Estimation) Let > O and N(O, c) be the covering number of θ with radius c. Then,
Pr
sup FoeFe
jRPC,
E [KPC.O
[JRPcc//22) ||20)
nE² 32M²
m²²
nE² 32U22
me²
<2,(O,
4p(1 + + 2(3 2(++
+ exx
+ exp
32M22²
+ +exp
+ +exp
32
Proof: For notation simplicity, we define the operators
P(f) Ex [z y)] and Pa(f) 1 n Ei f(r, y:)
. Q()) = Px Py [f w)] and @m = 1. 1
Hence,
jm. RPC.0 E |RPC.
PP(lf) P(fo) afm 10000:0--0000-----------00000000
< PP((f)) - P(fo) + - 11000-0001011000-------11111--10
15

---

Published as a conference paper at ICLR 2021
Let c' =
4n 40(1+0-2(4+)10) and T := ()(, = foz for with {01, 02. ,0
be such that oo (01, e¹), ..., (Or, E') are e' cover. Hence, for any fo Fe. there is an fer E C such that |0 - 8x S Sc.
Then, for any for C::
R - E [
< (foo P(fo) + a - P( +7 -
- -
S
+a (fo. Q(fo.)) Qm) - ++ (Q(f)) - Q(fo.))
+
+
P( | P2. (P())))))))) P(
+7
1mmh -00
Qm + Q(12))
+ p|0 (fo.) - - 0x1 + p110 0z1l
+ a
-U0a)) p -100 - 0all + p110- - 0zll
+ 8
P((E)) + 0xll + =pp 0x.1
+ (10m - - 081 + 20U1 0x.1
(1 = =IPr(fo.) - P(fo.) a - +
|| P,())) P(ff)) | 1
1m (fo,) Q(().)
+ + 228 70) 0x1l
a
((Pr( - (fmee Q(fe.)))e))) + 8 (..)) - (m -
where
Pa(fe) - Pa(fe < p|0 0xll due to Assumption 2, and the result also applies for P(fo.)
P(fo) -(fo))((fo.). )l Qm (fo) Qm. ), and (Q(f)) - Q(f..).) )l.
PP(ft) - P, < fell 0ull < 2pU due to Assumptions 1 and 2. The result also applies for P(f P(fe,) )l. (m)) Qm (fb,)) )/ and (Q(f)) ((),) )/-
Hence,
Pr
sup FoEFF
jm.. R
E [[JRcc. |
[Pa(Sor)) +a (m) Q(oa.) +B (P P( + (m. Q(12. -220 +
SPr
max EC
+
EC
-a - Q(faa)l + - Q(10) -
=Pr
max
+
EC
Jor EC
T
Pr -P((a)))) + -0a - P
+7 1m.
Q(3,1 ;
k=1
(PP((fo))-((f))) (((lfo))P((oo)) -( 9 + Pr (fl Q(fa 9)
Pr
Pr (L..) P(F2)) ) + Pr (7 (r(10m(()))) -Q((6) 2 9)
With Hoeffding's inequality,
16

---

Lemma (Approximation (Hornik et al., 1989)) Let > 0. There exists d E N and fam- where O is compact, such that Published as a conference paper at ICLR 2021
(r(( - P(fea) 2 8) - 2exp
ne
32M2
Pr(a)2m(1 Q(fo.)) )1 2 3) $ 2exp
me
32M
Pr(B)Pa(())) P( 3) 5 2exp(
321/2
Pr(1 Q(faa)) 8) S 2exp
me²
320
To conclude,
fm.n RPC,0 - E (irc.. 2r
Pr
sup fo
ne²
mc?
nc²
mc?
<2N(O)
4p(1 + ((++2(++))))
exp
32M2
+
eepl
32M22a²
+ exp
32U 32
+ texp(
32U222
Part II Approximation: Neural Network Universal Approximation. We leverage the univer- sal function approximation lemma of neural network
ily of neural networks Fe := (fo 0 DEO C RR]
inf E [R]C... JRPC S. foef
Part III BBringing everything together. Now, we are ready to bring the estimation and approxi- mation together to show that there exists a neural network 8" such that, with high probability. JRPC, can approximate JRPC with n'' min {n,, m) at rate of O(1/Vr)):
Proposition 3 With probability at least 1 -- 6, 30 O, |RRP where n' = min {n, m}.
jm.n RPC,
O(v
d+ log (1/8)
12'
Proof:
The proof follows by combining Lemma and 7.
First, Lemma suggests, 30 O
e E Jurc.o-1 - Jure S 2
Next, we perform analysis on the estimation error, aiming to find n, m and the corresponding prob- ability, such that
jRPe. E JRPC...
e 2
(N(O).) =
Applying
Lemma 6 with the covering number of the neural network:
o( (exp(dlog (1/()) (Anthony & Bartlett, 2009)) and let minfn, m):
Pr
sup feeT
jmve
E. P/Jnre.o0l 2
S2N(0, 8p + 2ol1+a+266 7)0)
exp
ne² 128
) + texp
128 2 + texp(
mne?
1281/2
ne²
exp(
128U2
me?
=O(exp(dlog (1/6) -n()))
17

---

Published as a conference paper at ICLR 2021
where the big- O notation absorbs all the constants that do not require in the following derivation. Since we want to bound the probability with 1 - 6, we solve the such that
exp(d (1/e) - n'6²) S 6.
With log (x)
n''² d(e 1) n'e² + dloge 2 log (1/0),
where this inequality holds when
d log (1/8)
0
n'
A.4 PROOF OF PROPOSITION 2 IN THE MAIN TEXT FROM AN ASYMPTOTIC VIEWPOINT
Here, we provide the variance analysis on RPC via asymptotic viewpoint. First, assum- ing the network is correctly specified, and hence there exists a network parameter 8" satisfying (" = fo (v. u) = (z, y)- Then we recall that jurC is a consistent estimator JRPC (see Proposition 3), and under regular conditions, the estimated network parameter 0 in RPC sat- isfying asymptotic normality in the large sample limit (see Theorem 5.23 (Van der Vaart, 2000)). We recall the definition of APC. in equation and let n' min(n, m). the asymptotic expansion of jm. has
jm.n
jmn (0" - 0) + o(100 Đ11
RPC,8
RPC,0
RPC,
= jmm
RPC.0 + jmn
(0" - 0) + op(
R
(5)
jmm
Opl
RPC,0
where jm.n RPC,
since 0 is the estimation from RPC
sup foeFe
JPPC
jm.
Next, recall the definition in equation 4:
8 Exx 1f((,,)) Ep py (x)
BIJ B| Exxy lfg y)] aEpx (f((,,)))
Likewise, the asymptotic expansion of E|JRPC,o] has
B|JRPC = E[JRPC EI 10- o(110 0-
= E[JRCC + BInc 0²) + 0pl
1
Vrv
(6)
E[JRPC opl
v
where E[JRrC = 0 since B|Jnrc, = JRPC and 0* satisfying f (*( so (x.
18

---

Published as a conference paper at ICLR 2021
Combining equations 5 and 6:
jm RPC,,
=mmm RFC,0- JRPC + Op(
1
Vn
(1) 1
31
Σ
(zi, Us)
? 1
Σ
(z'. vUj
2n
2m j=1
j=1
1=1
Exx [f<,)))) + a By (x, y)]
3
Brx
$2 Br 1y [r<( 0pl Vn'
2
(2,, v1)
1
Σ
T. (/),
31
( v1) - 2 m
r a
2 i=1
==
j=1
Exx [ra (2, aBpx (a. v)l + Bxxx (2,,,11,1) + 2 Exxx Py (2,,,(0,0)
B
+ 0pl
- _
B (0,, Mi) Epxr (T..: z g ((aa)())w))
ra, 9i)
2 (0
a (1' 11) - Epxpr (0 + 20. (c. m
aT (x/,) + 2,
2 a.
vm
Vm J=1
+ pp(
Therefore, the asymptotic Variance of RPC is
Varl 1 -Var (x, v) 2
B
2, (2,v)]+ 1 Var (ara, v) 2
2 a. (x, y)1 + of )
2
First, we look at Varpxy [ra, v) 2 v)]. Since and - S Ta,Brr S 1. simple 1
calculation gives us
20y+80"
20² I (T,,,, v)
3 22 (r. S 2 Hence,
Varpyylaanne v) _ "0 (z,,i) max { ( (207 Ba (2 2
21 23 S
27
Next, we look Varp By ( (x, + 3r2 a (x, v)]. Since 2 0,7 > 0 and - S To,8,y S
3. simple calculation gives us
o² 24
S a0r 3) + 3ra (2i v) 5
20 Hence,
20
a
20,3 2 (22)
lar, + 2 2 (z. v) < max
(222
27
2,82 f.
Combining everything together, we restate the Proposition 2 in the main text:
Proposition 4 (Asymptotic Variance of jap
Var[jm-a]
Varrxyy (o..n((,1)
8 8,2
(x,
1
Var Py (ara. v) + 2,
2 + o0 +
(x,
n 1
2
m
2
< max { (
(20r + 3a² 3022
2
1
aa²
(20,3 + 7/)
1
max
+ o
27
23
27
2,32
19

---

Published as a conference paper at ICLR 2021
A.5
PROOF OF PROPOSITION 2 IN THE MAIN TEXT FROM BOUNDNESS OF fo
As discussed in Assumption 1, for the estimation f can bound the function fo in Fo within 72 3l without losing precision. Then, re-arranging JPCC
sup faFFa
Σ
Solas, Ue)
Σ
m
Σ 2
56 (1.
- Σ 7 (/(c).11/)
n
1=1
j=1
1
Σ
(fo(T),Vs) Vi)
B 5 (1). + Σ (afo(a') 1/) 2 1o(e,
sup
JoFFa
2
im1
jm
Then, since - S fo(:))) S 3.. basic calculations give us
20y + Ba² S fo( v:)
2 L (zri, yı) s and
1
a²
a 1/)+ (z),,/))
2aB + + 2,32
272
2/3
2y
The resulting variances have
Var[Ju(as, wi) 2 S0 Uis vc)l S max { (207 + Ba²
3
1
272
2,3
and
Varlarfo w/y) + 10 w/11 S max
a 2 2
2ax3 +
?
2
27
2.32
Taking the mean of m, independent random variables gives the result:
Proposition 5 (Variance of junn
Var[Jmaal < -max (200
1 { 220 + fa²
1
(a2
(
/2a,
(2a3 7/ 21
max
n1
272
23
m
27
2,32
A.6 IMPLEMENTATION OF EXPERIMENTS
For visual representation learning, we follow the implementation in https://github.. com/ boogglereeearcc//eeeee
For speech representation learning, follow the imple- For MI estima-
mentation in https: //g/thub..///////// com/ htt.///bbooeeeeecccc/CCCCCCCC. tion, we follow the implementation in httppa://github. com www./yaoougg//ooonoooooo se. Sependency_Neurer Latiirall_sstiii httpa:/oi.cteeaaaaaaa/e...aaaaaa
A.7 RELATIVE PREDICTIVE CODING ON VISION
The whole pipeline of pretraining contains the following steps: First, a stochastic data augmentation will transform one image sample Ty to two different but correlated augmented views, 2'x-1 and dgs. Then base encoder f(() implemented using ResNet (He et al., 2016) will extract representa- tions from augmented views, creating representations h2k-1 and h2k- Later a small neural network g(-) called projection head will map h24-1 and hes to #2k-1 and 224 in a different latent space. For each minibatch of N samples, there will 2N views generated. For each image 2; there will be one positive pair 2'z2-1 and Z/yx and 2(N - 1) negative samples. The RPC loss between a pair of positive views, æ and x'j (augmented from the same image), can be calculated by the substitution fo(z), 2';) = (z, 2,)/T $1 (T is hyperparameter) to the definition of RPC:
RRPC - 2(N 1)
2N
3 S 2 2-2(N 1)
7
Σ
2N
For losses other than RPC, hidden normalization of st,j is often required by replacing 2, 23 with (z,- xx,)))//1 /2////2/zz// CPC and WPC adopt this, while other objectives needs it to help stabilize training variance. RPC does not need this normalization.
20

---

Published as a conference paper at ICLR 2021
Confidence Interval of JRCC and Jcrc
Objective
CIFAR 10
CIFAR 100
ImageNet
JCPC
(91.09%, 91.13%)
77..1 77.36%
(3.399% 73.48%)
JRPC
(91.16%, 91.47%)
(77.41%, 41, 77.98%)
(73.92%, 74.43%)
Table 4: Confidence Intervals of performances of JRPC and JePc on CIFAR-10/-100 and ImageNet.
A.8 CIFAR-10/-101 AND IMAGENET EXPERIMENTS DETAILS
ImageNet Following the settings in (Chen et al., 2020b;e), we train the model on Cloud TPU with 128 cores, with batch size of 4, 096 and global batch normalization (loffe & Szegedy, 2015). Here we refer to the term batch size as the number of images (or utterances in the speech experiments) we use per GPU, while the term minibatch size refers to the number of negative samples used to calculate the objective, such as CPC or our proposed RPC. The largest model we train is 1.22-laeer ResNet with selective kernels (SK) (Li et al., 2019) and 2x wider channels. We use the LARS optimizer (You et al., 2017) with momentum 0.9. The learning rate linearly increases for the first 20 epochs, reaching a maximum of 6.4, then decayed with cosine decay schedule. The weight decay is 10-4 A MLP projection head g-) with three layers is used top of the ResNet encoder. Unlike Chen et al. (2020c), we do not use a memory buffer, and train the model for only 100 epochs rather than 800 epochs due to computational constraints. These two options slightly reduce CPC's performance benchmark for about 2% with the exact same setting. The unsupervised pre-training is followed by a supervised fine-tuning. Following SimCLRv2 (Chen et al., 2020b;c), we fine-tune the 3-layer g(-) for the downstream tasks. We use learning rates 0.16 and 0.064 for standard 50-layer ResNet and larger 152-layer ResNet respectively, and weight decay and learning rate warmup are removed. Different from Chen et al. (2020c), we use a batch size of 4, 096, and we do not use global batch normalization for fine-tuning. For //PC we disable hidden normalization and use a temperature = 32. For all other objectives, we use hidden normalization and 0.1 following previous work (Chen et al., 2020c). For relative parameters, we use a = 0.3, 3 0.001, 7 = 0.1 and 0.3, B = 0.001, y 0.005 for ResNet-50 and ResNet-152 respectively.
CIFAR-10/-100 Following the settings in (Chen et al., 2020b), we train the model on a single GPU, with a batch size of 512 and global batch normalization (loffe & Szegedy, 2015). We use ResNet (He et al., 2016) of depth 18 and depth 50, and does not use Selective Kernel (Li et al., 2019) or a multiplied width size. We use the LARS optimizer (You et al., 2017) with momentum 0.9. The learning rate linearly increases for the first 20 epochs. reaching maximum of 6.4, then decayed with cosine decay schedule. The weight decay is 10-1 A MLP projection head g(-) with three layers is used on top of the ResNet encoder. Unlike Chen et al. (2020c), we do not use memory buffer. We train the model for 1000 epochs. The unsupervised pre-training is followed by a supervised fine-tuning. Following SimCLRv2 (Chen al., 2020b;e), we fine-tune the 3-layer g() for the downstream tasks. We use learning rates 0.16 for standard 50-layer ResNet, and weight decay and learning rate warmup are removed. For JRPC we disable hidden normalization and use temperature T = 128. For all other objectives, we use hidden normalization and r = 0.5 following previous work (Chen et al., 2020c). For relative parameters, we use a = 1.0, 3 = 0.005, and 7 = 1.0.
STL-10 We also perform the pre-training and fine-tuning on STL-10 (Coates et al., 2011) using the model proposed in Chuang et al. (2020). Chuang et al. (2020) proposed to indirectly approximate the distribution of negative samples so that the objective is debiased. However, their implementation of contrastive learning is consistent with Chen et al. (2020b). We use ResNet with depth 50 as an encoder for pre-training, with Adam optimizer, learning rate 0.001 and weight decay 10-6, The temperature is set to 0.5 for all objectives other than JRPC, which disables hidden normalization and use т = 128. The downstream task performance increases from 83.4% of JCPC to 84.1% of JRPC-
Confidence Interval We also provide the confidence interval of JRPC and Jerc on CIFAR-10, CIFAR-100 and ImageNet, using R8sNet-18, R8 and ResNet-50 respectively (95% confi-
For WPC (Ozair et al., 2019). the global hatch normalization during pretraining is disabled since enforce 1-Lipschitz by gradient penalty (Gulrajani et al., 2017).
21

---

Published as a conference paper at ICLR 2021
dence level is chosen) in Table 4. Both CPC and RPC the same experimental settings throughout this paper. Here we use the relative parameters (a = 1.0, 3 = 0.005. y = 1.0) in JRPC which gives the best performance on CIFAR-10 10. The confidence intervals of CPC do not overlap with the con- fidence intervals of RPC, which means the difference of the downstream task performance between RPC and CPC is statistically significant.
A.9 RELATIVE PREDICTIVE CODING ON SPEECH
For speech representation learning, we adopt the general architecture from Oord et al. (2018). Given an input signal #1:7 with T time steps, first pass it through encoder de parametrized by to produce a sequence of hidden representations {h1rr} where h, = (g(2x)) After that, we obtain the contextual representation C₁ at time step & with sequential model up, parametrized by p: c (h1, h,)) where C2 contains context information before time step t. For unsupervised pre-training, we use a multi-layer convolutional network as the encoder do. and an LSTM with hidden dimension 256 as the sequential model vp- Here, the contrastiveness is between the positive pair (h++1, 6e) where k is the number of time steps ahead, and the negative pairs (h,, C.), where h; is randomly sampled from N, batch of hidden representation of signals assumed to be unrelated to c.. The scoring function f based on Equation at step and look-ahead will be fx = fx (h, c₁) = (xp((h) W&C,), where We is learnable linear transformation defined separately for each & {I, {I...,,K }} and is predetermined as 12 time steps. The loss in Equation will then be formulated
RPC 1,k = (fa((hkkk ci)
a W
(1e4k Suhhi ci) - (heek Cx)
7
fhh cx)) (8) Σ K.EN
2|
MEN
We use the following relative parameters: a = 1, = 0.25, and y= 1, and we use the temperature 16 for Jarc. For Jcre follow the original implementation which sets T 1. We fix all other experimental setups, including architecture, learning rate, and optimizer. As shown in Table 3. JRPC has better downstream task performance, and is closer the performance from fully supervised model.
A.10 EMPIRICAL OBSERVATIONS ON VARIANCE AND MINIBATCUI SIZE
Variance Experiment Setup We perform the variance comparison of JDV, JNwJ and the pro- posed JRPC- The empirical experiments are performed using SimCLRv2 (Chen et al., 2020c) on CIFAR-10 dataset. We ResNet of depth 18, with batch size of 512. We train each objective with 30K training steps and record their value. In Figure 1, we use a temperature T = 128 for all objectives. Unlike other experiments, where hidden normalization is applied other objectives, we remove hidden normarlization for all objectives due to the reality that objectives after normalization does not reflect their original values. From Figure 1, JRPC enjoys lower variance and more stable training compared to JDY and JNWJ-
Minibatch Size Experimental Setup We perform experiments on the effect of batch size on downstream performances for different objective. The experiments are performed using SimCLRv2 (Chen et al., 2020c) on CIFAR-10 dataset, as well as the model from Rivière al. (2020) on LibriSpeech- 100h dataset (Panayotov et al., 2015). For vision task, we use the default tempera- ture т 0.5 from Chen et al. (2020c) and hidden normalization mentioned Section 3 for JCPc- For JRPC in vision and speech tasks we use temperature of 128 and 16 respectively, both without hidden normalization.
A.11 MUTCAL INFORMATION ESTIMATION
Our method is compared with baseline methods CPC (Oord et al., 2018), NWJ (Nguyen et al., 2010), JSD (Nowozin et al., 2016), and SMILE (Song & Ermon, 2019). All the approaches consider the same design of fxr y). which is 3-layer neural network taking concatenated (ar, y) the input. We also fix the learning rate, the optimizer, and the minibatch size across all the estimators for fair comparison.
We present results of mutual information by Relative Predictive Coding using different sets of rela- tive parameters in Figure 4. In the first row, we set B 10-3 7 1, and experiment with different
22

---

Published as a conference paper at ICLR 2021
RPC 0.
RPC
RPC
RPC
RPC 2.0
Cabic
Tiue
Puta) 20301
5232 333XX SHRRI 20233
5028 11231
19081 10000
Iec0 15803 1M8E 2000
5031 D0X
RPC 1c405
C.0001
RPC 0.001
2500 20300
5223
3030
15000 20230
5050 10030
19000
20000
5000 15892 ISACE 2000
5000
RPC
RPC
RPC
20300
5200
30300
2500
15000 20006
15802
5000
2000
Figure 4: Mutual information estimation by RPC performed 20-d correlated Gaussian distribu- tion, with different sets of relative parameters.
parameterized Logistic
12
12
10
6-
4-
Gouttion
10000
15000 20000
5000
10000
Figure 5: Mutual information estimation by DoE performed 20-d correlated Gaussian distribu- tion. The figure on the left shows parametrization under Gaussian (correctly specified), and the figure on the right shows parametrization under Logistic (mis-specified).
values. In the second row, we set = 1, 1 and in the last row we set = 1. 3 = 10 From the figure, a small around 10-3 and large 7 around 1.0 is crucial for an estimation that is relatively low bias and low variance. This conclusion is consistent with Section 3 in the main text.
We also performed comparison between JRPC and Difference of Entropies (DoE) (McAllester & Stratos, 2020). We performed two sets of experiments: in the first set of experiments we compare JRPC and DoE when MI large (> 100 nats). while in the second set of experiments we compare JRPC and DoE using the setup in this section (MI < 12 nats and MI increases by 2 per 4k training steps). On the one hand, when MI large (> 100 nats), we acknowledge that DoE is performing well on MI estimation, compared to JRPC which only estimates the MI around 20. This analysis is based on the code from https: /giit hub. com/ oarlstratos doo. On the other hand, when the true MI is small, the DoE method is more unstable than JRPC, shown in Figure 5. Figure 5 illustrates the results of the DoE method when the distribution is isotropic Gaussian (correctly specified) or Logistic (mis-specified). Figure 3 only shows the results using Gaussian.
23


Reviews:
[{"review_id": "uxG_1fi-7Y6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new contrastive representation objective that has good training stability, minibatch size sensitivity, and downstream task performance. This objective is a generalization of Chi-square divergence, the optimal solution is the density ratio of joint distribution and product of marginal distributions, the estimation is consistent and its variance goes to 0 as sample size goes to infinite, so the paper is theoretical sound. The authors conduct comprehensive experiments to show that the training based on this objective is stable, not sensitive to batch size and leads to good downstream task performance  in vision and phoneme and speaker classification. \n\nHowever the theoretical results don't provide any clue to explain why this estimation leads to better downstream task performance, where in mutual information (MI) estimator it is easy to understand since MI is a good measure of dependency of two random variables. Section 3.5 states the relation to MI estimation, but it is just a plug-in method that won't be able to say anything on the goodness. \n\nFurthermore the paper misses an important reference, David McAllester and Karl Stratos, Formal Limitations on the Measurement of Mutual Information, AISTATS 2020, where David and Karl propose a method called DoE that has a good estimation of mutual information even when the mutual information is large. So I think the authors should compare RPC with DoE in the synthetic data experiment in Section 3.5 when the mutual information is large, say 100, instead of 10. In the case of large MI, all other methods fail to provide a good estimation.\n\nFrom my experience, when CPC is applied to ASR, the batch size is not a sensitive factor to WER results. \n\nIn the notation and In section 2.1, second line from bottom, I don't think it is appropriate to use joint distribution for a related or positive pair, product of marginal distributions for an un-related or negative pair.  It is the density ratio matters. For positive pair (X,Y), its MI is large, and for negative pair (X,Y), its MI close to 0. \n\nIn the proof of Lemma 5, there is a typo in the second line from bottom, the second term should be expection of P(x)P(Y), not P(X,Y).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "s6BcvfYM79j", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose RPC (Relative Predictive Coding), which is supposed to improve training stability (with chi-squared distance based regularization), minibatch size sensitivity (avoid sampling large batches), and downstream task performance (show generalization).\nAuthors discuss estimation of MI and/or probability ratio (of related divided by unrelated egs). Proposed solution stably estimates this. Experiments are convincing. It is a good direction in self-supervised training with convenient training schemes.\nSome weaknesses:\n1. Fix alpha=0 and find ratio or values of beta and gamma which gives maximum performance. It would be interesting since low value of alpha is also giving good performance.\n2. While discussing sensitivity to batch size, larger batch sizes should be tried since it is discussed in the initial part of paper that SimCLRv2 requires huge batch size.\n3. Since the proposal is generic, can authors give a word on using this on something other than SimCLRv2?\n4. Rather than reporting specific values of alpha, beta, gamma (the proposed \"relative parameters\"), if results can be reported in graph format, it would be vastly more helpful. For e.g. fix alpha=0.001 and x and y axis of plot could be other two relative parameters. (this is related to my point 1)", "extra_scores": {}, "extra_text": {}}, {"review_id": "EfU1QGmi2qs", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new objective for self-supervised contrastive learning. In the general framework proposed by Tsai et al. (2020b), the proposed method boils down to using a divergence related to $\\chi^2$-divergence. Compared to other objectives for contrastive learning, the authors illustrate the advantages of the proposed one in training stability (or easiness to train), sensitivity to batch size, and downstream task performance. However, introducing three new hyperparameters is a cause of concern since they make it more difficult to select optimal hyperparameters. Also, some important details of the experiments are missing. For example, how many runs to obtain the results shown in Tables 2 & 3? What's the confidence interval on the results? Any test to establish the statistical significance? What are the settings for supervised training? When the authors compare the results among different methods, did they select the optimal hyperparameters (e.g., learning rate) separately for each method?", "extra_scores": {}, "extra_text": {}}, {"review_id": "1sN2gUI1jnw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors provide a clear review of different divergences used in contrastive learning and their relative strengths and weaknesses in terms of training stability, minibatch size dependence, and usefulness on downstream tasks. This motivates the need for a new divergence which they introduce based upon chi-squared divergence.\n\nThey provide strong empirical and theoretical support for the new divergence, with extensive experiments on large-scale image and speech classification tasks. They also perform comparison studies across batch size and training stability that support their earlier arguments, and a hyperparameter sweep across term weights to make it clearer how to tune them in later work. Further, they demonstrate the decreased bias and variance in MI estimation experiments.\n\nThe paper is well-written, and provides helpful context to not just motivate the value of the new technique, but quantitatively and qualitatively contrast with existing techniques that helps inform the reader about the broader field. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents new contrastive based self-supervised objective based on Chi squared divergence that helps with mini batch sensitivity, training stability and improved downstream performance.
An accept.
Decision: Accept (Poster)