SELF-SUPERVISED REPRESENTATION LEARNING WITH RELATIVE PREDICTIVE CODING
Year: 2021

Abstract:
This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives. We empirically verify the effectiveness of RPC on benchmark vision and speech self-supervised learning tasks. Lastly, we relate RPC with mutual information (MI) estimation, showing RPC can be used to estimate MI with low variance 1.

heading: 1 INTRODUCTION
text: Unsupervised learning has drawn tremendous attention recently because it can extract rich representations without label supervision. Self-supervised learning, a subset of unsupervised learning, learns representations by allowing the data to provide supervision (Devlin et al., 2018). Among its mainstream strategies, self-supervised contrastive learning has been successful in visual object recognition (He et al., 2020; Tian et al., 2019; Chen et al., 2020c), speech recognition (Oord et al., 2018; Rivière et al., 2020), language modeling (Kong et al., 2019), graph representation learning (Velickovic et al., 2019) and reinforcement learning (Kipf et al., 2019). The idea of self-supervised contrastive learning is to learn latent representations such that related instances (e.g., patches from the same image; defined as positive pairs) will have representations within close distance, while unrelated instances (e.g., patches from two different images; defined as negative pairs) will have distant representations (Arora et al., 2019).
Prior work has formulated the contrastive learning objectives as maximizing the divergence between the distribution of related and unrelated instances. In this regard, different divergence measurement often leads to different loss function design. For example, variational mutual information (MI) estimation (Poole et al., 2019) inspires Contrastive Predictive Coding (CPC) (Oord et al., 2018). Note that MI is also the KL-divergence between the distributions of related and unrelated instances (Cover & Thomas, 2012). While the choices of the contrastive learning objectives are abundant (Hjelm et al., 2018; Poole et al., 2019; Ozair et al., 2019), we point out that there are three challenges faced by existing methods.
The first challenge is the training stability, where an unstable training process with high variance may be problematic. For example, Hjelm et al. (2018); Tschannen et al. (2019); Tsai et al. (2020b) show that the contrastive objectives with large variance cause numerical issues and have a poor downstream performance with their learned representations. The second challenge is the sensitivity to minibatch size, where the objectives requiring a huge minibatch size may restrict their practical usage. For instance, SimCLRv2 (Chen et al., 2020c) utilizes CPC as its contrastive objective and reaches state-of-the-art performances on multiple self-supervised and semi-supervised benchmarks. Nonetheless, the objective is trained with a minibatch size of 8, 192, and this scale of training requires enormous computational power. The third challenge is the downstream task performance, which is the one that we would like to emphasize the most. For this reason, in most cases, CPC
1Project page: https://github.com/martinmamql/relative_predictive_coding
is the objective that we would adopt for contrastive representation learning, due to its favorable performance in downstream tasks (Tschannen et al., 2019; Baevski et al., 2020).
This paper presents a new contrastive representation learning objective: the Relative Predictive Coding (RPC), which attempts to achieve a good balance among these three challenges: training stability, sensitivity to minibatch size, and downstream task performance. At the core of RPC is the relative parameters, which are used to regularize RPC for its boundedness and low variance. From a modeling perspective, the relative parameters act as a `2 regularization for RPC. From a statistical perspective, the relative parameters prevent RPC from growing to extreme values, as well as upper bound its variance. In addition to the relative parameters, RPC contains no logarithm and exponential, which are the main cause of the training instability for prior contrastive learning objectives (Song & Ermon, 2019).
To empirically verify the effectiveness of RPC, we consider benchmark self-supervised representation learning tasks, including visual object classification on CIFAR-10/-100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and ImageNet (Russakovsky et al., 2015) and speech recognition on LibriSpeech (Panayotov et al., 2015). Comparing RPC to prior contrastive learning objectives, we observe a lower variance during training, a lower minibatch size sensitivity, and consistent performance improvement. Lastly, we also relate RPC with MI estimation, empirically showing that RPC can estimate MI with low variance.

heading: 2 PROPOSED METHOD
text: This paper presents a new contrastive representation learning objective - the Relative Predictive Coding (RPC). At a high level, RPC 1) introduces the relative parameters to regularize the objective for boundedness and low variance; and 2) achieves a good balance among the three challenges in the contrastive representation learning objectives: training stability, sensitivity to minibatch size, and downstream task performance. We begin by describing prior contrastive objectives along with their limitations on the three challenges in Section 2.1. Then, we detail our presented objective and its modeling benefits in Section 2.2. An overview of different contrastive learning objectives is provided in Table 1. We defer all the proofs in Appendix.
Notation We use an uppercase letter to denote a random variable (e.g., X), a lower case letter to denote the outcome of this random variable (e.g., x), and a calligraphy letter to denote the sample space of this random variable (e.g., X ). Next, if the samples (x, y) are related (or positively-paired), we refer (x, y) ∼ PXY with PXY being the joint distribution of X × Y . If the samples (x, y) are unrelated (negatively-paired), we refer (x, y) ∼ PXPY with PXPY being the product of marginal distributions overX×Y . Last, we define f ∈ F for F being any class of functions f : X ×Y → R.

heading: 2.1 PRELIMINARY
text: Contrastive representation learning encourages the contrastiveness between the positive and the negative pairs of the representations from the related data X and Y . Specifically, when sampling a pair
of representations (x, y) from their joint distribution ((x, y) ∼ PXY ), this pair is defined as a positive pair; when sampling from the product of marginals ((x, y) ∼ PXPY ), this pair is defined as a negative pair. Then, Tsai et al. (2020b) formalizes this idea such that the contrastiveness of the representations can be measured by the divergence between PXY and PXPY , where higher divergence suggests better contrastiveness. To better understand prior contrastive learning objectives, we categorize them in terms of different divergence measurements between PXY and PXPY , with their detailed objectives presented in Table 1.
We instantiate the discussion using Contrastive Predictive Coding (Oord et al., 2018, JCPC), which is a lower bound of DKL(PXY ‖PXPY ) with DKL referring to the KL-divergence:
JCPC(X,Y ) := sup f∈F
E(x,y1)∼PXY ,{yj}Nj=2∼PY [ log ef(x,y1)
1 N ∑N j=1 e f(x,yj)
] . (1)
Then, Oord et al. (2018) presents to maximize JCPC(X,Y ), so that the learned representations X and Y have high contrastiveness. We note that JCPC has been commonly used in many recent self-supervised representation learning frameworks (He et al., 2020; Chen et al., 2020b), where they constrain the function to be f(x, y) = cosine(x, y) with cosine(·) being cosine similarity. Under this function design, maximizing JCPC leads the representations of related pairs to be close and representations of unrelated pairs to be distant.
The category of modeling DKL(PXY ‖PXPY ) also includes the Donsker-Varadhan objective (JDV (Donsker & Varadhan, 1975; Belghazi et al., 2018)) and the Nguyen-Wainright-Jordan objective (JNWJ (Nguyen et al., 2010; Belghazi et al., 2018)), where Belghazi et al. (2018); Tsai et al. (2020b) show that JDV(X,Y ) = JNWJ(X,Y ) = DKL(PXY ‖PXPY ). The other divergence measurements considered in prior work are DJS(PXY ‖PXPY ) (with DJS referring to the Jenson-Shannon divergence) and DWass(PXY ‖PXPY ) (with DWass referring to the Wassersteindivergence). The instance of modeling DJS(PXY ‖PXPY ) is the Jensen-Shannon f-GAN objective( JJS (Nowozin et al., 2016; Hjelm et al., 2018) ) , where JJS(X,Y ) = 2 ( DJS(PXY ‖PXPY ) −
log 2 ) .2 The instance of modeling DWass(PXY ‖PXPY ) is the Wasserstein Predictive Coding(
JWPC (Ozair et al., 2019) ) , where JWPC(X,Y ) modifies JCPC(X,Y ) objective (equation 1) by searching the function from F to FL. FL denotes any class of 1-Lipschitz continuous functions from (X × Y) to R, and thus FL ⊂ F . Ozair et al. (2019) shows that JWPC(X,Y ) is the lower bound of bothDKL(PXY ‖PXPY ) andDWass(PXY ‖PXPY ). See Table 1 for all the equations. To conclude, the contrastive representation learning objectives are unsupervised representation learning methods that maximize the distribution divergence between PXY and PXPY . The learned representations cause high contrastiveness, and recent work (Arora et al., 2019; Tsai et al., 2020a) theoretically show that highly-contrastive representations could improve the performance on downstream tasks.
After discussing prior contrastive representation learning objectives, we point out three challenges in their practical deployments: training stability, sensitivity to minibatch training size, and downstream task performance. In particular, the three challenges can hardly be handled well at the same time, where we highlight the conclusions in Table 1. Training Stability: The training stability highly relates to the variance of the objectives, where Song & Ermon (2019) shows that JDV and JNWJ exhibit inevitable high variance due to their inclusion of exponential function. As pointed out by Tsai et al. (2020b), JCPC, JWPC, and JJS have better training stability because JCPC and JWPC can be realized as a multi-class classification task and JJS can be realized as a binary classification task. The cross-entropy loss adopted in JCPC, JWPC, and JJS is highly-optimized and stable in existing optimization package (Abadi et al., 2016; Paszke et al., 2019). Sensitivity to minibatch training size: Among all the prior contrastive representation learning methods, JCPC is known to be sensitive to the minibatch training size (Ozair et al., 2019). Taking a closer look at equation 1, JCPC deploys an instance selection such that y1 should be selected from {y1, y2, · · · , yN}, with (x, y1) ∼ PXY , (x, yj>1) ∼ PXPY with N being the minibatch size. Previous work (Poole et al., 2019; Song & Ermon, 2019; Chen et al., 2020b; Caron et al., 2020) showed that a large N results in a more challenging instance selection and forces JCPC to have a better contrastiveness of y1 (related instance for x) against {yj}Nj=2 (unrelated instance for x). JDV, JNWJ, and JJS do not consider
2JJS(X,Y ) achieves its supreme value when f∗(x, y) = log(p(x, y)/p(x)p(y)) (Tsai et al., 2020b). Plugin f∗(x, y) into JJS(X,Y ), we can conclude JJS(X,Y ) = 2(DJS(PXY ‖PXPY )− log 2).
the instance selection, and JWPC reduces the minibatch training size sensitivity by enforcing 1- Lipschitz constraint. Downstream Task Performance: The downstream task performance is what we care the most among all the three challenges. JCPC has been the most popular objective as it manifests superior performance over the other alternatives (Tschannen et al., 2019; Tsai et al., 2020b;a). We note that although JWPC shows better performance on Omniglot (Lake et al., 2015) and CelebA (Liu et al., 2015) datasets, we empirically find it not generalizing well to CIFAR-10/100 (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015).

heading: 2.2 RELATIVE PREDICTIVE CODING
text: In this paper, we present Relative Predictive Coding (RPC), which achieves a good balance among the three challenges mentioned above:
JRPC(X,Y ) := sup f∈F
EPXY [f(x, y)]−αEPXPY [f(x, y)]− β
2 EPXY
[ f2(x, y) ] −γ
2 EPXPY
[ f2(x, y) ] ,
(2) where α > 0, β > 0, γ > 0 are hyper-parameters and we define them as relative parameters. Intuitively, JRPC contains no logarithm or exponential, potentially preventing unstable training due to numerical issues. Now, we discuss the roles of α, β, γ. At a first glance, α acts to discourage the scores of PXY and PXPY from being close, and β/γ acts as a `2 regularization coefficient to stop f from becoming large. For a deeper analysis, the relative parameters act to regularize our objective for boundedness and low variance. To show this claim, we first present the following lemma:
Lemma 1 (Optimal Solution for JRPC) Let r(x, y) = p(x,y)p(x)p(y) be the density ratio. JRPC has the optimal solution f∗(x, y) = r(x,y)−αβ r(x,y)+γ := rα,β,γ(x, y) with − α γ ≤ rα,β,γ ≤ 1 β .
Lemma 1 suggests that JRPC achieves its supreme value at the ratio rα,β,γ(x, y) indexed by the relative parameters α, β, γ (i.e., we term rα,β,γ(x, y) as the relative density ratio). We note that rα,β,γ(x, y) is an increasing function w.r.t. r(x, y) and is nicely bounded even when r(x, y) is large. We will now show that the bounded rα,β,γ suggests the empirical estimation of JRPC has boundeness and low variance. In particular, let {xi, yi}ni=1 be n samples drawn uniformly at random from PXY and {x′j , y′j}mj=1 be m samples drawn uniformly at random from PXPY . Then, we use neural networks to empirically estimate JRPC as Ĵ m,n RPC:
Definition 1 (Ĵm,nRPC, empirical estimation of JRPC) We parametrize f via a family of neural networks FΘ := {fθ : θ ∈ Θ ⊆ Rd} where d ∈ N and Θ is compact. Then, Ĵm,nRPC = supfθ∈FΘ 1 n ∑n i=1 fθ(xi, yi)− 1 m ∑m j=1 αfθ(x ′ j , y ′ j)− 1n ∑n i=1 β 2 f 2 θ (xi, yi)− 1m ∑m j=1 γ 2 f 2 θ (x ′ j , y ′ j).
Proposition 1 (Boundedness of Ĵm,nRPC, informal) 0 ≤ JRPC ≤ 1 2β + α2 2γ . Then, with probability at least 1− δ, |JRPC − Ĵm,nRPC| = O( √ d+log (1/δ) n′ ), where n ′ = min {n,m}.
Proposition 2 (Variance of Ĵm,nRPC, informal) There exist universal constants c1 and c2 that depend only on α, β, γ, such that Var[Ĵm,nRPC] = O ( c1 n + c2 m ) .
From the two propositions, whenm and n are large, i.e., the sample sizes are large, Ĵm,nRPC is bounded, and its variance vanishes to 0. First, the boundedness of Ĵm,nRPC suggests Ĵ m,n RPC will not grow to extremely large or small values. Prior contrastive learning objectives with good training stability (e.g., JCPC/JJS/JWPC) also have the boundedness of their objective values. For instance, the empirical estimation of JCPC is less than logN (equation 1) (Poole et al., 2019). Nevertheless, JCPC often performs the best only when minibatch size is large, and empirical performances of JJS and JWPC are not as competitive as JCPC. Second, the upper bound of the variance implies the training of Ĵm,nRPC can be stable, and in practice we observe a much smaller value than the stated upper bound. On the contrary, Song & Ermon (2019) shows that the empirical estimations of JDV and JNWJ exhibit inevitable variances that grow exponentially with the true DKL(PXY ‖PXPY ). Lastly, similar to prior contrastive learning objective that are related to distribution divergence measurement, we associate JRPC with the Chi-square divergence Dχ2(PXY ‖PXPY ) =
EPXPY [r2(x, y)] − 1 (Nielsen & Nock, 2013). The derivations are provided in Appendix. By having P ′ = ββ+γPXY + γ β+γPXPY as the mixture distribution of PXY and PXPY , we can rewrite JRPC(X,Y ) as JRPC(X,Y ) = β+γ2 EP ′ [r 2 α,β,γ(x, y)]. Hence, JRPC can be regarded as a generalization of Dχ2 with the relative parameters α, β, γ, where Dχ2 can be recovered from JRPC by specializing α = 0, β = 0 and γ = 1 (e.g., Dχ2 = 2JRPC|α=β=0,γ=1 − 1). Note that JRPC may not be a formal divergence measure with arbitrary α, β, γ.

heading: 3 EXPERIMENTS
text: We provide an overview of the experimental section. First, we conduct benchmark self-supervised representation learning tasks spanning visual object classification and speech recognition. This set of experiments are designed to discuss the three challenges of the contrastive representation learning objectives: downstream task performance (Section 3.1), training stability (Section 3.2), and minibatch size sensitivity (Section 3.3). We also provide an ablation study on the choices of the relative parameters in JRPC (Section 3.4). On these experiments we found that JRPC achieves a lower variance during training, a lower batch size insensitivity, and consistent performance improvement. Second, we relate JRPC with mutual information (MI) estimation (Section 3.5). The connection is that MI is an average statistic of the density ratio, and we have shown that the optimal solution of JRPC is the relative density ratio (see Lemma 1). Thus we could estimate MI using the density ratio transformed from the optimal solution of JRPC. On these two sets of experiments, we fairly compare JRPC with other contrastive learning objectives. Particularly, across different objectives, we fix the network, learning rate, optimizer, and batch size (we use the default configurations suggested by the original implementations from Chen et al. (2020c), Rivière et al. (2020) and Tsai et al. (2020b).) The only difference will be the objective itself. In what follows, we perform the first set of experiments. We defer experimental details in the Appendix.
Datasets. For the visual objective classification, we consider CIFAR-10/-100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and ImageNet (Russakovsky et al., 2015). CIFAR-10/-100 and ImageNet contain labeled images only, while STL-10 contains labeled and unlabeled images. For the speech recognition, we consider LibriSpeech-100h (Panayotov et al., 2015) dataset, which contains 100 hours of 16kHz English speech from 251 speakers with 41 types of phonemes.
Training and Evaluation Details. For the vision experiments, we follow the setup from SimCLRv2 (Chen et al., 2020c), which considers visual object recognition as its downstream task. For the speech experiments, we follow the setup from prior work (Oord et al., 2018; Rivière et al., 2020), which consider phoneme classification and speaker identification as the downstream tasks. Then, we briefly discuss the training and evaluation details into three modules: 1) related and unrelated data construction, 2) pre-training, and 3) fine-tuning and evaluation. For more details, please refer to Appendix or the original implementations. . Related and Unrelated Data Construction. In the vision experiment, we construct the related images by applying different augmentations on the same image. Hence, when (x, y) ∼ PXY , x and y are the same image with different augmentations. The unrelated images are two randomly selected samples. In the speech experiment, we define the current latent feature (feature at time t) and the future samples (samples at time > t) as related data. In other words, the feature in the latent space should contain information that can be used to infer future time steps. A latent feature and randomly selected samples would be considered as unrelated data. . Pre-training. The pre-training stage refers to the self-supervised training by a contrastive learning objective. Our training objective is defined in Definition 1, where we use neural networks to parametrize the function using the constructed related and unrelated data. Convolutional neural networks are used for vision experiments. Transformers (Vaswani et al., 2017) and LSTMs (Hochreiter & Schmidhuber, 1997) are used for speech experiments. . Fine-tuning and Evaluation. After the pre-training stage, we fix the parameters in the pre-trained networks and add a small fine-tuning network on top of them. Then, we fine-tune this small network with the downstream labels in the data’s training split. For the fine-tuning network, both vision and speech experiments consider multi-layer perceptrons. Last, we evaluate the fine-tuned representations on the data’s test split. We would like to point out that we do not normalize the hidden representations encoded by the pre-training neural network for loss calculation. This hidden nor-
malization technique is widely applied (Tian et al., 2019; Chen et al., 2020b;c) to stabilize training and increase performance for prior objectives, but we find it unnecessary in JRPC.

heading: 3.1 DOWNSTREAM TASK PERFORMANCES ON VISION AND SPEECH
text: For the downstream task performance in the vision domain, we test the proposed JRPC and other contrastive learning objectives on CIFAR-10/-100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and ImageNet ILSVRC-2012 (Russakovsky et al., 2015). Here we report the best performances JRPC can get on each dataset (we include experimental details in A.7.) Table 2 shows that the proposed JRPC outperforms other objectives on all datasets. Using JRPC on the largest network (ResNet with depth of 152, channel width of 2 and selective kernels), the performance jumps from 77.80% of JCPC to 78.40% of JRPC.
Regarding speech representation learning, the downstream performance for phoneme and speaker classification are shown in Table 3 (we defer experimental details in Appendix A.9.) Compared to JCPC, JRPC improves the phoneme classification results with 4.8 percent and the speaker classification results with 0.3 percent, which is closer to the fully supervised model. Overall, the proposed JRPC performs better than other unsupervised learning objectives on both phoneme classification and speaker classification tasks.

heading: 3.2 TRAINING STABILITY
text: We provide empirical training stability comparisons on JDV, JNWJ, JCPC and JRPC by plotting the values of the objectives as the training step increases. We apply the four objectives to the SimCLRv2 framework and train on the CIFAR-10 dataset. All setups of training are exactly the same except the objectives. From our experiments, JDV and JNWJ soon explode to NaN and disrupt training (shown as early stopping in Figure 1a; extremely large values are not plotted due to scale constraints). On the other hand, JRPC and JCPC has low variance, and both enjoy stable training. As a result, performances using the representation learned from unstable JDV and JNWJ suffer in downstream task, while representation learned by JRPC and JCPC work much better.

heading: 3.3 MINIBATCH SIZE SENSITIVITY
text: We then provide the analysis on the effect of minibatch size on JRPC and JCPC, since JCPC is known to be sensitive to minibatch size (Poole et al., 2019). We train SimCLRv2 (Chen et al., 2020c) on CIFAR-10 and the model from Rivière et al. (2020) on LibriSpeech-100h using JRPC and JCPC with different minibatch sizes. The settings of relative parameters are the same as Section 3.2. From Figure 1b and 1c, we can observe that both JRPC and JCPC achieve their optimal performance at a large minibatch size. However, when the minibatch size decreases, the performance of JCPC shows higher sensitivity and suffers more when the number of minibatch samples is small. The result suggests that the proposed method might be less sensitive to the change of minibatch size compared to JCPC given the same training settings.

heading: 3.4 EFFECT OF RELATIVE PARAMETERS
text: We study the effect of different combinations of relative parameters in JRPC by comparing downstream performances on visual object recognition. We train SimCLRv2 on CIFAR-10 with different combinations of α, β and γ in JRPC and fix all other experimental settings. We choose α ∈ {0, 0.001, 1.0}, β ∈ {0, 0.001, 1.0}, γ ∈ {0, 0.001, 1.0} and we report the best performances under each combination of α, β, and γ. From Figure 2, we first observe that α > 0 has better downstream performance than α = 0 when β and γ are fixed. This observation is as expected, since α > 0 encourages representations of related and unrelated samples to be pushed away. Then, we find that a small but nonzero β (β = 0.001) and a large γ (γ = 1.0) give the best performance compared to other combinations. Since β and γ serve as the coefficients of `2 regularization, the results imply that the regularization is a strong and sensitive factor that will influence the performance. The results here are not as competitive as Table 2 because the CIFAR-10 result reported in Table 2 is using a set of relative parameters (α = 1.0, β = 0.005, γ = 1.0) that is different from the combinations in this subsection. Also, we use quite different ranges of γ on ImageNet (see A.7 for details.) In conclusion, we find empirically that a non-zero α, a small β and a large γ will lead to the optimal representation for the downstream task on CIFAR-10.

heading: 3.5 RELATION TO MUTUAL INFORMATION ESTIMATION
text: The presented approach also closely relates to mutual information estimation. For random variables X and Y with joint distribution PXY and product of marginals PXPY , the mutual information is defined as I(X;Y ) = DKL(PXY ‖PXPY ). Lemma 1 states that given optimal solution f∗(x, y) of JRPC, we can get the density ratio r(x, y) := p(x, y)/p(x)p(y) as r(x, y) = γ/β+α 1−βf∗(x,y) − γ β . We
can empirically estimate r̂(x, y) from the estimated f̂(x, y) via this transformation, and use r̂(x, y) to estimate mutual information (Tsai et al., 2020b). Specifically, I(X;Y ) ≈ 1n ∑n i=1 log r̂(xi, yi) with (xi, yi) ∼ P⊗nX,Y , where P ⊗n X,Y is the uniformly sampled empirical distribution of PX,Y .
We follow prior work (Poole et al., 2019; Song & Ermon, 2019; Tsai et al., 2020b) for the experiments. We consider X and Y as two 20-dimensional Gaussians with correlation ρ, and our goal is to estimate the mutual information I(X;Y ). Then, we perform a cubic transformation on y so that y 7→ y3. The first task is referred to as Gaussian task and the second is referred to as Cubic task, where both have the ground truth I(X;Y ) = −10log (1 − ρ2). The models are trained on 20, 000 steps with I(X;Y ) starting at 2 and increased by 2 per 4, 000 steps. Our method is compared with baseline methods JCPC (Oord et al., 2018), JNWJ (Nguyen et al., 2010), JJS (Nowozin et al., 2016), SMILE (Song & Ermon, 2019) and Difference of Entropies (DoE) (McAllester & Stratos, 2020). All approaches use the same network design, learning rate, optimizer and minibatch size for a fair comparison. First, we observe JCPC (Oord et al., 2018) has the smallest variance, while it exhibits a large bias (the estimated mutual information from JCPC has an upper bound log(batch size)). Second, JNWJ (Nguyen et al., 2010) and JJSD (Poole et al., 2019) have large variances, especially in the Cubic task. Song & Ermon (2019) pointed out the limitations of JCPC, JNWJ, and JJSD, and developed the SMILE method, which clips the value of the estimated density function to reduce the variance of the estimators. DoE (McAllester & Stratos, 2020) is neither a lower bound nor a upper bound of mutual information, but can achieve accurate estimates when underlying mutual information is large. JRPC exhibits comparable bias and lower variance compared to the SMILE method, and is more stable than the DoE method. We would like to highlight our method’s low-variance property, where we neither clip the values of the estimated density ratio nor impose an upper bound of our estimated mutual information.

heading: 4 RELATED WORK
text: As a subset of unsupervised representation learning, self-supervised representation learning (SSL) adopts self-defined signals as supervision and uses the learned representation for downstream tasks, such as object detection and image captioning (Liu et al., 2020). We categorize SSL work into two groups: when the signal is the input’s hidden property or the corresponding view of the input. For the first group, for example, Jigsaw puzzle (Noroozi & Favaro, 2016) shuffles the image patches and defines the SSL task for predicting the shuffled positions of the image patches. Other instances are Predicting Rotations (Gidaris et al., 2018) and Shuffle & Learn (Misra et al., 2016). For the second group, the SSL task aims at modeling the co-occurrence of multiple views of data, via the contrastive or the predictive learning objectives (Tsai et al., 2020a). The predictive objectives encourage reconstruction from one view of the data to the other, such as predicting the lower part of an image from
its upper part (ImageGPT by Chen et al. (2020a)). Comparing the contrastive with predictive learning approaches, Tsai et al. (2020a) points out that the former requires less computational resources for a good performance but suffers more from the over-fitting problem.
Theoretical analysis (Arora et al., 2019; Tsai et al., 2020a; Tosh et al., 2020) suggests the contrastively learned representations can lead to a good downstream performance. Beyond the theory, Tian et al. (2020) shows what matters more for the performance are 1) the choice of the contrastive learning objective; and 2) the creation of the positive and negative data pairs in the contrastive objective. Recent work (Khosla et al., 2020) extends the usage of contrastive learning from the selfsupervised setting to the supervised setting. The supervised setting defines the positive pairs as the data from the same class in the contrastive objective, while the self-supervised setting defines the positive pairs as the data with different augmentations.
Our work also closely rates to the skewed divergence measurement between distributions (Lee, 1999; 2001; Nielsen, 2010; Yamada et al., 2013). Recall that the usage of the relative parameters plays a crucial role to regularize our objective for its boundness and low variance. This idea is similar to the skewed divergence measurement, that when calculating the divergence between distributions P and Q, instead of considering D(P ‖Q), these approaches consider D(P ‖αP + (1 − α)Q) with D representing the divergence and 0 < α < 1. A natural example is that the Jensen-Shannon divergence is a symmetric skewed KL divergence: DJS(P ‖Q) = 0.5DKL(P ‖ 0.5P + 0.5Q) + 0.5DKL(Q ‖ 0.5P + 0.5Q). Compared to the non-skewed counterpart, the skewed divergence has shown to have a more robust estimation for its value (Lee, 1999; 2001; Yamada et al., 2013). Different from these works that focus on estimating the values of distribution divergence, we focus on learning self-supervised representations.

heading: 5 CONCLUSION
text: In this work, we present RPC, the Relative Predictive Coding, that achieves a good balance among the three challenges when modeling a contrastive learning objective: training stability, sensitivity to minibatch size, and downstream task performance. We believe this work brings an appealing option for training self-supervised models and inspires future work to design objectives for balancing the aforementioned three challenges. In the future, we are interested in applying RPC in other application domains and developing more principled approaches for better representation learning.

heading: ACKNOWLEDGEMENT
text: This work was supported in part by the NSF IIS1763562, NSF Awards #1750439 #1722822, National Institutes of Health, IARPA D17PC00340, ONR Grant N000141812861, and Facebook PhD Fellowship. We would also like to acknowledge NVIDIA’s GPU support and Cloud TPU support from Google’s TensorFlow Research Cloud (TFRC).

heading: A APPENDIX

heading: A.1 PROOF OF LEMMA 1 IN THE MAIN TEXT
text: Lemma 2 (Optimal Solution for JRPC, restating Lemma 1 in the main text) Let
JRPC(X,Y ) := sup f∈F
EPXY [f(x, y)]−αEPXPY [f(x, y)]− β
2 EPXY
[ f2(x, y) ] −γ
2 EPXPY
[ f2(x, y) ] and r(x, y) = p(x,y)p(x)p(y) be the density ratio. JRPC has the optimal solution
f∗(x, y) = r(x, y)− α β r(x, y) + γ := rα,β,γ(x, y) with − α γ ≤ rα,β,γ ≤ 1 β .
Proof: The second-order functional derivative of the objective is
−βdPX,Y − γdPXPY ,
which is always negative. The negative second-order functional derivative implies the objective has a supreme value. Then, take the first-order functional derivative ∂JRPC∂m and set it to zero:
dPX,Y − α · dPXPY − β · f(x, y) · dPX,Y − γ · f(x, y) · dPXPY = 0.
We then get
f∗(x, y) = dPX,Y − α · dPXPY β · dPX,Y + γ · dPXPY = p(x, y)− αp(x)p(y) βp(x, y) + γp(x)p(y) = r(x, y)− α βr(x, y) + γ .
Since 0 ≤ r(x, y) ≤ ∞, we have −αγ ≤ r(x,y)−α βr(x,y)+γ ≤ 1 β . Hence,
∀β 6= 0, γ 6= 0, f∗(x, y) := rα,β,γ(x, y) with − α
γ ≤ rα,β,γ ≤
1 β .
A.2 RELATION BETWEEN JRPC AND Dχ2
In this subsection, we aim to show the following: 1) Dχ2(PXY ‖PXPY ) = EPXPY [r2(x, y)] − 1; and 2) JRPC(X,Y ) = β+γ2 EP ′ [r 2 α,β,γ(x, y)] by having P ′ = ββ+γPXY + γ
β+γPXPY as the mixture distribution of PXY and PXPY .
Lemma 3 Dχ2(PXY ‖PXPY ) = EPXPY [r2(x, y)]− 1
Proof: By definition (Nielsen & Nock, 2013),
Dχ2(PXY ‖PXPY ) = ∫ (dPXY )2
dPXPY − 1 = ∫ ( dPXY dPXPY )2 dPXPY − 1
= ∫ ( p(x, y) p(x)p(y) )2 dPXPY − 1 = ∫ r2(x, y)dPXPY − 1
= EPXPY [r2(x, y)]− 1.
Lemma 4 Defining P ′ = ββ+γPXY + γ β+γPXPY as a mixture distribution of PXY and PXPY , JRPC(X,Y ) = β+γ 2 EP ′ [r 2 α,β,γ(x, y)].
Proof: Plug in the optimal solution f∗(x, y) = dPX,Y −α·dPXPYβ·dPX,Y +γ·dPXPY (see Lemma 2) into JRPC:
JRPC = EPXY [f∗(x, y)]− αEPXPY [f∗(x, y)]− β
2 EPXY
[ f∗2(x, y) ] − γ
2 EPXPY
[ f∗2(x, y) ] = ∫ f∗(x, y) · ( dPXY − α · dPXPY ) − 1
2 f∗2(x, y) ·
( β · dPXY + γ · dPXPY ) =
∫ dPX,Y − α · dPXPY β · dPX,Y + γ · dPXPY ( dPXY − α · dPXPY ) − 1 2 ( dPX,Y − α · dPXPY β · dPX,Y + γ · dPXPY )2( β · dPXY + γ · dPXPY ) = 1
2 ∫ ( dPX,Y − α · dPXPY β · dPX,Y + γ · dPXPY )2( β · dPXY + γ · dPXPY ) = β + γ
2 ∫ ( dPX,Y − α · dPXPY β · dPX,Y + γ · dPXPY )2( β β + γ · dPXY + γ β + γ · dPXPY ) .
Since we define rα,β,γ = dPX,Y −α·dPXPY β·dPX,Y +γ·dPXPY and P ′ = ββ+γPXY + γ β+γPXPY ,
JRPC = β + γ
2 EP ′ [r2α,β,γ(x, y)].

heading: A.3 PROOF OF PROPOSITION 1 IN THE MAIN TEXT
text: The proof contains two parts: showing 0 ≤ JRPC ≤ 12β + α2 2γ (see Section A.3.1) and Ĵ m,n RPC is a consistent estimator for JRPC (see Section A.3.2).
A.3.1 BOUNDNESS OF JRPC
Lemma 5 (Boundness of JRPC) 0 ≤ JRPC ≤ 12β + α2 2γ
Proof: Lemma 4 suggests JRPC(X,Y ) = β+γ2 EP ′ [r 2 α,β,γ(x, y)] with P ′ = ββ+γPXY + γ
β+γPXPY as the mixture distribution of PXY and PXPY . Hence, it is obvious JRPC(X,Y ) ≥ 0. We leverage the intermediate results in the proof of Lemma 4:
JRPC(X,Y ) = 1
2 ∫ ( dPX,Y − α · dPXPY β · dPX,Y + γ · dPXPY )2( β · dPXY + γ · dPXPY ) = 1
2
∫ dPX,Y ( dPX,Y − α · dPXPY β · dPX,Y + γ · dPXPY ) − α 2 ∫ dPXPY ( dPX,Y − α · dPXPY β · dPX,Y + γ · dPXPY ) = 1
2 EPXY [rα,β,γ(x, y)]−
α 2 EPXPY [rα,β,γ(x, y)].
Since −αγ ≤ rα,β,γ ≤ 1 β , JRPC(X,Y ) ≤ 1 2β +
α2 2γ .

heading: A.3.2 CONSISTENCY
text: We first recall the definition of the estimation of JRPC:
Definition 2 (Ĵm,nRPC, empirical estimation of JRPC, restating Definition 1 in the main text) We parametrize f via a family of neural networks FΘ := {fθ : θ ∈ Θ ⊆ Rd} where d ∈ N and Θ is compact. Let {xi, yi}ni=1 be n samples drawn uniformly at random from PXY and {x′j , y′j}mj=1 be m samples drawn uniformly at random from PXPY . Then,
Ĵm,nRPC = sup fθ∈FΘ
1
n n∑ i=1 fθ(xi, yi)− 1 m m∑ j=1 αfθ(x ′ j , y ′ j)− 1 n n∑ i=1 β 2 f2θ (xi, yi)− 1 m m∑ j=1 γ 2 f2θ (x ′ j , y ′ j).
Our goal is to show that Ĵm,nRPC is a consistent estimator for JRPC. We begin with the following definition:
Ĵm,nRPC,θ := 1
n n∑ i=1 fθ(xi, yi)− 1 m m∑ j=1 αfθ(x ′ j , y ′ j)− 1 n n∑ i=1 β 2 f2θ (xi, yi)− 1 m m∑ j=1 γ 2 f2θ (x ′ j , y ′ j) (3)
and E [ ĴRPC,θ ] := EPXY [fθ(x, y)]−αEPXPY [fθ(x, y)]− β
2 EPXY [f2θ (x, y)]−
γ 2 EPXPY [f2θ (x, y)]. (4)
Then, we follow the steps:
• The first part is about estimation. We show that, with high probability, Ĵm,nRPC,θ is close to E [ ĴRPC,θ ] , for any given θ.
• The second part is about approximation. We will apply the universal approximation lemma of neural networks (Hornik et al., 1989) to show that there exists a network θ∗ such that E [ ĴRPC,θ∗ ] is close to JRPC.
Part I - Estimation: With high probability, Ĵm,nRPC,θ is close to E [ ĴRPC,θ ] , for any given θ. Throughout the analysis on the uniform convergence, we need the assumptions on the boundness and smoothness of the function fθ. Since we show the optimal function f is bounded in JRPC, we can use the same bounded values for fθ without losing too much precision. The smoothness of the function suggests that the output of the network should only change slightly when only slightly perturbing the parameters. Specifically, the two assumptions are as follows:
Assumption 1 (boundness of fθ) There exist universal constants such that ∀fθ ∈ FΘ, CL ≤ fθ ≤ CU . For notations simplicity, we let M = CU − CL be the range of fθ and U = max {|CU |, |CL|} be the maximal absolute value of fθ. In the paper, we can choose to constrain that CL = −αγ and CU = 1 β since the optimal function f ∗ has −αγ ≤ f ∗ ≤ 1β .
Assumption 2 (smoothness of fθ) There exists constant ρ > 0 such that ∀(x, y) ∈ (X × Y) and θ1, θ2 ∈ Θ, |fθ1(x, y)− fθ2(x, y)| ≤ ρ|θ1 − θ2|.
Now, we can bound the rate of uniform convergence of a function class in terms of covering number (Bartlett, 1998):
Lemma 6 (Estimation) Let > 0 and N (Θ, ) be the covering number of Θ with radius . Then,
Pr ( sup fθ∈FΘ ∣∣∣Ĵm,nRPC,θ − E[ĴRPC,θ]∣∣∣ ≥ )
≤2N (Θ, 4ρ ( 1 + α+ 2(β + γ)U ) )(exp(− n 2 32M2 ) + exp ( − m 2 32M2α2 ) + exp ( − n 2 32U2β2 ) + exp ( − m 2 32U2γ2 )) .
Proof: For notation simplicity, we define the operators • P (f) = EPXY [f(x, y)] and Pn(f) = 1n ∑n i=1 f(xi, yi)
• Q(f) = EPXPY [f(x, y)] and Qm(f) = 1m ∑m j=1 f(x ′ j , y ′ j)
Hence,∣∣∣Ĵm,nRPC,θ − E[ĴRPC,θ]∣∣∣ = ∣∣Pn(fθ)− P (fθ)− αQm(fθ) + αQ(fθ)− βPn(f2θ ) + βP (f2θ )− γQm(f2θ ) + γQ(f2θ )∣∣ ≤ |Pn(fθ)− P (fθ)|+ α |Qm(fθ)−Q(fθ)|+ β ∣∣Pn(f2θ )− P (f2θ )∣∣+ γ ∣∣Qm(f2θ )−Q(f2θ )∣∣
Let ′ = 4ρ ( 1+α+2(β+γ)U ) and T := N (Θ, ′). LetC = {fθ1 , fθ2 , · · · , fθT }with {θ1, θ2, · · · , θT } be such that B∞(θ1, ′), · · · , B∞(θT , ′) are ′ cover. Hence, for any fθ ∈ FΘ, there is an fθk ∈ C such that ‖θ − θk‖∞ ≤ ′. Then, for any fθk ∈ C:∣∣∣Ĵm,nRPC,θ − E[ĴRPC,θ]∣∣∣ ≤ |Pn(fθ)− P (fθ)|+ α |Qm(fθ)−Q(fθ)|+ β
∣∣Pn(f2θ )− P (f2θ )∣∣+ γ ∣∣Qm(f2θ )−Q(f2θ )∣∣ ≤ |Pn(fθk)− P (fθk)|+ |Pn(fθ)− Pn(fθk)|+ |P (fθ)− P (fθk)|
+ α ( |Qm(fθk)−Q(fθk)|+ |Qm(fθ)−Qm(fθk)|+ |Q(fθ)−Q(fθk)| ) + β
( ∣∣Pn(f2θk)− P (f2θk)∣∣+ ∣∣Pn(f2θ )− Pn(f2θk)∣∣+ ∣∣P (f2θ )− P (f2θk)∣∣ ) + γ
( ∣∣Qm(f2θk)−Q(f2θk)∣∣+ ∣∣Qm(f2θ )−Qm(f2θk)∣∣+ ∣∣Q(f2θ )−Q(f2θk)∣∣ ) ≤ |Pn(fθk)− P (fθk)|+ ρ‖θ − θk‖+ ρ‖θ − θk‖
+ α ( |Qm(fθk)−Q(fθk)|+ ρ‖θ − θk‖+ ρ‖θ − θk‖ ) + β
( ∣∣Pn(f2θk)− P (f2θk)∣∣+ 2ρU‖θ − θk‖+ 2ρU‖θ − θk‖) + γ
( ∣∣Qm(f2θk)−Q(f2θk)∣∣+ 2ρU‖θ − θk‖+ 2ρU‖θ − θk‖) = |Pn(fθk)− P (fθk)|+ α |Qm(fθk)−Q(fθk)|+ β
∣∣Pn(f2θk)− P (f2θk)∣∣+ γ ∣∣Qm(f2θk)−Q(f2θk)∣∣ + 2ρ ( 1 + α+ 2(β + γ)U ) ‖θ − θk‖
≤ |Pn(fθk)− P (fθk)|+ α |Qm(fθk)−Q(fθk)|+ β ∣∣Pn(f2θk)− P (f2θk)∣∣+ γ ∣∣Qm(f2θk)−Q(f2θk)∣∣+ 2 ,
where
• |Pn(fθ)− Pn(fθk)| ≤ ρ‖θ − θk‖ due to Assumption 2, and the result also applies for |P (fθ)− P (fθk)|, |Qm(fθ)−Qm(fθk)|, and |Q(fθ)−Q(fθk)|.
• ∣∣Pn(f2θ )− Pn(f2θk)∣∣ ≤ 2‖fθ‖∞ρ‖θ−θk‖ ≤ 2ρU‖θ−θk‖ due to Assumptions 1 and 2. The result also applies for
∣∣P (f2θ )− P (f2θk)∣∣, ∣∣Qm(f2θ )−Qm(f2θk)∣∣, and ∣∣Q(f2θ )−Q(f2θk)∣∣. Hence,
Pr ( sup fθ∈FΘ ∣∣∣Ĵm,nRPC,θ − E[ĴRPC,θ]∣∣∣ ≥ )
≤Pr (
max fθk∈C |Pn(fθk)− P (fθk)|+ α |Qm(fθk)−Q(fθk)|+ β ∣∣Pn(f2θk)− P (f2θk)∣∣+ γ ∣∣Qm(f2θk)−Q(f2θk)∣∣+ 2 ≥ ) = Pr
( max fθk∈C |Pn(fθk)− P (fθk)|+ α |Qm(fθk)−Q(fθk)|+ β ∣∣Pn(f2θk)− P (f2θk)∣∣+ γ ∣∣Qm(f2θk)−Q(f2θk)∣∣ ≥ 2 )
≤ T∑ k=1 Pr ( |Pn(fθk)− P (fθk)|+ α |Qm(fθk)−Q(fθk)|+ β ∣∣Pn(f2θk)− P (f2θk)∣∣+ γ ∣∣Qm(f2θk)−Q(f2θk)∣∣ ≥ 2) ≤
T∑ k=1 Pr ( |Pn(fθk)− P (fθk)| ≥ 8 ) + Pr ( α |Qm(fθk)−Q(fθk)| ≥ 8 ) + Pr ( β ∣∣Pn(f2θk)− P (f2θk)∣∣ ≥ 8)+ Pr(γ ∣∣Qm(f2θk)−Q(f2θk)∣∣ ≥ 8) .
With Hoeffding’s inequality,
• Pr ( |Pn(fθk)− P (fθk)| ≥ 8 ) ≤ 2exp ( − n 2 32M2 ) • Pr ( α |Qm(fθk)−Q(fθk)| ≥ 8 ) ≤ 2exp ( − m 2 32M2α2
) • Pr ( β ∣∣Pn(f2θk)− P (f2θk)∣∣ ≥ 8) ≤ 2exp(− n 232U2β2)
• Pr ( γ ∣∣Qm(f2θk)−Q(f2θk)∣∣ ≥ 8) ≤ 2exp(− m 232U2γ2)
To conclude,
Pr ( sup fθ∈FΘ ∣∣∣Ĵm,nRPC,θ − E[ĴRPC,θ]∣∣∣ ≥ )
≤2N (Θ, 4ρ ( 1 + α+ 2(β + γ)U ) )(exp(− n 2 32M2 ) + exp ( − m 2 32M2α2 ) + exp ( − n 2 32U2β2 ) + exp ( − m 2 32U2γ2 )) .
Part II - Approximation: Neural Network Universal Approximation. We leverage the universal function approximation lemma of neural network
Lemma 7 (Approximation (Hornik et al., 1989)) Let > 0. There exists d ∈ N and a family of neural networks FΘ := {fθ : θ ∈ Θ ⊆ Rd} where Θ is compact, such that
inf fθ∈FΘ ∣∣∣E[ĴRPC,θ]− JRPC∣∣∣ ≤ . Part III - Bringing everything together. Now, we are ready to bring the estimation and approximation together to show that there exists a neural network θ∗ such that, with high probability, Ĵm,nRPC,θ can approximate JRPC with n′ = min {n,m} at a rate of O(1/ √ n′):
Proposition 3 With probability at least 1 − δ, ∃θ∗ ∈ Θ, |JRPC − Ĵm,nRPC,θ| = O( √ d+log (1/δ) n′ ), where n′ = min {n,m}.
Proof: The proof follows by combining Lemma 6 and 7.
First, Lemma 7 suggests, ∃θ∗ ∈ Θ,∣∣∣E[ĴRPC,θ∗]− JRPC∣∣∣ ≤ 2 .
Next, we perform analysis on the estimation error, aiming to find n,m and the corresponding probability, such that ∣∣∣Ĵm,nRPC,θ − E[ĴRPC,θ∗]∣∣∣ ≤ 2 . Applying Lemma 6 with the covering number of the neural network: ( N (Θ, ) =
O ( exp ( d log (1/ ) )) (Anthony & Bartlett, 2009) ) and let n′ = min{n,m}:
Pr ( sup fθ∈FΘ ∣∣∣Ĵm,nRPC,θ − E[ĴRPC,θ]∣∣∣ ≥ 2 )
≤2N (Θ, 8ρ ( 1 + α+ 2(β + γ)U ) )(exp(− n 2 128M2 ) + exp ( − m 2 128M2α2 ) + exp ( − n 2 128U2β2 ) + exp ( − m 2 128U2γ2 )) =O ( exp ( d log (1/ )− n′ 2 )) ,
where the big-O notation absorbs all the constants that do not require in the following derivation. Since we want to bound the probability with 1− δ, we solve the such that
exp ( d log (1/ )− n′ 2 ) ≤ δ.
With log (x) ≤ x− 1,
n′ 2 + d( − 1) ≥ n′ 2 + dlog ≥ log (1/δ),
where this inequality holds when
= O
(√ d+ log (1/δ)
n′
) .

heading: A.4 PROOF OF PROPOSITION 2 IN THE MAIN TEXT - FROM AN ASYMPTOTIC VIEWPOINT
text: Here, we provide the variance analysis on Ĵm,nRPC via an asymptotic viewpoint. First, assuming the network is correctly specified, and hence there exists a network parameter θ∗ satisfying f∗(x, y) = fθ∗(x, y) = rα,β,γ(x, y). Then we recall that Ĵ m,n RPC is a consistent estimator of J RPC (see Proposition 3), and under regular conditions, the estimated network parameter θ̂ in Ĵm,nRPC satisfying the asymptotic normality in the large sample limit (see Theorem 5.23 in (Van der Vaart, 2000)). We recall the definition of Ĵm,nRPC,θ in equation 3 and let n
′ = min{n,m}, the asymptotic expansion of Ĵm,nRPC has
Ĵm,nRPC,θ∗ = Ĵ m,n RPC,θ̂ + ˙̂ Jm,n RPC,θ̂ (θ∗ − θ̂) + o(‖θ∗ − θ̂‖)
= Ĵm,n RPC,θ̂ + ˙̂ Jm,n RPC,θ̂ (θ∗ − θ̂) + op( 1√ n′ )
= Ĵm,n RPC,θ̂ + op( 1√ n′ ),
(5)
where ˙̂Jm,n RPC,θ̂ = 0 since θ̂ is the estimation from Ĵm,nRPC = sup fθ∈FΘ Ĵm,nRPC,θ.
Next, we recall the definition in equation 4:
E[ĴRPC,θ̂] = EPXY [fθ̂(x, y)]− αEPXPY [fθ̂(x, y)]− β
2 EPXY [f2θ̂ (x, y)]−
γ 2 EPXPY [f2θ̂ (x, y)].
Likewise, the asymptotic expansion of E[ĴRPC,θ] has
E[ĴRPC,θ̂] = E[ĴRPC,θ∗ ] + E[ ˙̂ JRPC,θ∗ ](θ̂ − θ∗) + o(‖θ̂ − θ∗‖)
= E[ĴRPC,θ∗ ] + E[ ˙̂JRPC,θ∗ ](θ̂ − θ∗) + op( 1√ n′ )
= E[ĴRPC,θ∗ ] + op( 1√ n′ ),
(6)
where E[ ˙̂JRPC,θ∗ ] = 0 since E[ĴRPC,θ∗ ] = JRPC and θ∗ satisfying f∗(x, y) = fθ∗(x, y).
Combining equations 5 and 6:
Ĵm,n RPC,θ̂ − E[ĴRPC,θ̂] =Ĵ m,n RPC,θ∗ − JRPC + op( 1√ n′ )
= 1
n n∑ i=1 f∗θ (xi, yi)− α 1 m m∑ j=1 f∗θ (x ′ j , y ′ j)− β 2 1 n n∑ i=1 f2θ∗(xi, yi)− γ 2 1 m m∑ j=1 f2θ∗(x ′ j , y ′ j)
− EPXY [f∗(x, y)] + αEPXPY [f∗(x, y)] + β
2 EPXY
[ f∗2(x, y) ] + γ
2 EPXPY
[ f∗2(x, y) ] + op(
1√ n′ )
= 1
n n∑ i=1 rα,β,γ(xi, yi)− α 1 m m∑ j=1 rα,β,γ(x ′ j , y ′ j)− β 2 1 n n∑ i=1 r2α,β,γ(xi, yi)− γ 2 1 m m∑ j=1 r2α,β,γ(x ′ j , y ′ j)
− EPXY [rα,β,γ(x, y)] + αEPXPY [rα,β,γ(x, y)] + β
2 EPXY
[ r2α,β,γ(x, y) ] + γ
2 EPXPY
[ r2α,β,γ(x, y) ] + op(
1√ n′ )
= 1√ n · 1√ n n∑ i=1
( rα,β,γ(xi, yi)− β
2 r2α,β,γ(xi, yi)− EPXY
[ rα,β,γ(x, y)− β
2 r2α,β,γ(x, y)
])
− 1√ m · 1√ m m∑ j=1
( αrα,β,γ(x ′ j , y ′ j) + γ
2 r2α,β,γ(x ′ j , y ′ j)− EPXPY
[ αrα,β,γ(x, y) + γ
2 r2α,β,γ(x, y)
])
+ op( 1√ n′ ).
Therefore, the asymptotic Variance of Ĵm,nRPC is
Var[Ĵm,nRPC] = 1
n VarPXY [rα,β,γ(x, y)−
β 2 r2α,β,γ(x, y)] + 1 m VarPXPY [αrα,β,γ(x, y) + γ 2 r2α,β,γ(x, y)] + o( 1 n′ ).
First, we look at VarPXY [rα,β,γ(x, y)− β 2 r 2 α,β,γ(x, y)]. Since β > 0 and−αγ ≤ rα,β,γ ≤ 1 β , simple calculation gives us − 2αγ+βα 2
2γ2 ≤ rα,β,γ(x, y)− β 2 r 2 α,β,γ(x, y) ≤ 12β . Hence,
VarPXY [rα,β,γ(x, y)− β
2 r2α,β,γ(x, y)] ≤ max {(2αγ + βα2 2γ2 )2 , ( 1 2β )2} .
Next, we look at VarPXPY [αrα,β,γ(x, y) + γ 2 r 2 α,β,γ(x, y)]. Since α ≥ 0, γ > 0 and−αγ ≤ rα,β,γ ≤ 1 β , simple calculation gives us − α2 2γ ≤ αrα,β,γ(x, y) + γ 2 r 2 α,β,γ(x, y) ≤ 2αβ+γ 2β2 . Hence,
VarPXPY [αrα,β,γ(x, y) + γ
2 r2α,β,γ(x, y)] ≤ max {(α2 2γ )2 , (2αβ + γ 2β2 )2} .
Combining everything together, we restate the Proposition 2 in the main text:
Proposition 4 (Asymptotic Variance of Ĵm,nRPC)
Var[Ĵm,nRPC] = 1
n VarPXY [rα,β,γ(x, y)−
β 2 r2α,β,γ(x, y)] + 1 m VarPXPY [αrα,β,γ(x, y) + γ 2 r2α,β,γ(x, y)] + o( 1 n′ )
≤ 1 n max {(2αγ + βα2 2γ2 )2 , ( 1 2β )2} + 1 m max {(α2 2γ )2 , (2αβ + γ 2β2 )2} + o( 1 n′ )
A.5 PROOF OF PROPOSITION 2 IN THE MAIN TEXT - FROM BOUNDNESS OF fθ
As discussed in Assumption 1, for the estimation Ĵm,nRPC, we can bound the function fθ in FΘ within [−αγ , 1 β ] without losing precision. Then, re-arranging Ĵ m,n RPC:
sup fθ∈FΘ
1
n n∑ i=1 fθ(xi, yi)− 1 m m∑ j=1 αfθ(x ′ j , y ′ j)− 1 n n∑ i=1 β 2 f2θ (xi, yi)− 1 m m∑ j=1 γ 2 f2θ (x ′ j , y ′ j)
sup fθ∈FΘ
1
n n∑ i=1 ( fθ(xi, yi)− β 2 f2θ (xi, yi) ) + 1 m n∑ j=m ( αfθ(x ′ j , y ′ j) + γ 2 f2θ (x ′ j , y ′ j) )
Then, since −αγ ≤ fθ(·, ·) ≤ 1 β , basic calculations give us
−2αγ + βα 2
2γ2 ≤ fθ(xi, yi)−
β 2 f2θ (xi, yi) ≤ 1 2β and −α
2
2γ ≤ αfθ(x′j , y′j)+
γ 2 f2θ (x ′ j , y ′ j) ≤ 2αβ + γ 2β2 .
The resulting variances have
Var[fθ(xi, yi)− β
2 f2θ (xi, yi)] ≤ max {(2αγ + βα2 2γ2 )2 , ( 1 2β )2} and
Var[αfθ(x ′ j , y ′ j) +
γ 2 f2θ (x ′ j , y ′ j)] ≤ max {(α2 2γ )2 , (2αβ + γ 2β2 )2} .
Taking the mean of m,n independent random variables gives the result:
Proposition 5 (Variance of Ĵm,nRPC)
Var[Ĵm,nRPC] ≤ 1
n max {(2αγ + βα2 2γ2 )2 , ( 1 2β )2} + 1 m max {(α2 2γ )2 , (2αβ + γ 2β2 )2} .
A.6 IMPLEMENTATION OF EXPERIMENTS
For visual representation learning, we follow the implementation in https://github.com/ google-research/simclr. For speech representation learning, we follow the implementation in https://github.com/facebookresearch/CPC_audio. For MI estimation, we follow the implementation in https://github.com/yaohungt/Pointwise_ Dependency_Neural_Estimation/tree/master/MI_Est_and_CrossModal..

heading: A.7 RELATIVE PREDICTIVE CODING ON VISION
text: The whole pipeline of pretraining contains the following steps: First, a stochastic data augmentation will transform one image sample xk to two different but correlated augmented views, x′2k−1 and x′2k. Then a base encoder f(·) implemented using ResNet (He et al., 2016) will extract representations from augmented views, creating representations h2k−1 and h2k. Later a small neural network g(·) called projection head will map h2k−1 and h2k to z2k−1 and z2k in a different latent space. For each minibatch of N samples, there will be 2N views generated. For each image xk there will be one positive pair x′2k−1 and x ′ 2k and 2(N − 1) negative samples. The RPC loss between a pair of positive views, x′i and x ′ j (augmented from the same image) , can be calculated by the substitution fθ(x ′ i,x ′ j) = (zi · zj)/τ = si,j (τ is a hyperparameter) to the definition of RPC:
`RPCi,j = −(si,j − α
2(N − 1) 2N∑ k=1 1[k 6=i]si,k − β 2 s2i,j −
γ
2 · 2(N − 1) 2N∑ k=1 1[k6=i]s 2 i,k) (7)
For losses other than RPC, a hidden normalization of si,j is often required by replacing zi · zj with (zi ·zj)/|zi||zj |. CPC and WPC adopt this, while other objectives needs it to help stabilize training variance. RPC does not need this normalization.

heading: A.8 CIFAR-10/-100 AND IMAGENET EXPERIMENTS DETAILS
text: ImageNet Following the settings in (Chen et al., 2020b;c), we train the model on Cloud TPU with 128 cores, with a batch size of 4, 096 and global batch normalization 3 (Ioffe & Szegedy, 2015). Here we refer to the term batch size as the number of images (or utterances in the speech experiments) we use per GPU, while the term minibatch size refers to the number of negative samples used to calculate the objective, such as CPC or our proposed RPC. The largest model we train is a 152-layer ResNet with selective kernels (SK) (Li et al., 2019) and 2× wider channels. We use the LARS optimizer (You et al., 2017) with momentum 0.9. The learning rate linearly increases for the first 20 epochs, reaching a maximum of 6.4, then decayed with cosine decay schedule. The weight decay is 10−4. A MLP projection head g(·) with three layers is used on top of the ResNet encoder. Unlike Chen et al. (2020c), we do not use a memory buffer, and train the model for only 100 epochs rather than 800 epochs due to computational constraints. These two options slightly reduce CPC’s performance benchmark for about 2% with the exact same setting. The unsupervised pre-training is followed by a supervised fine-tuning. Following SimCLRv2 (Chen et al., 2020b;c), we fine-tune the 3-layer g(·) for the downstream tasks. We use learning rates 0.16 and 0.064 for standard 50-layer ResNet and larger 152-layer ResNet respectively, and weight decay and learning rate warmup are removed. Different from Chen et al. (2020c), we use a batch size of 4, 096, and we do not use global batch normalization for fine-tuning. For JRPC we disable hidden normalization and use a temperature τ = 32. For all other objectives, we use hidden normalization and τ = 0.1 following previous work (Chen et al., 2020c). For relative parameters, we use α = 0.3, β = 0.001, γ = 0.1 and α = 0.3, β = 0.001, γ = 0.005 for ResNet-50 and ResNet-152 respectively.
CIFAR-10/-100 Following the settings in (Chen et al., 2020b), we train the model on a single GPU, with a batch size of 512 and global batch normalization (Ioffe & Szegedy, 2015). We use ResNet (He et al., 2016) of depth 18 and depth 50, and does not use Selective Kernel (Li et al., 2019) or a multiplied width size. We use the LARS optimizer (You et al., 2017) with momentum 0.9. The learning rate linearly increases for the first 20 epochs, reaching a maximum of 6.4, then decayed with cosine decay schedule. The weight decay is 10−4. A MLP projection head g(·) with three layers is used on top of the ResNet encoder. Unlike Chen et al. (2020c), we do not use a memory buffer. We train the model for 1000 epochs. The unsupervised pre-training is followed by a supervised fine-tuning. Following SimCLRv2 (Chen et al., 2020b;c), we fine-tune the 3-layer g(·) for the downstream tasks. We use learning rates 0.16 for standard 50-layer ResNet , and weight decay and learning rate warmup are removed. For JRPC we disable hidden normalization and use a temperature τ = 128. For all other objectives, we use hidden normalization and τ = 0.5 following previous work (Chen et al., 2020c). For relative parameters, we use α = 1.0, β = 0.005, and γ = 1.0.
STL-10 We also perform the pre-training and fine-tuning on STL-10 (Coates et al., 2011) using the model proposed in Chuang et al. (2020). Chuang et al. (2020) proposed to indirectly approximate the distribution of negative samples so that the objective is debiased. However, their implementation of contrastive learning is consistent with Chen et al. (2020b). We use a ResNet with depth 50 as an encoder for pre-training, with Adam optimizer, learning rate 0.001 and weight decay 10−6. The temperature τ is set to 0.5 for all objectives other than JRPC, which disables hidden normalization and use τ = 128. The downstream task performance increases from 83.4% of JCPC to 84.1% of JRPC.
Confidence Interval We also provide the confidence interval of JRPC and JCPC on CIFAR-10, CIFAR-100 and ImageNet, using ResNet-18, ResNet-18 and ResNet-50 respectively (95% confi-
3For WPC (Ozair et al., 2019), the global batch normalization during pretraining is disabled since we enforce 1-Lipschitz by gradient penalty (Gulrajani et al., 2017).
dence level is chosen) in Table 4. Both CPC and RPC use the same experimental settings throughout this paper. Here we use the relative parameters (α = 1.0, β = 0.005, γ = 1.0) in JRPC which gives the best performance on CIFAR-10. The confidence intervals of CPC do not overlap with the confidence intervals of RPC, which means the difference of the downstream task performance between RPC and CPC is statistically significant.

heading: A.9 RELATIVE PREDICTIVE CODING ON SPEECH
text: For speech representation learning, we adopt the general architecture from Oord et al. (2018). Given an input signal x1:T with T time steps, we first pass it through an encoder φθ parametrized by θ to produce a sequence of hidden representations {h1:T } where ht = φθ(xt). After that, we obtain the contextual representation ct at time step t with a sequential model ψρ parametrized by ρ: ct = ψρ(h1, . . . ,ht), where ct contains context information before time step t. For unsupervised pre-training, we use a multi-layer convolutional network as the encoder φθ, and an LSTM with hidden dimension 256 as the sequential model ψρ. Here, the contrastiveness is between the positive pair (ht+k, ct) where k is the number of time steps ahead, and the negative pairs (hi, ct), where hi is randomly sampled fromN , a batch of hidden representation of signals assumed to be unrelated to ct. The scoring function f based on Equation 2 at step t and look-ahead k will be fk = fk(h, ct) = exp((h)>Wkct), where Wk is a learnable linear transformation defined separately for each k ∈ {1, ...,K} and K is predetermined as 12 time steps. The loss in Equation 2 will then be formulated as:
`RPCt,k = −(fk(ht+k, ct)− α |N | ∑
hi∈N
fk(hi, ct)− β
2 f2k (ht+k, ct)−
γ 2|N | ∑
hi∈N
f2k (hi, ct)) (8)
We use the following relative parameters: α = 1, β = 0.25, and γ = 1, and we use the temperature τ = 16 for JRPC. For JCPC we follow the original implementation which sets τ = 1. We fix all other experimental setups, including architecture, learning rate, and optimizer. As shown in Table 3, JRPC has better downstream task performance, and is closer to the performance from a fully supervised model.

heading: A.10 EMPIRICAL OBSERVATIONS ON VARIANCE AND MINIBATCH SIZE
text: Variance Experiment Setup We perform the variance comparison of JDV, JNWJ and the proposed JRPC. The empirical experiments are performed using SimCLRv2 (Chen et al., 2020c) on CIFAR-10 dataset. We use a ResNet of depth 18, with batch size of 512. We train each objective with 30K training steps and record their value. In Figure 1, we use a temperature τ = 128 for all objectives. Unlike other experiments, where hidden normalization is applied to other objectives, we remove hidden normarlization for all objectives due to the reality that objectives after normalization does not reflect their original values. From Figure 1, JRPC enjoys lower variance and more stable training compared to JDV and JNWJ.
Minibatch Size Experimental Setup We perform experiments on the effect of batch size on downstream performances for different objective. The experiments are performed using SimCLRv2 (Chen et al., 2020c) on CIFAR-10 dataset, as well as the model from Rivière et al. (2020) on LibriSpeech-100h dataset (Panayotov et al., 2015). For vision task, we use the default temperature τ = 0.5 from Chen et al. (2020c) and hidden normalization mentioned in Section 3 for JCPC. For JRPC in vision and speech tasks we use a temperature of τ = 128 and τ = 16 respectively, both without hidden normalization.

heading: A.11 MUTUAL INFORMATION ESTIMATION
text: Our method is compared with baseline methods CPC (Oord et al., 2018), NWJ (Nguyen et al., 2010), JSD (Nowozin et al., 2016), and SMILE (Song & Ermon, 2019). All the approaches consider the same design of f(x, y), which is a 3-layer neural network taking concatenated (x, y) as the input. We also fix the learning rate, the optimizer, and the minibatch size across all the estimators for a fair comparison.
We present results of mutual information by Relative Predictive Coding using different sets of relative parameters in Figure 4. In the first row, we set β = 10−3, γ = 1, and experiment with different
α values. In the second row, we set α = 1, γ = 1 and in the last row we set α = 1, β = 10−3. From the figure, a small β around 10−3 and a large γ around 1.0 is crucial for an estimation that is relatively low bias and low variance. This conclusion is consistent with Section 3 in the main text.
We also performed comparison between JRPC and Difference of Entropies (DoE) (McAllester & Stratos, 2020). We performed two sets of experiments: in the first set of experiments we compare JRPC and DoE when MI is large (> 100 nats), while in the second set of experiments we compare JRPC and DoE using the setup in this section (MI < 12 nats and MI increases by 2 per 4k training steps). On the one hand, when MI is large (> 100 nats), we acknowledge that DoE is performing well on MI estimation, compared to JRPC which only estimates the MI around 20. This analysis is based on the code from https://github.com/karlstratos/doe. On the other hand, when the true MI is small, the DoE method is more unstable than JRPC, as shown in Figure 5. Figure 5 illustrates the results of the DoE method when the distribution is isotropic Gaussian (correctly specified) or Logistic (mis-specified). Figure 3 only shows the results using Gaussian.

Review summaries:
This paper presents a new contrastive representation objective that has good training stability, minibatch size sensitivity, and downstream task performance. This objective is a generalization of Chi-square divergence, the optimal solution is the density ratio of joint distribution and product of marginal distributions, the estimation is consistent and its variance goes to 0 as sample size goes to infinite, so the paper is theoretical sound. The authors conduct comprehensive experiments to show that the training based on this objective is stable, not sensitive to batch size and leads to good downstream task performance  in vision and phoneme and speaker classification. 

This paper proposes a new objective for self-supervised contrastive learning. In the general framework proposed by Tsai et al. (2020b), the proposed method boils down to using a divergence related to $\chi^2$-divergence. Compared to other objectives for contrastive learning, the authors illustrate the advantages of the proposed one in training stability (or easiness to train), sensitivity to batch size, and downstream task performance. However, introducing three new hyperparameters is a cause of concern since they make it more difficult to select optimal hyperparameters. Also, some important details of the experiments are missing. For example, how many runs to obtain the results shown in Tables 2 & 3? What's the confidence interval on the results? Any test to establish the statistical significance? What are the settings for supervised training? When the authors compare the results among different methods, did they select the optimal hyperparameters (e.g., learning rate) separately for each method?