Decomposing Mutual Information for Representation Learning
Authors: Alessandro Sordoni
Nouha Dziri
Hannes Schulz
Geoff Gordon
Remi Tachet des Combes
Philip Bachman
Venue:  (2021)
Keywords: Mutual Information
Self-supervised learning

Abstract: Many self-supervised representation learning methods maximize mutual information (MI) across views. In this paper, we transform each view into a set of subviews and then decompose the original MI bound into a sum of bounds involving conditional MI between the subviews. E.g.,~given two views $x$ and $y$ of the same input example, we can split $x$ into two subviews, $x^{\prime}$ and $x^{\prime\prime}$, which depend only on $x$ but are otherwise unconstrained. The following holds: $I(x; y) \geq I(x^{\prime\prime}; y) + I(x^{\prime}; y | x^{\prime\prime})$, due to the chain rule and information processing inequality. By maximizing both terms in the decomposition, our approach explicitly rewards the encoder for any information about $y$ which it extracts from $x^{\prime\prime}$, and for information about $y$ extracted from $x^{\prime}$ in excess of the information from $x^{\prime\prime}$. We provide a novel contrastive lower-bound on conditional MI, that relies on sampling contrast sets from $p(y|x^{\prime\prime})$. By decomposing the original MI into a sum of increasingly challenging MI bounds between sets of increasingly informed views, our representations can capture more of the total information shared between the original views. We empirically test the method in a vision domain and for dialogue generation.

Under review as a conference paper at ICLR 2021
DECOMPOSING MUTUAL INFORMATION FOR REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Many self-supe representation learning methods maximize mutual informa- tion (MI) across views. In this paper, we transform each view into a of subviews and then decompose the original MI bound into a sum of bounds involving condi- tional MI between the subviews. F.g. given two views x and y of the same input ex- ample, we can split z: into two subviews, x' and x", which depend only on but are otherwise unconstrained. The following holds: (xr;)) y) 2 [xx"; y) + [x² yla"), due to the chain rule and information processing inequality. By maximizing both terms in the decomposition, our approach explicitly rewards the encoder for any in- formation about y which it extracts from x" and for information about y extracted from x' in excess of the information from x". We provide a novel contrastive lower bound conditional MI, that relies sampling contrast sets from p(yl=" (y)=") By decomposing the original MI into a sum of increasingly challenging MI bounds between sets of increasingly informed views, our representations can capture more of the total information shared between the original views. We empirically test the method in vision domain and for dialogue generation.
INTRODUCTION
The ability to extract actionable information from data in the absence of explicit supervision seems to be a core prerequisite for building systems that can, for instance, learn from few data points or quickly make analogies and transfer to other tasks. Approaches to this problem include generative mod- els (Hinton) 2012 Kingma & Welling 2014) and self-supervised representation learning approaches, in which the objective is not to maximize likelihood, but to formulate a series of (label-agnostic) tasks that the model needs to solve through its representations (Noroozi & Favaro 2016 Devlin ct al., 2019: Gidaris et al. 2018 Hjelm et al. 2019). Self-supervised lcarning includes successful models leveraging contrastive learning, which have recently attained comparable performance to their fully-supervised counterparts (Bachman et al. 2019 Chen et al. 2020a).
Many self-supervised learning methods train an encoder such that the representations of pair of views x and y derived from the same input example are more similar to each other than to representations of views sampled from contrastive negative sample distribution, which is usually the marginal distribution of the data. For images, different views can be built using random flipping, color jittering and cropping (Bachman et al., 2019 Chen et al. (0220a). For sequential data such as conversational text, the views can be past and future utterances in a given dialogue. It can be shown that these methods maximize a lower bound on mutual information (MI) between the views, (a;; u). w.r.t. the encoder, i.e. the InfoNCE bound (Oord et al. 2018). One significant shortcoming of this approach is the large number of contrastive samples required, which directly impacts the total amount of information which the bound can measure (McAllester & Stratos 2018 Poole et al. 2019).
In this paper, we consider creating subviews of x by removing information from it in various ways, e.g. by masking some pixels. Then, we use representations from less informed subviews as source of hard contrastive samples for representations from more informed subviews. For example, Fig. [ one can mask pixel region in x' to obtain x" and ask (the representation of) x" to be closer to y than to random images the corpus, and for x' to be closer to than to samples from ply (y)
This corresponds to decomposing the MI between and y into I(x;y) 2 I(>": : y) [(x': y ()) The conditional MI measures the information about that the model has gained by looking at x' beyond the information already contained in x" In Fig. (left), standard contrastive approaches

---

Under review as a conference paper at ICLR 2021
<As Beausifll day, whar's your rame? <B> Jarry yours ?
y
what day!
X
<Bo Jarry yours ?
X
Figure I: A demonstration of our approach in vision (leff) and dialogue (right). (leff) Given two augmentations and y. we fork x into two subviews, x' which an exact copy of and x", an information--rsstriieed view obtained by occluding some of the pixels in x'. We can maximize [xx; v) 2 (z":y) + yll")) using a contrastive bound by training x" to be closer to y than to other images from the corpus, and by training x' to be closer to y than to samples from (x), ie. we can use x" to generate hard negative samples for x'. The conditional MI term encourages the encoder to imbue the representation of z' with information it shares with y beyond the information already in x". (right) z' and represent past and future in a dialogue respectively and x" is the "recent past". In this context, the encoder is encouraged to capture long-lerm dependencies that cannot be explained by the most recent utterances.
could focus on the overall "shape" of the object and would need many negative samples to capture other discriminative features. In our approach, the model is more directly encouraged to capture these additional features, e.g. the embossed detailing. In the context of predictive coding on sequential data such dialogue, by setting x" to be the most recent utterance (Fig. right), the encoder is directly encouraged to capture long-term dependencies that cannot be explained by x". We formally show that, by such decomposition, our representations can potentially capture more of the total information shared between the original views I and y.
Maximizing MI between multiple views can be related recent efforts in representation learning, amongst them AMDIM (Bachman et al. 2019). CMC (Tian et al.) 2019) and SWAV (Caron et al. 2020). However, these models maximize the sum of MIs between views I [(fx, x"):y) [[=: T E.g., in Bachman et al. 2019), x' and x" could be global and local representations of an image, and in Caron et al. 2020). x' and x" could be the views resulting from standard cropping and the aggressive multi-crop strategy. This equality is only valid when the views x' and x" are statistically independent, which usually does not hold. Instead, we argue that a better decomposition is / [{x',, xz" xz); v) I ( + (2'; ; ("), which always holds. Most importantly, the conditional MI term encourages the encoder to capture more non-redundant information across views.
To maximize our proposed decomposition,
present novel lower bound conditional MI in Section 3. For the conditional MI maximization, we give a computationally tractable approximation that adds minimal overhead. In Section 4, we first show in synthetic setting that decomposing MI and using the proposed conditional MI bound leads to capturing more of the ground-truth MI. Finally, we present evidence of the effectiveness of the method in vision and in dialogue generation.
PROBLEM SETTING
The maximum MI predictive coding framework (McAllester 2018 Oord et al. 2018 Hjelm et al. 2019) prescribes learning representations of input data such that they maximize MI. Estimating MI is generally a hard problem that has received a lot of attention in the community (Kraskov et al. 20044] Barber & Agakov) 2003). Let and y be two random variables which can generally describe input data from various domains, e.g. text, images or sound. We can learn representations of x and by maximizing the MI of the respective features produced by encoders f.g X -> RRd, which by the data processing inequality, bounded by (r;y)::
arg max ((f()))(()) $ I(x:v). f.9
(1)
We assume that the encoders be shared, i.e. f = g. The optimization in Eq. iis challenging but can be bower-bounded. Our starting point is the recently proposed InfoNCE lower bound on MI (Oord et al. 2018) and its application to self-supervised learning for visual representations (Bachman
2

---

Under review as a conference paper at ICLR 2021
et al. 2019
Chen al., 2020a). In this setting, and y are paired imput images, or independently- augmented copies of the same image. These are encoded using a neural network encoder which is trained such that the representations of the two image copies closer to each other in the embedding space than to other images drawn from the marginal distribution of the corpus. This can be viewed as contrastive estimation of the MI (Oord al. 2018). We present the InfoNCE bound next.
2.1 INFONCE BOUND
InfoNCE (Oord et al. 2018 is a lower-bound on [(x;)) y) obtained by comparing pairs sampled from the joint distribution æ, y1 p(z, to set of negative samples, y2:K p(12.)) III=2 also called contrastive, independently sampled from the marginal:
e E(x..))
()) = Epta, (ptzzk)) log K
S (x,
(2)
HE k= eb(a.v))
where E is critic assigning real valued score 2; pairs. We provide an exact derivation for this bound in the Appendix For this bound, the optimal critic is the log-odds between the conditional
distribution p(y(az) and the marginal distribution of y. Ex, y) log
plulz) + c(x) (Oord et al.
p(y)
2018 Poole et al. 2019). The InfoNCE bound is loose the true mutual information 7 (x; is larger than log K In order to overcome this difficulty, recent methods either train with large batch sizes (Chen et al. 2020a) or exploit an external memory of negative samples in order to reduce memory requirements (Chen et al. 2020b Tian al. 2020). These methods rely uniform sampling from the training set in order to form the contrastive sets. For further discussion of the limits of variational bounds of MI, see McAllester & Stratos (2018)
3 DECOMPOSING MUTUAL INFORMATION
By the data processing inequality: T(x;y)= 2 I([=1,...... NN } v). where {z1...., NNZ are different subviews i.e., views derived from without adding any exogenous information. For example, {zI...., IN can represent exchanges in longer dialog x, sentences in document or different augmentations of the same image æ. Equality is obtained when the set of subviews retains all information about z, c.g. if z is in the set.
EN
Without loss of generality, we consider the case N = 2, (x;z) 2 [((x', ((aa,,"":3 ;));), where {x',""} indicates two subviews derived from the original æ. We can apply the chain rule for MI:
(x; v) 2 I(tx', a"):)) ((=":)) + yll;")""),
where the equality obtained if and only [z; yl(",,
(=,,,"")", x²)) = 0. i.e x doesn't give any information about y in excess to {x', xx") } This suggests that we can maximize [(x; y) by maximizing each of the MI terms in the sum. The conditional MI term can be written as:
p(yla', x")
I(x':yl=" "yl"")
Exire .))
log
p(yl=")
This conditional MI is different from the unconditional MI, (x';v), insofar it measures the amount of information shared between x' and y which cannot be explained by x". Note that the decomposition holds for arbitrary partitions of x', x.. e.g. I(fx', v) [(z':v) + I(=": [(=":v/z').
When X' is high- sigh-dimensional, the amount of mutual information between and y will potentially be larger than the amount of MI that INC' can measure given computational constraints associated with large K and the poor log scaling properties of the bound. The idea that we put forward is to split the total MI into sum of MI terms of smaller magnitude, thus for which INCE would have less bias for any given K, and estimate each of those terms in turn. The resulting decomposed bound can be written into sum of unconditional and conditional MI terms:
INCES (x; y) INCE(="; v) + ICNCE(a' "//"" S (x;,),
(5)
1 'The derivation in Oord An alternative, exact derivation the bound can be found |Poole et al. (2019)
2018 presented an approximation and therefore was properly bound.
For proof of this fact, it suffices to consider T(( x', F:y) [ R xx ((<x"=""):""" given
that [( x', x²):v) v) [(x: v), equality is obtained iff [x;; (x
3

---

Under review as a conference paper at ICLR 2021
where ICNCE is lower-bound conditional MI and will be presented in the next section. Both conditional (Eq. ( and unconditional bounds on the MI (Eq. [4] can capture at most log K nats of MI. Therefore, the bound that arises from the decomposition of the MI in Eq. | potenniially allows to capture up to N log K' nats of MI in total, where N is the number of subviews used to describe This shows that measuring mutual information by decomposing it in sequence of estimation problems potentially allows to capture more nats of MI than with the standard INCE, which is bounded by log K.
CONTRASTIVE BOUNDS ON CONDITIONAL MUTUAL INFORMATION
One of the difficulties in computing the decomposed bound is measuring the conditional mutual information. In this section, we provide bounds and approximations of this quantity. First, we show that we can readily extend InfoNCE.
Proposition 1 (Conditional InfoNCE). The following is a lower-bound the conditional mutual information [(' ()(") and verifies the properties below:
e .1)
IeNca(e':gla" E,K) Eppat all a (n)
log A K - eB(a"
(6)
1. lONCE S (<<;vv/=")
2. E =argsup:: ICNCE log p(yl=",'") + c( x²)). p(ul=")
3. When K + oo and F. = F.,, we recover the true conditional MI: limg ICNCE(1' y|o", F K) T(xl:y)a") ula").
The proof can be found in Sec. A.2 and follows closely the derivation of the InfoNCE bound by applying a result from Barber Agakov (2003) and setting the proposal distribution of the variational approximation to [ly="") An alternative derivation of this bound
also presented in parallel in Foster et al. (2020) for optimal experiment design. Eq. 66 shows that a lower bound on the conditional MI can be obtained by sampling contrastive from the proposal distribution p(y] ==)) Indeed, since we want to estimate the MI conditioned on x", we should allow our contrastive distribution condition on z". Note that is now function of three variables.
Computing Eq. requuirs access to a large number of samples from p(ylæ"), which is unknown and usually challenging to obtain. In order to overcome this, we propose two solutions.
4.1 VARIATIONAL. APPROXIMATION
The next proposition shows that is possible to obtain a bound the conditional MI by approximating the unknown conditional distribution p(yla" with a variational distribution Tyyl=").
Proposition 2 (Variational (CNCE). For any variational approximation T(ylx") in lieu of p(yl xy))")),
e E(u""x''
TrAR(o' vll"" E,T, K) = Ep(a' a" af (1
log
K E (B(e",a'
-Eplar)
KL KL (plulze) ([
with p(-1x") << <<<<<<<<ff>>- ((-1="))or any x", we have the following properties:
1. IYAR S I(>';u/=").
2. If((y/=") = plylz"), IVAR = ICNCE-
3. limx- supg Tv A u F,T,KK
Taa::la"")
See Sec. A.3 for the proof. This bound side-steps the problem of requiring access to an arbitrary number of contrastive samples from the unknown p(y [ by i.i.d. sampling from the known and
4

---

Under review as a conference paper at ICLR 2021
tractable ())(="). We prove that as the number of examples goes to co, optimizing the bound w.r.t E converges to the true conditional MI. Interestingly, this holds true for any value of r, though the choice of r will most likely impact the convergence rate of the estimator.
Eq. 3 is superficially similar to the ELBO (evidence lower bound) objective used to train VAES (Kingma & Welling 2014), where plays the role of the approximate posterior (although the KL direction in the ELBO is inverted). This parallel suggests that 7 ""(y|=") = =p(y|=") may not be trivially that if we ignore the dependency of the first expectation term on and only optimize T to minimize the KL term, then it is
the optimal solution for some values of K and E. However, we
guaranteed that T [y ()) = p(yla") for any K and E. Thus, by the second property in Proposition 2 optimizing AR E, T K) w.r.t E will correspond to optimizing ICNCE.
In practice, the latter observation significantly simplifies the estimation problem as one can minimize a Monte-Carte approximation of the KL divergence w.r.t 7 by standard supervised learning: we can efficiently approximate the KL by taking samples from p(y(="). Those can be directly obtained by using the joint samples from p(z, y) included in the training set and computing x" from x!
4.2 IMPORTANCE SAMPLING APPROXIMATION
Maximizing AR
AR can still be challenging it requires estimating distribution over potentially high-dimensional inputs. In this section, we provide an importance sampling approximation of ICNCE that bypasses this issue
We start by observing that the optimal critic for INCE (x"; yy| K) KKis is E(x", y) = log p(alar")
ply) + +c(x"),
for any c. Assuming we have appropriately estimated E(x", y). is possible use importance sam- pling to produce approximate samples from p(y]==). This is achieved by first sampling VLLM ~ p(v) and resampling S M (K 0) examples i.i.d. from the normalized importance distribution exp E(a" .pa)
(STR(U)) = W E Fi.as), where We FM exp B(x" This process is also called "sam- pling importance resampling" (SIR). As M/K + 0o, it is guaranteed to produce samples from p(yl=") (Rubin 1987p. The SIR estimator is written as:
1 K
e Ela" aa)))
}.
ISIR(Z' y/ E, Ep(a" .xt sn..)))ll ) (vsik(vaxx)
log
(8)
ΣK F(
where we note the dependence of Q on wa and hence E. SIR is known to increase the variance of the estimator (Skare et al., 2003) and is wasteful given that only a smaller set of K examples actually used for MI estimation. Hereafter, we provide a cheap approximation of the SIR estimator.
The key idea is to rewrite the contribution of the negative samples in the denominator of Eq. as an .yx)
average (K - 1) Ek x ,x), and use the normalized importance weights wy. to estimate that term under the resampling distribution. We hypothesize that this variant has less variance as does not require the additional resampling step. The following proposition shows that as the number of negative examples goes to infinity, the proposed approximation converges to the true value of the conditional MI.
Proposition 3 (Importance Sampling LCNCE). The following approximation of ISIRE
BB(a",a ,a' (1))
Iis(I' ,yl"", E, K) = Eppa" ae an ())2)) log
R (e + (K - 1) Ek wk E( ar mn)
(9)
where wk Ek= exp Elx"" and E argsup INCE(I", K,, verifies:
exp B()"
vv)
1. lim supp (2':2)/22 E, I (x': y/ xz"),
2. lim sug 1i5 = log p(yl=",z')
p( + c( , =").
'The ability to perform that computation usually a key assumption in self-supervised learning approaches.
5

---

Under review as a conference paper at ICLR 2021
Estimated MI
4-
True
logik)
6
2logik)
3
10
InfoNCE
InfoNCE,
2
4
5
InfoNCR, 18
2
10a
107
10
10
10
10a
10a
107
10a
104
10?
10
Figure 2: We plot the value of the MI estimated by INCE and INCES bounds for three Gaus- sian covariates x', y as function of the number of negative samples K. We sample different covariances for fixed true MI (green horizontal line) and report emror bars. "InfoNCE" com- putes INCE(1' u)); "InfoNCEs" computes INCe(>";)) + --LNCceL': "InfoNCEs IS" computes INCE(*": + [1s; y
The proof can be found in Sec. A.4 This objective up-weights the negative contribution to the normalization term of examples that have high probability under the resampling distribution. This approximation is cheap to compute given that the negative samples still initially come from the marginal distribution p(y) and avoids the need for resampling. The proposition shows that in the limit E' converges the conditional MI and the optimal E converges
of K 30, optimizing IıS to the optimal ICNCE solution. We also note that we suppose E The Iıs approximation provides general, grounded way of sampling "harder" negatives by filtering samples from the easily-sampled marginal p(3).
EXPERIMENTS
We start by investigating whether maximizing the decomposed MI using our conditional MI bound leads to better estimate of the ground-truth MI in synthetic experiment. Then, we experiment self-supervised image representation learning domain. Finally, we explore an application to natural language generation in sequential setting, such as conversational dialogue.
5.1 SYNTHETIC DATA
We extend Poole al. 2019)* two variable setup three variables. We posit that {a', ...v) three Gaussian co-variates, x' x". (O, Σ) and we choose Σ such that we can control the total mutual information T ((<r, x y) such that 7 {5, 10, 15, 20) (see Appendix for pseudo-code and details of the setup). We aim to estimate the total MI [({x', x" }: y) and compare the performance of our approximators in doing so. For more details of this particular experimental setting, see App.
y)
In Figure 2 we compare the estimate of the MI obtained by:
1. InfoNCE, which computes INCk(fx',e)
((=',='))]'])_" x²") El E, K) and will serve as our baseline;
2. InfoNCEs, which probes the effectiveness of decomposing the total MI into a sum of smaller terms and computes INCE(*" , E K/2) ICNCE(22 . E, K/2), where K/2 samples are obtained from p(y) and K/2 are sampled from p(y[=");
3. InfoNCEs IS, the decomposed bound using our importance sampling approximation to the conditional MI IıS, i.c. INCE(=", y E, K) + Iıs("', ,y,"",""" E, K). This does not require access to samples from p(yl=") and aims to test the validity of our approximation in empirical setting. Both terms reuse the same number of samples K.
For 2., we use only half as many samples as InfoNCE to estimate each term in the MI decomposition (K/2), so that the total number of negative samples is comparable InfoNCE. Note that use K samples in "InfoNCE IS". because those are reused for the conditional MI computation. All crities E are parametrized by MLPs explained in Sec. B Our results in Figure show that. for larger amounts of true MI, decomposing MI as we proposed can capture more nats than InfoNCE with an order magnitude less examples. We also note that the importance sampling estimator seems

---

Under review as a conference paper at ICLR 2021
Table 1: Accuracy on ImageNet linear evaluation. x + y denotes standard contrastive matching between views. In "InfoNCEs", we use the same base InfoMin Aug. architecture but augments the loss function with conditional MI maximization across views (x < y). All models use a standard Resnet-50 architecture. (1) represents improvement over InfoMin Aug.
Model
Views
Epoch
Top-1
Top-5
SimCLR
+ I +
200
66.6
MocoV2
200
67.5
InfoMin Aug.
I + y
200
70.1
89.5
+InfoNCEs (a" cx)(
x + y, x" y, x Ox" y
200
70.6 (1)
89.8 (1)
+InfoNCEs (a" = crop(z))
+ x² + y, Ox!
200
70.9 (1)
90.1 (1)
without cond. MI (æ" crop(e))
I + y, x $
200
69.9 (L)
89.5 (-)
Additional Losses / More Epochs
SWAV
200
72.0
ByOL
800
74.3
91.6
estimate MI reliably. Its empirical behavior for MI {5, 10) could indicate that InfoNCEs is valid lower bound on MI, although we couldn't 't prove it formally.
5.2 VISION
Imagenet We study self-supervised learning of image representations using 224x224 images from ImageNet. The evaluation is performed by fitting a linear classifier to the task labels using the pre-trained representations only, that is, we fix the weights of the pre-trained image encoder f. Each input image is independently augmented into two views æ and y using stochastically applied transformation. For the base model hyper-parane pareametrssss and augmentations, we follow the "InfoMin Aug." setup (Tian et al.] 2020). This uses random resized crop, color jittering, gaussian blur, rand augment, color dropping, and jigsaw as augmentations and uses a momentum-contee contrastive memory buffer of K 65536 examples (Chen al. 2020b).
view We fork into two sub-views {a' ":": set x' A and x" to be information- of x. We found beneficial to maximize both decompositions of the MI: I(=': y) + I[=": y|z') = + y(= contained in v', our encoder f' is trained to maximize:
[==:y)==") By noting that yla"" is likely zero given that the information of x" is
L AINCE(( :lf, + A1 (INCE(E' K) + Lis(' {.
(10)
Note that if x J, then our decomposition boils down to maximizing Therefore, InfoMin Aug. is recovered by fixing À = 1 by setting x" z. The computation of the conditional MI term does not add computational cost as it can be computed by caching the logits used in the unconditional MI terms (see Sec. B)
standard InfoNCE bound.
We experiment with two ways of obtaining restricted information views x": cut, which applies cutout to x, and crop which is inspired by/Caron al. (2020) and consists in cropping the image aggressively and resizing the resulting crops to 96x96. To do so, we use the CandomResizedCrog from the torchvision. transforms module with parameters: s (0.05, 0.14). Results are reported in Table 1. Augmenting the InfoMin Aug, base model with our conditional contrastive loss leads to 0.8% gains on top-1 accuracy and 0.6% on top-5 accuracy. We notice that the crop strategy seems to perform slightly better than the cut strategy. One reason could be that cutout introduces image patches that do not follow the pixel statistics the corpus. More generally, we think there could be information restricted views that are better suited than others. In order to isolate the impact on performance due to integrating an additional view z". i.c. the INCE(*"1 y K) term in the optimization, we set the conditional mutual information term to zero in the line "without cond. MI We see that this does not improve over the baseline InfoMin Aug., and its performance is 1% lower than method, pointing observed gains. We also include the very recent results of SwAV (Caron et al.) (2020) and ByOL (Grill et al. 2020) which use larger number of views ASwAV) and different loss functions (SwAV, ByOL.)
the fact that maximizing conditional MI across views provides the
7

---

Under review as a conference paper at ICLR 2021
and thus we think are orthogonal to our approach. We think our approach is general and could be integrated in those solutions as well.
We also experiment on CIFAR-I0 building upon SimCLR (Chen et al. 2020b), which uses standard ResNet-50 architecture by replacing the first 7x7 Conv of stride 2 with 3x3 Conv of stride 1 and also remove the max pooling operation. In order to generate the views, we use Inception crop (flip and resize 32x32) and color distortion. We train with learning rate 0.5, batch-size 800, momentum coefficient of 0.9 and cosine annealing schedule. Our energy function is the cosine similarity between representations scaled by a temperature of 0.5 (Chen et al. 2020b). We obtain top-1 I accuracy of 94.7% using linear classifier compared to 94.0% as reported in Chen et al. (2020b) and 95. % for a supervised baseline with same architecture.
CIFAR-10
5.3 DIALOGUE
For dialogue language modeling, we adopt the predictive coding framework EElias 1955 McAllester Stratos 2018) and consider past and future a dialogue as views of the same conversation. Given L utterances x (Ii,...,, ,T)), we maximize TNCs( (rggs; x>* If; K), where past 2i = (311...,, .&) and future Ty = (Ik-1;-..- FT. are obtained by choosing a split point 1 k < L. We obtain f(xxxx) (* ) by computing forward pass of the fine-tuned "small" GPT2 model (Radford et al. 2019) on past and future tokens, respectively, and obtaining the state corresponding to the last token in the last layer.
We evaluate our introduced models against different baselines. GPT2 is a basic small pre-trained model fine-tuned the dialogue corpus. TransferTransfo (Wolf et al. 2019) augments the standard word prediction loss in GPT2 with the next-sentence prediction loss similar to Devlin et al. (2019). Our baseline GPT2+InfONCE maximizes (xc=(E=g;;>>> K) in addition to standard next- word prediction loss. In GPT2+InfNNCSs, we further set x' = TSa and x" = 2t. the recent past, To maximize the conditional MI bound, we sample contrastive
and maximize (Z futures from p Ix: 0urrz), using GPT2 itself as the variational approximation"
We fine-tune all models on the Wizard of Wikipedia (WoW) dataset (Dinan et al., 2018 with early stopping validation perplexity. We evaluate our models using automated metries and human evaluation: we report perplexity (ppl), BLEU (Papineni et al., 2002). and word-repetition-baar metrics from Welleck et al. (2019). specifically: seq-rep-n measures the portion of duplicate n-grams and seq-rep-avg averages over n E {2, 3, 4, 5, 6]. We measure diversity via dist-n (Li et al. 2016). the number of unique n-grams, normalized by the total number of n-grams.
Table 4 shows results on the validation set. For the test set results, please refer to the Appendix. Incorporating InfoNCE yields improvements all metries Please refer to the Appendix for sample dialogue exchanges. We also perform human evaluation on randomly sampled 1000 WoW dialogue contexts. We present the annotators with a pair of candidate responses consisting of GPT2+InfoNCEs responses and baseline responses. They were asked to compare the pairs regarding interestingness, relevance and humanness, using a 3-point Likert scale (Zhang et al. 2019). Table Lists the difference between fraction of wins for GPT2+InfNNCEs and other models as H-rei, H-hum, and I1-int. Overall, GPT2+InfoNCEs was strongly preferred over GPT2, TransferTransfe and GPT2+InfONCE, but not the gold response. Bootstrap confidence intervals and p-values (t-test) indicate all improvements except for GPT2+InfoNCE on the relevance criterion are significant at a=0.55.
DISCUSSION
The result in Eq. is reminiscent of conditional noise-contrastive estimation (CNCE) Ceylan & Gutmann 2018) which proposes a framework for data-conditional noise distributions for noise contrastive estimation (Gutmann & Hyvärinen] 20122). Here, we provide an alternative interpretation in terms of bound on conditional mutual information. In CNCE, the proposal distribution is obtained by noising the conditional proposal distribution. would be interesting to investigate whether
4 'The negative sampling of future candidates is done offline. Note that our results are not directly comparable with Li et al. 2019 as their model is trained from scratch on not publicly available Reddit-based corpus.
8

---

Under review as a conference paper at ICLR 2021
Table Results for perplexity, sequence-level metric, token-level metrics, BLEU, diversity metrics and human evaluation on the valid data of the Wizard of Wikipedia dataset
(Dinan et al. 2018).
Model
ppl
seq-rep
rep
wrep
uniq
dist-1
dist-2
BLEU
H-rel
H-hum
H-int
GPT2
19.21
0.057
0.133
0.128
8276
0.065
0.389
0.776
0.20
0.12
0.35
TransferTransfo
19.32
0.077
0.132
0.129
7735
0.064
0.390
0.754
0.17
0.09
0.27
GPT2+InfONCE
18.85
0.054
0.132
0.131
8598
0.064
0.382
0.798
0.05
0.10
0.08
GPT2+InfNNCEs
18.70
0.053
0.131
0.129
8673
0.066
0.401
0.816
0
0
0
Ground Truth
0.052
0.095
9236
0.069
0.416
-0.32
-0.34
-0.14
is possible form information-restrice views by similar noise injection, and whether "optimal" info-restricted views exist.
Recent work questioned whether MI maximization itself is the of the recent success in representation learning (Rainforth et al., 2018 Tschannen et al.] 2019). These observed that models capturing a larger amount of mutual information between views do not always lead to better downstream performance and that other desirable properties of the representation space may be responsible for the improvements (Wang Isola 2020). Although we acknowledge that various factors can be at play for downstream performance, we posit that devising more effective ways to maximize MI will still prove useful in representation learning, especially paired with architectural inductive biases or explicit regularization methods.
REFERENCES
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Proc. Conf. Neural Information Processing Systems (NeurlPS), pp. 15509- 15519. 2019.
David Barber and Felix Agakov. The im algorithm: A variational approach to information maximization. In Proc. Conf. on Neural Information Processing Systems (NIPS), pp. 201-208, 2003.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised
learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006 09882, 2020.
Ciwan Ceylan and Michael U Gutmann. Conditional noise-contrastive estimation of unnormalised models. arXiv preprint arvi: www.:1806.002000 906603664, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05700 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint wtt.Xxv:20030022999 2020b.
Chris Cremer, Quaid Morris, and David Duvenaud. Reinterpreting importance-weighted autoencoders. Proc. Int. Conf. on Learning Representations (ICLR), 2017.
Jacob Devlin, Ming- Wing-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proc. Conf. of the North American Chapter of the Assoc. for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. arXiv preprint 181.001211,,, 2018.
Peter Elias. Predictive coding-i. IRE Transactions on Information Theory, 1(1):16-24, 1955.
Adam Foster, Martin Jankowiak, Matthew O' Meara, Yee whye Teh, and Tom Rainforth. A unified stochastic gradient approach to designing hayesian-optimal experiments. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 2959-2969, Online, 26-28 Aug 2020. PMLR. URL httt:://proceedings mlr, www.:pp..s//e..m......e...e press/ 2110/foster20a..tt
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations, arXiv preprint 1rXiv:1803.077 803.07728, 2018.

---

Under review as a conference paper at ICLR 2021
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre II Richemond. Elena Buchatskaya. Carl Doersch, Bermardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13(ccb):307-311, 2012.
Geoffrey Hinton. A practical guide training restricted boltzmann machines. In Neural networks: Tricks of the trade, pp. 599-619. Springer, 2012.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchile Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In Proc. Int. Conf. on Learning Representations (ICLR), 2019.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations. ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. http:/ttt:///rx// barg//ba//ii 2b111611..111
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E. 6966),66638,, 2004.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-poomotin promoting objective function for neural conversation models. In Proc. Conf. of the North American Chapter of the Assoc. for Computational Linguistics: Human Language Technologies (NAACL-HLT), pp. 110- 119, 2016.
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason We- ston. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. arXiv preprint arXiv:191 1.03860, 2019.
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. arXiv preprint ##Xiv:180900180 2018.
David McAllester. Information theoretic co-training. arXiv preprint arXiv:1802.07572 2018.
David McAllester and Karl Stratos. Formal limitations the measurement of mutual information. arXiv
preprint arXiv:1811.00m 004251, 2018.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Aaron den Oord, Yazhe Li, and Orial Vinyals. Representation learning with contrastive predictive coding. arXiv preprint S.kiv:1807.037770 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of
machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. pp. 31 1-318. Association for Computational Linguistics, 2002.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On variational bounds of mutual information. In Proc. Int. Conf. on Machine Learning (ICML), 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Suiskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Tom Rainforth, Adam R Kosiorek, Tuan Anh Le, Chris J Maddison, Maximilian Igl. Frank Wood, and Yee Whye Teh. Tighter variational bounds are not necessarily better. arxiv preprint arXiv: 1802.04537 2018
Donald B. Rubin. The calculation posterior distributions by data augmentation: Comment: noniterative sampling/mpoorance resampling altermative to the data augmentation algorithm for creating a few imputations when fractions of missing information are modest: The SIR algorithm. Journal of the American Statistical Association, 82(398):543-546 1987.
Øivind Skare, Erik Bølviken, and Lars Holden. Improved sampling-importan resampling and reduced bias importance sampling. Scandinavian Journal of Statistics, 10(4)7719-737, 2003.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. McXiv:190.005849, 2019.
Contrastive multiview coding. arXiv preprint
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for gond views for contrastive learning. arXiv preprint arXiv:2005.108 10243, 2020.
10

---

Under review as a conference paper at ICLR 2021
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint 19Xiv:1077.1622 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. arXiv preprint arXiv:2005.10242 2020.
Sean Welleck, Ili Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv: 1901199998899,,,11111 2019.
Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. Transfertransfo: A transfer learning
approach for neural network based conversational agents. In Proc. Conf. Neural Information Processing Systems (NcurIPS) CAI Workshop, 2019.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan, Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.000 1.0036,, 2019.

---

Under review as a conference paper at ICLR 2021
A DERIVATIONS
A.1 DERIVATION OF INFONCE. INCE
We start from Barber and Agakov's variational lower bound on MI (Barber Agakov 2003). [1; v) can be bounded as follows:
[(x; v) Ep(r..) log p(ulz) p(y)
2 Ep(z.s) log ((v(>)
(11)
p(y)
where is arbitrary distribution. We show that the InfoNCE bound (Oord al. 2018 corresponds a particular choice for the variational distribution & followed by the application the Jensen inequality.
Specifically, g(3/ x) is defined by independently sampling a set of examples f1/,,..,,,4 from proposal
distribution (1) and then choosing from {.n....; yk } in proportion to the importance weights My
Ex Ee where E is a function that takes and y and outputs scalar. In the context of representation
learning, E is usually a dot product between some representations of and y, e.g. f(x)² f(v (Oord et al.
2018). The unnormalized density of y given specific of samples y2.k {......, yK. and is:
K B(xa)
4 ar v2.k) (vy) (f(=.)) Eh e((.v)))
(12)
where we introduce factor which provides "normalization expectation". By normalization in expectation, we mean that taking the expectation of q(yl æ. y2:K) with respect to resampling alternatives y2.k from I (y) produces normalized density (see Sec. A.1.1 for a derivation):
q x))
Ex( (1y1( y2.k) )]
(13)
where (ty: ) = I (3u)). The InfoNCE bound OOrd et al. distribution as the marginal distribution, (3) = p(v) and applying Jensen's 's inequality, giving:
2018 is then obtained by setting the proposal
I(x; v) 2 Ep(r.s) log Epier ( / l2, B22.k) 2 Bp(a.w)
Fp(ea)) log p(v) K wy
ply)
p(y)
K F((),))
= Ep(x.)) Eplense) log ek(e,v) >xaz (((..)))
log K EE= (((,,))))
B(x.y)
INCE(E; s)E, log K,
(14)
where the second inequality been obtained using Jensen's 's inequality.
A.1.1 DERIVATION OF NORMALIZED DISTRIBUTION
We follow Cremer al. 2017) to show that g(yl>) = Ep. (d(y/a, ye.k )] normalized distribution:
[ gly)x) dy / Eyz.k =(y)
F(re))
dy
((a.v))) ) + c((x,))
2=2
1, +(y)
((w,))
dy
k2 (laive)) + cB(()))
F(x,g)
K 2im (((.)))) + B(aa))
B(z,v)
=Ba(ruk)
k=1 (((()))))
R
c F(x.21)
K E.(rnk)
k= E(orve))
Elann))
Br(rix) x)
Σk=1 (k(..)))
K F(rm)))
Bx(rx)
(15)
Σk= (((a.)))
12

---

Under review as a conference paper at ICLR 2021
A.2 PROOFS FOR ICNCE
Proposition (Conditional InfoNCE). following a lower-bound on the conditional mutual information [(x' y)) and verifies the properties below:
Ela" .nn)
(eNcE(''sul"" E, K)- Eppcd (1)(maaatl")
log
(6)
2k== x'
1. leNCE S [[<':1/>=")
2. E" = argsupp lONCE log viul"" vly x")
afx' x²).
When dc and E., we recover the true conditional MI; limx- ICNCE( u/z", EE,, K) 1(x'; y")"")
Proof. We begin with I., the derivation is as follows:
[a' :yl"" = Epta", ,0,,)) log p(yla",>' p(ul=")
2 Bptan sa log aly 20 ,x))
(16)
p(u)<=")
(p(a log Epirax (onenwl , M2.k) p(ul=")
(17)
plyla") K wy p(ul=")
2 Ep(a" wo Epienakee) log
(18)
K Kee Mla
Ep(e" xz Ep(sz log
(19)
Ei=1 BBla" or ma)
Bla"
= Epts" of Rpion log
(20)
K e 126)
R 2==1
= LeNce (4's w|=",""""""""".""""""""""" K.,),
(21)
where used in Eq. T6 Jensen's inequality following Barber and Agakov's bound (Barber & Agakoy 2003 used p(yla") as our proposal distribution for the variational approximation
For 2., we rewrite LeNce by grouping the expectation w.r.t x":
E(x" m1)
E
Ep(a"))
log
(22)
Ek=n Ala",ative
Given that both distributions in the inner-most expectation condition the same x". this term has the same form INCe and therefore the optimal solution is E log x' =²) can can(a") (a') (Ma & Collins 2018
p(ul=")
The optimal for JONCE is thus obtained by choosing E(s" x v)) Ee"" for each x". giving E.
log plu s' ""'"") c(x' ==").
p()l=")
For proving 3., we substitute the optimal critic and take the limit K + co. We have:
plyl Kull"".r" ,x) Mo)a"")
lim Ep(e" of (p(eaell"") log K000
(23)
plys Jar xx)) plun [x")
>x=z
pluel=" (o0
plunla"
From the Strong Law of Large Numbers, we know that as R as K + 00 a.s., therefore (relabeling y = y1):
EK
pluel "aa""" pluselio!r)
1,
plyl=" Plyla"")
ICNCE ~o00000 Ep(x" m))
log p(yle"" =²) Diyla")
+ K 1)
(24)
R
log plylæ",
K
~o000 Bp(a")a""
log
(25)
p(y)=")
p(yla" of) Plal=")
K
~o [('', "y(a"))
(26)
where the last equality is obtained by noting that the second term + 0.
13

---

Under review as a conference paper at ICLR 2021
A.3 PROOFS FOR IVAR
Proposition 2 (Variational LONCE). For any variational approximation T(yl²") in lieu of ply
Ble" wl)
Iv y/ E, Ep(zt,a (ri))aa
log
Ex e (llu"ra'
R
Bylcot KL (p(ul=") (I =(((l=))
with p(-l=") << T(-(=") for any x², we have the following properties:
1. IVAR - u(l')"
2. I/ [y((""==) = plu ), IVAR = LONCE.
3. limx- supg Iv (x(2's ull"", E,
B, K, 1(x112/x")
Proof. For I., we proceed as follows:
log q(vl=",a") (<"()"===""
[<<'i0/a") 2 Ex(x.))
p(yl=")r(ule""
log g(vl=", xx')
Ep(e.))
Ep(e) ([L(o(ll<""1><<>>>""""
T(yla=")
2 Ep(e.ni)roe
log R
Ep(x) [KL(p(yle") (+(()=>))
K km ca(
= Tvan(a y/=" E,
(27)
where the last step has been obtained in Eq. Proving 2. is straightforward by noting that if p. (LLppy) to ICNCE.
18
(<(p(yl=")llr(y)" and first term corresponds
Proving 3. goes as follows:
sup Epcr ae (ntt log e ,''".=
KL (p(ul=") I| ((]l=))
(28)
2==1 CE('" (ataaa))
log pluil 1z.",
p(v1 (x0))
plunlo',
-Eple" (nr(taii)))
log
log
(29)
7832 (xx")
7(h (xr)
K
7 (xx)
plun le', x(")
=I(x',ulur" Epiin an
log
(30)
K
- [a', y(e").
(31)
This is obtained by noting that (1) for any K and argsup lv AR plyla",,'' T(p (because the KL doesn't
depend on E) and (2) the second term in the last line goes to 0 for K + oo (a straightforward applica- tion of the Strong Law of Large Numbers shows that for samples y2.K drawn from 7 kl"),, we have: R DK plve ='."")
(WE 2") K 1).
A.4 PROOFS FOR I1S
We will be using the following lemma.
Lemma 1. For any x", s' and and any sequence Ex such that ||E - El - 0:
KC Ex(e"" x²
lim Epivai)) log eFx(""='s (K 1) k "xx a' 4000
(32)
KeF x
lim E bet) log
cBl"" x' + ΣK Bla"
(33)
where wa exp B(c" .v))
Σk= exp Eix" mnd for E(x", ,n) argsupe INCE(*", V,E, K) = p(vel=")
p(3))
14

---

Under review as a conference paper at ICLR 2021
Prouf. We that almost surely, for y2:k p(-):
Σ K (iuel=")
Ex(a" (x" mm)
Whe "xle"x" xx'
p(ae)
KK000 Bolylung B(z" (t/1))
(34)
k=
MOke
where we applied the Strong Law of Large Numbers to the denominator. For the numerator, we write:
pluel=") Fxla"aa'vv
p(ul(=")
Eia" a'
K
p(ya)
K
plyn)
K p(yel=")
p(yn) (e (x Bidl ar mk))
K
and note that the first term is the standard IS estimator using p(yx) proposal distribution and tends "ta" .') from the Strong Law of Large Numbers, while the second term goes to Ex tends E Epcylarye E(="
uniformly.
Ke (c This gives limx-+0 Bp(vax) log Bxgce" +(K 1) El wge mflme log
Following the same logic, without the importance-sampea
demonstrates that:
KeE(=", .z
E(x" xx
lim Ep("a)il"") log eR("" + En e Flael,
log
K
Ep(n w/ B(a" (a
which concludes the proof.
Proposition 3 (Importance Sampling (ONCE). The following approximation of ISIR:
Blx" xe' w1
Irs(s', ,,,",,, K) Egcu" of (n)))(n(x log (e + (K - 1) Ex uge aa
K
where Wh
exp B(x" wc) and = arg INCE("", F, K), verifies:
Ex=2 exp B(a"
1. lim supg Uis(''sulrr K,K)
2. lim argsupg 118 log plula",z') plyla") + c(x', x²).
Proof. By applying Lemma with F.R- F., we know for any Fi:
Ke B(x"."" 22.00
lim Lis(a' iss(a':ll=" E, K, +0000
lim Eptx" s)p(v. (x")) log Bla"" + Ekzz eEls", un)
In particular, the RHS of the equality corresponds to limx-+20 loNcEla' ylu", E, K). That quantity is smaller than [a",, s/x"", with equality for F. F This guarantees that: lim sup Lis('' (x's K00 E
y/=" K,E,K) lim Iis(' vol"", E. K) 1( vlz").
(35)
We now prove the reverse inequality. let 2c limg- supg L1s(x' : E, B. assume toward a contradiction that e We know that:
(ee, "læ"), and
3Ko, AK 2 Ku, sup Lis(2' vy)""" E, E, 2 1(x' y/("))))))) te.
Now, AK 2 Ko, Fxx he such that:
Ins(a's ule", Ex. K.K) inplisldiill E, E. K) 2'
E
and thus: AK 2 Iis Ex, E, K) x
Since Ex E RIIXIIIIIII) {Ex] contains a subsequence that converges certain Exc XI Without loss of generality, assume that VK, Væ", Vz', Bpgg]ER (z". ,',v), = 0 which implics that (x"",, x',, y)] = (similarly to INCE, IıS is invariant to constants added E).
In particular, this guarantees that F 1100 cc. Otherwise, would have (x"" x', v) -oo for given y, which would then imply [1s(x' v/a" Foc, E,, K) oo and give contradiction.
We can now apply Lemma to [Ex) and Ex to show that lim (a'::/lll"" EK, K) lim JeNeE(E (2', y/a", Exe. K). and get a contradiction: the first term is larger than [(x', "g/=" + while the second smaller than [<',y)>"*
15

---

Under review as a conference paper at ICLR 2021
B PSEUDOCODE
B.1 LOSS COMPUTATION
We provide pseudo-code for the loss computation which uses MocoV2 backbone comprising memory of contrastive examples obtained using muomentum-averagee encoder (Chen al. 2020b).
compute_ (o xpp, y, f, e_ema, memory, lam=0.5)
Args:
app: info-restricted
up: vtew
y: a view
f: standard encoder
f- ema:: momentum averaged encoder
memory: memory bank of representations
Returns:
lam mi (zp; y) (1 lam) + (mi (app; y) mi (xp; / app))
encode up upp with standard encoder, (1, dsm)
q_xP, x_xpp (x_p), {(x_pp)
# encode with momentun-averaged encoder, (1, dim)
k_y = f_ema(y) (.).yydddch(h)))
# (1 n_mem,), first is mpp-y score
logits_xpp_y dot (q_xp, cat memory))
# (1 n_mem,), first is up-y score logits_xp_] do cat ck_y, memory))
# infonce bound between up and y woftmax sittss_ppyy_ll [0] infonce bound between upp and nce_ nce_xpp_y = -log_aoftmax(( www.fftttaaxoooooom (oootttaaaa [0]
nce _x__pp_y
K len(logits_xpp_y)
# compute resampling importance weights
w_pp-y softmax (logits_xpp__[1:)
# form opprozimation to the partition function (Eg. 12)
z_xp-y = (K 1) w_PP-y exp(logits_xp_y(t.)
2_xp-y 2_xp_y.auml
exp(logits_xppy(O]]
# infonce bound on the conditional mutual information
nce_xp_y_1 -p-y_I_xpp -logits_xp_ylle + log(2.xp_y)
# compose final loss
loss lam nce_xp-y
loss += (1-lam) (nce_xpp_y nce_xp_y_I.xpp
return loss
B.2 SYNTHETIC EXPERIMENTS
Here, provide details for Sec. 5.1 In this experiment, each x², =" and y 20-dimensional. For each dimension, sampled (ar,, 2/' V. from correlated Gaussian with mean and covariance matrix cov, For given value of MI, mi = {5, 10, 15, 20), we sample covariance matrices cov, = sample_ cocppleccvlveee (mi,), such that Σ, mi; = mi, mi; chosen at random. We optimize the bounds by stochastic gradient descent (Adam, lcarning rate 10 10- encoders / are multi-layer perceptrons with single hidden layer ReLU activation. Both hidden and output layer have size 100.
InfoNCE computes:
e (le /lE',="') """"""" f(w) Ep log (f f(v) + Σk= (l(
log K, y2:k ~ p(v),
where the proposal is the marginal distribution p(y). E' is chosen to be dot product between representations, F, denotes expectation w.r.t. the known joint distribution p(æ', x²" and is approximated with Monte-Carla
[ar ,""""" denotes concatenation and is 1-hidden layer ML.P.
16

---

Under review as a conference paper at ICLR 2021
InfoNCEs computes:
Epts" slp(la)))
log
(36)
ellwnry f(0)
E Ennge ellarr (olllr)ssivv) sive]
Alla" ,/1)²" 1(x)
Bptx" 21 (e"))
log ef(""""'")" f(v) Ex "f)"" sive)
+ 2 log K
where (xx is just [[z, 0}) in order to re-use MLP parameters for the two terms. The negative samples of the conditional MI term come from the conditional distribution p(yl="), ), which is assumed to be known in this controlled setting. We maximize both lower bounds with respect to the encoder f.
We report pseudo-code for sample cov, used generate 3x3 covariance matrices for fixed [([a' ,x" (a"):) b) and uniformly sampled [(=" /)/I((a [=(a[[([["=)))"" (oa'"x""):))) v)::)):
def sample_cov (mi):
alpha random. uniform (0. 0.9)
params rranommnnomal (0, 1ς)
# use black box optimizer (Nealder--ead) to determine opt_parame opt_param ming residual(params, return project.posdef (opt_params)
def project R to positive defénate matriz cov zeros 3)
(x):
[v[tril_indicem (3)] xx
cov /= callumn_norm(ee return dot cov.T)
def analytical_micon (cov))
# compute analytical NI of covariate Gaussion variables
cov_01 cov[:2, :2]
cov_2 cov[2:3, 2:3]
mi_xp.xpp-y 0.5 (dooo((cc)o)_11) log(det(cov_2)) log (det (cov)))
cov_1 cov[1:2, 1:2]
cov_23 cov [ov[1: 1:]
mi_xp_y 0.5 (coo_(((ooo))))) + log(detccov_2) (coo mi dd_xp_xxp__3 mi_xp_y
def residual (x, mi, a):
# penalize difference between analytical mi and target mi, ami project_poadef (x)
mi_xp-y, mi_xp-y analytical pacicallmiimae (cov)
return _mi_xp_xpp_y mi) 2 + _mi__p__y - a mi) **
C EXPERIMENTS ON DIALOGUE
C.I INFONCE DETAILS
For all InfoNCE terms, given the past, the model is trained to pick the ground-truth future among set of N future candidates. This candidate includes the ground-truth future and N negative futures drawn from different proposal distributions. To compute InfoNCE( (* f*>>*)), we consider the ground truth future of each sample as negative candidate for the other samples in the batch. Using this approach, the number of candidates N is equated to the batch size. This ensures that negative samples are sampled from the marginal distribution p(==*)) ). To compute the conditional information bound InfoNCEs. sample negative futures p(y|æs) xx)) ) by leveraging GPT2 model itself, by conditioning model only on the most recent utterance æs the past.
C.2 EXPERIMENTAL SETUP
Given memory constraints, proposed models are trained with a batch size per GPU over epochs, considering up to three utterances for the future and five utterances in the past. All the models are trained on 2 NVIDIA V100s. The models early-stop in the 4th epoch. We use the Adam optimizer with learning rate of 6.25 x 10 which we linearly decay zemo during training. Dropout is set to 10% all layers. InfoNCE/InfoCEF
weighted with factor 0.1 the loss function.
17

---

Under review as a conference paper at ICLR 2021
Table 3: A sample dialogue between speaker A and speaker B from the Wizard of Wikipedia dataset. The four rows from top bottom are: (1) the "past" dialogue up utterance k (2) the ground-trute utterance for the next turn ++1 (3) generations for the next turn sampled from the "restricted context" conditional "future" distribution ply (rx) ) (4) future candidates sampled from the groundtruth "future" distribution. We can see that p(y| is semantically close but incoherent w.r.t to the dialogue history as it was conditioned solcly on the immediate past utterance 2g. However, we can notice that p is semantically distant from z as it sampled randomly from the data distribution. The highlighted text in green correspond to the topic of the conversation. Speaker B indicates that has never done either parachuting skydiving. p(y(æx)) corresponds to the set of hard negatives that closely related to the conversation. B₁ corresponds to the utterance generated based on the restricted context TR. The utterance is on-topic but completely contradictory what speaker B has said in the past. On the other hand B is randomly sampled from other dialogues. We can observe that the utterance is clearly irrelevant to the conversation. Therefore, easier to the model discriminate between B and Bgt.
A: like parachuting or skydiving
I've never done either but they sound terrifying, not a fan heights.
A:
But it is interesting game. This first parachute jump in history was made by Andre Jacques.
B: Oh really ? Sounds like french name, what year did he do it
It done in October 22 1797. They tested his contraption by leaping from a
hydrogen balloon.
B: Was he successful did he kick the bucket off that stunt?
A: think its a success. The military developed parachuting tech.
p()(
Byt Yeah nowadays they are lot more stable and well made.
31.N
B1 That is great. I've been skydiving for days now
How is it ?
p(y|Z*)
Bz: Oh have never flown but I'm glad to know.
By: I've been dying for it since I was kid.
B4: Yes, that is why NASA had an advanced mechanics tech for months.
B5: went parachuting last Sunday and enjoyed it.
VIEN p(y)
B1
think science fiction an amazing genre for anything
B
Can you imagine the world without internet access ?
By:
am just finishing my university course and will be qualified pharmacist.
B;
don't t know how to be romantic. have trouble expressing emotional
attraction.
B5:
think Krav Maga is martial art sport. That 's the reason I picked it
Table 4: Results for perplexity. sequence-level metric, token-level metrics, BLEU and diversity metrics on the test data of the Wizard of Wikipedia dataset. Results demonstrate that the proposed InfoNCE and InfoNCEs bounds archive lower perplexity, reduce next- next-token repetition and increase the number of unique next-tokens compared the baselines GPT2 and TransferTrransf
Model
ppl
seq-rep
rер
wrep
uniq dist-1
dist-2
BLEU
GPT2
19.24
0.064
0.130
0.132
7393
0.064
0.392
0.775
TransferTransfo
19.33
0.078
0.134
0.132
7735
0.058
0.386
0.752
GPT2+InfNNCE (ours)
18.88
0.065
0.126
0.131
8432
0.065
0.390
0.799
GPT2+InfoNCEs (ours)
18.76
0.050
0.120
0.128
8666
0.070
0.405
0.810
Ground Truth
0.052
0.095
9236
0.069
0.416
18

---

Under review as a conference paper at ICLR 2021
Table 5:
Selected responses to the same context from different methods fine-tuned the Wizard of Wikipedia datasets.
Context
Generated Responses
A: recently took up archery and think love it Do you know anything about Archery ?
GPT2+InloNCEs: Archery originated from ancient
Mesopotamia, and is actually practiced internationally in over 200 countries. GPT2+1nlNNCE: I am not sure but many traditions and
B: Archery sport skill of using a bow to propel arrows.
principles arose during the Mesolithic and Paleolithic era.
A: And a great sport it is Do you know where archery originated from?
TransferTransfe Yep you just use it for skill and using it to shoot arrows.
GPT2: don't know, but know that the old French called it archer's art.
A: What is your motivation?
GPT2+InloNCEs:
Yes, definitely! You should check out Mark Twain's 's famous motivational book ! There how important it is to meditate.
B: That's tough question. I 'd say my own progress. like seeing how much improve with something. You
theories that point out
A: am retired now. Are you familiar with the work of Mehr and Meyer, well known psychologists
GPT2+InfONCE:
They said that in psychology re- search, theories and evidence, partici- pants who enjoyed continuous experi- ence could be more likely be success- ful.
" Motivation is a word that is part the popular culture as few other psychological concepts are. Do you agree with what they said? ?
B: They said that
TransferTransfor
Absolutely, disagree. Are you an author or a commentator?
GPT2: That's awesome. So what types differences
exist between "good motivation and bad?
C.3 HUMAN EVALUATION
We closely follow the protocol used in Zhang et al. (2019). Systems paired and each response pair was presented to 3 judges random order a point Likert scale. We majority vote for each response pair decide whether systeml, system2, or neither, performed better. We then bootstrap the set majority votes to obtain a 95% confidence interval on the expected difference between systeml and system2. If this confidence interval contains O, the difference is deemed insignificant. We also compute p-values from the confidence intervals
the following tables, "pivot" is always the system given by our full InfoNCEs model. Pairings where the pairwise confidence interval marked with "g" have significant difference between systems.
Human Evaluation:
Which response is more relevant?
pivol_wo wins
pivot_CI WC
cmpsys_wing wins
empsys_Ce __
pairwise_ CI
p
emp_sys
GPT2
0,48726
(0.46, 0.52]
0.28662
(0.26, 0.32)
(0.15, 0.26]*
1.24835e- 12
GPT2MMI
0.65833
(0.62, 0.7]
0.16250
(0.13, 0.2]
(0.43, 0.56]
6. 6.11888e-42
GPT2_NSP
0.46888
(0.44, 0.5]
0.30043
(0.27, 0.33)
(0.11, 0.22]*
6.67922e-09
InfoNCE
0.41711
(0.39, 0.45]
0.36748
(0.34, 0.4]
(-0.01, 0.11]
8.09387c-02
gold response
0.22679
0.25)
0.54325
(0.51. 0.58]
(-0.37, -0.27
3.26963e-23
httpp:://ww..... bmj
http://wwww...bcccccccaatt/t//// wccmm//onten//343/mmm. .2304
19

---

Under review as a conference paper at ICLR 2021
Human Evaluation:
Which response more humanlike?
pivot_wing wins
pivot_CI
empsys_ wins
empsys_co CI
pairwise_ce CI
emp_sys
GPT2
0.45084
(0.42, 0.48]
0.32636
(0.3, 0.36]
(0.07, 0.1
1.17277c-05
GPT2MMI
0.61734
(0.57, 0.66]
0.18393
(0.15, 0.22]
(0,36, 0.5]*
1.73160e-30
@PP2_NSP
0.43617
(0.41, 0.47]
0.35000
(0.32, 0.38)
(0.03, 0.14]*
2.92302e-03
InfoNCE
0.44630
(0.42, 0.48]
0.34515
(0.32, 0.38)
(0.04, 0.16]*
4.45383e-04
respooss
0.22164
(0.2, 0.25]
0.56608
(0.53, 0.6]
(-0.4, -0.29]*
9.29316e-28
Human Evaluation: Which response more interesting?
pivot_wins
pivot_CI
empsys_wins
empsys_co
pairwise_CC
emp_sys
GPT2
0.56157
(0.53, 0.59]
0.21444
(0.19, 0.24]
(0.3, 0.4]
2.13032e-36
GPT2MMI
0,68750
(0.65, 0.73]
0.12292
(0.09, 0.15]
(0.5, 0.63]*
6.66687e-63
GPT2 NSP
0,51931
(0.49, 0.55]
0.24571
(0.22, 0.27]
(0.22, 0.33J*
2.30585e-22
InfoNCE
0,41288
(0.38, 0.44]
0.33580
(0.31, 0.37]
(0.02, 0.13]*
5.84741e-03
gold_respont
0.32384
(0.29, 0.35]
0.46624
(0.44, 0.5]
(-0.2, +0.09]
1.08781e-03
20


Reviews:
[{"review_id": "C2R8VJHX6Mn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a contrastive learning approach where one of the views, x, is converted into two subviews, x' and x'', and then separate InfoNCE style bounds constructed for each of I(x'';y) and I(x';y|x'') before being combined to form an overall training objective.  Critically, the second of these is based on the conditional MI, I(x';y|x''), distinguishing it from previous work using multiple views that just take the marginal I(x';y).  Estimating this conditional MI transpires to be somewhat trickier due to the additional intractability from p(y|x''), with approximations suggested to get around this.  Experiments are performed on both vision and NLP problems.\n\n\n*Review Overview*\n\n\nI believe the premise of the paper is quite sensible and the general idea potentially useful, albeit somewhat incremental.  However, I am rather unconvinced by the approximations used to try and estimate p(y|x''); this rather lets the paper down as I feel this is the main technical challenge with the other contributions being rather straightforward.  Further, I feel the experimental results are somewhat weak, both in terms of the fact that the gains shown are quite small, but also because there are doubts as to whether these gains are actually statistically significant at all or just a somewhat \"engineered\" win that does not accurately reflect the underlying approaches.\n\n\n*Strengths*\n\n+ As far as I'm aware, the suggested approach has not appeared in the literature before.\n+ This is a highly active research area that is very much of interest to the ICLR community.\n+ The high-level idea of using subviews and breaking down the MI in this way seems sensible and potentially useful.\n+ The toy experiment in Figure 2 is helpful in showing that the approach can be helpful if p(y|x'') can be estimated well.\n+ It is good to see experiments on both vision and NLP problems.\n+ The paper is mostly quite easy to follow.\n\n\n*Weaknesses (in roughly decreasing order of importance)*\n- The experiments do not follow procedures that are clearly statistically sound and the gains could easily just be a result of luck and /or tweaking the approach until it works.  In particular, no error bars are provided for any of the results; different approaches are run for different problems without justification such that it feels like things are somewhat cherry-picked; additional heuristics are added to what is actually run without clear a priori motivation such that they feel rather based on trial and error and undermines whether the gains are actually from the suggested approach or just engineering the exact setup until it gives an improvement; some important details seem to be missing such as how lambda is chosen. \n- The experimental gains are rather modest.\n- The approximation methods for p(y|x'') are rather unconvincing: the variational bound requires a very complex learning problem in itself to encapsulate p(y|x'') while permitting optima that do not produce approximations that are close to this target at all, while the importance sampling approach is very much just a heuristic with rather flimsy justification.  More explanation on this is given later.\n- Much of the technical content, other than how one approximates p(y|x''), is very straightforward and is not enough to warrant publication on its own.  For example, the first time anything new is introduced is not until section 3.1 at the bottom of page 4, while the conditional InfoNCE bound/proposition 1 is trivial and arguably not even new to the literature (for example, equivalent bounds are already employed in conditional MI estimation in a Bayesian experimental design context, see e.g. http://proceedings.mlr.press/v108/foster20a/foster20a.pdf).\n- Though not bad, the writing of the paper could definitely be improved.  In particular, the abstract and introduction could be much easier to follow and have a clearer narrative, while the initial explanations of what x, y, and the sub-views x' and x'' represent could be much clearer.\n- [Minor] I'm not a fan of the title.  \"Decomposing MI\" suggests you are making manipulations to the standard MI to ease estimation or focus on different things, rather than introducing additional views and then introducing separate bounds for each (the approach isn't even a true decomposition as its a bound on the original MI, not an equality).\n\n\n*Approximations*\n\n\nAs previously mentioned, one of my biggest misgivings is with the approximations employed to try and draw samples from p(y|x'').  This is at the core of the problem and arguably why most papers have steered clear of dealing with conditional MI directly.\n\nStarting with the variational approach, I have two issues:\n- Estimating p(y|x'') in this way is extremely difficult and is almost like learning a decoder in that we have to learn how to generate full data examples.  Though this is far from fatal in the context of the overall approach, it feels like if we are able to learn a good approximation for p(y|x''), we have already solved much of the challenge contrastive learning was trying to address in the first place.\n- More immediately worryingly, the resulting objective introduced (Eq 10) does not appear to guarantee that an effective contrastive learning scheme will be learned as there are unwanted ways this objective can be optimized.  \n\nTo be more precise about the latter point, the paper claims about Eq 10\n\n\"This suggests that using to sample contrastive sets retains the properties of being a lower bound on the conditional MI.\"\n\nThis is a strange phrase and not properly quantified: my interpretation of Eq (10) would be that the KL term being positive means using a variational approximation does not produce a valid lower bound in any useful sense.  In particular, because \\tau appears in both terms, it seems perfectly possible for there to be optima of Eq (10) that do not correspond to \\tau(y|x'') = p(y|x'') and consequently critics E that do not behave in the desired manner.  Essentially, it seems like the approach, at least in principle, can learn a tau that allows for easy discrimination to the contrastive samples rather than which represents the actual distribution; the first term could potentially just dominate the KL divergence (such a phenomenon is well documented in VAEs) and avoid learning anything useful.\n\nThis is not the end of the world if it does actually train properly in practice, but I think this needs to be directly demonstrated and the potential pitfalls that are there discussed.\n\nOn the other hand, with the \"importance sampling\" approach, there are many large and not fully justified jumps being made such that the final objective used (i.e. Eq 11) is more of an importance sampling inspired heuristic than any principled importance sampling estimate.  There are a lot of approximations going on here, from the weights themselves (they are only valid weights if E=E* and its not clear they will give anything particularly sensible when E\\neq E*)  to the rather unjustified jump from these to Eq 11.  In particular, deriving Eq 11 from the weights somewhat boils down to making the assumption E[1/X] = 1/E[X] which is typically a poor approximation (though interestingly might become exact in the limit K->\\inf). In short, I would suggest that the approach Eq (11) requires far more careful justification for it to be convincing that this is an approach one would a priori expect to behave in the intended manner.  Some sort of ablation study into how reasonable the approximation it provides is would also be extremely helpful (this could also be done for the variational approach).\n\n\n*Other Specific Comments *\n\n\n1. It would be good to explain what you mean by \"hard\" contrastive samples in the intro.\n\n2. The sentence \n\n\"We note that the bound (Oord et al., 2018) corresponds InfoNCE to a particular choice for the variational distribution q.\"\n\nis not quite true: the InfoNCE bound can be viewed as a lower bound on Eq (2) (not Eq (2) itself) for a particular choice of q as it still requires an extra invocation of Jensen's inequality.\n\n\n*Questions etc *\n\n\na. Is a stop gradient is used with the w_k to stop E from being indirectly trained through these?\n\nb. I would like to see the results of using the variational approach with the vision problem and the importance sampling approach with the NLP problem, or at the very least a justification for why this hasn't been done.  Without this the results feel a little cherry-picked.\n\nc. What is the performance you get when setting lambda=0?  This was the originally proposed method and, presumably, experiments were run on this so it is important to include them in the results, both so we can see how important the extra heuristic it conveys actually is, and because it relates to the statistical interpretation of the results (tuning lambda reduces the significance of the results as its an extra degree of freedom the baselines don't have).  Moreover, how was lambda actually chosen?  This seems to be missing from the paper.\n\nd. The experimental results are pretty meaningless without any sort of error bars.  The gains are quite small and it impossible to know at present whether they are the result of pure chance, or actual statistically significant improvements.  I would like to see repeat runs and the variability in the performance across these.\n\ne. Are the same adjustments used for the dialogue experiment as for the vision experiment?  In general, section 4.3 is quite hard to follow and seems to lack complete details for the exact setup used for the suggested approach (section 4.2 still has a few things missing as well like setting lambda).\n\nf. What is done with the bootstrap confidence intervals?  This was difficult to understand from the shortness of the explanation, while it is rather unsatisfactory to just quote that the results were significant at the 5% level rather than actually providing the variations, p values, and exact details of the test being done.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8Ej6x4zev3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes to decompose mutual information estimation into subtasks, based on the chain rule of MI, i.e. Eq.(6).\nA novel bound for the proposed decomposition insight is achieved based on the classic InforNCE bound. Higher total information is expected to be captured with the proposed method. \n\n\nThe proposed insight is understandable and reasonable to achieve the desired goal. it is\n also presented to be effective to achieve higher MI and results in better representations for classification. \nI have one question regarding this work.\n\n1) Work [1]  also presents analysis on estimating mutual information in decomposing, or hierarchy, manner, i.e. Eq(19) in its paper. Can you discuss this work with yours? Can I understand your work as a variant of this work performed with InforNCE estimation? I appreciate your opinion here. \n\n\n[1] Gao, Shuyang, et al. \"Auto-encoding total correlation explanation.\" The 22nd International Conference on Artificial Intelligence and Statistics. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4TjMm4peOsB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposed a lower-bound on the mutual information by introducing a conditional mutual information term.\nThe authors claim that their conditional mutual information term can avoid capturing redundant information shared among different data samples and therefore is more efficient.\n\n\nReason for the score:\n\nI think the authors could improve their paper by comparing with more methodologies such as the information bottleneck (IB), and various approximations such as MINE from ICLR 2018 (https://openreview.net/pdf?id=rJHOuiqaf). \n\n\n\nPro:\n- The authors provided experiments in vision, dialogue, and synthetic datasets on different models.\n\nCon/Questions:\n- Some experimental setups are not too clear. For example, how did the authors choose information-restricted views?\n- I think the authors could compare their method with other relevant information theoretic methods such as the Information Bottleneck (IB). Many recently applications in vision, language, and molecules found IB very useful in balancing between compression and information retention.\n- I think the paper is a bit difficult to follow in its current format particularly with its current choices of words. For example, the authors use both $x, y$ to denote data features instead of data features and data labels which is quite uncommon. \"views\" is also used frequently in this paper which I can only assume that the authors meant \"data samples\".\n\n-----------------------------------------------------------------\nPost Rebuttal:\n\nMany thanks for the authors to update their original paper addressing some of my questions and concerns.\nI have now updated the score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a learning approach based on mutual information maximization. By considering a view x, and two subviews, x’ and x’’, the authors provide a bound on MI by combining two InfoNCE-like bounds on I(x’’; y) and I(x’; y | x’’). The authors show that optimising this (approximate) bound leads to improvements in several tasks covering NLP and vision.

This paper is aiming to address a significant problem for the ICLR community and provide a novel solution. The manuscript is well written and the main idea is clear. The reviewers appreciated the fact that the experimental setup covers both vision and NLP. On the negative side, the reviewers raised several major issues, both with the presented theory and the experimental setup. From the theoretical point of view, the approach hinges on a good  approximation for p(y|x''), which could arguably be as hard as the original problem. The author's response is definitely a step in the right direction, but the changes to the original manuscript are quite substantial and there is no time for a thorough validation of the updated claims. I will hence recommend rejection and strongly suggest that the authors incorporate the reviewers’ feedback and submit the manuscript to a future venue.
Decision: Reject