KNIFE: Kernelized-Neural Differential Entropy Estimation
Authors: Georg Pichler
Pierre Colombo
Malik Boudiaf
Günther Koliander
Pablo Piantanida
Venue: ICLR 2022 Submitted (2022)
Keywords: differential entropy estimation
differential entropy
mutual information
kernel estimation

Abstract: Estimation of (differential) entropy and the related mutual information has been pursued with significant efforts by the machine learning community. To address shortcomings in previously proposed estimators for differential entropy, here we introduce KNIFE, a fully parameterized, differentiable kernel-based estimator of differential entropy. The flexibility of our approach also allows us to construct KNIFE-based estimators for conditional (on either discrete or continuous variables) differential entropy, as well as mutual information. We empirically validate our method on high-dimensional synthetic data and further apply it to guide the training of neural networks for real-world tasks. Our experiments on a large variety of tasks, including visual domain adaptation, textual fair classification, and textual fine-tuning demonstrate the effectiveness of KNIFE-based estimation.

Under review as a conference paper at ICLR 2022
DIFFERENTIAA KNIFE: KERNELIZED-NEURAL DIFFERENTIAL ENTROPY ESTIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Estimation of (differential) entropy and the related mutual information has been pursued with significant efforts by the machine learning community. To address shortcomings in previously proposed estimators for differential entropy, here we introduce KNIFF, fully parameterized, differentiable kernel-based estimator of differential entropy. The flexibilityy of our approach also allows us to construct KNIFE-based estimators for conditional (on either discrete or continuous variables) differential entropy, well mutual information. We empirically validate our method on high-dimensiont synthetic data and further apply it to guide the training of neural networks for tasks, including visual domain adaptation, textual fair classification, and textual fine-tuning demonstrate the effectiveness of KNIFE-based estimation.
tasks. Our experiments
large variety of
INTRODUCTION
Learning tasks requires information (Principe et al., 2006) in the form of training data. Thus, information measures (Shannon, 1948) (e.g. entropy, conditional entropy and mutual information) have been a source of inspiration for the design of learning objectives in modern machine learning (ML) models (Linsker, 1989; Torkkola, 2006). Over the years, a plethora of estimators have been introduced estimate the value of the aforementioned measures of information and they have been applied to many different problems, including information and coding theory, limiting distributions, model selection, design of experiment and optimal prior distribution, data disclosure, and relative importance of predictors (Ebrahimi et al., 2010). In these applications, traditional research focused on both developing new estimators and obtaining provable guarantees on the asymptotic behavior of these estimators (Liu et al., 2012; Verdú, 2019).
However, when used for training deep neural networks, additional requirements need to be satisfied. In particular, the estimator needs to be differentiable w.r.t. the data distribution (R1), computationally tractable (R2), and rapidly adapt to changes in the underlying distribution (R3). For instance, Mutual Information (MI). fundamental measure of dependence between variables, only became a popular (standalone or regularizing) learning objective for DNNs once estimators satisfying the above requirements were proposed (Poole et al., 2019; Barber & Agakov, 2003). Although MI is notoriously difficult to estimate in high dimensions (Kraskov et al., 2004; Pichler et al., 2020; McAllester & Stratos, 2020), these estimators have demonstrated promising empirical results in unsupervised representation learning (Krause et al., 2010; Bridle et al., 1992; Hjelm et al., 2019; Tschannen et al., 2020), discrete/invarian representations (Hu et al., 2017: Ji et al., 2019), generative modelling (Chen et al., 2016; Zhao et al., 2017), textual disentangling (Cheng et al., 2020b; Colombo al., 202 1), and applications of the Information Bottleneck (IB) method (Mahabadi et al., 2021; Devlin et al., 2018 8; Alemi et al., 2016) 6) among others. Compared to MI, Differential Entropy (DE) has received less attention from the MI. community while also having interesting applications.
In this paper, we focus on the problem of DE estimation as this quantity naturally appears in many applications (e.g. reinforcement learning (Shyam et al., 2019; Hazan et al., 2019; Ahmed et al., 2019; Kim et al., 2019), IB (Alemi et al., 2016), mode collapse (Belghazi et al., 2018)). Traditional estimators of DE often violate at least one of the requirements (R1)- (R3) listed above (e.g. k- nearest neighbor based estimators violate (R1)). As consequence, the absence of DE estimator for arbitrary data distributions forces deep learning researchers to either restrict themselves to special cases where closed-form expressions for DE are available (Shyam et al., 2019) use MI proxy

---

Under review as a conference paper at ICLR 2022
(Belghazi et al., 2018). In this work, we introduce Kernelized Neural d[Fferential Entropy (KNIFE) estimator, that satisfies the aforementioned requirements and addresses limitations of existing DE estimators (Schraudolph, 2004; McAllester & Stratos, 2020). Stemming from recent theoretical insights (McAllester & Stratos, 2020) that justify the use of DE estimators as building blocks to better estimate MI, we further apply KNIFF MI estimation. In the context of deep neural networks with high dimensional data (e.g. image, text), KNIFE achieves competitive empirical results in applications where DE MI is required.
1.1 CONTRIBUTIONS
Our work advances methods in DE and MI estimation in several ways.
1. We showcase limitation of the existing DE estimators proposed in Schraudolph (2004); McAllester & Stratos (2020) with respect to desirable properties required for training deep neural networks. To address these shortcomings, we introduce KNIFE, fully learnable kernel-based estimator of DE. The flexibility of KNIFE allows us to construct KNIFE-based estimators for conditional DE, conditioning on either discrete or continuous random variable.
2. We prove learnability under natural conditions on the underlying probability distribution. By requiring fixed Lipschitz condition and bounded support we are not only able to provide an asymptotic result, but also confidence bound in the case of a finite training set. This extends the consistency result by Ahmad & Lin (1976).
3. We validate on synthetic datasets (including multi-modal, non- Gaussian distributions), that KNIFE addresses the identified limitations and outperforms existing methods on both DE and MI estimation. In particular, KNIFE more rapidly adapts to changes in the underlying data distribution.
We conduct extensive experiments on natural datasets (including text and images) to compare KNIFF--bsedd MI estimators to most recent MI estimators. First, we apply KNIFE in the IB principle to fine-tune a pretrained language model. Using KNIFE, we leverage a closed-form expression of a part of the training objective and achieve the best scores among competing MI estimators. Second, on fair textual classification, the KNIFE- based MI estimator achieves near perfect disentanglement (with respect to the private, discrete label) at virtually degradation of accuracy in the main task. Lastly, in the challenging scenario of visual domain adaptation, where both variables are continuous, KNIFF-bbsed MI estimation also achieves superior results.
1.2 EXISTENT METHODS AND RELATED WORKS
Existing methods for estimating DE fit into one of three categories (Beirlant et al., 1997; Hlavakovv-SShhidlee et al., 2007; Verdú, 2019): plug-in estimates (Ahmad & Lin, 1976; Györfi & Van der Meulen, 1987), estimates based on sample-spacings (Tarasenko, 1968), and estimates based on nearest neighbor distances (Kozachenko & Leonenko, 1987; Tsybakov & Van der Meulen, 1996); (Berrett et al., 2019). Our proposed estimator falls into the first category and we will thus focus here on previous work using that methodology. Excellent summaries of all the available methods can be found in the works (Beirlant et al., 1997:
DE estimation.
et al.,
2007; Wang et al., 2009; Verdú, 2019). In Ahmad & Lin (1976), a first nonparametric estimator of DE was suggested and theoretically analyzed. It builds on the idea of kernel density estimation using Parzen-Rosenbles
windowing (Rosenblatt, 1956; Parzen, 1962). More detailed analysis followed (Joe, 1989; Hall & Morton, 1993) but the estimator remained essentially unchanged. Unfortunately, this classical literature is mostly concerned with appropriate regularity conditions that guarantee asymptotic properties of estimators, such (asymptotic) unbiasedness and consistency. Machine learning applications, however, usually deal with a fixed-often very limited -number of samples.
Differentiable DE estimation. A first estimator that employed a differential learning rule was introduced in Viola et al. (1996). Indeed, the estimator proposed therein is optimized using stochastic optimization, it only used a single kernel with a low number of parameters. An extension that uses a heteroscedastic kernel density estimate, i.e., using different kernels at different positions, has been proposed in Schraudolph (2004). Still the number of parameters was quite low and vary- ing means in the kernels or variable weights were not considered. Although the estimation of DE remained a topic of major interest as illustrated by recent works focusing on special classes of distributions (Kolchinsky & Tracey, 2017; Chaubey & Vu, 2021) and nonparametrie estima- tors (Sricharan et al., 2013; Kandasamy et al., 2015; Moon et al., 2021), the estimator introduced in Schraudolph (2004) not further refined and hardly explored in recent works.
2

---

Under review as a conference paper at ICLR 2022
Differentiable MI estimation. In contrast, there has been recent surge new methods for the estimation of the closely related MI between two random variables. The most prominent examples include unnormalized energy-based variational lower bounds (Poole et al.. 2019). the lower bounds developed in Nguyen et al. (2010) using variational characterization of f-divergence, the MINE- estimator developed in Belghazi et al. (2018) from the Donsker- Varadhan representation of MI which can be also interpreted as an improvement of the plug-in estimator of Suzuki et al. (2008), the noise-contrastive based bound developed in van den Oord al. (2018) and finally contrastive upper bound (Cheng et al., 2020a). McAllester & Stratos (2020) point out shortcomings in other estimation strategies and introduce their own Differences of Entropies (DoE) method.
KNIFE
In this section identify limitations of existing entropy estimators introduced in Schraudolph (2004); McAllester & Stratos (2020). Subsequently, we present KNIFE, which addresses these shortcomings.
2.1 LIMITATIONS OF EXISTING DIFFERENTIAL ENTROPY ESTIMATORS
Consider a continuous random vector X p in R. Our goal is estimate the DE h(X) - p(z) p(r) dz. Given the intractability of this integral, we will rely on Monte-Carlo estimate of h(X), using N i..d. samples D₂ { to obtain
1 Σ log p(xn).
LORACLE (Dx) =
(I)
Unfortunately, assuming access to the true density is often unrealistic, and we will thus construct an estimate á that can then be plugged into (1) instead of p. If p is smooth, the resulting plug-in estimator of DE is differentiable (R1).
Assuming access to an additional -ddallllll-dealllyy independent set of M i.idd samples fa/m 1MM we build upon the Parzen- Rosenblatt estimator (Rosenblatt, 1956; Parzen, 1962)
1
p(a; w. E)
K
and M
W
where w > denotes the bandwidth and k is a kernel density. The resulting entropy estimator when replacing p in (1) by (2) was analyzed in Ahmad & Lin (1976). In Schraudolph (2004), this approach
was extended using the kernel estimator
PSCHRAU (I;
1 M
Σ
KAm. (x s'm))
(3)
where A == (Ai...., As) are (distinct, diagonal) covariance matrices and ( = (2;; 0, A) is a centered Gaussian density with covariance matrix A.
The DoE method of McAllester & Stratos (2020) is MI estimator that separately estimates DE and conditional DE. For DE, a simple Gaussian density estimate PDO (a;; 0) = KA 11) is used, where 0 = (A, n4) are the training parameters, the diagonal covariance matrix A and the mean μ..
While both SCHRAU. and DoE yield differentiable plug-in estimators for DE, they each have major disadvantage. The strategy of Schraudolph (2004) fixes the kernel mean values at E, which implies that the method cannot adapt to a shifting input distribution (R3). On the other hand, DOE allows for rapid adaptation, but its simple structure makes it inadequate for the DE estimation of multi-modal densities. We illustrate these limitations in Section 3.
2.2
KNIFE ESTIMATOR
In KNIFE, the kernel density estimate is given by M PKNIE (I; 0) = Σ
Um Am (x am):
(4)
m=l
3

---

Under review as a conference paper at ICLR 2022
where 0 == (A, a, u) and the additional parameters 0 S (U1, Ug..... uM)) with u 1 and a = (aj..... a.) are introduced. Note that PKNNE (2; 0) is a smooth function 0, and so is our proposed plug-in estimator
IKNIFE (Dx:0) = N
Σ
log PKNIFE (Z,;; 0)-
(5)
a=1
KNIFE combines the ideas of Schraudolph (2004); McAllester & Stratos (2020). It is differentiable and able to adapt to shifting imput distributions, while capable of matching multi-modal distributions. Thus, as we will see in synthetic experiments, incorporating Um and shifts am in the optimization enables the use of KNIFE in non-stationary settings, where the distribution of X evolves over time.
Learning step:
Stemming from the observation that, by the Law of Large Numbers (LLN),
fIKNIr (Dx, 0)
LLN
-E [oog 0) h(X) + DKL (pllkksurd (- ()) 2 h(X),
(6)
we propose to learn the parameters by minimizing lINNIFE. where & may be used to initialize a. Although not strictly equivalent due to the Monte-Carlo approximation, minimizing lINIFFE can be understood minimizing the Kullback-Leeibe between hRSIFR and h(X). In fact, lięNие can also be interpreted as the standard maximum likelihood objective, widely used in modern machine learning. It is worth to mention that the KNIFE estimator fully differentiable with respect to O and the optimization can be tackled by any gradient-based method (e.g., Adam (Kingma & Ba, 2014) AdamW (Loshchilov & Hutter, 2017)).
(KI.) divergence in (6), effectively minimizing the gap
2.3 CONVERGENCE ANALYSIS
Note that the classical Parzen-Rosenblatt estimator (Dxx w), where (2) is plugged into (I), is a special case of KNIFE. Thus, the convergence analysis provided in (Ahmad & Lin, 1976, Theorem 1)
also applies and yields sufficient conditions for hiNuree (Dx: 0) h(X). In Appendix C, we extend
this result and, assuming that the underlying distribution is compactly supported on X [0, 1]d and L-Lipschitz continuous, the following theorem is proved.
Theorem 1. For any 0, there exists a function e(N, M, w) such that, with probability at least
1 1-8, [Dxx; w) h(X) I =(NN M, w). Additionally, e M, w) - 0 as M.N oc and w + 0 provided that
N log N 2n
Nw +
and
0,
where M and N denote the number of samples in E and Dx, respectively.
The precise assumptions for Theorem and an explicit formula for ((N, M, w) are given in Theorem in Appendix C. For instance, Theorem provides bound on the speed of convergence for the consistency amalysis in (Ahmad & Lin, 1976, Theorem 1).
2.4 ESTIMATING CONDITIONAL DIFFERENTIAL ENTROPY AND MUTCAL INFORMATION
Similar to (McAllester & Stratos, 2020), the proposed DE estimator can be used to estimate other information measures. In particular, we can use KNIFE to construct estimators of conditional DE and MI. When estimating the conditional DE and MI for a pair of random variables (X, Y) p. we not only use D = {æ,, IN drawn according to p.
n= but also the according i.i.d. samples D,- [ where (Ias Dn.) are
Conditional Differential Entropy. We estimate conditional DE h(X YY by considering 0 to be parameterized function (y)) of y. Then all relations previously established naturally generalize and
(vnlvm 1 Σ log PKNIFE Iyu +)-
PREMFE (z) yy: θ PKNIFF (v)), TIKNIFE (D DD, θ =
(8)
Naturally, minimization of (6) now performed over the parameters of @ If Y is a continuous random variable, E y is a discrete random variable, we have one parameter 0 for each y E y, i.e., O (0,lvey and PRNIFE (aly;; +) PRNIFE (x;; (v)) PXNIF (x; 0y))
artificial neural network @yy) taking y as its input. On the other hand, if
4

---

Under review as a conference paper at ICLR 2022
Mutual Information. To estimate the MI between random variables X and Y (either discrete or continuous), recall that MI can be written as [[X; Y) = h(X)= h(X Therefore, we use the marginal and conditional DE estimators (5) and (8) to build KNIFE- MI estimator
T (Dx, Dy; + TINNIFE ("xx; [IxNIEe +).
EXPERIMENTS USING SYNTHETIC DATA
3.1 DIFFERENTIAL ENTROPY ESTIMATION
In this section we apply KNIFE for DE estimation, comparing to (3), the method introduced in Schraudolph (2004), subsequently labeled "SCHRaU.". It worth mention that we did not perform the Expectation Maximization algorithm, as suggested in (Schraudolph, 2004), but instead opted use the same optimization technique for KNIFE to facilitate a fair comparison.
3.1.1 GAUSSIAN DISTRIBUTION
As a sanity check, we test KNIFE on multivariate normal data in moderately high dimensions, comparing SCHRAU, and DoE, which we trained with the exact same parameters. We performed these experiments with d = 10 and d = 64 dimensional data. KNIFE yielded the lowest bias and variance both cases, despite DOE being perfectly adapted to matching a multivariate Gaussian distribution. Additional details can be found in Appendix A.I.
In order to use a DE estimation primitive in a machine learning system, must be able to adapt to changing input distribution during training (R3). As already pointed out in Section 2.1, this is severe limitation of SCHRAU., as re-drawing the kernel support can be either impractical or at the very least requires a complete re-training of the entropy estimator. Whereas in (4), the kernel support a trainable and can thus adapt to a change of the input distribution. order to showcase this ability, utilize the approach of Cheng al. (2020a) and successively decrease the entropy, observing how the estimator adapts. We perform this experiment with data of dimension d = 64 and repeatedly multiply the covariance matrix of the training vectors with factor of 1 The resulting entropy estimation is depicted in Figure 1. It is apparent that SCHRAU. suffers from a varying bias. The bias increases with decreasing variance, as the kernel support fixed and cannot adapt as the variance of Dz shrinks. DoE is perfectly adapted to single Gaussian distribution and performs similar KNIFE.
Differential Entropy
140
KNIFE
0.8
KNIFE
Differential Entropy
20
KNIFE
120
Schrau.
Schrau.
15
Schrau.
100
DoE
DoE
DoE
W 0.5
80
10
60
0.3
40
20
1250
2500
3750
5000
10
0
10000 20000
30000
40000
Iterations
Iterations
Figure 2: Left: PDF when estimating DE of triangle mixture in dimension. Right: Training run when estimating DE of 2 2-component triangle mixture dimensions.
Estimating DE of Gaussian data with decreasing variance.
Figure 1:
3.1.2 TRIANGLE MIXTURE
KNIFE is able to cope with distributions that have multiple modes. While (3) is also capable of matching multi-moda distributions, DoE is unable to do so, as it approximates any distribution with multivariate Gaussian. We illustrate this by matching a mixture of randomly drawn triangle distributions. The resulting estimated PDFs as well as the ground truth when estimating the entropy of I-dimensional mixture of triangles with 10 components can be observed in Figure (left). With increasing dimension the difficulty of this estimation rises quickly as dimensions, the resulting PDF of independent c-mmponent triangle mixtures has cd modes. To showcase the performance of KNIFE in this challenging task, we ran 10 training runs for DE estimation of 2-component triangle mixtures in 8 dimensions. An example training run is depicted in Figure 2 (right).
5

---

Under review as a conference paper at ICLR 2022
CLUB
DOE
InfoNCE
MINE
NWJ
KNIFE
5kk
10
15 20
5K 10 15
20k
5K
10. 15 Steps
20
10 5k
15 Steps
20
5k
15k Steps
10
20
10k 15K 20 Steps
Steps
Steps
CLUB
DOE
InfoNCE
MINE
NWJ
KNIFE
16
10
5
16 Steps
20 Steps
5k Steps
15 Steps
Steps
Steps
CLUB
DOE
InfoNCE
MINE
NWJ
KNIFE
6k 10 15%
20
6K 10 16
20
5k 10 16
20
0
5k 10 16k
20
0
5k 16 15k
20
10 15k 20
Steps
Steps
Steps
Steps
Steps
Steps
Figure 3: Top: Estimation of I(X"; Yd), where (X, Y) are multivariate Gaussian with correlation coefficient P in the i-th epoch and d 20. Middle: Estimation of I d))) Bottom: Estimation of 1(x4; yd) for uniform (X, E)) and Y p.X + vI p E the i-th epoch.
3.2 MUTUAL INFORMATION ESTIMATION
Multivariate Gauss We repeat the experiments in (Cheng al., 2020a), stepping up the MI T( T("1; between i.i.d. copies of joint normal random variables Y) by increasing their correlation coefficient, i.c., (X, Y) are multivariate Gaussian with correlation coefficient p; in the i-th epoch. A training run is depicted in the top of Figure 3. As in (Cheng et al., 2020a), we also repeat the experiment, applying cubic transformation Y. The estimation of MI between i.i.d. copies of X and y can be observed in the middle row of Figure 3. The MI is unaffected by this bijective transformation. In Appendix A.3, the bias and variance are depicted separately.
Sum of Uniformly Distributed Variables In order to the ability of KNIFE to adapt to distribu- tions substantially different from the Gaussian kernel shape, we apply it in MI estimation of I(x4; yd) with uniformly distributed data. To this end, let X and E be centered, uniformly distributed random variables with E F and define Y p; + / / the i-th epoch. One training run with d = 20 is shown in Figure (bottom). Details about the source distribution well as details of the experiments can be found in Appendix A.3.
4 EXPERIMENTS ON NATURAL DATA
In this section, we benchmark our proposed KNIFE- based MI estimator on three practical applications, spanning textual and visual data. We reproduce and compare our method the most recent MI estimators including MINE (Belghazi et al., 2018), 1), NWJ (Nguyen et al., 2010), InfoNCE (van den Oord et al., 2018), CLUB (Cheng al., 2020a), and DoE (McAllester Stratos, 2020). We do not explicitly include the SMILE estimator Song & Ermon (2019) in our comparison as it has the same gradient as NWJ.
Common notation: In all following applications, will use Φ X' Z to denote encoder, where X' the raw input space (i.e., texts or images), and denotes a lower dimensional continuous will use C Z - ) to denote shallow classifier from the latent
feature space. Additionally, space Z to a discrete or continuous target space y for classification or regression, respectively. We will use to denote the parameters of both models, ax and Cø. CE denotes the cross entropy loss.

---

Under review as a conference paper at ICLR 2022
Table Fine-tuning on GLUE. Following (Lee al., 2019; Dodge al., 2020), mean and variance are computed for 10 seeds.
MRPC
STS-B
RTE
F1
Accuracy
Pearson
Spearmann
Accuracy
BERT (Devlin et al., 2018)
83.4 +0.9
88.2
+0.7
89.2
:0.4
88.8
=0.4
69.4 =0.4
CLUB (Cheng et al., 2020s)
85.0 2:04
89.0
1:07
89.7
1002
89.4 20.1
70.7 20.1
InfoNCE (van den Oord et al., 2018)
84.9 +0.5
88.9
#0.6
89.4
+0.4
89.7 =0.6
70.6 =0.1
MINE (Belghazi al., 2018)
80.0 :2.5
85.0
10.9
88.0 +0.7
88.0
NWJ (Nguyen et al., 2010)
84.6 +0.5
88.1
+0.7
89.8 #:01
89.6
69.6 =0.7
VUB/VIBERT (Alemi et al,, 2016)
85.1 10.5
89.1
10.3
90.0
1.0.2
89.5 20.3
70.9 1001
DOE (McAllester Strutos, 2020)
84.1 4.02
88.3 1..2
89.6
1:0.2
89.5 =0.2
69.6 :0022
KNIFE
85.3
90.1
90.3
90.1
72.3
4.1 INFORMATION BOTTLENECK FOR LANGUAGE MODEL FINETUNING
IB has recently been applied to fine-tune large-scale pretrained models (Mahabadi et al., 2021) such as BERT (Devlin et al., 2018) and aims suppressing irrelevant features order reduce overfitting.
Problem statement. Given textual input X E X' and a target label Y E V, the goal to learn the encoder 2y and classifier Cz., such that & (XX) retains little information about X, while still producing discriminative features, allowing the prediction of Y. Thus, the loss of interest is:
X 111101gg (X):
1 (X):).
(10)
compression term
downstream term
where À controls the trade-off between the downstream and the compression terms.
Setup. Following Mahabadi et al. (2021) (relying on VUB), we work with the VIBERT model, which uses a Gaussian distribution as prior. L is implemented as a stochastic encoder & (x()) = Z ~ (X), (X)). Details the architecture of μ. and Ei can be found in Appendix B. The classifier Cpp is composed of dense layers. To minimize L. the second part of the objective (10)
bounded using the variational bound from Barber & Agakov (2003). Since we
Gaussian prior,
h(]lX) can be expressed in closed form.' Thus, when using KNIFE, I(X; Z) = h(Z) - h(2lX)
can be estimated by using hKvare to estimate h(2). We compare this KNIFE-based MI estimator with aforementioned MI estimators and the variational upper bound (VUB). For completeness, we also compare against a BERT model trained by direct minimization of CE loss.
We closely follow the protocol of (Mahabadi et al., 2021) and work on the GLUE benchmark (Wang et al., 2018) originally composed of 5 datasets. However, following (Mahabadi et al., 2021), choose to finetune neither on WNLI (Morgenstern & Ortiz, 2015) nor on CoLA (Warstadt et al., 2019) due reported flaws in these datasets. The evaluation is carried out the standard validation splits as the test splits are not available. Following standard practice (Liu et al., 2019; Yang et al., 2019), we report the accuracy and the Fl for MRPC, the accuracy for RTE and the Pearson and Spearman correlation coefficient for STS-B.
reports our results on the GLUE benchmark. We observe that KNIFE obtains the best results on all three datasets and the lowest variance on MRPC and STS-B. The use of a Gaussian prior in the stochastic encoder & could explain the observed improvement of KNIFE-based estimation over Ml-estimators such as CLUB, InfoNCE, MINE, DoE, or NWJ.
Results. Table
4.2 FAIR TEXTUAL CLASSIFICATION
In fair classification, we would like the model to take its decision without utilizing private information such gender, age, race. For this task, MI can be minimized disentangle the output of the encoder Z and a private label S E S (e.g., gender, age, or race).
"h(?)) In (X)) In(2we), where is the dimension of X and denotes the determinant.
7

---

Under review as a conference paper at ICLR 2022
0.68
0.675
Accuracy
Pfvvale Teek Apcurecy
0.M4
Main Task Aczuracy
0.50
Accuracy
0.65
Akk
0.75
0.60
MNU
CLUA
INNNCE
0.55
0.80
0.58
0.600
0,575
0.55
0.54
0.52
0.06
0.50
0.60
0.50
Randam
0.500
10
(a) Y (sentiment)
(b) S (sentiment)
(c) Y (mention)
(d) S (mention)
Figure 4: Results on the fair classification task for both main (Figures 4a and 4c) and private task
(Figures 4b and 4d) for both mention and sentiment labels. Results of MINE are not reported because of instabilities that prevent the network from converging. Figures 4b and 4d are obtained by training an offline classifier to recover the protected attribute S from 4 (X).
Problem Statement. Given input text X, discrete target label Y and private label S, the loss given by
C CE(Y; (X)) +X I( (X):
(11)
disentangled
where À controls the trade-off between minimizing MI and CE loss. In this framework, a classifier is said to be fair to achieve perfect privacy if no statistical information about S can be extracted from y (X) by an adversarial classifier. Overall, a good model should achieve high accuracy on the main task (i.e., prediction of Y) while removing information about the protected attribute S. This information is measured by training an offline classifier to recover the protected attribute S from
4 (X).
Setup. We compute the second term of (11) with competing MI estimators, as well as the model from Elazar & Goldberg (2018), which will be referred to as "Adv", as it utilizes an adversary to recover the private label from the latent representation Z. For KNIFE-based MI estimation, we use two DE estimators (as is a binary label), following the approach outlined in Section 2.4. All derivations detailed in Appendix B.
We follow the experimental setting from Elazar & Goldberg (2018); Barrett et al. (2019) and use two datasets from the DIAL corpus (Blodgett et al., 2016) (over 50 million tweets) where the protected attribute S is the race and the main labels are sentiment or mention labels. The mention label indicates whether a tweet is conversational or not. We follow the official split using 160 000 tweets for training and two additional sets composed of 10000 tweets each for development and testing. In all cases, the labels S and Y are binary and balanced, thus a random guess corresponds 50% accuracy.
Results. Figure 4 gathers results on the fair classification task. The upper dashed lines represent the (private and main) task accuracies when training model with only the CE loss (case À in (11)). This shows that the learned encoding 4 (X) contains information about the protected attribute, when training is only performed for the main task. On both the sentiment and mention task, achieve perfect privacy (see Figures 4b and 4d) with
observe that KNIFE-based estimator nearly no accuracy loss in the main task (see Figures 4a and 4c). The other MI estimators exhibit different behavior. For sentiment labels, most MI estimators fail reach perfect privacy (CLUB, NWJ, DOF, and Adv) while others (InfoNCE) achieve perfect privacy while degrading the main task accuracy (10% loss on main accuracy). For mention labels, CLUB can also reach perfect privacy with almost no degradation of the accuracy of the main task. Overall, it is worth noting that KNIFE-based MI estimation enables better control of the degree of disentanglement than the reported baselines.
4.3 UNSUPERVISED DOMAIN ADAPTATION
In unsupervised domain adaptation, the goal is to transfer knowledge from the source domain (S) with a potentially large number of labeled examples to target domain (I'), where only unlabeled examples are available.
Problem Statement. The learner is given access to labeled images from source domain (I.. y) ~ (Xs, Y) X's x y and unlabeled images from target domain St Xy E Xr. The goal is to
8

---

Under review as a conference paper at ICLR 2022
Table 2: Domain adaptation results: M (MNIST), MM (MNIST M). U (USPS), SV (SVHN), C (CIFAR10) and S (STLIO). Results are averaged over 3 seeds.
M + MM
S C
U-
M
M
M-U
U
C-+S
SV - M
Mean
Source only
51.9 10.88
58.3 ..2
91.1 10.7
93.5 10.6
72.3 10..5
54.7 12.8
70.3 10.9
CLUB
79.1 12.2
59.9 11.
96.0 10.2
96.8
10.5
71.6
11.
83.8 13.4
81.2
11.7
DOE
82.2 12.6
58,9
97.2
10.3
94.2
10.9
68.8
1..4
86.4 15.4
81.3
11.9
INFONCE
77.3 #0.5
61.0 =0.1
97.4
#0.2
97.0
#0.3
70.6
+0.8
89.2
#4.1
82.1
#1.0
MINE
76.7 #0.4
61.2 =0.3
97.7
+0.1
97.3
#0.1
70.8
110
91.8 +0.8
82.6
#0.4
NWJ
77.1 1:066
61.2 00.
97.6
1:0.1
97.3
#0.5
72.1
1.07
91.4
40.8
82.8 20.5
KNIFE
78.7 #0.7
61.8
0.5
97.7
+0.3
97.4
#0.4
71.2
1188
93.2
#0.2
83.4
#0.6
learn a classification model ( Cup) } that generalizes well to the target domain. Training models on the supervised source data only results in domain-specific latent representations Φ (X) leading to poor generalization (when X is chosen randomly from Xr)). In order to make the latent representations domain-agnostic as possible, we follow the information-theorett method proposed by Gholami et al. (2020), and used in Cheng et al. (2020a). The idea is to learn an additional binary model ( Cd), whose goal is guess the domain D E {0. 1} of X. The latent representation information that we would like the
learned by de will therefore contain all the main encoder 4ep to discard. In other words, we would like (X)) and (X) to completely disentangled, which naturally corresponds to the minimization of I( (X); & (X)). Concretely, the domain classifier is trained to minimize the CE between domain labels D and its predictions, whereas the main classifier is trained to properly classify support samples while minimizing the MI between 1 as (X) and (X). Using fo = Cø o D and fø == Cs pyp, the objectives are
min (E((:
and
min CE(Y; (Xs)) + A 1(2 (X); (X))
(12)
Setup. The different MI estimators compared based their ability guide training by estimating I( (X): (X)) in (12). We follow the setup of Cheng et al. (2020a) as closely as possible, and consider total of 6 source/target scenarios formed with MNIST (LeCun & Cortes, 2010), MNIST-M (Ganin et al., 2016), SVHN (Netzer et al., 201 1), CIFAR-10 (Krizhevsky et al., 2009), and STL-10 (Coates et al., 201 1) datasets. We reproduce all methods and allocate the same budget for hyper-paranan parameter tuning to every method. The exhaustive list of hyper- parameters can be found in Appendix B.
Results. Results are presented in Table 2. The KNIFE-based estimator is able to outperform MI estimators in this challenging scenario where both (X) and (X)) are continuous.
5 CONCLUDING REMARKS
We introduced KNIFE, a fully learnable, differentiable kernel-based estimator of differential entropy. designed for deep learning applications. We constructed a mutual information estimator based on KNIFE and showcased several applications. KNIFE is a general purpose estimator and does not require any special properties of the learning problem. It can thus be incorporated as part of any training objective, where differential entropy or mutual information estimation is desired. In the case of mutual information, one random variable may even be discrete.
Despite the fundamental challenges in the problem of differential entropy estimation, beyond limita- tions arising from the use of a finite number of samples, KNIFE has demonstrated promising empirical results in various representation learning tasks.
Future work will focus on improving the confidence bounds given in Theorem I. In particular, tailoring them towards KNIFE using tools from (Birge & Massart, 1995; Singh & Poczos, 2014). Another potential extension is direct estimation of the gradient of entropy, when PKXIFE (2; 0) has been learned (Mohamed et al., 2020; Song et al., 2020). This could be applied after the learning phase of KNIFF and is left for future work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
I. Ahmad and Pi-Erh Lin. A nonparametric estimation of the entropy for absolutely continuous distributions. IEEE Trans. Inf. Theory, 22())372--355, May 1976.
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. 97:151-160, 09-15 Jun 2019.
Alexander A. Alemi, Ian Fischer. Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. arXiv, abs/1612.00441 2016.
David Barber and Felix Agakov. The im algorithm: A variational approach to information maxi- mization. In Proceedings of the 16th International Conference on Neural Information Processing Systems, NIPS'03 03, pp. 201-208, Cambridge, MA, USA, 2003. MIT Press.
Maria Barrett, Yova Kementchedjhieva, Yanai Elazar, Desmond Elliott,, and Anders Søgaard. Ad- versarial removal of demographic attributes revisited. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6331-6336, 2019.
Jan Beirlant, Edward J. Dudewicz, László Györfi, and Edward C. Van der Meulen. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6(1):17-39, 1997.
Ishmael Belghazi, Sai Rajeswar, Aristide Baratin, R. Devon Hjelm, and Aaron C. Courville. MINE: mutual information neural estimation. arXiv, abs/ 0///1801.04066 2018.
Thomas B Berrett, Richard J Samworth, and Ming Yuan. Efficient multivariate entropy estimation via n-nearest neighbour distances. The Annals of Statistics, 47(1))288-311 -18, 2019.
Lucien Birge and Pascal Massart. Estimation of Integral Functionals of a Density. The Annals of Statisties, 231 29, 1995. doi: 10.1221/4aa 1e.111/4/o///11 176244452. URL https: //doi. .rgg 1214/ 117732247222
Su Lin Blodgett. Lisa Green, and Brendan O'Connor. Demographic dialectal variation in social media: case study of african-american english. arXiv, abs/1608 00886. 2016.
John Bridle, Anthony Heading, and David MacKay. Unsupervised classifiers, mutual information and 'phantom targets. J. Moody, S. Hanson, and R. P. Lippmann (eds.), Advances in Neural Information Processing Systems, volume 4. Morgan-Kauffaan Kaufmann, 1992.
Yogendra P Chaubey and Nhat Linh Vu. On the estimation of entropy for non-negative data. Journal of Statistical Theory and Practice, 15(2). 2021.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. pp. 2180-2188, 2016.
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. CLUB: A contrastive log-ratio upper bound of mutual information. 119, 13-18 Jul 2020a.
Pengyu Cheng, Martin Renqiang Min, Dinghan Shen, Christopher Malon, Yizhe Zhang, Yitong Li, and Lawrence Carin. Improving disentangled text representation learning with information- theoretic guidance. arXiv, abs/2006.00693, 2020b.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011.
Pierre Colombo, Chloe Clavel, and Pablo Piantanida. A novel estimator of mutual information for learning to disentangle textual representations. arXiv, abx/2105.02685, 2021.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv, abs/1810.04805, 2018.
10

---

Under review as a conference paper at ICLR 2022
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannanch Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv, abs/2002.06305, 2020.
Nader Ebrahimi, Ehsan S. Soofi, and Refik Soyer. Information measures in perspective. International Statistical Review, 78(3):383 412, 2010.
Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. arXiv, abs/1 808.06640, 2018.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1), 2016.
Behnam Gholami, Pritish Sahu, Ognjen Rudovic, Konstantinos Bousmalis, and Vladimir Pavlovic. Unsupervised multi-target domain adaptation: An information theoretic approach. IEEE Transac- fions Image Processing, 29, 2020.
László Györfi and Edward C. estimators of entropy. Computational Statistics Data Analysis, 5(4):425-436, 1987.
Van der Meulen. Density-free convergence properties of various
Peter Hall and Sally Morton. On the estimation of entropy. Annals of the Institute of Statistical Mathematics, 1993.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. 97:2681-2691, 09-15 Jun 2019.
Devon Hjelm, Alex Fedorov, Samuel Lavoie Mariie-arrchi,onn, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. 2019.
Katerina Hlaváčková SShhnddeer,, Milan Paluš, Martin Vejmelka, and Joydeep Bhattacharya. Causality detection based on information-thewte approaches in time series analysis. Physics Reports, 441 (1), 2007.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 1558 1567. PMLR, 06-11 Aug 2017.
Xu Ji, Joao Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Harry Joe. Estimation of entropy and other functionals of multivariate density, Annals of the Institute of Statistical Mathematics, 41(4)6683-697. 1989.
Kirthevasan Kandasamy, Akshay Krishnamurthy, Barnabas Poczos, Larry Wasserman, and james m robins. Nonparametric von mises estimators for entropies, divergences and mu- tual informations. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 28, Curran Asso- ciates, Inc., 2015. URL https: http://pr//eeee//// ngs neurips. cerrips.cee httppcccpppppeeeeeeeeeeeppppppep cccppaper/01////1 06138bc5a 662.364 6ede0e1 1711ea11111111a1111 pdf.
Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. EMI: exploration with mutual information. 97:3360-3369, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2014.
Artemy Kolchinsky and Brendan Tracey. Estimating mixture entropy with pairwise distances. Entmpy, 19(7), 2017.
11

---

Under review as a conference paper at ICLR 2022
LF Kozachenko and Nikolai N Leonenko. Sample estimate of the entropy of random vector. Problemy Peredachi Informatsii, 23(2)99-16, 1987.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Phys. Rev. E, 69, Jun 2004.
Andreas Krause, Pietro Perona, and Ryan Gomes. Discriminative clustering by regularized informa- tion maximization. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune
large-scale pretrained language models. arXiv, abs/1909. 1299, 2019.
Ralph Linsker. How generate ordered maps by maximizing the mutual information between input and output signals. In Neural Comput., 1989.
Han Liu, Larry Wasserman, and John Lafferty. Exponential concentration for mutual information estimation with application to forests. In F. Pereira, J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv, 0bs//997.1116 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv, abs/ 171110111111, 2017.
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. Variational information bottle- neck for effective low-resource fine-tuning. ICLR. OpenReview. net, 2021.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 875-884, PMLR, 26-28 Aug 2020.
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. J. Mach. Learn. Res., 21(112): 2020.
Kevin R. Moon, Kumar Sricharan, and Alfred O. Hero. Ensemble estimation of generalized mutual information with applications to genomies. IEEE Transactions on Information Theory, 67(9): 5963-5996, 2021. doi: 10. 10.1100//1T.22211311000
Lcora Morgenstern and Charles Ortiz. The winograd schema challenge: Evaluating progress in commonsense reasoning. In Seventh IAAI Conference, 2015.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 2010.
Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6), 2003.
Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathemat- ical statistics, 33(3):1065-100 1962.
12

---

Under review as a conference paper at ICLR 2022
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 2019.
Georg Pichler, Pablo Piantanida, and Günther Koliander. On the estimation of information measures of continuous distributions. arXiv, abs/2002.02851,, 2020.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceed- ings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5171-5180. PMLR, 09-15 Jun 2019.
Jose C Principe, Dongxin Xu, John Fisher, and Simon Haykin. Citeseer, 2006.
Information theoretic learning.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. arXiv preprint arXiv:2004.0766 2020.
Murray Rosenblatt, Remarks on some nonparametric estimates of a density function. Ann. Math. Statist., 27(3):832-837, 1956.
N. N. Schraudolph. Gradient-based manipulation of nonparametric entropy estimates. IEEE Transac- tions on Neural Networks, 15(4), 2004.
C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3): 379 373,423, July 1948.
Pranav Shyam, Wojeiech Jaskowski, and Faustino Gomez. Model-based active exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp- PMLR, 09-15 Jun 2019.
Shashank Singh and Barnabas Poczos. Exponential concentration of density functional estimator. In Z.. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Asso- ciates, Inc., 2014. URL https::////// pestts::/prooooo /tttpp::/prooeedingg .nerrie cc/papper/214/filee. 4ad.881444444444444444 Paper.pdl
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. arXiv, abs/ Abb//190.00622, 2019.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp. 574-554-550 584. PMLR, 2020.
Kumar Sricharan, Dennis Wei, and Alfred O. Hero. Ensemble estimators for multivariate entropy estimation. IEEE Transactions Information Theory, 59(7):4374 4388, 2013. doi: 10. 2013.2251456.
Taiji Suzuki, Masashi Sugiyama, Jun Sese, and Takafumi Kanamori. Approximating mutual informa- tion by maximum likelihood density ratio estimation. In New challenges for feature selection in data mining and knowledge discovery, pp. 5-20. PMLR, 2008
F.P. Tarasenko. On the evaluation of an unknown probability density function, the direct estimation of the entropy from independent observations of continuous random variable, and the distribution- free entropy test of goodness-of-fit Proceedings of the IEEE, 56(1 2)2152220222053, 1968.
Kari Torkkola. coammation-Thhorerte Methods in "Feature Extraction: Foundations and Applica- fions", pp. 167-185. Springer, Berlin, Heidelherg, 2006.
13

---

Under review as a conference paper at ICLR 2022
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. International Conference on Learning Representations, 2020.
Alexandre B. Tsybakov and E. C. Van der Meulen. Root-n consistent estimators of entropy for densities with unbounded support. Scandinavian Journal of Statistics, pp. 75-83, 1996.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv, abs/1 8071 007.03748, 2018.
Sergio Verdú. Empirical estimation of information measures: A literature guide. Entropy, 21(8), 2019.
Paul Viola, Nicol N Schraudolph, and Terrence J Sejnowski. Empirical entropy manipulation for real-world problems. Advances in neural information processing systems, 1996.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv, abs/1804 804.07461, 2018.
Qing Wang. Sanjeev R. Kulkarni, and Sergio Verdú. Universal estimation of information measures for analog sources. Foundations and Trends in Communications and Information Theory, 5(3): 265-353 2009.
Alex Warstadt, Amanpreet Singh, and Samuel Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv, abs/1910.03771, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoe Le. XInet: Generalized autoregressive pretraining for language understanding. arXiv, abs/1906.08237,, 2019.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Info Information maximizing variational autoencoders. arXiv, abs/l 70.62226, 2017.
14

---

Under review as a conference paper at ICLR 2022
APPENDIX
A EXPERIMENTAL DETAILS OF EXPERIMENTS WITH SYNTHETIC DATA
Implementation of KNIFE in PyTorch (Paszke et al., 2019) is rather straightforward. The constraint on the weights be satisfied by applying soft transformation. The covariance matrices were parameterized by the lower-triangular factor in the Cholesky decomposition of the precision matrices, guaranteeing the definiteness constraint to be satisfied.
A.1 DIFFERENTIAL ENTROPY ESTIMATION OF GAUSSIAN DATA
In Section 3.1.1, the estimation of the entropy h(X) 2 log 2 for X ~ (O, Ia) was performed with the hyperparameters given in Table The mean error and its empirical standard deviation are reported in Table 5 over 20 runs, where an independently drawn evaluation set with the same size as the training set is used. At d = 10 we have the entropy / = log 2Te- = 14.19, while for the higher dimension, d 64 we find h 90.81.
In the experiment depicted in Figure I, entropy is decreased after every epoch by letting X; ~ (0, a² Ia), where = ...... is the epoch index. That is, X; = Ga where G is standard normal random variable, resulting in an decrease of the DE by A = log = 22.18 for = with every epoch. We start h(X,)) = á' log 2nre 90.81 and successively decrease until h(Xa) h(Xa) + 4A = 2.1. Additional parameters can be found Table
Computational Resources. Training was performed on an NVidia V100 GPU, Taken together, training for the first experiments of entropy estimation in dimensions d = 10, 64, as well as the experiment depicted in Figure used GPU time of less than 5 minutes.
A.2 DIFFERENTIAL ENTROPY ESTIMATION OF TRIANGLE MIXTURES
In Section 3.1.2, perform an estimation of the entropy of c-component triangle mixture distribu- tions. The PDF of such a c-component triangle-mixture, is given by
x z -
p(z)=
(13)
where (x) == max 4s/x/} is centered triangle PDF with width 0. The scales s = ($),.... s.) and weights w (w)....s W.) satisfy 0 < the experiment, we choose w uniformly random from the e-probability simplex and the scales are chosen uniformly at random in [0.1, 1.0). An example for 10 is the true PDF depicted in Figure 2
< and Σ w; 1. Before
Table 4: Experimental details of the experiment depicted in Figure 1.
Table 3:
Experimental details of first experi-
ment in Section 1.1.1.
Parameter
Value
Parameter
Value
Source Distribution
X N10, for i O....,
Source Distribution X
X No Ia)
Dimension d
10 and 64
Dimension d
64
Optimizer
Adam
Factor a
Learning Rate
0.01
Optimizer
Adam
Batch Size N Kernel Size M Iterations per epoch
128
Learning Rate
0.01
128
Batch Size N Kernel Size M Iterations per epoch
128
200
128
Epochs Runs
1
1000
20
Epochs
Runs
15

---

Under review as a conference paper at ICLR 2022
Table 5: Results of first experiment in Section 3.1.1 with Gaussian data. We provide the average distance h Ti| and the empirical standard deviation. Experimental details are given Table 3.
d 10
d 64
DoE
0.8388 1.0045
3.3170 1.8281
SCHRAU.
0.7301 + 0.0428
9.8919 + 0.1604
KNIFE
0.0461 0.0139
2.8045 0.0796
Table 7: Experimental details of the experiment resulting in the training depicted Figure (right).
Table 6: Experimental details of the experiment resulting in the PDF in Figure (left).
Parameter
Value
Parameter
Value
Source Distribution X
c-component triangle mixtures
Source Distribution X
c-component
triangle mixtures
Components
10
Components
2
Dimension d
1
Dimension d
8
Optimizer
Adam
Optimizer
Adam
Learning Rate
0.1
Learning Rate Batch Size N Kernel Size M Iterations per epoch
0.001
Batch Size N Kernel Size M Iterations per epoch Epochs Runs
128
128
128
128
100
1000
10
Epochs Runs
20
1
10
(left). For d 1, we perform the estimation on d i.idd. copies. Note that the triangle mixture with components in d-dimensional space has cd modes, i.e., the support can be partitioned into ed disjoint components.
The parameters of the experiment yielding Figure (left) are given in Table 6, while the details the experiment depicted in Figure (right) can be found in Table 7. In the latter experiment, entropy was estimated to an accuracy of 1.6563 + 0.8528 by KNIFE, accurate to 2.4445 = 0.5439 using (3) and with an accuracy of 7.1070 + 2.7984 by DoE. This is the mean absolute error and its empirical standard deviation over all 10 runs, where the evaluation set was drawn independently from the training set and has the same size as the training set.
ten runs,
Computational Resources. Training was performed an NVidia V100 GPU. Training in d 1 dimension, that resulted in Figure (left) can be performed in seconds, while all training required for producing Figure (right) used approximately 1.5 hours GPU time.
A.3 MUTUAL INFORMATION ESTIMATION
In Section 3.2, we estimate T(Xd, yd) and I(X"; (Y33y)) where (X, Y) are multivariate correlated Gaussian distributions with correlation coefficient, O; the i-th epoch. Subsequently, we estimate I(xd;yd) where X, E ul-v3, v3] are independent and Y given by = P.X VI In both cases, Ps is chosen such that I(X4; yd) = 2i in the i-th epoch.
All neural networks are randomly initialized. The bias, variance, and MSE during training function of the MI, can observed Figure 5.
The estimation is performed in 10 runs, randomly choosing the training meta-parameters proposed by McAllester & Stratos (2020). Figure (bottom), we present the best rum for each method, selected by distance from the true MI at the end of training. The bias, variance, and MSE during training, as a function of the MI, can be observed in Figure 6. Details about the source distribution as well as details of the experiments be found in Table 8. During experimentation it turndd out be
16

---

Under review as a conference paper at ICLR 2022
Gaussian
Cubic
8
Bias
Bias
2
10
10a
10a
Variance
Variance
10a
10-1
10-1
10-2
10-2
2
2
8
100
CLUB
CLUB
60
DOE
80
DOE
InFoNCE
InfoNCE
MSE
40
MINE
MSE
60
MINE
NWJ
NWJ
30
KNIFE
KNIFE
10
2
8
MI Values
MI Values
Figure 5: Left: Estimation of I(X;Y); Right: Estimation of I(X; Y (cubic transformation).
17

---

Under review as a conference paper at ICLR 2022
Table 8: Experimental details of the training depicted in Figure 3 (bottom).
Parameter
Value
Dimension d
20
Optimizer
Adam
Learning Rates
0.01, 0.003, 0.001, 0.0003
Batch Size N Kernel Size M
128
128
Iterations per epoch Epochs Runs
20 000
10
beneficial to train the parameters O and in (9) separately and substantially increase the learning rate for the training 0. Thus, we increase the learning rate for the training by a factor of 10
Model Architecture for @ We utilize the feed-forward architecture, also used in McAllester & Stratos (2020). It is simple architecture with two linear layers, one hidden layer using tanh activation, immediately followed by an output layer. The number of neurons the hidden layer is meta-parameter selected randomly from (64, 1 128, 256} } for each training run. Three models with this architecture are used for the three parameters (A, a, u), as described by (4), where only the output dimension is changed fit the parameter dimension.
Computational Resources. Training was performed, using about hours of GPU time on an NVidia V100 GPU to carry out the experiment depicted in Figure (bottom).
B EXPERIMENTAL DETAILS OF EXPERIMENTS ON NATURAL DATA
B. ON THE PARAMETER UPDATE
In Section 4, rely on two different types of models: pretrained (e.g., fine tuning with VIBERT) and randomly initialized (e.g., in fair classification and domain adaptation). When working with randomly initialized networks the parameters are updated. However, is worth noting that in the literature the pretrained model parameters (i.e. 1)) are not always updated (see Ravfogel et al. (2020)). In our experiments: We always update the parameters (even for pretrained models), and (ii) did not change the way the parameters were updated in concurrent works (to ensure fair comparison). Specifically,
for language model finetuning (Appendix B.2), followed Mahabadi et al. (2021) and did joint update;
for the fair classification task (Appendix B.3), we followed common practice and used the algorithm described Algorithm which rely on an alternated update;
for the domain adaptation task (Appendix B.4), we followed common practice and used a joint method.
B.2 INFORMATION BOTTLENECK FOR LANGUAGE MODEL FINETUNING
For this experiment we follow the experimental setting introduced in Mahabadi et al. (2021) and work with the GLUE data?.
Model Architecture. We report in Table the multilayer perceptron (MLP) used compute the compressed sentence representations produced by BERT. Variance and Mean MLP networks are composed of fully connected layers.
see httph::////// httpa///////bbeeeeeeemmeee wwweenchaaaaee.eem com/ cam/aag
18

---

Under review as a conference paper at ICLR 2022
6
Bias
2
8
10
Variance
10-1
10-2
10-3
2
8
CLUB
DOE
InfoNCE
60
MINE
MSE
NWJ
40
KNIFE
20
MI Values
Figure 6: Bias, variance, and MSE for MI estimation on uniformly distributed data,
Algorithm 1 Disentanglement using a MI-based regularizer
1: INPUT Labelled training set D { 83, U2 E [n 1, N]}; independent set of samples E;
0 parameters KNIFE; / parameters of network.
2: INITIALIZE parameters 0, db
3: OPTIMIZATION
4: while (0, not converged do
5: for E [1, Unroll] do
Learning Step for KNIFE
6: Sample batch B from E
7: Update using ((9)).
8:
end for
9:
Sample a batch B' from D
10:
Update with B' ((11)..
11:
end while
12: OUTPUT Encoder and classifier weights
19

---

Under review as a conference paper at ICLR 2022
Table 10: Experimental details on Informa- tion Bottleneck.
Table 9: Architecture of the model used in the IB finetuning experiment. We use ReLU an activation function.
Parameter
Value
Learning Rate
See Appendix B.2
Layer type
Input shape
Output shape
Optimizer Warmup Steps
AdamW
Fully connected
768
2
0.0
765+ K
Fully connected
2904-K
Dropout
0.0
Batch Size
32
Table 11: Datasets from the GLUE as used in our experiments.
#Labels
Train
Val.
Test
RTE
2
2.5k
0.08k
STS-B MRPC
1 (regression)
5.8k
1.5k
1.4k
2
3.7k
0.4k
1.7k
Model Training. For model training, all models are trained for epochs and we use early stopping
10-5, (best model is selected on validation set error), For IB, À is selected in {10-+,10-3 1005,,10 10-6) 6] and K is selected (144, 192, 288, 384]. We follow (Alemi et al., 2016) where the posterior is averaged over 5 samples and a linear annealing schedule is used for X. Additional hyper-parameters are reported in Table 10.
Dataset Statistics.
Table 11 reports the statistics of the dataset used in finetuning experiment.
Computational Resources. For all these experiments rely on NVidia-P100 with 16GB of RAM. To complete the full grid-search on 10 seeds and on the three datasets, approximately 1.5k hours are required.
B.3 FAIR TEXTUAL CLASSIFICATION
In this section, we gather the experimental details for the textual fair classification task.
B.3.1 DETAILS OF THE KNIFE-BE BASEEB ESTIMATOR
In this experiment, we estimate the MI between continuous random variable, namely Z = and a discrete variable, denoted by S Section 2.4 for estimating the conditional DE h S) However, we will reuse the estimate of the conditional PDF ( +) to compute an estimate of the DE as
21, (|]]. We follow the strategy outlined in
h(Z) ~
1 N
log Σ (x AES
(14)
where p(s) = (n s, = =}} used indicate the empirical distribution of S in the training set D..3 our experiments, with [S] = 2, we found that estimating the DE h(Z) based on the KNIFE estimator learnt for h(2[S) increases the stability of training. We adopted the same strategy for DoE.
B.3.2 EXPERIMENTAL DETAILS
Model Architecture. For the encoder, and input dimension set to 128. We use LeakyReLU as the activation function. The classification head is composed of fully connected layers of input dimension 256. We use a learning rate of 0.0001 for AdamW. The dropout rate is set to 0.2. The number of warmup steps is set to 1000.
bidirectionnal GRU with two layers with hidden
As work with balanced batches, will have p(s) ST
20

---

Under review as a conference paper at ICLR 2022
Computational Resources. For all these experiments, rely NVIDIA-PI00 with 16GB of RAM. Each model is trained for 30k steps. The model with the lowest MI is selected. The training of single network takes around hours.
B.4 UNSUPERVISED DOMAIN ADAPTATION
We follow the experimental setup given in Cheng et al. (2020a) as closely as possible, i.e., we pick hyperparameters given in the paper, if provided, those set the code:"
Model Training. We use Adam optimizer for all modules with learning rate of 0.001. Batch size set to 128. set the weighting parameter À = 0.1. The original code Cheng et al. (2020a) uses 15 000 training iterations, but we found most methods had not properly converged at this stage, and hence use 25 000 iterations instead. Similar to other experiments, we set the kernel size M 128.
Model Architecture. Table 12 summarizes the architectures used for the different modules. For the MI network of each method, the best configuration, based on the validation set of the first task MNIST MNIST-M, is chosen among 4 configurations: with without LayerNorm and with ReLU or tanh activation.
Computational Resources. For these experiments, we used cluster of NVIDIA-VION with 16GB of RAM. Each training (i.e., 25k iterations) single task requires average hours. Given that we have tasks, and repeat the training for 3 different seeds, on average hours computation time required for each method.
C BOUNDING THE ERROR
In the following, fix L > 0 and let Pa be the set of L-Lipschitz PDFs supported on X' := [0, ie., Jx p(x) da 1, and
Vx, E R Ip(x) p(v)) S L| - yll
(15)
for E PL. o ||>||| = Ee mxl
Assume p PL and let be a PDF supported on X. In order to show that estimation of h(X) achievable, we use standard in (2). The entropy estimate is then defined by the empirical average
is
estimator p(æ; w) Mac ΣM
x
[x-X" X
as
T(D:; w) =
Σ log p (nn w).
(16)
Further, define the following quantities, which assumed to be finite:
Pmax == max (p(x) ) :
2},
(17)
C = p(x) log" p(æ)dr,
(18)
C2 : 4 ||u|| k(u)du,
(19)
Kmax max(x(()
X}.
(20)
Note that is easily seen that Pmax < 1/2 and C1 < max Pmax log² Pmax 4e le= by our assumptions. The requirement C2, Kmax represents mild condition on the kernel function
We can now show the following.
httpss: / / thub, com/ i.i nea r 95/ Anyy known compact support suffices. An affine transformation then yields X 00 1Js, while possibly resulting a different ipsschitz constant.
'The & norm is chosen to facilitate subsequent computations. By the equivalence of norms Rd, any norm suffices.
21

---

Under review as a conference paper at ICLR 2022
Table 12: Architectures used for the Unsupervised Domain Adaptation experiments. For the MI network of each method, we chose the best performing configuration between with or without LayerNorm layer and best activation between ReLU and tanh, using the validation set of MNIST-M.
Encoder (both & and 4d)
Layer type
Input shape
Output shape
Details
Convolution sequence
(3, H. W)
(64, H. W)
Cf. below
Noisy downsampling
(64, H, W)
(64, H 2, W If 2)
Cf. below
Convolution sequence
(64, H 1/ W // 2)
(64, H I/ 2, // 2)
Cf. below
Noisy downsampling
(64, H 2/ W 2)
(64, H II 4,
W II 4)
Cf. below
Convolution sequence
(64, H 4, ()
(64, H 4/4,
II 4)
Cf. below
Global Average Pool
(64, / W 4/4)
(64.)
Main classifier C
Domain classifier Cd
Layer type
Input shape (64.)
Output shape
Layer type
Input shape (64.)
Output shape
Fully connected
(10,)
Fully connected
(2.)
Convolution sequence
Layer type
Input shape
Output shape
Parameters
2D convolution
(3, H. W)
(64, H. W)
3x3, 64 channels, Stride=1, Padding=l
2D BatchNorm
(3, H, W)
(64, H. W)
Activation
(3, H, W)
(64, H. W)
LeakyRelu 0.1 3x3, 64 channels, Stride=1, Padding=1
2D convolution
(64, H, W)
(64, H, W)
2D BatchNorm
(64, H. W)
(64, H. W)
Activation
(64, H, W)
(64, H. W)
LeakyRelu 0.1 3x3, 64 channels, Stride=1, Padding=l
2D convolution
(64, H, W)
(64, H. W)
2D BatchNorm
(64, H, W)
(64, H. W)
Activation
(64, H. W)
(64, H. W)
LeakyRelu 0.1
Noisy downsampling
Layer type
Input shape
Output shape
Parameters
MaxPool
(64, H. W)
(64, H 2/ /
2x2, Stride=2
Dropout
(64, H // 2, W 2)
(64, H // 2, 2
p=0.5
Noise
(64, H // 2. W 2/2)
(64, H 2, /
Gaussian with
MI network
Layer type
Input shape
Output shape
Details
LayerNorm
(Cin.)
(Cin.)
Optional
Fully connected
(Cin.)
(64.)
Activation = [ReLU, tanh]
LayerNorm
(Cin.)
(64,)
Optional
Fully connected
(64.)
(Cout.)
Optional
Theorem 2. With probability greater than 1 we have
3.N K max
GN log
3NCw
/CG
h(X) - -(Px: log
3
+
V
(21)
wd8
2M
8
No¹
the expression in the logarithm is positive.
22

---

Under review as a conference paper at ICLR 2022
In particular, the estimation error approaches zero N + + - - M = M(N) are chosen such that
Nw + 0, N2 log N 2 M
(22)
-
(23)
We prove Theorem in several Lemmas.
Lemma 3. Fix and zo E Then, with probability greater than 6,
K Kmax
log 2M
(p(#o) - f(x00) S
+ C2w.
(24)
wdd
Proof.
First, we can show that
M
[lp(2))) - p(æo)1
=
Ma
To
w
- x
p(x)dox - p(zo)
(25)
m=1
wvd
K
30
p(z)dæ p(an))
(26)
w
f n. (u) p(an
wu)du p(xo)
(27)
/f n. (u) [p(aro wu) - p(x du
(88)
/ K. (u) Ip(2o wu) p(ao))du
(29)
K (u) Luilul/de
(30)
wC2.
(31)
Next, note that
K Kmax to
log 8
Blp(zo)) -(x00) <
(32)
2M
holds with probability greater than 1 - the requirements of McDiarmid's inequality (Paninski, Kmax and thus P([E[p(zo) p(zo)l 2 =} < with 2003, Sec. 3) are satisfied with 0j M
K max wd
log a
(33)
2MM
Combining (31) and (32) gives (24).
Lemma 4. For any continuous random variable X supported on X and 2 have
P(p(X) I <a} a.
(34)
Proof. We apply Markov's inequality to the random variable Y p(X and observe that
P(p(X) s a} = P(Y 2 a- s vol(X)a a.
(35)
Lemma 5. of>>0,2a>0.000000 1, and |r yl 5 < a, then
a
log x - log yl S log a- - log
(36)
23

---

Under review as a conference paper at ICLR 2022
Proof. Case 2 y. We can write y a at and b-c 0 and 0 S
8
1+ e
log = log
(37)
a+b)
[1 =) s log
8 1+=1
S log
(38)
a
Furthermore,
8 log (a + 6) - 6)
log
a aa-8)
log
1
(39)
log a² 82
(40)
1
2 log
-2 2ogaa 0.
(41)
a²
Case x < <.g, Here, we can write y = a and x = y - c = a + e for b and 0 S e a.
log
log
(42)
y
+ b
log
(43)
< log
(44)
6 1--
< log
loog
1
(45)
Proof of Theorem 2. We apply Lemma 3 N times and use the union bound to show that with proba- bility greater than 1 - we have for every n E [N]
K Max w
log
(((,)) -(X,))
S
+ Cyw.
(46)
2M
Similarly, by Lemma 4, we have with probability greater than 1 -
3 that
6
p(X,) 2
(47)
3N
for all n E [N].
Again by the union bound, we have that with probability greater than 1 both (46) and (47) hold for all n E [N], and thus, by Lemma 5, we obtain
i(D2; w) + 1 N
N
((,,))) (Xx))
Σ log logg(X.)) N n=1
log
(48)
N 12
Kmax wnst
log x
C2w
- log
1
2M1
(49)
3N
3NN Kmax
6N log
3N 3NC2w
- log
1
&
(50)
wdd 8
2M
24

---

Under review as a conference paper at ICLR 2022
provided the argument in the logarithm is positive. Finally, we have the upper bound the variance
N log p( (X.))
1 N
E
h(X) +
B((h(X) + log
(51)
N2 #2m
(Ellog p(X) - h(()))
(52)
Ci
(53)
and apply Chebychev's inequality, showing that with probability greater than 1 1-
N
h(X)
1
Σ log (X,)
S
3C1
(54)
N
N6
The union bound and the triangle inequality applied to (50) and (54) yields the desired result.
D LIBRARIES USED
For experiments, we built upon code from the following sources.
VIBERT (Mahabadi et al., 2021) at gi thub. wtt.c/.bccm//rbbbeekk/////////
TRANSFORMERS (Wolf et al., 2019) at gihaua.cwwwmuggiinttaeeettttttoe
DOE (McAllester & Stratos, 2020) at github. com/kaalsstraatos//be.
SMILE (Song Ermon, 2019) at github. ctmm://mmmonggogggammmmmmmeeeeee
InfoNCE, MINE, NWJ, CLUB (Cheng al., 2020a) at gg1thbb.ccom//neee555/0....
25


Reviews:
[{"review_id": "iemgWDZ88bk", "reviewer": "Reviewer_ABE4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Original Review\n\nThe methodological novelty of this work is the learnable weighting coefficients and centroids, i.e., modifying the Parzen estimator $\\frac{1}{M}\\sum_i k_{A_i}(\\cdot - x_i)$ as $\\sum_i u_i k_{A_i}(\\cdot - a_i)$, where the parameters $(u_i, a_i)$ are chosen to minimize the KL divergence w.r.t. the data distribution, and $A_i$ is the learnable bandwidth.  It is argued that, in ML tasks where the data $x_i$ are the learned representations and change during training, the modification allows the estimator to be both flexible and efficient.\n\nThe argument will be reasonable in problems where learnable bandwidth to be necessary, yet more flexible estimation methods (e.g. with neural energy based models) are too expensive.  The strength of the method mostly lies in its practicality (in representation learning tasks): it does not provide proper lower or upper bounds for mutual information, which is the central quantity of interest in the downstream tasks.  The novelty is limited, but this is fine if there is consistent improvement in empirical performance.\n\nI don't have any major issues with this work. There are nonetheless a few questions that need clarification:\n\n1. Consistency of the results with previous work:\n    * Table 1 doesn't appear consistent with any tables in Mahabadi et al (2021), e.g. in the MRPC experiments the accuracy is consistently higher than F1 in Mahabadi et al (2021), but not here. \n    * Table 2 doesn't appear consistent with Table 2 in Cheng et al (2020a), the baseline performance here seems consistently worse, and the difference is significant with the only exception of M->U/U->M.\n\nIn both cases, this paper claims to follow the setup in the corresponding previous work closely, so an explanation on the difference will be appreciated.\n\n2. Usefulness of MI regularization in downstream tasks: while I can imagine MI-like/-inspired regularization can be useful, I'm less certain if accurate estimation of MI always translate to improvements.  For example, from a quick scan it appears Mahabadi et al (2021) used a very heuristic estimation of MI, by fitting multivariate Gaussians on the joint and conditional distributions, yet demonstrated a similar level of improvement in GLUE as here.  While the difference in experiment setup prevents any definitive conclusion, it would be convincing if the authors could implement the heuristic in Mahabadi et al (2021) in the GLUE experiment here and report the performance.\n\n3. For entropy/MI regularization for downstream tasks, the gradient (score) estimators in the variational inference literature may serve the same purpose.  See, for example, Section 9.6 in Mohamed et al (2020).  While evaluation may be difficult to fit in the rebuttal timeline, it would greatly strengthen the paper if the authors could eventually compare with some works in this line, e.g., Song et al (2019) and Zhou et al (2020), as they usually claim applicability in problems with a similar complexity.\n\nReferences:\n\n* Mohamed et al (2020), Monte Carlo Gradient Estimation in Machine Learning, in JMLR.\n* Song et al (2019), Sliced Score Matching: A Scalable Approach to Density and Score Estimation, in UAI.\n* Zhou et al (2020), Nonparametric Score Estimators, in ICML.\n\n## Post-rebuttal Update\n\nThe authors' response addressed my questions about the experiments and the updated proof appears correct.  The contribution of this work is largely empirical -- the listed requirements and the new estimator do not appear very novel to me, although it is understandable that the characteristics of the downstream tasks may prevent the development of more flexible methods.  The experiments clearly demonstrate improvements over past MI-based methods.  However, as I'm not familiar with the evaluation tasks, I cannot evaluate their significance in the broader context; that task will have to be left to the other reviewers.  Therefore, I'm changing my score to 6, in light of the resolved questions about the theory and experiments.", "summary_of_the_paper": "This work proposes an entropy / mutual information estimator that is suitable for representation learning, by extending the Parzen-Rosenblatt estimator with learnable centroids, bandwidth matrices and coefficients.  The authors prove consistency, and demonstrate that on various downstream tasks where MI-based regularization is needed, the proposed method outperforms previous work on entropy / MI estimation.", "summary_of_the_review": "+ Simple method with consistent improvement in empirical performance.\n\n- Uncertainties around the experiment results.\n\nI will change my score if the two questions about the experiments are clarified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "d4jRbhZv8b", "reviewer": "Reviewer_Y1rk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the empirical results are promising, ultimately this paper is incomplete. First, the authors completely ignore the many estimators of information theoretic measures that have good empirical results and strong theoretical results. These include [R1-R5]. The authors should include these in the discussion of prior work at the very least and compare the empirical results to the ones that are also differentiable.\n\nSecond, the theoretical results are very weak and may even be wrong. Minimax estimation results for estimating entropy, divergence, and mutual information have been established [R1,R2,R6]. All of these results show that lower bounds on the estimation accuracy depend on the dimension of the data as well as the smoothness of the densities. The authors' convergence rate results do take into account the density smoothness but appear to be independent of the data dimension. The authors should resolve this apparent discrepancy.\n\nOther comments:\nIn Section 1.2 the authors state: \"In machine learning applications, however, the use of asymptotic results is not realistically justified.\" I disagree with this. Asymptotic results often relate to finite sample results, i.e. many estimators with good asymptotic theory often have better empirical results as well. \n\nWhile it's nice that KNIFE adapts to new data, it seems that most nonparametric estimators would automatically adapt. Thus the lack of adaptability seems to be more a problem with parametric estimators and wouldn't be an issue with the estimators given in [R1-R5]. \n\nI'm somewhat skeptical about minimizing the LHS of (6). While the explanation provided makes intuitive sense, in practice, due to finite samples, it does seem like underestimating the entropy could still happen. Perhaps some kind of concentration inequality could be used to obtain a more accurate bound instead of using the LLN?\n\n[R1] Moon et al, \"Ensemble estimation of generalized mutual information with applications to genomics,\" IEEE Trans. on IT, 2021.\n[R2] Kandasamy et al., \"Nonparametric von Mises Estimators for Entropies, Divergences and Mutual Informations,\" NeurIPS, 2015.\n[R3] Singh and Poczos, \"Exponential Concentration of a Density Functional Estimator,\" NeurIPS, 2014.\n[R4] Sricharan et al., \"Ensemble Estimators for Multivariate Entropy Estimation,\" IEEE Trans. on IT, 2013.\n[R5] Berrett et al, \"Efficient multivariate entropy estimation via k-nearest neighbour distances,\" Annals of Statistics, 2019.\n[R6] Birge and Massart, \"Estimation of Integral Functionals of a Density,\" Annals of Statistics, 1995.\n\nPost-rebuttal update: I have read the authors' response to my review and the other reviewers. I appreciate the revisions the authors have made up to this point, especially regarding the theoretical results. However, I do believe that comparisons to other estimators should be done before I can recommend publication. In their comment, the authors claim that some of these estimators are not well-suited for the proposed use-cases. However, many of these estimators are based on plug-in approaches similar to the KNIFE estimator. Thus I believe that they can be compared to as well.\n\nI have thus raised my score to be marginal, leaning towards reject. \n", "summary_of_the_paper": "This paper provides a new approach to estimating differential entropy called KNIFE that is also applied to mutual information estimation. The authors define their estimator using a parametric model based on estimating a KDE. The authors provide some theoretical analyses of the estimator and multiple empirical experiments where the proposed estimator outperforms several other estimators.", "summary_of_the_review": "The authors do not mention nor compare to other important estimators of differential entropy and mutual information. Furthermore, there are important unanswered questions about the theoretical results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "clnSMGY2_8A", "reviewer": "Reviewer_ug32", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strength]\\\nEmpirical evidence demonstrates the effectiveness of the proposed method in terms of its estimation error and adaptation to underlying data distribution shift. The results seem fairly consistent, at least for the tasks and baselines the authors included.\n\n[Weakness]\\\nThe technical significance of the proposed method seems incremental: The Schraudolph estimator proposed before already includes the covariances as learnable parameters, and there does not seem to be many technical challenges of making the anchor and mixture probability learnable as well. But I'll take the author's defense on this into consideration.\n\n[Other comments]\\\n$\\theta = (A, a, u)$ can be viewed as a parameter for the loss function How is $\\theta$ updated, say for the VIBERT example? Is it updated together with the model parameters or in a bilevel fashion (alternating)?. If it is the former, how would it prevent mode collapse? Usually, in the AutoML literature where one wants to learn a surrogate loss function, people would alternatively update loss function parameters and the model parameters to prevent mode collapse or overfitting.\n\n---- post rebuttal update\nThe author's response and the revised draft addressed my questions. The background information provided in the statement of novelty improves my view on the matter, though I hold to the opinion that extending the Schraudolph estimator does not seem to be particularly challenging. That been said, the proposed method is well motivated and its effectiveness is sufficiently backed by empirical evaluations. Overall it is a paper of quality. I raise the tech contribution score to 3 and confidence to 3 and am inclined towards acceptance.", "summary_of_the_paper": "This paper studies differentiable proxy estimators for density function, which is in turn used to compute various information metrics such as (conditional) entropy and mutual information. The main contribution of this paper is that it generalizes previous kernel-based density estimators by parameterizing the anchors and mixture probability of kernels. The advantage of the resulting method is that it 1) has increased capacity 2) it can adapt to input data distribution shift.\nThe author provides convergence of the resulting entropy, as well as providing extensive empirical studies on synthetic and real data, such as BERT+text classification and Unsupervised Domain Adaptation.\n", "summary_of_the_review": "The technical aspect of this paper is not very significant in my opinion. However, empirical evidence seems sufficient and the improvements look consistent. My rating for this paper lies on the borderline for the moment. It is likely that I'll make changes after the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9v_-6p3yCMd", "reviewer": "Reviewer_8iox", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles the important problem of estimating commonly used information-theoretic quantities like differential entropy and mutual information. For a random variable $X$ with density $p$, differential entropy is $h(X) := - \\int p(x) \\log p(x) \\, \\mathrm{d}x$. Given $n$ i.i.d. samples $\\\\{x_i\\\\} _ {i=1}^n$ of $X$, a naive monte carlo estimate $-\\frac{1}{n} \\sum _ {i=1}^n \\log p(x_i)$ cannot be calculated since $p$ is unknown. Therefore, a common solution to this problem is to estimate the density $p$ using an i.i.d. sample $\\\\{x'_i\\\\} _ {i=1}^m$ of $X$ _independent_ of $\\\\{x_i\\\\} _ {i=1}^n$. This paper uses a modification of standard kernel density estimators to estimate $p$, and therefore gets\n$$\n\\widehat{h} _ {\\text{KNIFE}} = -\\frac{1}{n} \\sum _ {i=1}^n \\log \\widehat{p} _ {\\text{KNIFE}}(x _ i; \\theta).\n$$\n\nIt's easier to criticize, so let me do that first. The core of the paper, Section 2, where the proposed estimator is discussed is devoid of necessary details. For example, what is $\\mathbf{a}$ in the proposed estimator for $\\widehat{p} _ {\\text{KNIFE}}$? It is unclear how $\\widehat{p} _ {\\text{KNIFE}}$ is estimated: what data are used (is it a subset of training data to maintain independence?, is it all training data? why?)? In the paragraph \"Learning step:\" of Section 2.2, why would _minimizing_ $\\widehat{h} _ {\\text{KNIFE}}$ be desirable? isn't the goal to get $\\widehat{h} _ {\\text{KNIFE}}$ close to $h(X)$? Although an interesting result, how is Theorem 1 relevant for KNIFE estimator? The estimator $\\widehat{h}$ used in Theorem 1 is not $\\widehat{h} _ {\\text{KNIFE}}$. The selling point of the paper is the added flexibility provided by $\\widehat{h} _ {\\text{KNIFE}}$ over other estimators like $\\widehat{h}$.\n\nThe biggest strength of the paper is the extensive experiments to justify that the proposed estimator is in fact better than other estimators on many tasks.", "summary_of_the_paper": "The paper proposes an estimator $\\widehat{h} _ {\\text{KNIFE}}$ for differential entropy suited for applications in deep learning. A set of desirable requirements is specified all of which are satisfied by $\\widehat{h} _ {\\text{KNIFE}}$ but not by other commonly used estimators. Extensive experiments are performed to justify the new estimator.", "summary_of_the_review": "The paper needs some rewriting of the technical sections, but otherwise is a good paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1abSgAmFRzz", "reviewer": "Reviewer_5zVE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Estimation of (differential) entropy and the related mutual information are topics in machine learning of fundamental importance, which drive a wide range of applications.\n2. The proposed method is relatively easy to implement. \n3. Experiments on the real-world dataset have decent coverage of important applications concerning mutual information. \n4. This paper is written with clarity and is fairly easy to follow. \n\nWeaknesses:\n1. Originality. There is not much novelty in this work. The proposed solution is a simple plug-in estimator based on adaptive kernel density estimation. The techniques used are very standard and there is no new theory developed.  \n2. Lack of comparisons, the following competing estimators should be covered in the discussion or compared in experiments.\n - Plug-in estimators based on likelihood-ratio estimates, like neural estimators (or JSD estimator) and ML/least-square estimators (see [1] and reference therein)\n - Nearest-neighbor estimator [2]. \n3. Lack of in-depth discussions. Variational schemes have been proposed to address the inadequacy of plug-in estimators in such settings (likelihood estimation for complex distributions in high-dimensions is a long-standing challenge in statistics and machine learning), at least the paper should expand the discussion on that point. The experiments have focused on low-dimensional setups, so the proposed method works okay. But performance on high dimensions random variables is unknown. \n\n[1] Suzuki, Taiji, et al. \"Approximating mutual information by maximum likelihood density ratio estimation.\" New challenges for feature selection in data mining and knowledge discovery. PMLR, 2008.\n[2] Berrett, Thomas B., Richard J. Samworth, and Ming Yuan. \"Efficient multivariate entropy estimation via $ k $-nearest neighbour distances.\" The Annals of Statistics 47.1 (2019): 288-318.\n", "summary_of_the_paper": "This paper introduces a differentiable kernel-based estimator of differential entropy, named KNIFE.  \nKNIFE-based estimators can be applied to both conditional (on either discrete or continuous variables) differential entropy and mutual information. In essence, KNIFE leveraged a kernel-based nonparametric likelihood estimator for the plug-in estimate of differential entropy, where the basis and covariance parameters are learned via MLE. The proposed method is validated on high-dimensional synthetic data and guiding the training of neural networks for real-world tasks, including domain adaptation and fair learning. \n\n", "summary_of_the_review": "The quality is okay but falls below the ICLR threshold because of the lack of originality and in-depth discussions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The focus of the submission is the estimation of the Shannon differential entropy (DE). The authors propose a differentiable DE estimator referred to as KNIFE (Kernelized Neural diFFerential Estimator): it is a plug-in method (5) using KDE (kernel density estimation; (4)). KNIFE has parameters including the locations (a), weights (w) and covariances (A) in KDE, which are tuned according to the upper bound heuristic in (6). The approach is illustrated on toy examples and in the context of training neural networks.

Estimating information theoretical quantities is a current topic of machine learning. Unfortunately, as assessed by the reviewers
1) the submission lacks context and comparison to available entropy estimators,
2) the estimator closely follows Schraudolph (2004); the technical novelty is quite limited.

More work and major revision are required.
Decision: Reject