A Variance Reduction Method for Neural-based Divergence Estimation
Authors: Jeremiah Birrell
Markos A. Katsoulakis
Yannis Pantazis
Dipjyoti Paul
Anastasios Tsourtis
Venue: ICLR 2022 Submitted (2022)
Keywords: Divergence estimation
Variational formulas
Variance reduction
Representation learning

Abstract: A central problem in machine learning is the computation of similarity or closeness between two (data) distributions. The applications span from generative modelling via adversarial training, representation learning, and robustness in out-of-distribution settings, to name a few. A palette of divergences, mutual information, and integral probability metrics are indispensable tools for measuring the ``distance'' between distributions and these are made tractable in high dimensional settings through variational representation formulas. Indeed, such formulas transform an estimation problem into an optimization problem. Unfortunately, the approximation of expectations that are inherent in variational formulas by statistical averages can be problematic due to high statistical variance, e.g., exponential for the Kullback-Leibler divergence and certain estimators. In this paper, we propose a new variance penalty term that acts directly on the variance of each component of the statistical estimator. The power of the variance penalty is controlled by a penalty coefficient which trades off bias and variance.  We tested the proposed approach on several variational formulas and synthetic examples and showed that the overall error is decreased about an order of magnitude relative to the baseline statistical estimator. Impressive results are obtained for R\'enyi divergence with large order values due to the improved stability of the proposed estimator. Furthermore, in real biological datasets we are able to detect very rare sub-populations with a moderate sample size. Finally, we obtain improved (in terms of objective measures) disentangled representation of speech signals into text, speaker, and style components via variance-penalized mutual information minimization.

Under review as a conference paper at ICLR 2022
A VARIANCE REDUCTION METHOD FOR NEURAL- BASED DIVERGENCE ESTIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
A central problem in machine learning the computation of similarity or close- ness between two (data) distributions. The applications span from generative mod- elling via adversarial training. representation learning, and robustness out-of- distribution settings, to name few. A palette of divergences, mutual information, and integral probability metrics are indispensable tools for measuring the "dis- tance" between distributions and these are made tractable in high dimensional settings through variational representation formulas. Indeed, such formulas trans- form estimation problem into optimization problem. Unfortunately, the ap- proximation of expectations that are inherent in variational formulas by statistical averages be problematic due high statistical variance, e.g., exponential for the Kullback-Leibler divergence and certain estimators. In this paper, we propose new variance penalty term that directly on the variance of each component of the statistical estimator. The power of the variance penalty is controlled by a penalty coefficient which trades off bias and variance. We tested the proposed ap- proach on several variational formulas and synthetic examples and showed that the overall error is decreased about an order of magnitude relative to the baseline sta- tistical estimator. Impressive results are obtained for Rényi divergence with large order values due to the improved stability of the proposed estimator. Furthermore, in real biological datasets we are able to detect very rare sub-populations with a moderate sample size. Finally, we obtain improved (in terms of objective mea- sures) disentangled representation of speech signals into text, speaker, and style components via variance-penalized mutual information minimization.
INTRODUCTION
Divergences such as Kullback-Leible (KL) divergence, f-divergences, Hellinger divergence, a- divergences and Rényi divergences, which were initially developed in the fields of information theory and statistical physics, are indispensable tools a growing number of machine learning applications. They have been used in adversarial training of generative models (Goodfellow et al. 2014 Nowozin et al., (016),, in the estimation of generalization errors (Esposito et al.] (2021) and hypothesis testing (Broniatowski & Keziou| 2009), to ticular, which is defined as the KI. divergence between the joint distribution and their marginals (and can be generalized to divergences other than KL), plays crucial role in Bayesian networks and (conditional) independence (Cheng et al. 2002). self-supervised learning via contrastive losses (van den Oord al. 2018 Le-Khac et 2020) well in representation
few. Mutual information (MI), in par-
pair of variables
learning (Hjelm et al. Chen et al. 2016)..
Classical divergence estimators perform reasonably well for low dimensional cases, however they scale poorly to large, high dimensional datasets which are typically encountered in modern machine learning. The most compelling estimation approach of a divergence is via the optimization of a lower variational bound parametrized by neural networks. These lower bounds, which are likelihood-free approximations, are maximized in order to compute the divergence value at the optimizer. Well- known variational representations are the Legendre transformation of an f-divergence (Broniatowski & Kezioul 2006 Nguyen et al. 2010) as well as the Donsker- Varadhan (DV) variational formula (Donsker & Varadhan 1983) for KL divergence and its extension to Rényi divergence (Birrell et al. 2020b). Their tractability stems from their objective functionals, which are computed from expected values and approximated using statistical averages from the available generated samples.

---

Under review as a conference paper at ICLR 2022
Despite the scalability and tractability, the estimation of a divergence based on variational formulas is a notoriously difficult problem. One challenge stems from the potentially high bias, since any approximation for the worst case scenario requires exponential number of samples in order to attain the true divergence value (McAllester & Stratos 2020). Additionally, the statistical variance, which scales exponentially with respect to the divergence's is value for certain variational estimators (Song& & Ermon 2019), is often prohibitively high. Focusing on the elevated MI, there are several further lower bounds (Barber & Agakov 2003; Belghazi et al. 2018 van den Oord et al.] 2018 Poole et al. 2019, Guo et al.) (021) and a few upper bounds (Cheng et al.] 2020] Poole et al, 2019] which aim to provide more reliable estimates of MI in the low sample size regime. However, the majority of these MI estimators are not transferable to the general estimation of divergences and frequently produce instabilities during training which are further magnified by the small batch and/or sample size.
In this paper, we propose to reduce a divergence estimator's s variance via an explicit variance penalty (VP) which is added to the objective functional. Our contributions are summarized as follows:
. We present a novel variance reduction penalty for f-divergence and expand it via the delta method to the nonlinear setting, including the DV formula for KL divergence as well as the variational formula for the Rényi divergences. The proposed VP is able to flexibly trade off bias and variance.
We present numerical evidence on synthetic datasets that the proposed approach improves both mean squared error (MSE) and median absolute error (MedAE) in range of sample sizes and types of divergences. Furthermore, we implemented the proposed VP in several other lower and upper bounds of MI, showing that our variance reduction approach is not restricted to particular variational formulas but it is generic and applicable to the majority of existing variational representations.
When applied to real datasets, we demonstrate the ability of the proposed approach to re- duce the variance of the estimated Rényi divergence, thus enabling the detection of rare biological sub- populations which are otherwise difficult to identify. Interestingly, the base- line estimator is unstable when the order value is above one, but it becomes stable when the VP is added.
We also applied the VP to the disentangled representation learning of speech into its text, speaker, and style components. Results on objective evaluation metries showed that the addition of the VP generally improves the training performance, as much as 18% relative to the baseline systems.
1.1 RELATED WORK
There are several general-purpor variance reduction techniques in Monte Carlo stochastic sam- pling, with the most popular approaches being antithetic sampling more broadly coupling meth- ods, control of variates and importance sampling (Robert & Casella 2005 Glasserman] [004 Srini- vasan 2013). These methods have not been explicitly applied for the variational divergence estima- tion problem. We speculate that either they are not applicable due to the unavailability of analytical probability density formulas or they are inefficient (c.g.. the control of variates approach requires a second estimator and potentially a second parametric model in order to be applied).
Another way to reduce the variance is restrict the function space to more smooth and/or controlled test (or critic) functions, balancing again between bias and variance. For instance, the restriction to Lipschitz continuous functions has the potential reduce the variance since there exist favorable concentration inequality results for the Lipschitz space (Wainwright (019). In the GAN literature, Wasserstein GAN (Gulrajani et al., 2017) and spectral normalization (Miyato et al., 2018) impose Lipschitz continuity which resulted in signigicant gains in terms of training stability. Similarly, the restriction of test functions to an appropriately designed reproducing kernel Hilbert space could reduce the variance (Sreekar et al. 2020). Such approaches can be combined with our proposed variance penalties, as our formulation allows for general test-function spaces. However, we do not focus on this point here.
Given the importance of MI, several estimators aim towards improved statistical properties. Lower bounds such MINE (Belghazi et al. 2018), which uses the DV variational formula with expo-
2

---

Under review as a conference paper at ICLR 2022
nential moving average. NWJ estimator (Nguyen al. 2010) and BA estimator (Barber & Agakov 2003) as well as upper bounds such as CLUB (Cheng et al. 2020) still have high variance. InfoNCE van den Oord et al.) 2018) is one of the few MI estimators that has low variance, but at the cost of either high bias or high computational cost due to the need for many negative samples and thus large batch size. PPoole et al. (2019) and Guo et al. (202T) aim elarify the relationships and trade-offs between those variational bounds. A different approach to reducing variance is by appropriately working on the gradients of the objective function (Wen al. 2020 2021
Finally, we discuss the approach of truncating the test function inside a bounded region as proposed in (Song & Ermon| 2019). The determination of the truncation threshold is quite difficult since requires an priori understanding of the log- leglliid ratio. Moreover, a high truncation thresh- old will not affect the estimation since a high threshold implies no real benefit in terms of variance reduction. On the other hand, a low threshold will result in large bias. Overall, using high trunca- tion threshold in order to avoid extreme values a good practice even though will have limited impact variance reduction.
2 BACKGROUND ON VARIATIONAL FORMULAS FOR RÉNYI AND f-DIVERGENCES.
While variance reduction method can be applied to any divergence that possesses variational formula, here our focus will be on the Rényi and f-divergences, including the KL divergence. For Rényi divergences appropriate objective functional can be constructed from difference of cu- mulant generating functions (Birrell et al. 2020b)
Ra (QllP)
sup (2M)
log Eale(a-19))
log pp [e a + 0,
(1)
Here Q and P are probability distributions on the set Ω Eg and Ep denote the expectations with respect to Q and P respectively, and Mo( is the space of bounded measurable real-valued func-
tions on Ω. For / divergences, I being a lower semicontinuous convex function with f(1) 0, one
has the well-known Legendre transform variational formula (Broniatowski & Keziou
2006 Nguyen
et al. 2010)
D)(Q1PP)
sup (Eolg) Ep(s"())) gEM.((R)
where ("(v)= = suPzER fyz- f(x)) is the Legendre transform of f- Here and in the following, the function of g that being optimized will be called the objective functional. Equation (2) can be generalized to the (f. -idveeeeeesss (Birrell et al. 2020a), where c Mc([2) is restricted test-function space
DF(QIIP) sup b/ geT
Aflgl = inf fv Ep -v v)]}
(4)
PER
In particular, if fx)( = x log(x) corresponds to the KL divergence then APlg)
A fa. [g] log log(Ep(exp(g))
(5)
is the classical cumulant generating function and equation (with r = Mo(()) becomes the
Donsker-Varadhan variational formula (Dupuis & Ellis. 1997 A Appendx C.2)
(kl(Q] sup {Eq [g] log 1 g.MM((0)
(6)
For general f. we will often write equation (3) as
(TP( sup
{Eolg - b)]}
gET,
and if I' is closed under the shifts g > DF(Q1IP)= sup Eg|g] Erlf*
v, R then we can write it simply
(8)
In particular, rf M2) then DF = Dj. The generalizations of Rényi and KL divergence obtained by using a restricted space in place of M2(22) in equation []] or equation (6) will be denoted by R. and DE respectively.
3

---

Under review as a conference paper at ICLR 2022
3 STATISTICAL ESTIMATORS AND VARIANCE REDUCTION
Variational representations of divergences are especially useful for creating statistical estimators in data-driven setting; a naive estimator is obtained by simply replacing expectations with the corresponding statistical averages in any of the equations [ (2). (3) etc. More formally, the naive estimators can be written as (Qnl P.). R Pn), etc., where I is some parameterized space of functions (e.g., neural network). Qn and Pv. the n-sample empirical measures from Q and P respectively (i.e., Ep, [g] = 1 Σj= gXx)) where X, are i.i.d. samples from P and similarly for iEq.; we also assume that the samples from Q and P are independent of one another), and the divergences are expressed in terms of the variational formulas from Section 2 However, in practice these naive methods often suffer from high variance (Song & Ermon 2019 Birrell et al., 2020b).
We address this via variance- penaieeedd divergences, which are constructed by introducing variance penalty into the objective functional of the variational representation, e.g.
(Q11)) = sup [Bolg) [p((" XV PI
ge.M.(0)
where the variance penalty, XV, is proportional to the variance of Eq,, [g] - Ep, (g)) with strength O. Using this, we construct the following divergence estimator
(Ep, Ep. (f,) \g Qn Pal]
(10)
where In is a neural network with parameters v7. Similar variance penalties can be derived to other divergences with variational representations.
3.1 VARIANCE PENALTY
In this subsection we provide details on the variance penalty for (J. diiiieergggss the KL- divergence, and Rényi divergences. The same framework can be repeated to other divergences with a variational representation.
To introduce the variance penalty, first consider the (f, didvverrgeeee representation equation (8) Our goal is to penalize g's for which Eq,, [g] or Ep, (f),((() (g)] have large variance, hence we introduce a penalty term proportional (Varq denotes variance with respect etc.)
1
Var [e, [a] + Ep,, (5"(0))) = (Varolg) + Varp(/"
(11)
Specifically, for > 0 we define the variance-penalizee (f, divvergeee
(QIP)) sup (Eglg v) [r w)] - X XVarrllg v) + Varp -)])}
ger
(12)
As noted above, if is invariant under constant shifts then the optimization over v can be omitted. A similar result to equation (12) can be derived for any objective functional that is a linear combination
of expectations, e.g., integral probability metrics (Müller 1997 Sriperumbudur et al. 2012) such as the Wasserstein metric.
For nonlinear objective functional terms of the generic form ((E)))(g))), such as appear in equa- tion (1) and equation (6. cannot compute the variance of the corresponding statistical estimator at finite n but we can use the delta method to obtain the asymptotic variance
lim nar [G [b(g))) = (M))) R0
(13)
Thus, we propose for the nonlinear case to use the above asymptotic variance as penalty and obtain the following variance-penalized KL and Rényi divergence variational formulas:
Eqq [g] log Ep(es] A (Varalg) Varzle)/((p((9))))] }
(Q)(P)))) = sup 9ET
(14)
1
1 log [c
(Q)(() sup 3ET
log Eole(a- (a-1)
(15)
-
Varg (a (a-1))
Varple
-X
(a 1)a [BQ fe aala-11) 121)
a² (Lp (eag))))
4

---

Moreover, under fairly general assumptions holds that lim Under review as a conference paper at ICLR 2022
Remark 1. Both equation (II) and equation (3) suggest that the statistical estimators for the above penalized divergences should use a variance penalty strength that decays with the sample size X = >n/n, n, though other forms of n-dependencc may be useful in practice.
Though the variance penalty introduces bias, as 0 the penalized divergence converges to the corresponding non non-penalized divergence, made precise by the following theorem.
Theorem 2. Let C MM((2). We have the following convergence results:
lim (Q)))) (Q1IP))
(16)
2
lim (011)) = =[((Q)P),
(17)
x
and if (y) < do for all y E IR then
lim X-01
(18)
Dii (QIP)) X-000
x = lim R (QIIP) = lim DJ (Q1P))= X
(19)
Remark 3. Note that the corresponding statistical estimators, D (Q.)PPa), etc., have additional bias due to the supremum over g. We present partial results on bias bounds in Appendix D.
The proof of Theorem is given in Appendix for the zero limit and Theorem 9 for the infinity limit. The same proof techniques can be applied to other divergences with variational characterization.
Finally, for non-zero À the penalized divergences (22. 14 15 retain the divergence property and are therefore appropriate for quantifying the "distance" between probability distributions:
Theorem 4. Under fairly general assumptions on f and (see Appendix for details) and letting D denote any of D.,,, DEA or R we have (FI) 2 and DE-AQQ)))) 0 if and only if P.
The proof of Theorem can be found in Appendix B.
3.2
VARIANCE-
-ERRUNEE-REDUEDD REDUCED DIVERGENCE ESTIMATION ALGORITHM
We now propose the following divergence neural estimation (DNE) methods with variance penalty, generalizing equations (1) (DNE-VPA)
H Qn, Px] AV Qn Pol]
(20)
We will compare the above method to the non-penalized estimator (i.e., with À = 0) (DNE)
sup H [gns Qn, Pn]
(21)
In the above, the test function space is a neural network r = (g., 17 E E} with parameters and H denotes the objective functional the divergence, e.g., for the Rényi divergences (T)
Ha Q1
log Eole!o (ofon-1)) l)9] log Ep [e 0,1
(22)
and for divergences
H Q. P1 Eglsl -
Epls"(
(23)
Finally, V is the variance penalty corresponding to the chosen divergence (see Section 3.1 e.g. for Rényi divergences
Varq,, (e(a,,e((-11))) [e(0-119] + a (Er, (2 a 3 0,
Varp, [o
Valg: Qn, Px]
(24)
(a 1)2 (Eq,, [e(a- 110
and for divergences
1g Qn Px] = Varq, [g] + Varp, [f*
(25)
We solve equation method.
via Adam algorithm (Kingma Ba 2014); a stochastic gradient descent
5

---

Under review as a conference paper at ICLR 2022
4 RESULTS ON
S SYNTHETIC DATASETS
Figure Seuurepresene presents the statistical estimation of Rényi divergence between two one-dimensional Gaus- sians which both have zero mean but different variance values. The order of Rényi divergence, a, controls how much weight to put on the tails of the distributions, thus it can become very sensitive the few samples from the tails. The same conclusion can be deduced from the variational for- mula (i.e., equation (1) where multiplies the exponentials' argument). Therefore, larger value implies larger statistical variance. Indeed, high estimation variance is observed with DNE (upper leftmost panel of Figure [ despite the fact that we applied truncation as proposed by [Song & Fr- mon (2199) with truncation threshold set to I. In contrast, the DNE-VP, estimator with À = 0.1 greatly reduces the statistical variance even when is large (lower leftmost panel). For fairness, imposed the same truncation operation in the output of DNE-VPx- We report a 80% reduction of variance for 2 which becomes 99% for 10.
a = 0.5
DNE DNN-VP DNE-VP
Reelslive ModAE
Relative MedAI
R(QQ)P)
10
0.02
0.5
0.01
0
0.5
2
5
10
10
10
10%
10
N
DNE-VE ANNEVP.A.
0.05
10
1000 DNE-VP.
-INE
0.04
MedAE
10
Relative MelAE
10
DNE-VP.
R.(Q1P)
0.03
0.02
100
0.01
0
10
10 10
101
A
x10
Figure 1: Comparison between the estimator without VP (DNE) and with VP (DNE-VPA) for Rényi divergence between two one-dimensional Gaussians with Q (O,, and p = (0, 1). We use N 5K sample size, 512 as batch size and results are averaged over 100 i...d. runs. Left column: DNE and DNE-VP, estimators for increasing values of a. The variance of DNE becomes uncontrollably high for a > 3. Middle column: Relative MedAE (the lower, the better) for varying penalty coefficient and two values of a. The relative MedAE for large values of À is close to one which implies that the estimated value of DNE-VP, approaches zero Right column: Relative MedAE for increasing sample size N. We additionally present penalty coefficient that varies with 2000 for 0.5 and 10, respectively).
sample size, shown in blue (AN 500 and AN
N
The proposed approach introduces an additional hyper-parameter which controls the strength of the VP. Our theory suggests that À should depend on the sample size (and perhaps also the other parameters), therefore we perform two sets of experiments. In the first experiment, we explore the range of optimal values for À terms of MedAr As is evident from the middle panels of Figure | X-values in the vicinity of 0.1 are a reasonable compromise between variance and bias. In the second experiment, we demonstrate the performance in terms of MedAE as function of the sample size, N. As suggested in Remark [ monotone performance is obtained when À is inversely proportional N (blue dashed line rightmost upper panel of Figure [I]
Our second synthetic example constitutes the estimation of MI using various approaches with and without VP. Here, we let Q be a zero-mean multivariate correlated Gaussian random vector of di- mension d. We impose element-wii wise correlation, i.e., corr(r,, Ø +)) 8i,5 to the samples ~ Q
Recall that MedAF stands for median absolute error and is more rohust-to-outliers metric.

---

Under review as a conference paper at ICLR 2022
where i.j 1,..., and o., is Kronecker's delta. With P we denote the product of the marginals, which in this case is simply a zero-mean standardized multivariate Gaussian. Figure 2 presents the estimated MI per training step. We consider the Renyi-based MI with 0.5 as well as the stan- dard MI using the DV variational formula. Notice that these two variants result in different true values (black lines in F). The plotted results demonstrate the successful reduction of variance when VP is added to the objective functional. Interestingly, the extension of VP to InfoNCE and CLUB estimators (second row panels Figure 2) implies that our approach can be applied to any MI estimators, thus offering a general variance reduction framework. Bias, variance and MSE plots as well several more experiments can be found in Appendix F.
DNE-VP (Ra3()
DNE (KL
DNE-VP
Aud. (KL
14
14
12
Divergences
0-
6-
0
10000
20000
30000
40000
0
10000 20000 30000 40000
30000
20000
40000
10000
Steps
InfoNCE
InfoNCE-VP AAH.11
CLUB
CLUB-VP A+6.1
1.0
1.5
14
34
Estimatad
Exact
12-
Divergences
10000 20000 20000 40CD0
10000 20000 30000
40000
10000 20000 30000
AB0D0
10000 20020 30000 20000
Steps
Figure 2: Performance comparison of several MI estimation approaches 40-dimensional corre- lated Gaussian random vector. The number of samples set to 512K and batch size to 64. Panels with Ra= in their titles present the Rényi-base based MI with = 0.5 whereas the rest of the methods estimate the standard MI (i.e., the KL divergence). In each panel, the true values are shown as a step function (black line). The correlation coefficient of the Gaussian, p. for each step is: 0.3084, 0.4257, 0,5091, 0,5741, 0.6273 and 0.6717. The running estimates per minibatch are displayed as shadow blue curves. The dark blue curves shows the moving average of the estimated MI, with bandwidth equal to 200 steps.
5 REAL DATA APPLICATIONS
5.1 DETECTING RARE BIOLOGICAL SUB- POPULATIONS
Using the dataset from Levine al. 2015)) test the efficacy of DNE-VP, in discriminating cell populations which are contaminated with a rare sub-population with distinguishable statistical properties. Specifically, we consider single-cell mass cytometry measurements on 16 bone marrow protein markers (i.e., d = 16) coming from healthy and diseased individuals with acute myeloid leukemia. For each run we created three subsets of healthy samples with sample size which we denote by P and one dataset as a mixture of 99% healthy and 1% diseased samples which denoted by Q. Notice that the actual number of diseased samples is only 200 thus it considered as rare sub- cobb-popultiio..
20K
For Rényi divergence with a 0.5 (left panels in Figure 3 both DNE and DNE-VE are stable. Despite the improvement in the separation of the two histograms, the observed variance reduction
Data was accessed from

---

Under review as a conference paper at ICLR 2022
DNE
DNR
0.15
R PM
PP
Ra 2 P)
Probability
0.2
0.1
0.05
0.05
1 110
6.05
0.06
0.07
0.08
0.09
0.1
1000
2000
3000
DNE-VP
DNE-VP
0.15
PP
(Q1 P))
Probability
0.2
0.05
0.05
0.05
0.06
0.07
0.08
0.09
0.1
8.04
0.06
0.08
0.1
0.12
Divergence value
Divergence value
Figure 3: Comparison of DNE and DNE-VPX estimators for Rényi divergence on biological data. The histograms the estimated divergence value are constructed from 100 i.i.d. runs between datasets of N = 20 samples each. Healthy dataset's distribution is denoted by P whereas healthy + 1% diseased dataset's by Q. Left columm: Rényi divergence with a = 0.5. Neither DNE nor DNE-VPA are able discriminate between the healthy and the 1 1% contaminated dataset. Right col- umn: Rényi divergence with a = 1.1. For this a value, VP is compulsory for a stable estimation of Rényi divergence. Furthermore, are able discriminate between healthy and 1% contaminated distributions with high accuracy (87.5%).
of DNE-VP. VPx minimal and not enough to discriminate between the healthy and the contaminated with 1% diseased samples distributions. When considering Rényi divergence with 1.1, we observe that DNE fails to produce stable estimates. In contrast, DNE-VPX always computes stable estimates. Additionally, the two histograms of a are crucial, provided there is a way to handle the statistical variance. For completeness, Table I rrporrs the first and second order statistics of the histograms shown in Figure
satisfactorily separated, implying that larger values
Table Mean values and standard deviation for the histograms shown in Figure 3
Divergence
DNE
DNE-VPA V
mean
std
mean
std
Ra (P
0.0765
0.0066
0.0695
0.0053
R. R.=
0.0789
0.0039
0.0720
0.0036
R. Ra-1.1 (PIPP)
676
515
0.0890
0.0089
Ro=1.1 (Q] 1445
1165
0.1000 0.0120
5.2 DISENTANGLED REPRESENTATION LEARNING IN SPEECH SYNTHESIS
important application of MI disentangled representation learning. In the context of representa- tion disentanglement, the extraction of meaningful latent features for high-dimensional data is chal- lenging, especially when explicit knowledge needs to be distilled into interpretable representations. One popular approach to enforce representation disentanglement is via MI minimization. Moreover, a superior disentanglement will allow a greater degree of interpretability and controllability, espe- cially for generative models maintaining high production capacity. In this section, we employ the proposed DNE- VP, estimator for MI estimation in order to lear disentangled representation, and, particularly, in the context of speech synthesis and analysis.
8

---

Under review as a conference paper at ICLR 2022
A universal sext-to-ppeech synthesizer can generate speech from text with speaker factor and speak- ing style similar to a reference signal. Previous works aimed to encode the information from refer- ence speech into fixed-length style and speaker embedding using trainable encoders (Wang al. 20188 Tjandra et al. 2020; Chien et al. 2021; TTnn et al. 2021). The major challenges for such speech synthesizers
controllability and generalisability, especially when trying generalize the models with multiple speakers and multiple styles. During training, content information is leaked into the style embeddings ("content leakage") and speaker information into style embeddings ("style leakage"). Thus at inference, when the reference speech has different content from the input text, the decoder expects the content from the style vector ignoring some part of the content text. More- over, speaker information could be expected from the style encoder leading to completely different speaker attribute. To alleviate that, Paul et al. (202T) suggested replacing the KL--based MI with Rényi-based MI and minimizing the Rényi divergence between the joint distribution and the product of marginals for the content-style and style-speaker pairs. However, reliable estimation of Rényi divergence was problematic due to high statistical variance. Taking advantage of the proposed vari- ance reduction technique, we employ a VP term in the loss function which denoted as DNE-VP, (Ra). By doing so, content, style, and speaker spaces become representative and (ideally) inde- pendent of each other. We introduce two variations of this framework: sum of three Rényi diver- gences DNE(R + 0.5 R, (i.e., sum of the corresponding objective functionals) and DNE(R.=0.5). We tested several different À values, aiming to reduce the statistical variance of the adversarial component. Notice that larger À values were helpful this application.
Table 2: Objective evaluation tests. Lower scores indicate better performance.
Methods
No Shuffle
Shuffle
RMSE-FO
MCD
WER(%) %)
RMSE-FO
MCD WER(%)
DNE (Ra=o + Ra=D.5 + Ra=1)
28.59
5.35
21.6
45.75
6.39
28.7
DNE (Ro=0.6)
28.59
5.27
18.3
47.26
6.60
26.6
DNE VP (Ra=0 + Ra=0.5 + Ro=1)
30.29
5.23
21.2
48.15
6.39
27.3
DNE- VEE-VPx=100 10 (Ro==0 + Ri= + Ro=1)
27.76
5.36
18.1
47.62
6.48
28.7
DNE- VPC-VPP== 5= (Ra=o.s)
28.69
5.87
17.3
46.53
6.72
25,4
DNE- VP (R.=
29.71
5.33
22.8
45.47
6.54
26.2
We evaluate the performance of disentanglement strategies using three performance scores from 100 random samples shown in Table [ Mel-cepstral distortion (MCD) measures the spectral distance between the synthesized and reference mel-spectrum features. Root mean squared error (RMSE) evaluates the similarity in FO modeling between reference and synthesized speech. Lastly, the con- tent preservation criterion is evaluated by word error rate (WER). During inference, evaluate the performance on two conditions: 'no shuffle' and 'shuffle'. During inference, 'no shuffle' feeds the same reference speech into style and speaker encoders and its corresponding text predict the speech features, whereas 'shuffle' feeds random speech. We observe that the proposed DNE-VP VPx variants outperform baseline approaches without VP in terms of all evaluation metrics. Our pro- posed systems greatly reduced content leakage by improving the word error rate by approximately 5-18% relative to the baseline systems. Furthermore, RMSE-FO and MCD scores show that the dis- entanglement module during training assists the TTS to achieve more accurate rendering of prosodic patterns as well as synthesizing proper speech content to its corresponding text without any signifi- cant leakage issues.
REFERENCES
David Barber and Felix V. Agakov. The IM algorithm: A variational approach to information maximization. In NIPS, pp. 201-208, 2003. ht :tattt::::/pooceo http:///oocceedddee nww.iis.......m ips. ca/
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Proceedings of the 35th International Conference Machine Learning, volume 80, pp. 531-540, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL httt:://proceedingg mlr, htt.a//.ss.mm..e.......esss press/ v8C/bel wwazelggaaiiii 18a. html

---

Under review as a conference paper at ICLR 2022
Jeremiah Birrell, Paul Dupuis, Markos A. Katsoulakis, Yannis Pantazis, and Lue Rey-Bellet. (f. )- Divergences: Interpolating between f-Divergences and Integral Probability Metrics. arXiv e- prints, art. (wX.x:20111009999 November 2020a.
Jeremiah Birrell, Paul Dupuis, Markos A. Katsoulakis, Luc Rey-Bellet, and Jie Wang. Varia- tional Representations and Neural Network Estimation of Rényi Divergences. arXiv e-prints, art. arXiv:2007.0388 033844 July 2020b.
Michel Broniatowski and Amor Keziou. Minimization of divergences sets of signed measures. Studia Scientiarum Mathematicarum Hungarica, 43(4):403-442, 2006.
Michel Broniatowski and Amor Keziou. Parametric estimation and tests through divergences and the duality technique. Journal of Multivariate Analysis, 100(1):16-36, 2009. ISSN 0047-259X. doi:
https://dooi.or//100 https://dooooo////00000////aaaap 03.011. URL https: /www. sci sciencediii rect. com/
X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. InfoGAN: Interpretable
representation learning by information maximizing generative adversarial nets. In Proceedings of the International Conference Neural Information Processing Systems, pp. 2180-2188, 2016.
Jie Cheng, Russell Greiner, Jonathan Kelly, David Bell, and Weiru Liu. Learning bayesian networks from data: An information-thest theoryy based approach. Artificial Intelligence, 137:43 90, 2002.
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In ICML 2020, July 2020. club-a-cont ve- htt.:/aaeeeeeooooooeererrrrrrrar -ttprppper--ouuudd------t--ee-e- www.oouuuuuuaaoooooouooooooouu www.mattll...omaaiiii.......
Chung Ming Chien, Jheng Hao Lin, Chien Yu Huang, Po Chun Hsu, and Hung Yi Lee. Investigating on incorporating pretrained and learnable speaker representations for multi-speaker multi-style text-to-speech. arXiv preprint arXiv:2103.0444 103.04088, 2021.
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. IV. Communications on Pure and Applied Mathematics, 36(2):183- 212. 1983.
P. Dupuis and R.S. Ellis. A Weak Convergence Approach to the Theory of Large Deviations. Wiley series in probability and statistics. John Wiley & Sons, New York, 1997. ISBN 0-471-07672-4. URL http:://papcceece inria. feteria.frrreeee htt.effrorooooooo0225555.0
A Inley-Interscience Publication.
Amedeo Roberto Esposito, Michael Gastpar, and Ibrahim Issa. Generalization error bounds via rényi-, f-divergences and maximal leakage. IEEE Transactions on Information Theory, 67:4986- 5004, 2021.
Glasserman. Monte Carlo Methods in Financial Engineering. Applications mathematics stochastic modelling and applied probability. Springer, 2004. ISBN 9780387004518. URL tttp://ooooo..oooleggggggggggg//
https://bood googloooooo..oooo.ooooooooooooo www.lel.g/booooo..eooooooooooooo btbpoooksss.d-eeeeossskkkkee
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. Generative Adversarial Nets. In Proceedings of the Annual Conference on Neural Information Processing Systems, pp. 2672-2680, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved Training of Wasserstein GANs. In Proceedings of the 31st International Conference Neural Information Processing Systems, NIPS' 17, pp. 7769,5779, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Qing Guo, Junya Chen, Dong Wang, Yuewei Yang, Xinwei Deng, Lawrence Carin, Fan Li, and Chenyang Tao. Tight mutual information estimation with contrastive fenchel-legendre optimiza- tion. CoRR, abs/2107.01131. 2021.
10

---

Under review as a conference paper at ICLR 2022
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR 2019. ICLR, April 2019. URL httts:: net/
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv:1412.6988) 2014.
Phuc H. Le-Khac, Graham Healy, and Alan F. Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 8:193907 2020. ISSN 2169-3536. doi: 10.1109/ 200.33..2000333154..
URL/http: / /x doi. org/ 10. 1119/11CEESSSS. 2020, 3031549
Jacob H. Levine, Erin F. Simonds, Sean C. Bendall, Kara L. Davis, El Ad D. Amir, Michelle D. Tadmor, Oren Litvin, Harris G. Fienberg, Astraea Jager, Eli R. Zunder, Rachel Finck, Amanda L. Gedman, Ina Radtke, James R. Downing, Dana Pe er, and Garry P. Nolan. Data-Driven Phe- notypic Dissection of AML Reveals Progenitor-like Cells that Correlate with Prognosis. Cell, 162(():184-197, 2015. ISSN 10974172. doi: 10. 10.1016/j.cell.2000 ce0.6....1.200150550047. URL http: 7/dx. 1tt.:/.../.....1111111111.111./0
2015. 05 047
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual informa- tion. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third Interna- fional Conference on Artificial Intelligence and Statistics, volume 108 Proceedings of Machine
Learming Research, pp. 875-884. PMLR, 26-28 Aug 2020. URL https: .ttll
mlr.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. hhaatpps//peeneer.eee.eeeeeeeees
Alfred Müller, Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429 443, 1997. URL http:://ww: jstor org/stable/
1428011
X. Nguyen, J. Wainwright, and M. I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions Information Theory, 56(11 1)54777)888)) 2010.
S. Nowozin, B. Cseke, and R. Tomioka. f-GAN: Training generative neural samplers using varia- tional divergence minimization. In Proceedings of the Advances in Neural Information Processing Systems, pp. 271-279, 2016.
Dipjyoti Paul, Sankar Mukherjee, Yannis Pantazis, and Yannis Stylianou. A Universal Multi- Speaker Multi-Style Text-to-Speech via Disentangled Representation Learning Based on Rényi Divergence Minimization. In Proc. Interspeech 2021, pp. 3625-3629, 2021. doi: 10.21437/ Interspeech.2021-666 2021-666.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceed- ings of Machine Learning Research, pp. 5171-5180, PMLR, 09-15 Jun 2019. URL https: //proceedings.o ml wtw.eeddiinn.m.........eeee.eeee v9
html
Christian P Robert and George Casella. Monte Carlo statistical methods; 2nd ed. Springer texts in statistics. Springer, Berlin. 2005. URL/ttts:://cda
www.dddww..eeeeeeeooooe.eeeeeeee cern. ch/ ctt./eeoooc///111777
Jiaming Song and Stefano Ermon. Understanding the Limitations of Variational Mutual Information Estimators. arXiv e-prints, art. arXiv:1910.0622) October 2019.
P. Aditya Sreekar, Ujjwal Tiwari, and Anoop M. Namboodiri. Reducing the variance of variational estimates of mutual information by limiting the critic's hypothesis space to RKHS. In ICPR, pp. 10666-10674. IEEF, 2020.
11

---

Under review as a conference paper at ICLR 2022
R. Srinivasan. Importance Sampling: Applications in Communications and Detection. Springer Berlin Heidelberg. 2013. ISBN 9783662050521. URL https://bocks google. com/
Bharath K. Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert
R. G. Lanckriet. On the empirical estimation of integral probability metrics. Electron. J. Statist, 6:1550-1599, 2012. doi: 10.1214/12-E1S72.. U 12-EUS722
Daxin Tan, Hingpang Huang, Guangyan Zhang, and Tan Lee. CUHK-EE voice cloning system for ICASSP 2021 M2VoC challenge. arXiv preprint arXiv:2103. 04,999 2021.
Andros Tjandra, Ruoming Pang, Yu Zhang, and Shigeki Karita. Unsupervised learning of disentan- gled speech content and style representation. arXiv preprint (XXi::2010.19973) 2020.
Aäron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, (lbs/180..00 807.03387, 2018. BuL_tttp./aaxxv..t///////////// 037448
Martin J. Wainwright. High-Dinmensional Statistics: A Non-Asymplotic Viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019, doi: 10.1017/ 9781 1086277777
Yuxuan Wang. Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In International Conference on Machine Learning, pp- 5180-5189. PMLR, 2018.
Liangjian Wen, Yiji Zhou, Lirong He, Mingyuan Zhou, and Zenglin Xu. Mutual information gradi- ent estimation for representation learning. International Conference on Learning Representa- lions, 2020. URL httts:://ppeneeet
Liangjian Wen, Haoli Bai, Lirong He, Yiji Zhou, Mingyuan Zhou, and Zenglin Xu. Gradient estimation information measures deep learning. Knowledge- Based Systems, 224:107046, 2021. ISSN 0950-7051. doi: http:://dl....m//.....o.....a.as
URL https:
//www. sc scwencediiiccccccooooooooo rect. wtt.ccom.cccc.. htt.a//cw......aaaa...aaooooaoaa
12


Reviews:
[{"review_id": "rUDascNV49G", "reviewer": "Reviewer_gwrQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper a highly topical problem is tackled, how to obtain low-variance divergence estimator. Authors decide to use variance penalty. Idea is quite good and synthetic experiments do seem to validate that reduction works. \n\nI have only following points: \n- About Fig 2, can you comment about why for CLUB VP does not seem to help so much as for DNE? Variance reduction, at least visually, seems to be quite minimal.\n- It appears to be that there were no other variance reduction techniques compared, why not to compare against for example control variates?\n- Why there is no Conclusions ? \n", "summary_of_the_paper": "In this paper authors propose to an new neural-based divergence estimator using variance penalty.", "summary_of_the_review": "Good contribution on an important topic.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ed7Xx3pcnTP", "reviewer": "Reviewer_3SLV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(1) In Section 3.1, $G$ in Eq. (13) seems to override $f^*$ in previous equations, and $h$ is defined in the context. It is vague to understand how to derive Eq. (14) and Eq. (15) from Eq. (13). The authors should carefully take care of notations. For example, from (13) to (14), the author should at least denote $G=f^*$ and $h=\\exp$.\n(2) Theorem 1 is missing. Theorem 2 could be inappropriate and insufficient to be considered as a theorem.\n(3) The assumption for Theorem 4. is not well described. The authors defer the details in Appendix B., but the proof which the author indicated only proves the divergence property of Renyi divergences instead of other divergences.\n(4) In Section 3.2, $E$ is not defined and is ambiguous with $E$ which represents the expectation. The author should consider using other notations.\n(5) Algorithm 1 is overly simplified and incorrect. Line 3 in Algorithm 1 implies that the proposed variance penalty is not included during optimization; therefore the variances would not be suppressed. Also, since $\\alpha$ is used by Renyi divergence, the authors should denote the parameter of Adam by another symbol other than $\\alpha$.\n(6) Figure 1 did not comprehensively analyze the trade-off of bias and variance. In addition, it is necessary to explain why the authors choose MedAE instead of MSE.\n(7) The trade-off between bias and variance is unsatisfactory. For instance, VP does not affect InfoNCE. Moreover, to greatly reduce variance, DNE-VP exhibits large biases.\n(8) The second synthetic experiment seems problematic. First, as we know that InfoNCE is bounded above by $\\log N$, where $N$ is 64 according to the caption in Figure 2. Therefore, InfoNCE should saturate to about 4.159, but the authors' experiment shows that the estimated MI saturates below 4. Second, the author examined estimators under different scales of the ground-truths.\n(9) The authors might have overlooked that as they adopted the Delta method, the estimators should be guaranteed to be ${\\it consistent}$. The authors did not prove that all estimators included in their experiments are consistent.\n(10) The authors did not introduce the model architectures and datasets used in the speech synthesis experiments (for both speech synthesizer and automatic speech recognition systems).  Without such information, readers cannot reproduce the results easily, and thus the results reported in the paper are not convincing.  \n", "summary_of_the_paper": "The paper proposes to adopt a variance penalty term to manipulate the divergence estimators. Experiments first investigated the correlations of the variance penalty and the bias and variance trade-offs. Then, the authors tested the proposed approaches on two real-world application (biological and speech synthesis) datasets. The results show that the proposed approach can effectively reduce the variance as compared to systems without including the variance penalty. ", "summary_of_the_review": "The proposed idea is reasonable. However, some key information (including notation definitions and experimental setups) is not well described, which may make readers difficult to follow the main ideas and reproduce the results.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DciGPDstodb", "reviewer": "Reviewer_KwzF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\nThe paper clearly motivates the problem of high-variance in the estimation of the statistical divergences . Despite being heavy on notation, the manuscript is well-written and easy to follow, and the proposed solution is clearly laid-out. \n\nThe description of the method seemed mostly correct to me, although I did not verify the mathematics in detail.  The theoretical results are also a useful addition.\n\nThe experimental section assesses some of the properties of the proposed estimator sufficiently.\n\n\n**Weaknesses:**\n\nOne of my concerns is that this paper does not compare to other variance reduction methods. The paper mentions that there are some other methods in the literature that apply techniques such as antithetic sampling to reduce the variance, but I did not see a direct comparison to such methods. How would they fair against the variance penalty method? Can the VP be used to supplement them?\n\nRegarding the bias issue of the proposed estimator, would decaying the regularisation coefficient during the optimisation process help with alleviating this issue? If yes, I think this point can be explored in the experiments as a “best-of-both-worlds” approach.\n\nI spotted some linguistics errors, e.g. \"such as appear in equation (1)\nand equation (8)” and  “there we also prove that”.\n\nMinor point: Missing error bars on MedAE in Figure (1).\n", "summary_of_the_paper": "This paper takes the variational view of the statistical estimation of statistical divergences. Standard variational formulas suffer from high variance when estimated with finite samples, this paper proposes to address the problem by adding a variance penalty regularisation term to the objective functional, thus trading variance for bias. The bias-variance trade-off is controlled via the regularisation coefficient. From this new formulation, the paper derives a new low-variance Neural Divergence Estimation Algorithm with the variance regularisation technique.", "summary_of_the_review": "In my opinion, this work is original and well-presented. The paper discussed the presented method in-depth and the experimental evaluation was well-constructed. I think, overall, this work is a useful and significant contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AASaSRwac2W", "reviewer": "Reviewer_JtE9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper present an effective method to reduce the variance of divergence estimators based on variational bounds. In summary, the strengths and weaknesses of the paper are listed below: \n\n** strengths **\n- The investigated problem is relevant and the paper is well-written and the contributions clearly presented. \n- For the considered synthetic scenarios the proposed approach improves the mean square error which seems to validate the underlying hypothesis of the regularzation term introduced to correct the variance. \n- Although the main idea of exploiting the asymptotic expression of the variance (based on the delta method) to introduce a penalty term that controls the variance is not surprising, the application within the framework of estimation of diverges based on variational bounds and the effectiveness of the method seems to be interesting.  \n\n** weaknesses **\n- The main misconception of this work relies on the fact that the synthetic experiments provided in Section 4 are too simplistic to be able to validate the proposed method. More precisely,  (i) the regimes under which the estimators are investigated are rather optimistic since a large amount of samples are considered (e.g. 512K); (ii) there is no experiments on multimodal (mixtures) distributions or other distributions than Gaussian distributions (e.g., there are many possible examples where the underlying information measure does not change but the the distribution of the samples can be very much different by introducing complex transformations); (iii)  there is no studies analysing the effects of the batch size, the effects of the underlying optimisers used to obtain the variational  lower bounds (e.g., what about the use of batch normalisation?; (iv) there is no investigation of the varice reduction under realistic high dimensional datasets (e.g. SVHN, imagenet, etc.)\n- The empirical results are compared with some previous works but not state-of-the-art estimators. For example, CLUB has been improved in [A] and see references in therein for other missing methods and possible additional non-Gaussian scenarios to study. Indeed, this estimator is also applied to the problem of disentanglement and thus, it is relevant to compare with the results obtained on Section 5.2. \nA comparison with SOTA methods is fundamentally important to validate the effectiveness of the proposed method. Which optimiser is used for the experiments in Fig. 2 ?\n- The resulting estimator after the variance regularization should also be compared to the work in [B] which  studies for a particular case the asymptotic variance the f-divergence estimators. What can be said about the variance of the proposed estimation with the framework of [B] ? \n- It is claimed that \"the majority of the existent estimators for MI are not transferable to the general estimation of the divergences and frequently produce instabilities during training\". This claim is not valid or at least not clear enough since the estimation methods that will be consider in this paper follow the same ideas (i.e., they are based on variational bounds). \n- Similar comments apply to the empirical results presented in sections 5.1 and 5.2. For example, there is no comparison of the variance reduction compared to the reference [C] which is particular relevant for the investigated framework in Section 5.2. \n\n** References ** \n[A] A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations by P. Colombo et al, ACL 2020. \n[B] Practical and Consistent Estimation of f-Divergences by Paul K. Rubenstein et al., NeurIPS 2019. \n[C] Adversarial Disentanglement of Speaker Representation for Attribute-Driven Privacy Preservation by Paul-Gauthier Noé et al., Interspeech 2021\n", "summary_of_the_paper": "This paper introduces a novel variance regularization to reduce the statistical variance of the  variational based estimators for f-divergences and Renyi's divergence. The proposed regularization is based on the well-known delta method which provides the asymptotic variance and  depends on the specific from of the variational bound under consideration. The approach is tested on different synthetic data sets for which the numerical results show that the variance is decreased relatively to the baseline estimator. In particular, this effects appears be more visible for high-orders of the Renyi's divergence. Finally, an application is provided for real biological datasets, disentanglement for speech signals into text, speaker and style components.  ", "summary_of_the_review": "The paper introduces some novel and interesting ideas. However, the numerical experiments  provided  in this paper are neither complete nor  always relevant. In particular, comparison with state-of-the-art estimators are clearly missing, the synthetic datasets are too simplistic  to validate the proposed method, the regimes (in terms of samples) are not relevant and many other effects related to the choice and impact of the optimiser, batch size, etc... are missing. I cannot recommend the acceptance of this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new class of divergences that are also sensitive to the variance of the estimator. The proposed additional variance penalty term introduces a bias term and acts directly on each component of the statistical estimator. By choosing the penalty parameter one can trade bias versus variance. 

The results on synthetic examples look promising and suggest that with this technique, it is feasible to decrease the estimation error relative to the baseline statistical estimator. This is demonstrated to be particularly pronounced for certain Renyi divergences in the large order parameter alpha regime. Two applications (detection of subpopulations and disentangled representation learning in speech) are provided.

The opinions about the work were fairly divided. Both positive reviews have lower confidence and are rather short and do not fully justify the high rating. Two high confidence reviews are negative and raise several critical points. In a nutshell, reviewer JtE9 complains mainly about the insufficient experimental evaluation while 3SLV raises several concerns regarding readability, mathematical notation, lacking details of the proofs, as well as technicalities regarding the consistency. The authors have partially answered the concerns.

While there seems to be a consensus that the paper is interesting and makes a valid contribution, the introduction of the VP term defines a new estimation problem and both the choice and interpretation of lambda becomes critical. In particular, the key question is understanding the effect of lambda for various tasks where divergence estimation is crucial and I am not fully convinced if the chosen applications are the best for convincingly demonstrating the utility as these require somewhat application specific motivation. I would rather see result of standard benchmark datasets (such as estimating the divergence between two subsets of MNIST images to detect subtle distribution shifts).

The synthetic experiments are good but this section could be improved as well to get the message accross. Rather than delving directly to the findings, this section could first justify what needs to be measured and what are the control variables (number of samples, Renyi order etc)

In light of the comments raised by the reviewers, I feel that this paper can benefit from a further iteration and clarification of the experimental section before being accepted to a venue like ICLR.
Decision: Reject