Function-space regularized Rényi divergences
Authors: Jeremiah Birrell
Yannis Pantazis
Paul Dupuis
Luc Rey-Bellet
Markos Katsoulakis
Venue: ICLR 2023 poster (2023)
Keywords: Rényi divergence
integral probability metrics
variational formulas
worst-case-regret

Abstract: We propose a new family of regularized Rényi divergences parametrized not only by the order $\alpha$ but also by a variational function space. These new objects are defined by taking the infimal convolution of the standard Rényi divergence with the integral probability metric (IPM) associated with the chosen function space. We derive a novel dual variational representation that can be used to construct numerically tractable divergence estimators. This representation avoids risk-sensitive terms and therefore exhibits lower variance, making it well-behaved  when $\alpha>1$; this addresses a notable weakness of prior approaches. We prove several properties of these new divergences, showing that they interpolate between the classical Rényi divergences and IPMs. We also study the $\alpha\to\infty$ limit, which leads to a regularized worst-case-regret and a new variational representation in the classical case. Moreover, we show that the proposed regularized Rényi divergences inherit features from IPMs such as the ability to compare distributions that are not absolutely continuous, e.g., empirical measures and distributions with low-dimensional support. We present numerical results on both synthetic and real datasets, showing the utility of these new divergences in both estimation and GAN training applications; in particular, we demonstrate significantly reduced variance and improved training performance.

Published as a conference paper at ICLR 2023
FUNCTION-SPACE REGULARIZED RÉNYI DIVERGENCES
Jeremiah Birrell', Yannis Pantazis", Paul Dupuis Luc Rey-Bellet', Markos A. Katsoulakis' 'University of Massachusetts, Amherst, Foundation for Research & Technology Hellas, 3 BBrown University, fjbirrell, luc, markos} @ @umass.edu, pantazis @ fach.forth.gr, poul__dpuis@ee @ brown.edu
ABSTRACT
We propose a new family of regularized Rényi divergences parametrized not only by the order but also by variational function space. These new objects are defined by taking the infimal convolution of the standard Rényi divergence with the integral probability metric (IPM) associated with the chosen function space. We derive a novel dual variational representation that can be used to construct numerically tractable divergence estimators. This representation avoids risk-sensitive terms and therefore exhibits lower variance, making it well-behaved when a 1; this addresses notable weakness of prior approaches. We prove several properties of these new divergences, showing that they interpolate between the classical Rényi divergences and IPMs. We also study the - so limit, which leads to a regularized worst-case-regre and a new variational representation in the classical case. Moreover, show that the proposed regularized Rényi divergences inherit features from IPMs such as the ability to compare distributions that are not absolutely continuous, e-g., empirical measures and distributions with low- dimensional support. We present numerical results on both synthetic and real datasets, showing the utility of these new divergences in both estimation and GAN training applications: in particular, we demonstrate significantly reduced variance and improved training performance.
INTRODUCTION
Rényi divergence, Rényi 1961. is a significant extension of Kullback-Leibler (KL) divergence for numerous applications; see, c.g.. Van Erven & Harremos (2014). The recent neural- estimators for divergences Belghazi et al. 2018) along with generative adversarial networks (GANs) Goodfellow et al. (2014) accelerated the use of divergences in the field of deep learning. The based divergence estimators are feasible through the utilization of variational representation formulas. These formulas are essentially lower bounds (and, occasionally, upper bounds) which are approximated by tractable statistical averages. The estimation of a divergence based on variational formulas is a notoriously difficult problem. Challenges include potentially high bias that may require an exponential number of samples McAllester & Stratos (2020) or the exponential statistical variance for certain variational estimators (Song& & Ermon (2019), rendering divergence estimation both data inefficient and computationally expensive. This is especially prominent for Rényi divergences with order larger than 1. Indeed, numerical simulations have shown that, unless the distributions /' and Q are very close to one another, the Rényi divergence R., (PIQ) is almost intractable to estimate when > 1 due to the high variance of the statisically-approoimaaee risk-sensitive observables Birrell et al. (021), see also the recent analysis in Lee & Shin (2022). A similar issue has also been observed for the KL divergence, /Song& & Ermon (2019). Overall, the lack of estimators with low variance for Rényi divergences has prevented wide-spread and accessible experimentation with this class of information-theoree theoretic tools, except very special cases. We hope our results here will provide suitable set of tools to address this gap in the methodology.
One approach to variance reduction the development of new variational formulas. This direction is especially fruitful for the estimation of mutual information van den Oord et al. 2018); Cheng et al. (2020). Another approach is to regularize the divergence by restricting the function space of the variational formula. Indeed, instead of directly attacking the variance issue, the function space of the variational formula can be restricted, for instance, by bounding the test functions or more appropriately by bounding the derivative of the test functions. The latter regularization leads to

---

Published as a conference paper at ICLR 2023
Lipschitz continuous function spaces which are also foundational to integral probability metrics (IPMs) and more specifically to the duality property of the Wasserstein metric. In this paper we combine the above two approaches, first deriving new variational representation of the classical Rényi divergences and then regularizing via an infimal-convolution as follows
R..1C (PIIQ) = in (PIl)) + W WT n),
(I)
where P and Q are the probability distributions being compared, the infimum is over the space of probability measures, Rs is the classical Rényi divergence, and W is the IPM corresponding to the chosen regularizing function space, I.
The new family of regularized Rényi divergences that are developed here address the risk-sensitivity issue inherent in prior approaches. More specifically, our contributions are as follows.
We define a new family of function-space regularized Rényi divergences via the infimal convolution operator between the classical Rényi divergence and an arbitrary IPM (T. The new regularized Rényi divergences inherit their function space from the IPM. For instance, they inherit mass transport properties when one regularizes using the I-Wasserstein metric.
We derive a dual variational representation [T] of the regularized Rényi divergences which avoids resk-sensitive terms and can therefore be used to construct lower- variance statistical estimators.
We prove a series of properties for the new object: (a) the divergence property, (b) being bounded by the minimum of the Rényi divergence and IPM, thus allowing for the comparison of non-absolutely continuous distributions, (c) limits as a - from both left and right, (d) regimes in which the limiting cases R. (PIIQ) and (V, are recovered.
We propose rescaled version of the regularized Rényi divergences (16) which lead to a new variational formula for the worst-case regret (i.e., a - oo). This new variational formula does not involve the essential supremum of the density ratio as in the classical definition of worst-case regret, thereby avoiding risk-sensitive terms.
We present a series of illustrative examples and counterexamples that further motivate the proposed definition for the function-spar space regularized Rényi divergences.
We present numerical experiments that show (a) that we can estimate the new divergence for large values of the order a without variance issues and (b) train GANs using regularized function spaces.
Related work. The order of Rényi divergence controls the weight put on the tails, with the limiting cases being mode-covering and meleeselection Minka (2005). Rényi divergence estimation is used
in a number of applications, including Sajid et al. (2022) (behavioural sciences), Mironov| (011)) (differential privacy), and [i & Turner (2016) (variational inference); in the latter the variational
formula is an adaptation of the evidence lower bound. Rényi divergences have been also applied in the training of GANs Bhatia et al. (2021) (loss function for binary classification discrete case) and in Pantazis et al. (2022) (continuous case, based on the Rényi-Donskee Varahdan variational formula in Birrell al. (202T)). Rényí divergences with a: are also used in contrastive representation learning, Lee & Shin (2022. as well as in PAC-Bayesian Bounds, Bégin et al. (2016). In the context of uncertainty quantification and sensitivity analysis, Rényi divergences provide confidence bounds for rare events. Atar et al. (2015): Dupuis et al. 2020), with higher rarity corresponding to larger a.
Reducing the variance of divergence estimators through control of the function space have been recently proposed. In |Song & Ermon (2019) an explicit bound to the output restricts the divergence values. A systematic theoretical framework on how to regularize through the function space has been developed in DDpuis, Paul & Mao, Yixiang (2022): Birrell et al. (2022a) for the KL and fddivergences. Despite not covering the Rényi divergence, the theory in Dupuis, Paul & Mao, Yixiang (2022); Birrell formulation clearly inspired the current work. However, adapting the infimal-convolution method to the Rényi divergence setting requires two new conjugate variational formula for the classical Rényi divergence in Theorem 2.1 (see also Fig. D. allowing us to apply infimal- convolution tools develop the new I'-Rényi divergences in Theorem 3.4 (b) We study the a + oo limit of (a) to obtain a new low-variance variational representation of worst-case regret in Theorem 2.2 and study its I r-regularization in Theorem 4.5
et al. (2022a) and particularly the
technical innovations: (a) We develop a new low-variance
2

---

Published as a conference paper at ICLR 2023
2 NEW VARIATIONAL REPRESENTATIONS OF CLASSICAL RÉNYI DIVERGENCES
The Rényi divergence of order E (0.1) U (1, oo) between P and Q, denoted R. can be defined as follows: Let be a sigma-finite positive measure with dP pdv and dQ = qdv. Then
if < a < 1 or and P < Q i and P Q,
Ra (PIQ) = a[- log pa
(2)
+00
where P & Q denotes absolute continuity of P with respect to Q. There always exists such (e.g.. ++Q) and one can show that the definition (2) does not depend on the choice of v. The Ra provide notion of 'distance' between and Q2 in that they satisfy the divergence property, i.e. they are non-negative and equal zero iff Q = P. The limit of R., as ar approaches or 0 equals the KI. or reverse KI. divergence respectively Van Erven & Harremos (2014).
An alternative representation of R.. the so-called Rényi-Donsker- Varadhan variational formula, was derived from (2) Birrell et al. (202I).
aa-1 log / ela-1)odP log / (eved)) e.Q)P())). (3)
1
Ra (PI sup (EM)())
Here M) denotes measurable space, M&(Ω)) the space of bounded measurable real-values functions on St and P(2) the space of probability measures on Ω2. By a change of variables argument this can be transformed into the following variational representation;
Theorem A.2
in Appendix A for a proof. We call the convex-conjugate Rényi variational formula (CC-Rényi).
Theorem 2.1 (Convex- Connex-CCnjuuatee Rényi Variational Formula), Let P. P().) and a (),1)u
(1, oo). Then
{ gdQ - log 19/ + = + 1). (4)
(P(l)))==
sup (EMa(R)aao
If (2, M) is metric space with the Borel a-algebra then (4) holds with My(S2) replaced by (s))), the space of bounded continuous real-valued functions on $2.
The representation 44 is of convex-conjugate type, which will be key in development of function- space regularized Rényi divergences. It is also of independent interest as it avoids risk-sensitive terms, unlike (3) which contains cumulant-generating -tnuuttttiinn-ftnctioon... This makes (4) better behaved in estimation problems, especially when a > 1; see the example in Section 6. below.
We also obtain new variational formula for worst-case regret, defined by Van Erven & Harremos (2014)
log 100.
dP ess supp aQ
P P
(x = lim aR,
(5)
P P
In contrast to (5) which requires estimation of the likelihood ratio, the new variational formula (6) below avoids sensitiie terms.
Theorem 2.2 (Worst-case Regret Variational Formula). Let P, E P(Q). Then
{ gdQ + log |oldP] +1.
Dx
sup gEMs(D))c0
If Q is a metric space with the Borel a-algebra then 16) holds with Mo(S) replaced by Cs()).
See Theorem A.5 in Appendix A for a proof. Equation 66) is a new result of independent interest and will also be useful in our study of the a - oo limit of the function- space regularized Rényi divergences that we define in the next section.
Remark 2.3, Alternative variational formulas for Dro on finite alphabet were derived in Kurri al. 2022
3

---

Published as a conference paper at ICLR 2023
3 PRIMAL AND DUAL FORMULATIONS OF THE INFIMAL-CONVOLUTION r-RÉNYI DIVERGENCES
We are now ready to define the function-space regularized Rényi divergences and derive their key properties. In this section, X will denote a compact metric space, P(X) will denote the set of Borel probability measures on X, and ((X) will denote the space of continuous real-values functions on X. We equip CXX) with the supremum norm and recall that the dual space of C(X) is C(X)* ( the space of finite signed Borel measures on X (see the Riesz representation theorem, c.g., Theorem 7.17 in Folland (013)).
Definition 3.1. Given a test-function space I C C(X), we define the infimal-convolution r-Rényi divergence (i.e., IC-I--Rényi divergence) between P,Q P(X) by RC
inf (R. W (Q.1)), (1,)), TEP(X)
where WT denotes the T-IPM
/ gdp gdy- gdv), 1, E M(X).
W (H, = supl geT
(8)
Remark 3.2. The classical Rényi divergence is convex in its second argument but not in its first when > Van Erven & Harremos (2014). This is the motivation for defining the IC-T-Rény: divergences via an infumal convolution in the second argument of Ra; convex analysis tools will be critical in deriving properties of R.. below. For a E (0. 1) one can use the identity R., (PllQ) = Rí R1-a(Q1)P) to rewrite as an infimal convolution in the first argument.
The definition 7 can be thought of as a regularization of the classical Rényi divergence using the T-IPM. For computational purposes it is significantly more efficient to have dual formulation, iee, a representation of RT.IC in terms of a supremum over a function space. To derive such a representation we begin with the variational formula for Ra from Theorem 2.1 If define the
convex mapping Ag C(X) + (0),
(01 log f 12)10 dP + a (log + 11) 1gco:
A [g] 1
then (4 from Theorem 2.1 can be written as a convex conjugate
1f /ddQ A [9]].
(a(P)Q) [Q] == sup
(10)
gec(x)
One can then use Penchel-Rockafelle duality to derive a dual formulation of the IC-I-Rényi divergences. To apply this theory we will need to work with spaces of test functions that satisfy the following admissibility properties. These properties are similar to those used in the construction of regularized KL and f-divergences in Dupuis, Paul & Mao, Yixiang (2022) and Birrell et al. (2022a).
Definition 3.3. We will call r c C(X) admissible it is conver and contains the constant functions. We will call an admissible I' strictly admissible if there exists ) detternninn set V C(X) such that for all vé E & there exists E R, > O such that + oáp E Т. Recall that V being P P(X) -etermminn means that for all Q, P E P(X), if bdQQ f @dP for all up then Q P.
Putting the above pieces together one obtains the following variational representation.
Theorem 3.4. Let I' C. C(X) be admissible, P.Q P(X), and (0, (1) Then:
1.
RD RC (P(()) = sup ger'o<o
(/ gdQ
log / Igl + a + 1).
(11)
2. If 11 is finite then there exists n) E P(X) such that
PE..C((P()) = inf (Ra(P) + WT (Q.1)) (a(())))) (.]][n=+++)]) + (T(Q...) (12)
TEP(X)
4

---

Published as a conference paper at ICLR 2023
3. (PIC(( S (in
4. IfT is strictly admissible then R. has the divergence property.
See Theorem B.3 in Appendix B for detailed proofs of these results as well as several additional properties. We note that there are alternative strategies for proving the variational formula (11) which make different assumptions; further comments on this can be found in Remark B.4 Important examples of strictly admissible I' include the following:
1. r = C(X), which leads to the classical Rényi-divergences.
2. r = Lip'(X), i.e. all I-Lipschitz functions. This regularizes the Rényi divergences via the Wasserstein metric.
3. r g R.g E C(X), [gl S 1). This regularizes the Rényi divergences via the total-variation metric.
4. r = {cc 1c+gg: CE R,g E Lip' (X), g S 1}. This regularizes the Rényi divergences via the
Dudley metric.
5. l' = {c+ CER.gee regularizes the Rényi divergences via MMD.
||g|lv < 1}, the unit ball in a RKHS V C C(X). This
In practice, uniform bounds can be implemented using an appropriately chosen final NN layer. Lipschitz bounds can be implemented using spectral normalization of neural networks Miyato et al. (2018), or using a soft gradient penalty Gulrajani et al. (2017). The function space I for structure- preserving GANs discussed in the Appendix implemented using equivariant neural networks, Birrell et al. 2022b If I is ball in an RKHS space the implementation is carried out using the same tools used in, e.g., MMD distances and divergences, Gretton et al. (2012); Glaser et al. (02TT)
The IC-F-Rényí divergences also satisfy data processing inequality. See Theorem B.8 in Appendix for a proof as well as details regarding the notation.
Theorem 3.5 (Data Processing Inequality). Let (0.1) (L, oo). Q,P P(X), and K be probability kernel from X to Y such that K[g] E C(X) for all g E C(X,Y). IfT c C(Y) is
admissible then R..IC (K(P]IKIQ) S Ra KTT,, (P(I)) IfT c C(X is admissible then R RD.C (P & K|[Q & K)= S R IT (PIQQ).
If T is strictly contained in I then the bounds in Theorem 3.5 can be strictly tighter than the classical data processing inequality Van Erven & Harremos (2014) Data- pata-processing inequalities are
important for constructing symmetryy-reserring GANs; Birrll al. (022b) and Section D.
LIMITS, INTERPOLATIONS AND REGULARIZED WORST-CASE REGRET
Next we use Theorem 3.4 to compute various limits of the IC-I'-Rényi divergences. First we show that they interpolate between R,, and W" in the following sense (see Theorem B.5 for a proof). Theorem 4.1. Let I' C C(X) be admissible, P, Q E P(X), and a (0, 1) U (1)
1. lim (P = ().QPP).
2. IfT is strictly admissible then lim R RTT.IC(PIIQ) = R. (PIQ).
Now we discuss the limiting behavior in a. These results generalize several properties of the classical Rényi divergences Van Erven & Harremos 2014). First we consider the + limit; see Theorem B.6 for a proof.
Theorem 4.2. Ler I c C(X) be admissible and P,Q E P(X). Then
lim RC (PI[Q) (C = inf
(R(Pl)n) + (
(13)
a-1+
MPP(X):
38>1,R((Pin)
lim R (PI()) = inf
πEP(X) (R(P)))) +
(14)
a-11-
sup { /fdd0. /ddQ+//ooloddp)) logloldP) ser
(15)
5

---

Published as a conference paper at ICLR 2023
transforms 15) into the Legendre-transfen transform variational formula for R R(P]]Q):: see equation (1) in Birrell et al. (2022c) with f(x) r log(z). Eq. 414y is an infimal convolution of the reverse KL-divergence, opposed to the results in Dupuis, Paul & Mao, Vixiang 2022) which apply to the (forward) Kl.-divergence.
Remark 4.3. When I = C(X), changing variables -
Function-space regularized worst-case regret. Next investigate the IC-T-Rényi divergences, which will lead to the function-space regularized worst-case regret. First recall that some authors to the one used in this paper by D-(-Il-) = aR (-I-)- This alternative definition has the useful property of being non-decreasing in at see Van Erven & Harremos (2014), Appropriately rescaled, the IC-I-- Re-nyi divergence also satisfies this property, leading to the following definition. Definition 4.4. For C C(X), (0, 1) (1, and P,Q P(X) we define
limit of the
alternative definition of the classical Rényi divergences, related
(PI()) / /O. (PIIQ))
(16)
Note that //, / (PIIQ) is non-decreasing in at see Lemma B. for a proof. We now show that the divergences JC are well behaved in the + oo limit, generalizing (5). Taking this limit
provides a definition of function-spact spaccnn-spceeeeeeee regularized worst-case regret, along with the following dual variational representation.
Theorem 4.5. Let r C C(X) be admissible and P,Q E P(X). Then
(PPI(() lim 1
inf nEP(X)
(Pol) + (Q,
(17)
{ 9QDQ + log (slde) +1.
sup geTgg<D
(18)
We call DF.IC the infimal-convolution l'worrt-case regret (i.e.,
The method of proof
of Theorem 4.5 is similar to that of part (1) of Theorem (344, see Theorem B.7 in Appendix for
details. Theorem 4.5 suggests that D is the appropriate a-scaling when is large and find this to be the case in practice; see the example in Section 6.3.I
5 ANALYTICAL EXAMPLES AND COUNTEREXAMPLES
In this section we present several analytical examples and counterexamples that illustrate important properties of the IC-I--Rényi divergences and demonstrate weaknesses of other attempts define regularized Rényi divergences. In particular, we show that other attempts at regularizing Rényi divergences fail to inherit important properties from the I-IPM. More details on the computations can be found in Appendix C
Infimal convolution and scaling limits: First present simple example that illustrates the infimal convolution formula and limiting properties from Sections Jad Let P do, Qx,e
for e E (0, 1). x > O, and let I = Lip'. Then for L > 0 one can compute
aIr <1 a - cLa + + log(a<L)) 1 aLx < 1/c
(1 c)
R.r (PIlQz.))
(19)
a log(1/), aLx 1/c
In particular, it is straightforward to show that ( (1- WLT P). lim IL (P)))ee) = lim,-o+ (1 - Lr = 0, and lim RLT,IC (PI]Que))
log(1/c) R. (z,)) We can also rewrite this in terms of the solution to the infimal convolu- tion problem and take the worst-case-reee regret scaling limit as follows
wrr Whr (Qr.e, 0 aLx <1
R RLT.IC(P]]Q... =
R. (PIl + WyL((Qu,) (u.1/oLa)); 1 1 aIx $ 1/c
Ra (PllQz.))) aLx > 1/c
(T..r P), 0<
lim /a///.CC (P)(Qx.)) = (PI(( + WT (Qz,es Q2,1/5), 15x5 1/c (20)
a-00c
Doo (P((Qu)))))) 1/c

---

Published as a conference paper at ICLR 2023
I- T-Rényi-Donsker- Daaniii-onnse---aaadhaann counterexample: As an alternative to Definition 3.1 one can attempt to regularize the Rényi divergences by restricting the test- rest-function space in the variational representation 3 leading to the r- T-Rényi-Donsker- Varadhan (T' (T--ényi-DV) divergences
1 a aa-1 log
ela-1)) 1)d dP log / c00 dQ
1
(PIIQ) == sup R
(21)
NET
The bound log f e dP 2 c odP for all á E T.,C E R implies that R. I WT for a E (0, 1). making (2T) a useful regularization of the Rényi divergences in this case; this utility was demonstrated in Pantazis et al. (2022), where it was used to construct GANs. However, estimators built from the representation (2) (i.e., replacing P and Q by empirical measures) are known to be numerically provide counterexample showing that, unlike for the IC-I"-Rény: divergences, Rt DV z W in general when a > 1. We conjecture that this is a key reason for the instability of Le---Reni-DDossee-WWWe Varaaaaaaaee-eaaaaaa Varadhan estimators when o >> 1.
unstable when > I. Below
Let Pa.c. = con + (1 c) Q = dn for x > 0, E (0. and F Lip". Then for a >> 1 we have R...DV (Px.c (Qx.cllQ)
log (c + (1 - c) exp((a - 1))r)) and WTz (Pa,c. Q) (1 c) Lx. Using strict concavity of the logarithm one can then obtain the bound
RE (Pa,ell)) > (F'(Pa,. Q)
(22)
This shows that, when a > 1, Rénéyi--D violates the key property that allows the IC-I-Rényi divergences inherit properties from the corresponding I-IPM. Another alternative is to begin with (3) and then reduce the test-function space to _ log(T), where the logarithm is introduced to eliminate the exponential functions in (2I). However, this definition also fails to provide an appropriate regularized Rényi divergence; in particular, it is incapable of meaningfully comparing Dirac distributions. See Appendix C.3 for details. These counterexamples lend further credence to our infimal-conollution based regularization approach (7).
6 NUMERICAL EXPERIMENTS
In this section we present numerical examples that demonstrate the use of the IC-I'-Rényi divergences for both estimation and training of GANs (additional examples can found in Appendix [D). All of the divergences considered in this paper have variational representation of the form D(PIIQ) =
ger H [g; P, Q] for some objective functional H;; we use the corresponding estimator
D. = sup H [o: Pa, Qn] o
(23)
where Pn. Qn are n-sample empirical measures and go is a family of neural networks (NN) with parameters O E O For Lipschitz function spaces we weaken the Lipschitz constraint to a soft 1-sided gradient penalty (see Section 4.1 of Birrell et al. (2022a)). Optimization is performed using the Adam optimizer Kingma& & Ba (2014). For the infimal convolution divergences we enforce negativity of the test function (i.e., discriminators) using a final layer having one of the following forms: 1) I)-abs(1) or 2)- - (11/(1 2) + (1 =2)2) The latter, which term poly- poly-softplus, is C' and decays like (z as x x------ -00
6.1 VARIANCE OF RÉNYI ESTIMATORS
As first example, compare estimators of the classical Rényi divergences (i.e., without regular- ization) constructed from DV-Rényi (3) and CC-Rényi (4) in simple case where the exact Rényi divergence is known. We let Q and P be 1000-dimensional Gaussians with equal variance and study (PllQ) as a function of the separation between their means. The results are shown in Figure] | We see that the estimator based on the convex- conjuute Rényi variational formula 4has smaller variance and meaansquared error (MSE) that the Rényi-Donsker Varadhan variational formula 3 with the difference becoming very large when a >> or when and Q are far apart (i.e., when 14g is large). The Rényi-Donsker Varadhan estimator only works well when M4 and a are both not too large, but even in such cases the convex-conjugate Rényi estimator generally performs better. We conjecture that this difference is due to the presence of risk- sensitive terms in which were eliminated in the new representation 4 We note that the NN for the convex-come conjx--co Rényi estimator used the poly-softplus final layer, as found the -abs final layer to result in significant percentage of failed runs (i.c., NaN outputs) but this issue did not arise when using poly-sofiplus. We do not show results for either DV- or CC-WCR here the exact divergence is infinite in this example.
7

---

Published as a conference paper at ICLR 2023
DV-Ra
DV-R2
10
CC-R
DV-Ra
DV-R2
OC-Ha
CC-Rs
DV Rio
DV-R8e
103
OC--ie
10
CC Roa
13V-Rut
CC-Rie
MSE
10
10
10
10-10
10
10 10 1E 10"
10" 10"
101
Figure 1: Variance and MSE of estimators of the classical Rényi divergence between 1000- dimensional Gaussians. DV-R,, refers to Rényi divergence estimators built using (3) while CC-R, refers to estimators built using our new variational representation (4) We used NN with one fully connected layer of 64 nodes, ReLU activations, and a soly-softplus final layer (for CC-Rényi). We trained for 10000 epochs with a minibatch size of 500. The variance and MSE were computing using data from 50 independent runs. Note that the CC-Rényi estimator has significantly reduced variance and MSE compared to the DV-Rényi estimator, even when a is large. Strikingly, the 1-D case exhibits the same behavior (see Figure 3 in Appendix D.2), demonstrating that the DV-Rényi estimator is unsuitable even in low dimensions.
6.2 DETECTION OF RARE SUB-POPULATIONS IN SINGLE-CELL BIOLOGICAL DATASETS
A critical task in cancer assessment is the detection of rare sub-populations subsumed in the overall population of cells. The advent of affordable flow and mass cytometry technologies that perform single cell measurements opens a new direction for the analysis and comparison of high-dimenssonal cell distributions Shahi et al. (017)) via divergence estimation. We consider single cell mass cytometry measurements 16 bone marrow protein markers (d 16) coming from healthy and disease individuals with acute myeloid leukemia et al. 2013). Following Weber et al. (2019). we create datasets: with only healthy samples and another with decreasing percentage of sick cells and compute several divergences. Considering the estimated divergence value as the score compute the ROC curve and the respective area under the ROC curve (AUC)
of binary classifier, for any pair of sample distributions. More specifically, true negatives correspond to the divergence values between two healthy datasets while true positives correspond the divergence between healthy and a diseased dataset. Thus, the AUC is 1.0 when the divergence estimates are completely separable while AUC is 0.5 when they completely overlap. Table reports the AUC values for the scaled IC-I-Réény divergences (16). various levels of rarity and two sample sizes for the datasets. The best performance in the Rényi family is obtained for a = oo using the IC-I-WCR variational formula (18). IC-I-WCR also outperforms the Wasserstein distance of first order in both sample size regimes.
6.3 IC-T-RÉNYT GANS
Finally, we study pair of GAN examples (the second example is presented in Appendix D). Here the goal is to learn a distribution P using family of generator distribution Qu ~ hy (X) where X is noise source and he is a family of neural networks parametrized by v E V, i.c., the goal to solve
inf (PIQ@w), NEW
(24)
where D,, is divergence estimator of the form 23 In particular, will study the GANs constructed
from the newly introduced IC-I--Rényi and IC- WCR GANs and compare them with Wasserstein GAN Gulrajani al. (2017) Arjovsky al. (017
8

---

Published as a conference paper at ICLR 2023
Table AUC values (higher is better) for several divergences and various levels of rarity. The AUC values have been averaged from 50 independent runs. The neural discriminator has 2 hidden layers with 32 units each and ReLU activation. The D.. divergences used the poly- softplus final layer.
Sample size
100K
20K
Probability (%)
0.1
0.2
0.3
0.4
0.5
1.0
10.0
0.1
0.3
0.5
1.0
10.0
a =
0.51
0.55
0.62
0.64
0.70
0.92
1.00
0.48
0.58
0.58
0.60
1.00
Dr
0 =
0.66
0.70
0.71
0.72
0.74
0.80
1.00
0.32
0.37
0.43
0.38
0.91
0
0.57
0.50
0.62
0.64
0.49
0.59
1.00
0.48
0.48
0.43
0.47
0.74
a 00
0.64
0.89
0.96
0.99
1.00
1.00
1.00
0.58
0.71
0.79
0,91
1.00
Wasserstein
0.63
0.58
0.58
0.51
0.57
0.55
1.00
0.46
0.40
0.45
0.40
1.00
6.3.1 CIFAR-10
In Figure we demonstrate improved performance of the IC-I'-Rényi and IC-I-WCR GANs, as com- pared to Wasserstein GAN with gradient penalty (WGAN-GP), on the CIFAR-I0 dataset Krizhevsky al. 22099 The IC GANs also outperform Ré GAN [I], as the latter is highly unstable when a > and so the training generally encounters NaN after small number of training epochs (hence we omit those results from the figure). We use the same ResNet neural network architecture as in (Gulrajani et al. 2017, Appendix F) and focus on evaluating the effect of different divergences. Here we let I be the set of 1-L.isschitz functions, implement via gradient penalty. Note that D performs significantly better than R with large a, and the rescaled DF.IC-GAA -GAN performs better that R2..C---AA -GAN when a is large.
8.5
R GAN, FID: 113
R GAN, FID: 11.
JAGNN FID: 1&6
MAGN, FID: 15.3
R.J0 GAN. 57.9
R JAN-GAN, FID:
WGAN-CPL, FID: 19.0
D
10 GC.GNN FID: 16.0
7.5
WGAN-GP2, FID: 19.0
7.5
1.0 .GAN, FID: 57.9
Incegiton Soxe
GAN. FID: 16.7
7-
DE
D. CGAN, FID: 16.1
Incepsiio
6.5
6.5
6-
5.5
5.5
Mmy
5
103
10
10 10
10*
10
Epuch
Epuch
Figure 2: Comparison between IC-r-Rényi GAN, IC-F-WCR GAN, and WGAN-GP (both 1 and 2-sided) on the CIFAR-10 10 dataset. Here we plot the inception score as a function of the number of training epochs (moving average over the last 5 data points, with results averaged over 5 runs). We also show the averaged final FID score in the legend, computed using 50000 samples from both P and Q. For the IC GANs we enforce negativity of the discriminator by using final layer equal to -abs. The GANs were trained using the Adam optimizer with an initial learning rate of 0.0002. The left pane shows that the IC-I'Rééyi GANs outperform WGAN while the right pane shows that GANs based on the rescaled DI.IC divergences (16) perform better when a is large, including in the > oc limit, i.e., IC-F-WCR (17). In both cases the IC GANs outperform the RéRényi-DV GANs
with a > 1 21 the latter fail to converge due to the presence of risk-sensitive terms.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The research of J.B., M.K. and L.R.-B. was partially supported by the Air Force Office of Scientific Research (AFOSR) under the grant FA9550-21-1-035- The research of M. K. and L.R.-B. was partially supported by the National Science Foundation (NSF) under the grants TRIPODS CISE- 1934846.. The research of P.D. was partially supported by the NSF under the grant 1MS-9994992 and by the AFOSR under the grant FA-9550-21- 1-1444. The work of Y.P. was partially supported by the Hellenic Foundation for Research and Innovation (HFRT) through the "Second Call for HFRI Research Projects to support Faculty Members and Researchers" under Project 4753. This work was performed in part using high performance computing equipment obtained under a grant from the Collaborative R&D Fund managed by the Massachusetts Technology Collaborative.
MMMS2000990 and
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on muchine learning, pp. 214-223. PMLR, 2017.
R. Atar, K. Chowdhary, and P. Dupuis. Robust bounds on risk-sensitive functionals via Rényi divergence. SIAM/ASA Journal on Uncertainty Quantification, 3(1):18-33) -3,,, 2015. doi: 10.1137/ 130939730. URL http::///ooio.r10.111111//1/////
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Proceedings of the International Conference on Machine Learning, pp- 531-540, 2018.
Himesh Bhatia, William Paul, Fady Alajaji, Bahman Gharesifard, and Philippe Burlina. Least kth-order and Rényi generative adversarial networks. Neural Computation, 33(9):2473-2510, 2021.
Jeremiah Birrell, Paul Dupuis, Markos A. Katsoulakis, Lue Rey-Bellet, and Jie Wang. Variational representations and neural network estimation of Rényi divergences. SIAM Journal on Mathematics of Data Science, 3(4): 2021. doi: 10.1 369113 10. 00.113222M1368222
htthttts:://// //ooi org/
Jeremiah Birrell, Paul Dupuis, Markos A. Katsoulakis, Yannis Pantazis, and Luc Rey-Bellet. ()- divergences: Interpolating between f-divergences and integral probability metries. Journal of Machine Learning Research, 23739):1-70, 2022a. BRLhhtt::// httt::/jmmr........////.// wammr..rg//ae org/papers/v23/
21 0100..0000000 1
Jeremiah Birrell, Markos Katsoulakis, Lue Rey-Bellet, and Wei Zhu. Structure- preeourvi-peeevvingggg GANs. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 1982-2020. PMLR, 17-23 Jul 2022b. URL http/vvl2/bbbb /tt.appeeeee/11.///rrraaaaaaaaa
httpps:/prroeeeinnggg
Jeremiah Birrell, Markos A. Katsoulakis, and Yannis Pantazis. Optimizing variational representations of divergences and accelerating their statistical estimation. IEEE Transactions on Information Theory, 68(7):4553 4572, 2022c. doi: 10.1109/TT.2202231166555
J. Borwein and Q Zhu. Techniques of Variational Analysis. CMS Books in Mathematics, Springer New York, 2006. ISBN 9780387282718. URL /tttok:/bbokss... google. www.coom//oobeee
id=Fquue 78 7KkJe3C
Luc Bégin, Pascal Germain, François Laviolette, and Jean-Francis Roy. PAC-Bayesian bounds based on the Rényi divergence. In Arthur Gretton and Christian C. Robert (eds.). Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pp. 435 444, Cadiz, Spain, 09-11 May 2016. PMLR. URL htttps://proceedingg mlm press http///////////eee/e/ee/ee html
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In ICML 2020, July 2020. http:////ww...aaoooooooo//////// abberppartatttttteoorrrrrrrrrrre
10

---

Published as a conference paper at ICLR 2023
Neel Dey, Antong Chen, and Soheil Ghafurian. Group equivariant generative adversarial networks. In International Conference on Learning Representations, 2021. URL URUpttpp://oppeeeeeeae
net/ peet/Eoze Wwwouooidorrre..... NuJHHXv
Paul Dupuis, Markos A. Katsoulakis, Yannis Pantazis, and Luc Rey-Bellet, Sensitivity analysis for rare events based on Rényi divergence. Ann. Appl. Probab., 30(4):1507-1533, 2020. ISSN 1050-5164. doi: 11221/19-AAP1444
AAP AAP1468
Dupuis, Paul and Mao, Yixiang. Formulation and properties of a divergence used to compare probability measures without absolute continuity. ESAIM: COCV, 28:10, 2022. doi: 10. 105///ccc// 2022002. URL
G.B. Folland. Real Analysis: Modern Techniques and Their Applications. Pure and Applied Mathe- matics; A Wiley Series of Texts, Monographs and Tracts. Wiley, 2013. ISBN 9781 118626399.
URL https:
googlo com
wwcom/boooeee boo.s w 4I
Pierre Glaser, Michael Arbel, and Arthur Gretton. KALE flow: A relaxed KL gradient flow for probabilities with disjoint support. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and
J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. neuriss cc/
8018-8031. Curran Associates, Inc., 2021.
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. Generative Adversarial Nets. In Proceedings of the Annual Conference on Neural Information Processing Systems, 2672-2680, 2014.
A. Gretton, M. Borgwardt, M. J. Rasch, Schölkopf, and A. Smola. kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723-777 2012.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved Training of Wasserstein GANs. In Proceedings of the s1 International Conference Neural Information Processing Systems, NIPS' 17, pp. 7779, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6900 2014.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Gowtham R. Kurri, Oliver Kosut, and Lalitha Sankar. A variational formula for infinity-Rémn
divergence with applications to information leakage. In 2022 IEEE International Symposium on Information Theory (ISIT), pp. 2493-2498. IEEE Press, 2022. doi: 10. 100111196577065666..0 2022. 9834358. URL https:://doo org/ 0. 1w1111101000506.. 2022. 9834358
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(1 (6(11):2278-233) 1998.
Kyungmin Lee and Jinwoo Shin. RényiCL: Contrastive representation learning with skew Rényi divergence. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (cds.), Advances in Neural Information Processing Systems, 2022. URL net/ 7
Jacob Levine, Erin Simonds, Sean C. Bendall, Kara Davis, El Ad D. Amir, Michelle D. Tadmor, Oren Litvin, Harris G. Fienberg, Astraea Jager, Eli R. Zunder, Rachel Finck, Amanda L. Gedman, Ina Radtke, James R. Downing, Dana er, and Garry Nolan. Data-Driven Phenotypic Dissection of AML Reveals Progenitor-like Cells that Correlate with Prognosis. Cell, 162(1): 184-197, 2015. ISSN 10974172. doi: 10. ce166f.cel1.2015.05..
URL http::htt/// doil
org/10. 00t//101106633 1l 11.1111..111111 2015..0 05.005.. 05.001 047
Yingzhen Li and Richard E Turner. Rényi divergence variational inference. In Proceedings of the Advances in Neural Information Processing Systems, pp. 1073-1081, 2016.
11

---

Published as a conference paper at ICLR 2023
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In Silvia Chiappa and Roberto Calandra (eds.). Proceedings of the Twenty Third International Conference Artificial Intelligence and Staristics, volume 108 of Proceedings of Machine Learning Research, pp. 875-884, PMLR, 26-28 Aug 2020. URL https: htt.tttsss/pprceeedaitt ings
mlr www.ppsss/vll//// httaa///e00///aaaaeeeeee/0/ Tester20a .tmml
Tom Minka. Divergence measures and message passing. Technical Report MSR-TR-2005-173, Jan-
uary 2005. http:tt::: http://wwww.iiiii....oo www.soff.......oommmmoo com wttp:/euuuuuuuueetieeeeeee
d
llya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263-275, 2017. doi: 10.1 101100SSF:20...1....
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Yannis Pantazis, Dipjyoti Paul, Michail Fasoulakis, Yannis Stylianou, and Markos A. Katsoulakis. Cumulant GAN. IEEE Transactions Neural Networks and Learning Systems, pp. 1-12, 2022. doi: 10. R0.109/NNLSS.0223311111
Alfréd Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability. Volume 1: Contributions to the Theory of Statistics, pp. 547-561, Berkeley, Calif., 1961. University of California Press. URL https: //projecteucl org.llid.oorggeeee idd.illdddc.... http:///ws.....p.pppp.ppppp blsmpp/100011111111
Noor Sajid, Francesco Faccio, Lancelot Da Costa, Thomas Parr, Jürrenn Schmidhuber, and Karl Friston. Bayesian Brains and the Rényi Divergence, Neural Computation, 34(4):829-855, 2022. ISSN 0899-76677. doi: 10.1162/neco._ a_ 01444. C1484
P Shahi, SC Kim, JR Haliburton, ZJ Gartner, and AR Abate. Abseq: protein profiling with droplet microfluidic barcoding. Sci Rep, 7:44447, 2017.
single cell
Jiaming Song and Stefano Ermon. Understanding the Limitations of Variational Mutual Information Estimators. arXiv e-prints, art. arXiv:1910.0622.. October 2019.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CORR, //////077037444 2018. UtL.ittpp///arwwv..orr///b///// 03748
Tim Van Erven and Peter Harremos. Rényi divergence and Kullback-Leibler divergence. IEEE Transactions Information Theory, 60(7):3797-3820, 2014.
Lukas M Weber, Malgorzata Nowicka, Charlotte Soneson, and Mark D. Robinson. diffcyt: Dif- ferential discovery in high-dimensional cytometry via high-resolution clustering. Communi-
cations Biology, 2(183), 2019. ISSN 3649. doi: 10. 000100884200000194411555 URL
A DERIVATION OF VARIATIONAL FORMULAS FOR THE CLASSICAL RÉNYI DIVERGENCES
In this appendix we provide several variational formulas for the classical Rényí divergences, some of which are new. In the following we let (Ω, M) denote measurable space, M(1)) be the space of measurable vellev functions on Ω, M2( the subspace of bounded functions, and P(2) will be the space of probability measures on Ω.
First we recall the Rényi-Donsked Varadhan variational formula derived in Birrell et al, (022)) This generalization of the Donsker-Varadhan variational representation of the KI. divergence.
12

---

Published as a conference paper at ICLR 2023
Theorem A.I (Rényi-Donsker Varadhan Variational Formula). Let P and Q be probability measures on M) and a R. a + 0, 1. Then for any set of functions, D. with Mb(2)) c C M(2) we have
1
log ela-1)o 1 a
log [e0
(PIlQ)
sup pc
(25)
where interpret 5o 0o and -00 +
If in addition (Q,M) metric space with the Borel o-algebra then (25) holds for all & that satisfy Lip,, (St) C & C. M(()), where (it, is the space of bounded Lipschitz,unctions functions on Q2 (i.e., Lipschitz for any Lipschitz constant L E (0, 20)).
Using Theorem A.1 we can derive new variational representation that takes the form of convex conjugate.
Theorem A.2 (Convex-Conjugal Rényi Variational Formula), Let P. Q P(0) and a (0, 1) U
oo). Then
(P (i),(()) { gdQ
{d
1 - log / 191/0-11/0 dP + a 'logaaaaaa 1). (26)
If (Q. is metric space with the Borel a-algebra then (26) holds with Ms((I) replaced by (h()., the space of bounded continuous real-valued functions fuancvolnedunnctionn on St.
Proof. Let Theorem A. implies
1a---oo(-11
Mosl2), h < 0}. We have Mc(S2) c + C M(Q), hence
cla-1)6 log / eaoaa)
Ra (PIIQ) = sup ΦE
log
(27)
a -
sup (EM
1
log
logl log(-1))) claa-)aa-log((hh) dP-
log / Log(a)
logl
/ 11ll--1//op-- - 1 log (-h)de)) /
log /
sup (E)
a
Note that the second term is finite but the first term is possibly infinite when (0, 1). Next use the identity
log(c)
inf {a ce "e Ce (0)
(28)
zER
in the second term write
aa-1 log / [1110-1//000
inf (z tee / (-njjao}
(PIIQ) sup
(9
a xER
(29)
sup log [b|la-13/0 13/0 dP 2-1
+ +a aale-=
=uu sER (5)
For each E IR make the change variables h axee g. g E Mo( 9 in the inner supremum derive
a a log / [ (a- dp 2-1 + ta -e- / ae geeQ
Ra = sup
sup
=ER gE M,
(30)
a { log 1111/ (a -I 1) / )doo)
sup sup =ER (Q)
1
/ |g| 1a + - + +1).
sup
gdQ
log
(f)gg0)
This completes the proof of
26 The proof of the metric-space version in nearly identical.
13

---

Published as a conference paper at ICLR 2023
Remark A.3. To reverse the above derivation and obtain 25 (with +b {Ø E M(Ω)
is bounded above)) from 26 change variables + -ce & D, c > in 26 and then maximize over
Corollary A.4. If X is a compact metric space with the Borel o-algebra, P,Q E P(X), and
(0,1) 1) U (1,og) then Cx(X) = C(X) and so
{ gdQ R. (PIIQ) sup 2EC(X):300
1
log 1gl 1ggla-1/aal 1//a dP) + alogar +1).
(31)
a
Next we derive variational formula for the worst case regret, defined by
Dxc lim (Pa(P)))).
(32)
Theorem A.S. Let P,Q E P(L)). Then
{ gdQ + log / lglaP] +1
Dxc (PlIQ)
sup gEMs((0)sE0
(33)
S a metric space with the Borel a-algebra then (33 holds with Mc(S2) replaced by (2
Remark A.6. Note that compact metric space, the space of bounded continuous functions is the same as the space of all continuous functions.
Proof. Recall Van Erven Harremos (2014)
dP flog log (ess supp eq oc,
P&Q P p
(PIlQ)
(34)
First suppose P & Q.
Then there exists measurable set A with Q(A) and P(A) Let
In = A - Lcc. Then
{ gdQ + log / [1//r] ] +12 sup gEM,((0);g<0
/SndQ gndQ + log IgnldP +1
(35)
-Q((AA) Q(A9) + log(nP(A) + P(A9) + (oo((o(()) P(A")) +
(36)
as n + cc. Therefore
/ + log / [slar) + x- +
sup 2E
(37)
Now suppose P < Q. Using the definition 32 along with Theorem A.2 and changing variables g = ğ/a we have
Doo (PIIQ)
lim al, (PI[Q)
(38)
{ agdQ -- log / a (11aa-1/0000 1//ap (log a + 1)
lim ax
sup gEMo(
lim 7x0
gdQ + log
(a/// + (log 1)
f gdQ + log
10ka-11/a +1]
lim ax
gdQ + log
|0ldP +1 for all g E My, g <
Here used the dominated convergence theorem to evaluate the limit. Hence, by maximizing over I we obtain
{ gdQ + log / [olap] + (E
(PIIQ) 2
sup
(39)
14

---

Published as a conference paper at ICLR 2023
To prove the reverse inequality, take any E (0, ess pdPP/d)) By definition of the essential supremum we have P(dP/dQ dQ > r) > O. We also have the bound
P(dPIdQ>1)
-aaaraa2>> Lap/dQ>> dQ
dP dQ
Jarraqs. rQ(dP/QQ >)). (40)
For e,e c, > 0 define gese -caar/Q2rr These satisfy Ic,e Mb(s2), Jo,e < and so
{1 gdQ + +log / (oldP) } +12 f Be + log / 19c. (41)
sup
0EMu(0):000
= Q(dP/ dQ > r) c log(cP(dP/dQ > r) +c
2 - cP(dP/ > r) c log(cP(dP//Q) > r) +1
Letting + 0+ we find
{ gdQ log / 1sldp/ 2-12-cP(PP/dll > to//P r))
sup (0)g0000
(42)
for all c O. We have P(dP/dQ > r) > O, hence by maximizing over c >> 0 and changing variables to cP(dP/dQ > r) we obtain
{ gdQ + log loldp/ + sup(-2/r log(z) + 11 = log(r)) (EM,(R))
sup
(43)
s>0
This holds for all ess sup /QQ, therefore can take / conclude
css sus and use 34 to
{ gdQ + log / gldP +1
sup gE)Mg(():
log(ess = (Po(P))).
(44)
Combining this with 39) completes the proof of
Now suppose Ω is metric space. We clearly have
{ gdQ + log
/ (olap) +1
(PI()) = sup gEMo(9))ac0
(45)
{ gdQQ + log / [19lap]
sup (0)gg0
+1.
To prove the reverse inequality, take any g E MM((2)) with g < 0. By Lusin's theorem, for all e > 0 there exists a closed set E, and hte E ((t) such that P(Eş) S e, Q(E)) S e, helF, = 9. and
inl g < h < 0. Define ge = tie e. Then Je < 0. Je E Co( and we have
{ gdQ log f [oldp]
sup (EC5(0))850
g.dQ + log luldd
(46)
=/ gdQ + 1(he ((he-1)) (1he-g)1 eQ - - -lllog/ lolap+ / (1hhl- (g1) + +c)
gdQ - (sup inf 9nf()) + +lgog Iolde + inf gP((s) - E)
2 f gdQ (sup g inf g)e e + log( / lgldP -nu inf ge
Taking the limit 0 therefore obtain
{
gdQ + log / loldP
gdQ + log lsldp.
sup g0)gg0
(47)
This holds for all g E (2) with < 0. hence by taking the supremum over we obtain the reverse inequality (45 This completes the proof.
15

---

Published as a conference paper at ICLR 2023
B PROOFS
In this appendix we provide a number of proofs that were omitted from the main text. Recall that X denotes compact metric space,
Lemma B.1. Let P C C(X) and P.Q. P(X). Then aRa fo (PIIQ) is non-decreasing in E (1, 1 P then A also non-decreasing.
Proof. If0 E then W 2 hence
aR. (PIIQ)
inf (aRa(Pll)) + + (P(()())))) nEP(X)
(48)
where both + (P and + aWT(Q: 17) are non-decreasing. Therefore the infimum is as well. The proof for + aR /a,IIC (PIl)) is similar, though it doesn't 't require the assumption 0 r due to the identity aW = WT.
Next we prove a key lemma that is used in our main result. First recall the definition
dP 1 (a-11 log / (ol0--1// 1)/o dP + a "log 1gco, O (49)
A = o0lgg0
Lemma B.2. Ag is convex and is continuous on {g C(X) :9 < 0}, an open subset of C(X).
Proof. First we prove convexity. Let go. 91 E (X( g < 0} and X (0, 1). For (0, 1)
can use the inequality da + (1 A)6 2 a~b2-> for all a, > 0 to compute
1
log / |1991 + (1 \)gn (a- (a-1)/a dP a
1
log / (191 A " -o1--1aa- (50)
Using Hölder's inequality with exponents p 1/2, q 1/(1 X) we then obtain
1 -
log (10a0"
(51)
log (fl11la-11aa / 1001 1001(-11///pp-1))
a
- log [9n (a- + 2) (1)0-00))
1
a 1 (-a- log [gol (a- (11aa11/aap)
Therefore g a log (g] (111-1)/a is convex on {g < 0}. This proves A is convex when (0, 1).
Now suppose a 1. The map > t - 11a-1)/a 1)/a is concave and log is decreasing and convex, hence
log / |oo + +(1 (1 (a-1/aa
(52)
log ( 19n Do +(1
12) log f 191 10- dp) + (1 (1- 2) a
11 (11-)) l1-laa1)//ad))
0
1
(---1110 log
[gol (a aP)
a
This proves that A is also convex when 1. Openness of {g 0} follows from the assumption that X is compact and so any strictly negative continuous function is strictly bounded away from zero. Continuity on {g < 0} then follows from the dominated convergence theorem.
Now we prove our main theorem, deriving the dual variational formula and other important properties of the IC-I Ré-n'- divergences.
Theorem B.3. Let T C(X) be admissible, P.Q E P(X), and a E (0, U (1,20) Then:
16

---

Published as a conference paper at ICLR 2023
1.
{f gdQ - a-1 log / [g|( 19ll--D/odP) + aa- a a + 1).
Re (P(((())))) = sup
ger
(53)
2. If 53) is finite then there exists 7/= E P(X) such that
(PII))
inf nEP(X)
(P(l)) w (Q, n)) (Pll)))))) (FQQ.))) (54)
3. (P(()) is convex in Q. Ifa E (0,1) then 12 (PIIQ) is jointly corvex in (P,Q).
4. (P,Q) + (P((O())))) is lower semicontinuous.
5. RE (PI)Q) 2 0 with equality if == Q.
6. (CI((())))) S (P)))) (T (Q.
7. I is strictly udmissible then REIIC has the divergence property.
Proof.
1. Define F; G:: C(X) + (-00, oc by F == Ag and G[g. co - Eo]gg]. Using the assumptions I' along with Lemma B.2 we that F' and G are convex, 11] oc, G[-1] < oc, and F is continuous at -1. Therefore Fenchel-Rockafellar duality (see, e.g.. Theorem 4.4.3 in Borwein & Zhu (2006) along with the identity C(X)* (XX gives
{F + G*1-n)> sup [-F[gl - G[g]} inf SEC(X)
(55)
NEM(X)
and if either side is finite then the infimum on the right hand side is achieved at some m, E (X). Using the definitions, we can rewrite the left hand side as follow
sup (EC(X)
f-Figl G[g)}
(56)
{ odQQ+ a log 11/ + a 3 +o++l 1).
sup gETge<o
We can also compute
G-[-n]= = sup
{- f-/ gdn (ool - Eg(g))} = w (Q.n)
(57)
9EC(X)
Therefore
inf (EM(X) {(A)))()) + -.
(58)
{f gdQ
log [g/ } +a + "log -1 1).
sup ger'geD
Next show that the infimum over M(X) can be restricted P(X). First suppose
7) E (X) with (xX + 1. Then, using the assumption that I contains the constant functions, have
WT Eq[#n] - / +ndn -n (1-111nx)))
(59)
as 12 >> Do (for appropriate choice of sign). Therefore WT (VT(Q.)) = x0 if (X) + 1. This implies that the infimum be restricted to {n E M(X) nxX)=((])) 1).
Now suppose 7) E (X) is not positive. Take a measurable set A with (A) < 0. By Lusin's 's theorem, for all > 0 there exists closed set F,, C X and continuous function
17

---

Published as a conference paper at ICLR 2023
9e C(X) such that |n](E;) c, 0, - and ge 1A. Define In.e nge.
n E Zt Then In,e E {9 E C(X) 9 < <0). hence
(Ag)*()) 2 =f dn + a-
- log /
1
I I (a-1/aa
(log + +1
(60)
= -- --)((A) + +nn(An nn E - ge gel (x))
a-1 log / Inge + 1|(0-11/0 dP P+ + a- ' 1)
1
1 - log / |Ing. + 11001)/a + a 1).
2n([ - 26) - n(() +
a a
If > 1 then log Inge + 11(0-1)/0 11 2 and if E (0, then log Ing. 11-11/adP & 0. In either case we have 1
a-1 log / Ing. + 1/ 2 0 and so
y (n(((AA) 2e) a a +1)
+
(61)
By choosing e < (n)l/22 and taking n - oo see that (A oo whenever 1 E ((X is not positive. Therefore the infimum can further be restricted to positive measures. Combining these results we find
{f gdQ + a 1 log 111 dp + a aa'ooooo + 1)
sup 9
(62)
inf EXP(
{(A W (Q.n)}.
For 7) E P(X), equation
10 implies (m) = R. (Pll7). This completes the proof.
2. The existence of minimizer follows from Benchel-Rockafellee duality; again, see Theorem 1.33 in Borwein Zhu (206).
3. This follows from (33 convex and y
together with the fact that the supremum of convex functions log(y) is convex when e (0,
4. Compactness of X implies that g and (-/ bounded and continuous whenever 9 I satisfies g 0. Therefore Q gdQ and P + J g) 1)/a (m11// dP are continuous in the weak topology on P(X). Therefore the objective functional in (53 continuous in (P, Q). The supremum is therefore lower semicontinuous.
5. This easily follows from the definition
6. RA, is divergence, hence is non-negative. I contains the constant functions, hence WT > Therefore R. ICC 2 0. If Q = P then 0 s (C((C()))Q) S Ra WT = O.
hence 1C (PI = 0.
7. Suppose is strictly admissible. Due to part of this theorem, we only need to show that if (PIIQ) = 0 then P Q. If (C((() then part 2 implies there exists
7. E P(X) such that
0 - R. + (F(.(-1))))
(63)
Both terms are non-negative, hence R, (PI = WT n))) R. has the divergence property, hence 7/ = P. So W (Q, P) = Therefore 0 > gdQ f gdP for all y E r. Let Vi be as in the definition of strict admissibility and let v É W. There exists c E RR e > 0 such that c eω and so 0 2 =<(ʃ vdQQ vddP). Therefore pdQQ = vdP for all &b E W. v is P(X))-etermining hence Q =
Remark B.4. The Fenchel- RockffllR Theorem applies under different sets of assumptions: the first assumes both mappings are lower semicontinuous (LSC) while the second applies when one mapping is continuous at point where both are finite. The mapping AP. as defined by (49) and
18

---

Published as a conference paper at ICLR 2023
appearing in TO), is not LSC but it is continuous on its domain, hence used the second version of Manchel-Rockafellrr in our proof of Theorem [.3 For a > one could alternatively redefine An along the boundary of {g < 0f to make it LSC while still maintaining the relation (0)) and thereby utilize the first version of extending the theorem non-compact spaces, using the methods from Dupuis, Paul & Mao, Vixiang )0 Birrell et al. (2022a). However, these methods do not apply to E (0, With this in mind, in order to provide a simple unified treatment of all a E (0, (1, o0) we structured our proof around the second version of Fenchel- Rackkll--
This alternative approach is also amenable to
Despite the fact that A is not LSC, the Fenchel- Rockaflll Theorem does imply that convex duality holds at all points of continuity in the domain, i.e., one has
Sf gdn gdnn Ra(Pll)) for all < O, nEM(X)
N Na [g]] = sup
(64)
but this duality formula doesn't 't necessarily hold if K O. Here, R. (Pll)) for general vj E M(X is defined via the variational formula
R. :0 (1))
sup gEC(X)
nd A
(65)
and one can rewrite this in terms of the classical Rényi divergence as follows
00 in ž 0r 0, log ||lll in is nonzero positive measure.
(P(l))
(66)
Ra (Pll
Next we prove the limiting results from Theorem Theorem B.5. Let C C(X) ) be admissible, P.Q E P(X), and a E (1, Then
lim 1 Rár.Ic 1C = (PP1Q)==((Q,1)) 8-0+
(67)
and T is strictly admissible have
lim RLT.IC (pIlQ) (PIIQ) L-000
(68)
Proof. It is straightforward to show that the scaled function spaces are admissible and Wer = cWT for all c O. First we prove 667. From the definition 7 we have
8- -- (PIIQ) = inf and so 8-1
1 RR - W nEP(X)
(Q,
(69)
is non-increasing in Therefore
lim 8~ R.5r (PII)) sup 8-1 Rér. (PI[Q)
(70)
6-
8>0
and
lim 8- JC 1PP S 6-+0
(71)
We will assume this inequality is strict and derive a contradiction. This assumption, together with (70), implies RC (eP(()(P))()) < oo for all > Part (2) of Theorem 3.4 then implies the existence of
N..8 P(X) such that
8- RTT
= R- - WT(Q n..8) 2 (Q,
(72)
Take sequence 5n 0+ We have assumed X is compact, hence P(X) is also compact and so there exists a weakly convergent subsequence Ne,5n, + n. From the variational formulas (25) and
g
we see that R. (PI).) and WT(Q,,) are lower semicontinuous, hence lim inf, W WT(Q Me,6n,
WT WT(Q, n.) and
Rs (PI[m=) S <limini inf R. (PIliT, S lim inf 8n, (6,,) R., (PIlNe,ss)) ) + WT B,,sm)))
(73)
=lim 8n, (6m, Ro. P.IC (PIIQ)) 0,
(74)
19

---

Published as a conference paper at ICLR 2023
where the last equality follows from the assumed strictness of the inequality divergence property for the classical Rényi divergences implies Ra (P)))) and P = 17.- Combining the above results we obtain
70 Therefore the
fn P.IC lim 8 RIT.IC (PIIQ) = lim ong Ro
(PI()) 2 lim inf W (Q, Mr,n,)
(75)
j_000
(Q. = (Q,
This contradicts (71) and therefore we have proven the equality (67)
Now we assume I' is strictly admissible and will prove (68 via similar reasoning. From the definition
we that
R R.r.o((lIQ)) = inf (Pln) + (W"(Q,)) - R. (PllQ) nEP(X)
(76)
and RLC,IC is non-decreasing in L. Hence lim IC (PIIQ) = suP IC (PIIQ) and
lim +
Rd (PI])) S (a
(77)
Suppose this inequality is strict. Then (PI(I co for all L and can part (2) of Theorem 3.4 to conclude there exists 7, E P()) such that
RLT.IC (PIIQ) = R₂, (n..)) LW (Q.
(78)
Take Tip 20. Compactness of P(X) implies the existence of weakly convergent subse- quence M/..j = Mrling + E P(X). Lower semicontinuity of (Pll.) and WT (Q, imply
lim inf, Ra (P(l))))))) 2 (a(l))))) and
(T. inm WT (Q, (1-5)) lim inf L
W (Q,))s)
(79)
< Iim L R. R. I,,,TIIC D.IC (PII)) 0.
where the last equality follows from the assumed strictness of the inequality WT (Q, 0. Г is strictly admissible, hence Q = (see the proof of part (7) of Theorem
Therefore
3.4 Combining these results we see that
lim R.T.IC (PIlQ) lim R
Ann IIC
(PIQ) lim (R (al (Pll),, + Ln, w WF(Q (Q, (r.n))))))) (80)
3
)
2 lim R. (PI 2 R. (-) = R. (P.(P)))
3
This contradicts the assumed strictness of the inequality (77 and hence is equality. This completes the proof.
Next we prove Theorem 4.2 regarding the a - limit of the IC-I'-Rényi divergences.
Theorem B.6. Let T c C(X) ) he admissible and P.Q E P(X). Then
lim IC (PIIQ) = inf
{R (R(P))n) + (F( 3Bt:1,oo(Pll)
(81)
a+1
7X)(
lim IC ([(()( = inf nEP(X)
(R(PI)n) + (R(Plln)+W*(Q... +W
(82)
a-11-
/ggdQ + /1 /10gloldP)) [gldP] +
sup gEr'yc0
(83)
Proof. Lemma B.1 implies + 1C is non-decreasing on (1, therefore lim aRt. IC (PII)) = inf a++
a>1 (C(I()
(84)
inf inf (aRa (Q.)
a> nEP(X)
inf inf (afa + aW (Q.n) nEP(X)a>1
inf lim aR, (P)m) WT (Q. )Q.))) nEP(X) a++
20

---

Published as a conference paper at ICLR 2023
From Van Erven & Harremos (2014) have
[(P])))
ir > 1, (g(lnn <0
lim R. (PI))) a=lt
(85)
00
otherwise
and so we can conclude
lim R (PI()) = lim R
inf TEP(X): 7BP 18>1, <m)
(R(PI)n) + (Q. (86)
a-11
This proves
Now we compute the limit as + 1 Note that the limit exists due to the fact that + aR 10 (PII)) is non-decreasing. From the definition (7) for rj E P(X) we have
lim (PI(())))) < lim (P + W (Q n) R(Pl))) + (T,
(87)
Here we used the fact that lima-+1- (P(()) = R(Plln) (see Van Erven & Harremos (0144) Maximizing over then gives
R (E < inf a=11
lim
( WF(Q.)))
(88)
nEP(
To prove the reverse inequality, use part of Theorem 3.4 to compute
lim a=11
RC (PIIQ)
= lim aRD (P(I()
(89)
a-11
{a gdQ a- log / lgl log lim sup
a gel:geu
a- a - log [g/[ /g/l//)/
lim gdQ
f gdQ + dy
w=n log / e golssslld +1
log
/ gdQ log lg|d
for all g E T, < 0. Therefore, maximizing over gives
{ 9dQ / log lg|dP +1.
lim RT.I (C 2
sup nETig<0
(90)
We now use Fenchel-Rockafelle duality (Theorem 4.4.3 Borwein & Zhu (0066) compute the dual variational representation of the right hand side of 90 Define F.G. C(X) (-o0, oc] by
F(g] = , log [g] and Gig) = ool ,er Folg
g It is stratightforward to show that F
and G are convex, F1-1] < G[-1] < co, and F is continuous -1. Therefore
inf (F(g) Glg)} SEC(X)
sup (- - EEE-
(91)
i...
sup (Folg) + [ogl I gel'sec0
inf (F W TEM(X)
(92)
where (1) SuPpeC(X) gdn + log [g]dP] Now show the infimum can be restricted
to P(X): If (X)) # then by taking g = en we find
(F(Q,n) m) 2 Q(() - xXX)] x
(93)
0c. Therefore (T. = if nx) #
Now suppose 17 E M(X) is not positive. Take a measurable set A with (TA) < O. By Lusin's theorem, for all > 0 there exists closed set F, X and continuous function ge C(X)
21

---

Published as a conference paper at ICLR 2023
such that (E.) < 0 < Oe 1, and gelE. 14. Define In.e nge 1, Zz Then
In.e E {g E C(X): : 9 < 0). hence
(n) 2 f In +
flog log lan,eldP
(94)
ngg. - -Idn / log(nse + 1)dP
(n([((A)] / (9) - -a))l (X) 2n(]((A) 6) -n(().
Letting = |n(A)//2 and taking n gives (n) co. Therefore we conclude
inf (F*(n) + m)) nEP(X)
inf (F) (Q
(95)
(EM(X)
To evaluate (n) for nj P(X) we make change of variables g exp(h 1), h E C(X) to
obtain
1ʃ hdPP e 11= -1= R(PlIn)
Fn) sup kEC(X)
(96)
Here we used the Legendre-transfore variational representation of the KL divergence; see equation (1) in Birrell al. (2022c) with f(x) 2 log(()). Combining these results obtain
inf (R(PIln)+ W (Q, 2 lim nEP(X)
RE.IC (PIIQ)
(97)
a-1
{ gdQ + flogg |loglo (oglg/p)) +1
sup sET:
inf TEM(X)
(F" W ( (Q, +1
inf TEP(X)
(R(PlIn) w (*( n))
This completes the proof.
Now we prove Theorem 4.5
regarding the a -> oo limit of the IC-I -én-iRényí divergences.
Theorem B.7. Let C(X) be admissible and P,Q P(X). Then
nEP(X) lim a aR R./ I/ (PIIQ) inf {Dx (P (m) + WT
(98)
a=
sup gEr-<co { gdoQ+ log / [olde +
(99)
Proof. First that
RT.. Jo. (PIIQ) = inf
+ +
(100)
(EP(X)
is nondecreasing in therefore for E P(X) we have
lim aRT/a.1C a0000
(PI(Q) sup at //.1CC (PI[Q)
(101)
< sup(aRt (Pl (T.)))))
= =Do( + wT
Maximizing over y gives the upper bound
lim /.I//a.CC /C (PI())(P)Q)) a-200
<
inf (Doo (PlIn) + -T nEP(X)
(102)
22

---

Published as a conference paper at ICLR 2023
To prove the reverse inequality, the variational formula (53 to write
/R.//.IC (PPl)))))ll( =a
{ g/adQ+
a a-1 log 19// + loga+1 (103)
sup gEr:g<0
sup g
{
gdQ
+ a
log
/ 19lla- ) +1.
Therefore, for all g E E.g < we can use the dominated convergence theorem to compute
lim a../ (PIIQ) 2 / gdQ + lim a-700
a a a a-1 log (glla-1) 1g +1
(104)
gdQ + log lgddP +1. f
Maximizing over g then gives
{ gdQ +og log
/ (gldP) +
lim /R
IC (PIIQ)
sup gET'g<D
(105)
Next we use the Ronchel- duality to derive a dual formulation of the right hand side of (105). Define F.F: C(X) + oo), Gig] I L9gT Eq [g]. F(g] 1g log f l... It iS straightforward to prove that G, F are convex and G < 00, F[-1 < oc and F is continuous
Therefore Fenchel-Roote Rockafellar duality implies
inf [Fig) + G[gl} = sup SEC(X)
(106)
nEC(X)*
i..
Eolgi log [olde] inf (F + (Q(0)))): nEM(X)
sup a
(107)
where [M] sup (euPyee gdn + log IgldP]. We now prove that the infimum over (
can be restricted to P(X). First suppose (X) # 1. Then, because I contains the constant functions, we have
(T(Q.)) 2 In - (())) + 0a
(108)
as n + oo for appropriate choice sign. Therefore W (Q, oa when r7( *
Now suppose 77 (() is not positive. Take measurable set A with (A) 0. By Lusin's theorem, for all e > 0 there exists closed set E, C. and a continuous function ge e C(X) such that (EE) < < 9r 1, and gelE, 1A. Define In.e -gg. 1, Z+. Then In.e E {g E C(X): : < 0}. hence
F' > on.edn + log / I9n,eldP
(109)
=n([n(A)] - / (gc (A))=1)) n(X) log(n / +.dP+1) 2n((((4) c) m)(X).
Letting In(A)l/2 and then taking aa that F*[n] when n is not positive.
Together these results imply
inf (F"(1) + (T( inf (EM)
(F* wT
(110)
TEP(X)
Finally, using Theorem A.3 we see that
1f ydn + log (EC(X))00
loldp)+ +1 (Plln)
Fd [l + +1=
sup
(111)
23

---

Published as a conference paper at ICLR 2023
for all 1y P(X). Combining these results gives
{/ gdQ + log / lolde +1
lim aBa / (PIIQ) 2 sup 0-+00
(112)
ger'gec
inf {F* + (Q: +1
nEM(x)
inf (Dx(P)ln) + W: (Q.m) 2
lim aR I///a.CC (PII)). a+x
nEP(X)
This completes the proof.
Finally, prove Theorem 3.3 the data-proceet peeeeprooceinng inequality for the IC-I IC-T-Rényi divergences. First
we introduce the following notation: Let Y be another compact metric space and K be a probability kernel from X Y. Given P E P(X)
denote the composition of P with K by P K (a probability measure on X x Y) and we denote the marginal distribution on Y by K[P]. Given g E C(X Y) we let [g] denote the function on X given by x + f g(z, y) Kx
Theorem B.8 (Data Processing Inequality). Let E (0, (1, oo), Q. E P(X), and K be probability kernel from X to Y such that Kig] E C(X) all C(X,Y).
1. IʃT C C(Y) is admissible then
RIIIC (K(P][K[Q) S RKTNUC (PIIQ).
(113)
2. IfT C. C(X X Y) is admissible then
RC & B3QKQQQ & K) I R RKTTIC(PIIQ)
(114)
Proof. It is straightforward to show that admissiblility of implies admissibility of K[T]. Hence we write
R RKIFLIC (K(F)C(PPIQ=
{ .dQ++ a log (o (loga (115) G
1
sup
KlgldQ + a- log f [1K19](001/<app + + +
1
sup gELig<o
Using Jensen's 's inequality we can derive
/ ((v) (a-1//
11/a /
a (0.
(116)
/ ((y) K2 (dy)) (a-1//
(alv)((a 1)/a a a>
(117)
Combining 1115
117 with the monotonicity + log(y) we arrive 113).
The proof of
(14 is similar.
C DETAILS ON ANALYTICAL EXAMPLES AND COUNTEREXAMPLES
In this appendix we present several details regarding the analytical examples found in Section
C.I INFIMAL CONVOLUTION SCALING LIMITS
First we present a simple example that illustrates the infimal convolution formula and limiting properties from Sections and
Let P = 80, Qz,e cōu + (1 c)&2 for e E (0,1), > 0, and let I = Lip". Then for L > 0 one
can compute
Ra(1((Q =a -1 1 log(1/c)
(118)
(Q P) =(1 Lx
(119)
24

---

Published as a conference paper at ICLR 2023
and
RLI.IC
(P))Q,.))
sup (Lea L(1 -
a log([La)) - (loga+1) (120)
a b/s
supp Lea L(1 min 0} ta log (L|a)) +a "loga +
a<0
+ y aLæ
+ + sup =0
(1 - (1-c)ala - + logy, > aLx
y>0
(1 La, < aLæ <1
a cLx a log(alx) 1 ale S 1/0
log(1/)), aLx > 1/c
In particular, it is straightforward to show that
ver.Io(PllQ... & WLT (Qz,e, P),
(121)
lim ( = lim (1 - c) La = 0, x
(122)
lim ( = log(1/)) = Raa (P11Qz.)) Lox
(123)
We also rewrite this in terms of the solution to the infimal convolution problem follows
WL (Qz,e P), O aLw <1 Ro (I(1111111)))))) + WL (Qa,c: Qu2/(aLa)), 13 aLx S 1/c (PIlQ.))))) aLx 1/c
Rer,1G (P]]Qx.))
(124)
Taking the worst-case-regret scaling limit we find
c)., 0<<<1 1 1- cr cr 1 1 15s$1/e log(1/c), x> x> 1/c
lim aRT//.10(PPIQ... (P])Qa,e)
(125)
00000
(T,Q2,es P), 0<r<] <<
Doo (P1 + (Qr,er Q2.122) 1 1 x 1/10
Doe (P((Qz.)))) x> 1/c
where Doc (PI[Qz.c) log(1/c).
C.2 T-RÉNYI-DONSKER VARADHAN COUNTEREXAMPLE
As an alternative to Definition 3.1 one can attempt to regularize the Rényi divergences by restrict- ing the test-function space the variational representation (3), leading the I-Rényi-Donsker- Varadhan divergences
log f (cl-1))ddP dP log / (e)
DV = sup TET (a
(126)
The bound
log / cedd of odP, o dET,CER
(127)
implies that DD.DV < WT for a E 10, making 126) a useful regularization of the Rényi divergences in this case: this utility was demonstrated in Pantazis et al. 20022) where it was used to construct GANs. However, the representation (126) is known to be poorly behaved when a > 1. Here provide a counterexample showing that, unlike for the IC-I- divergences, RT. & WT in general when a 1. We conjecture that this fact is the reason for the poor behavior pr.bv when 1.
25

---

Published as a conference paper at ICLR 2023
Let P2.c cōo + (1 c)62. Q do for 0, (0.1) and rz Lip". Then for a > 1 we have
(Pa,cllQ)= L.
log(cexp((a (1)) (1 c) exp((a--1)) 1}b))
sup
a.bER:Ja-b|ss b/
(128)
=sup aE - log Dog(cep((aa-1a) + c) exp((a 1 1)(Lx + a)))
1
log (c + (1 c) cxp(((11)))
a
WTz a.c. Q) la-b SLe
sup {ca + (1 - c)b a} (1 LL
(129)
Note that the condition a > 1 was crucial in computing the supreumum over b in 128 Using strict concavity of the logarithm one can then obtain the bound
R.. (Pr,ellQ) (FL(Pr,; Q)
(130)
C.3 Sog-T-RENYI-DDNSKRR-WOE VARRDDHAN COUNTEREXAMPLA
second alternative to Definition 3.1 to 1/ log((')
again start with and then reduce the test-function space
sup 1 log / g( 1/ 1 log mdo) 1a-1
R Aog log DV (PIIQ)=
(131)
gEL
However, show below, this definition fails to provide regularized divergence; in particular, it incapable of meaningfully comparing Dirac distributions.
Let P = óg. Q₂ = 6g, x > 0, rz = Lip". Then straightforward computations using the variational definition gives
R DV (PIQ..) =a sup log(O(0)/o(z))
(132)
dEr
= -
sup 6>0
sup log(a/b)
log z/b) 30.
620
In contrast have
IC (EL.IC(P(Q..))
(133)
{Lb a looo L + a log((a))) + a- (log(a) + 1)
sup 1.00
1
supp Lb + log( [b- 1 x1)] + c log I + a a (log
b<0
a (og(aa))) tt ス 1/(a)) x < 11/((T))
Lx,
In particular,
RC (P((()) < -== WT (P...(PP.Q.) Qx), lim 1C 1 (PIQ2)= = 0,
(134)
(135)
a1
showing that RT2.CC is able to capture the convergence of Q2 to as x + 0", while R logg fails to do
DV
26

---

Published as a conference paper at ICLR 2023
D ADDITIONAL EXAMPLES
D.1 TRAINING SYMMETRY-PRESERVING GANS ON ROTMNIST
When learning distribution P that is invariant under symmetry group (e.g.. rotation invariance for images without preferred orientation) one can greatly increase performance by using GAN that incorporates the symmetry information into the generator and the discriminator space T Dey et al.) (202T). A theory of such symmetry-presernvee new divergences introduced in this paper satisfy assumptions required apply that theory. In Table we demonstrate this effectiveness the RotMNIST dataset, obtained from randomly rotating the original MNIST digit dataset LeCun et al. (1998). resulting in an rotation rotation-invariant invariant distribution. Note that incorporating more symmetry information into the GAN (i.e., progressing down the rows of the table) results in greatly improved performance, especially in the low data regime.
GANs was developed in BBrrell al. (2022b) and the
Table 2: The median of the FIDs (lower is better), calculated every 1,000 generator update for 20,000 iterations, over three independent trials. The number of the training samples used for experiments varies from 1% (600) to 10% (6,000) of the RoMMNIST training set. The NN structure and hyperparameters are the same as those used in Section 5.4 of Birrell al. (2022b). Eqv (resp. Inv D) denotes that the symmetry information was incorporated into the generator (resp. discriminator) while CNN implies that a convolutional NN was used (without rotational symmetry). E denotes the rotation group used, where Cn denotes rotations by being integer multiples of 2 //n.
Architecture
1%
5%
10%
Reverse RI..
CNN G&D
357
325
298
Eqv G CNN
D, Σ C4
464
271
263
CNN G+ Inv
D, = C4
366
321
302
Eqv G+ Inv
D, CA
151
105
89
Eqv G Inv
D, = Cs
114
71
62
D.2 VARIANCE OF RÉNYT ESTIMATORS
Here compare the DV-Rényi and CC-Rényi estimators the Gaussian test problem from Section 6.1 except in lower dimensions (1-D and 100-D). Qualitatively, the behavior is similar, In particular, striking that the DV-Rényi estimator performs extremely poorly in the 1-D case (see Figure
3) while the RénRé estimator has much lower variance and MSE when the separation between the distributions becomes larger (i.e., Hg increases).
27

---

Published as a conference paper at ICLR 2023
DV-Re
DV-R
10
CC-Ra
Rg
DV-R;
CC-R,
CCC
10
DV-Ra
CC
CC-RRo
DV Rom
CC-Raa
Rue
MB8
10
10
10
10
10
10 10
10
10 107
Figure 3: Variance and MSE estimators of the classical Rényi divergence between I-dimensional Gaussians. DV-Ra refers to Rényi divergence estimators built using 3 while CC-Ra refers to estimators built using our new variational representation (4). We used a NN with one fully connected layer of 64 nodes, ReLU activations, and poly-softplus final layer (for CC-Rényi). We trained for 10000 epochs with a minibatch size of 500. The variance and MSE were computing using data from 50 independent runs. Note that the CC-R CC-Rényi estimator has significantly reduced variance and MSE compared to the DV-Rényi estimator, even when a is large.
10'0
CC-Rs
10
DV-R
CC-R
DV-Ra
DV-Rs
CC-Rs
DV-Ra
DV-Rie
105
(C--Bo
CC
DV Ruz
CC- Roo
CC-Rua
Yarsore
AIRE
169
109
10
10
10
10
Figure 4: Variance and MSE of estimators of the classical Rényi divergence between 100-dimensional Gaussians. DV-R2, refers to Rényi divergence estimators built using 3 while CC-Ra refers to estimators built using our new variational representation (4). We used a NN with one fully connected layer of 64 nodes, ReLU activations, and a poly-softplus final layer (for CC-Rényi). We trained for 10000 epochs with minibatch size of 500. The variance and MSE were computing using data from 50 independent runs. Again, the CC-Rényi estimator has significantly reduced variance and MSE compared to the DV-Rényi estimator, even when is large.
28


Reviews:
[{"review_id": "nEp2lRr93W", "reviewer": "Reviewer_Fzvh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses a regularized family of Renyi divergences using a variational function form. For this family of divergences the paper discusses various properties relating to limits, interpolations, data processing inequality. Examples of the importance of using the new variational Renyi divergences is demonstrated through examples (and counterexamples). Finally, the paper ends with a discussion of numerical applications: a) the variance of the convex-conjugate Renyi divergence estimators are much lower than that of estimators designed using the Renyi-Donsker-Varadhan variational formula when risk sensitive terms are detrimental to the DV-Renyi estimators; b) proposed infimal-convolution divergences are more suitable in the detection of rare sub-populations for a particular dataset; c) the proposed infimal-convolution divergences can perform better than Wasserstein GAN on the CIFAR-10 dataset; and d) the proposed divergences provides improvement on the ROTMNIST dataset. ", "summary_of_the_review": "Below is a short summary of justification, for more details please refer to the above sections. \n\n1. The paper is clearly written and states the background and main results adequately. Main ideas such as the addition of regularization to Renyi divergences are nicely conveyed. \n2. The paper introduces interesting new results for the important field of Renyi divergences, and is closely related to the scope of ICLR.\n3. Overall, no major issues were identified in the main text.  ", "strength_and_weaknesses": "Overall the paper is nicely written and describes well Renyi divergence and its background, and provides a good description of the paper's main results. In particular, the paper provides an interesting development to the study of Renyi divergences by introducing a new family of regularized Renyi divergences. \n\nSome minor comments on the strengths of the paper are:\n1. The paper includes clear description of definitions and main results. The highlights of contributions on Page 2 were very helpful to understanding the paper.\n2. The paper addresses Renyi divergences, an interesting and important topic, and provides interesting development to available results. The variational formulations of the Renyi divergence and the corresponding regularizations using an IPM (i.e., (6)) are very interesting. \n3. The paper considers a variety of numerical experiments, demonstrating flexibility in applications. \n\nSome minor comments on the weaknesses of the paper are:\n1. In the first experiment (Section 6.1), it would be interesting to see if CC-Renyi also performs better than DV-Renyi when distributions are close. In the experiment shown the two Gaussians are chosen to be N(2,0.5) and N(0,1) respectively and relative variance and MSE numbers are quite extreme for DV-Renyi -- it would be interesting if CC-Renyi also improves upon DV-Renyi for scenarios where say the two Gaussians are N(0,1) and N(0.1,1), for example. \n2. The first section in Section 5 may benefit from some more explanation. \n3. It may be helpful to adjust some of the citation formatting to make the paper more readable. For example, in the very first sentence of the Introduction, the two citations are a bit confusing to the overall readability of the sentence.\n4. The details to the last experiment seem to be slightly lacking; it would be helpful to add some more description to the main text. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gq7CFhBhHp", "reviewer": "Reviewer_ht2a", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new divergence which is a convolution between Renyi divergence and other IPM.\n", "summary_of_the_review": "A new divergence is introduced, some its properties are explored, however the real need of new divergence for applications is not well justified. ", "strength_and_weaknesses": "Strength:\n\n1. The paper proposes a new divergence which include the Renyi divergence and other IPM. It is new and have some interesting properties.\n2. The author derives some properties of the new divergence and show its dual form for computation. \n3. They derive the limit formula of divergence and present some special case of the new divergence.\n4. Empirical results show that the new divergence is more stable the Renyi divergence when $\\alpha >1$. \n\nWeaknesses:\n\n1. The derivations of mathematical results of the paper are quite straightforward from known results.\n2. It is not clear for me the popularity of using new divergence in reality when $\\alpha >1$. For $\\alpha <1$, I do not see any advantage to have new divergence.\n3. In the application on biological datasets, the best results are obtained when $\\alpha = \\infty$. It means that we do not need to care about other values of $\\alpha \\in (1,\\infty)$, then why do we need to define the new divergence. \n4. In the experiment with GAN, the proposed divergence is used to improve the performance of generative model. The results are shown through the FID scores. We need to choose a particular $\\alpha$ such that it obtains the lowest FID. In this case, is there any explanation of using the proposed divergence rather than searching something in a larger space will obtain a lower minimum.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "e6mnge4B26", "reviewer": "Reviewer_qiWv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new family of regularized Renyi divergences by infimal-convolution with integral probability metric (IPM). The authors derive its dual variational representation and provide several theoretical results about limits, interpolations, regularized worse-case regret. The authors evaluate the proposed divergence on both synthetic and real datasets, and its application on GANs.", "summary_of_the_review": "The proposed divergences and their properties are interesting. However, the authors may need to elaborate with more details to highlight the contribution and distinguish with results in the literature (especially Birrell et al, JMLR'2022a)", "strength_and_weaknesses": "(*) Strength:\n+ The authors propose a new family of regularized Renyi divergences by infimal-convolution with IPM.\n+ The authors derive several theory results for the proposed divergence\n\n(*) Weaknesses:\n+ It seems that some results in the submission appear in the Birrell et al, JMLR'2022a. \n+ It is better to give definitions/explanations for some notations before using them.\n+ It is unclear how to choose the hyperparameters of the proposed divergences (it seems that these parameters play important roles in applications)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster