$f$-Divergence Thermodynamic Variational Objective: a Deformed Geometry Perspective
Authors: Jun Li
Ping Li
Venue: ICLR 2022 Submitted (2022)

Abstract: In this paper, we propose a $f$-divergence Thermodynamic Variational Objective ($f$-TVO). $f$-TVO generalizes the Thermodynamic Variational Objective (TVO) by replacing Kullback–Leibler (KL) divergence with arbitary differeitiable $f$-divergence. In particular, $f$-TVO approximates dual function of model evidence $f^*(p(x))$ rather than the log model evidence $\log p(x)$ in TVO. $f$-TVO is derived from a deformed $\chi$-geometry perspective. By defining $\chi$-exponential family exponential, we are able to integral $f$-TVO along the $\chi$-path, which is the deformed geodesic between variational posterior distribution and true posterior distribution. Optimizing scheme of $f$-TVO includes reparameterization trick and Monte Carlo approximation. Experiments on VAE and Bayesian neural network show that the proposed $f$-TVO performs better than cooresponding baseline $f$-divergence variational inference.

Under review as a conference paper at ICLR 2022
f JDIIVERGENCE THERMODYNAMIC VARIATIONAL OB- JECTIVE: A DEFORMED GEOMETRY PERSPECTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we propose a f-divergence Thermodynamic Variational Objective (f-TVO). f-TVO generalizes the Thermodynamic Variational Objective (TVO) by replacing Kullback-Leibler (KL) divergence with arbitary differeitiable f- divergence. In particular, f-TVO approximates dual function of model evidence (p(())) rather than the log model evidence p(x) in TVO. f-TVO is derived from a deformed X-geometry perspective. By defining x-exponential family ex- ponential, we are able to integral ʃ-TVO along the x-path, which is the deformed geodesic between variational posterior distribution and true posterior distribution. Optimizing scheme of f-TVO includes reparameterization trick and Monte Carlo approximation. Experiments on VAE and Bayesian neural network show that the proposed f-TVO performs better than cooresponding baseline f-divergence variational inference.
INTRODUCTION
Variational inference (VT) is core technique in probabilistic machine learning (Mnih & Gregor 2014, Mnnh & Rezende 2016; Tucker et al. 2017; Maddison et al.] 2017 Paisley et al. 2012
Salimans et 2013: Ranganath al. [014 Kucukelbir al., 2015). Normal Variational inference
tries to perform a tractable posterior distribution approximation for log model evidence estimation log p(x). Commonly, this is done by introducing divergence 2[g(22>)][p(2
between variational
distribution q( (=][z) and true posterior p (=|a:r as regularization term. Then the evidence lower bound (ELBO): log p(x) Dlg(= [x)])(=](>) used as a surrogate optimization objective for maximum
likelihood parameter estimation (Hoffman & Johnson 2016 Duan et al.) 2017 Yang 2017).
Recently, there are serveral directions that generalize normal variational inference, Interested readers can refer to VI's background and progression in (Blei et al. 2017 Zhang et al. 2018). One direction to explore tighter variational bounds (Burda et al.) 2016; Chen et | Masrani et al. 2019
Brekelmans et al. 2020). Among these efforts, the recent Thermodynamic Variational Objective (TVO) approach connects variational inference with thermodynamic integration. In paricular. TVO interprets the log model evidence estimation as 1D integral along the geodesie path between joint distribution of data and hidden variable p(æ, 2) and variational hidden variable distribution (=+[x). By properly choosing intergral intervals of Riemannian sum approximations, the TVO approximates the log likelihood with a series of upper and lower bounds, which are tighter surrogates than the normal variational inference objectives for model parameter likelihood maximization (Masrani et al. 2019
Another direction is trying to extend the normal VI framework by replacing KL divergence with other statistical divergences. Among these efforts. Rényi's a-divergence and x-divergence can be regarded as the root divergences, which makes many existing divergence as special cases including KL-ddiveggence Rényi's 's a- VI (Li & Turner (016) and x- VI (Dieng et al., 2017) are stochastic VI that generalized from stochastic VI (Hoffman et al. 2013) and black-box VI (Ranganath et al.) 2014), and outperform the classical KL-VI in Bayesian regressions and image reconstruction some benchmarks. More generally, recent work f f-divergence variational inference (f-VI) (Wan) et al. 2020) gives general framework that includes all divergences from f-divergence family. The variational objective of f f-divergence VI can be regarded as the surrogate bound of dual function of model evidence f (p(x)))

---

Under review as a conference paper at ICLR 2022
Divergences
f(t)
(t)
Variational Bound
KL divergence
t blog
log t flog t t"
ELBO
KL divergence
logt
EUBO
General x" divergence
ビlーn t, R {0,
CUBO,, TVD RVB
Total Varaition divergence
1 - 1|
|1 1--18 1 logl [ + (a 1)H,
Rényi a divergence
Do (glg))) (a 111
Table 1: Difference divergences and their corresponding f function. Rényi a-divergence is not f-divergence, but it can formulated with Hellinger divergence H. (alpp)) which is an f-divergence.
In this paper, we combine the above two directions together by proposing f-TVO that generalizes the original TVO to arbitary f-divergence in unified framework. f-TVO approximates dual function of model evidence (p((z)) rather than the log model evidence log p(x) TVO. The estimation of f-TVO is established from a deformed x-geometry prospective. By defining x-exponential family distribution, the computation of f-TVO is integrated over the x-path, which is the geodesic between p(z, 2) and q(2 |z) under X-geometry. The proposed f-TVO is tighter bound of dual function of model evidence f (p(x)) rather than the normal variational lower bound in TVO.
Our contributions include the following:
We enrich the TVO by proposing unified f-TVO framework that compatible with an amount existing divergence, such as KL, a divergence, x divergence;
We derive unified f-TVO bound equiped with the upper/llwer bound criteria that improves many existing bounds, such as ELBO, CUBO, RVB;
Experiments show that our proposed f-TVO better optimization surrogate than corre- sponding f-VI.
The rest of the paper is organized as follows: Sec. 2 reviews preliminary knowledges for f-divergence and Thermodynamic Variational Objective, Sec. presents our f-TVO, Sec. eextends approach to more variants f-TVO, and Sec. presents our experiment results.
2 PRELIMINARY
In this section, we will review the preliminary knowledge for f-divergence and Thermodynamic Variational Objective.
2.1 f F-IIVERGENCE
f-divergence is measures of difference between two probability distributions. be defined as follows (Sason & Verdú 2016).
Definition 1. Given convex function ʃ with S(1) 0, then f-divergence from probability distri- bution q(2) to p(z) is defined as:
g(2) / ((222)) b(z)dz Ep p(2)
(()) p(z)
Dy(ella)=
(I)
Many existing well-know divergence be regarded special cases of f-divergence by properly choose the generation function f. Table [ shows the relationship between the sample divergences and their corresponding generation function f.
Wan et al. [Wan et al. 2020) proposed a o -ddivergence variational inference (f-VI) that generalizes normal variational inference to all f-divergences in unified framework. f-VI framework is primarily based on reverse f-divergence (Wan et al.) 2020). One can connect forward f-divergence (y(pl14) with the reverse fddivergence 1y(g]
(p) via dual function, f" as follows.
2

---

Under review as a conference paper at ICLR 2022
Ex, [
log pix)
ELBO
B
B
B
Figure 1: Thermodyyamic Variational Objective (TVO). Lefttevidence lower bound (ELBO); middle:
TVO a Riemannian sum approximation of p(x); right: log model evidence log) TVO is tighter bound of log p(z) than ELBO.
Definition 2. Given a function f. its dual funtior f is defined as:
(*) -t-)/t)
(2)
The dual function f" has the following properties: (D)S*)* (2) is convex iff f is convex, (3) = iff /(1) = 0. f-divergence in equation can be reformulated with the dual function f":
Dy(ella) = (((((3) p(z)
p()))) (p(z)d= = (r (()))) Dy- (gl
(3)
9(2)
As shown in the theorem below, f-variational bound is approximation of the dual function f" of the model evidence p(z):
Theorem 1. The dual function of model evidence f (p(x)) is bounded by the f-variational bound Cy (g.x)
p(z, x))
Ly(g,x) (g( [/"
(x(x))
(4)
(=|xr)
where equality is obtained when q(2(x)) p(2(x:).
The proof of the theorem straightforward as f is convex (Wan et al. 2020
Many well- variation bounds can be represented in the f-VI framework. For example, f-VI recovers to ELBO when f /= <llog I, and f-VI recovers to CUBO,, when f = 11-n - More details shown the Table
2.2 THERMODYNAMIC VARIATIONAL OBJECTIVE
The evidence lower bound (ELBO) is a lower bound of the log evidence of a generative model p(r, 2). can formulated as the difference between log evidence of model evidence log p(x) and KL divergence of variational posterior distribution g(=[<r) and true posterior distribution p(2(xr):
ELBO logr(x) D (<<=|=)]l<<<()))
(5)
The Thermodynamic Variational Objective (TVO) gives a tighter bound of log p(x) than ELBO. As shown in Fig. TVO a Riemannian sum of integral along the geometric path between the model and inference network:
K
p(z, 2(z,2) < L' Blog ((2(z)
p(z, 2) dB log p(x)
ELBO +
E Bmay
log
(6)
K
q(21x))
1=1
TVO
Note:dual function here is different from Fenchel conjugate function: f*( suup.cdom 100 f(z))

---

Under review as a conference paper at ICLR 2022
where [3,, B.,... ,8k] is partition between [0.1], where Po 0, and 3x 1, T is
the normalized geometric combination of p(z,2) 2) and q(z(z), i.. To -==|=)=~ p(x, 2) and
T = /23 where ZB is a normalization constant for each 3. One can verify that xg is member of exponential family distribution, no matter what (=2(2r) and p(x, 2) are:
{ < exp T(x, 2) <(z; 8)}
((]))) p(2; 3) is the log partition function that normalize over where Txx,2) log p(m.s)
X- THERMODYNAMIIC VARIATIONAL OBJECTIVE
In this section, we establish our theory for f- Thermodynamic Variational Objective (f-TVO). We first connect f-divergence with X-geometry. Then, we present f-TVO as an integration along the X-path. Lastly, we specify the computation process of f-TVO and its gradients.
3.1
X- LOGARIITH AND X E-EPONENNTIAL
We first define generalized x-logarithm function for a positive non-increasing function X:
log, (u) J
x(v)du
which is different from the definition in Nauudts 2011 Amari 2016). Notice that equation 8 gives
normal log function log when x(v) The inverse of the loggarii is called x-exponential:
/ ()()dd
expx (u) 1+
where &b is auxiliary function. One determine y for each log, (u), e.g. vb(v) exp(v) when log, (u) log u. x-Logarithm and X X-Exponential have the following properties:
d (1) exp (u) v'(),, (2) = exp, (u),, du
(3)
(log,
x(u)
For any divergence with convex diffrentiable generation function f with f(0) = 1, there exists a continuous positive non-increasing function X that connects the dual function f":
[in)=-( S, x(u)dv
(*(u) log, = (u)=
(10)
then f-divergence can be represented
Dy(al(p) = (pllg) - / p(z) log
g(2)
dz
(11)
p(2)
3.2 THERMODYNAMIC INTEGRATION
Using the x-logarithm function and x-exponential function defined in equation X and equation can define a exponential family distribution under x-geometry:
9
we
Definition 3. x-Exponential Family Distribution:
p(x,0) 0) c(x)
logx
-. xxxx-((8)
(12)
or equivalently:
p(x, 0) exx)exxx(0 (00))
(13)
where x is random variable, is parameter. (()) is the function,, c(x) is the normalization
term.
4

---

Under review as a conference paper at ICLR 2022
Given two unnormalized distribution Ro (z) and #1 (2), define an unnormalized distribution that is the log x interpolation between #2(2) and #1 (z):
( = (1 - 3) log, (mo(z)) + log, (f1(=))
(14)
Its normalized distribution defined as:
*z(2))
T
(15)
Zg
where Z3 (z)d. It's easy show that: #=(=)) on the log x geodesic between #o(2) and T1 (1(2) with x-Exponential Family Distribution as:
Theorem 2. By is also an x-exponential family distribution, no malter tz( and कं (z).
Proof.
#2(2)= expx 3) log, mo(z) + 8 Blog, #1(2))
(1og #0(z)= + Bllog #1(z) - -logx To())))
(16)
By defining the potential energy function: Ug(2) log, (2), then we can estimate the log, the ratio of the normalizing constants via thermodynamic integration:
log, Z1 log Zo [,']
x(Z)) x(+x(=)) (log, #o(2) log #1 =1(2))
(17)
Notice that variational inference can be connected with thermodynamic integration if we set:
#) q(21x)) Zo / ((2(2)d2
Z1 / p(z, 2)x = p(x)
(=) = p(z, z),
(18)
then the log, of model evidence is the thermodynamic integration (TI):
log, p(x)(( S(' Szd3
(19)
x(Zx) where S3 = (=))(=) (log, <(2|=) log, p(x, =))
Recall the relation between f-divergence and function in equation 10 then log, p(x) = f (p(x))
dual function f the model evidence, which corresponds the /-v in (Wan al. 2020
To approximate f-TVO defined in equation 17 we pick the a list of ordered partitions , [Bo, Bi, ,3x], where 30 31 < 3K. 0, 3x 1, then f-TVO the left Riemannian approximation of TI:
f-TVO'
Σ S9
(20)
km
where A BR Bk-1-
3.3 COMPUTATION
In this section, we show how to efficiently compute the f-TVO.
We first rewrite Sn in equation 17 into
S3 /
x(Z3) (log, #o(2) - loogy #1(2)) X (2))
x(Z3)
B xx) (log, #o(2) -logy (z(2))/(((
(21)
5

---

Under review as a conference paper at ICLR 2022
We apply the reparameteriizatoon trick to estimate Sg. We assume there is mapping go(c) that
satisfies: 2 = (c(.). Then the expectation of arbitrary function F(z) over distribution g(2) can be computed as (((=)(((=) (F In particular, use the prevalent Gaussian repa- rameterization: 2 N(, Σ), ie. 2 = Σte,e ~ N(O, I). Then Sy can be reformulated
as:
x(Z3)
S E. x aT (g)((e))) (logx p(c) log, plz, go(e))
(22)
where we define (2(22) = p(e) and p(x,z) = p(z, (x))) in equation Empirically, we can compute Sg as:
S3 Σ w (log p(en) log, p(t,))((e))))
(23)
where w/, = x(En1 R (gg(()))))/ ((tpp)(((((((((())) In practice, we split the computation of S3 into two parts: S3 = I - R:
L
logy p(en)]
N n=
R
Ww logy p(z, (e())))
(24)
This is because we use p(r, ga and log during our computation to avoid numerical overflow or underflow. All the intermediate sum operations are performed by logsumexp operation until we get log I, and log R, then we use exponential operation to complete the computation.
4 GENERALIZING THERMODYNAMES OBJECTIVES
Recall f-TVO in equation 20 is left Riemannian approximation of Thermodynamic integration. Similarly, We can also have right Riemannian approximation of f-TI:
K
f-TVO (K) = Σ S
(25)
1=1
Both left and right Riemannian approximation corresponds existing variational bounds. For example, left single Riemannian approximation of the KL-TVO equals the ELBO, while the right single Riemannian approximation equals to the EUBO. Also, it easy to verify if the dual function I* is increasing (decreasing). then left Riemannian approximation is upper (lower) bound of Thermodynamic integration, while right Riemannian approximation is lower (upper) bound of Thermodynamic integration.
In addition to the left or right Riemannian approximation of Thermodynamic integration, we can also define a "zig-zag" (z) approximation of Thermodynamic integration, which combines left and right Riemannian approximation. As shown in Flg. 2 the "zig-zag" Thermodynamic objective can be formulated as:
ʃ-TVO² Σ
Ax(1(k%2 1)8A, 1(k%2 0)S,
(26)
where 1(A) is indicator function, i.e. 1(A) when condition A true, 1(A) = 0 when condition is false.
EXPERIMENTS
In this section, we evaluate the effectiveness and the wide applicability of f-TVO on two tasks. f-TVO is first used as a surrogate optimization objective f-VI in a Bayesian neural network for linear regression, then used in VAE for image reconstruction and generation. For both tasks, Adam as our optimizer with recommended parameters in (Da 2014). For all experiments, we use partition of the temperature for Riemannian integration.

---

Under review as a conference paper at ICLR 2022
(a)/-TVOL
/TVO-
[c) /-Vo
Figure 2: Variants of f- H-Thermodynamie Variational Objective (f-TVO). (a)Left Riemannian approxi- mation; (b) Right Riemannian approximation; (c)zig-zap bag Riemannian approximation.
Dataset
KL-VI
KL-TVO
X-VI
X-TVO
a-VI
a-TVO
Sai
fa-TVO
Caltech 101
73.72
73.51
73.84
73.61
74.95
74.23
74.90
74.62
Frey Face
160.85
160.58
160.61
160.32
161.06
160.85
160.73
160.35
MNIST 59.03
58.89
62.15
61.77
61.88
61.42
59.47
59.21
Omniglot 109.65
109.52
110.57
109.89
110.75
110.12
108.31
108.09
Table 2: Test reconstruction enrors of f-TVO VAEs. outperform corresponding f-VI.
Lower is better. In most cases, f-TVO
5.1 BAYESIAN VARIATIONAL AUTOENCODER
f-TVO is used in Bayesian VAE for image reconstruction task datasets including Caltech 101 Silhouettes (Cal), Frey Face (Fre), MNIST (MNI). and Omniglot (Omn). We have replaced the conventional ELBO loss function of VAE with the proposed f-TVOs that correspond to flexible divergences. We test and compare the f-TVO VAEs associated with three well-known f-divergences (KL-divergence, Renyi's a-divergence with a 3, and x-divergence with customized divergence which is defined by its dual function fá (1) log" + log
2). We also try
We use the left Riemannian approximation with 5 partitions, i.e. f-TvO (5) in our setting. As shown in Table 2 the proposed f-TVOs outperforms the corresponding baselines in most cases.
Effects of numbers of partitions. Here, we study the effects of different numbers of partitions on Bayesian variational autoencoder in our experiments. We try partition numbers K 1, 5, 10 in the f-TVO, respectively. As shown in Fig. 3. we can see that even partition number is = 2 can improve the baseline performance.
Effects of f-TVO variants. We also study the effects of f-TVO variants defined in Sec. We try all variants f-TVO- "t and f-TVO² on Bayesian variational autoencoder in our experiments. As shown in Fig. three f-TVO variants achieve similar performance.
5.2 BAYESIAN NEURAL NETWORKS
Our experiments of Bayesian neural network are performed on regression tasks. We generally follow the experiments settings as in (Wan al. (020). In particular, We use twelve datasets that are collected from the UCI dataset repository, where each dataset is randomly split into 90%/10% for training and testing. The model is single-layer neural network with 50 hidden units (ReLUs). We use (0; 0, 1) as a Gaussian prior of the network weights and Gaussian approximation to the true posterior.
In the experiments, we try different f-TVOs, including corresponds to KL TVO, a-divergence TVO, where 3, x-square divergence TVO. Following (Wan al. 2020), also define custom divergence, which defined by its dual function Sa (*(() (1), where f*(()

---

Under review as a conference paper at ICLR 2022
181.0
16..
190.
1013
(a)
110.5
Acorddncconn E11
635
955
106.5
(c)
(d)
Figure 3: Effect of partition numbers. We evaluate f-TVO with partition numbers [1,2,5,10] on datasets (a)Caltech (b) Frey Face (c) MNIST (d) Omniglot.
Dataset
KL-VI
KL-TVO
X-VI
X-TVO
a-VI
c-TVO
fe2
fe2-TVO
Airfoil
2.18
2.07
2.36
1.96
2.43
2.14
2.35
2.20
Aquatic
1.17
1.11
1.25
1.12
1.16
1.11
1.14
1.05
Boston
2.88
2.69
2.99
2.80
2.89
2.73
2.86
2.73
Building
1.56
1.38
2.81
2.36
1.87
1.64
1.85
1.65
CCPP
4.19
3.96
4.26
3.96
4.20
3.86
4.35
4.08
Concrete
5.40
5,38
3.51
3.37
5.40
5.24
5.26
5.11
Fish Toxicity
0.95
0.88
0.92
0.89
0.91
0.86
0.89
0.85
Protein
1.96
1.91
2.46
2.24
1.85
1.74
2.00
1.89
Real Estate
7.53
7.39
7.51
7.33
7.50
7.40
7.62
7.45
Stock
3.95
3.78
3.93
3.74
3.99
3.84
3,95
3.81
Wine
.658
.648
.647
.639
.643
.635
.665
.649
Yacht 0.79
0.76
1.23
1.23
1.05
0.95
1.22
1.18
Table 3:
Test RMSE of Bayesian Neural Network for f-VIs and f-TVOs. Lower is better. In most cases, fTVOOs outperform corresponding f-VIs.
-1/6 (logt + tn) - 1/2 (logt partitions, i.e. (-TVO-(5) in setting. As shown in Table the proposed f-TVOs outperform the corresponding baselines.
t + tn) - 1. We use the left Riemannian approximation with
CONCLUSION
In this paper, have proposed general f-TVO framework that is tighter evidence bounds than f-VI. f-TVO unifies all differential f-divergence into TVO from a X X-geometry perspective. By connecting f-divergence with X-geometry, f-TVO is the Riemanian sum of thermodynamic
8

---

Under review as a conference paper at ICLR 2022
101.
214
103.
190.
ITVO Virierss
LTVO Varters
aa)
(b)
515
130.60
106.75
200.22
200.00
138.2
106.50
106.55
(c)
(d)
Figure 4: Effect of f-TVO variants. We evaluate f-TVO with all 3 variants: left riemannian approximation, right Riemannian approximation, "zig-zag" Riemannian approximatin,, on datasets (a)Caltech (b) Frey Face (c) MNIST (d) Omniglot.
Dataset
KL-VI
KL-TVO
X-VI
X-TVO
a-VI
a-TVO
la-VI
f2-TVO
Airfoil
2.22
2.14
2.30
2.02
2.40
2.21
2.29
2.16
Aquatic
1.57
1.52
1.59
1.47
1.54
1.48
1.54
1.39
Boston
2.49
2.32
2.54
2.40
2.48
2.36
2.49
2.38
Building
6.90
6.71
6.85
6.75
6.79
6.56
6.74
6.58
CCPP
2.84
2.65
2.93
2.76
2.84
2.57
2.99
2.80
Concrete
3.23
3,02
2.75
2.51
3.12
2.87
3.10
2.83
Fish Toxicity
1.35
1.36
1.27
1.25
1.31
1.26
1.32
1.27
Protein
2.12
1.91
2.11
1.97
2.14
1.90
2.25
2.06
Real Estate
3.55
3.40
3.69
3.45
3.59
3.38
3.65
3.51
Stock
-1.08
-1.12
-1.08
-1.06
-1.10
-1.08
-1.14
-1.18
Wine
.975
.974
.970
.966
.971
.965
.978
.970
Yacht
1.72
1.68
1.88
1.82
1.87
1.87
2.02
1.95
Table 4: Test negative log-likelihood of Bayesian Neural Network for f-VIs and f-TVOs. Lower is better. most cases, f-TVOs outperform corresponding f-VIs.
integration along the geodesic between q(2(2r) and p(z, =) under X-geometry. Empirical experiments of some instances of f-TVO on the popular benchmarks show the superior performance than state- of-the art results, which imply the flexibility and effectiveness of f-TVO. Future work f-TVO may include more efficient T-TVO optimization methods, and more explorations on the properties of x-geometry.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Caltech 101 silhouettes dataset.
ct umass. htt..//............ma.m.aaaa..
shtm1
Frey face dataset.
Mnist dataset.
http://wee..eenn..c///xxxm/mm/ss
Omniglot dataset.
OMNIGLOT
Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):85--877 2017.
Rob Brekelmans, Vaden Masrani, Frank Wood, Greg Ver Steeg, and Aram Galstyan. All in the exponential family: Bregman duality in thermodynamic variational inference. In International Conference on Machine Learning, pp. 1111-1122. PMLR, 2020.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. ICLR (Poster), 2016.
Liqun Chen, Chenyang Tao, Ruiyi Zhang, Ricardo Henao, and Lawrence Carin Duke. Variational inference and model selection with generalized evidence bounds. In International conference machine learning, pp. 893-902. PMLR, 2018.
Kingma Da. A method for stochastic optimization. arXiv preprint arXiv: 1412.6990, 2014.
Adji B Dieng, Dustin Tran, Rajesh Ranganath, John Paisley, and David M Blei. Variational inference via x upper bound minimization. Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 9729-2738, 2017.
Huiping Duan, Linxiao Yang, Jun Fang, and Hongbin Li. Fast inverse-free sparse bayesian learning via relaxed evidence lower bound maximization. IEEE Signal Processing Letters, 24 2017.
Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, volume 1, pp. 2, 2016.
Matthew Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14(5), 2013.
Alp Kucukelbir, Rajesh Ranganath, Andrew Gelman, and David M Blei. Automatic variational inference in stan. arXiv preprint arXiv: 1 1506.03431, 2015.
Yingzhen Li and Richard E Turner. R Rnny divergence variational inference. arXiv preprint arXiv:1602.02311, 2016.
Chris J Maddison, Dieterich Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, Andriy Mnih, Arnaud Doucet, and Yee Whye Teh. Filtering variational objectives. arXiv preprint arXiv:1705.09279, 2017.
Vaden W Masrani, Tuan Anh Le, and Frank Wood. The thermodynamic variational objective. 2019.
Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In International Conference Machine Learning, pp. 1791-1799, PMLR, 2014.
Andriy Mnih and Danilo Rezende. Variational inference for monte carlo objectives. International Conference on Machine Learning, pp. 2188-2196 2196. PMLR, 2016.
Jan Naudts. Generalised thermostatistics. Springer Science & Business Media, 2011 1.
10

---

Under review as a conference paper at ICLR 2022
John Paisley, David Blei, and Michael Jordan. Variational bayesian inference with stochastic search. arXiv preprint 1rxiv:1206.64333 2012.
Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Artificial intelligence and statistics, pp. 814-822. PMLR, 2014.
Tim Salimans, David Knowles, al. Fixed-form variational posterior approximation through stochastic linear regression. Bayesian Analysis, 8(4):837-882 2013.
Igal Sason and Sergio Verdú. f-divergence inequalities. IEEE Transactions Information Theory,
62(11):5973-6006, 2016.
George Tucker, Andriy Mnih, Chris Maddison, Dieterich Lawson, and Jascha Soht-Dickstein. Rebar: Low-variance, unbiased gradient estimates for discrete latent variable models. arXiv preprini arXiv:1703.07370, 2017.
Neng Wan, Dapeng Li, and Naira Hovakimyan. f-divergence variational inference. Advances in Neural Information Processing Systems, 33, 2020.
Xitong Yang. Understanding the variational lower bound, 2017.
Cheng Zhang, Judith Bütepage, Hedvig Kjellström, and Stephan Mandt. Advances in variational inference. IEEE transactions pattern analysis and machine intelligence, 41(8):2008-200 2026, 2018.


Reviews:
[{"review_id": "_4X1CjCyp6G", "reviewer": "Reviewer_Qe9J", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is well written, however, it lacks some more motivation (and insights) about the extension enabled by deformed geometry. Moreover, the experiments should be improved.\n\nstrengths:\n - novel extensions of TVO towards f-divergences\n - solid theoretical work  \n\nweakness: \n- experiments should be improved and better discussed\n- lack of discussion (and comparison) that connects to related TVO work [1]\n\n\n- It is not immediately clear what are the clear benefits of f-TVO, wrt other objectives that can be defined within TVO (ELBO, EUBO, Renyi, CUBO etc) [1]. \n- The related work section can be improved, discussing work on TVO that already have shown improvements (such as refined partitioning in [1]). \n- The experiments can also be improved, showing standard (from multiple runs) to begin with, and providing more discussion on the results. As of now, here isn't some significant or practical benefits between f-TVO variants and f-VI. The authors also introduce two additional divergences (c1 and c2) which are not motivated and discussed in more detail.\n\nOther comments:\n- Table 1: What is H_alpha ()? (Hellinger?)\n- How were the number of partitions (K=5) and the f-TVO variant (left) chosen for the experiments? It seems that the performance can slightly vary wrt. the choice of divergence and dataset? Is there any intuition behind the choice?\n\n[1] Brekelmans et al (2020). All in the exponential family: Bregman duality in thermodynamic variational inference.", "summary_of_the_paper": "The paper proposes a novel f-divergence Thermodynamic Variational Objective (f-TVO) framework for VI, that extends the TVO towards, a more general, family of f-divergences. The authors propose to use a $\\chi$-deformed exponential distribution, which casts the f-TVO objective as integral along the $\\chi$-path between p(x,z) and q(z|x) (rather than the geometric path in TVOs) under $\\chi$-geometry. The authors propose different variants of f-TVO, that vary between the type of the f-divergance used, as well as how this integral is approximated (K-partitioned ) left-Riemann sum (related to ELBO), right-Riemann sum (related to EUBO) or a 'zig-zag' that alternates between the two. Besides theoretical justifications, results from two sets of experiments show that, in general, the proposed f-TVO perform comparable to or slightly better than the f-VI counterparts, but without clear conclusion wrt the choice of the f-divergence.", "summary_of_the_review": "Overall the paper is well written and (mostly) easy to follow. The ideas and contributions are novel and solid: connecting f-divergances with TVO. However, it lacks some more motivation (and insights) about the connection with deformed geometry. The empirical results are not convincing, since the reported performances, while similar across methods, do not include standard error nor more discussion, which would better highlight the robustness and benefits of the proposed f-TVOs. \n\n----\nPost-rebuttal update:\nMany thanks to the authors for addressing some of my concerns. However, I still think that the experimental setup should be improved and some design choices (eg. wrt the custom divergences) better motivated and evaluated. With this in mind, as well as reading the other reviews (and responses), my initial recommendation will remain the same.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DibGZ44fZD", "reviewer": "Reviewer_6bLb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The general idea does not seem surprisingly novel. Nevertheless, developing the objective and its calculation are technically nontrivial and worth a contribution. Still, I think the current paper does not seem to implement the idea in a comprehensive and compelling way.\n\nPresentation issues:\n* The main technique content introduces what the bound is, how is it developed and how to do the calculation. But I expect more comprehensive (unnecessarily deep and involved) technical investigations that help readers grasp a general understanding of the developed bound. For example, why is it better (e.g., how is it tighter if it is) than existing bounds, particularly f-VI and TVO? When is the bound tight? How does it relate to other bounds? These issues are also related to the motivation for developing this bound and explaining why the new bound achieves a better performance.\n* Eq. (17) seems to be the central technical contribution in this paper, but its derivation is missing. I took some effort and verified the equation myself by the facts listed in the paper, but I still think missing the derivation is a problem. The derivation is even more important than the proof of Theorem 2.\n* The potential energy function $U_\\beta(z)$ is introduced but not used anywhere.\n* I do not quite understand why Theorem 2 is explained as that \"$\\tilde\\pi_\\beta(z)$ is on the log $\\chi$ geodesic between $\\tilde\\pi_0(z)$ and $\\tilde\\pi_1(z)$\". To me, it just shows how $\\tilde\\pi_\\beta(z)$ interpolates between the two ends. If geodesic is mentioned, then (1) conceptually, specify the metric or Riemannian structure that defines the geodesic, and (2) why consider the geodesic, i.e. how the properties of geodesic relate to our desideratum.\n* Some claims are expecting proof or reference. For example, Eq. (6) (there is no reference in Sec. 2.2), the statement above Eq. (10), and Eq. (17).\n* Above Eq. (7), $Z_\\beta$ is a normalizing constant over $(x,z)$ or $z$?\n* There are some grammar and spelling typos (\"differeitiable\" and \"able to integral\" in abstract, \"knowledges\" in introduction, \"a exponential family\", \"two unnormalized distribution\" and \"pick the a list\" in Sec. 3.2).\n\nTechnical problems:\n* In Eq. (21), what is $q(z)$? To make the equality, it should be $q(z|x)$ right? If yes, then why does the reparameterization map $g_\\phi$ does not also take $x$ as an input?\n* Eq. (22) seems problematic. \"Defining $q(z|x) = p(\\epsilon)$\" may break the equality. Instead, $p(\\epsilon)$ in Eq. (22) should be $q(z=g_\\phi(\\epsilon) | x)$, which differs from $p(\\epsilon)$ by a factor $1 / \\sqrt{|\\Sigma|}$ (this can also be derived by the rule of change of variables), which cannot be ignored since it depends on $\\phi$ (and $x$).\n* The calculation of $w_n$ also seems problematic. In the equation below Eq. (23), the $1/N$ factor seems missing in the argument of the firxt $\\chi$. Also, the summation estimates $\\int \\tilde\\pi_\\beta(z) q(z|x) \\\\, \\mathrm{d}z$, but not $Z_\\beta$ which is $\\int \\tilde\\pi_\\beta(z) \\\\, \\mathrm{d}z$.\n* How is $\\tilde\\pi_\\beta(z)$ evaluated? It requires calculating the $\\exp_\\chi$ map, which by definition is rather implicit. So is it evaluated by iteratively solving for the inverse of $\\log_\\chi$, or is known in closed-form? If the latter, how is it known?", "summary_of_the_paper": "The paper presents a new bound as the objective for variational inference. The bound combines the recent progress of thermodynamic variational object which gives a tighter bound than the conventional ELBO, and the f-divergence which induces more possible distribution metrics. Experiments show its better performance on some Bayesian inference tasks.", "summary_of_the_review": "Up to my understanding, there are some technical flaws and the paper does not investigate the proposed bound comprehensively (particularly, the motivation is not supported and the advantage is not clear).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C8BnuLZMoUw", "reviewer": "Reviewer_BE5D", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors provided an interesting theoretical extension of TVO to more general f-divergence families. However, this extension seems not surprising but straightforward. The critical issue of this work is that the authors did not present the motivation for the extension. I expected that numerical experiments support the motivation of extending the f-VI to f-TVO. But the presented numerical experiments are too weak, and I could not find the usefulness of the proposed method. It seems that standard KL-TVO still works very well. I strongly recommend adding additional experiments to support the usefulness of the f-TVO.\n\nPros\n-  Combine two important concepts in VI, TVO and f-VI.\n\nCons\n- Numerical experiments are too weak to support the usefulness of the proposed method. I could not find the practical benefit of the generalization of TVO to f-VI. \n- Experiments are only done for over-parametrized models. But there is no explanation why f-TVO is recommended to use for over-parametrized models.\n- The details of the experimental settings are lacking or insufficient to reproduce the results.\n\nComments and Questions:\n- Please provide the proof for the statement before Eq10\n- No proof of Eq17\n- Below Eq22, $q(z|x)=p(\\epsilon)$ seems strange, should be $p(g(\\epsilon))$ ?\n- Could not follow the discussion below Eq24. Please explicitly write how the authors used the logsumexp trick here. Moreover, if $R$ and $L$ are negative, $\\ln R$ and $\\ln L$ cannot be defined.\n\n- In the numerical experiments, the authors introduced the customized f-divergence, called $f_{c2}$. I know that they were introduced in the previous work [29], but the authors should explain why it is used in numerical experiments and what distinguished properties they have compared to other f-divergences so that non-experts can understand them.\n- What is the reason for introducing the zig-zag integral? In VAE experiments, it did not show significant improvements.\n- In VAE experiments, I could not find the network architectures. Also, I would like to know the performance in the more meaningful measures, such as the test log-likelihood. Moreover, the authors should compare the results with IWAE, since IWAE also uses the multi-sample bound. \n- Based on all the numerical experiments, it seems that KL-TVO seems to show the best performance on average and could not confirm why we use f-TVO.\n\n- All the experiments are done in over-parametrized models. Why?\n- I would like to see results of simple models, for example, GLM, mixture models, or Gaussian processes, in which we can estimate true posterior distributions.  We can understand the solution of f-TVO more intuitively in those models than in over parametrized models.", "summary_of_the_paper": "This paper proposed new variational inference that combines f-divergence variational inference and the thermodynamic variational objective. The authors introduced several new concepts of exponential families to extend TVO. Finally, the authors provided the estimator of the gradient of the objective function based on the reparametrization trick.", "summary_of_the_review": "The extension of TVO to f-TVO is an interesting direction, but I think the experimental results are too weak to support the usefulness of the proposed method. Moreover, the description of the experimental settings is limited to reproduce the results. I recommend improving writing and experimentation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "40hEkrDtAGv", "reviewer": "Reviewer_QFxg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The generalized $\\chi$ exponential family part is interesting and new to me. But it seems inconsistency in the definition, the authors need to make careful and clear explanations and check every detail before submission.\n\n\nWeakness:\n- The definition of f-divergence is different from the commonly used one, $D_f(p||q) = \\int q f(p/q) dz$.\n\n- Equation 8 does not hold when $\\log_{\\chi}(u) = \\log u$, $\\chi(v)=1/v$, there is an additional constant 1.\n- What is the domain of the function $\\chi$? \n- Convex differentiable generation function f with $f(0)=1$ or $f(1)=0$?\n- How to guarantee the existence of the function \\chi for any generation function f? For example, when $f^*(u)=u^2-1$ is convex differentiable and $f^*(1)=0$. When u is larger than 1, $f^*(u)$ is positive while $-\\int_1^u \\chi(v)dv$ is negative based on the condition on function $\\chi$.\n- Equation 11 $D_f(p||q) = -\\int p(z) \\log_\\chi (\\frac{p(z)}{q(z)})dz$. \n- Can you provide the derivation of equation 17? I do not think this is a direct result. \n- I did not find the monotonicity of $S_\\beta$ in the paper. One key point in TVO is the integrand is monotonically increasing, thus the $\\log$ probability can be estimated by the Riemann sum. \n\ntypo: \n- page 2: $f$-divergence is a measures --> measure\n- page 2: $f$-TVO in contributions should use the same font as the other part of the paper.\n- page 4: we can define a exponential --> an exponential\n-page 4: $x$ should be $\\textbf{x}$ in definition 3.\n\n", "summary_of_the_paper": "This paper proposed an $f$-divergence TVO, which includes some existing works e.g., RVB, CUBO, ELBO into a unified framework. This paper's main idea is to transform $f$-divergence into a generalized $\\chi$-exponenetial family and integral TVO along the $\\chi$-path. The paper provides some theoretical analysis and the optimization methods of the suggested framework and supports the proposed $f$-TVO with numerical results. ", "summary_of_the_review": "In all, the theoretical part is intriguing but not convincing enough. There are many typos and inconsistencies in the notations and computations, I expect the authors to provide some supplementary materials to support the theory, including all the mathematical derivations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The four reviewers believed the paper was below threshold for acceptance to ICLR. They raised concerns with the experimental evaluation and thought that the paper could benefit from another edit to help with the clarity.
Decision: Reject