Variational Learning of Fractional Posteriors
Authors: Kian Ming A. Chai
Edwin V. Bonilla
Venue: ICML 2025 poster (2025)
Keywords: Variational Inference
Fractional Posterior
Primary area: probabilistic_methods->variational_inference
TL;DR: A one-parameter variational objective that lower bounds the data evidence and enables the estimation of approximate fractional posteriors.

Abstract: We introduce a novel one-parameter variational objective that lower bounds the data evidence and enables the estimation of approximate fractional posteriors. We extend this framework to hierarchical construction and Bayes posteriors, offering a versatile tool for probabilistic modelling. We demonstrate two cases where gradients can be obtained analytically and a simulation study on mixture models showing that our fractional posteriors can be used to achieve better calibration compared to posteriors from the conventional variational bound. When applied to variational autoencoders (VAEs), our approach attains higher evidence bounds and enables learning of high-performing approximate Bayes posteriors jointly with fractional posteriors. We show that VAEs trained with fractional posteriors produce decoders that are better aligned for generation from the prior.

Variational Learning of Fractional Posteriors
Kian Ming A. Chai
Edwin V. Bonilla"
Abstract
posteriors (Grünwald & van Ommen 2017 Bhattacharyal ct 2019) have emerged generalization of Bayesian in- ference to address these. Unlike the Bayes posterior, which fully incorporates the likelihood, a fractional posterior has an exponent that weighs the likelihood to temper its influ- ence. This approach has been shown to enhance robustness in misspecified models and has strong connections to PAC- Bayesian bounds (Bhattacharya et al.) 2019), which control generalization error in statistical learning.
We introduce a novel one-parameter variational objective that lower bounds the data evidence and enables the estimation of approximate fractional posteriors. We extend this framework to hierar- chical construction and Bayes posteriors, offering a versatile tool for probabilistic modelling. We demonstrate two cases where gradients can be ob- tained analytically and a simulation study on mix- ture models showing that fractional posteriors can be used to achieve better calibration compared to posteriors from the conventional variational bound. When applied to variational autoencoders (VAFs), our approach attains higher evidence bounds and enables learning of high-performing approximate Bayes posteriors jointly with frac- tional posteriors. We show that VAEs trained with fractional posteriors produce decoders that are better aligned for generation from the prior.
This work introduces new variational framework that gen- eralises conventional variational inference (VI) by allow- ing the approximation of fractional posteriors, enabling im- proved posterior flexibility and calibration. As standard VI based on ELBO maximisation, our approach provides lower bound on the marginal likelihood and extends to hierarchical construction (Ranganath et al. (0166) and Bayes posteriors. Thus it offers flexible trade-off between evi- dence maximization and posterior calibration, bridging the gap between standard VI and fractional Bayesian inference.
We explore both analytical and empirical insights into vari- ational learning of fractional posteriors. First, we identify cases where gradients can be derived analytically, elim- inating the need for gradient estimators (Roeder et al., 2017). Next, we perform a simulation study on mixture models, demonstrating that fractional posteriors achieve better-calibrated uncertainties compared to conventional VI. Finally, we consider variational autoencoders (VAEs) Kingma & Welling 2014), showing that fractional posteri- ors not only give higher evidence bounds but also enhance generative performance by aligning decoders with the prior known issue in standard VAEs (Dai & Wipi 2019).
1. Introduction
Exact Bayesian inference is intractable for most models of interest in machine learning. Variational methods (Jordan
1999 Minka 200T Opper & Winther 2005 Blei al.
al. 2017) address this by casting the required integration as optimisation. These methods have two objectives: to estimate the marginal likelihood or data evidence (MacKay 2003) for model comparison or model optimisation; and to obtain an approximate Bayes posterior for prediction.
The widely used evidence lower bound (ELBO) (Jordan let al., 1999) often leads to underestimated uncertainty and suboptimal posterior calibration (Wang & Tittering- ton 2005 Bishop, 2006) Yao al. 2018). These deficien- cies can be compounded by challenges inherent to general Bayesian modelling, such as misspecificationn Fractional
This work advances the field of approximate Bayesian infer- ence with a theoretically grounded and empirically validated approach to fractional variational inference. It demonstrates that fractional posteriors can improve model calibration and yield better generative models, and it offers alternative standard variational inference and learning approaches.
'DSO National Laboratories, Singapore *CSIRO's Data61, Australia. Conresoondence to: Kian Ming A. Chai <ckianmin dso.org.sg>, Edwin V. Bonilla <ed- win.bonilla @ csiro.au>.
Notation We use letter p for model distributions and let- ters and r for approximate distributions. The tilde (*) accent is for the unnormalised version of the distribution that it modifies; and the asterisk (*) superscript is for the optimised versions. The unaccented letter Z is for the nor-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Variational Learning of Fractional Posteriors
malising constant, and 2 is for an arbitrary scaling constant. We reserve ELBO for the conventional bound (Jordan al.. 1999). Boldfaces are used only when needed to distinguish vectors from scalars. We omit the approximate qualifier from approximate posterior, unless the context requires it. When comparing evidence bounds, we use the adjective tighter if we can ascertain that the bound is closer to a fixed evidence value; and we use the adjective higher if bounds cannot be compared for tightness because their correspond- ing fixed evidences are different. The latter is limited to sections 5.2 and 5.3 when we optimise the likelihood of the model. Table 3 in section A lists the bounds in this paper.
The derivation does not require ((=) to be distribution. However Y must be in L to apply the Hölder's inequality, that is, supp S supp p. Hence, the must be consistent with the prior p. desideratum. If ((x) ) is a distribution q(2), that is, f <(=)dz = 1, then the second term of the bound is the Rényi divergence D1/y(a(=)llp(x)))
Our objective C, in eq. ①I can be seen as an example of the generalized variational inference framework (Knoblauch) et al. 2022 However, it is uniquely derived as lower bound to the log-evidence, so it naturally encodes the Oc- cam's 's razor principle and can be used for model optimisa- tion (MacKay) 2003 Chapter 28). We can relate this ob- jective to the conventional ELBO (Lemma A shows that lim b. = LELBO using the I.' L''ôôital's rule and the convergence of Rényi KL divergence) and also provide two related upper bounds (Lemmas A. and A.2
2. A Lower Bound with Hölder's Inequality
The log-evidence Levd def DD) of data D for genera- tive model involving auxillary variable is bounded by the variational Rényi (lower) bound 2R (Li & Turner 2016 Theorem 1) for a > 0:
2.1. Fractional Posteriors
The optimal Cy is tight at (* = p(D)>>''(())/ where 2 can be the normalising constant:
Levd 2 LB def 1 log / (mD):))))))))) -a dz.
1
The Kullback-Leible divergence (KI., + 1) is the only case where the chain rule of conditional probability holds exactly to get the conventional ELBO LELBO dud I q=)) log p(D]=)d J q(2) volves the expected log conditional likelihood (first term) that encourages data fitting and the KL term between the approximate posterior (2) and the prior p(z) (second term) that acts as a regulariser to bias the posterior towards the prior. This decomposition is not possible for other values of a, so L generally cannot be expressed in such terms.
1 log 2-1 / p(D)=))(z)d=
1
7 log 2 1/1 / pl D(D)z)p(2)dz
1/7
This in-
1-7
llgg_ p(D)2))(2)dz
Levdd
Since C. is lower bound, this already proves the optimality of q" (section gives variational derivation). In con- trast, the gap between Leva and C is the Rényi divergence D Da(9(2) ( D)] (section A.2), so C I optimal g(2) is the exact Bayes posterior p(= DD. Nonetheless, L. is related to CB by suitable change of distributions (section A.3)
To this end, we revert to the original log-evidence Levd and apply Hölder's 's inequality (Rogers 1888) in the manner of (B[(x]=11)*
8 2 |[|XY|]/ (IC (I[I))))))
" where 8 B+y=
The above shows that the bound is tight at a fractionaal pos- terior where the data- likelihook is weighted with y E (0. 1) (Bhattacharya et al., 2019). This is more generally known as the Gibbs posterior (Zhang) 1999| Alquier et al. 2016). the power posterior (Friel & Pettiti 2008), or the tempered posterior (Pitas & Arbel 2024). As mentioned in the in- troduction, fractional posteriors are related to robustness in misspecified models (Bhattacharya et al. 2019).
1 and B,7 (0, 1), with
EH] del [ p(z) dz,
X Ehl ((D)2) Y thd (<))/((()
to obtain
1 = 3 100 log (INE))(D)ss))) 9
Levd
2 1
((())
We have obtained the fractional posterior directly from op- timising Ly. which allows approximations to the unnor- malised fractional posterior via optimisation within an as- sumed family of non-negative functions. It is achieved without relying on PAC-Bayes or modifying the likelihood, and it follows from optimising a lower bound on Leva. is an alternative to the approach by Alquier al. (2016).
log
3
(o)()((()))())) 177 dz)
1 log Za
1 1-~ log Ze
7
def E,y,
(1)
where we have expressed B in y, and we have data-fitting and regularisation (complexity) terms
(z) d (()))(
(z(z) der p(2))1//>
2.2. On the choice of
Za def / [
Z duf / (z)))dz.
It is generally futile to seek a a giving the tightest bound: for a given probabilistic model and fixed posterior q, 7
2

---

Variational Learning of Fractional Posteriors
depends on q. For, if q is close the exact Bayes posterior, then >" 1: if q is close an exact fractional posterior, then 7- is that fraction. The situation is the same if we learn q within a family Q. If Q contains all the exact posteriors, Bayes and fractionals, then all values of ys are optimal because all give Levd after optimisation. If, however, Q can approximate only certain fractional posteriors well, then the corresponding yss will give the tightest bounds.
g(u) is known to be the delta distribution located for opti-
mal {(=/u) (Yin & Zhou 2018, Proposition 1). However, deviations from this property may happen in practice (see
sections B.2
and C.4)
2.3.2 2. BAYES POSTERIOR
We can bound the data term in C, using Jensen's 's incquality with another variational distribution r(z):
Nonetheless, if one applies approximate inference analyt- ically to a problem that is specified with an explicit prior, such as the normal distribution, it is typical to choose Q to be in the same family as the prior, such that Q includes the prior and its neighbourhood. In this case, we expect optimising with small y to give consistently tighter bounds. This is shown in section 5.1 empirically.
log Za 2 (1 (1
/ r())) log ((D((()))
- r((2) log
r(2) dz
q(z)
so that logarithm over the product of likelihoods becomes sum over the logarithms of each likelihood, and we may sample over the data points. Section 3.1.3 gives a different but more specific bound for the mixture model.
In similar fashion, for challenging data sets for which we use neural networks, we expect smaller 7 to give tighter bounds for simpler neural networks that can better approxi- mate the prior and the fractional posteriors than the Bayes posterior. This is supported in section C.4 empirically
The above bound is exact at (=) xx ((z) p(D))) If (()) is also optimal, then (z) x ((z))(D)z) which is the Bayes posterior. Combining the above bound with eq. [I] gives a bound on Levd involving both KL and Rényi diver- gences. We shall denote this bound by Ch
Considerations other than bounds may influence the choice of y. In section [5.]] we use calibration; in section 5.3 we want posterior that is close to the prior.
If we fix r(z) regardless of its optimality, and then optimise for q(=), we obtain q" (z) x r(=)"p(z)²> interpolating between the fixed r(2) and the model prior p(=). This shows that r(z) has a constraining effect on q(=). so the fractional posteriors approximated by C. are in general different from those approximated by using Ly In particular, if r(2) itself is fractional posterior with fraction 7', then ((2) has at best fraction 7/7. This is shown empirically in section 5.2.1 5.2.1
2.3. Extensions to Hierarchical Constructions
We give two extensions to Cy to allow for more expressive fractional and Bayes posteriors using mixing. We show in sections 1 and A.5.2 that degeneracy of the mixing distribution is not necessary for optimality, in contrast to the case for ELBO (Yin & Zhou 2018 Proposition 1).
2.3. 1. FRACTIONAL POSTERIOR
For a fixed r(z), C.), is upper bounded by LELRO (see Lemma A.4). so Ch in itself has limited use. However, we can use it with the hierarchical posterior model (sec- tion 2.3. to give more expressive posteriors. For this, we have to go beyond just applying the hierarchical model on r(z) because this will result in degenerate mixing distri- bution for r(2), since the terms involved are exactly the same as in ELBO. To prevent degeneracy, we apply hier- archical model to both q(2) and r(z), with the same mix- ing distribution q(u). That is, 7(z) def f [(()()))(( (u))d and r(z) def J r(zlu)q(u)dd Under this setting, we apply eq. ( on the Rényi divergence term and the convexity on the KI. term to obtain bound we call Cb (see section A.4)
Let @(2) S ʃ (u) be a hierarchical model of the posterior distribution using mixing variable Jensen's 's inequality for convexity of powers above unity gives
1/7 Ze = / (f ((shu)atwdd)) p(2))-1/76)
$ / (/
p(z) -1/)
= / (/ ((=))) q(u)du) p(2)2-1/ndz=
= -f ( g()))
(2)
We may substitute this into eq. obtain another bound, which we shall call Ct. This lower bound allows Monte Carlo estimates of the integral by only using samples from the posterior (z/[u) (see section 4p..
Leva 2 lf r(z) (()l))d(w) log DD== dzdu
1-7 // ((all)))(u) log r(=lu) dzdu
1
q(2lu)
A similar approach has been used to lower bound the ELBO
7 log // (fotelaatt))
((<]|)) 1/7-1
p(z) dzdu.
Yin & Zhou, 2018 Theorem 1). There, the optimal for
1 1
3

---

Variational Learning of Fractional Posteriors
3. Learning
Where the exact fractional posterior could be complicated, we assume functional form for the approximate posterior.
Let q be parameterised by 0. Under regularity conditions,
Consider the model with a multinomial logit likelihood for
@f.,
1 / (aa(2)) /ea(t)) - {o(2)) (c)=)) 2 Alog
C classes and a standard Gaussian prior:
dz, 00
00
where ga(z) _ (z and (=()) def dc(=)/Ze are nor- malised distributions, and we have used the log-derivative trick Ölog //00- = (1/4)aQ/00) At the optimal q", both gc(2) and (=(=) equates the exact Bayes posterior p==]D)) Setting the gradient to zero entails matching the expectations of the gradients of log q under qe and ga.
p(x|2) = Ee exp &e
exp= E
p(z) = II
V2T
1
exp
2
c=1
For n data points, we choose 1/7 = 1 + 1/n and let
n/(n+)
[I [I
C
((z) =
exp 2
Gradients for Cty, and Chy can be similarly expressed.
Then
(c(z) (Te=I M(zel (Eez exp 2c)
3.1. Case Studies
Eee 2 te [el=:-Warr imee, (8.))
We study three cases of applying C, analytically. The first case where exact inference is possible is illustrative. The other cases, where exact inference is not, demonstrate where using C, can be useful.
(a(z) x [TTe--NZellee o2)) Ili= exp (Zz, / (n+ 1))
x IT N (zellic + noo? nof..??)
3.1.1. EXPONENTIAL FAMILY
where
me Jer Me(n 1)
no² + ob
Consider D to be a collection of independent data {æ1...., 3n in the exponential family with the conju- gate prior setting: p(æ;|z)) = h(x;) exp (zTt(z;) - a(z)); p(z(u,n) x) = g(v, k)) exp (xTV~ - ka(z)); and (( A) exp (z μ >a(z)). with t being the sufficient statistic, the natural parameter, the log-partition function; v and x the parameters for the prior; and μ and x the pa- rameters for the posterior. Then &log /0p) = z and a log /(z) = -az),, and
2 del
n + 1 o2
and ne def Σ 8(c. is the number of data points of class c. The last expression of qe has normalising constant Σe exp(m, + s²2/2,, and the last expression for is normalised because it is a product of independent Gaussian distributions. The gradients with respect to the parameters and the required expectations are given in section A.6 This is an example where we need only the unnormalised density q during optimisation.
(=(2) x exp(=" T (1/7 + - - (z()) x exp(z" TT (μ + (1 -1 E.= ((x))) - k(n)a(=))
3.1.3. MIXTURE MODEL
where k())) कu X+ (1 - 0. The sufficient statistics for conjugate distributions are z and -a -a(z), so
A common model in the Bayesian literature is the mixture model. For n samples {n;)²= and K components with parameters {ux) K independently drawn from p(ux). ). the evidence is P(c)_ f p(u)
E.g. [z] = //7 + (1 1/7)z/
[a(=a(z)] = k(k://>
(()))) I[= p(=, [c,, u. u)du (Blei et al., 2017 Equation 9), where G E (1,..., K} is the latent as- signed cluster for the ith sample, and the c,s are independent. The outer sum over KT cluster assignments makes exact inference intractable.
Egg [z] = μ + (1 7) Σ, t(x:)
aa()) = k(n).
Zeroing gradients 0pp and 00x gives the follow-
ing parameters for q as expected:
# v + Ei_ t(x,)
A= m.
Assume mean-field approximation for the posterior: q(u, c) = I g(ux) ([=_1((s)) Like ELBO, we may apply the L. bound to convert the innermost product to an outer sum for the likelihood term. Alternatively, we can first apply C., for the variational posterior q(u) of the compo- nent means, and then the ELBO for the cluster assignments c, so that the Za term in eq. ( is lower bounded by
The parameters interpolate between the prior and the Bayes posterior, as a consequence that exact inference is achiev- able. This is not true for more general models and approxi- mate inference may be required.
3.2.2. MULTINOMIAL DATA WITH GAUSSIAN PRIOR
In the previous case where exact posteriors can be obtained, it is not necessary to derive the gradients, since we already know the functional form of these posteriors.
/ (q(u) III II (olzl ()(cs))(())))))))))))))))))))))) (17(())) du.
1=1c=1
4

---

Variational Learning of Fractional Posteriors
Define the variational parameters PPu = ((c = k). i ,...., n, k ,...., K. Simplifying with the mean-field independence assumption, the lower bound is (section A.7
not be known explicitly, but are required to be able to draw the (u,, =)) samples and to know the conditional g(zlu) exactly. This particular model for g(2) is the semi-
implicit hierarchical construction (in & Zhou 2018).
log ((ux))((z))ux) p)s dug
For Cbb, we also have N₂/Nng samples z'4 ~ r(2lus). Sec-
tion provides more details.
Pik log (pst/plc= = k))
5. Experiments
log / (p(u))))(
We provide three experiments. The first uses analytical up- dates to infer the posteriors for given model, a variational inference task; the second and third, Monte Carlo sampling with the reparameterisation trick (Kingma & Welling 2014) to infer the posteriors and also learn the hyperparameters of the model, a variational learning task. For y 1.0 we use the standard ELBO implementation directly. We refer the reader to Table E in section [A] as reminder of the bounds used in the paper and evaluated in this section.
1 7
Identifying terms with Cnp optimality gives g(ux)) x p(us) I[ p( ((l..//)) For piu. we first define distri- butions % (2x) oc q((x)) (((,))p(s))))))) (eslss//))((e (1=)(_,,, which introduces the proportion of the ith sample omitted in q(ux). Then PPtk x p(c,) exp (E,g [log [log p(a)/2x))) - in this way, q(c,) approximates the Bayes posterior by using the full contri- bution of likelihood due to Tyg- A full ELBO solution is obtained by setting q: (1x) = ( ) for all i.
for the updates; and in particular,
5.1. Calibration Study for Mixture Models
We evaluate the quality of the learnt fractional posteriors by examining the calibration diagnostics for a one-dimensional mixture model. We use the Gaussian mixture model (GMM) of Blei et al. [017 $2.1). where each observation is drawn with white noise (that is, variance 2 = 1) from one of the K components with equal probability, and the component means have independent and identical Gaussian priors. We infer posteriors over component means using variational inference on set of observations. For a given significance level a, actual coverage is the long-term frequency that the - a credible interval from a posterior includes the true component mean. The credible interval is calibrated when the actual coverage is 1 a. is known that the posteriors from ELBO are overconfident (Wang & Titterington 2005).
If each conditional likelihood p(z; (1)) is in the exponential family and the prior p(ux) is conjugate to it, then g(ux) is in the same conjugate family that have the sufficient statistics of the data weighted by Pak and 7. Consider the Gaussian mixture model of Blei et al. [2117] $2.1), where the com- ponent priors are identically normal with mean zero and variance o²: the assignment priors are identically uniform; and the likelihood is unit variance normal centered at ug. Then q(us) is normal with mean and variance
7 El Pitte
and 1/02 + Pak 1/02 + + Ei Psk
The same expressions are obtained from the approxi- mate Bayes posterior r(ux) and the prior p(u+) using p(u)))(()))>==) but the assignment probabilities pas within are different. Section C.2 illlustras the difference.
We use K 2 components centered at μ1 -2 and μ22 2, and we compute the empirical coverage K over 5, 000 repli- cas of n = 400 observations (Syring & Martin 2019. $S2). For each replica, we obtain the approximate fractional pos- teriors for 7 = 0.1 to 0.9 in intervals of 0.1, and the ap- proximate Bayes posterior using ELBO (see section 3.1.3). Using a 0.05, we find that the posteriors from BLO and ? 0.9 are overconfident, that is, < 1- ax; and those from < 0.8 are conservative (first set of results in Table D. Moreover, the interval lengths ts decrease with 7. These findings conform to our expectations of fractional posteriors, and they demonstrate that optimising C. gives approximate fractional posteriors with the intended properties.
4. Monte Carlo Estimates
If we have N₂ samples 2,8 from distribution q(2) = 7(=)/Z for known normalising constant Z, we can use them to esti- mate Ze and Za in eq. O If we only have the unnormalised density, then estimating C., requires the normalising factor Z. An alternative is to introduce a mixing distribution and use the Ch bound:
Ch. log N. Σ ((D)=()))))) 1-7
To see the effect of the interval length l on k, we measure the coverages when, for each replica, the component means are from the Bayes posterior but the component variances are from fractional posteriors with y = 0.1, 0.5 or 0.9. We find the coverages for such conflated models C,,s match that
7 log N, Σ, Ed(tt (u:) (m))p((z)))))))))))
where there are now Nº samples from U4- ~ g(u) followed by No/N/ samples zij ~ ((2lu4). In this setting, ((2) need
5

---

Variational Learning of Fractional Posteriors
Table I: Calibration study of GMM at a = 0.05 signif- icance. The empirical coverages k (higher is better) and average interval lengths & (shorter better) for each of the [2i: means are shown. The last column gives the bounds to Levd (higher better). The first set of results is from modelling with different ys (C1.0 is ELBO). Results for y E {0.2, 0.4, 0.6, 0.8) are omitted for brevity, but the trend remains. The second set is from conflated models combining the means from ELBO and the variances from C. The third set is from calibrations of using the first set of results. The y values for R& and R. are 0.785 and 0.798.
In practice, when replications of data sets are not available, hootstrapping can be used (Syring & Martin 2019). Both Rs and R. are shown to be effective, and which to use in practice will depend primarily on the nature of data collec- tion. Section C.1 gives additional examples; here we note that analysis for K > components is complicated by the complex marginal likelihood landscape (Jin et al. 2016).
Bounds With current experimental settings, we perform importance sampling using 1,000 samples to estimate the log-evidence to be -827.2. So, while La.1 is the tightest (last column in Table I in this scenerio, it is still rather loose. Section provides more details.
11
P12
K
L
bound
Co.1
1.0000
0.8515
1.0000
0.8987
-832.5
5.2. Variational Autoencoder
fa.
0.9994
0.4924
0.9988
0.5200
-833.3
Variational autoencoder (VAE) (Kingma & Welling 2014)
Ca.s
0.9876
0.3816
0.9860
0.4029
-834.0
provides a variational objective learn the the encoder and decoder neural networks of autoencoder. Though there are many variations (for example, Tomezak & Welling Higgins et al. (2017), we compare with the standard VAE since our aim is to investigate differences with ELBO. VAE is a local latent variable model that separately applies ELBO to each datum. This will be the same for our bounds.
Co.7
0.9694
0.3225
0.9606
0.3406
-834.4
fa.
0.9438
0.2845
0.9334
0.3004
-834.8
L1.
0.9278
0.2699
0.9182
0.2850
834.9
Co.1 1.0000
0.8515
1.0000
0.8987
839.3
Co.5
0.9878
0.3816
0.9858
0.4029
834.5
Co.9
0.9436
0.2845
0.9334
0.3004
-834.8
Re 0.9588
0.3046
0.9474
0.3216
834.6
We follow the experimental setup and the neural network models of |Ruthotto & Haber (202T) for the gray-scale
Rx
0.9570
0.3021
0.9458
0.3190
834.6
MNIST dataset (Lecun et al. 1998). The latent space is two-dimensional, so we can inspect the posteriors visually. We make three changes (see section C.4]: most significantly we use the continuous Bernoulli distribution (Loaiza-Ganem) & Cunningham 2019) as the likelihood function. We use four values of ys: 1.0 (ELBO), 0.9, 0.5 and 0.1.
of the corresponding fractional posteriors in general, but for a minority of replicas changing the variances is insufficient (compare the coverages of C,s to L., in Table
Here, we investigate two calibration strategies, one using interval lengths is and the other using coverages ks. Given our knowledge of the model, sans the locations of the com- ponents, we expect n K observations per component, so their sample mean has variance K Kohhs/n. n.. Combining this with the critical value for a provides an interval length e that we expect in the ideal case. For each replica and each component k we perform linear regression on 7 against using the results from C. to predict " at C*. We average the T to obtain 7 for that replica. The model that optimises Cy- in then computed, for each replica. We call this Re.
For the posterior family, use an explicit normal dis- tribution (following Ruthotto & Haber (2021)) and also semi- comi-impiicit distribution. For the latter we use three-layer neural network for the implicit distribution, similar to that used by Yin & Zhou (2018) (details in section C.4p. The choice of posterior family affects only the encoder structure.
In VAE, the log-evidence Leva depends the prior and the likelihood, which in turn depends on the learnt VAE decoder. Hence, optimising the decoder parameters can be seen as an ML-I1 procedure (Wang et al., 2019). Therefore, when comparing the evidence bounds after optimisation, we can only say that the optimised models provide certain guaran- tees on Levd, with higher bounds giving better guarantees. For tightness of bounds, see section C.4 (Table 7
The other strategy, called Rx, has to be performed after computing the ks over the replicas. This regresses linearly ? against using the results from C.y to predict the % at 1 - a coverage, for each component k. We then obtain single 7" by averaging the r/s. For each replica, the model that optimises L. is then computed.
We first look at the effect of and posterior families, and where C.y uses the explicit distributions and C/ uses the semi-implicit distributions. While the learning objectives are on the training set, we also examine them on the test set to assess generalisation. We observe smaller ys gives higher final evidence bounds (third and fourth columns of Table 2
In this study, we find both R₂ and R2 to provide coverages close to - a (last set of results in Table []. For Re, the average value of 7 0.78, while for R. the value is 0.80.
6

---

Variational Learning of Fractional Posteriors
Table 2: Average log-evidences (higher better) over data samples, and its breakdown for VAE on MNIST data sets. We give the mean and three standard deviations of these averages over ten experimental runs, For Monte Carlo averages, 1,024 samples are used (32 x 32 for semi-implicit posteriors). For y = 1.0, the figures are the same under Test using Objective and Test using ELBO. For the first eight rows, the columns under Test using ELBO are solely for diagnostics to understand the learnt posteriors using the same metries: they are not performance measures. For C we show the addition of the KL divergence and Rényi divergence for Test using Objective, and for Test using ELBO evaluate the Bayes posterior r.
Test using Objective
Test using ELBO
Objective
Train (Total)
Total
data
div
Total
data
div
by
1.0
1614.3+11.
1588.5
5.3
1583.2=14a
1588.5=14.
5.3
0.9
1648.5=s.1
1639.3+4..
1641.9+4.s
2.6+0.4
1452.6+4s.
1455.0+4s.
2.4+o.s
0.5
1675.925.
1672.825.
1674.726.0
1.920.
1318.7 12.
1320. 1242
1.4
0.1
1680.1=2.9
1677.2#3.
1679.5#a.4
2.3+0.a
1322.8=49.a
1324.2=40.
1.3
1.0
1639.6:14.
1609.6=20..
1614.6#20.b
5.0+a.a
1609.7
1614.6
5.0
0.9
1657.7+a.
1647.8+5..
1651.1+6.6
3.3+o.a
1534.6+sa
1537. 8
3.2+
0.5
1677.424.
1674.425.0
1676.425.o
2. 2.1
1366.0 2266002zz.
1367. 2622
1.720.2
0.1
1681.422.7
1678.72a.1
1681.2+3.2
2.5
1355.62a7.
1357.1
1.620.a
Chh
0.9
1636.2 11..6
1608.8#2s.a
1613.4+22.n
4.0+0.2 + 0.6+0.s
1607.7=2s.0
1613.4+22..
5.7+1.0
0.5
1635.2+10.
1608.0=25.
1612.7
4.3+0.2 + 0.4+0.2
1607.4425.
1612. 7
5.3+0.9
0.1
1633.5212.0
1607.5216.1
1612.4216.1
4.6 + 0.3
1607.3:1e.
1612.4218.
5.1
first two sets of results), showing that Cy and Ch can be better than LELBO. This implies that using a range of ys useful for model selection, comparison and optimisation. Moreover, Lo.9 with the simpler explicit posterior already gives higher bound than Ch.o (ELBO) with the semi-implicit posterior (compare their fourth columns), illustrating that is more impactful than the posterior family. Nonethe- less, for the same y, the semi-implicit posterior family gives higher evidence bounds (compare the first two sets of re- sults), showing that Ch is viable approach to learning within the semi-implicit family.
in most cases (Figs. 6b to 6d) This demonstrates that learn- ing with our bounds is a viable alternative to other methods
Yii & Zhou 2018 Titsias & Ruiz 2019 Uppal et al., 20033 to prevent collapse of the implicit distributions.
Because of the degeneracy of hierarchical posterior for the ELBO, we have expected to find the results of Ct.o to be very similar to that of L However, this is not the case here. We postulate that the different gradients and the ad- ditional implicit samples have led to different learning dy- namics and allow C)., escape local optimas in the neural network parameter space. The large variances in the train objectives across the ten experimental runs support this.
The Rényi and KI. divergences generally increase with y (sixth and ninth columns in Table In particular, the trend for KL validates that we are learning fractional posteriors closer to the prior for smaller yss The means of the explicit posterior distributions have also less spread for smaller ys (see Fig. in in section C samples from these distributions, which depends the learnt variances, also demonstrate the same (Fig. [5] section C.. This also means that the data is less fitted for smaller yss, which is generally shown from the data fit term in ELBO (eighth column in Table
Bounds For limited comparison on the tightness of the bounds with respect to common Levd, we take single run of LELBO for the explicit posterior and uses its decoder as the fixed decoder to train the encoders (or posteriors) for L.y. With smaller ys, we obtain tighter bounds and posteriors closer to the prior. Details are in section C.4
1.2.1. COMPARING Cbb WITH Ch
There are more clumps in samples from the semi-explicit posteriors than from the explicit ones (Fig. [ section ( demonstrating the mixing property of the former. The sam- ples from the implicit posteriors shows that the implicit distribution for y 1.0 (ELBO) are mostly concentrated (Fig. 6a section C) suggesting frequent degeneracy to the delta distribution, in broad agreement to theory (Yin & Zhou 2018). In contrast, those for 7 < 1, we find diverse samples
We examine the joint learning of Bayes posterior and fractional posterior q using the bound Chh,, where the posteriors are in the same semi-implicit family. We compare with using Ch, in both the bounds and the posteriors. The neural network settings are the same as for Cty-
The train and test objectives of Chh do not perform better than those from C.), (and L.y, for y o 1.0; last three rows in
7

---

Variational Learning of Fractional Posteriors
Random samples
(h) Ly 7 1; or ELBO
(c) C.y, 10-6
(d) Prior distribution
Figure/ 1: We train VAEs on the Rashion-MNIST dataset using L, for different yss We obtain mean images from the decoding latent variables that are systematically sampled by coordinate-wiss inverse-CDF (standard normal) transform from a unit square. Fig. b shows the images using the Bayes posterior (learnt with ELBO), and Fig. с shows those using a fractional posterior very close to the prior. The last image the heat map of the corresponding prior densities.
Table 2). even though there are more parameters - those for the Bayes posterior r. In particular, they do not perform better than Cl.o- as would be suggested by Lemma A.4 but we qualify that the decoders and hence the probabilistic models are probably different.
distances (FIDs, Heusel et al. 2017 Seitzer 2020) to the test set: with decreasing y the distances are 83.5, 69.5, 67.8 and 68.8 (smaller is better). This shows that fractional posteriors can train better decoders for generative modelling.
The 3- VAE objective (Higgins et al. 2017) with the appro- priate parameters also gives fractional posteriors. However, this objective can be unstable during optimisation, espe- cially when we seek fractional posteriors very close to the prior. For the same fractional posteriors with y set 10- 10-3 and 10-5, obtain FIDs 77.3, 334.7 and 342.3. Sec- tion C.5 provides the details.
We find [[[]]] to be large, and it increases with smaller 7, as expected (first summand the sixth column in Table When we evaluate the Bayes posteriors rs with ELBO, we find that they are competitive with those obtained by directly optimising C).o (seventh column).
Comparing the Rényi divergences of the fractional posteri- ors learnt with cha to those learnt with Chy (sixth column in Table 2), find those learnt with Chh significantly closer to the prior. This shows that the fractional posteriors from chà are constrained significantly by the Bayes posteriors when learnt jointly (see third paragraph in section 2.3.2).
6. Related Work
Generalised variational inference (Knoblauch et al., 2022 provides an optimisation framework that generalises ELBO. Being generic, one need concretise the individual terms before applying to specific cases. One example is 8- 3-VAE (Higgins et al. (017) for learning disentangled represen- tations in variational autoencoders (VAEs): it weighs the divergence term more heavily. By construction, the 3AV bound is provably not tighter than ELBO, and optimising it gives fractional posterior. The importance weighted EL ELBO is also not tighter than ELBO (Domke & Sheldon 2018 eq. 8). In contrast, the variational Rényi bound (Li & Turner 2016) can be tighter than ELBO, and optimising it gives the Bayes posterior. This paper provides bound that can be better than ELBO, especially for simpler assumed family distributions, and optimising gives a fractional posterior. We show this with the calibration and VAE study.
5.3. Improving VAE Decoder via Fractional Posteriors
The decoder of VAE is learnt with latent samples from the encoder. An encoder from a fractional posterior gives samples closer to the prior than the Bayes posterior. Hence, when we generate images from the VAE decoder using samples from the prior-that is, without using the encoder that need input data expect the decoder learnt with smaller y to provide better images.
We illustrate this with the Fashion-MNIST dataset (Xiao ct al. 2017), training with C., for y taking values 1.0 (for Bayes posterior), 10-1, 1,0-1,00 10-3 and 10 (for fractional poste- rior close to prior). Using latent samples from the prior, the decoder trained with the Bayes posterior provides images that are of lower quality then the decoder trained with the fractional posterior with = 10-5 (Fig. and Fig. 7 in section C.5). To quantify, we generate 10,000 images from each trained decoder and measure their Fréchet inception
In a standard VAE, the decoder is trained with samples from the posteriors encoded using the training data, but these unavailable for pure generative tasks. Current approaches overcome this by learning prior that is accessible during generation, with the objective for matching the prior to
8

---

Variational Learning of Fractional Posteriors
the posteriors Makhzanni et al. 2016 Tomezak & Welling 2018, Tran et al.] )021). The alternative is to train the decoder via distribution close to the prior. While the 3- VAE implies such distribution, its looser bound suggests that the decoder parameters may be learnt suboptimally. Our approach uses fractional posteriors and gives bounds higher than ELBO empirically.
the main alternative Monte Carlo approaches. When the assumed variational family does not include the exact Bayes posterior, some-but not all-also consider this misspeci- fication (Chen et al.] 2018 Knoblauch et al., (022). This paper addresses this by expanding the possibility afforded by the conventional ELBO, so that we may also have ap- proximate fractional posteriors as the optimal solutions. At this point, there is no single recipe to select y (section (.2): we opine that this should be application dependent. For ex- ample in section [.T] the best ys are selected for calibration and not for the tightness of the corresponding bounds.
Variational inference can be seen as intentionna model mis- specification (Chen et al. 2018). Fractional posteriors is one approach to overcome misspecification (Grünwald & van Ommen 2017). Such posteriors can be obtained by sampling with down-weighted likelihood; or one can adjust the scale parameter of the Bayes posterior (Syring & Martin 2019). Alternatively, one can optimise the B-VAE objec- tive (Alquier al., 2016 Higgins et al. 2017). We have provided an alternative variational approach to approximate fractional posteriors, and we have demonstrated calibration using them within regression procedures. More complex calibration procedures (Grünwald & van Ommen 2017
Posterior collapse With small % we might seem to be en- couraging posterior collapse (Wang et al. (02T). However, Fig. 4d for 7 = 0.1 demonstrates that while the fractional posteriors as a whole aggregate towards the prior, the pos- terior for every data point is different. This topic demands more investigation and discussion than possible here.
Syring & Martin 2019) can be explored.
We identify three limitations with Egp. First, the conventional ELBO using the KL divergence is usually more mathematically elegant and convenient. This is be- cause it can convert a log-sum (or log-integral) to sum-log (or integral-log), and sum integral is neccessary for marginalisation. In particular, the variational inference for the mixture model must rely on the ELBO at some point (section B.1.3). Second, the Rényi divergence is finite in less cases than the KL (Gil et al. 2013). It may be necessary to consider this when optimising the parameters of the approx- imate posteriors, though we have not needed to do this for the presented experiments. Third, when using Monte Carlo estimates, more than one sample is required to be effective (section B.Ib. This can be unrealistic for huge datasets.
Limitations
7. Discussions and Limitations
Misspecification If the prior and likelihood are correctly given, and if exact inference is possible, then in principle Bayesian only needs to compute the exact Bayes posterior. This seldom happens in practice (Faden & Rausser 1976). If either the prior or likelihood or both are misspecified, then Bost-Bayesianim efforts, such as generalised Bayes (Knoblauch et al. 2022), robust Bayesian (Miller & Dunson
2019) and PAC-Bayes (Masegosa 2020 Morningstar al. 2022), seek to ameliorate the situation. This paper does not directly address the goals of post-Bayesianism. It has not evaluated when either the likelihood or the prior is misspec- ified. Although those for MNIST and Fashion-MNISS are most probably misspecified, we have not compared to when they are not. We rely on the works of others to address such goals. Nonetheless, we make two connections here.
Power posteriors This paper focuses on estimating an ap- proximate fractional posteriors with the lower bound C.y on the log-evidence. For power posteriors with 7 > 1, we have upper bounds (Lemma A.II. Similarly, there is no fixed cri- terion to select such rys (section 2.2p. Estimating these pos- teriors involves minimising the upper bounds, but machine learning applications will typically require maximising the minimised upper bounds; executing this is probably more involved than what is done in this paper.
First, using fractional posteriors is a proposed solution for misspecification (Grünwald & van Ommen 2017 Bhat- tacharya et al., 2019: Medina et al.) 2022), and our ob- jective does this naturally through a lower bound on the log-evidence. While the objective is not as intuitive as, say, the 3-VAE objective, lower bound like Cy that can be tighter than ELBO can help in selecting or optimising appro- priately parameterised priors in an empirical Bayes manner (Berger& & Berliner 1986). Second, our objective uses the Rényi divergence to quantify the closeness of the prior to the posterior, and this divergence is well-behaved for robustness to prior misspecification (Knoblauch et al., [002 $5.2.1).
8. Conclusions
We have presented a novel one-parameter variational lower bound for the evidence. Maximising the bound within an assumed family of distributions estimates approximate frac- tional posteriors. We have given analytical updates for ap- proximate inference in two intractable models. Empirical results for calibration and VAE show the utility of our ap- proach. For the Fashion-MNIST dataset, VAE decoders learnt with our approach can generate better images.
When exact inference is not possible, we can use approxi- mate inference by way of variational optimisation, which is

---

Variational Learning of Fractional Posteriors
Acknowledgements
Society Series B: Statistical Meshodology, 703))589-607, 2008.
We thank Xuesong Wang and Rafael Oliveira for their inputs, and the reviewers for their constructive comments. This research was conducted while Chai was visiting CSIRO's Data61, and the Machine Learning and Data Science Unit in the Okinawa Institute of Science and Technology (OIST).
Gelman, A. and Meng, X.-L. Simulating normalizing con- stants: from importance sampling to bridge sampling to path sampling. Statistical Science, 13(2):163-185 1998.
Gil, M., Alajaji, F., and Linder, T. Rényi divergence mea- sures for commonly used univariate continuous distribu- tions. Information Sciences, 249: 124 131, 2013.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Grünwald, P. and van Ommen, T. Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it. Bayesian Analysis, 12644)1099 2017. doi: 1.11214/1-BB1088
References
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by time-scale update rule converge to local Nash equilibrium. In Proceedings of the 31 International Conference Neural Informa- tion Processing Systems, pp. 6629-6644 Red Hook, NY, USA, 2017. Curran Associates Inc.
Alquier, P., Ridgway, J., and Chopin, N. On the properties of variational approximations of Gibbs posteriors. Journal of Machine Learning Research, 17(236):1-4) 2016.
Berger, J. and Berliner, L. M. Robust Bayes and empirical Bayes analysis with e-contaminated priors. The Annals of Statistics, ((2))461-486 1986.
Higgins, I., Matthey, L.,, Pal, A., Burgess, C. P., Glorot, X., Botvinick, M. M., Mohamed, S., and Lerchner, A. f-- VAE: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, 2017.
Bhattacharya, A., Pati, D., and Yang, Y. Bayesian fractional posteriors. The Annals of Statistics, 47(1):39-66) 66, 2019.
Bishop, C. M. Puttern Recognition and Machine Learning (Information Science and Statistics). Springer- Verlag, Berlin, Heidelberg. 2006. ISBN 0387310738.
Jin, C., Zhang, Y., Balakrishnan, S., Wainwright, M. J., and Jordan, M. L. Local maxima in the likelihood of Gaus- sian mixture models: structural results and algorithmic consequences. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 4123-4131, Red Hook, NY, USA, 2016. Curran Associates Inc.
Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. Varia- tional inference: A review for statisticians. Journal of the American Statistical Association, 1 12(5188)859-877 2017.
Chen, Y.-C., Wang, Y. S., and Erosheva, E. A. On the use of bootstrap with variational inference: Theory, interpre- tation, and two-sample test example. The Annals of Applied Statistics, 12(2):846-876, 2018
Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. An introduction to variational methods for graphical models. Machine Learning, 37((): 1999.
Kingma, D. P. and Welling, M. Auto-encoding variational Bayes. In International Conference on Learning Repre- sentations, 2014.
Dai, B. and Wipf. D. Diagnosing and enhancing VAE mod- els. In International Conference on Learning Representa- tions, 2019.
Domke, J. and Sheldon, D. R. Importance weighting and variational inference. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Gar- nett, R. (eds.), Advances in Neural Information Process- ing Systems, volume 31. Curran Associates, Inc., 2018.
Knoblauch, J., Jewson, J., and Damoulas, T. An optimization- view on Bayes' rule: Reviewing and generalizing variational inference. Journal of Machine Learning Research. 23(132):1-109, 2022.
Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(1 1):2278 -324, 1998.
Faden, A. M. and Rausser, G. C. Econometric policy model The post-Bayeste approach. Annals of Economic and Social Measurement, 5(3):349-362, 1976.
construction:
Li, Y. and Turner, R. E. Rényi divergence variational in- ference. In Advances in Neural Information Processing Systems, volume 29, pp. 1081-1089, 2016.
Friel, N. and Pettitt, A. N. Marginal likelihood estimation via power posteriors. Journal of the Royal Statistical
10

---

Variational Learning of Fractional Posteriors
Loaiza-Ganem, G. and Cunningham, J. The continuous Bernuuli:: fixing a pervasive error in variational autoen- coders. In Advances in Neural Information Processing Systems, volume 32, 2019.
on Machine Learning, volume 48 of Proceedings of Ma- chine Learning Research, pp. 324-333, New York, New York, USA, 20-22 Jun 2016. PMLR.
Roeder, G., Wu, Y., and Duvenaud, D. Sticking the landing: Simple, lower- gradient estimators for variational inference. In Advances in Neural Information Processing Systems, volume 30, pp. 6925-6934, 2017.
Maas, A. L., Hannum, Y., and Ng. Rectifier non- linearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech, and Language Processing (WDLASL 2013), 2013.
Rogers, L. J. An extension of a certain theorem in inequali- ties. Messenger of Mathematics, New Series, XVII(10): 145-150, 1888.
MacKay, J. C. Information theory, inference and learning algorithms. Cambridge University Press, 2003.
Makhzani, A., Shlens, J., Jaitly, N., and Goodfellow, Adversarial autoencoders. International Conference on Learning Representations (Workshop Track), 2016.
Ruthotto, L. and Haber, E. An introduction to deep genera- tive modeling. GAMM-Mintillngenn 44(2)x2001000008, 2021.
Masegosa, A.
Learning under model misspecification: Applications to variational and ensemble methods. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Process- ing Systems, volume 33, pp. 5479-5491. Curran Asso- ciates, Inc., 2020.
Seitzer, M. pytorch-fidt FID Score for PyTorch, https: github. com/ /wt.///ee//eeettrtt...t// fid August 2020. Version 0.3.0.
Syring, N. and Martin, R. Calibrating general posterior credible regions, Biometrika, 106(2):479-488) 2019.
Medina, M. A., Olea, L. M., Rush, C., and Velez, A. On the robustness to misspecification of a-posterioors and their variational approximations. Journal of Machine Learning Research, 23(147):1-51, 2022.
Titsias, M. K. and Ruiz, F. Unbiased implicit variational inference. In Proceedings of the Twenty-Secone Interna- tional Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 167-176, 2019.
Miller, J. W. and Dunson, D. B. Robust Bayesian infer- ence via coarsening. Journal of the American Statisticul Association, 114(527) 2019.
Tomezak, J. M. and Welling, M. VAE with VampPrior. In Storkey, A. J. and Pérez-Cruz, F. (eds.), International Conference on Artificial Intelligence and Statistics, AIS- TATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, volume 84 of Proceedings of Ma- chine Learning Research, pp. 1214-1223. PMLR, 2018.
Minka, T. P. Expectation propagation for approximate Bayesian inference. Uncertainty in Artificial Intelligence, pp. 362-369, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001.
Morningstar, W. R., Alemi, A., and Dillon, J. V. PAC"M Bayes: Narrowing the empirical risk gap in the misspeci- fied Bayesian regime. In Camps-Valls. G., Ruiz, F. J. R., and Valera, (eds.), Proceedings of The 25th Interna- tional Conference on Artificial Intelligence and Statis- fics, volume 151 of Proceedings of Machine Learning Research, pp. 8270-8298. PMLR, 2022.
Tran, B.-H., Rossi, S., Milios, D., Michiardi, P., Bonilla, V., and Filippone, M. Model selection for Bayesian autoencoders. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 19730-19742. Curran Associates, Inc., 2021.
Uppal, A., Stensbo-Smidt, K., Boomsma, W., and Frellsen, J. Implicit variational inference for high-dimensional posteriors. In Advances in Neural Information Processing Systems, volume 36, pp. 73793-7388 2023.
Opper, M. and Winther, O. Expectation consistent approxi- mate inference. Journal of Machine Learning Research, 6(73):2177-2204, 2005.
Pitas, K. and Arbel, J. The fine print on tempered posteriors. In Proceedings of the 15th Asian Conference on Machine Learning, volume 222 of Proceedings of Machine Learn- ing Research, pp. 1087-1102. PMLR, 2024.
van Erven, T. and Harremos, P. Rényi divergence and Kullback-Leibler divergence. IEEE Transactions on In- formation Theory, 60(7):3797-3820, 2014.
Ranganath, R., Tran, D., and Blei, D. Hierarchical varia- tional models. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference
Wang, B. and Titterington, D. M. Inadequacy of interval estimates corresponding to variational Bayesian approx- imations. In Cowell, R. G. and Ghahramani, Z. (eds.),

---

Variational Learning of Fractional Posteriors
Proceedings of the Tenth International Workshop Arti- ficial Intelligence and Statistics, volume R5 of Proceed- ings of Machine Learning Research, pp. 373-380. PMLR, 16-08 Jan 2005.
Wang, Y., Miller, A. C., and Blei, D. M. Comment: Varia- tional autoencoders empirical Bayes. Statistical Sci- ence, 22::2292333, 2019.
Wang. Y., Blei, D., and Cunningham, Posterior collapse and latent variable non-identifiability. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. Inc., 2021.
Curran Associates,
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-MNIST: novel image dataset for benchmarking machine learning algorithms. CoRR, 0777770 2017.
Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. Yes, but did it work?: Evaluating variational inference. In Proceedings of the 35th International Conference on Ma- chine Learning, pp. 5581-5590 PMLR, 2018
Yin, M. and Zhou, M. Semi-implicit variational inference. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5660-5669, 2018.
Zhang, T. Theoretical analysis of class of randomized regularization methods. In Proceedings of the Twelfth Annual Conference on Computational Learning Theory, pp. 156-163, 1999.
12

---

Variational Learning of Fractional Posteriors
A. Proofs
This section collects the proofs for the main paper. For reference, T the bounds used this paper.
Lemma A.1. Levd is upper bounded the same expression as eq. but with > 1.
Proof. Similar to the proof for > E (0, 1), but we use the reverse Hölder's inequality the manner of IE [ix or
S
[[ IE [Muule" where 3' + 7 = and 8 < 0, with the same expressions for E[], X and Y. Set 7 = 7
3- 0.
Lemma A.2. Levd is upper bounded the same expression as eq. but with < O.
Proof. Similar to the proof for Lemma A.1 but now we use 3 1, 80 that 7 >= =1 >=/==1 11 <0.
Lemma A.3. lim,-1 by LELBO.
Proof. The data term in C, converges to the expectation of the log likelihood under approximate posterior g when we apply the L' L'Hôpital's rule:
log [(==) (D)))
log po
lim 7-1
lim 7-1
1-7
[((z) (D)=))
S ((2)) log log)(D)=)dz
f ((z)dz
= ((>) log pog
Moreover, as 7 - 1, the Rényi divergence converges to the KI. divergence (van Erven & Harremos 2014). Lemma A.4. For a fix approximate Bayes posterior, Cb < LELBO-
Proof. For a fixed r(2), L. is optimal at (z) x r<)))(())>~ Substituting q² (²) into Cb recovers ELBO. Hence, Cb upper bounded by ELBO.
A.1. Variational Derivation of Saddle Point C.y
To obtain the functional derivative / we introduce scalar and function (1z) and consider
(ā ( + hn)
1==0 -1- f [<)))
n( Mmt
dh
hm)
1-7 {x()) h-0
[n(=))
[((=))
1-7
J
[d(=))/"p(2))-1/1d
/ 1-7 (TE))
p(D)2)
p(2)) [ 7(=)de,
where the integrand sans 7²2) the required derivative. Equating to zero give
- J [((3) 7/(1->) (
a(=) p())))
Substituting ((z) into the RHS of 2 gives
- [p(E)p(D(e)dd//
7/(1->)
Z.
[DDDDabpleedt//211
13

---

Variational Learning of Fractional Posteriors
Table 3: List of lower bounds. Some are expressed differently from the main paper to ease comparison among the bounds. The 3-VAE objective is the weighted KL divergence (Knoblauch et al.
2022 $B.3.1).
Description
Notation
Expression
/ p(z)p(D|z)dz
Log-evidence
Levd
logp log
/ ((>) (()) / ((()) log 7(2) p(z)
ELBO (evidence lower bound)
LELBO
dz
/ (2) (o)) - 3 (()) log g(3)
Weighted KL divergence (3-VAE objective)
dz
p(z)
(p(D)=)p(2) p(2)
Variational Rényi bound
CR
1-a log {(2)
de
((2)
1 log / ((3)p(D(2) 7 log ((2)
((2)
1/4-1
Our primary bound
C
de
1-7
(p(z)
/f [((eaa)g(tu)((1a)))>>
with bierarchical fractional posterior
Ch
log
1 1->
7 log // ((zlu)q(w)
[((zlu) p(=)
1/7-1
dz du
I ((2)))g
1-7 / r(z r(2)log log
r(2)
with Bayes posterior
Cb
dz
((2)
Y log 7(2)
7(2))
1/7-1
dz
p(z)
|f ((zlu)q(u) log (D)2) dzdu
with hierarchical fractional
chi
and Bayes posteriors
1
// ((zlu)q(u) log
r(2lu)
dzdu
?(=|u)
1 log /f ((zlw)a(())
((slu)
1/7-1
p(=) dzdu
|| r(zlu)g(u) loggg((uooggl (D)) dzdu
with hierarchical fractional and Bayes posteriors (alternative bound; see section A.4
Cbb-alt
/|/ r(= r(=lw)((u)(((() log r(zlu) dzdudu'
1-
q(= w²)
Y log // ((zlw))(w)
[(2|u) 1/7-1
p(z) dzdu
1
14

---

Variational Learning of Fractional Posteriors
which is self-consistent. The solution is true for any Z, so may choose 2 as the normalising constant. Because of this, and because the optimal solution is already non-negative, it is necessary to introduction Lagrange multipliers for constrained optimisation.
A.2. Gap between Log-evidence and Variational Rényi Bound
1 logp(1) D)- - log / ((()) ((D),)(())))(
1 log
p(D) [q(z) (p(D, dz
1 a
((z) (p(D, 2)/g(z)) dz
a log
D(D)
1
log / [(())
a
A.3. Divergences
Following the definition of Levd and its lower bound Cy, we may define divergence Difac frac from distribution to distribution P1 with respect to an underlying distribution Po:
free |p2llp) der
Levd Lyy
Here, Po participates as the prior, (z) x {())' (o(2) as the target fractional posterior with likelihood {(2). and q as the approximating posterior. By definition, the divergence is non-negative because L., is lower bound, and the divergence zero when P2 = Pi because C, tight.
Let Z be the normalising constant p1. We have {(z) (p(()(()))) By substitution and simplification,
1 Levd log Z + log /
1
?
1
1-7 log / (x
1
1/- 1p((2) Y log / 1/7
C, log Z +
1->
Therefore,
Drree [22ll11] log
pi (((3))/pp(3)))-///
1- log / (3()) 2/o////-ppo))-/1/11111)
1
+ 7 log /
1 1-
and Z is not required.
We may also obtain a divergence without the notion fractional posterior. Continuing from above, let p.(2) def (p))) (=>1/>///// for 1, 2 and where Z,8 normalising constants. Then, by substituting into and simpli- fying expressions in the right side of the above equation, we have
7 1 log (=)))"p((=)
which is the Rényi divergence Dy(22lli.1
Observe that (2) ((=)po(>) is the Bayes posterior. So, in minimising the Rényi divergence, obtain fo2 an approximate Bayes posterior. We recover an approximate fractional posterior p2 by using the definition of ý2, where the prior Po is needed. Therefore, if we are to change the subject of optimisation from the fractional posterior to the Bayes posterior, we recover the variational Rényi bound (Li & Turner 2016).
Throughout this section, the precise definition of C. has allowed normalising constants to cancelled. Also, we can derive f., as lower bound retrospectively by reading this section in reverse.
15

---

Variational Learning of Fractional Posteriors
A.4. Derivations for 2
The data term is because p)) dependent on u. The Rényi divergence term follows from eq. divergence term below.
We address the KL
We use the convexity of KL[pp]g] in the pair (p.g). This provides - f r(=) log(r())/g(=))d== > - (f r(zlu) bog(r(//l))/(((/uddd q())du for the KL divergence term in Lbb. The overall bound requires a dou- ble integral because we have a hierarchical construction, involving random variables u and given v.
An alternative derivation gives chh-al in the last row in Table 3 The function -r log x is concave, so we have - r()) loggllog)))) r(=)dz 2 -f (f r(alu) log (=//))d2)) g(u)du. Similarly, log is concave, so we also have [((= log [(2)d= 2 r(() (J ((') log ((=])((d du'. Introducing u' to the entropy term and u to the negative cross- entropy term and then summing the two gives an alternative KL divergence term in Lbh. The triple integral in the KL term comes from the independently mixing of rzzlu) and ((2lu) with the same distribution q(u).
A.5. Non-degenyracy of the Implicit Distributions within the Semi-implicit Distributions
This section shows the existence of non-degenerate implicit distributions when optimising for L.), Clyy and Ch A common theme is that the Rényi divergence is expressed as a log-integral rather than an integral-log, so the optimsation for the implicit distribution cannot be factored out.
1. FOR FRACTIONAL POSTERIORS USING C.)
For this section, let f(u) def [ ([)) and g(u) én
(22//d,, so that Ch(())
(log f (()))(u)du)/(( 7) - (log J (())g(())dd (1))(())))(1111111 - 7), where is just for the distribution of u. We introduce scalar and function nu)) The functional derivative ach, log is the integrand sans n7(1) of the expression
d.C, q + hn) dh
=
(i-JJFiaaaa 1 1-7 g(u')a(z')dd
f(0)
g(u)
g(u)m(u)du
h=
Together with the normalisation constraint which introduces Lagrange multiplier A. require
(1 1 [1- f(u')q(u')d 1 1-7 g(u')q(u')dd) + > g(uu) =
f(u)
g(u)
Integrating with respect to yield 1 A = 0, have
(1- S(u)
7 1->J J gll g(u')g -1) (())= =0
g(u)
This means that quu) will collapse to zero if the left term not zero. Though it is not necessary that q(u) degenerates to delta distribution, delta distribution satisfies the above constraint readily.
For further illustration, consider quu) supported at only two locations ū1 and u2 Using di fo and gi denote evaluations of f and g at these locations, we have
- f292 f192 7f291
(1 1111 f291 + 7yf192 Q2
(1 - (f1 [2) 92)
(1-7)(/ (2) - 92)
which is satisfiable with different values of fis and g,s.
A.5.2. FOR BAYES AND FRACTIONAL. POSTERIORS USING Cbb For this section, let
f(u) def f r(zlu) log (D|2(D
g(u) del
f
huu) te /
d(u) del / [ (((ll)) log g(2lu)d2,
16

---

Variational Learning of Fractional Posteriors
so that
(g) = f())g(u)du - / h())g(u)du + ( d(w)g(u)du- log / g())g(u)du,
1 T-
1 T- /
where y is just for the distribution of u. Taking derivative with respect to log g(u) and imposing normalisation constraint with the Lagrange multiplier A, we require
7 g(u)
2) a(uu) O.
f(u)
I-7 n(u) 1-7 d(u)
1 - J (ll')a(l' -
(4)
Integrating with respect to u fix
1 == / (w)g(u)d -1- / Mu))(())du
- / d(w)g(u)du
delta distribution satisfies eq. (4) readily. However, other solutions are also possible in general. As an example, consider q(u) supported at only two locations in and ug, and let Af der f(u)) - f(2)). Ah der h(u1)- h(22). Ad del d(ui) - d(u2), and Ag g(u)) g( (u In these settings, and using (un) + (u2)) eq. may be written as
Ah Since ((11) # 0 and q(u2) + O, the first term must be zero. This can be expressed as
(as As
1
1-7 Ad 7
Ag
((u1)q(u2) = 0
1-
u (4))))(()))))))))))) + ( + ((22)
^gg
((v1)q(u)) ((u2)g(u)) (1-7)Af Ah Ad
Using ((u2) 1 q(11), the explicit expression for g(u)) is
((u1) (1- Ah Ad g(u)).
A.5.3. FOR BAYES AND FRACTIONAL POSTERIORS USING C. -hb-ALT
We define f. and h as for ch in section A.5.2 but we now have d(u, u') der f (z(2u) log ((2//(()d)) so that
bb -at(a) / f(u)g(u)du
1 1- /
M())g(u)du+ 1- 1 1-,, /f d(u, ul (d)((())(())ddaa)))
log / g(u)g(w)du
where 9 is just for the distribution of u. Taking derivative with respect to log q(u) and imposing normalisation constraint with the Lagrange multiplier X, we require
1 1-7 h(u) + (
((w)) ((w))11-
1- -,,/ / (dlu, + d(u', ()), - 1- g( -
7 g(u)
-x) >)a() -.
(5)
Integrating with respect to u fix
x= f flu)g()ddu - / h(w)glu)du +1=,
2_ /f d(u u' (d)g(())(()))dduu
Y 1-7
A delta distribution satisfies eq. S readily. However, other solutions are also possible in general. As an example, consider q(u) supported at only two locations u, and U2 and let Af de f(u,) f(2)). Ah बt h(u,) - h(u2), Ad def [d((u,, u') + d(u', (1))))g (1)))( [ [d((u,, u') + d(u', uz))q((')ddd) and Ag de g(uz) - g(uz). We proceed as the example for section A.5.2 under these definitions.
17

---

Variational Learning of Fractional Posteriors
A.6. Gradients and Expectations for the Multinomial Example
The gradients with respect the parameters are
log q(=) Ze
Me
i log q(z)
(ze MC)
+
pp
0o?
20
20
and the required expectations are Eg. [2c]] = m. + P..
Egg [c)
= Sc + m + 5 2- 2mc)
Eg [=o] = ue + oe no/ No/(nn + 1)
E
[
[cc
0 + (Me + ~ no/(n + 1))
where
exp(m + 2²/22)
Pe
e'= exp(me + s²: (2)
These can be used for gradient ascend to learn the parameters of q.
A.7. Derivation for the Mixture Model The marginal likelihood evidence is
exp Levd = f p(u) E(())
p(c) /plu)
p(ode u)du.
i=1
Applying eq.
① on Leva focusing on the posterior for p(u) gives
Levd 2 log / q(u)
Σ p(c) II plalee u)
du - 1-7
2 log / q(u))//
1-10/ 1/7
(6)
Applying ELBO on the logarithm of the term within the brackets above and then exponentiating the result get us to
Levd 2 1-9 log f ((u)
K II -1=1
p(a))
a(c,) 1-7
du
1 7
log
-//7du.
(css)
The argument of the logarithm in the first summand is the first displayed expression in section 3.1.3 In the first summand, we bring the products out of the logarithm:
log S ((()) (plastue)) p(c:) ((())
(1-2)((e)
7 log p(u) 1-
Levd
du
In the first summand, the density ratios p(ci)/q(c.) are independent u and taken out to give the KL divergence. So the bound written
log f q(()) p(z: ((zilua,)) (1-)), 1=1 c==1
p(c)) 1- a(ci) log (G))) 7 - log
1-7 i=l c=
Finally, use the mean-field approximation for q(u) and rewriting the indexing for the o; as indexing for k to give
K
/ (a(u))((zsuu ( Dele ---e
K
k)) log q(a = k)
log
g(c
1
p(c, k))
Im 1m
log / p(um)
Swapping the order of the summations and using variational parameter (Pu for q(c, = k) gives the second displayed expression in section 3.1.3
18

---

Variational Learning of Fractional Posteriors
A.7.1. FRACTIONAL POSTERIORS FOR CLUSTER ASSIGNMENTS
For approximate inference to be tractable for the mixture model, it seems that we ultimately cannot avoid using ELBO for c. Nonetheless, this does not preclude from also having fractional posterior for Instead of applying ELBO on eq. (6)
apply the Sy in section 2.3.2 to the same term:
r(c) g(c)
log Σ p(c) Il plasee, u) 2
Σ r()) log II plad u) 1-
Er(c)
((c) log
log E((c) p(e) Dt
where r(c) is the approximate Bayes posterior and q(c) the approximate fractional posterior. Following through derivations similar to before with mean hean-field approximations, we obtain
log / (((uz) (1x) (1 due
r(c: k-) log r(c: = k)) g(c; = h:)
Levd
(=1 k=1
K
qcc = 1/ p(a==()))) = 1k) -1)
7
Σ log / ((ux))/"p(uu 1/1 p-ux)=-1/ddx
log
As reasoned in section 2.3.2 the optimal fractional posteriors ((;))s is interpolate between the r(c;)s and the p(c,)s g(q) x r(c)) p(c))((')))) 1->"_ At this setting, we recover eq. 7 with a change notation for the approximate Bayes posterior. This is expected since there no constraint on fractional posteriors ((c)ss other than normalisation.
B. Monte Carlo Estimates
Suppose we have N, samples 2;8 from distribution ((2) =[22//Z for known normalising constant Z. Then
C, ~
log
E((D)=)'
7 log 1-7 N₂ [(((e)/(())))/
Ng
Ideally, one should draw separate samples for estimating Z, and Za, but in practice, one trades this off with computational efficiency. We currently employ this approach.
If we only have the unnormalised density, then approximating the lower bound requires the normalising factor Z:
L ~ log 7 log N EM(DE=) 1 1- log 1 N
Σ
((z,)
1/7-1
1
p(2;)
The normalising constant cannot be avoided, and one may estimate it with, for example, importance sampling from a distribution for which the normalising constant is known (Gelman & Meng 1998).
An alternative is to introduce a mixing distribution and use the Cy bound:
( (u:) 1/7-1 p(zis)
1 log N₂ ΣΣr(Dlag) ((D)za)) 7 log ΣΣ :
ch ~
N
where there are now No samples from u, q(u) followed by No/Nz samples 25y q(2/u:). In practice, there can be different number of samples for each u, but we simplify the notation here. In this setting, q(z) need not be known explicitly, but
required to be able to draw the (u,, 2 samples and know the conditional ((2|u) exactly.
For Cbh, we similarly have N samples from u, q(u) and N₂/N¹ samples zij <(2lus), but we now also have No/Ng samples z/4 ~ r(z(u:):
bb ~
1 N₂
ΣΣ log (DD
ΣV
log r(slelus)
1 log N p(2())
ΣΣ
(1(=))lue) 1/7-1
1 N
q( (2))
1
19

---

Variational Learning of Fractional Posteriors
For the alternative L -alt bound (last row in Table 30, we suggest the following mechanism. We have the same samples, but further, for each uy sample, we associate with it subset U; from the set {ui..... uNg J. Then we estimate this alternate bound with
log (ogp(D)<(k)
Σ
log (( l2s)
N
N₂
log (2)a))))) 1-7 Ng
7 log ΣΣ
((siglue)
1/y-1 p(2gg)
N
k ByEU,
If q(u) is continuous distribution such that there is zero probability of having two samples with the same value, then it is important that Ug excludes uç Similarly, if [U,] is small, then systematic instead of random selection of the subsamples should be better. The method of unbiased implicit variational inference (Titsias & Ruiz 2019) similarly requires nested summation of independent samples from q(u). but the proposal there is to sample U/ separately. The trade-off between computational cost and more faithful Monte Carlo estimates depends on, for example, the cost of sampling from q(u). Yet another approach the Gaussian approximation based on linearisation (Uppal et al. 2023).
B.1. On Single Samples
If N2 = 1, as commonly done for local latent variable models (see section 5.2). then the above estimates for Cy and Ch revert to ELBO because the exponents within the logarithms cancel the multiplicative factors on the logarithms. Therefore, these bounds require N₂ >
be effective.
For Ch. the 1/(1 7) factor remains for the KL divergence from to q. The implications may be subject to future work.
B.2. Considerations for Learning
If we are using the above estimates within an automatic differentiation procedure for learning the parameters of the distributions, it is critical that any sampling distributions (that is, q(2), or (2(uu) and q(u) be driven from standard distributions with fixed parameters so that we may apply the Law of the Unconscious Statistician, also known as the reparameterisation trick (Kingma & Welling. 2014).
In addition, for the semi-implicit constructions, we find learning to be effective when N./N¹ > 1, that is more than one sample of z for each sample of u.
For ELBO with semi-implicit posterions, although delta distribution is known be optimal (Yin Zhou 2018 variances in sampling may lead mixture of delta distributions located the optimal and the near-optimal locations. This is further exacerbated by the variance in stochastic gradient methods. The parameterisation of the implicit posteriors may also give region of lower probabilities "bridging" these locations. Fig. in section C.4 provides an illustration based on VAE on the MNIST data set.
C. Experiments
This section collects additional details and results for the experiments.
C.1. Calibration
For the calibration experiment (section 5.1 we initialise the estimated posterior with means -1 and 1, and variances n/ K. The prior standard deviations of the component means The experiment sensitive the precise settings of these quantities.
C.I.I. STRATEGY USING INVERSE SQUARED LENGTII
For a prior p(z) and a corresponding exact Bayes posterior D() the exact fractional posterior is given by the interpolation ">===D)) For Gaussian distributions, the precision this fractional posterior is (1 - 7)/ + 7/af. where of and of are the variances of the prior and the Bayes posterior. In the context of calibration, precision is inversely proportional to the square of the interval length.
20

---

Variational Learning of Fractional Posteriors
Precision
7
12
0
0.2
0.4
0.6
0.8
logio mean-squared differences in precisions
(b) An example of precisions for one data set. The straight line (with squares) is the interpolated precision, while the curved line (with circles) obtained from the posterior learnt with Ly-
(a) The histogram of the logarithm of the mean-squared differences.
Figure 2: Differences between the precisions of the fractional posteriors for the first component of the mixture model.
This suggests us to perform linear regression against 2 2 to target the required interval length, using the results from the approximate posteriors. For the experimental setup in section 5.1 this strategy, called Rę-2, gives As (resp. εs) with 0.9320 (resp. 0.2735) and 0.9220 (resp. 0.2888) for j4 and μ22 While Rg-2 does gives interval lengths closest to the ideal of 0.2772, the coverages lower. The for this 0.974 with evidence bound 834.99
That strategy Ry-2 is not necessary best in all respects reminds us that are dealing with mixture model and not Gaussian model. The difference is empirically elaborated in section
C.1.2. DIFFERENT EXPERIMENTAL SETTINGS
We find the results and conclusions to be similar when the number of observations n per data set is different, albeit with different interval lengths. Table 4a gives the results for same experiment with n = 30 (versus the 400 section [. This is similar for closer component means 1/2 and 1/2, though now with different coverages (Table (Table44b). and only the more comprehensive R, that gives 7 0.38 is effective for calibration in this more difficult setting.
The picture is more complex with K = 4 components, centred at 22 -1/2, 1/2 and 2 the coverages for components at 11/2 are noticeably smaller than for components at +2, more so for larger (Table 4c). Again, only R,, that gives = 0.26 is close to effective for calibration.
This extended study highlights the need to consider fractional posteriors, especially for difficult problems.
C.2. Differences with Fractional Posteriors obtained by Interpolation
We seek to illustrate that fractional posterior obtained by our bounds is in generally different from that obtained by interpolating between the prior and the Bayes posterior (see section C.I.IJ. We follow section ann section C.T but with K 2, n 20 and component centres 1/2 and 1/2 in order to make the differences more noticeable.
With compute the approximate Bayes posterior learnt by optimising ELBO (C1.0), interpolate with prior to obtain fractional posteriors pl-3y77 This is compared with the approximate fractional posterior 4, obtained from optimising L.y. It is sufficient to examine the precisions for the first component to show the differences.
We compute mean-squared differences between the precisions the two set of fractional posteriors at E (0.1, 0.2, 0.3,0.3000 0.4, 0.5, 0.6, 0.7, 0.9). Figure 2a plots the histogram, across the 5000 data sets, of the base-10 log- arithm of the mean- -quared differences; and Fig. 2b plots the precisions for particular data set. The plots demonstrate that the interpolated posteriors and the learnt posteriors are different, in general.
'The posterior means are initialised +1 and +1/4.
21

---

Variational Learning of Fractional Posteriors
Table 4: Calibration study of the Gaussian mixture model at 0.05 significance level, for different settings.
(a) For = 30. The values for Re. Ri-2 and Ha are 0.789.
(b) For = -1/2. μr 1/2. The values for He. Re-z and Rs 0.785, 0.998 and 0.379.
0.965 and 0.849.
H22
μ12
bound
E
i
bound
Co.1
1.0000
3.0111
1.0000
3.1774
69.2
Co.1
0.9996
0.8740
0.9986
0.8743
625.8
Co.2
0.9998
2.1644
0.9998
2.2901
69.1
Co.2
0.9946
0.6188
0.9898
0.6190
624.9
Co.3
0.9990
1.7770
0.9978
1.8822
69.4
Co.3
0.9828
0.5055
0.9682
0.5057
624.9
Co.4
0.9968
1.5433
0.9936
1.6355
69.7
Co.4
0.9642
0.4379
0.9434
0.4380
625.0
Co.5
0.9922
1.3826
0.9876
1.4658
69.9
Co.s
0.9382
0.3917
0.9176
0.3918
625.2
C0.6
0.9844
1.2636
0.9780
1.3399
-00.2
Lo.6
0.9082
0.3576
0.8884
0.3577
625.3
Co.7
0.9746
1.1708
0.9672
1.2418
-70.4
Ca.t
0.8816
0.3311
0.8598
0.3312
625.4
fo.8
0.9620
1.0958
0.9524
1.1624
-70.5
Co.s
0.8558
0.3097
0.8378
0.3098
625.6
Co.9
0.9498
1.0336
0.9382
1.0966
-70.7
Co.9
0.8320
0.2920
0.8212
0.2921
625.7
L1.
0.9328
0.9810
0.9234
1.0408
-70.8
L1.0
0.8126
0.2771
0.7972
0.2771
625.8
Co.1 1.0000
3.0111
1.0000
3.1774
-74.7
Co.1
0.9990
0.8740
0.9978
0.8743
630.2
Co.5
0.9920
1.3826
0.9874
1.4658
-70.4
Co.5
0.9372
0.3917
0.9176
0.3918
625.4
Co.9
0.9494
1.0336
0.9378
1.0966
-70.7
Co.9
0.8314
0.2920
0.8212
0.2921
625.7
Rg
0.9628
1.1035
0.9538
1.1706
-70.5
Rg
0.8596
0.3127
0.8414
0.3128
625.6
Ry-2
0.9398
0.9983
0.9296
1.0595
-70.8
Rş-2
0.8128
0.2773
0.7974
0.2774
625.8
Rx 0.9570
1.0642
0.9464
1.1290
-70.6
R₂
0.9686
0.4501
0.9486
0.4503
625.0
(c) For K 4, with component means -2, 1122, The values for Re, Re-z and R, are 0.785, 0.995 and 0.263.
11
P22
113
P4
K
E
K
K
bound
Co.1
0.9990
1.2259
0.9876
1.2369
0.9696
1.2377
0.9994
1.2309
-820.7
Co.2
0.9942
0.8712
0.9046
0.8757
0.8904
0.8757
0.9956
0.8740
-817.3
Lo.3
0.9862
0.7126
0.7980
0.7152
0.8310
0.7151
0.9814
0.7147
-816.7
Co.4
0.9710
0.6176
0.7188
0.6195
0.7864
0.6194
0.9660
0.6194
-166.7
Lo.5
0.9514
0.5527
0.6558
0.5542
0.7410
0.5540
0.9434
0.5542
816.8
Co.6
0.9328
0.5047
0.6076
0.5059
0.7018
0.5057
0.9234
0.5061
887.00
Lo.7
0.9120
0.4674
0.5746
0.4684
0.6656
0.4682
0.9020
0.4687
-81.22
Co.8
0.8878
0.4373
0.5436
0.4382
0.6384
0.4380
0.8846
0.4385
-87.44
Co.9
0.8654
0.4123
0.5126
0.4131
0.6126
0.4130
0.8660
0.4134
817.6
C1.0
0.8422
0.3912
0.4864
0.3919
0.5970
0.3918
0.8484
0.3923
817.8
Co.1
0.9988
1.2259
0.9830
1.2369
0.9684
1.2377
0.9996
1.2309
826.4
Co.5
0.9504
0.5527
0.6530
0.5542
0.7412
0.5540
0.9460
0.5542
817.0
Co.9
0.8650
0.4123
0.5106
0.4131
0.6174
0.4130
0.8664
0.4134
-87.6
Re
0.8922
0.4414
0.5478
0.4423
0.6412
0.4421
0.8860
0.4426
-17.4
R
0.8432
0.3922
0.4884
0.3929
0.5926
0.3927
0.8500
0.3932
-1778
R.,
0.9904
0.7610
0.8404
0.7642
0.8506
0.7641
0.9870
0.7633
-166.8
22

---

Variational Learning of Fractional Posteriors
Table Importance sampling to estimate the log-evidence Leva of Gaussian mixture models using approximate posteriors. We the four experimental settings from section 5.1 and section C.1.2 For each setting, and for ten values of (including ? 1.0 for ELBO), we have the bound Ey from the variational optimisation, Levd, and the coefficient of variation CV of
the weights used to compute Levd.
400, 14
+2
30, 14
+
400, 14 +1/2
n 400, components
4
Levd
CV
Ey
Levd
CV
Levd
CV
4
Levd
CV
0.1
-13.00
806.9
2.2
65.77
-60.2
2.0
602.1
593.1
84.0
805.2
-786.3
87.6
0.2
812.9
806.9
1.5
64.6
60.2
1.3
601.0
593.4
23.7
801.8
-786.3
83.0
0.3
-83.22
806.9
1.1
-4.55
-60.2
1.0
600.9
593.4
19.8
801.1
-786.2
62.8
0.4 813.5
806.9
1.0
64..5
-60.2
0.8
601.0
-99.4
31.5
801.1
-786.4
36.8
0.5 -813.7
806.9
0.9
-64.6
-60.2
0.6
601.2
-93.4
24.3
801.2
786.66
19.4
0.6 - 881.9
806.9
0.9
66..7
-60.2
0.5
601.3
592.8
163.0
801.4
786.2
83.9
0.7 814.1
806.9
1.0
-6.88
60.2
0.4
601.4
592.8
130.9
801.6
786.6
26.6
0.8 814.3
806.9
0.8
-64.9
60.2
0.3
601.6
593.5
12.0
801.8
786.8
16.9
0.9
814.4
806.9
65.0
60.2
0.3
601.7
593.2
96.9
802.0
-786.8
17.0
1.0 8145
806.9
1.2
65.1
60.2
0.3
601.8
593.6
15.5
802.2
-786.8
23.0
C.3. The Tightness of Bounds for the Gaussian Mixture Models
We use importance sampling estimate the evidence of the Gaussian mixture models (GMMs) using approximate posteriors. For the log-evidence of -827 24 reported in section 5.1 we draw N₂ = 1, 000 samples from the approximate posteriors ((u, e) to estimate the log-evidence Levd def log Σ i= /Ng, where weights wy def p(u,, c, )(p(((((,, c4//((1). c4). The average log-evidence of - -28. 24 is obtained over the 5, 000 replications. The same value, up to five significant figures, is obtain with every one of the ten fractional or Bayes posteriors.
We perform more experiments with four settings: the one in section 5.1 and the three in section C.1.2, We use one of the 5,000 replications for this. For each setting, we draw N₂ 100, 000 importance samples from the approximate posteriors q(u, c) to estimate Levd- We also compute the coefficient variation (CV) of the weights to give an indication of the quality of the estimates. By central limit theorem, this is VNg times the the coefficient of variation of the evidence exp Lova
The results demonstrate that there is no guarantee that lower y will give better bounds (columns C. in Table 5 as we have reasoned in section 2.2 though the best bounds are typically not with ELBO ( = 1.0).
There is also substantial gap between Leva and the best bounds (comparing columns L, and Levá in Table 5 We opine that this is because (1) the approximate posterior quu, c) is factorised into (u))((c) where only q(u) is the approximate fractional posterior while g(c) still the approximate Bayes posterior; and (2) in our GMM settings c has a larger role because there are 30 data points while for u we have most one-dimensional components.
For the quality of importance sampling, the CVs are significantly smaller when the true means more separated (first two columns of CV versus the last two columns of CV in Table This is expected because better separation suggests multi-modality in the true Bayes posterior is less pronounced. In addition, for the purpose of better importance sampling in attribute this again q(c) being approximate Bayes
GMMs to obtain Levd, using C, has only slight advantage posterior and that is a larger role.
If the end goal is to perform better importance sampling using fractional posteriors, we may use the fractional posteriors for the cluster assignments, which are interpolations between the approximate Bayes posteriors and the priors (section A.7.I).
C.4. Variational Autoencoder
We make three experimental changes from Ruthotto & Haber (2021) One, we use the continuous Bernoulli distribution (Loaiza-Ganie Cunningham] 2019) as the likelihood function instead of the cross-entropy loss function. Two, we draw 100 samples from the approximate posterior per datum during training instead of the one sample that they use, because otherwise there is no difference between the ELBO and some of our bounds (see section B.I Three, we train for 500
23

---

Variational Learning of Fractional Posteriors
Table 6: Average log-evidences (higher better) over data samples, and its breakdown for VAE on MNIST data sets, for 2 and Sy -hah We give the mean and three standard deviations of these averages over ten experimental runs. For Monte Carlo averages, 1,024 samples are used (32 x 32 for semi- immiiitplicit posteriors). We show the addition of the KL divergence and Rényi's divergence for Test using Objective; and for Test using ELBO we evaluate the Bayes posterior r.
Test using Objective
Test using ELBO
Objective
7
Train (Total)
Total
data
div
Total
data
div
cbb
0.9
1636.2 22
1608.8293.4
1613.4 4123.0
4.020.2 + 0.620.3
1607.7
1613.4223.0
5.721.0
0.5
(635.2+10.
1608.0 0225.4
1612.7
4.3 + 0.4
1607.4=2s.a
1612.7
5.3
0.1
1635.5 5212.
1607.5 16.
1612.4416.1
4.62o.1 + 0.320.2
1607.3216.a
1612.4_16.1
5.
Cty -hb-
0.9
1639.4 .40.4
1609.1+21.1
1613.6+21.0
4.0+o.1 + 0.6+0.2
1608.0+21.
1613.6+21.
5.6+0.s
0.5
1639.4+10.6
1608, 2
1612.9
4.3+ + 0.4+0.2
1607.6 66.7.6624.5
1612.9#244
5.3+0.5
0.1
1639.2 10.44
1608.8
1613.7 7
4.7 + 0.3 1608.6=2ed
1613. =223.
5.1
epochs instead of the 50 epochs that they have used, so that we obtain results closer to convergence to reduce doubts on the comparisons.
For the semi-implicit posterior family, we use three layers neural network for the implicit distribution, similar to that used by Yin & Zhou (0188) with the following changes: we reduce the noise dimensions to 15, 10 and 5, and the hidden dimensions to 28, 14 and 2 so that we can visualise the samples from the implicit distribution; we use normal with mean 0.5 and standard deviation 1 instead of Bernoulli for the noise distribution to better match the gray-scale images that we use; we use leaky ReLU activations (Maas et al. 2013) for the hidden units to reduce degeneracy due to the learning dynamics, and we use sigmoid for the output unit so that we can visualise the distribution within a unit square.
For training the explicit posteriors with Ly, we use 100 samples per datum. For the semi-implicit posteriors with L.., we draw 10 samples from the implicit distribution per datum, then 10 latent variables from the explicit distribution per implicit sample. For Cbh involving semi-ine amii-impiicit fractional and Bayes posteriors, we additionally use 5 of the 10 implicit samples to estimate the cross entropy term. Batch size of 64 is used for training with the Adam optimizer, where the learning rate and weight decay are set to 10-3 3 and 10-5.
We use ten experimental runs to obtain Table The neural networks in each run are iniitlised with a different seed for the pseudo-random number generator. The results in the table are the mean and three standard deviations of these ten runs. The overall relatively small variations among the ten runs suggests the stability of the results.
We also perform the same experiments for the Cy -h bound (last row of Table 3), and the results are compared with those of Cbl in Table 6 We find their results to be very similar.
Bounds We take single run of LELBO for the explicit posterior and uses its decoder as the fixed decoder train the encoders (or posteriors) for L. for y E {0.1, 0.5, 0.9, 1.0]. The encoder neural networks are initialised randomly and optimised for 50 epochs with the same number of Monte Carlo as before during training. Since the decoder and hence likelihood models are now fixed together with the prior, we may compare the train and test objectives as bounds on a single fixed log-evidence. With smaller 7, we obtain tighter evidence bounds and posteriors closer to the prior (second, third and eighth columns of Table 77. The results for the reference LELBO (that is, first row in the table with (reference) L.0) are that for joint optimisation with the decoder for 500 epochs, and are used for Table Comparing the figures for the two instances of 7- 1.0, we that the dynamics of jointly training decoder and encoder can give better objectives.
Image generation We perform further investigation by plotting figures from a single run each. Figure 3 gives the images from the learnt decoders for the VAE experiments using latent variables sampled from the prior in a regular manner. We do not see any significant quality to the decoded images from the different values of ry tried. Differences are more visible for the harder not sharper: (1) we use simple neural networks for the decoder and encoders (Ruthotto & Haber 2021) with only 88,837 parameters for the case of Cy; and (2) the images are mean images from the decoded latent variables, that is, the parameters of the continuous Bernoulli distributions, and not samples.
data set (section 5.3). There are two reasons why the images in the figure (except for Fig. 3i) are
24

---

Variational Learning of Fractional Posteriors
7
7
7
7
9
9
9
9
2 2 1
1
1
1 7
/ /
3
3
1
1
1
/
2
3
3
3
1
1
1
8
4
9
9
9
9
9
2
7
7
7
4 4
3
8
3
7
7
7
9
4 4
2
3
8
8
8
8
9
8
8
8
9
9
9
9
9
4
2
7
7
7
9
4
8
3
8
8
7
7
9
7
4
4
2
3
8
8
7
9
7
8
8
8
8
3
8
4
2
2
8
9
9
7
9
9
8
3
8
8
9
9
9
7
9
5
2
3
8
8
7
7
7
5
5
5
3
3
5
5
3
8
8
9
9
7
8
3
8
4
7
7
9
5
2
2
5
9
7
7
7
5
5
5
3
2
3
8
3
9
7
4
9
5
3
3
6
9
7
4
6
2
3
5
8
9
9
9
9
9
5
5
6
2
2
2
3
8
3
9
1
4
8
5
6
8
3
6
6
4
4
4
0
0
6
4
4
4
4
4
4
4
6
6
6
6
6
6
6
6
3
8
6
4
9
4
5
5
6
5
2
6
6
6
4
9
0
0
0
6
6
6
6
5
5
4
4
6
6
6
6
6
6
8
6
6
6
0
0
0
0
6
2
2
2.
6
6
O
O
O
0
6
6 C,
6
5
5
5
(a) C. 7
1.0
(b) C. 7
0.9
(c)
7,
0.5
(d)
0.1
1
1
5
2
3
1
9
9
9
4
4
7
8
8
6
6
0
2
8
8
8
4
7 7
/
7
7
9
9
4
7
7
7
9
9
7
8
8
8
3
6
6
6
33
8
4
4
9
7
8
8
8
3
7
7
7
9
7 58
5
7
7
9
9
4
7
8
80810
8
2
6
6
3
8
4
9
7
772
3 38
888
8
8
7.
9
9
4
4
4
7
4
8
8
3
3
2
2
6
3
3
8
9
9
7
7
5 5
7
9
5 8
4
9
9
4
9
9
9
5
8
8
5
3
2
2 2 3
6
5
3
8
9
9
9 4
2
8
5 5
5
9
4 4
8
8
8
9
9
9
9
9
9
8
8
5 3
3
3 5
6
5
3
9
4
4
5
2
2
8
5
5
6
6
4
3 335
3
3
6
6
6 6
9
4
8
8
6
4
4
5
5
3
3
3
6
6
6
33
5
3
2
2
6
6
9
9
6
5
5
6
6
6
5
6
2
3
3
5
0
0
6
3
3
2
2
2
2
0
0
(e)
7
1.0
(f), 7
0.9
(g)
L"
7 0.5
(h) L",
7
0.1
7
2
1
0
4
4
9 5
7
7
7
9
4
4
5
8
9
7 7
7
9
4
4
7
7 8 8 3
9
9
4
4
4
9
0
6
9
7 4
9
8
8
8
5
5
5
5
5
3
4
4
4
8
9
8
9
/ 8
8 8 8
8
5
5
8
8
4
9
5
0 7
8
8
8
8
8
5
5 3
8
8
8
8
8
8
8 8
8
5
5
5
8
4
0
1
3
3
47
8
3
3
5 5
8
8
8 8
8
3
3
3
3
5
8
8
5
5
7
2
1
7
2
5
3
3
5
5 5
5
5
3
3
3
3
3
5
5
5
3
3
3
5
5
5
\
2
4
4
6355
5
3
3
3
3
3
3
2
3
3
3
3
3
3
3
5
5
3
3
3
3 2
3
6
0
4
1
9
5
7 /
8 9
5
5
3
3
2
2
2
2
2
5
3
3
3
3
2
2
2
8
5 5
5
3
2
2
2
2 0
3
7
2
1
4
9
6
6
3
2
2
2
2
2
2
5
5
3
2
2
2
2
2
2
5
5
3
2
2
2
6
5
9
0
6
9
0
1
9
6
6
6
2
2
6
6
6
6
6
6
6
2
2 2
6
6
6
6
6
(i) Test samples
(i)
ch
Y = 0.9
(k)
bb
7 = 0.5
())
? hb
bb
0.1
7
7
9
4
4 4
5
5'
8
1
7
7
7
9
4
4
/
7
7
9
9
9
4
8
8
8
8
5
5
5
5
8
4
4
8
8
8
8
8
8
5
4
8
8
9
9
9
9
9
9
8
8
8
8
8
5
5
5
3
4
8
8
8
8
8
8
3
5
4
8
8
8
8
8
8
3
9
5
8
3
8
8
8
5
5
5
8
8
8
8
3
3
3
3
5
4
8
8
3
S
3
3
3
3
5
5
5
3
3
3
3
3
3
5
5
8
3
3
3
3
5
S
5
3
3
3
3
3
3
3 3
5
3
3
3
3
3
2
5
3
3
3
3
3
3
2
5
3
3
3
2
2
3
3
2
5
5
3
3
2
2
2
2
3
5 5
3
3
3
2
2
2
5
6
2
2
2
2
2
2.
5
5
6
2
2
2
2
2
5
6
6
2
2
2
2
6
6
6
6
2
6
6
6
6
6
6
6
6
6
22
6
6
6
5 6
6
6
6
4
O
0
(m) Lh
(l)
alt, 7 = 0.9
(n) cht, 7 = 0.5
(o) Chhh 7 = 0.1
Figure 3: Mean images from decoded latent variables obtained by coordinate-wist inverse-CDF (standard normal) transform from unit square. For the last row, the first image are samples from the test set. The quality of the images are visually similar across ys and posterior families, and they are all not as sharp as the real test images. The VAE is trained on the MNIST dataset.
25

---

Variational Learning of Fractional Posteriors
Table 7: Log-evidences (higher better) over data samples for single run, and its breakdown for VAE on MNIST data sets, for L, where the decoders are fixed to be the same as that optimised for ELBO (first row in table). For Monte Carlo averages, 1,024 samples are used.
Test using Objective
Test using ELBO
Y
Train (Total)
Total
data
div
Total
data
div
(reference)
1.0
1616.814
1591.401
1596.695
5.294
1591.407
1596.700
5.293
1.0
1580.006
1558.188
1562.939
4.751
1558.189
1562.940
4.751
0,9
1625.566
1620.629
1624.170
3.541
1551.088
1554.442
3.354
0.5
1638.652
1636.510
1639.442
2.933
1451.295
1453.317
2.022
0.1
1641.655
1639.472
1642.944
3.472
1427.995
1429.871
1.876
(0) 1.0
(b) 7 0.9
(c) 7 0.5
(d) 7 0.1
Figure 4: Means of the explicit posteriors for 5,000 sampled MNIST test images, colour-coded by the class labels. All axes ranges from -4 to 4.
Posteriors Figure 4 gives the means of the explicit posterior, and Fig. gives the samples from the posteriors, explicit and semi-implicit. For L. and Ch;, the posteriors are visually closer to the prior for smaller gamma, except for between 7 = 0.5 and 0.1: their KL divergences are similar in Table 2 The scatter plots for C/)h and Chh--ltt (Figs. 5i to 5n are for the Bayes posteriors, and they seems have clumps than Ch.o (Fig. Sep.
Figure plots the samples from the implicit distributions of Cly, Lbb and Chhh-att separately for 16 test images, that is, one plot for each test image in each setting. We see diversity in the implicit distributions in majority of the cases for C4, with < 1.0 (Figs. to to 6d) This is diversity is seldom for ELBO (Ch. oo Fig. 6a). but not totally absent despite the theory for otherwise (Yin & Zhou 2018), probably because of noise in learning with Monte Carlo samples and the neural network parameters giving similar optimum values.
For Cbh and Cb -hlt (Figs. 6e to 6j}. we see almost lack of diversity in the implicit samples. We attribute this to the larger magnitude of the lata-fit term in the objective over the divergence (about 340 times larger in Table 2 causing the theoretical degeneracy of the ELBO to be prominent.
To have broad overview the distributions of the implicits, we compute the sample covariance for the implicit distributions of each test image with 500 samples after transforming with aresine suuuene--uurrrooot. Each sample covariance is used compute the generalised variance and total variation, and we summarise them over the 10,000 test images using the following descriptive statistics (Table median, maximum, mean, coefficient of variation (CV) and skewness (Fisher-Pearson coefficient). We caution that this assumes single modes in the two-dimensional implicit distributions. We find the medians, maximums and means of the generalised variances to be significantly smaller than that of the total variations, which indicates that most distributions approximately degenerate and can hardly be considered two-dimensional distributions. Moreover, looking at the medians of the total variations, total degeneracy to delta-distributionn occurs for more than half of the implicit distributions of Ch with 7 = 1.0 (ELBO), ChI with 7 0.1 and Cbb-alt with 7 0.9. These agree with the plots in Fig. and show that our approach cannot prevent degeneracies. Nonetheless, if we compare among the statistics for the Chss, we find that settings with 7 < 1 give less degenerate distributions than ELBO (y = 1.0).
The coefficient of variations are at least 1.9, indicating very different implicit distributions for the test samples. The skewness
26

---

Variational Learning of Fractional Posteriors
(8) 1 1.0
(b) y 0.9
(0) L, 0.5
(d) y = 0.1
(0) ( 1.0
(1) - 0.9
(g) L 0.5
(h) 7
( C bb 7 0
(i) bb = 0.5
(k) bb ,7 0.1
(1 bb 7
(m) bb -lt, 0.5
(n) c -lt. 7
Figure 5: 5,000 samples from the posteriors of the MNIST test images. For Lb and abb the Bayes posteriors used. All axes ranges from -4 to 4.
27

---

Variational Learning of Fractional Posteriors
(a) 7 1.0
(b) L, 0.9
(c) L" y 0.5
(d) L", y 0.1
(c) ch Y 0.9
(f) hh 7 0.5
L 7 0.1
(h) alt, y 0.9
(i) bb 0.5
bb 0.1
Figure 6: 500 samples from the implicit posteriors for 16 MNIST test images: one small square is for one image. All axes ranges from to 1, which is the range of the samples by design.
28

---

Variational Learning of Fractional Posteriors
Table 8: Descriptive statistics, over the test images, of the generalised variance and total variation of the transformed implici distributions. Figures less than 10 are treated as zero. For a number a x 10", we express as nb except that we use O when a = 0 and, when b 0.
Generalised Variance
Total Variation
Objective
Median
Max
Mean
CV
Skew
Median
Max
Mean
CV
Skew
Ch
1.0
5.4 3
3.2
3.0
4.5
0
8.2
2.0
3
1.1
1.8
0.9
1.9 -1
13
6.8
1.1
-3
5.5
2.3
3.6
0.5
1.9
1.8 1.83
1.0 1.0
1.9
0.1
1.5 1
1.3 1.3
3.8
5.9
2.3
Cbh
0.9
1.1 -
1.5
7.4
9.7
4.5
9
4.5
1.3 3
9.9
1.6
0.5
1.5 1 2
2.5
6.3
8.7
3.9
1.7
3
1.0
1.7
0.1
8.2 8.2-2 2
2.4 2.43
4.3
6.6
0
5.9
7.2
3
5.9
8.1
abb
0.9
1.5 1.5- 2
3.2
5.2
7.9
0
5.6 5.6-1
4.3 3
7.8
1.1
0.5
3 3
-
4.5
7.4
14
7
5.3
1.8
1.0
1.71
0.1
7.4 3
3
4.1
3.2
4.1
3.0
6
5.6
4.2
3
1.1
are positive, indicating high proportion very low variance implicit distributions, and this also shown by the medians being smaller than the means. In particular, the skewness for ELBO about five times more than for C with 7 = 0.9.
C.5. Improving the VAE Decoder by Learning Fractional Posterior
Figures 7a to Visually, the best samples are provided by = 10-5 5 (Fig. 7d.. For decreasing y, the FIDs are 83.5, 69.5, 67.8 and 68.8 (smaller is better). While the FIDs for the fractional posteriors are similar, they are all significantly better than the Bayes posterior's.
7d provide sample images for the decoders trained with 7 taking values 1.0 (for ELBO), 10-1, 10-3 and 10-5,
Since the 3-VAE at its theoretical optimum also gives power posterior, also evaluate training with its objective, called f The fractional posterior for C.g corresponds directly to that for L. with 3 = 1/7, so we use 10 103 and 10 for B.
The FIDs in increasing 3 are 77.3, 334.7 and 342.3. While the FIDs for Lit (corresponding to 7 = 10 1001) improves over the 83.5 of ELBO's, it significantly worse than the 69.5 of L10 1. Moreover, increasing to 10 and 10 appears to cause
degeneracy to a different and worse optima, contrast to the stability afforded by Ly. Figures 7 to 7h provide the sample images. We further tried and 10 for 3, giving FIDs 78.4 and 99.1.
Table 9 provides the statistics of the bounds to the data evidence for the trained VAEs. Similar the results for MNIST (Table Ly with smaller give tighter bounds and the learnt posteriors are closer the prior. For Lie the bounds are looser than ELBO's, as expected. For C3 with 33 E {103, 105), the direct multiplication of the divergence term in 2 has cause instability such that the empirical divergence computed by sampling becomes negative more samples than the 1,024 used here could resolve this issue for 3-VAE.
All the preceding results are with two-dimensional latent space. We tested a case of four-dimensional latent space using our bound with set to 1.0 (for ELBO), 10-3 and 10-5, With this more expressive model, the evidences are larger (last three row Table 9.. and the FIDs are better at 58.3, 56.8 and 55.8. Again, we see an advantage for < 1.0, though now the benefits are much less significant.
C.6. Source Codes and Data Sets
Other than the standard Python and Py Torch fatts::/pytoorch.orgg including Torchvision, packages, we take reference from and make use of the following source codes:
VAE Mtttep//tttiiicoorrrrrrr/r//eeea
SIVI https://gthhbb.ccmmmmiiiaaaaaagp
29

---

Variational Learning of Fractional Posteriors
7.
(b) L. -0
(c) 7 10
(d) L. 10
(e) Test samples
(f) C 3 10
(g) C 8 10
(h) L B 10
Figure 7:
Mean images from decoded latent variables obtained by coordinate-wise inverse-CDF (standard normal) transform from a unit square. The VAE is trained on the Fashion-MNIST dataset. The top row uses the bound introduced this paper; the bottom row (sans the first figure) uses the objective from 3-VAE.
Table 9: Average log-evidences (higher better) over data samples, and its breakdown for VAE on Fashion-MNIST data sets. For Monte Carlo averages, 1,024 samples are used. For 1.0, the figures are the same under Test using Objective and Test using ELBO. The columns under Test using ELBO are solely for diagnostics to understand the learnt posteriors using the same metrics: they are not performance measures. For ease of comparison, we also add the last column FID scores of the 10,000 generated images.
Test using Objective
Test using ELBO
dim(=)
Obj.
7
Train (Total)
Total
data
div
Total
data
div
FID
2
C
1152.30
1118.15
1123.11
4.96
1118.15
1123.11
4.96
83.5
10-1
1180.08
1171.38
1173.97
2.59
902.79
904.56
1.77
69.5
10-3
1179.68
1171.05
1173.91
2.87
915.56
917.30
1.74
67.8
10
1183.97
1175.29
1178.04
2.75
853.45
855.16
1.71
68.8
C
5.0
1135.83
1103.29
1121.04
17.76
1117.50
1121.05
3.55
78.4
10'
1120.62
1086.02
1117.13
31.11
1114.01
1117.12
3.11
77.3
102
937.29
921.26
1067.22
145.95
1065.74
1067.20
1.46
99.1
103
754.34
752.08
598.54
153.54
598.69
598.54
00.15
334.7
105
15946.58
15952.71
598.53
15354
598.69
598.54
00.15
342.3
4
C,
1.0
1220.04
1200.40
1207.98
7.58
1200.40
1207.98
7.58
58.3
10-3
1231.16
1219.11
1225.14
6.04
1147.20
1151.29
4.09
56.8
10-5
1231.02
1219.24
1225.36
6.12
1147.87
1152.03
4.16
55.8
30

---

Variational Learning of Fractional Posteriors
FID http:://gtthbb.ccmmmmaaaattttttt
The above PyTorch code for FID recommended by the original authors FID at https://githu: com/bioinn-jjku/TruR Our source codes are available at https://githul. com/ csiiociiiroooo funmi/ They are executable on single NVIDIA T4 GPU, which are available free (with limitations) Google Collab, Kaggle and Amazon SageMaker Studio Lab the point of writing. A single training run of the 500 epochs for the VAE experiments for Cyy is currently achievable within 12 hours on Kaggle.
The MNIST and Fashion-MNiST data sets are available via Torchvision.
31


Reviews:
[{"review_id": "BulvXNTGfD", "reviewer": "Reviewer_H3En", "summary": "The classical variational inference (VI) often underestimates the uncertainty, motivating the research of generalized Bayesian inference. Nevertheless, the theoretical connections between generalized Bayesian inference bounds and the marginal likelihoods are only established approximately/asymptotically, hindering careful use in practice. This work shows that a family of generalized Bayesian inference bounds are lower bounds to the log marginals with an application of the Hölder’s inequality. The bound is tight when the approximate posterior is a fractional posterior, which interpolates between the true posterior and the prior. There are two main applications of the bound. The calibration of the approximate posterior from such a bound can be adjusted by setting a parameter $\\gamma$, and the bound can also be utilized for learning generative models such as VAEs.\n\nHierarchical models are widely used in practical Bayesian inference. This work further derives two objectives with fractional posteriors for hierarchical models, an objective with structure and an objective that allows subsampling. There are also cases where components in the objectives can be derived analytically and the work shows three of them. \n\nIn the experiments, it is first shown that with proper setting of $\\gamma$, the fractional posteriors can achieve the correct coverage given a confidence level, while having an interval similar to that from ELBO. For MNIST image modeling problems, tuning $\\gamma$ is also useful for having a better model learning capability. On FashionMNIST, it is shown that the generation quality increases when $\\gamma$ is reduced, while retaining the theoretical aspects of a variational lower bound.\n\n## update after rebuttal\n\nI was leaning towards acceptance for this work. I still think this work is worthy of acceptance after reading through other reviewers' comments. I keep my score, but do not increase it because I also foresee huge efforts in the revision.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Methods And Evaluation Criteria\n> For the methodology, only the derivation of $\\mathcal{L}^{bh}$  is not clear to me. As in A.3, there are two possible objectives, but the main text uses the one with both $\\boldsymbol{u}$ and $\\boldsymbol{u}$’  in the integrand. I do not see why this one is chosen.\n\nThis one is chosen because we have its experimental results at time of submission. We now have experimental results for the alternate and simpler bound in A.3 — the empirical results are very similar (for example, the test objective for $\\gamma=0.5$ is $1608.0\\pm25.4$), and the implementation is simpler and does not involve triple integrals. We also have proof showing non-degeneracy of the implicit distributions is not neccessary, similar to A.4.2.\n\nWe will replace the currently chosen bound with the simpler bound in the main text. The current bound will be placed in the appendix. The essential arguments in the experimental section remain the same.\n\n> For a theoretical paper like this, I think the benchmark datasets are enough to support the claims. However, it would be much better if the comparison with $\\beta$-VAE is also demonstrated in the experiments.\n\nThe $\\beta$-VAE is demonstrated in C.4 for generating images from the prior, and this is referenced from the main text in section 6 (line 409, right column). In addition to the parameters for $\\beta$-VAE currently in C.4, we have additionally tried with $\\beta$ taking values 5 and $10^2$, giving FIDs 78.4 and 99.1. The conclusions are the same. We will include these additional results in the final version.\n\nSubject to space constraints, we will move some of these results to the main text in the camera-ready version.\n\nSince $\\beta$-VAE for fractional posteriors is provably less tight than ELBO, we do not include the results in section 5.2/Table 2.\n\n### Theoretical Claims\n> Section 3.1 contains three cases where components in the objective could be derived analytically. Given that modern Bayesian models can be written as probabilistic programs with program tracing and autodiffs, I suppose most of the derivations could be automated in practice.\n\nYes indeed, for the most part. In addition, we believe knowing the derivations and derived formulae have pedagogical value, and can give rise to new algorithms and/or more efficient updates in future work. For example, autodiff will not be able to choose $1/\\gamma = 1 + 1/n$ (section 3.1.2); nor is it able to decide to apply $\\mathcal{L}_\\gamma$ first on $q(\\boldsymbol{u})$ and then ELBO for $p(\\boldsymbol{c})$ (section 3.1.3), and in this order. After these are determined, and perhaps with further simplification of the bounds and equations, then autodiffs can be applied readily.\n\n### Essential References Not Discussed:\nWe will mention [1]-[3] and relate them to our work.\n\n### Questions For Authors:\nAnswered above.", "claims_and_evidence": "I think the theoretical claims and empirical evidences are strong.", "methods_and_evaluation_criteria": "- The discussion of the methods is thorough and strong. In addition to the variational objectives, this work also provides recipes for stochastic optimization, as well as a second-step variational objective when the first-step approximate posterior is not normalized.\n- For the methodology, only the derivation of $\\mathcal{L}^{bh}$ is not clear to me. As in A.3, there are two possible objectives, but the main text uses the one with both $u$ and $u'$ in the integrand. I do not see why this one is chosen.\n- For a theoretical paper like this, I think the benchmark datasets are enough to support the claims. However, it would be much better if the comparison with $\\beta$-VAE is also demonstrated in the experiments.", "theoretical_claims": "- It is neat to have a generalized Bayesian inference objective that is a lower bound to the log marginal. I think this contribution is strong enough to establish the work.\n- Section 3.1 contains three cases where components in the objective could be derived analytically. Given that modern Bayesian models can be written as probabilistic programs with program tracing and autodiffs, I suppose most of the derivations could be automated in practice.", "experimental_designs_or_analyses": "Most experiments are on the effects of changing $\\gamma$ in the fractional posterior framework. The designs and analyses are sound to me.", "relation_to_broader_scientific_literature": "From what I can see, this work extends the generalized variational bounds from Li & Turner, 2016 in the context of generalized Bayesian inference, and is heavily influenced by Yin & Zhou, 2018, for implicit approximate posterior.", "essential_references_not_discussed": "There is another line of works in generalized Bayesian inference with empirical losses and regularizers [1-3]. This work has similarly structured objectives as those works (though they do not produce tight lower bounds). I don't think they should be compared, but they are worth mentioning.\n\n[1] Masegosa, A. (2020). Learning under model misspecification: Applications to variational and ensemble methods. Advances in Neural Information Processing Systems, 33, 5479-5491.\n\n[2] Morningstar, W. R., Alemi, A., & Dillon, J. V. (2022, May). PACm-Bayes: Narrowing the empirical risk gap in the misspecified Bayesian regime. In International Conference on Artificial Intelligence and Statistics (pp. 8270-8298). PMLR.\n\n[3] Lai, J., & Yao, Y. (2024). Predictive variational inference: Learn the predictively optimal posterior distribution. arXiv preprint arXiv:2410.14843.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No.", "questions_for_authors": "- Why is the objective with both $u$ and $u'$ chosen for $\\mathcal{L}^{bh}$? Is there a practical obstacle that keeps the other from being implemented?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AUV8daECQG", "reviewer": "Reviewer_F6u2", "summary": "The paper proposes a variational objective targetting fractional posteriors based on the Holder inequality instead of the more typical Jensen's inequality approach. Furthermore, for hierarchical models, minor variations of the variational objective are considered. The utility of the approach is demonstrated through coordinate-ascent variational inference of mixture models and gradient-based variational expectation maximization of deep latent variable models.\n\n## update after rebuttal\nThrough intense discussion with the authors, the points of disagreement have been sufficiently identified, and the authors have promised to address them. As such, I am now in favor of the paper to be accepted. However, the rather large number of changes promised by the authors makes me think that an additional round of review would be beneficial. As such, I will lean toward borderline.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We suspect misunderstandings. We ask for further clarifications below.\n \n### Claims & Evidence\n> contributions ... confounded\n\nOur key contribution is a lower bound that _also_ approximates fractional posteriors. Having both at once is new in ML and statistics. Moreover, ELBO is a special case. So, we fill a gap between standard VI (ELBO) and fractional Bayesian inference.\n\nFractional posteriors have been shown to be more robust (sec 1 2nd para), and have applications in calibration (sec 5.1).  A lower bound allows _principled_ maximisation of the hyperparameters (ML-II), common in Bayesian ML (sec 5.2). Having both at once benefits generation from VAE decoder (sec 5.3).\n\nWe bring the bound in sec 2 to expts in sec 5 by developing complex posterior constructions (sec 2.3), parameter updates (sec 3) and MC estimates (sec 4).\n\n> $L_{ELBO}^{\\gamma}$\n\n$L_{ELBO}^{\\gamma}$ essentially changes the likelihood of the model, so it is an ELBO to that changed model, and may not lower bound the original model (depends whether $p(D|z)>1$). In contrast, ours is a theoretical lower bound to the original model. It holds for all $\\gamma\\in(0,1)$ and all $q$.\n\nNonetheless, the optimal solution of $L_{ELBO}^{\\gamma}$ indeed approximates the fractional posterior of the original model. It's the same as $\\beta$-VAE’s objective, by dividing by $\\gamma$ (Table 3). If we only want a fractional posterior, then this suffices. If, in addition, we wish to estimate the hyperparameters of the model, as is common in ML, then our lower bound is effective (Table 2). \n\n> statistically better in any sense than naively using the ELBO ...?\n\nWe show our bound to be computationally more stable and to give better results than $\\beta$-VAE for the FashionMNIST in C.4 (referenced in sec 6). We have not analysed the properties such as convergence rates. Please clarify if we misunderstood the question.\n\n> computational benefit?\n\nNo. Indeed, KL is mathematically more convenient. Computationally, if implemented using sampling (e.g., for VAE), we swap the log and sum operations, include multiplicative and additive constants, and use more than one sample (line 807).\n\n> Works that propose new divergences …what is uniquely new or desirable about the proposed divergence\n\nWe provide a variational objective that lower bounds evidence _and_ enables estimation of approx. fractional posteriors. This is _not_ fulfilled by $L_{ELBO}^{\\gamma}$ (nor $\\beta$-VAE).\n\n### Methods & Evaluation\n> ... a surrogate for some divergence, but which one? ... equivalent to this divergence up to a constant ... a strict surrogate?\n\nOptimality is at the fractional posterior so it's not a divergence to/from the Bayes posterior. We do not think it’s equivalent to any known divergence up to a constant (the evidence). We can define a divergence (from the fractional posterior) based on our bound: $D(q||(p(z),p(D|z),\\gamma)) = L_{evd} - L_\\gamma$, where the triple defines the target posterior.\n\nWe can better answer if “strict surrogate” is defined.\n\n> $\\gamma$ is a specification of the model, not the inference\n\n$\\gamma$ is not a specification of the model. From a Bayesian viewpoint, the model is fully specified by likelihood $p(D|z)$ and prior $p(z)$. Given a data set, this fixes the exact marginal likelihood or data-evidence $p(D)$ (MacKay, 2003). \n\nOne may compute $p(D)$ and $p(z|D)$ using sampling, such as MCMC. For complex models in ML, variational methods are developed (see 1st para of paper). We propose a new variational method that leads to approximate fractional posteriors. Our method has a parameter $\\gamma$ for the inference, but it does not change the model.\n\nWe admit there are schools other than Bayesian. In particular, in the regularisation community, the end objective is treated as the model, like the reviewer alludes to. To put our paper in the correct setting, it begins with “Exact Bayesian inference is ...”.\n\n> Optimal in what sense?\n\nIn the sense of giving a tighter bound to the evidence (1st sent in para). We will reiterate at the end of para.\n\n> modifying the model … statistically sound.\n\nWe emphasise that the model is fixed (given hyperparameters) and evidence is fixed (given data) in the Bayesian view. Our lower bound to the evidence is mathematically sound. It changes neither $p(D|z)$ nor $p(z)$; max wrt $q$ also changes neither. But max wrt $p(D|z)$ or $p(z)$ changes the model.\n\n### Experiments\nOur evaluation is sound: **In sec 5.1, the model is fixed** and the bounds are comparable for relative tightness to the same exact evidence. **In sec 5.2, the model changes** because we optimise the decoder, changing $p(D|z)$ (right para starting line 318). Here, the bounds are comparable for the quality of optimised models within the same family (i.e., same NN architecture) on the same data.\n\nWe should have used the word “tighter” more carefully. We will update the abstract; 4th para in intro; last para in sec 2.2; intro to sec 5. 4th and 5th para in sec 5.2; and conclusion.", "claims_and_evidence": "The contributions of the paper are a bit confounded, and this is my main concern. The main claim appears to be that the paper proposes a variational objective that can handle fractional posteriors. What I don't get is that one can use plain-old evidence lower bound maximization for fractional posteriors too: what is wrong with using the ELBO\n$$\n\\mathcal{L}_{\\mathrm{ELBO}}^{\\gamma}(q) = \\int \\big\\\\{ \\gamma \\log p(\\mathcal{D} \\mid z) + \\log p(z) \\big\\\\} q(\\mathrm{d}z) + \\mathbb{H}(q) \n$$\nfor approximating a $\\gamma$-fractional posterior?\nTherefore, I do not find that being able to handle fractional posteriors can be claimed as a technical contribution. \n\nOn the other hand, it does appear that the paper is proposing a novel variational objective based on Holder's inequality. But it is unclear what the claimed benefits of this new objective are. Is this statistically better in any sense than naively using the ELBO as above? Is there a computational benefit? Works that propose new divergences need to articulate what is uniquely new or desirable about the proposed divergence. For instance, the initial claim about $\\alpha$- and $\\chi^2$-divergences [1,2] was that they reduce the mode-seeking behavior of the exclusive KL divergence. The long-standing argument for the exclusive KL divergence is that it is computationally convenient to optimize.\n\nUnless there is some serious misunderstanding on my end, I think the paper should reconsider its positioning and refine its technical claims.\n\n1. Li, Yingzhen, and Richard E. Turner. \"Rényi divergence variational inference.\" Advances in neural information processing systems 29 (2016).\n2. Dieng, Adji Bousso, et al. \"Variational Inference via $\\chi $ Upper Bound Minimization.\" Advances in Neural Information Processing Systems 30 (2017).", "methods_and_evaluation_criteria": "The same comments apply here. On a technical level, however, I would also like to point out that it is unclear what the objective $\\mathcal{L}_{\\gamma}$ is doing. It is fair to assume that it is a surrogate for some divergence, but which one? Is the proposed objective equivalent to this divergence up to a constant (The ELBO is exactly the same as the KL divergence up to a constant), or is it a strict surrogate?\n\nFurthermore, the discussion in Section 2.2 needs to be more subtle. Selecting $\\gamma$ is a specification of the *model*, not the *inference algorithm*. That is, the submission claims that if a variational approximation can \"approximate only certain fractional posteriors well, then the corresponding $\\gamma$s would be optimal.\" Optimal in what sense? This is essentially modifying the *model* so that some variational approximation better approximates it, but there is no reason to believe that this will be statistically sound. The model that is best approximated could be terrible. Instead, previous works took different approaches, from maximizing the predictive density [1] or taking the ABC perspective [2], which come with accompanying theoretical analyses.\n\n1. Grünwald, Peter, and Thijs Van Ommen. \"Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it.\" Bayesian Analysis (2017): 1069-1103.\n2. Miller, Jeffrey W., and David B. Dunson. \"Robust Bayesian inference via coarsening.\" Journal of the American Statistical Association (2019).", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "For reasons mentioned above, the evaluation is not entirely sound. The paper is proposing a variational objective and corresponding inference algorithms. Thus, the evaluation should focus on evaluating the following: \"Given a fixed value of $\\gamma$, how accurate is the obtained variational approximation?\" The experiments are not appropriate for doing this.\n\nThe experiments in Table 1, for example, are not doing this: coverage confounds the properties of the model and the algorithm, so they are not informative about the performance of the algorithm/variational objective. Furthermore, since the model changes depending on $\\gamma$, the variational bound values in between different values of $\\gamma$ cannot be compared, and individual bound values are not interpretable. Similar arguments apply to Table 2.", "relation_to_broader_scientific_literature": "The paper proposes a new variational objective, which extends the existing literature on developing alternatives to the ELBOs. Given the concerns above, it is unclear how the work is positioning itself within this context.", "essential_references_not_discussed": "* Section 2.3: The objectives specialized to hierarchical models in Section 2.3 are reminiscent of the \"locally-enhanced bounds\" proposed in [1]. I recommend taking a look for connections.\n* Line 228: The reparameterization gradient for variational inference was independently proposed by [2,3] as well.\n\n1. Geffner, Tomas, and Justin Domke. \"Variational inference with locally enhanced bounds for hierarchical models.\" arXiv preprint arXiv:2203.04432 (2022).\n2. Titsias, Michalis, and Miguel Lázaro-Gredilla. \"Doubly stochastic variational Bayes for non-conjugate inference.\" International conference on machine learning. PMLR, 2014.\n3. Rezende, Danilo Jimenez, Shakir Mohamed, and Daan Wierstra. \"Stochastic backpropagation and approximate inference in deep generative models.\" International conference on machine learning. PMLR, 2014.", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "* The term \"hierarchical posterior\" has been used throughout. This is a bit unusual as a posterior may not have any notion of hierarchy, but a *model* can be hierarchical.\n* Line 30 \"high-variance estimators\": Why is Roeder 2017 cited here? Roeder proposes a lower variance estimator, so this doesn't appear to be the right citation here. Furthermore, the fact that coordinate descent can be used does not necessarily fix everything since it can only be used with mean-field conjugate families.\n* Line 69-72 \"The Kullback-Leibler divergence ... is the only case where the chain rule of conditional probability\": I am not sure why this is relevant here.\n* Line 94 \"it is achieved without relying on PAC-Bayes or modifying the likelihood\": PAC-Bayes bounds are generalization bounds and not about variational inference algorithms. (Alquier *et al.* 2016 establish generalization guarantees for variational posteriors.) What did the authors intend by this sentence?", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TD30veYjNv", "reviewer": "Reviewer_SAc3", "summary": "The paper introduces a novel one-parameter variational objective that generalizes the standard evidence lower bound (ELBO) by enabling the estimation of fractional posteriors. By leveraging Hölder’s inequality, the authors derive a new bound L₍γ₎ which recovers the conventional ELBO in the limit as γ → 1. The framework is further extended to hierarchical and Bayes posteriors, and the paper provides both analytical gradient derivations for cases such as exponential family models and mixture models, as well as empirical studies. Experiments on Gaussian mixture models and variational autoencoders (VAEs) demonstrate that fractional posteriors yield better-calibrated uncertainties and improve generative performance, particularly in aligning the VAE decoder with the prior distribution.\n\n**Updates after Rebuttal**\n\nThanks for the rebuttal. I think most of my concerns have been addressed. I will keep my scores.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Relation To Broader Scientific Literature\n\nThank you. We will try our best to incorporate these into the paper.\n\n### Other Strengths And Weaknesses\n> The method involves nested integrations and Monte Carlo estimates (especially in the hierarchical and semi-implicit formulations), which may lead to high computational overhead. This can become particularly problematic in high-dimensional latent spaces or when dealing with complex models.\n\nAs in our reply to Reviewer H3En, the triple integral in section 2.3.2 can be reduced to a double integral (see section A.3 and last row of Table 3). The remaining double integrals are necessary since we have a hierarchical construction, involving $\\boldsymbol{u}$ and then $\\boldsymbol{z}|\\boldsymbol{u}$. Fortunately, depending on the application, one may not need such a construction if we set $\\gamma$ to be small so that the approximating family can be simple, as discussed in section 2.2.\n\n### Questions For Authors\nWe have done preliminary work on such extensions, but it is not trivial. We leave such matters for future work to keep the current paper focused. Subject to space constraints, we may suggest approaches in relation to other frameworks in an additional future work or discussion section, especially on pitfalls to avoid based on our experience.", "claims_and_evidence": "Yes all claims are supported by clear theoretical and empirical evidence", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I have checked all theoretical proofs, and I found no overt mistakes in the proofs.", "experimental_designs_or_analyses": "Yes I have checked the soundness and validity of experimental designs, and they are sound.", "relation_to_broader_scientific_literature": "The proposed approach is closely linked to the density estimation literature by tempering the likelihood to improve robustness and convergence properties. For instance, Friel and Pettitt (2008)  introduced power posteriors to facilitate marginal likelihood estimation in mixture models—a concept that underpins the use of fractional likelihoods in density estimation. Similarly, O’Hagan (1995) \nARXIV.ORG\n showed that raising the likelihood to a fractional power can yield more objective Bayes factors, particularly valuable in nonparametric settings where standard models are prone to overfitting. \n\nIn the context of variational autoencoders (Kingma & Welling, 2014), the paper’s findings relate to efforts such as β-VAE (Higgins et al., 2017) and IWAE (Domke & Sheldon, 2018), where modifying the ELBO has been proposed to improve disentanglement or tighten the variational bound.", "essential_references_not_discussed": "The paper covered most of the essential works in this field.", "other_strengths_and_weaknesses": "Strengths:\n* The experiments on VAEs, including improvements in evidence bounds and better alignment of decoder distributions for generative tasks, indicate potential practical benefits in generative modeling and beyond.\n\nWeaknesses\n* The method involves nested integrations and Monte Carlo estimates (especially in the hierarchical and semi-implicit formulations), which may lead to high computational overhead. This can become particularly problematic in high-dimensional latent spaces or when dealing with complex models.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "How would the proposed fractional posterior works when a generation framework other than VAE is applied? Say a flow-based framework.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kan9N05Q7B", "reviewer": "Reviewer_LKGu", "summary": "The authors introduce a novel variational inference method based on fractional posteriors, parameterized by a single scalar\nγ∈(0,1). This approach generalizes Bayesian variational inference by tempering the likelihood term, leading to fractional posteriors that provide improved calibration and robustness. It extends to hierarchical models and shows applicability in both analytic and empirical scenarios, including Gaussian mixture models (GMMs) and variational autoencoders (VAEs). The main contributions include derivations of novel variational lower bounds based on Holder’s inequality, analytic gradient expressions, and empirical demonstrations showing improved calibration and generative modeling performance over conventional methods such as ELBO.\n\n## update after rebuttal\n\nI would like to thank the authors and the other reviewers for the discussion.\nI keep my score, as explained in my comments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> some claims … insufficiently supported by the empirical\n\nLines 12 (right) and 87 (right) are general remarks on robustness of fractional posteriors, citing others. Statement on line 44 (right) follows these, but may be misunderstood. We will change to “an alternative to”.\n\nAlignment with prior is substantiated by Table 2 (last col), Fig 4&5, and  Fig 7 & Table 5.\n\n### Experimental\n> only two-component mixtures\n\nBeyond 2 components, the marginal likelihood landscape is complex (line 328, left, citation therein). The main text uses 2 components for clarity and focus. Beyond this, we need more exposition to separate the confounding effects due to the complex landscape. Nonetheless, C.1.2 gives results for 4 components.\n\n> linear regressions … lack sufficient theoretical justification\n\nIndeed, it’s more justified to regress against $\\ell^{-2}$ (C.1.1). This gives better interval lengths, but worse coverages. Other strategies are possible (last sent. in sec. 6).\n\n### Comments (numbered)\n1. Yes. See caption (also Fig 4 of Kingma&Welling, 2022]. marginal-CDF$\\in [0.1, 0.9]$\n2. All results are based on the variational approximation optimised on train set. The empirical error-bars are for 10 runs, a different seed each to initialise the NN.\n\nWith “ELBO/div”, we know that the fractional posteriors are closer to the prior for smaller $\\gamma$ in the same sense of KL. This can't be seen from “Objective/div” because Renyi-divergence differs with $\\gamma$. See also line 358 (left)\n\n3. Ok\n4. We haven't systematically investigated. Table 2 hints at stability from the relatively small error bars (3 x stddev) by using 100 samples per datum and 10 runs (details in C.3).\n\n### Questions (numbered)\n1. Sensitivity analysis of $\\gamma$ must be done within posterior families (sec 2.2). Current results show the quality of the bound and the closeness of the approx. fractional posterior to the prior.\n\n2. We will acknowledge this experimental limitation in an additional section.\n\nCurrent 2d latent space allows posterior illustrations (Fig 4 to 6). We use NNs from [Ruthotto & Haber, 2021], with only 88,837 parameters for 2d latent space, so results can be obtained on free platforms (sec C.5). We need richer NNs for more complex data.\n\nWe are also careful and use the correct likelihood: continuous Bernoulli. Otherwise, bounds in Table 2 are not convincing.\n\nWe expect similar benefits compared with existing objectives such as ELBO or $\\beta$-VAE’s, when the goal is either better evidence bounds, or more stable computation of fractional posteriors, or both.\n\n3. Degeneracy isn't necessary for optimality; but it isn't avoided and is also a solution, as stated in A.4. There, Eqs 3&4 say if we explicitly enforce $q(u)\\neq0$ at multiple locations, optimality is still possible. So, we can design $q(u)$ to be non-degenerate if we wish. Further work can be on crafting training dynamics towards non-degeneracy.\n\n4. For FashionMNIST, using 4d latent space and the following $\\gamma$s for our primary bound:\n* 1 (ELBO): train/test bound=1220/1200; FID=58.3\n* $10^{-3}$: train/test bound=1231/1219; FID=56.8\n* $10^{-5}$: train/test bound=1231/1219; FID=55.8\n\nSo for 4d, bound for smaller $\\gamma$ gives better results. Overall 4d results better than 2d (see C.4 for existing values), as expected for a richer model with more parameters.\n\nWe haven't analysed the MC approx error.\n\n5. We don't directly address posterior collapse. If we must, we might seem to be encouraging collapse, but it's more complicated and demands more discussion. Figure 4(d) gives a prelude, where the fractional posteriors as a whole aggregate towards the prior, but posterior for every data point is different.\n\n6. With respect to image generation from prior (sec 5.3), we learn a decoder that operates well with the prior, which can be simple. At the same time and within the same objective, the NN are optimised with respect to the fractional posteriors that are close to the prior.\n\nPost learning the empirical prior requires its model to be sufficiently flexible to approx the posteriors, and a separate procedure to learn its parameters.\n\nSo we believe our proposal is simpler.\n\nWhich is more practical depends on context within a wider ML/AI sys — e.g., if posteriors exist from previous intensive training/tuning, the empirical prior approach might be preferred. \n\n7. The samples are noisy also because NN architectures for the encoder & decoder are simple with 89K parameters for 2d latent space. For CIFAR10, we add one CNN layer (total 3) each for encoder & decoder, total 530K parameters for 32d latent space. Number of train/eval samples reduced to16/128; epochs reduced to 300 (C.3 gives current settings). Results using SageMaker StudioLab:\n* $\\gamma=1$ (ELBO): train/test bound=1229/1172; FID=141\n* $\\gamma=10^{-5}$: train/test bound=1238/1184; FID=135\n\nOur bound for $10^{-5}$-posterior gives better results. Better results need more complex architectures, e.g., ResNet-1001 with 10.2M parameters.", "claims_and_evidence": "Most claims are supported through theoretical derivations and by limited empirical results . The authors convincingly show the benefit of fractional posteriors for uncertainty calibration in GMMs and generative modeling with VAEs. However, some claims about robustness to model misspecification and improved alignment with priors are insufficiently supported by the empirical evaluations provided. Specifically, the paper claims robust performance improvement, but the supporting evidence is limited to simplified experiments that might not generalize to more complex scenarios.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are mostly appropriate. The use of simple benchmarks like MNIST and Gaussian mixtures is effective for demonstrating conceptual advantages, though it somewhat limits the impact. For the calibration study on the GMM, the chosen metric (credible intervals and coverage probabilities) is appropriate. However, broader applicability could have been shown with additional, more complex datasets or larger-scale problems to demonstrate practical relevance clearly.", "theoretical_claims": "The main theoretical derivations involving the lower bounds via Hölder’s inequality (Equations (1) and (2)) appear mathematically sound. However, the proof that the fractional posterior exactly corresponds to an optimal lower bound (Section 2.1) should be elaborated further to highlight clearly any assumptions required. Additionally, the limits as γ approaches 1 (recovering ELBO) are adequately proven, though readers might benefit from more intuitive explanations (which are indeed present in the supplementary materials).", "experimental_designs_or_analyses": "The empirical designs have notable issues. The calibration experiments with Gaussian mixtures provide insights into fractional posteriors' theoretical properties; however, only two-component mixtures are explored, which significantly limits the generalizability of results to more complex, realistic scenarios. The linear regressions used to select optimal γ values (Rℓ and Rκ) lack sufficient theoretical justification—why linear regression should capture the relationship between coverage intervals and fractional parameters?\n\nIn VAE experiments, it is unclear whether the observed improvements in generative quality generalize beyond the simple MNIST and Fashion-MNIST datasets, or the limited dimensionality (two-dimensional latent space). Experiments involving higher-dimensional latent spaces or more complex datasets are needed to substantiate the broader claims about generative performance improvements.", "relation_to_broader_scientific_literature": "The paper situates its contributions clearly within the context of existing literature on fractional posteriors, variational inference (VI), and related methods (e.g., β-VAE, importance-weighted ELBO, and general fractional posterior methods). It explicitly connects its novel bounds with existing concepts like PAC-Bayes and generalized variational inference (Knoblauch et al., 2022). However, the paper downplays related work that explores tempering or regularization in posterior inference (e.g., mitigating posterior collapse), which could contextualize the novelty more comprehensively.", "essential_references_not_discussed": "Several relevant approaches are noticeably absent. Examples:\n\n* Recent advances in \"cold posteriors\" (e.g., \"How Good is the Bayes Posterior in Deep Neural Networks Really?\" by Wenzel et al.) share conceptual similarities with fractional posteriors in controlling posterior calibration but are not discussed adequately.\n* \"A Simple Baseline for Bayesian Uncertainty in Deep Learning\" by Maddox et al. which propose  Stochastic Weight Averaging (SWA) approach that also deal with tempered posteriors and could offer valuable comparisons.\n\nAlso many papers on mitigating posterior collapse are missing, such papers present an alternative to improving sample quality, especially when combined with learning an empirical prior post-training.\n\nExamples:\n* \"Preventing Posterior Collapse with delta-VAEs\" by Razavi et al.\n* \"beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework\" by Higgins et al.\n* \"Variational Lossy Autoencoder\" by Chen et al.\n\nInclusion of such works would clarify the positioning of this paper's contributions relative to established posterior inference approaches and methods known for calibration.", "other_strengths_and_weaknesses": "Strengths:\n* Clearly derived and theoretically justified new variational lower bounds.\n* Novelty in combining fractional posterior methods with variational inference.\n* Empirical demonstration that fractional posteriors improve generative tasks in VAEs.\n\nWeaknesses:\n* Experiments limited in complexity (simple datasets, low-dimensional settings).\n* Lack of comprehensive evaluation on scalability or applicability to large-scale practical tasks.\n* Limited sensitivity analysis regarding the γ hyperparameter.", "other_comments_or_suggestions": "* Figure 1 - clarity could be improved by adding explanation to each of the labels (a-d) explicitly. (b,c) looks a lot like latent interpolation, is that the case?\n* Table 2 - captions and discussions should explicitly clarify the implications of differences between empirical evidence bounds and variational approximations. If \"Test using ELBO are solely for diagnostics to understand the learnt posteriors using the same metrics\" then can you please explain what do we learn from it or alternatively remove it?\n* Line 069 - should be \"(KL, α → 1)\" and not \"(KL, α = 1)\"\n* Line 252 - what is the error for the MC estimates of Zc and Zd? Can you provide any bound? How will large error affect the learned parameters of a VAE model?", "questions_for_authors": "* How sensitive are results to different choices of γ? Would a systematic exploration (beyond the limited set used) reveal significantly different findings regarding calibration or performance?\n\n* Why did you not test your approach on higher-dimensional latent spaces or more complex data? Would you expect similar benefits in such scenarios, and if not, why?\n\n* Can you clarify the conditions under which the introduced hierarchical approach (section 2.3) avoids the degeneracy problem of ELBO? An explicit analysis would substantially strengthen the theoretical contribution.\n\n* The various approximations might limit the practicality of the proposed method to simpler dataset and lower dimensions. Have you tested the MC approximation error for latent space with more dimensions and larger and more complex images?\n\n* What is the advantage of the proposed method compared to many other methods that address posterior collapse in VAE? \n\n* Learning an empirical prior after training can also provide high quality samples (assuming posterior collapse was mitigated) and might be a simpler and a more practical solution. Can you explain why would the proposed method be beneficial here?\n\n* In  section C.4 clearly demonstrates areas of the prior (explored via interpolation) where the samples are noisy (Figures 3,6). \nGiven the very simple datasets (MNIST, fashion-MNIST), it is not clear how well the advantages of the proposed method will scale to more complex and higher dimensional data. Have you tried the proposed method with such datasets (e.g., ImageNet or even CIFAR10)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: We thank the authors for their compelling contribution.  Reviewers agreed that this work contains both interesting theoretical claims and compelling empirical evidence.  The main idea is novel, and the authors showcase a variety of settings where it can be applied.  The productive back and forth with reviewer F6u2 clarified a number of key questions, and the authors are encouraged to incorporate these points of clarification into the manuscript.
Decision: Accept (poster)