Loss function based second-order Jensen inequality and its application to particle variational inference
Authors: Futoshi Futami
Tomoharu Iwata
Naonori Ueda
Issei Sato
Masashi Sugiyama
Venue: NeurIPS 2021 Poster (2021)
Keywords: Variational inference
Bayesian inference
Jensen inequality

Abstract: Bayesian model averaging, obtained as the expectation of a likelihood function by a posterior distribution, has been widely used for prediction, evaluation of uncertainty, and model selection. Various approaches have been developed to efficiently capture the information in the posterior distribution; one such approach is the optimization of a set of models simultaneously with interaction to ensure the diversity of the individual models in the same way as ensemble learning. A representative approach is particle variational inference (PVI), which uses an ensemble of models as an empirical approximation for the posterior distribution. PVI iteratively updates each model with a repulsion force to ensure the diversity of the optimized models. However, despite its promising performance, a theoretical understanding of this repulsion and its association with the generalization ability remains unclear. In this paper, we tackle this problem in light of PAC-Bayesian analysis. First, we provide a new second-order Jensen inequality, which has the repulsion term based on the loss function. Thanks to the repulsion term, it is tighter than the standard Jensen inequality. Then, we derive a novel generalization error bound and show that it can be reduced by enhancing the diversity of models. Finally, we derive a new PVI that optimizes the generalization error bound directly. Numerical experiments demonstrate that the performance of the proposed PVI compares favorably with existing methods in the experiment.

Loss function based second-order Jensen inequality and its application to particle variational inference
Futoshi Futami, Tomoharu Iwata, Naonori Ueda
NTT Communication Science Laboratories (futoshi futami. uk, tomoharu. iwata. gy, naonori ueda @rco att.co...
Issei Sato, Masashi Sugiyama
The University of Tokyo
satolg. ecc, u-t.kuu..kkccoo ac. jp. sup,uugiiee te.k.uutoon u-tokyo..c....
Abstract
Bayesian model averaging, obtained as the expectation of likelihood function by posterior distribution, has been widely used for prediction, evaluation uncertainty, and model selection. Various approaches have been developed to efficiently capture the information the posterior distribution; one such approach is the optimization of set of models simultaneously with interaction to ensure the diversity of the individual models the same way as ensemble learning. A representative approach is particle variational inference (PVT), which uses an ensemble of models as an empirical approximation for the posterior distribution. PVI iteratively updates each model with repulsion force to ensure the diversity of the optimized models. However, despite its promising performance, a theoretical understanding of this repulsion and its association with the generalization ability remains unclear. In this paper, we tackle this problem light of PAC- Bayessan analysis. First, we provide new second-order Jensen inequality, which has the repulsion term based the loss function. Thanks to the repulsion term, it is tighter than the standard Jensen inequality. Then, derive novel generalization error bound and show that it can be reduced by enhancing the diversity of models. Finally, we derive a new PVI that optimizes the generalization error bound directly. Numerical experiments demonstrate that the performance of the proposed PVI compares favorably with existing methods in the experiment.
Introduction
Bayesian model averaging (BMA) has been widely employed for prediction, evaluation of uncertainty, and model selection Bayesian inference. BMA obtained as the expectation of likelihood function by a posterior distribution and thus it contains information of each model drawn from the posterior distribution [21). Since estimating the posterior distribution is computationally difficult in practice, various approximations have been developed to efficiently capture the diversity in the posterior distribution 2
One of these recently proposed approaches involves optimizing a set of models simultaneously with interaction to ensure the diversity of the individual models, similar ensemble learning. One notable empirical
example is particle variational inference (PVI) [17 28). which uses an ensemble approximation for the posterior distribution. Such PVI methods have been widely employed in variational inference owing to their high computational efficiency and flexibility. They iteratively update the individual models and the update equations contain the gradient the likelihood function and the repulsion force that disperses the individual models. Thanks to this repulsion term, the obtained ensemble can appropriately approximate the posterior distribution. When only one model
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

used in PVI, the update equation is equivalent to that of the maximum a posterioni (MAP) estimation. Other methods have been developed apart from PVI, especially for latent variable models, which have introduced regularization to the MAP objective function to enforce the diversity in the ensemble.
notable example of such methods is the determinantal point process (DPP) [29].
Despite successful performances of these methods in practice [22
18
17
28
5 27 a theoretical
understanding of the repulsion forces remains unclear. Some previous studies considered PVI as gradient flow in Wasserstein space with an infinite ensemble size 16 12 and derived the convergence
theory. However, an infinite ensemble size is not a practical assumption and no research has been conducted to analyze the repulsion force related to the generalization.
BMA can be regarded as special type of ensemble learning 26]. and recent work has analyzed the diversity of models in ensemble learning in light of the PAC-Bayesian theory [19]. They reported that the generalization error is reduced by increasing the variance of the predictive distribution. However, the existing posterior approximation methods, such as PVI and DPPs, enhance the diversity with the repulsion of the parameters or models rather than the repulsion of the predictive distribution. We also found that the analysis in a previous work [19] cannot be directly extended to the repulsion of parameters or loss functions (see Appendix F.. In addition, when the variance of the predictive distribution is included in the objective function in the variational inference, the obtained model shows large epistemic uncertainty, which hampers the fitting of each model to the data (see Section 5).
Based on these findings, this study aims to develop a theory that explains the repulsion forces in PVI and DPPs and elucidates the association of the repulsion forces with the generalization error. To address this, we derive the novel second-order Jensen inequality and connect it to the PAC-Bayesian generalization error analysis. Our second-order Jensen inequality includes the information of the variance of loss functions. Thanks to the variance term, our bound is tighter than the standard Jensen inequality. Then, we derive a generalization error bound that includes the repulsion term, which means that enhancing the diversity is necessary to reduce the generalization error. We also show that PVI and DPPs can be derived from our second-order Jensen inequality, and indicate that these methods work well from the perspective of the generalization error. However, since these existing methods do not minimize the generalization error upper bound, there is still room for improvement. In this paper, we propose a new PVI that directly minimize the generalization error upper bound and empirically demonstrate its effectiveness.
Our contributions are summarized as follows:
1. We derive a novel second-order Jensen inequality that inherently includes the variance of loss functions. Thanks to this variance term, our second-order Jensen inequality is tighter than the standard Jensen inequality. We then show that enhancing the diversity is important for reducing the generalization error bound in light of PAC-Bayesian analysis.
2. From second-order Jensen inequality, demonstrate that these methods work well even at a finite ensemble size, since their objective functions includes valid diversity enhancing terms to reduce the generalization error.
derive the existing PVI and DPPs. We
3. We propose new PVI that minimizes the generalization error bound directly. We nu- merically demonstrate that the performance of our PVI compares favorably with existing methods.
2 Background
In this section, we briefly review PVI, DPPs, and PAC-Bayesian analysis.
2.1 Particle variational inference
Assume that training dataset = (#1. .........2.. ..D) is drawn independently from unknown data gener- ating distribution v(z). Our goal is to model u(x) by using a parametrized model Inp(z)6). where 0 O Idd. We express p(D(0) Ed In p(aa())) and assume probability distribution over parameters. In Bayesian inference, we incorporate our prior knowledge or assumptions into prior distribution (0). This is updated to posterior distribution p(0 x (D)0) (O), which incorpo- rates the observation D. Let us consider the approximation of p(0 (D) with (0). We predict new data point by predictive distribution p(x) = (( (). where denotes the expectation. This expectation over the posterior is often called BMA (11]

---

Table 1: Particle variational inference methods. / is an N x N identity matrix and Kig == K(0,, 0,).
Methods SVGD[17]
210)
N و KiyOb, log log ( (TT))(()()()) + do Kis 80, Ky
ΣN to, Kzi
W-SGLD
0e, log (Og((D)))))()) Σド= Ii= KG*
+ Σx== Kik
L5
Without the second term in w-SGLD
GFSF
Oa, log p(D) (( x Σラ ((K + cl) +c)===))/x))) ),, Kag
Assume that we draw N models from the posterior distribution and calculate BMA. We denote those drawn models distribution that has a mass at 0,. We also refer to these N models as particles. The simplest approach to obtain these N particles is MAP estimate that updates parameters independently with gradient
empirical distribution pe( t Σ òe, (0), where óg, (0) is the Dirac
descent (GD) as follows 28 :
guew f- Bold + n00g In p( p ) (B:1)))
(1)
where 7) E R+ is a step size. In BMA, we are often interested in the multi-modal information of the posterior distribution. In such a case, MAP estimate is not sufficient because N optimized particles do not necessarily capture the appropriate diversity of the posterior distribution. Instead, particle variational inference (PVI) [7 [8] approximates the posterior through iteratively updating the empirical distribution by interacting them with each other:
0pee t 0
To(((()))) ((
where u( v(())) is the update direction and explicit expressions are summarized Table | Basically, v is composed of the gradient term and the repulsion term. In Table 1 the repulsion terms contain the derivative of the kernel function K, and the Gaussian kernel 23 is commonly used. When the band-
width of K is h, the repulsion term is expressed as 0g (0,, 02) =- (0;-00,e - ((6k)) -20-0) 0,
where || || denotes the Euclidean norm. We refer to this as a parameter repulsion. Note that the repulsion term depends on the distance between particles, and the closer they are, the stronger force applied. This moves Ag away from f). and thus particles tend not to collapse to a single mode.
For over peer-paamettized models such as neural networks, since the repulsion in the parameter space is not enough for enhancing the diversity, function space repulsion force for supervised tasks was developed (28]. We call it function space PVI (f-PVI). Pairs of input-output data are expressed (x; where [(;; 0) is dimensional output function parametrized by 0 and x is an input. Furthermore, we consider the distribution over f and approximate it by size-N ensemble of f, which means that we prepare N parameters (particles) {0: (e We define fi(r) == f(z; 0z). When we input the minibatch with size b into the model, we express it as f(( = (f.(z),....... (((E))))..(.. E Rob Then the update equation is given
DD {<r. ]a) We consider the model p(y|or, 0)
(f1()
Opew <- Gold
(3)
00,
0.
where v((fe)) is obtained by replacing In (D(0))=() with (D/b) Ed Inp(yalfi(aa) (Si). where x xf() is a prior distribution over f and the Gram matrix K0,, 02) is replaced with K K(J, 19) in Table [ See appendix C.I for details. Then, f-PVI modifies the loss signal so that models are diverse. We refer the repulsion term of f-PVI as a model repulsion. We express f f(( for simplicity. When we use the Gaussian kernel, the model repulsion is expressed as
On, KUU 13) -h h- - f3) 1st 1211))(2))))))))))))))))))))))) 00.
(4)
Thus, the model repulsion pushes model fa away from fj-
2.2 Regularization based methods and determinantal point processes
Another common approach for enhancing the diversity for latent variable models is based on regular- ization. A famous example is the determinantal point process (DPP) (2). in which we maximize
Epg In p(D () =z (0) + In det
(5)
where K the kernel Gram matrix defined by KJ K(0,,0,) This log-determinant term essentially repulsion term that enhances the diversity in the parameter space.
3

---

2.3
PAC-Bayesian theory
Here, we introduce PAC-Bayesian theory
7 We define the generalization error as the cross-entropy:
CE = En))] In (()))())))
which corresponds to the Kullback-Leibler (KL) divergence. Our goal is to find (()) that minimizes the above CE. In many Bayesian settings, often minimize not CE but surrogate loss [9 that is obtained by the Jensen inequality:
Ex(x) - [x)))(2()0] S -0
Since the data generating distribution is unknown, we approximate with training dataset as Ex(z))a(o)]= l= In p(pp((())))) [0)] se Egt 1o E p( The PAC-Bayesian generalization error anal- ysis provides the probabilistic relation for this approximation as follows:
Theorem 1 (Germain 77) For any prior distribution a over θ independent of 'D and for any E (0, 1) and c with probability least & over the choice of training data D (&D(x),
for all probability distributions q over O we have
(- s (00
1
Σ
D
I-
KL(g.z) x) Inn V.. Vc, In p(a))))))
D
cD
d=1
where v (c, D) = In F. T. D exp lng E Eedillnpe Inp(c.10))-
The Bayesian posterior is the minimizer of the right-hand side of the PAC- Bayesaan bound when
1. Recently, the PAC-Bayesian bound has been extended so that it includes the diversity term Under the same assumptions as Theorem and for all xr, 0, p(=(0) < co, we have
19
[ + V(ra)) KL(g x) Ing
CE -PESa (( [(r)] s -B, D
eD
where V(x)= := (2 maxy p(z ((z))))))''Enp [(p(=|0) (x(0) is the variance of the predictive distribution and V W (c. D) is the modified constant of V (c, D) (see Appendix C.2 for details). A similar bound for ensemble learning, that is. q(0) as empirical distribution, was also previously proposed [19 (see Appendix C.3). This bound was derived directly from the second-order Jensen inequality derived in another work [14]. Furthermore, the diversity comes from the variance of the predictive distribution, which different from PVI and DPPs because their repulsion in the parameter model space. Note that we cannot directly change the variance of the predictive distribution to that of PVI or DPPs because requires an inequality that contrary to the Jensen inequality. We also found that directly optimizing the upper bound Eq. referred to as PAC results in a too large variance of the predictive distribution which is too pessimistic for supervised learning tasks (see Section 5).
Method
Here, we derive our novel second-order Jensen inequality based on the variance loss functions and then derive a generalization enror bound. Then, we connect our theory with existing PVI and DPPs.
3.1 A novel second-order Jensen inequality
First, we show the second-order equality, from which we derive our second-order Jensen inequality.
Theorem 2. Let be a twice differentiable monotonically increasing concave function on R+ be random variable on R+ that satisfies E| < cc, and its probubility density be (z(2). Define constant μ = 3 (0 (Z))) Then, we have
dob(()) dz
[dPp(e(z)) [ dz²
E μ
12² pz(2)dz,
(10)
where e(2) is a constant between z and jr that is defined from the Taylor expansion (see Appendix p.1 for details).

---

Proof sketch: There exists a constante(z) between und.t =f=)== +()))+((++++)) d (a (3- / n)²
from the Taylor expansion. Then we take the expectation. Full proof given Appendix D.
This theorem states the deviation of EZ from j when d is applied to Z. By setting (b()) - In(-) and ==p(x (0) and applying In to both hand sides of Eq.(TO) we have the following equality:
Corollary 1. for all a and 6, p(x=0) < oo, we have
Ey0 In p(=)0) - (e (0)) - In (1 + (0o (2g(0, (e - Be -n 1oplel (11)
where g(0,x) x) is constant berween p(x (0) and Fuco In p(x))))) that is defined from the Taylor expansion (see Appendix(D.2 details).
Remark 1. Recall that the standard Jensen inequality is Eg In p(x[0) IIn Exp(a and its gap is called the Jensen gap. In [I), the second term of the right-hane side is always positive. Thus, this term corresponds to the Jensen gap. Remarkably when we use the standard Jensen inequality, this information is lost. We clarify the meaning of this term below. Also note that our second-order equality is different from those of the previous works 19 14 (see Appendix[Figg details).
Next, we show our first main result, loss function based second-order Jensen inequality:
Theorem 3. Under the same assumption as Corollary L
By0 In ()) S In Ext0)p(=10) Ey(0) Imp(x - Eyco) Inp(a100)
(12)
2h(x, 0)
R(a, h))
where Mxx00)22 exp [mp((=10) + Eg(0) In (mp((00) - 2 max In p(x))))
(13)
Proof sketch: Apply vaBs
a- for any a, Oto Eq. Full proof given in Appendix D.3
In
Remark 2. is the weighted variance of loss functions, and it always positive. Thus, this inequality is always tighter than the Jensen inequality, and the equality holds if the weighted variance is zero. Compared to the results of the previous works [19] [4] that used the predictive variance in the inequality, bound focuses on the variance of loss functions. We refer to repulsion term R as a loss repulsion.
Then, by rearranging Eq. 12 and taking the expectation, we have the following inequality:
CE <
(e 0)] (a)) R(x, h) < ((())) [[x]]]]
(14)
Using this inequality, obtain the second-order PAC-Bayesian generalization error bound:
Theorem 4. (See Appendix(D.4e for the complete statement) Under the same notation and assumptions
as Theorems 7 and with probability at least 1 - -5, we have
Σ [Inp(aa0)+ R((a,hh))) KL(g. d=1
CE CE (eu--E
(15)
cD
where T (c, D) is the modified constant of Vr D) and R(x,hm) is R(x,h) in Eq. maxeln p( ()).
12 replacing
h(x, 0 of Eq. 13) with (x, == p( (0) mwnglinnmmmiiiieemmmmmm
where (2hm (z. - (In p(x 22 p(=(0) In (mxz(()) p (8)) In p()(0")) Then, we apply the same proof technique as Theorem 1 [ to the loss function 0( 8²") with
Proof sketch. We express Byee) [In p(x R(x,h))) as (ete)
[(x,0,0'0") :n
= 3cD. Full proof is given in Appendix D.4
Remark 3. To reduce the upper bound of the generalization error, Eq. (5). we need to control the trade-off between the data fitting term of the negative log- nikkeelldood and enhancing the diversity of the models based on the loss repulsion term R.
Remark 4. In the defimition of him mingln p(z](0) is too pessimistic in some cases. we additionally assume that there exists a positive constant s.I. Eggoy [In ((]]))) < M < oc, we can replace
mingln p( (0) with Mediang (In p(z(0)) M in hm in Theorem 4 (see Appendix D.4.1)
5

---

Compared to Eq.9), our bound focuses on the variance of loss functions, which has direct connection the repulsion PVI and DPPs (see Section 3.2). Furthermore, we show that optimizing the upper bound in Eq. (5) shows competitive performance with the existing state-of-the-arr PVI (see Section S Note that this inequality not restricted to the case where g(0) an empirical distribution. We can also use this for parametric variational inference [21]
3.2 Diversity in ensemble learning and connection to existing methods
In the following, we focus on the ensemble setting and use a finite set of parameters as pr(() = N Dim 1 0g, (0) and discuss the relationship of our theory and existing methods. We show the summary of the relationships in Appendix
I
3.2.1 Covariance form of the loss repulsion
To emphasize the repulsion between models, we upper-bound Eq. (2) using the covariance:
Theorem 5. Under the same assumption as Corollary 7 we have
E (0) Imp(x < In po 2(2hm 02
Σ
(Inp(210,) - Inp(x10,)) (16)
where uw (x, 0)- exp (min, In ((x)0.) 1n=11im(x// - max, lnp(z)0,))
See Appendix D.5 for the proof. We can also show generalization error bound like Theorem for the ensemble learning setting (see Appendix E.I). In existing PVI and DPPs, the repulsion is based
not on the loss function but the parameters or models, as seen in Section 2 We derive the direct connection between our loss repulsion and the model and parameter repulsion below.
3.2.2 Relation to w-SGLD and model repulsion
First, from Eq. [6). we derive the direct connection w-SGLD, which is an instance of PVI introduced in Section Let us define an N kernel Gram matrix G whose (i,j) element is defined as
G₁ = exp (8hh (Inp(x)0,) - Inp((=))))))
(17)
Applying the Jensen inequality to [6. we obtain
N
N
G N
In (p(()))()(0)) 2 (e 1np(2
In
2 E p In pnp(0)))
(18)
N
This is tighter than the standard Jensen inequality. To derive the relation to w-SGLD optimize the middle part of T8) by gradient descent. We express T.(O,) == In p(>p (.)) and do not consider the dependency of and for simplicity. By taking the partial derivative with respect 0,, have
Gij EE Gjı
ΣSe1 OL(Os) Gji
0o In (()))))))
Oo. I(O.)-
(19)
Σn Ga
The second term corresponds to the repulsion term, which is equivalent to that of w-SGLD shown in Ta The difference that our Gram matrix G in Eq.419 depends on the loss function rather than the parameter model. Using the mean value theorem, it is casy to verify that there exists constant C' such that || In ((x - In p(x(0,))) 10) = | CC(,, - 0,)l| (see Appendix for details), thus we can easily transform the loss repulsion to the parameter model repulsion.
However, since we cannot obtain the explicit expression of the constant C, is difficult understand the intuitive relation between our loss repulsion and the parameter or model repulsion. Instead, here we directly calculate Og Gt, and discuss the relation. Due to the space limitation, we only show the relation to the model repulsion in the regression task of f-PVI. See Appendix G for the complete statement including the classification setting f-PVI and the parameter repulsion of PVI. Following the setting in Section 2.1 for regression problem, we assume that f(y).(x;; 0)) is the

---

Gaussian distribution with unit variance for simplicity. We define L(f.) == (n and dLip := 0z (,, The derivative of the Gram matrix G is expressed as
0o, Gig ((f - fi)lddGbll 112 + +0, L.i -f3ll foll) GG,
(20)
The first term i) corresponds to the model repulsion of f-PVI shown in Eq. (30 and the second term ti) is the correction term based on the loss function. Thus, our loss repulsion can be translated the model repulsion of f-PVI plus the correction term using the loss function.
In conclusion, we have confirmed that w-SGLD is directly related to our theory. For Eq. [q.188... we can also derive a generalization eгrоr bound like Theorem 4 (see Appendix E.2 for details). This means that w-SGLD controls the trade-off between the model fitting and enhancing the diversity by optimizing the generalization error bound. This explains the reason why w-SGLD still works well even with finite ensemble size.
3.2.3 Relation to other PVIs
Here, we derive other PVI and DPPs from our theory. First, we derive GFSF shown in Table
I We
express the identity matrix with size N as I. We obtain the following upper bound from Eq. T6:
Theorem 6. Under the same assumption as Corollary 7 for any real value E (1, N1 1) and for any positive conslant h we have
2 de K) + 2nn N hN
In E. (0))(((()))) 2 R (6) In lnp(=
2 PeeC In pnp()))),
(21)
hN where K is an N x N kernel Gram matrix of which (i,j) element is defined as
Ky == exp (- N(4h.))) (In p(x: ((x1)))))²
(22)
The proof is shown in Appendix D.6 This is tighter than the standard Jensen inequality.
Remark& 5. Eq. (T) holds for any positive bandwidth h. h defines the tightness Eq. (2I)
Remark 6. In Eq. (2). used the scaling of In N to define K. This is motivated by the median trick of the existing PVI, which tunes the bandwidth as In N m/mmiiann..
2 This scaling implies that, for each i, Ky R 1 holds. We We found that using this scaling is necessary to obtain the bound Eq. (T). We conjecture that this the reason why sealing the bandwidth is important for PVI in practice.
In the same way as Eq. for w-SGLD, we also optimize the middle term in Eq.21 21 by GD. By taking the partial derivative, we have the following update equation:
(p log p()10,))
2 AN
Σ(K+ ccij ' r(, Ki, Ko L(O.).
(23)
See appendix D.7 for the proof. The second term is the repulsion force, which is the same as that in the update equation of GFSF in Table
Next we consider the relation to DPPs. Using the trace inequality g to 16 and using a Gram
matrix G whose (i.j) element G in Eq.(T7) we obtain
2 In dedG N N
In Epp,((1))) > Epp Inp(x10) +
(24)
The proof is shown in Appendix D.8 The lower bound term of Eq 24 is equivalent to the objective function of the DPP introduced in (5)
For Eqs. 21 and (24). we can derive PAC- PAC-Bayesian generalization error bound like Theorem
(see Appendix E.2 for details). Moreover, we can connect our loss repulsion in Eqs.(2) (1 and (24)
to model and parameter repulsions in the same way w-SGLD. Accordingly, GFSF and DPP closely related to the recond-order generalization error bound.
In conclusion, we derived PVI and DPPs from our theory based on the second-order Jensen inequality. On the other hand, we found it difficult to show its relation SVGD, since has the kernel smoothing coefficient in the derivative of the log- likeliill function. We leave it for future work.
7

---

Discussion and Related work
In this section, we discuss the relationship between our theory and existing work.
4.1 Theoretical analysis of PVI
Originally, PVI was derived as an approximation of the gradient flow in Wasserstein space and its theoretical analysis has been done with an infinite ensemble size [16] [2]. In practice, however, an infinite ensemble size is not realistic and various numerical experiments showed that PVI still works well even with a finite ensemble size [17 28). Accordingly, our analysis aimed to clarify why PVI works well with such a finite-size ensembles. On the other hand, as discussed Section 3.2.2] there is a difference between our loss repulsion and the parameter and model repulsion used in the existing works, and thus it would be an interesting direction to extend our theory to fill such a gap.
4.2 Relation to existing second-order Jensen inequality
Liao and Berg 14 derived a second-order Jensen inequality, and Gao et al. [6] worked on higher-order inequality. Masegosa 19] combined the earlier result 14 with the PAC-Bayesian theory. Although our bound is also a second-order Jensen inequality, its derivation and behavior are completely different from them. In
Recently, some works derived tighter Jensen inequalities
14 6
Liao [[4,, their second-order Jensen inequality includes the term variance of a random variable, and Masagosa [19] considered p(x(0) to be corresponding random variable that depends on 0. Thus, the second-order inequalities depend on the variance of the predictive distribution Epto) [p(x [)]] On the other hand, our bound is based which leads to the variance loss function as shown in Theorem 3 By using the variance of loss funetions,
novel second-order equality shown in Theorem 2
directly connect theory to existing PVI and DPPs as shown in Section 3.2 Moreover, as shown in Section including the predictive variance in the objective function results in large epistemic uncertainty. which means that individual models do not fit well. On the other hand, ours does not show this phenomenon. Consequently, our result can be regarded as an extension of the earlier work [4
19 that directly focuses loss function in machine learning.
Masagosa 19 showed that the second-order PAC-Bayesiin generalization error is especially useful under misspecified models, i.e., for any 0, p(x10) & p(r). Our theories can also be extended to such a setting (see Appendix H for further discussion).
Other closely related work is an analysis of the weighted majority vote in multiclass classification 20). which uses a second-order PAC-Bayesian bound. While their analysis specific to the majority vote of multiclass classification, our analysis has been carried out in more general setting based on the second-order Jensen inequality derived from Theorem 2 and Theorem
5 Numerical experiments
According to our Theorem 4 is important to control the trade-off between the model fitting and diversity enhancement in order reduce the generalization error. Therefore, minimize generalization error bound Eq. (15) directly and confirm that the trade-off is controlled appropriately. Our objective function is
N D
F((0,) =
In pxx R(xa.h)) + KL(PE, T),
(25)
N
d=
which is obtained setting (0) (0( and c= 1 in Eq. (5). We call our approach Variance regularization, and express as VAR. We compared the performances of our methods with MAP, PAC and f-PVI(f-VGDD) on toy data and real data. Our experimental settings are almost same as that of the previous work (8 and the detailed settings and additional results are shown in Appendix
As for the KL divergence, since Pe is an empirical measure, we need to carefully define a prior distribution so that the KI. divergence between /E and и can be defined properly. We used certain prior distribution as introduced previously [19] (see Appendix C.3 for details). Moreover, to climinate such a limitation, we considered smoothing the gradient using SVGD. We call this approach
8

---

MAP(Ereemi
SVGD
FSVGD
WRRIM)Piousee)
0-
-35
DACK
(Pheoon)))
Figure 1: Uncertainty of the regressions. Blue curve is the predictive mean, light shaded area is the predictive cretible interval and dark shaded area is the credible intervals for the mean estimate respectively.
VAR-SVGD, which update equation given as
N Σ 1=1
Brew t 8ohl + 17
Kopa (log (x + R(D, + Ogpii Kij.
(26)
N
where R(D, h1) = Ed= R(xa, 1t) and Kúj is the Gram matrix using the Gaussian kernel defined similarly to the model repulsion in Section 2.1 We tuned its bandwidth by the median trick.
5.1 Toy data experiments
First, using toy data, we visualized the model fitting and diversity in various methods. We considered regression task and we randomly generated the I-dimensional data points and fit them by using feed-forward neural network model with 2 hidden layers and ReLU activation, which has 50 units. used 50 ensembles for each method. The results are shown in Figure which visualizes 95% credible intervals for the prediction and mean estimate corresponding to aleatoric and epistemic uncertaintics.
Hamilton Monte Carlo (HMC) is the bascline method used to express the Bayesian predictive uncertainty properly. MAP and SVGD methods give an evaluation the uncertainty that too small. Our method and f-SVGD showed very similar evaluations for uncertainty. PACE [19] showed large epistemic uncertainty, which is expressed as dark shaded area in Figure [ PACE includes the predictive variance the objective function and the enhanced diversity is too large. Described below, PACE shows slightly worse results real data experiments than those of other approaches. This might be because including the predictive variance in the objective function does not result in a better trade-off of between model fitting and enhancing diversity in practice.
We conjecture it is because
5.2 Regression task on UCI
We did regression tasks on the UCI dataset The model is single-layer network with ReLU activation and 50 hidden units except for Protein data, which has 100 units. We used 20 ensembles. Results of 20 repetition are shown in Table We found that our method compares favorably with f-SVGD. We also found that PACE shows worse performance than those of other methods. We conjectured that this is because the predictive variance in the objective function enhances too large diversity as shown in Figure which hampers the fitting of each model the data.
5.3 Classification task on MNIST and CIFAR
We conducted numerical experiments on MNIST and CIFAR 10 datasets. For MNIST, we used a feed- forward network having two hidden layers with 400 units and ReL. activation function and used 10 ensembles. For CIFAR 10, used ResNet-32 19, and used 6 ensembles. The results are shown

---

Table 2: Benchmark results test RMSE and negative log likelihood for the regression task.
Avg. Test RMSE
Avg.
Test negative log likelihood
Dataset
MAP
PACE
[-SVGD
VAR
VAR-SVGD
MAP
PACE
r-SVGD
VAR
VAR-SVGD
Concrete
5.19:003
549220.3
4.32:0.1
433==0
4.35:002
3.11:0.12
3.1620.00
2.560002
2:83:0000
2.00:0066
Bostoa
2.98:0.4
400-005
254
254=0.3
2.52:0.3
262=022
261
2.6200.1
2.39=02
2.35-022
Wine
0.65:10.11
1:00:10009
0,61-0000
06110001
0611:0.03
0.7.10.07
12610.00
0.90
0.8910.04
0.89
Power
3.44:000
5.012021
3:77=000
3.7660003
2.79:00005
3.170001
276=0006
2.79:0003
2.7::003
Yachi
0.86.10.0
0.70.1021
0.59_00.9
0.5910005
0.58..0.12
1.0000005
0.00.0.4
0.66.03
0.7_003
081102
Protein
4.25:00.7
4.17::0.05
3,98-003
3,95:0005
3.93:0007
2,95:00000
2.84:0001
2,80
2.1:0001
2.00:0.01
Table 3: Benchmark results on test accuracy and negative log likelihood for the classification task.
Test Accuracy
Test log likelihood
Dataset
MAP
PACE
I-SVGD
VAR
VAR-SVGD
MAP
PACE
F-SVGD
VAR
VAR-SVGD
MNIST
0.981
0.986
0.987
0.988
0.988
0.057
0.042
0.043
0.040
0.041
CIFAR 10
0.935
0,919
0.927
0.928
0.928
0.215
0.270
0.241
0.238
0.240
Table 4: Cumulative regret relative to that of the uniform sampling.
Dataset
MAP
PACE
F-SVGD
VAR
VAR-SVGD
0.129:1.098
0.037:0:012
0.029:10.010
0.03700.012
Financial
0.79110.2119
0.189:0.025
0.15410.017
0.155:10.224
0.176:0.023
Statlog
0.675 +:02877
0.052=0.0025
0.01020.8003
0.006
0.0070000004
CoverType
0.610::0.011
0.396:00.066
0.372:40007
0.289
0.320=0.000
in Table For both datasets, our methods show competitive performance compared to f-SVGD. For CIFAR TO, as reported previously (28]. f-SVGD worse than the simple ensemble approach. We also evaluated the diversity enhancing property by using the out of distribution performance test. is hypothesized that Bayesian models to capture uncertainty. Thus, we generated attack samples and measured the vulnerability to those samples. We found that, shown in Figure in Appendix [] our method and f-SVGD showed more robustness compared to MAP estimation in each experiment.
robust against adversarial examples due their ability
5.4 Contextual bandit by neural networks on the real data set
Finally, we evaluated the uncertainty of the obtained models using contextual bandit problems [24]. This problem requires the algorithm to balance the trade-off between the exploitation and exploration, and poorly evaluated uncertainty results in larger cumulative regret. We consider the Thompson sampling algorithm with Bayesian neural networks having 2 hidden layers and 100 ReLU units, and used 20 particles for each experiment. Results of 10 repetition shown in Table 4 We can that our approach outperform other methods.
Conclusion
In this work, we derived a novel second-order Jensen inequality that includes the variance of loss functions. We also derived a PAC-Bayesian generalization error bound. Our error bound shows that both model fitting and enhancing diversity are important for reducing the generalization error. Then, we derived the existing PVI and DPPs from our new Jensen inequality. Finally, we proposed a new PVI that directly minimizes current state-of-thes PVI. In future work, it would be interesting to apply our second-order Jensen inequality to general variational inference optimal control problems.
PAC-Bayesian bound. It shows competitive performance with the
Other interesting direction to derive an upper-bous bound of the Jensen gap. Gao et al. 6 derived In Ep(x: ()- E In p(z(() < Variance(p( (0)) which uses the predictive distribution. This means that the larger the predictive variance is, the larger upper bound of the Jensen gap we have. We leave it for future work to upper-bound the Jensen gap with the variance of loss function using our theory.
Acknowledgments and Disclosure of Funding
FF was supported by JST ACT-X Grant Number JPMIAX190R.
10

---

References
[1] Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
[2] David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: review for statisticians. Journal of the American Statistical Association, 1 11((518):859 877, 2017.
[3] Changyou Chen, Ruiyi Zhang, Wenlin Wang, Bai Li, and Liqun Chen. unified particle- optimization framework for scalable bayesian sampling. arXiv preprint 1805./ 2018.
[4] Dua Dheeru and Efi Karra Tamiskidou. UCI machine learning repository. 2017.
[5] Mohamed Elfeki, Camille Couprie, Morgane Riviere, and Mohamed Elhoseiny. GDPP: Learning diverse generations using determinantal point processes. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Leurning Research, pages 1774-1783. PMLR, 09-15 Jun 2019.
[6] Xiang Gao, Meera Sitharam, and Adrian E Roitberg. Bounds on the jensen gap, and implications for mean-concentraated distributions. arXiv preprint arXiv: 2017.
[7] Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien Pac-bayesian theory meets bayesian inference. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 1884-1892, 2016.
[8] Gene H Golub and Charles F Van Loan. Matrix computations, volume 3. JHU press, 2012.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[10] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[11] Chin-Wa Wei Huang and Aaron Courville. Note on the bias and variance of variational inference, 2019.
[12] Anna Korba, Adil Salim, Michael Arbel, Giulia Luise, and Arthur Gretton. A non-asymptotic analysis for stein variational gradient descent. Advances in Neural Information Processing Systems, 33, 2020.
[13] Yingzhen Li and Richard E. Turner Gradient estimators for implicit models. In International Conference on Learning Representations, 2018
[14] JG Liao and Arthur Berg. Sharpening jensen's inequality. The American Statistician, 2018
[15] Chang Liu, Jingwei Zhuo, Pengyu Cheng, Ruiyi Zhang, and Jun Zhu. Understanding and accelerating particle-based variational inference. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4082-4092, Long Beach, California, USA, 09-15 Jun 2019, PMLR.
[16] Qiang Liu. Stein variational gradient descent as gradient flow. In Advances in neural information processing systems, pages 3118-3126 2017.
[17] Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances In Neural Information Processing Systems, pages 2378-22 2378-2386, 2016.
[18] Yang Liu, Prajit Ramachandran, Qiang Liu, and Jian Peng. Stein variational policy gradient. In 33rd Conference on Uncertainty in Artificial Intelligence, UAI 2017, 2017.
[19] Andres Masegosa. Learning under model misspecification: Applications to variational and ensemble methods. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5479-5491. Curran Associates, Inc., 2020.

---

[20] Andres Masegosa, Stephan Lorenzen, Christian Igel, and Yevgeny Seldin. Second order pac- bayesian bounds for the weighted majority vote. Advances in Neural Information Processing Systems, 33, 2020.
[21] Kevin P Murphy, Machine learning: probabillistic perspective. 2012.
[22] Yunchen Pu, Zhe Gan, Ricardo Henao, Chunyuan Li, Shaobo Han, and Lawrence Carin. Vae learning via stein variational gradient descent. In Proceedings of the 31st International Conference Neural Information Processing Systems, pages 4239-4248, 2017.
[23] Carl Edward Rasmussen and Christopher KI Williams. Gaussian processss for machine learning, volume 1. MIT press Cambridge, 2006.
[24] Carlos Riquelme, George Tucker, and Jasper Snock. Deep bayesian bandits showdown: An empirical comparison bayesian deep networks for thompson sampling. In International Conference Learning Representations, 2018.
[25] Jiaxin Shi, Shengyang Sun, and Jun Zhu. Kernel implicit variational inference. arXiv preprini arXiv: 101 19, 2017.
[26] E Ke Tang, Ponnuthurai N Suganthan, and Xin Yao. An analysis of diversity measures. Machine learning, 65(1):247 271, 2006.
[27] Dilin Wang and Qiang Liu. Nonlinear stein variational gradient descent for learning diversified mixture models. In Kamalika Chaudhuri and Ruslan Salakhutdinov. editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages
PMLR, 09-15 Jun 2019.
[28] Ziyu Wang, Tongzheng Ren, Jun Zhu, and Bo Zhang, Function space particle optimization for bayesian neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[29] James Yang Zou and Ryan Prescott Adams, Priors for diversity in generative latent variable models. 2012.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section |
(b) Did you describe the limitations of your work? [Yes] See Appendix A
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Our paper conforms to the ethics review guidelines.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See each theorem in the paper.
(b) Did you include complete proofs of all theoretical results? [Yes] See each theorem in the paper.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URL)? [Yes] See the supplemental code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix and the supplemental code.
(c) Did you report error bars (e-g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Appendix and the supplemental code.
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix | and the supplemental code.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 5 Appendix[[ and the supplemental code.
(b) Did you mention the license of the assets? [Yes] See the supplemental code.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "SqFk38deRM1", "reviewer": "Reviewer_ELqV", "summary": "This paper derives a tighter Jensen Inequality and applied it into the framework of particle variational inference. Different from previous work, it tackles this problem in light of PAC-Bayesian analysis, and provides a new second-order Jensen inequality, which has the repulsion term based on the loss function. In addition, the authors derive the existing PVI and DPPs and demonstrate that these methods work well even at a finite ensemble size, since their objective functions includes valid diversity enhancing terms to reduce the generalization error. Finally, the numeral experiments show the better performance than the most related work PAC2.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and the last figure in the supplementary clearly illustrates the main idea of the loss function based second-order Jensen inequalities.\n\nThe theoretical analysis is sound to me, even I didn't check the detailed proof in the supplementary materials. The idea seems to be enlighten by [16], but the methods and mathematical tools used in this paper are different. It also builds the connection to the existing methods such as w-SGLD. From the theoretical perspective, this is good paper and should be interested to the area of probabilistic inference. However, the overall contribution seems incremental based on the previous works Stein variational gradient descent (SVGD) and \"Function Space Particle Optimization for Bayesian Neural Networks\". Basically, it is still a simple variant of SVGD. Besides, I would like to see the performance on generative or energy models. The regression or classification tasks are too simple to see the real improvement over SVGD.\n\nA typo Line 221: $\\ln p(x|\\theta_i)$ -> $\\ln p(x|\\theta_j)$", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "GFxd1WO1JF", "reviewer": "Reviewer_LgZb", "summary": "This paper studies particle variational inference (PVI), which is a mix between ensemble and VI (the variational posterior is assumed to be a mixture of Dirac distributions, and an additional \"repulsion term\" is added to give better diversity), and proposes a novel generalization to Jensen's inequality. From here, the authors propose an upper bound to the first term of the ELBO (the expected log-likelihood term), which is tighter compared to the standard Jensen inequality. Empirical findings show that the proposed loss performs well in various cases.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is very well written. The proposed method is soundly and rigorously justified. I think this is a good and solid paper. \n\nNevertheless, one criticism from me is that the authors did not discuss the comparison of their method (i.e. their loss in eq. 25) with the standard ELBO objective. From eq. 25 it is clear that without the term $R(x_d, h)$, this objective reverts back to the standard ELBO with $q(\\theta) := 1/K \\sum_{k=1}^K \\delta(\\theta_k)$. But, when the $R(x_d, h)$ term is kept in, how valid is it to consider eq. 25 as an ELBO, and thus the proposed method as a VI method? I hope to see a discussion from the authors in the rebuttal period.\n\n**Minor comments**\n\n* Thm. 1: \n  * Do this instead: \\begin{theorem}[\\citeauthor, \\citeyear]\n  * The statement is very hard to parse. Please consider rephrasing it\n  * The notation $\\nu^{\\otimes D}$ is undefined \n* All proofs: the qed symbols should be flush-right\n* L.197: What does it mean by \"$q(\\theta)$ is a set of parameters\"\n* L.212: \"a kind of PVI\" -> \"an instance of PVI\"\n* Fig. 1:\n  * \"Blue line\" -> \"Blue curve\"\n  * (Also L.316) How do \"mean estimate\" differ from the predictive mean?\n* L.326, among other places: \"ReLu\" -> \"ReLU\"\n* Tab. 2, 3, 4: Inconsistency with the bolding. I.e. Tab. 2 and Tab. 3 also need some bold faces\n* Tab. 2: \"Mnist\", \"Cifar\" -> \"MNIST\", \"CIFAR\"\n* L.359: \"Variacne\" -> \"Variance\" or just \"Var\"\n\n\n---\n\n**Post-Rebuttal**\n\nAfter reading the authors' rebuttal and other reviews, I decided to keep recommending an acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed both limitations and societal impacts. However, they do so in the appendix. The limitations should instead be discussed in the main text.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "75dFuTt_jFv", "reviewer": "Reviewer_hCAh", "summary": "The paper presents a tighter PAC-Bayes bound based on the second-order Jensen inequality. Remarkably, minimizing the new bound under various relaxations/approximations recovers various PVIs and imposing a DPP prior. This novel view of PVIs explains why their repulsive force helps generalization (also, this is a valuable finite-sample regime analysis of PVIs), and also inspires new algorithms for PVIs. Experiment verifies the utility and the benefits of the new methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* I much appreciate the idea and its originality. I didn't expect a theory as fundamental as the PAC-Bayes analysis could explain specific algorithms for the Bayesian posterior. It is exciting if it is possible for an analysis at this level, as PAC-Bayes is the basic theory showing the generalization of Bayesian methods while PVIs are practical algorithms realizing the Bayesian approach, so such an analysis reveals PVIs have a theoretical root and verifies the generalization benefits of PVIs from the root (remarkably, this is a finite-particle analysis).  \nThe analysis is based on a finer development of PAC-Bayes bound, which is also novel.\n* The paper reads clear and enjoying, although it involved heavy equations. The meaning of complicated math expressions are well explained. I did not find problems regarding math deduction.\n\nCons:\n* I find my major concern is on the proposed new PVI, presented at the beginning of Sec. 5.\n  - Regarding the KL term in the objective Eq. (25), it is inappropriate to directly plug-in the empirical distribution $\\rho_E$. KL would be undefined (or at least, infinite) if the first distribution is not absolutely continuous w.r.t the second distribution, which is the case for this $\\rho_E$ and a common choice of $\\pi$ (e.g., a standard Gaussian). From another perspective, if the entropy term of $\\rho_E$ in the KL is treated as a constant, then minimizing this KL would collapse all particles onto the modes of $\\pi$, which is not desired for diversity. This is also the reason why PVIs cannot be derived by treating the particles as the parameters of the variational distribution and minimizing the -ELBO objective (Interestingly, Eq. (25) is in the form of an ELBO!), and why kernels are introduced for smoothing the empirical distribution [Liu et al., 2019].\n  - I did not find how $h$ and $h_m$ are calculated in the algorithm. I suppose they are estimated directly using their definitions, but this then involves a likelihood-model $p(x|\\theta)$ optimization for every data sample $x$ in every iteration. Will it be too expensive?\n* Literature inconsistency. In Table 1, both GFSD and GFSF are originated from [Liu et al., 2019]. Also, GFSD is not the sum of SVGD and w-SGLD. It is w-SGLD without the second term.\n* Minor issues.\n  - Notations. In Eq. (2), $v$ should also depend on the specific particle $\\theta_i$. So it'd be in the form $v(\\theta_i, \\{ \\theta_{i'}^{old} \\}_{i'=1}^N)$ (I cannot type the curly bracket). In Line 110, $p(x_d | \\theta)$ may be also needed to adapt to e.g. $p(y_d | f, x_d)$. Would be better to unify notations of $\\mathbb{E} Z$ and $\\mathbb{E} [Z]$.\n  - The paper mentioned that there is not finite-particle analysis before. But it seems that [11] showed an analysis in this case.\n  - Possible typos. Line 214: there may not be \"to\". Line 359: \"Variacne\".\n  - Discussion. The paper showed the connection to the median method for bandwidth selection. The authors may also consider the connection to the heat-equation bandwidth selection method by [Liu et al., 2019] based on an equivalent dynamics principle, which achieves a better performance than the median method.\n\n[Liu et al., 2019] \"Understanding and Accelerating Particle-Based Variational Inference\". ICML 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mentioned the limitation in explaining SVGD. I think the computational burden to estimate $h$ and $h_m$ in the proposed algorithms (the second item in \"Cons\") may also be a limitation.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission uses a tighter inequality for the likelihood than Jensen's inequality by a second order expansion.  The additional terms are then interpreted as a diversity term.  A PAC-Bayesian analysis is then developed based on this inequality.  The submission initially received a mix of scores straddling the acceptance threshold.  An external expert was consulted giving a mixed opinion (see copy of comments below), while the lower scoring reviewer was convinced to revise their score upwards during the rebuttal process.  On the balance, the paper is viewed positively, but has some important issues that should be corrected prior to publication.  These relate to incorporating reviewer comments (see below), but importantly to avoid obvious technical inaccuracies such as "MAP estimation can be used to obtain samples from the posterior" (section 2, from expert opinion below).

Additional expert opinion:
"The idea is simple overall, they use a thighter inequality than Jensen For the likelihood by exploiting second order expansion. This results in an additional terms which is interpreted as a diversity term. They combine this bound with a pac-bayesian bound to define a suitable loss for sampling from posterior in a particle fashion. Then they proceed to relate this loss to existing methods in particular w-SGLD and DPP, showing that their bound can result in methods that are formally similar to these prior works I didn’t find the paper particularly nice to read, for instance **some important details raised by reviewer 2 was only discussed in the appendix without mentioning them in the main ** The experiements did not compare with very standard benchmarks such as ELBO or even SMC methods which is also a particle methods. In the text they say that they used to HMC as a baseline, but I couldn’t see the performance of HMC in the results Also some inaccuracies in section 2.1: they claim that MAP estimation can be used to obtain samples from the posterior, but of course this is wrong... this is on the negative side on the positive side: The PAC-bayesian people might be interested in such result as it relies on a tighter bound than Jensen to derive the PAC-bayesian bound. And the connection to other methods is also interesting at a high level, although it is unclear to what extend this connection can say anything specific about w-SGLD and DPP in terms of generalization bound, etc If the paper is accepted, then the remarks by Reviewer 2 should be taken into account and in general the paper should mention in the main text what is going on in the appendix"
Decision: Accept (Poster)