Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper derives a tighter Jensen Inequality and applied it into the framework of particle variational inference. Different from previous work, it tackles this problem in light of PAC-Bayesian analysis, and provides a new second-order Jensen inequality, which has the repulsion term based on the loss function. In addition, the authors derive the existing PVI and DPPs and demonstrate that these methods work well even at a finite ensemble size, since their objective functions includes valid diversity enhancing terms to reduce the generalization error. Finally, the numeral experiments show the better performance than the most related work PAC2.
main_review: The paper is well written and the last figure in the supplementary clearly illustrates the main idea of the loss function based second-order Jensen inequalities.

The theoretical analysis is sound to me, even I didn't check the detailed proof in the supplementary materials. The idea seems to be enlighten by [16], but the methods and mathematical tools used in this paper are different. It also builds the connection to the existing methods such as w-SGLD. From the theoretical perspective, this is good paper and should be interested to the area of probabilistic inference. However, the overall contribution seems incremental based on the previous works Stein variational gradient descent (SVGD) and "Function Space Particle Optimization for Bayesian Neural Networks". Basically, it is still a simple variant of SVGD. Besides, I would like to see the performance on generative or energy models. The regression or classification tasks are too simple to see the real improvement over SVGD.

A typo Line 221: $\ln p(x|\theta_i)$ -> $\ln p(x|\theta_j)$
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a tighter PAC-Bayes bound based on the second-order Jensen inequality. Remarkably, minimizing the new bound under various relaxations/approximations recovers various PVIs and imposing a DPP prior. This novel view of PVIs explains why their repulsive force helps generalization (also, this is a valuable finite-sample regime analysis of PVIs), and also inspires new algorithms for PVIs. Experiment verifies the utility and the benefits of the new methods.
main_review: Pros:
* I much appreciate the idea and its originality. I didn't expect a theory as fundamental as the PAC-Bayes analysis could explain specific algorithms for the Bayesian posterior. It is exciting if it is possible for an analysis at this level, as PAC-Bayes is the basic theory showing the generalization of Bayesian methods while PVIs are practical algorithms realizing the Bayesian approach, so such an analysis reveals PVIs have a theoretical root and verifies the generalization benefits of PVIs from the root (remarkably, this is a finite-particle analysis).  
The analysis is based on a finer development of PAC-Bayes bound, which is also novel.
* The paper reads clear and enjoying, although it involved heavy equations. The meaning of complicated math expressions are well explained. I did not find problems regarding math deduction.

Cons:
* I find my major concern is on the proposed new PVI, presented at the beginning of Sec. 5.
  - Regarding the KL term in the objective Eq. (25), it is inappropriate to directly plug-in the empirical distribution $\rho_E$. KL would be undefined (or at least, infinite) if the first distribution is not absolutely continuous w.r.t the second distribution, which is the case for this $\rho_E$ and a common choice of $\pi$ (e.g., a standard Gaussian). From another perspective, if the entropy term of $\rho_E$ in the KL is treated as a constant, then minimizing this KL would collapse all particles onto the modes of $\pi$, which is not desired for diversity. This is also the reason why PVIs cannot be derived by treating the particles as the parameters of the variational distribution and minimizing the -ELBO objective (Interestingly, Eq. (25) is in the form of an ELBO!), and why kernels are introduced for smoothing the empirical distribution [Liu et al., 2019].
  - I did not find how $h$ and $h_m$ are calculated in the algorithm. I suppose they are estimated directly using their definitions, but this then involves a likelihood-model $p(x|\theta)$ optimization for every data sample $x$ in every iteration. Will it be too expensive?
* Literature inconsistency. In Table 1, both GFSD and GFSF are originated from [Liu et al., 2019]. Also, GFSD is not the sum of SVGD and w-SGLD. It is w-SGLD without the second term.
* Minor issues.
  - Notations. In Eq. (2), $v$ should also depend on the specific particle $\theta_i$. So it'd be in the form $v(\theta_i, \{ \theta_{i'}^{old} \}_{i'=1}^N)$ (I cannot type the curly bracket). In Line 110, $p(x_d | \theta)$ may be also needed to adapt to e.g. $p(y_d | f, x_d)$. Would be better to unify notations of $\mathbb{E} Z$ and $\mathbb{E} [Z]$.
  - The paper mentioned that there is not finite-particle analysis before. But it seems that [11] showed an analysis in this case.
  - Possible typos. Line 214: there may not be "to". Line 359: "Variacne".
  - Discussion. The paper showed the connection to the median method for bandwidth selection. The authors may also consider the connection to the heat-equation bandwidth selection method by [Liu et al., 2019] based on an equivalent dynamics principle, which achieves a better performance than the median method.

[Liu et al., 2019] "Understanding and Accelerating Particle-Based Variational Inference". ICML 2019.
limitations_and_societal_impact: The authors mentioned the limitation in explaining SVGD. I think the computational burden to estimate $h$ and $h_m$ in the proposed algorithms (the second item in "Cons") may also be a limitation.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies particle variational inference (PVI), which is a mix between ensemble and VI (the variational posterior is assumed to be a mixture of Dirac distributions, and an additional "repulsion term" is added to give better diversity), and proposes a novel generalization to Jensen's inequality. From here, the authors propose an upper bound to the first term of the ELBO (the expected log-likelihood term), which is tighter compared to the standard Jensen inequality. Empirical findings show that the proposed loss performs well in various cases.
main_review: This paper is very well written. The proposed method is soundly and rigorously justified. I think this is a good and solid paper. 

Nevertheless, one criticism from me is that the authors did not discuss the comparison of their method (i.e. their loss in eq. 25) with the standard ELBO objective. From eq. 25 it is clear that without the term $R(x_d, h)$, this objective reverts back to the standard ELBO with $q(\theta) := 1/K \sum_{k=1}^K \delta(\theta_k)$. But, when the $R(x_d, h)$ term is kept in, how valid is it to consider eq. 25 as an ELBO, and thus the proposed method as a VI method? I hope to see a discussion from the authors in the rebuttal period.

**Minor comments**

* Thm. 1: 
  * Do this instead: \begin{theorem}[\citeauthor, \citeyear]
  * The statement is very hard to parse. Please consider rephrasing it
  * The notation $\nu^{\otimes D}$ is undefined 
* All proofs: the qed symbols should be flush-right
* L.197: What does it mean by "$q(\theta)$ is a set of parameters"
* L.212: "a kind of PVI" -> "an instance of PVI"
* Fig. 1:
  * "Blue line" -> "Blue curve"
  * (Also L.316) How do "mean estimate" differ from the predictive mean?
* L.326, among other places: "ReLu" -> "ReLU"
* Tab. 2, 3, 4: Inconsistency with the bolding. I.e. Tab. 2 and Tab. 3 also need some bold faces
* Tab. 2: "Mnist", "Cifar" -> "MNIST", "CIFAR"
* L.359: "Variacne" -> "Variance" or just "Var"


---

**Post-Rebuttal**

After reading the authors' rebuttal and other reviews, I decided to keep recommending an acceptance.
limitations_and_societal_impact: The authors discussed both limitations and societal impacts. However, they do so in the appendix. The limitations should instead be discussed in the main text.
review_title: null

Meta-review: This submission uses a tighter inequality for the likelihood than Jensen's inequality by a second order expansion.  The additional terms are then interpreted as a diversity term.  A PAC-Bayesian analysis is then developed based on this inequality.  The submission initially received a mix of scores straddling the acceptance threshold.  An external expert was consulted giving a mixed opinion (see copy of comments below), while the lower scoring reviewer was convinced to revise their score upwards during the rebuttal process.  On the balance, the paper is viewed positively, but has some important issues that should be corrected prior to publication.  These relate to incorporating reviewer comments (see below), but importantly to avoid obvious technical inaccuracies such as "MAP estimation can be used to obtain samples from the posterior" (section 2, from expert opinion below).

Additional expert opinion:
"The idea is simple overall, they use a thighter inequality than Jensen For the likelihood by exploiting second order expansion. This results in an additional terms which is interpreted as a diversity term. They combine this bound with a pac-bayesian bound to define a suitable loss for sampling from posterior in a particle fashion. Then they proceed to relate this loss to existing methods in particular w-SGLD and DPP, showing that their bound can result in methods that are formally similar to these prior works I didn’t find the paper particularly nice to read, for instance **some important details raised by reviewer 2 was only discussed in the appendix without mentioning them in the main ** The experiements did not compare with very standard benchmarks such as ELBO or even SMC methods which is also a particle methods. In the text they say that they used to HMC as a baseline, but I couldn’t see the performance of HMC in the results Also some inaccuracies in section 2.1: they claim that MAP estimation can be used to obtain samples from the posterior, but of course this is wrong... this is on the negative side on the positive side: The PAC-bayesian people might be interested in such result as it relies on a tighter bound than Jensen to derive the PAC-bayesian bound. And the connection to other methods is also interesting at a high level, although it is unclear to what extend this connection can say anything specific about w-SGLD and DPP in terms of generalization bound, etc If the paper is accepted, then the remarks by Reviewer 2 should be taken into account and in general the paper should mention in the main text what is going on in the appendix"
Decision: Accept (Poster)