Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper aims to improve the numerical stability of stein mixture in variational inference. To achieve this, the proposed EBLO-within-stein method allows the choice of a class of hierarchical attractives forces indexed by the hyper-parameter $\alpha$ in the Renyi divergence. Different values of $\alpha$ acounts for different types of latent variables, which could be measured by the signal-to-noise ratios. The paper includes empirical results to show various effects of $\alpha$ on real data analysis.
strength_and_weaknesses: Strengths:

- The paper is overall well-written and the method is clearly explained.  The reviewer really like the writing in section 2 that builds up the knowledge step by step
- The literature review is thorough
- The connection with the Renyi divergence and the VR bound seems interesting

Weakness: 

- Although the connection with existing works is interesting, the reviewer is not fully convinced why ELBO-within-Stein is preferred over the existing ones.

- given a real data analysis, how should one pick the value of $\alpha$


clarity,_quality,_novelty_and_reproducibility: Some minor comments in terms of clarity:

- the third contribution mentionds global latent and local latent variables, could the authors clarify their definition. The explanation in section 3.1 is not crystal clear to the reviewer.

- on page 2, when the authors claim the motivation of the method, they state "for such estimates, we cannot expect XXX as stein force will compensate the gradient estimation error by a counter force", can the authors be more specifc?

summary_of_the_review: Although more empirical results can make the paper much stronger, this is an overall well-executed paper. The authors need clarify on a few raised questions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors generalized the Stein Mixtures approach of Nalisnick and Smyth 2017 to general alpha-divergences.  The paper is mostly nicely written and was a good read.  But it did not have a clear motivation nor impressive empirics.
strength_and_weaknesses: Strengths
* The explanation and demonstration of the trade-off between the need for a large number of Monte Carlo samples when estimating the gradients without too much bias / variance is nicely laid out and is clear.
* The introduction of alpha-divergences into Stein mixtures is a nice application of a compelling line of work on generalized variational inference and I was glad to see it explored here.
* The worked out toy example with the analytic fixed points is very nice and illuminating!

Weaknesses:
* The authors should better defend a key, undefended premise of their work about which I was unconvinced: that Stein mixtures alleviate an “exponential” dependence of SVGD on the dimensionality on the model.  This premise is not made precise nor is it attributed to previous work that establishes it more precisely.  Why is an exponential number of particles needed for SVGD?  This is certainly not the case for MCMC for example.
* If there is a situation where the dependence is exponential with dimension, would the authors please explain (ideally with a theoretical result) such a situation?
* Or if there is no theoretical justification or proof to be made about the better dependence on dimension, empirical validation of this claim is needed.  For example, it would help to have an illustrative example of a case (e.g. in simulation) where the dimension scaling is exponential with the dimension. E.g. perhaps a plot with an increasing dimension on the x axis and the number of particles needed to achieve some level of posterior precision on the y-axis (for all three of the SVGD method, Stein mixtures, and the new proposed method).
* Or alternatively, if this exponential dependence on dimension is known from prior theory or empirics, even replicating those prior results (with citation) would strengthen the paper.
* Or if the dependence is not actually exponential, the authors must either be precise about the way in which dimensionality poses a problem that is addressed by their approach or drop this as the premise of the method.

* Given that the paper is about an improvement upon Nalisnick and Smyth 2017 (whose applications and limitations most readers may not be very familiar with), the paper could benefit from more justification as to why this method is important/useful and why it must be improved.  At present, I found it challenging to see what problem the new methodology was trying to solve that Nalisnick and Smyth 2017 had not solved.
* On problems where the developed methodology is applicable, how does it compare to other Bayesian inference methods?  E.g. HMC, MFVB, Laplace approximations etc., in terms of computational cost and accuracy (experiments are not necessarily needed, but I think commentary at least is).  For the UCI comparisons, previously run baselines could be used.
clarity,_quality,_novelty_and_reproducibility: * I found the paper mostly well written but had trouble following it in some places.  For example, around equation 2 it is very unclear whether the $phi_j$ are part of the model or if they are hyper-parameters of the variational approximation since they are written to have a density under p.  This should be made much more explicit, as the notation in equation 2 suggests that $phi$ exists in the original model $p$.
* The work is original.
summary_of_the_review: The paper is an enjoyable read about extending Stein mixtures to use alpha divergences.  But I recommend rejection because in my assessment it falls short on:
* Motivating how or why the proposed method addresses limitations of existing methods (particularly dimension dependence) and
* Empirical demonstration of improvement relative to existing methods.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigate a pevious work on stein mixtures and show that it corresponds to inference with the Renyi $\alpha$-divergence for $\alpha=0$, and using other values of $\alpha$ may stablize the inference.
strength_and_weaknesses: Strength:

1. This paper is overall well-written and the literature review is thorough.

Weaknesses:

1. The motivation of the proposed method is vague. No comparison to other SVGD variants.
2. The orginality is low as the major contribution seems to be some empirical investigation of a hyperparameter.
3. The paper is a little bit hard to follow. What is the main difference between stein mixture and a simple mixture model?
clarity,_quality,_novelty_and_reproducibility: The paper is well written, mostly clear, and lack of originality.
summary_of_the_review: My main concern is that the original contribution of this work is not significant enough.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work considers an extension of Stein variational gradient descent to the the case where the training objective captures an f-divergence between the proposal and target distribution, and develops algorithmic machinery and convergence guarantees for such methods. 

The strengths of this work are in the writing quality, technical precision, and numerical validation of the proposed approach.

The weaknesses are in establishing the proposed generalization addresses a key shortcoming of prior work. In particular, both theoretically and numerically, it is not clear why the proposed generalization can address a gap in prior works, and fails to illuminate the practical significance of the additional generality. This is a key sticking point among the reviewers with which I tend to agree. For this reason, the paper is below the bar of acceptance.
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA
Decision: Reject