Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors analyze the underestimation issue of stein variational gradient descent (SVGD), and propose the maximum mean discrepancy (MMD) descent. From the perspective of the decomposition of the gradient term (driving force and repulsive force), this paper suggested to use another driving force term instead of the original one in SVGD. But the new driving force makes MMD-descent impractical since it depends on an intractable integral of the desired distribution $p$. In addition, the paper identify the log derivative driving force as the problematic term in SVGD, and propose a modified SVGD with particle resampling. They also argue that the proportional asymptotic limit is more relevant to understanding the variance collapse phenomenon. The theoretical dimensional analysis of SVGD on Gaussian also suggested another modified (damped) SVGD.
main_review: Strengths:
1. It is a good paper to analyze the limit and weaknesses of SVGD, especially the variance collapse issue on high-dimensional distributions. It is worth reading for the researchers working on Bayesian inference.
2. The proposed MMD-descent motivates some modifications to the original SVGD.
3. The experiments on toy datasets perfectly validate the claims of this paper.

Weaknesses:
1. Only toy datasets are used to validate the proposed approach. I would like to see a more practical experiment. E.g., in the original SVGD paper, Bayesian logistic regression and Bayesian Neural nets (BNN) on practical datasets are discussed. I would like to know on real scenario any advantage could be found with the proposed method.
summary_of_the_review: Overall, this is a good paper with solid analysis. It should be of interest to the area of Bayesian inference. But I think more practical validation should be addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides an understanding of the variance collapse phenomenon of SVGD. The paper first (1) introduces the reader to the most important concepts and phenomena, then (2) gives an explanation for why this problem occurs, thanks to a comparison with an accurate (yet computationally intensive) algorithm they call MMD-descent. Finally (3) the paper shows how to fix SVGD with damping. The paper provides experiments and theory, nicely combined.
main_review: I really enjoyed the paper and learned a lot from it. I was not familiar with this issue, but I appreciated the dedication and the in-depth study: it is rare to see a paper dedicated to "understanding" these days. All my congrats to the authors also for the nice writing and the nice figures.

My acceptance is only "weak" though since I would have loved to see the following:

1) a more thorough investigation of the "fixed SVGD": this only takes a page, and though some additional results are presented in the appendix I feel it deserves more space, or potentially also a few weeks of "rethinking" by the authors. Do you maybe have some ideas on how to make this section more complete? For instance, it would be great if the authors were able to show how damping provably fixes variance collapse in the high-dim Gaussian case. Examples are often underrated, yet these specified proofs often highlight nice ideas and concepts.

2) I feel that the method under investigation is only SVGD, yet many other methods exist to estimate probability densities. I think the paper would really profit from a comparison with other methods, just to set the right context.

All the rest is really good, though it maybe can be shortened a bit to leave space to more experimental results or more results (potentially a consistency theorem) for damped SVGD. 
summary_of_the_review: A great paper, well written. The topic is nice. Some things can be improved (taken a step forward) to make the paper truly valuable also for practical uses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work studies the variance collapse phenomenon of SVGD. By comparing to MMD-descend, the authors argue that the driving force of SVGD suffers from a bias caused by reusing data, and thus tends to underestimate the variance of the target distribution. Theory are developed in the setting of estimating standard Gaussian with a proposal limit (i.e., $d/n \to \gamma$), and explains the understanding in the overparameterized/high-dim setting (i.e., $\gamma > 1$). Experiments are also conducted to verify the understanding. Finally, motivated by the understanding, new algorithm is proposed to fix the issue of SVGD by damping the driving force term in SVGD.
main_review: # Pros
- The paper is well written and easy to follow.
- There is a complete set of experiments to support the authors' argument.
- Theory in simplified settings is provided to justify the understanding. 

# Cons
- The presented theory is a bit unsatisfactory in the sense that (1) the target data distribution is too simplified and (2) the proposional assumption ($d/n \to \gamma > 1$) weakens the theory for explaining the experimental results. In specific:
   *  Based on my understanding of the proof (note that I did not look in detail), (1) greatly simplifies the argument in that the studied problem is basically 1-dimensional. I believe the theory could be much more interesting if (1) can be relaxed to allow general PSD matrix as the covariance of the target distribution.
   * When I went through the paper, which is enjoyable process, I did not realize the necessity of  (2) until Section 5. In particular, the experimental observation, e.g., Figs 1 and 2, does not seem to satisfy (2), and in these experiments the variance collapse issue still exists in SVGD. It feels like (2) is invented only for the purpose of making up a theory. 


summary_of_the_review: First of all I am not an expert on SVGD thus I tend to be conservative in evaluating the paper. 
Overall I think the paper is written very well & I have enjoyed much going through it. The problem studied in the paper, the variance collapse of SVGD, seems to be important and interesting in my perspective. I think the authors have made sharp observation towards understanding this issue & how to mitigate it. The main pitfall of this work, in my perspective, is the theory part; as explained above, I think the presented theory has quite some gaps from the experimental findings. 

I am leaning to weak acceptance for now. Authors feedbacks are welcome as I may have missed some important points.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper analyzed the curse-of-dimensionality problem of the vanilla SVGD with Euclidean distance kernel in a qualitative and quantitative way. Specifically, the author first built a connection of SVGD to MMD-descent, where they share identical repulsive forces with different driving forces (if Euclidean distance kernel is adopted). Then, the author argued that the variance collapse problem is rooted in (1) high variance and (2) the deterministic bias of the driving force, which were confirmed by sampling from the isotropic Gaussian. Quantitatively, the author analyzed the stationary variance of MMD-descent and SVGD with isotropic Gaussian under the proportional limit, which confirms the curse-of-dimensionality problem of SVGD. 
main_review: ## Strength
This paper attempts to analyze one important problem of SVGD. Although this phenomenon has been identified in many previous works, this paper is the first one that theoretically quantifies the converged variance beyond the mean-field assumption. Qualitatively, the connection between MMD-descent and SVGD is intuitive (although I am still a bit confused about why introducing MMD-descent is necessary). Personally, I believe the theoretical analysis under the proportion limit is a significant contribution to the community, which is much closer to the actual empirical behaviour than the typical mean-field assumption. However, I still have several confusions.

## Weakness
My first question is why the connection between MMD-descent and SVGD is necessary? I understand that they share identical repulsive forces and similar driving forces, but they are still different. Thus, any finding from analyzing MMD-descent cannot be directly transferred to SVGD, no? For example, in section 4.1, proposition 2 revealed the large variance of the driving force in MMD-descent. However, you only analyze its effect on MMD-descent, not SVGD. Specifically, whether the driving force in SVGD has a large variance or its empirical effects are not analyzed. So what is the purpose of introducing MMD-descent and analyzing the large variance of the driving force?

Proposition 2 only confirms that the driving force in MMD-descent has a large variance. Is it still true for SVGD?

I also find that Section 4 and 5 are a bit disconnected. In Section 4, the argument is that the variance collapse problem is due to the deterministic bias of the driving force, which is confirmed empirically. However, the theoretical analysis in section 5 does not reveal why this bias causes the variance collapse problem. Also, I recommend that the author should add a sentence explaining where the deterministic bias assumption is used in the analysis. 

If I understand correctly, the entire analysis is based on the Euclidean distance kernel assumption. However, there are many types of kernels. Do they still suffer from the variance collapse problem? In other words, do you think the variance collapse problem is due to the SVGD itself or the kernel associated with it?

For the empirical verification of (A2), how do you select $\epsilon_d$? 

summary_of_the_review: Overall, this paper is written clearly and easy to follow. It also has significant theoretical contributions about the SVGD dynamics under the proportion limit. Although I am a little confused about the motivation of introducing MMD-descent, this paper has more merits than its drawbacks.
review_title: null

Meta-review: This paper proposes new analysis on Variance Collapse of SVGD in High Dimensions. The analysis provides some new insights despite of some limitations.
Decision: Accept (Poster)