Hybrid Kernel Stein Variational Gradient Descent

Authors: Anson MacDonald, Scott A Sisson, Sahani Pathiraja
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: Stein Variational Gradient Descent, Approximate Inference, Particle-based Variational Inference, Gradient Flow

Abstract:
Stein variational gradient descent (SVGD) is a particle based approximate inference algorithm. Many variants of SVGD have been proposed in recent years, including the hybrid kernel variant (h-SVGD), which has demonstrated promising results on image classification with deep neural network ensembles. In this paper, we demonstrate the ability of h-SVGD to alleviate variance collapse, a problem that SVGD is known to suffer from. Unlike other SVGD variants that alleviate variance collapse, h-SVGD does not incur additional computational cost, nor does it require the target density to factorise. We also develop the theory of h-SVGD by demonstrating the existence of a solution to the hybrid Stein partial differential equation. We highlight a special case in which h-SVGD is a kernelised Wasserstein gradient flow on a functional other than the Kullback-Leibler divergence, which is the functional describing the SVGD gradient flow. By characterising the fixed point in this special case, we show that h-SVGD does not converge to the target distribution in the the mean field limit. Other theoretical results include a descent lemma and a large particle limit result. Despite the bias in the mean field limiting distribution, experiments demonstrate that h-SVGD remains competitive on high dimensional inference tasks whilst alleviating variance collapse.

Decision: reject
Reviewer scores: 8, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730229118431

thread_type:
official_review

<thread_object>
creator: reviewer_rguy

<content>
summary:
This paper discusses h-SVGD as a method to address variance collapse by using separate kernels for the driving and repulsive terms. The authors extend the theoretical aspects of the original paper and demonstrate that h-SVGD does not converge to the target distribution in the mean field limit. The empirical results support the theoretical findings.

strengths:
- Theoretical contributions are rigorous and sound, including the existence of a solution to the hybrid Stein PDE and the establishment of the descent lemma. A kernelised Wasserstein gradient flow interpretation is thoroughly discussed with $k_2 = ck_1$.
- The experiments involve diverse datasets and metrics.
- The representation of experiments is clear and the paper is well-structured.
- Good to quantify that h-SVGD does not add additional computational costs to SVGD.

weaknesses:
- The title may be a bit misleading as it suggests that h-SVGD is a novel algorithm introduced in this paper, but it was previously proposed. For instance, it can reflect the paper's focus on theoretical analysis and empirical evaluation of h-SVGD, rather than introducing it as a new method.
- The main focus on $k_2 = ck_1$ may be somewhat limiting, as it potentially restricts the exploration of using truly distinct kernels for the driving and repulsive terms.
- While RMSE and LL have been assessed in Appendix B, h-SVGD does not show clear benefits over SVGD in these metrics. It would be better to provide a more detailed discussion in the main text about why DAMV is an appropriate metric for evaluating h-SVGD's performance, especially in relation to the variance collapse problem.
- Using "S-SVGD" and "SSVGD" interchangeably is slightly confusing and could be more consistent.

questions:
- The authors mention convergence issues in the mean field limit. Are there any ways to mitigate this bias?
- The authors focus on $k_2 = ck_1$ case. Have they tried using two completely different kernels?
- The authors reference h-SVGD's promising results on image classification. Have they conducted similar experiments in this paper?
- From an applications perspective, why is it important to avoid variance collapse?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730307445794

thread_type:
official_review

<thread_object>
creator: reviewer_2j3t

<content>
summary:
Ths paper gives more theoretical insights about a sampling method named hybrid kernel stein variational gradient descent (h-SVGD). It is demonstrated that this method effictively sample a law which is not the target law (but linked to it). Moreover, it provides a descent result and a discretization quantification. Finally, it presents experiments that show the empirical interest of h-SVGD.

strengths:
The paper provides interesting theoretical results on h-SVGD. It extend the known results on SVGD to h-SVGD. The experimental results effectively support the interest of h-SVGD.

weaknesses:
Major weaknesses:
- Assumption (A1) does not seem to be well written. To suppose that $\lim_{|x|\to +\infty} V(x) = +\infty$ must be a more realistic assumption to ensure that $\pi$ could be a probability measure. Is it a writting error ?
- There is a few intuitions and comments in the paper, making the paper hard to read. In particular, I suggest to discuss the assumptions in details. A reminder on Wasserstein gradient flow will be appreciable. Finally, Proposition 3.6 is not comment.

Minor weaknesses:
- Equation 6 : there is a typo, it might be $(x, \cdot)$ instead of $(\cdot, x)$.
- line 198 : in the paper notations, it might be $\mathcal{H}$ instead of $\mathcal{H}_1$.
- The remark in line 246 seems inconsistent because Assumption (A1) induce that $V$ is bounded.
- In line 250, the constant $L$ is said to be dependent of $k_1, k_2$ and $p$. However it is not clear what is the variable $p$.
- In line 329, the normal density do not verify Assumption (A1). In fact in this case $V$ is quadratic.
- line 226, I suggest to be more precise about the sense of "symetric function" in this context.

questions:
- In Assumption (A2), the first part can be deduce from Assumption (A1). In fact, Assumption (A1) induce that $V$ is bounded. Why do you precise this assumption ?
- Can you give an interpretation of the second part of Assumption (A2) ?
- Can you give an interpretation of Assumption (A3) ?
- Do you have examples of potentials $V$ that verifies Assumptions (A1)-(A2)-(A3)-(A4) ?
- According to your analysis of h-SSVD you have a proposition of a sampling strategy that does not suffer from dimensional collaps and sample the right distribution ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730582999103

thread_type:
official_review

<thread_object>
creator: reviewer_ytia

<content>
summary:
This paper explores the theoretical understanding of the hybrid kernel variant of Stein variational gradient descent (h-SVGD). Specifically, it provides a kernelized Wasserstein gradient flow representation for h-SVGD and, building upon this, offers the dissipation rate and large particle limit of h-SVGD. In numerical experiments, the authors demonstrate that h-SVGD significantly improves variance collapse compared to vanilla SVGD.

strengths:
- This paper provides a relatively comprehensive theoretical foundation for the empirical method hybrid kernel variant of SVGD.
- This paper offers a clear explanation of the relationships and distinctions between the theoretical results of h-SVGD and those of SVGD.

weaknesses:
- The experimental results presented in the paper offer relatively limited support. In results on the Protein and Kin8nm datasets, SVGD does not appear to suffer from variance underestimation issues, yet its performance is still inferior to that of h-SVGD. This observation suggests that the advantage of h-SVGD in BNN tasks may not primarily stem from mitigating variance underestimation.

questions:
- The result in Corollary 3.4 indicates that h-SVGD does not guarantee the distribution of particles converges to the target distribution. While this is reasonable, the metrics commonly used in Bayesian Neural Network (BNN) tasks, such as test RMSE and test LL, do not reflect this limitation. This raises the question of how to interpret the advantages of h-SVGD despite this theoretical constraint.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730640827578

thread_type:
official_review

<thread_object>
creator: reviewer_j5rt

<content>
summary:
This paper proposed studied the theoretical foundation of the hybrid kernel Stein variational gradient descent (h-SVGD) method, which is a variant of the vanilla SVGD method. Specifically, the authors demonstrated the ability of h-SVGD to alleviate variance collapse, and showed the existence of a solution to the hybrid Stein partial differential equation for h-SVGD. They also showed that h-SVGD does not converge to the target distribution in the mean field limit. Experiments have been provided to show the promising properties of h-SVGD.

strengths:
1. The theoretical foundation of h-SVGD is established, which is new and important to this research topic.

2. Besides the new results, some existing results are proved with relaxed and more practical assumptions.

weaknesses:
To be honest, since I am not familiar with the mathematical tools used in this work, I can only judge the paper based on what the authors have done, as they claimed, and I cannot say much about its weaknesses.

questions:
No further questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732065411206

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Main Response

comment:
We thank each of the reviewers for their time reviewing our submission, and their considered feedback. As a direct result, the following changes have been made to the manuscript.

- The title has been updated to "Convergence aspects of hybrid kernel SVGD".
- A note on the importance of variance estimation has been added to the second paragraph of the introduction.
- Equation (6) has been updated from $(\cdot,  x)$ to $( x, \cdot)$.
- A typo in Assumption (A1) has been corrected. The condition has been changed from $\lim_{| x|\to\infty}V(x)=0$ to $\lim_{| x|\to\infty}V(x)=+\infty$.
- A typo in Assumption (A2) has been updated. The left-hand side of the first inequality has been updated from $|V( x)|^q$ to $|\nabla V( x)|^q$.
- Some commentary on and interpretation of the assumptions in Section 3.1 has been added after they are introduced.
- The definition of symmetric has been included in Assumption (B1) for clarity.
- A typo in Proposition 3.1 has been updated to say that $L$ depends on $\pi$ instead of $p$.
- A reminder of the definition of a Wasserstein gradient flow has been included in Section 3.4, along with references to two standard texts.
- A comment on the interpretation of Proposition 3.6 has been added after the statement of the proposition.
- A comment linking the improved variance estimation in practice with the mean field variance overestimation in Corollary 3.4 has been added to the start of Section 4.
- In Section 4.2, we included further interpretation of the improved variance collapse and linked it to Corollary 3.4. We also included a comment about how the improved RMSE and LL may be due to factors other than increased DAMV.
- All instances of SSVGD have been changed to S-SVGD for consistency. This is primarily in Appendix B.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper claims to develop a theory of h-SVGD, an extension of SVGD that alleviates variance collapse without computational overhead. Using the developed theory, it is claimed that h-SVGD does not converge to the target distribution in the limit of having infinite particles. 

The strengths of this work are in establishing a theoretical foundation of h-SVGD. 

A main weakness of the work is the clarity and issues with the mathematical presentation as pointed out by the reviewers. Reviewers were not able to verify the mathematical correctness or soundness of the presented proofs.   Moreover, the experimental evaluation is limited and it remains unclear whether the advantage of h-SVGD really stems from mitigating variance underestimation.

I recommend rejection with a main reason being that the practical implications of the main theoretical result remain unclear. For a resubmission, I encourage the authors to simplify and improve the clarity of the theoretical results, as well as add numerical results which show a clearer connection to the established theorems.