SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We develop two computationally efficient variants of SVGD with provably fast convergence guarantees in the finite-particle regime, and present a wide range of improvements over prior work. A promising avenue of future work could be to establish convergence guarantees for SVGD with general non-logconcave targets, as was considered in recent works on LMC and SGLD [2, 12]. Other important avenues include establishing minimax lower bounds for SVGD and related particle-based variational inference algorithms. Beyond this, we also conjecture that the rates of GB-SVGD can be improved even in the regime  $n \ll KT$ . However, we believe this requires new analytic tools.
We thank Jiaxin Shi, Lester Mackey and the anonymous reviewers for their helpful feedback. We are particularly grateful to Lester Mackey for providing insightful pointers on the properties of Kernel Stein Discrepancy, which greatly helped us in removing the curse of dimensionality from our finite-particle convergence guarantees.
[1] L. Ambrosio, N. Gigli, and G. Savaré. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media, 2005.
[2] K. Balasubramanian, S. Chewi, M. A. Erdogdu, A. Salim, and S. Zhang. Towards a theory of non-log-concave sampling: first-order stationarity guarantees for Langevin monte carlo. In P.-L. Loh and M. Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 2896-2923. PMLR, 02-05 Jul 2022. URL https://proceedings.mlr.press/v178/balasubramanian22a.html.
[3] E. Bernton. Langevin monte carlo and jko splitting. In Conference On Learning Theory, pages 1777-1798. PMLR, 2018.
[4] S. Bobkov and M. Ledoux. Poincaré's inequalities and talagrand's concentration phenomenon for the exponential distribution. Probability Theory and Related Fields, 107:383-400, 1997.
[5] S. G. Bobkov and F. Götze. Exponential integrability and transportation cost related to logarithmic sobolev inequalities. Journal of Functional Analysis, 163(1):1-28, 1999.
[6] C. Carmeli, E. De Vito, A. Toigo, and V. Umanitá. Vector valued reproducing kernel hilbert spaces and universality. Analysis and Applications, 8(01):19-61, 2010.
[7] S. Chewi, T. Le Gouic, C. Lu, T. Maunu, and P. Rigollet. Svgd as a kernelized wasserstein gradient flow of the chi-squared divergence. Advances in Neural Information Processing Systems, 33:2098-2109, 2020.
[8] S. Chewi, M. A. Erdogdu, M. Li, R. Shen, and S. Zhang. Analysis of langevin monte carlo from poincare to log-sobolev. In P.-L. Loh and M. Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 1-2. PMLR, 02-05 Jul 2022. URL https://proceedings.mlr.press/v178/ Chewi22a.html.
[9] K. Chwialkowski, H. Strathmann, and A. Gretton. A kernel test of goodness of fit. In International conference on machine learning, pages 2606-2615. PMLR, 2016.
[10] J. B. Conway. A course in functional analysis, volume 96. Springer, 2019.
[11] A. Das, B. Schölkopf, and M. Muehlebach. Sampling without replacement leads to faster rates in finite-sum minimax optimization. Advances in Neural Information Processing Systems, 2022.
[12] A. Das, D. Nagaraj, and A. Raj. Utilising the clt structure in stochastic gradient based sampling: Improved analysis and faster algorithms. In Conference on Learning Theory, 2023.
[13] R. M. Dudley. Real analysis and probability. CRC Press, 2018.
[14] A. Duncan, N. Nusken, and L. Szpruch. On the geometry of stein variational gradient descent. arXiv preprint arXiv:1912.00894, 2019.
[15] A. El Alaoui, A. Montanari, and M. Sellke. Sampling from the sherrington-kirkpatrick gibbs measure via algorithmic stochastic localization. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS), pages 323-334. IEEE, 2022.
[16] S. Gershman, M. D. Hoffman, and D. M. Blei. Nonparametric variational inference. In Proceedings of the 29th International Conference on International Conference on Machine Learning, 2012.
[37] Y. Nesterov. Introductory lectures on convex programming volume i: Basic course. Lecture notes, 3(4):5, 1998.
[38] N. Nusken and D. Renger. Stein variational gradient descent: many-particle and long-time asymptotics. arXiv preprint arXiv:2102.12956, 2021.
[39] G. Parisi. Correlation functions and computer simulations. *Nuclear Physics B*, 180(3):378-384, 1981.
[40] M. Raginsky, A. Rakhlin, and M. Telgarsky. Non-convex learning via stochastic gradient Langevin dynamics: a nonasymptotic analysis. In Conference on Learning Theory, pages 1674–1703. PMLR, 2017.
[41] G. O. Roberts and R. L. Tweedie. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, pages 341-363, 1996.
[42] A. Salim, L. Sun, and P. Richtarik. A convergence theory for svgd in the population limit under talagrand's inequality t1. In International Conference on Machine Learning, pages 19139-19152. PMLR, 2022.
[43] J. Shi and L. Mackey. A finite-particle convergence rate for stein variational gradient descent. arXiv preprint arXiv:2211.09721, 2022.
[44] I. Steinwart and A. Christmann. Support vector machines. Springer Science & Business Media, 2008.
[45] L. Sun, A. Karagulyan, and P. Richtarik. Convergence of stein variational gradient descent under a weaker smoothness condition. In International Conference on Artificial Intelligence and Statistics, pages 3693-3717. PMLR, 2023.
[46] M. Talagrand. Pettis integral and measure theory. American Mathematical Soc., 1984.
[47] S. Vempala and A. Wibisono. Rapid convergence of the unadjusted Langevin algorithm: Isoperimetry suffices. Advances in neural information processing systems, 32, 2019.
[48] R. Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[49] C. Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
[50] D. Wang and Q. Liu. Learning to draw samples: With application to amortized mle for generative adversarial learning. arXiv preprint arXiv:1611.01722, 2016.
[51] D. Wang, Z. Zeng, and Q. Liu. Stein variational message passing for continuous graphical models. In International Conference on Machine Learning, pages 5219-5227. PMLR, 2018.
[52] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681–688, 2011.
[53] A. Wibisono. Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem. In Conference on Learning Theory, pages 2093-3027. PMLR, 2018.
[54] J. Zhuo, C. Liu, J. Shi, J. Zhu, N. Chen, and B. Zhang. Message passing stein variational gradient descent. In International Conference on Machine Learning, pages 6018-6027. PMLR, 2018.
[55] D. Zou, P. Xu, and Q. Gu. Faster convergence of stochastic gradient Langevin dynamics for non-log-concave sampling. In Uncertainty in Artificial Intelligence, pages 1152-1162. PMLR, 2021.