Convergence of Smoothed Empirical Measures With Applications to Entropy Estimation

This paper studies convergence of empirical measures smoothed by a Gaussian kernel. Specifically, consider approximating <inline-formula> <tex-math notation="LaTeX">$P\ast \mathcal {N}_\sigma $ </tex-math></inline-formula>, for <inline-formula> <tex-math notation="LaTeX">$\mathcal {N}_\sigma \triangleq \mathcal {N}(0,\sigma ^{2} \mathrm {I}_{d})$ </tex-math></inline-formula>, by <inline-formula> <tex-math notation="LaTeX">$\hat {P}_{n}\ast \mathcal {N}_\sigma $ </tex-math></inline-formula> under different statistical distances, where <inline-formula> <tex-math notation="LaTeX">$\hat {P}_{n}$ </tex-math></inline-formula> is the empirical measure. We examine the convergence in terms of the Wasserstein distance, total variation (TV), Kullback-Leibler (KL) divergence, and <inline-formula> <tex-math notation="LaTeX">$\chi ^{2}$ </tex-math></inline-formula>-divergence. We show that the approximation error under the TV distance and 1-Wasserstein distance (<inline-formula> <tex-math notation="LaTeX">$\mathsf {W}_{1}$ </tex-math></inline-formula>) converges at the rate <inline-formula> <tex-math notation="LaTeX">$e^{O(d)} n^{-1/2}$ </tex-math></inline-formula> in remarkable contrast to a (typical) <inline-formula> <tex-math notation="LaTeX">$n^{-\frac {1}{d}}$ </tex-math></inline-formula> rate for unsmoothed <inline-formula> <tex-math notation="LaTeX">$\mathsf {W}_{1}$ </tex-math></inline-formula> (and <inline-formula> <tex-math notation="LaTeX">$d\ge 3$ </tex-math></inline-formula>). Similarly, for the KL divergence, squared 2-Wasserstein distance (<inline-formula> <tex-math notation="LaTeX">$\mathsf {W}_{2}^{2}$ </tex-math></inline-formula>), and <inline-formula> <tex-math notation="LaTeX">$\chi ^{2}$ </tex-math></inline-formula>-divergence, the convergence rate is <inline-formula> <tex-math notation="LaTeX">$e^{O(d)} n^{-1}$ </tex-math></inline-formula>, but only if <inline-formula> <tex-math notation="LaTeX">$P$ </tex-math></inline-formula> achieves finite input-output <inline-formula> <tex-math notation="LaTeX">$\chi ^{2}$ </tex-math></inline-formula> mutual information across the additive white Gaussian noise (AWGN) channel. If the latter condition is not met, the rate changes to <inline-formula> <tex-math notation="LaTeX">$\omega \left ({n^{-1}}\right)$ </tex-math></inline-formula> for the KL divergence and <inline-formula> <tex-math notation="LaTeX">$\mathsf {W}_{2}^{2}$ </tex-math></inline-formula>, while the <inline-formula> <tex-math notation="LaTeX">$\chi ^{2}$ </tex-math></inline-formula>-divergence becomes infinite – a curious dichotomy. As an application we consider estimating the differential entropy <inline-formula> <tex-math notation="LaTeX">$h(S+Z)$ </tex-math></inline-formula>, where <inline-formula> <tex-math notation="LaTeX">$S\sim P$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$Z\sim \mathcal {N}_\sigma $ </tex-math></inline-formula> are independent <inline-formula> <tex-math notation="LaTeX">$d$ </tex-math></inline-formula>-dimensional random variables. The distribution <inline-formula> <tex-math notation="LaTeX">$P$ </tex-math></inline-formula> is unknown and belongs to some nonparametric class, but <italic>n</italic> independently and identically distributed (i.i.d) samples from it are available. Despite the regularizing effect of noise, we first show that any good estimator (within an additive gap) for this problem must have a sample complexity that is exponential in <italic>d</italic>. We then leverage the above empirical approximation results to show that the absolute-error risk of the plug-in estimator converges as <inline-formula> <tex-math notation="LaTeX">$e^{O(d)} n^{-1/2}$ </tex-math></inline-formula>, thus attaining the parametric rate in <italic>n</italic>. This establishes the plug-in estimator as minimax rate-optimal for the considered problem, with sharp dependence of the convergence rate both in <italic>n</italic> and <italic>d</italic>. We provide numerical results comparing the performance of the plug-in estimator to that of general-purpose (unstructured) differential entropy estimators (based on kernel density estimation (KDE) or <italic>k</italic> nearest neighbors (kNN) techniques) applied to samples of <inline-formula> <tex-math notation="LaTeX">$S+Z$ </tex-math></inline-formula>. These results reveal a significant empirical superiority of the plug-in to state-of-the-art KDE and kNN methods. As a motivating utilization of the plug-in approach, we estimate information flows in deep neural networks and discuss Tishby’s Information Bottleneck and the compression conjecture, among others.