SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {x \in V _ {1}} \pi (x) p ^ {\prime} \left(x, V _ {1} ^ {c}\right) \geq \frac {\sum_ {x \in \tau \left(V _ {1}\right)} \pi (x)}{4 d}. \tag {11}
$$
Further denote  $\tau(x, r)$  for some  $r \in \{0, 1, \dots, d\}$  as the set of elements at distance  $r$  from  $\tau(x)$  along the descending shortest paths. We provide an illustrative figure in 2. Then we have that
$$
\pi \left(V _ {1}\right) = \sum_ {\tau (x) \in \tau \left(V _ {1}\right)} \sum_ {r = 1} ^ {D} \tau (x, r). \tag {12}
$$
We note by reversibility that
$$
\pi (\tau (x, r)) p ^ {\prime} (\tau (x, r), \tau (x, r + 1)) = \tau (x, r + 1) p ^ {\prime} (\tau (x, r + 1), \tau (x, r)). \tag {13}
$$
Further by the fact that the function along the path is descending, we get that
$$
p ^ {\prime} (\tau (x, r), \tau (x, r + 1)) \geq p ^ {\prime} (\tau (x, r + 1), \tau (x, r)). \tag {14}
$$
$$
\pi (\tau (x, r)) \geq \pi (\tau (x, r + 1)). \tag {15}
$$
$$
\pi \left(V _ {1}\right) = \sum_ {\tau (x) \in \tau \left(V _ {1}\right)} \sum_ {r = 1} ^ {D} \tau (x, r) \leq D \pi \left(\tau \left(V _ {1}\right)\right) \tag {16}
$$
Therefore, when  $\pi (V_1)\leq 1 / 2$  we have
$$
\frac {\sum_ {x \in V _ {1}} \pi (x) p ^ {\prime} (x , V _ {1} ^ {c})}{\min  \left\{\pi (V _ {1}) , 1 - \pi (V _ {1}) \right\}} \geq \frac {1}{4 d D}.
$$
when  $\pi (V_1)\geq 1 / 2$  we have
$$
\begin{array}{l} \frac {\sum_ {x \in V _ {1}} \pi (x) p ^ {\prime} (x , V _ {1} ^ {c})}{\min  \left\{\pi (V _ {1}) , 1 - \pi (V _ {1}) \right\}} \geq 2 \sum_ {x \in V _ {1}} \pi (x) p ^ {\prime} (x, V _ {1} ^ {c}) \\ \geq \frac {\sum_ {x \in \tau (V _ {1})} \pi (x)}{2 d} \geq \frac {\pi (\tau (V _ {1}))}{2 d D} \geq \frac {1}{4 d D}. \\ \end{array}
$$
The theorem then follows by Cheeger's inequality.
Figure 2: An illustration for the definitions of  $\tau (x)$  and  $\tau (x,r)$
We still use  $\pi_3$  in 6.1 as our target distribution. We initially set  $n = 1, 10, 2000$  particles to run the LMC sampling algorithm respectively. We collect the locations of these particles after  $T = 1000$  iterations as valid samples. The target distribution is shown in Figure 3a; empirical distributions using  $n = 1, 10, 2000$  particles are shown in Figure 3b, 3c, and 3d.
We highlight two observations from the experiments. First when we use only one particle, it is always trapped in one Gaussian component and we never get samples from other modes. It has very bad global convergence, but still gets good conditional convergence: The convergence result conditioned
(a) target distribution
(b) 1 particle
(c) 10 particles
(d) 2000 particles
on the 1st quadrant is good, and the sample after  $T$  iterations has no probability distributed on the other 3 quadrants. Second, when the number of particles grows, the sample after  $T$  iterations has non-negligible probabilities distributed on all of the 4 quadrants, which means we could capture all of the Gaussian components. The results indicate that although restarting LMC may not directly lead to global convergence, it may help us capture the features in multi-modal distributions, which further implies we may empirically eliminate the "small probability mass" condition in Corollary 1 and mitigate the gap between sampling distribution and target distribution.
A broader implication could be on ensemble and stochastic averaging in neural network training, where ensemble follows the restart procedure where as stochastic averaging is closer to sampling from a single trajectory. Our theory and experiment suggest that the two can have very different distributions in the end.