SlowGuess/NeurIPS_2023_OCR huggingface dataset:
C.1.1 Target distribution and model parameterization
As target density we take
$$
\rho_ {*} (x) = \frac {e ^ {- \frac {1}{2} | x - a | ^ {2}} + e ^ {- z _ {*} - \frac {1}{2} | x - b | ^ {2}}}{\sqrt {2 \pi} (1 + e ^ {- z _ {*}})}. \tag {43}
$$
Here  $z_{*} \in \mathbb{R}$  parameterizes the mass of the second mode relative to the first and is the sole parameter of interest. The proportion of samples in both modes is indeed
$$
p _ {*} := \frac {1}{1 + e ^ {- z _ {*}}}, \quad q _ {*} := 1 - p _ {*} = \frac {e ^ {- z _ {*}}}{1 + e ^ {- z _ {*}}}. \tag {44}
$$
Our point is that when both modes are separated by very low-density regions, learning  $z_{*}$  without weight correction leads to an incorrect estimation of the mode probabilities ('no-learning') or to mode collapse depending on the initialization of the learning procedure, whereas using the Jarzynski correction does not. From now on, we will suppose that the modes are separated in the following sense:
$$
\left| a - b \right| = 1 0, \tag {A1}
$$
which will be enough for our needs. The more separated they are, the stronger our quantitative bounds will be.
The parameterization for our model potential  $U_{z}$  is consistent with (43):
$$
U _ {z} (x) = - \log \left(e ^ {- \frac {1}{2} | x - a | ^ {2}} + e ^ {- z - \frac {1}{2} | x - b | ^ {2}}\right) \tag {45}
$$
and the associated partition function and free energy are
$$
Z _ {z} = \sqrt {2 \pi} (1 + e ^ {- z}), \quad F _ {z} = - \log Z _ {z} = - \log (1 + e ^ {- z}) - \frac {1}{2} \log (2 \pi). \tag {46}
$$
The normalized probability density associated with  $U_{z}(x)$  is thus  $\rho_z(x) = e^{-U_z(x) + F_z}$ .
C.1.2 Learning procedures
Gradient descent on the cross entropy leads to the following continuous-time dynamics:
$$
\dot {z} (t) = \mathbb {E} _ {z (t)} [ \partial_ {z} U _ {z (t)} ] - \mathbb {E} _ {*} [ \partial_ {z} U _ {z (t)} ]. \tag {47}
$$
For simplicity, we will start this ODE at  $z(0) = 0$ . It corresponds to a proportion of  $\frac{1}{2}$  for both modes.
The gradient descent (47) is an ideal situation where the expectations  $\mathbb{E}_{z(t)},\mathbb{E}_{*}$  can be exactly analyzed. In practice however, the two terms of (47) are estimated; the second term using a finite
number of training data  $\{x_{*}^{i}\}_{i = 1}^{n}$ , and the first one using a finite number of walkers  $\{X_{t}^{i}\}_{i = 1}^{N}$ , with associated weights  $\{e^{A_t^i}\}_{i = 1}^N$ . For simplicity we set  $N = n$  and the empirical GD dynamics is thus
$$
\dot {z} (t) = \frac {\sum_ {i = 1} ^ {n} e ^ {A _ {t} ^ {i}} \partial_ {z} U _ {z (t)} \left(X _ {t} ^ {i}\right)}{\sum_ {i = 1} ^ {n} e ^ {A _ {t} ^ {i}}} - \frac {\sum_ {i = 1} ^ {n} \partial_ {z} U _ {z (t)} \left(x _ {*} ^ {i}\right)}{n} \tag {48}
$$
and the walkers evolve under the Langevin dynamic
$$
d X _ {t} ^ {i} = - \alpha \nabla U _ {z (t)} \left(X _ {t} ^ {i}\right) d t + \sqrt {2 \alpha} d W _ {t} ^ {i} \tag {49}
$$
for some fixed  $\alpha > 0$ . Now the nature of the algorithm varies depending on how the walkers are initialized and the Jarzynski weights are evolved.
1. The standard PCD algorithm sets  $X_0^i = x_*^i$ , that is, the walkers are initialized at the data points, and the weights are not evolved, that is  $A_t^i = 0$  at any time.
2. Alternatively, the walkers could be initialized at samples of the initial model:  $X_0^i \sim \rho_{z(0)}$  with the weights not evolved. We refer to this algorithm as the unweighted procedure.
3. Our algorithm 2 corresponds to initializing the walkers at samples of the initial model,  $X_0^i \sim \rho_{z(0)}$ , and uses the Jarzynski rule (17) for the weights updates.
For simplicity we analyze the outcome of these algorithms in the continuous-time set-up, i.e. using (48). This is an idealization of the actual algorithms, but it makes the analysis more transparent.
C.1.3 Perturbative analysis
Using simple approximations (see Appendix C for details), we show that in the three cases above, the dynamics (48) can be seen as a perturbation of a simpler differential system whose qualitative behaviour fits with our numerical simulations. These systems depend on the initialization of the walkers and are thus prone to small stochastic fluctuations. We introduce:
-  $\hat{q}_{*}$  the proportion of training data  $\{x_{*}^{i}\}$  that are close to mode  $b$  (a more precise definition is given in Appendix C), and we define  $\hat{z}_{*}$  as satisfying  $\hat{q}_{*} = e^{-\hat{z}_{*}} / (1 + e^{-\hat{z}_{*}})$ ;
-  $\hat{q}(0)$  the proportion of walkers at initialization that are close to  $b$ , and  $\hat{p}(0) = 1 - \hat{q}(0)$ .
Practically,  $\hat{q}_{*}$  is a random variable centered at  $q_{*}$  and with fluctuations of order  $n^{-1 / 2}$ , and  $\hat{q} (0)$  is centered at  $e^{-z(0)} / (1 + e^{-z(0)}) = \frac{1}{2}$  with fluctuations of the same order. In the limit where  $n$  is large, they can be neglected, but in more realistic training settings, the use of mini-batches leads to small but non-negligible fluctuations, as will be clear in Equation (53).
The arguments in Appendix C lead to the following approximations:
- In the model-initialized algorithm without Jarzynski correction ('unweighted'), (48) is a perturbation of
$$
\dot {z} _ {\mathrm {u n w}} (t) = \hat {q} (0) - \hat {q} _ {*}. \tag {50}
$$
This system has no stable fixed point since the RHS no longer depends on  $z_{\mathrm{unw}}(t)$ , leading to a linear drift  $z_{\mathrm{unw}}(t) = (\hat{q}(0) - \hat{q}_*)t$  and thus to a divergence of  $z_{\mathrm{unw}}(t)$ . Consequently, the mass of the second mode,  $q(t) = e^{-z_{\mathrm{unw}}(t)} / (1 + e^{-z_{\mathrm{unw}}(t)})$ , converges to 0 or 1. However, on longer time scales, this drift leads to a sudden transfer of all walkers in one the modes, then to a complete reversal of the drift of  $z_{\mathrm{unw}}$  which then diverges in the opposite direction, leading to a succession of alternating mode-collapses; see also Figure 8 and Remark C.4 below.
- In the continuous-time version of the standard PCD algorithm, (48) is a perturbation of the same ODE as (50). However, in this context, since the initial data  $\{X_0^i\}$  and the training data  $\{x_{*}^{i}\}$  are identical, we have  $\hat{q}(0) = \hat{q}_{*}$ , and
$$
\dot {z} _ {\mathrm {p c d}} (t) = 0. \tag {51}
$$
The parameters do not evolve and the system is stuck at  $z_{\mathrm{pcd}}(0)$  ('no-learning'). Note however that in this version of PCD, the walkers are initialized at the full training data. In practice, the number of walkers is smaller than the number of training data so that one often uses a small batch of training data to initialize them; in this case we can still have  $\hat{q}(0) \neq \hat{q}_{*}$ , falling back to the first case above.
(a) Unweighted dynamics: (50)
(b) PCD: (51)
Figure 14: Behaviors of the three learning algorithms leading to the ODEs (51), (50), and (53) The solid black lines represent the target parameters,  $z_{*}$  on top and  $q_{*} = e^{-z_{*}} / (1 + e^{-z_{*}})$  on the bottom. The blue lines show  $z(t)$  (top) and the corresponding  $q(t) = e^{-z(t)} / (1 + e^{-z(t)})$  (bottom). Figure 14b: PCD algorithm, which leads to no-learning, with the weights not evolving at all in agreement with the behavior predicted by (51). Figure 14a: Dynamics without weights, that leads to mode collapse, with  $z(t)$  diverging towards  $+\infty$  in agreement with (50) when the parameter  $\hat{\gamma} = \hat{q}(0) - \hat{q}_{*}$  is positive. Figure 14c: Jarzynski-corrected dynamics with weights included, (53). The green line represents (54) and features the small  $\log[\hat{q}(0) / \hat{p}(0)]$  correction due to the stochastic fluctuations at initialization between the walkers and the model. In all cases, we used 200 walkers and data points. The continuous-time dynamics were discretized using the Euler-Maruyama scheme with a step-size  $h = 0.01$  up to time  $T = 10^{4}$ .
(c) Jarzynski-corrected: (53)
- The continuous-version of Contrastive-Divergence is equivalent to the well-known Score-Matching technique ([24]). In this context, (48) is a perturbation of
$$
\dot {z} _ {\mathrm {c d}} (t) = 0, \tag {52}
$$
leading to the same 'no-learning' phenomenon.
- In the model-initialized algorithm using the Jarzynski correction, (48) is a perturbation of
$$
\dot {z} _ {\mathrm {j a r}} (t) = \frac {\hat {q} (0) e ^ {- z _ {\mathrm {j a r}} (t)}}{\hat {p} (0) + \hat {q} (0) e ^ {- z _ {\mathrm {j a r}} (t)}} - \frac {e ^ {- \hat {z} _ {*}}}{1 + e ^ {- \hat {z} _ {*}}} \tag {53}
$$
This system has a unique stable point  $\tilde{z}_{*}$  satisfying  $e^{-\tilde{z}_{*}} = \hat{p}(0)e^{-\hat{z}_{*}} / \hat{q}(0)$ , hence
$$
\tilde {z} _ {*} = \hat {z} _ {*} + \log \left(\frac {\hat {q} (0)}{\hat {p} (0)}\right). \tag {54}
$$
Note that the second term at the right hand side is a small correction of order  $O(n^{-1/2})$  since  $\hat{q}(0)/\hat{p}(0) = 1 + O(n^{-1/2})$ .
These predictions are in good agreement with the results of the simulations shown in Figure 14c.
C.2 Analysis of (47)
Using the Jarzynski correction
In continuous time, the Jarzynski-correction described in Proposition 2 exactly realizes (47). However, even for simple mixtures of Gaussian densities, the expectations in (43) do not have a simple closed-form that would allow for an exact solution. That being said, when  $a$  and  $b$  are sufficiently well-separated, the system can be seen as a perturbation of a simpler system whose solution can be analyzed.
First, we note that
$$
\partial_ {z} U _ {z} (x) = \frac {e ^ {- z} e ^ {- \frac {\left| x - b \right| ^ {2}}{2}}}{e ^ {- U _ {z} (x)}}. \tag {55}
$$
The main idea of the approximations to come is that  $\partial_z U_z(x)$  is almost zero when  $x$  is far from  $b$  (and in particular, close to  $a$ ), and is almost 1 if  $x$  is close to  $b$ . The next lemma quantifies this; from now on we will adopt the notation
$$
I _ {a} = [ a - 4, a + 4 ] \quad \text {a n d} \quad I _ {b} = [ b - 4, b + 4 ]. \tag {56}
$$
Lemma 1. Under Assumption (A1), for any  $v\in \mathbb{R}$
- if  $x \in I_{a}$ , then  $\partial_z U_v(x) \leq e^{-v - 10}$ ;
- if  $x \in I_b$ , then  $|\partial_z U_v(x) - 1| \leq e^{v - 10}$ .
Proof. From (55), we see that if  $x \in I_a$  then  $|x - b| > 6$  and consequently  $e^{-(x - b)^2 / 2} \leq e^{-18}$ . But the denominator of (55) is itself greater than  $e^{-(x - a)^2 / 2}$  which is itself greater than  $e^{-4^2 / 2} = e^{-8}$  since  $|x - a| < 4$ . This gives the first bound and the second is proved similarly.
We recall that if  $\xi \sim \mathcal{N}(0,1)$ , then  $\mathbb{P}(|\xi| > t) \leq e^{-t^2 / 2} / t$ , hence
$$
\mathbb {P} (| \xi | > 4) \leq e ^ {- 4 ^ {2} / 2} / 4 \leq 0. 0 0 0 1. \tag {57}
$$
Lemma 2. Let  $u, v \in \mathbb{R}$ . Under Assumption (A1), we have
$$
\left| \mathbb {E} _ {u} \left[ \partial_ {z} U _ {z} \right| _ {z = v} \right] - \frac {e ^ {- u}}{1 + e ^ {- u}} \mid \leq \varepsilon \tag {58}
$$
where  $|\varepsilon |\leq 0.0002 + 2e^{-10}e^{|v|}$
Proof. The integral is exactly given by
$$
\frac {1}{1 + e ^ {- u}} \mathbb {E} \left[ \frac {e ^ {- v - \left(\xi_ {a} - b\right) ^ {2} / 2}}{e ^ {- U _ {v} \left(\xi_ {a}\right)}} \right] + \frac {e ^ {- u}}{1 + e ^ {- u}} \mathbb {E} \left[ \frac {e ^ {- v - \left(\xi_ {b} - b\right) ^ {2} / 2}}{e ^ {- U _ {v} \left(\xi_ {b}\right)}} \right] \tag {59}
$$
where  $\xi_{a},\xi_{b}$  denote two Gaussian random variables with respective means  $a,b$  . By (57),  $\xi_{a}$  and  $\xi_{b}$  are respectively contained in  $I_{a} = [a - 4,a + 4]$  and  $I_{b} = [b - 4,b + 4]$  with probability greater than 0.999. Let us examine the first term of (59). The fraction inside the expectation is always smaller than 1, hence by Lemma 1,
$$
\left. \partial_ {z} U _ {z} (\xi_ {a}) \right| _ {z = v} \leq \mathbf {1} _ {\xi_ {a} \notin I _ {a}} + \mathbf {1} _ {\xi_ {a} \in I _ {a}} \left. \partial_ {z} U _ {z} (\xi_ {a}) \right| _ {z = v} \leq \mathbf {1} _ {\xi_ {a} \notin I _ {a}} + e ^ {- v - 1 0}.
$$
Consequently, by (57), the expectation is smaller than  $0.0001 + e^{-v - 10}$ . The second expectation in (59) is equal to
$$
1 - \mathbb {E} \left[ \frac {e ^ {- (X _ {b} - a) ^ {2} / 2}}{e ^ {- U _ {v} (\xi_ {b})}} \right]
$$
and by the same kind of analysis, the expectation here is smaller than  $0.0001 + e^{v - 10}$ . Gathering the bounds yields the result.
In particular, the right hand side of (47) can be approximated by  $\frac{e^{-z(t)}}{1 + e^{-z(t)}} - \frac{e^{-z_*}}{1 + e^{-z_*}}$  up to an error term smaller than  $0.0004 + 2e^{-10}(e^{|z(t)|} + e^{|z_*|})$ . If  $z(t), z_*$  are contained in a small interval  $[-C, C]$  with, say,  $C < 5$ , this error term is uniformly small in time, and one might see (47) as a perturbation of the following system:
$$
\dot {z} (t) = \frac {e ^ {- z (t)}}{1 + e ^ {- z (t)}} - \frac {e ^ {- z _ {*}}}{1 + e ^ {- z _ {*}}}, \tag {60}
$$
a system with only one fixed point at  $z(t) = z_{*}$ , the ground-truth solution.
Mode collapse in absence of Jarzynski corection
Now let us analyze in a similar fashion the dynamics without reweighting. Here, (47) is replaced by
$$
\dot {z} (t) = \mathbb {E} _ {z (t)} [ \partial_ {z} U _ {z (t)} (X _ {t}) ] - \mathbb {E} _ {*} [ \partial_ {z} U _ {z (t)} ], \tag {61}
$$
where the process  $X_{t}$  solves
$$
d X _ {t} = - \alpha \nabla U _ {z (t)} (X _ {t}) d t + \sqrt {2 \alpha} d W _ {t}, \qquad X _ {0} \sim \rho_ {z (0)}
$$
The probability density function  $\rho(t, x)$  of  $X_{t}$  satisfies a Fokker-Planck equation
$$
\partial_ {t} \rho = \alpha \nabla \cdot (\nabla U _ {z (t)} (x) \rho + \nabla \rho), \quad \rho (t = 0) = \rho_ {z (0)}
$$
which, in full generality, is hard to solve exactly, and thus exact expressions for the first term of (61) are intractable. However, depending on whether  $X_0$  is close to  $a$  or  $b$ , the process  $X_{t}$  can be well approximated by an Ornstein-Uhlenbeck process, hence  $\rho(t,x)$  can itself be approximated by a Gaussian mixture.
Proposition 3. Suppose that (A1) holds and that
$$
\exists T, C \in \mathbb {R} _ {+} \text {s u c h t h a t f o r a l l} t \in [ 0, \alpha^ {- 1} T ], \quad z (t) \in [ - C, C ]. \tag {A2}
$$
Then one has  $D_{\mathrm{KL}}(\rho (0)|\rho (t))\leq \delta t$  where  $\delta = 0.000025 + 100e^{-20}e^{2C}$
In other words,  $\rho(t)$  is approximately constant, up to reasonable time scales  $t = O(1/\delta)$ .
Proof.  $X_0$  is drawn from  $\rho_{z(0)}$ , a Gaussian mixture; the probability of it being sampled from a Gaussian with mean  $a$  is  $e^{-z(0)} / (1 + e^{-z(0)}) = 1/2$ . We will work conditionally on this event  $\mathcal{E}_a$  and we will note  $\rho^a(t)$  the density of  $X_0$  conditional on  $\mathcal{E}^a$ ; thus,  $\rho^a(0) = \mathcal{N}(a, 1)$ . We set  $V(x) = \frac{1}{2}|x - a|^2$  so that  $\nabla V(x) = (x - a)$  and we consider the following Ornstein-Uhlenbeck process:
$$
d Y _ {t} = - \alpha \nabla V (Y _ {t}) d t + \sqrt {2 \alpha} d W _ {t}, \quad Y _ {0} = X _ {0}
$$
whose density will be denoted  $\tilde{\rho}^a (t)$ . We use classical bounds on the divergence between  $\rho^a (t)$  and  $\tilde{\rho}^a (t)$ . For example, the bounds in [74, Lemma 2.20] directly apply and yield
$$
D _ {\mathrm {K L}} \left(\tilde {\rho} ^ {a} (t) \mid \rho^ {a} (t)\right) \leq \frac {1}{4} \int_ {0} ^ {t} \mathbb {E} \left[ \left| \nabla U _ {z (s)} \left(Y _ {s}\right) - \nabla V \left(Y _ {s}\right) \right| ^ {2} \right] d s.
$$
Since  $Y_{t}$  is nothing but an Ornstein-Uhlenbeck at equilibrium,  $Y_{t} \sim \mathcal{N}(a,1)$  for all  $t \geq 0$ . The term inside the integral is a Gaussian expectation and will be shown to be small:
$$
\mathbb {E} \left[ | \nabla U _ {z (s)} \left(Y _ {s}\right) - \nabla V \left(Y _ {s}\right) | ^ {2} \right] \leq 0. 0 0 0 1 + 4 0 0 e ^ {- 2 z (t) - 2 0}. \tag {62}
$$
$$
D _ {\mathrm {K L}} (\tilde {\rho} ^ {a} (t) | \rho^ {a} (t)) \leq t \frac {0 . 0 0 0 1}{4} + 1 0 0 e ^ {- 2 0} \int_ {0} ^ {t} e ^ {- 2 z (s)} d s.
$$
Under (A2), the overall bound remains smaller than  $t$  times  $0.000025 + 100e^{-20}e^{2C}$  as requested, thus proving that  $\tilde{\rho}^a (t) = \mathcal{N}(a,1)$  and  $\rho^a (t)$  are close with the same quantitative bound
Similarly,  $\rho^b (t)$ , the density of  $X_{t}$  conditional on  $X_0$  being sampled from a Gaussian with mean  $b$ , is close to  $\mathcal{N}(b,1)$  with the same quantitative bounds.
Overall, using the chain rule for KL divergences,
$$
D _ {\mathrm {K L}} (\tilde {\rho} (t) | \rho (t)) \leq \mathbb {P} \left(\mathcal {E} _ {a}\right) D _ {\mathrm {K L}} \left(\tilde {\rho} ^ {a} (t) | \rho^ {a} (t)\right) + \mathbb {P} \left(\overline {{\mathcal {E} _ {a}}}\right) D _ {\mathrm {K L}} \left(\hat {\rho} ^ {b} (t) | \rho^ {b} (t)\right) \leq \delta t.
$$
In other words,  $\rho(t)$  is close to a mixture of two Gaussians with modes centered at  $a, b$ , and the probability of belonging to the first mode is the probability of  $X_0$  belonging to the first mode, that is,  $e^{-z(0)} / (1 + e^{-z(0)}) = 1/2$ .
Proof of (62). We have
$$
\nabla U _ {z} (x) = \frac {(x - a) e ^ {- | x - a | ^ {2} / 2} + (x - b) e ^ {- | x - b | ^ {2} / 2 - z}}{U _ {z} (x)}. \tag {63}
$$
Using Lemma 1 and the fact that if  $x \in I_{a}$  then  $|x - b| < 16$  and  $|x - a| < 4$ , we get  $|\nabla U_{v}(x) - (x - a)| \leq 20\varepsilon$  with  $\varepsilon \leq e^{-v - 10}$ . Consequently,
$$
\begin{array}{l} \mathbb {E} \left[ | \nabla U _ {z (s)} \left(Y _ {s}\right) - \nabla V \left(Y _ {s}\right) | ^ {2} \right] \leq \mathbb {P} \left(Y _ {t} \notin I _ {a}\right) + (2 0 e ^ {- v - 1 0}) ^ {2} \\ \leq 0. 0 0 0 1 + 4 0 0 e ^ {- 2 v - 2 0}. \\ \end{array}
$$
As a consequence of the Proposition 3, the first term of (61) can be approximated by  $\mathbb{E}_{z(0)}[\partial_z U_{z(t)}(X_t)] = \mathbb{E}_{z(0)}[\partial_z U_{z(t)}]$ , which in turn can be approximated by  $e^{-z(0)} / (1 + e^{-z(0)})$  thanks to (58). Overall, (61) is therefore a perturbation of the system
$$
\dot {z} (t) = \frac {e ^ {- z (0)}}{1 + e ^ {- z (0)}} - \frac {e ^ {- z _ {*}}}{1 + e ^ {- z _ {*}}} = \frac {1}{2} - q _ {*} =: \gamma .
$$
Since the right hand side no longer depends on  $z(t)$ , this system leads to a constant drift of  $z(t)$ , that is  $z(t) = \gamma t$ , leading to mode collapse since  $(1 + e^{-z(t)})^{-1} \approx (1 + e^{\gamma t})^{-1}$  goes to either 0 or 1.
Mode collapse for Contrastive-Divergence and Score-Matching
The continuous-time limit of Contrastive Divergence (Algorithm 3) is equivalent to Score-Matching minimization ([28]). The objective function becomes the Stein score,
$$
\begin{array}{l} \operatorname {S M} (z) = \mathbb {E} _ {*} [ | \nabla \log \rho_ {z} (X) - \nabla \log \rho_ {z _ {*}} (X) | ^ {2} ] \\ = \mathbb {E} _ {*} [ | \nabla U _ {z} (X) - \nabla U _ {z _ {*}} (X) | ^ {2} ], \\ \end{array}
$$
which is in theory intractable due to the presence of the unknown parameter  $z_{*}$ ; a well-known computation from [24] shows that the gradient  $\partial_z\mathrm{SM}(z)$  can be estimated using the training samples without resorting to  $z_{*}$ . Note that in this context, there are no 'walkers'.
Now, the dynamics (47) is replaced by
$$
\begin{array}{l} \dot {z} (t) = \partial_ {z} \mathbb {E} _ {*} [ | \nabla U _ {z (t)} (X) - \nabla U _ {z _ {*}} (X) | ^ {2} ] (64) \\ = p _ {*} \partial_ {z} \mathbb {E} \left[ \left| \nabla U _ {z (t)} (\xi_ {a}) - \nabla U _ {z _ {*}} (\xi_ {a}) \right| ^ {2} \right] + q _ {*} \partial_ {z} \mathbb {E} \left[ \left| \nabla U _ {z (t)} (\xi_ {b}) - \nabla U _ {z _ {*}} (\xi_ {b}) \right| ^ {2} \right] (65) \\ \end{array}
$$
where here again  $\xi_{x}\sim \mathcal{N}(x,1)$  . From (62) and the triangle inequality, we have
$$
\begin{array}{l} \left(\mathbb {E} \left[ | \nabla U _ {z (t)} (\xi_ {a}) - \nabla U _ {z _ {*}} (\xi_ {a}) | ^ {2} \right]\right) ^ {1 / 2} \leq \\ \left(\mathbb {E} \left[ | \nabla U _ {z (t)} (\xi_ {a}) - (\xi_ {a} - a) | ^ {2} \right]\right) ^ {1 / 2} + \left(\mathbb {E} \left[ | \nabla U _ {z _ {*}} (\xi_ {a}) - (\xi_ {a} - a) | ^ {2} \right]\right) ^ {1 / 2} \\ \leq 0. 0 0 0 2 + 8 0 0 e ^ {- 2 z (t) - 2 0}. \\ \end{array}
$$
and the same approximation holds for the second part in (65). Overall, we get that for any reasonable  $z$ ,  $\mathrm{SM}(z) \approx 0$ : that is, every  $z$  minimizes the score. A similar analysis leads to  $\partial_z \mathrm{SM}(z) \approx 0$ . Consequently,  $\dot{z}(t) \approx 0$ : Score Matching and Contrastive Divergence leads to 'no-learning'.
C.3 Empirical gradient descent analysis of (48)
The gradient descent (47) represented an ideal situation where the expectations  $\mathbb{E}_{z(t)},\mathbb{E}_{*}$  can be exactly analyzed. In practice, the two terms of (47) are estimated; the second term using a finite number of training data  $\{x_{*}^{i}\}$ , and the first one using a finite number of walkers  $\{X_{t}^{i}\}$ , with associated weights  $e^{A_t^i}$  which are either evolved using the Jarzynski rule, or simply set to 1 in the PCD algorithm. Our goal in this section is to explain how these finite-size approximations do not substantially modify the previous analysis and lead to the behaviour presented in C.1. For simplicity we keep the time continuous.
We recall (48):
$$
\dot {z} (t) = \frac {\sum_ {i = 1} ^ {N} e ^ {A _ {t} ^ {i}} \partial_ {z} U _ {z (t)} \left(X _ {t} ^ {i}\right)}{\sum_ {i = 1} ^ {N} e ^ {A _ {t} ^ {i}}} - \frac {\sum_ {i = 1} ^ {n} \partial_ {z} U _ {z (t)} \left(x _ {*} ^ {i}\right)}{n}. \tag {66}
$$
The dynamics with Jarzynski correction leads to the correct estimation of the empirical weights
The continuous-time dynamics of the walkers and weighs in our method is given by
$$
d X _ {t} ^ {i} = - \alpha \nabla U _ {z (t)} \left(X _ {t} ^ {i}\right) d t + \sqrt {2 \alpha} d W _ {t} \tag {67}
$$
$$
\dot {A} _ {t} ^ {i} = - \partial_ {z} U _ {z (t)} \left(X _ {t} ^ {i}\right) \dot {z} (t). \tag {68}
$$
Let  $\hat{n}_{*}^{a}$  be the number of training data in  $I_{a}$  and  $\hat{p}_{*} = \hat{n}_{*}^{a} / n$  their proportion, and similarly  $\hat{q}_{*}$  the proportion in  $I_{b}$ , and  $r = 1 - \hat{p}_{*} - \hat{q}_{*}$ . By elementary concentration results, the remainder  $1 - \hat{p}_{*} - \hat{q}_{*}$  can be neglected: with high probability, it is smaller than, eg, 0.0001. We will note  $\hat{z}_{*}$  the parameter satisfying  $\hat{q}_{*} = \frac{e^{-\hat{z}_{*}}}{1 + e^{-\hat{z}_{*}}}$ . Using Lemma 1, the second term in (66) is approximated by  $\hat{q}_{*}$ . Now let us turn to the first term in (66). Still using Lemma 1, we see that the first term in (66) is well approximated by
$$
\frac {\sum_ {i :} x _ {t} ^ {i} \in I _ {b} e ^ {A _ {t} ^ {i}}}{\sum_ {i = 1} ^ {n} e ^ {A _ {t} ^ {i}}}. \tag {69}
$$
The second equation in (67) entails  $e^{A_t^i} = \exp \left(-\int_0^t\partial_zU_{z(s)}(X_s^i)\dot{z} (s)ds\right)$ . Now let us use Lemma 1: if  $X_{s}^{i}\in I_{a}$ , then  $\partial_zU_{z(s)}(X_s^i)\approx 0$ . Conversely, if  $X_{s}^{i}\in I_{b}$ , then  $\partial_zU_{z(s)}(X_s^i)\approx 1$ . Moreover, Proposition 3 and its proof essentially show that if  $X_0^i$  belongs to the first well (close to  $a$ ), then with high probability so does  $X_{s}^{a}$  for every  $s$ , and in particular  $X_{s}^{a}\in I_{a}$  with high probability for every  $s$ . Consequently,
$$
e ^ {A _ {t} ^ {i}} \approx \exp \left(- \int_ {0} ^ {t} 0 d s\right) = 1 \quad \text {i f} X _ {0} ^ {i} \in I _ {a}, \tag {70}
$$
$$
e ^ {A _ {t} ^ {i}} \approx \exp \left(- \int_ {0} ^ {t} \dot {z} (s) d s\right) = \exp (- z (t) + z (0)) = \exp (- z (t)) \quad \text {i f} X _ {0} ^ {i} \in I _ {b}. \tag {71}
$$
As already explained, the dynamics (67) leaves approximately constant the number of walkers in both modes; consequently, the proportion  $\hat{q}(t)$  of walkers  $X_{t}^{i}$  in  $I_{b}$  remains well approximated by the initial proportion, which is  $\hat{q}(0)$ , and we obtain that (69) is well approximated by
$$
\frac {\hat {q} (0) e ^ {- z (t)}}{\hat {p} (0) + e ^ {- z (t)} \hat {q} (0)} \tag {72}
$$
where we noted  $\hat{p}(0) = 1 - \hat{q}(0)$ . Note that since  $z(0) = 0$ , with high probability  $\hat{p}(0)$  and  $\hat{q}(0)$  are close to  $1/2$ . The random variable  $\hat{p}(0)/\hat{q}(0)$  is thus close to 1.
Overall, we obtain that the system (66) is a perturbation of the following system: