SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Davide Carbone
Dipartimento di Scienze Matematiche, Politecnico di Torino
Istituto Nazionale di Fisica Nucleare, Sezione di Torino
davide.carbone@polito.it
Mengjian Hua
Courant Institute of Mathematical Sciences, New York University
mh5113@nyu.edu
Simon Coste
LPSM, Université Paris-Cité
simon.coste@u-paris.fr
Eric Vanden-Eijnden
eve2@nyu.edu
Energy-based models (EBMs) are generative models inspired by statistical physics with a wide range of applications in unsupervised learning. Their performance is well measured by the cross-entropy (CE) of the model distribution relative to the data distribution. Using the CE as the objective for training is however challenging because the computation of its gradient with respect to the model parameters requires sampling the model distribution. Here we show how results for nonequilibrium thermodynamics based on Jarzynski equality together with tools from sequential Monte-Carlo sampling can be used to perform this computation efficiently and avoid the uncontrolled approximations made using the standard contrastive divergence algorithm. Specifically, we introduce a modification of the unadjusted Langevin algorithm (ULA) in which each walker acquires a weight that enables the estimation of the gradient of the cross-entropy at any step during GD, thereby bypassing sampling biases induced by slow mixing of ULA. We illustrate these results with numerical experiments on Gaussian mixture distributions as well as the MNIST and CIFAR-10 datasets. We show that the proposed approach outperforms methods based on the contrastive divergence algorithm in all the considered situations.
Probabilistic models have become a key tool in generative artificial intelligence (AI) and unsupervised learning. Their goal is twofold: explain the training data, and allow the synthesis of new samples. Many flavors have been introduced in the last decades, including variational auto-encoders [1, 2, 3] generative adversarial networks [4, 5], normalizing flows [6, 7, 8, 9, 10], diffusion-based models [11, 12, 13], restricted Boltzmann machines [14, 15, 16], and energy-based models (EBMs) [17, 18, 19].
Inspired by physics, EBMs are unnormalized probability models, specified via an energy function  $U$ , with the underlying probability density function (PDF) defined as  $\rho = \exp(-U)/Z$ , where  $Z$  is a normalization constant: this constant is often intractable but crucially it is not needed for data generation via Monte-Carlo sampling. In statistical physics, such PDFs are called Boltzmann-Gibbs densities [20, 21, 22] and their energy function is often known in advance; in the context of EBMs, the aim is to estimate this energy function in some parametric class using the available data. Such models benefit from a century of intuitions from computational physics to guide the design of the energy  $U$ , and help the sampling; hence, interpretability is a clear strength of EBMs, since in many applications the parameters of the model have a direct meaning. Unfortunately, training EBMS is a challenging task as it typically requires to sample complex multimodal (i.e. non-log-concave) distributions in high-dimension.
This issue arises because real-world datasets are often clustered into different modes with imbalanced weights. A skewed estimation of these weights can lead to harmful biases when it comes to applications: for example, under-representation of elements from a particular population. It is important to ensure that these biases are either avoided or compensated, leading to methods that ensure fairness ([23]); however, the training routines for EBMs often have a hard time properly learning the weights of different modes, and can often completely fail at this task.
Two popular classes of methods are used to train an EBM. In the first, the Fisher divergence between the model and the data distributions is taken as the objective to minimize, which amounts to learning the gradient of the energy function,  $\nabla U = -\nabla \log \rho$ . Although computationally efficient due to score-matching techniques [24], methods in this class are provably unable to learn the relative weights of modes when they are separated by low-density regions.
In the second class of methods, the cross-entropy between the model and the data distributions is used as the objective. Unlike the Fisher divergence, the cross-entropy is sensitive to the relative weights in multimodal distributions, but it unfortunately leads to training algorithms that are less justified theoretically and more delicate to use in practice. Indeed, gradient-based optimization procedures on the cross-entropy require sampling from the model distribution using Markov-Chain Monte-Carlo (MCMC) methods or the unadjusted Langevin algorithm (ULA) until mixing, which in a high-dimensional context or without log-concavity can be prohibitive, even considering the large computational power available today.
As a result, one typically needs to resort to approximations to estimate the gradient of the cross-entropy, for example by using the contrastive divergence (CD) or the persistent contrastive divergence (PCD) algorithms, see [25, 26, 27]. Unfortunately, the approximations made in these algorithms are uncontrolled and they are known to induce biases similar to those observed with score-based methods (the CD algorithm reduces to gradient descent over the Fisher divergence in the limit of frequent resetting from the data [28]). Many techniques have been proposed to overcome this issue, for example, based on MCMC sampling [29, 30, 31] or on various regularizations of CD [32, 33]. In practice though, these techniques still do not handle well multimodal distributions and they come with little theoretical guarantees.
Main contributions. In this work, we go back to the original problem of training EBMS using the cross-entropy between the model and the data distributions as objective, and:
- We derive exact expressions for the cross-entropy and its gradient that involve expectations over an ensemble of weighted walkers whose weights and positions evolve concurrently with the model energy during optimization. Our main tool is Jarzynski equality [34], an exact relation between the normalizing constants of two distributions linked by an out-of-equilibrium dynamics.
- Based on these formulas, we design sequential Monte Carlo sampling procedures [35] to estimate the cross-entropy and its gradient in a practical way at essentially no additional computational cost compared to using the CD or PCD algorithms with ULA.
- We show that reweighting the walkers is necessary in general and that training procedures based on the CD algorithm lead to uncontrolled biases whereas those based on using the PCD algorithm lead to mode collapse in general.
- We illustrate these results numerically on synthetic examples involving Gaussian mixture models where these effects can be demonstrated.
- We also apply our method on the MNIST and CIFAR-10 datasets. In the first, we intentionally bias the proportion of the different digits, to demonstrate that our method allows the retrieval of these proportions whereas the other methods fail to do so.
Related works. How to train EBMs is a longstanding question, and we refer e.g. to [17, 36] for overviews on this topic. The use of modern deep neural architectures to model the energy was then proposed in [37], and a relation between EBMs and classifiers has been highlighted in [38]. How to sample from unnormalized probability models is also an old and rich problem, see [39, 40] for general introductions.
Score-matching techniques and variants originate from [24, 41, 42]; their shortcoming in the context of EBM training is investigated in [43] and their blindness to the presence of multiple, imbalanced modes in the target density has been known for long: we refer to [44, 45] for discussions. Contrastive divergence (CD) algorithms originate from [25, 26, 27]. These methods only perform one or a few sampling steps of the algorithm, with random walkers that are repeatedly restarted at the data points. Persistent contrastive divergence (PCD) algorithm, introduced in [29], eliminates the restarts and evolves walkers using ULA. Unlike the approach proposed in this paper, these methods are known to give estimates of the gradient of the cross-entropy that have uncontrolled biases which are difficult to remove; there were many attempts in this direction, also in cooperation with other unsupervised techniques, see e.g. [46, 47, 48, 49, 50, 51, 52].
It is worth noting that the original papers on CD proposed to use an objective which is different from the cross-entropy, and their proposed implementation (what we call the CD algorithm) does not perform gradient descent on this objective, due to some gradient terms being neglected [33]. This sparked some debate about which objective, if any, was actually minimized by CD algorithms; for example, [53, 54] showed that the CD and PCD algorithms are essentially adversarial procedures; [33] introduced a way to approximate the missing term in the gradient of the CD objective; and [28, 55] showed that in the limit of small noise, CD is essentially equivalent to score matching. In contrast, there is no ambiguity about the objective used in the method we propose: it is always the cross-entropy.
Jarzynski equality (JE) was introduced in [34] and gives an exact expression relating the normalizing constants (or equivalently free energy [22]) of two distributions linked by out-of-equilibrium continuous-time dynamics. A discrete-time analog of JE is used in Neal's annealed importance sampling [56], which belongs to the framework of sequential Monte-Carlo methods [35]. These methods have been used in the context of generative models based on variational autoencoders [57, 58], normalizing flows [59, 60], and diffusion-based models [61, 62]. In contrast, here we use sequential Monte-Carlo method to train EBM on the cross-entropy directly.
2 Energy-Based Models
Setup, notations, and assumptions. The problem we consider can be formulated as follows: we assume that we are given  $n \in \mathbb{N}$  data points  $\{x_{i}^{*}\}_{i=1}^{n}$  in  $\mathbb{R}^d$  drawn from an unknown probability distribution that is absolutely continuous with respect to the Lebesgue measure on  $\mathbb{R}^d$ , with a positive probability density function (PDF)  $\rho_*(x) > 0$  (also unknown). Our aim is to estimate this PDF via an energy-based model (EBM), i.e., to find a suitable energy function in a parametric class,  $U_\theta : \mathbb{R}^d \to [0, \infty)$  with parameters  $\theta \in \Theta$ , such that the associated Boltzmann-Gibbs PDF
$$
\rho_ {\theta} (x) = Z _ {\theta} ^ {- 1} e ^ {- U _ {\theta} (x)}; \quad Z _ {\theta} = \int_ {\mathbb {R} ^ {d}} e ^ {- U _ {\theta} (x)} d x \tag {1}
$$
is an approximation of the target density  $\rho_{*}(x)$ . The normalization factor  $Z_{\theta}$  is known as the partition function in statistical physics [22]. This factor is hard to estimate and one advantage of EBMs is that they provide generative models that do not require the explicit knowledge of  $Z_{\theta}$  since Markov Chain Monte-Carlo (MCMC) methods can in principle be used to sample  $\rho_{\theta}$  knowing only  $U_{\theta}$  - the design of such MCMC methods is an integral part of the problem of building an EBM.
To proceed we will assume that the parametric class of energy we use is such that, for all  $\theta \in \Theta$ ,
$$
U _ {\theta} \in C ^ {2} (\mathbb {R} ^ {d}); \qquad \exists L \in \mathbb {R} _ {+}: \| \nabla \nabla U _ {\theta} (x) \| \leq L \quad \forall x \in \mathbb {R} ^ {d};
$$
$$
\exists a \in \mathbb {R} _ {+} \text {a n d a c o m p a c t s e t} \mathcal {C} \in \mathbb {R} ^ {d}: x \cdot \nabla U _ {\theta} (x) \geq a | x | ^ {2} \quad \forall x \in \mathbb {R} ^ {d} \setminus \mathcal {C}.
$$
These assumptions guarantee that  $Z_{\theta} < \infty$  (i.e. we can associate a PDF  $\rho_{\theta}$  to  $U_{\theta}$  via (1) for any  $\theta \in \Theta$ ) and that the Langevin equations as well as their time-discretized versions we will use to sample  $\rho_{\theta}$  have global solutions and are ergodic [63, 64, 65]. We stress that (2) does not imply that  $U_{\theta}$  is convex (i.e. that  $\rho_{\theta}$  is log-concave): in fact, we will be most interested in situations where  $U_{\theta}$  has multiple local minima so that  $\rho_{\theta}$  is multimodal. For simplicity we will also assume that  $\rho_{*}$  is in the parametric class, i.e.  $\exists \theta_{*} \in \Theta : \rho_{\theta_{*}} = \rho_{*}$ . Our aims are primarily to identify this  $\theta_{*}$  and to sample  $\rho_{\theta_{*}}$ ; in the process, we will also show how to estimate  $Z_{\theta_{*}}$ .
Cross-entropy minimization. To measure the quality of the EBM and train its parameters one can use the cross-entropy of the model density  $\rho_{\theta}$  relative to the target density  $\rho_{*}$
$$
H \left(\rho_ {\theta}, \rho_ {*}\right) = - \int_ {\mathbb {R} ^ {d}} \log \rho_ {\theta} (x) \rho_ {*} (x) d x = \log Z _ {\theta} + \int_ {\mathbb {R} ^ {d}} U _ {\theta} (x) \rho_ {*} (x) d x \tag {3}
$$
where we used the definition of  $\rho_{\theta}$  in (1) to get the second equality. The cross-entropy is related to the Kullback-Leibler divergence via  $H(\rho_{\theta},\rho_{*}) = H(\rho_{*}) + D_{\mathrm{KL}}(\rho_{*}||\rho_{\theta})$ , where  $H(\rho_{*})$  is the entropy of  $\rho_{*}$ , and its gradient with respect to the parameter  $\theta$  can be calculated using the identity  $\partial_{\theta}\log Z_{\theta} = -\int_{\mathbb{R}^d}\partial_{\theta}U_{\theta}(x)\rho_{\theta}(x)dx$ , to obtain
$$
\begin{array}{l} \partial_ {\theta} H \left(\rho_ {\theta}, \rho_ {*}\right) = \int_ {\mathbb {R} ^ {d}} \partial_ {\theta} U _ {\theta} (x) \rho_ {*} (x) d x - \int_ {\mathbb {R} ^ {d}} \partial_ {\theta} U _ {\theta} (x) \rho_ {\theta} (x) d x \tag {4} \\ \equiv \mathbb {E} _ {*} [ \partial_ {\theta} U _ {\theta} ] - \mathbb {E} _ {\theta} [ \partial_ {\theta} U _ {\theta} ]. \\ \end{array}
$$
The cross-entropy is more stringent, and therefore better, than other objectives like the Fisher divergence: for example, unlike the latter, it is sensitive to the relative probability weights of modes on  $\rho_{*}$  separated by low-density regions [36]. Unfortunately, the cross entropy is also much harder to use in practice since evaluating it requires estimating  $Z_{\theta}$ , and evaluating its gradient requires calculating the expectation  $\mathbb{E}_{\theta}[\partial_{\theta}U_{\theta}]$  (in contrast  $\mathbb{E}_{*}[U_{\theta}]$  and  $\mathbb{E}_{*}[\partial_{\theta}U_{\theta}]$  can be readily estimated on the data). Typical training methods, e.g. based on the CD or the PCD algorithms, give up on estimating  $Z_{\theta}$  and resort to various approximations to calculate the expectation  $\mathbb{E}_{\theta}[\partial_{\theta}U_{\theta}]$ —see Appendix A.5 for more discussion about these methods. While these approaches have proven successful in many situations, they are prone to training instabilities that limit their applicability. They also come with no theoretical guarantees in terms of convergence.
3 Training via sequential Monte-Carlo methods based on Jarzynski equality
In this section we use tools from nonequilibrium statistical mechanics [34, 56] to write exact expressions for both  $\mathbb{E}_{\theta}[\partial_{\theta}U_{\theta}]$  and  $Z_{\theta}$  (Sec 3.1) that are amenable to empirical estimation via sequential Monte-Carlo methods [35], thereby enabling gradient descent-type algorithms for the optimization of EBMs (Sec. 3.2).
3.1 Jarzynski equality in discrete-time
Proposition 1. Assume that the parameters  $\theta$  are evolved by some time-discrete protocol  $\{\theta_k\}_{k\in \mathbb{N}_0}$  and that (2) hold. Given any  $h\in (0,L)$ , let  $X_{k}\in \mathbb{R}^{d}$  and  $A_{k}\in \mathbb{R}$  be given by the iteration rule
$$
\left\{ \begin{array}{l l} X _ {k + 1} = X _ {k} - h \nabla U _ {\theta_ {k}} \left(X _ {k}\right) + \sqrt {2 h} \xi_ {k}, & X _ {0} \sim \rho_ {\theta_ {0}}, \\ A _ {k + 1} = A _ {k} - \alpha_ {k + 1} \left(X _ {k + 1}, X _ {k}\right) + \alpha_ {k} \left(X _ {k}, X _ {k + 1}\right), & A _ {0} = 0, \end{array} \right. \tag {5}
$$
where  $U_{\theta}(x)$  is the model energy,  $\{\xi_k\}_{k\in \mathbb{N}_0}$  are independent  $N(0_d,I_d)$ , and we defined
$$
\alpha_ {k} (x, y) = U _ {\theta_ {k}} (x) + \frac {1}{2} (y - x) \cdot \nabla U _ {\theta_ {k}} (x) + \frac {1}{4} h | \nabla U _ {\theta_ {k}} (x) | ^ {2} \tag {6}
$$
Then, for all  $k\in \mathbb{N}_0$
$$
\mathbb {E} _ {\theta_ {k}} \left[ \partial_ {\theta} U _ {\theta_ {k}} \right] = \frac {\mathbb {E} \left[ \partial_ {\theta} U _ {\theta_ {k}} \left(X _ {k}\right) e ^ {A _ {k}} \right]}{\mathbb {E} \left[ e ^ {A _ {k}} \right]}, \quad Z _ {\theta_ {k}} = Z _ {\theta_ {0}} \mathbb {E} \left[ e ^ {A _ {k}} \right] \tag {7}
$$
where the expectations on the right-hand side are over the law of the joint process  $(X_{k},A_{k})$
Algorithm 1 Sequential Monte-Carlo training with Jarzynski correction
1: Inputs: data points  $\{x_{*}^{i}\}_{i = 1}^{n}$ ; energy model  $U_{\theta}$ ; optimizer step  $\mathrm{opt}(\theta ,\mathcal{D})$  using  $\theta$  and the empirical CE gradient  $\mathcal{D}$ ; initial parameters  $\theta_0$ ; number of walkers  $N\in \mathbb{N}_0$ ; set of walkers  $\{X_0^i\}_{i = 1}^N$  sampled from  $\rho_{\theta_0}$ ; total duration  $K\in \mathbb{N}$ ; ULA time step  $h$ ; set of positive constants  $\{c_k\}_{k\in \mathbb{N}}$ .
2:  $A_0^i = 0$  for  $i = 1,\dots ,N$
3: for  $k = 0, \dots, K - 1$  do
4:  $p_k^i = \exp (A_k^i) / \sum_{j = 1}^N\exp (A_k^j)$  ▷ normalized weights
5:  $\tilde{\mathcal{D}}_k = \sum_{i=1}^N p_k^i \partial_\theta U_{\theta_k}(X_k^i) - n^{-1} \sum_{j=1}^n \partial_\theta U_{\theta_k}(x_*^j)$
6:  $\theta_{k + 1} = \mathrm{opt}(\theta_k,\tilde{\mathcal{D}}_k)$  optimization step
7: for  $i = 1,\dots,N$  do
8:  $X_{k + 1}^{i} = X_{k}^{i} - h\nabla U_{\theta_{k}}(X_{k}^{i}) + \sqrt{2h}\xi_{k}^{i},\qquad \xi_{k}^{i}\sim \mathcal{N}(0_{d},I_{d})$
9:  $A_{k + 1}^{i} = A_{k}^{i} - \alpha_{k + 1}(X_{k + 1}^{i},X_{k}^{i}) + \alpha_{k}(X_{k}^{i},X_{k + 1}^{i})$ $\triangleright$  weight update
10: end for
11: Resample the walkers and reset the weights if  $\mathrm{ESS}_{k + 1} < c_{k + 1}$ , see (13).  $>$  resampling step
12: end for
13: **Outputs:** Optimized energy  $U_{\theta_K}$ ; set of weighted walkers  $\{X_K^i, A_K^i\}_{i=1}^N$  sampling  $\rho_{\theta_K}$ ; partition function estimate  $\tilde{Z}_{\theta_K} = Z_{\theta_0} N^{-1} \sum_{i=1}^N \exp(A_K^i)$ ; CE estimate  $\log \tilde{Z}_{\theta_K} + n^{-1} \sum_{j=1}^n U_{\theta_K}(x_*^j)$
The proof of the proposition is given in Appendix A.2: for completeness we also give a continuous-time version of this proposition in Appendix A.1. We stress that the inclusion of the weights in (7) is key, as  $\mathbb{E}[\partial_{\theta}U_{\theta_k}(X_k)] \neq \mathbb{E}_{\theta_k}[\partial_{\theta}U_{\theta_k}]$  in general. We also stress that (7) holds exactly despite the fact that for  $h > 0$  the iteration step for  $X_{k}$  in (5) is that of the unadjusted Langevin algorithm (ULA) with no Metropolis correction as in MALA [66, 67]. That is, the inclusion of the weights  $A_{k}$  exactly corrects for the biases induced by both the slow mixing and the time-discretization errors in ULA.
Proposition 1 shows that we can evolve the parameters by gradient descent over the cross-entropy by solving (5) concurrently with
$$
\theta_ {k + 1} = \theta_ {k} + \gamma_ {k} \mathcal {D} _ {k}, \quad \mathcal {D} _ {k} = - \partial_ {\theta} H \left(\rho_ {\theta_ {k}}, \rho_ {*}\right) = \frac {\mathbb {E} \left[ \partial_ {\theta} U _ {\theta_ {k}} \left(X _ {k}\right) e ^ {A _ {k}} \right]}{\mathbb {E} \left[ e ^ {A _ {k}} \right]} - \mathbb {E} _ {*} \left[ \partial_ {\theta} U _ {\theta_ {k}} \right], \tag {8}
$$
where  $\gamma_{k} > 0$  is the learning rate and  $k\in \mathbb{N}_0$  with  $\theta_0$  given. We can also replace the gradient step for  $\theta_{k}$  in (8) by any update optimization step (via AdaGrad, ADAM, etc.) that uses as input the gradient  $\mathcal{D}_k$  of the cross-entropy evaluated at  $\theta_{k}$  to get  $\theta_{k + 1}$ . Assuming that we know  $Z_{\theta_0}$  we can track the evolution of the cross-entropy via
$$
H \left(\rho_ {\theta_ {k}}, \rho_ {*}\right) = \log \mathbb {E} \left[ e ^ {A _ {k}} \right] + \log Z _ {\theta_ {0}} + \mathbb {E} _ {*} \left[ U _ {\theta_ {k}} \right]. \tag {9}
$$
3.2 Practical implementation
Empirical estimators and optimization step. We introduce  $N$  independent pairs of walkers and weights,  $\{X_k^i,A_k^i\}_{i = 1}^N$ , which we evolve independently using (5) for each pair. To evolve  $\theta_{k}$  from some prescribed  $\theta_0$  we can then use the empirical version of (8):
$$
\theta_ {k + 1} = \theta_ {k} + \gamma_ {k} \tilde {\mathcal {D}} _ {k}, \tag {10}
$$
where  $\hat{D}_k$  is the estimator for the gradient in  $\theta$  of the cross-entropy:
$$
\tilde {\mathcal {D}} _ {k} = \frac {\sum_ {i = 1} ^ {N} \partial_ {\theta} U _ {\theta_ {k}} \left(X _ {k} ^ {i}\right) \exp \left(A _ {k} ^ {i}\right)}{\sum_ {i = 1} ^ {N} \exp \left(A _ {k} ^ {i}\right)} - \frac {1}{n} \sum_ {j = 1} ^ {n} \partial_ {\theta} U _ {\theta_ {k}} \left(x _ {*} ^ {j}\right), \tag {11}
$$
These steps are summarized in Algorithm 1, which is a specific instance of a sequential Monte-Carlo algorithm. We can also use mini-batches of  $\{X_k^i, A_k^i\}_{i=1}^N$  and the data set  $\{x_*^j\}_{j=1}^n$  at every iteration (see Algorithm 2 in Appendix A.3), and switch to any optimizer step that uses  $\theta_k$  and  $\mathcal{D}_k$  as input to get the updated  $\theta_{k+1}$ . During the calculation, we can monitor the evolution of the partition function and the cross-entropy using as estimators
$$
\tilde {Z} _ {\theta_ {k}} = Z _ {\theta_ {0}} \frac {1}{N} \sum_ {i = 1} ^ {N} \exp \left(A _ {k} ^ {i}\right), \quad \tilde {H} _ {k} = \log \tilde {Z} _ {\theta_ {k}} + \frac {1}{n} \sum_ {j = 1} ^ {n} U _ {\theta_ {K}} \left(x _ {*} ^ {j}\right) \tag {12}
$$
These steps are summarized in Algorithm 1, which is a specific instance of a sequential Monte-Carlo algorithm. We adapted our routine to mini-batches in Algorithm 2 without any explicit additional source of error: in fact, the particles outside the mini-batch have their weights updated too. The only sources of error in these algorithms come from the finite sample sizes,  $N < \infty$  and  $n < \infty$ . Regarding  $n$ , we may need to add a regularization term in the loss to avoid overfitting: this is standard. Regarding  $N$ , we need to make sure that the effective sample size of the walkers remains sufficient during the evolution. This is nontrivial since the  $A_{k}^{i}$ 's will spread away from zero during the optimization, implying that the weights  $\exp(A_{k}^{i})$  will become non-uniform, thereby reducing the effective sample size. This is a known issue with sequential Monte-Carlo algorithms that can be alleviated by resampling as discussed next.
Resampling step. A standard quantity to monitor the effective sample size [40] is the ratio between the square of the empirical mean of the weights and their empirical variance, i.e.
$$
\mathrm {E S S} _ {k} = \frac {\left(N ^ {- 1} \sum_ {i = 1} ^ {N} \exp \left(A _ {k} ^ {i}\right)\right) ^ {2}}{N ^ {- 1} \sum_ {i = 1} ^ {N} \exp \left(2 A _ {k} ^ {i}\right)} \in (0, 1 ] \tag {13}
$$
The effective sample size of the  $N$  walkers is  $\mathrm{ESS}_kN$ . Initially, since  $A_0^i = 0$ ,  $\mathrm{ESS}_0 = 1$ , but it decreases with  $k$ . At each iteration  $k_r$  such that  $\mathrm{ESS}_{k_r} < c_{k_r}$ , where  $\{c_k\}_{k\in \mathbb{N}}$  is a set of predefined positive constants in  $(0,1)$ , we then:
1. Resample the walkers  $X_{k_r}^i$  using  $p_{k_r}^i = e^{A_{k_r}^i} / \sum_{j=1}^N e^{A_{k_r}^j}$  as probability to pick walker  $i$ ;
2. Reset  $A_{k_r}^i = 0$
3. Use the update  $Z_{\theta_k} = Z_{\theta_{k_r}}N^{-1}\sum_{i = 1}^N\exp (A_k^i)$  for  $k\geq k_r$  until the next resampling step.
This resampling is standard [35] and can be done with various levels of sophistication, as discussed in Appendix A.4. Other criteria, based e.g. on the entropy of the weights, are also possible, see e.g. [68].