SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Afra Amini $^{1}$  Li Du $^{2}$  Ryan Cotterell $^{1}$
$^{1}$ ETH Zürich  $^{2}$ Johns Hopkins University
{afra.amini, ryan.cotterell}@inf.ethz.ch
leodu@cs.jhu.edu
Gradient-based sampling algorithms have demonstrated their effectiveness in text generation, especially in the context of controlled text generation. However, there exists a lack of theoretically grounded and principled approaches for this task. In this paper, we take an important step toward building a principled approach for sampling from language models with gradient-based methods. We use discrete distributions given by language models to define densities and develop an algorithm based on Hamiltonian Monte Carlo to sample from them. We name our gradient-based technique Structured Voronoi Sampling (Svs). In an experimental setup where the reference distribution is known, we show that the empirical distribution of svs samples is closer to the reference distribution compared to alternative sampling schemes. Furthermore, in a controlled generation task, svs is able to generate fluent and diverse samples while following the control targets significantly better than other methods.
https://github.com/AfraAmini/svs
Gradient-based sampling algorithms such as Hamiltonian Monte Carlo [HMC; 32] and Langevin dynamics [46] are widely used in Bayesian inference due to their efficiency in drawing samples from high-dimensional space [4]. Such algorithms construct a Markov Chain that has the desired distribution as its stationary distribution and use the gradient information of this distribution to efficiently navigate the state space. Additionally, gradient-based sampling schemes have recently been deployed in computer vision to generate high-quality images from state-of-the-art models [13, 43], and are a popular choice for tasks such as image synthesis [5] and image-to-image translation [40].
In natural language processing, there have been several attempts to apply gradient-based sampling techniques to sampling text from neural language models [21, 23, 38]. The motivation behind this approach to text generation is to sample from energy-based probabilistic models, where the normalization factor is not tractable. One such case is in controlled text generation, where energy functions are usually defined as a linear combination of LM probabilities and the probability of satisfying a set of predefined constraints [21, 23, 38]. In contrast to computer vision, however, applying gradient-based sampling schemes to text generation is nuanced as text, in contrast to images, is discrete.
Upon closer inspection, none of the proposed algorithms actually defines a valid Markov Chain Monte Carlo [MCMC; 12, 29] scheme that will draw samples from the model in the limit. For instance, Qin et al. [38] relax the language model from a distribution over strings to a distribution over logits. While the relaxation does transform the language model into a continuous distribution, it introduces bias. Kumar et al. [MUCoLA; 21] take a different approach. They derive a constrained gradient-based
sampler where the constraint is enforced through a projection. However, the projection invalidates the MCMC procedures, leading to an algorithm without guarantees. $^{1}$
We derive a principled Hamiltonian Monte Carlo scheme for generating text from language models, which we call a structured Voronoi sampler. Our scheme consists of two steps. First, we give a recipe for encoding discrete distributions as densities over  $\mathbb{R}^d$ ; we term the resulting encoding Voronoi measures. Second, we derive a refractive Hamiltonian Monte Carlo algorithm [30] for sampling from an arbitrary Voronoi measure. In our theoretical analysis, we show that, despite the presence of discontinuities, we are able to give proof that our sampler satisfies the detailed balance condition and, thus, is a correct MCMC scheme.
To empirically evaluate the performance of structured Voronoi sampling, we begin by applying it to a toy example where the exact reference probability distribution is known. We compare the empirical distribution of drawn samples with the reference distribution and show Voronoi sampler's distribution is closer to the reference distribution than MUCoLA or unconstrained HMC. Furthermore, we use our sampling scheme for controlled generation, where the goal is to use GPT-2 to generate restaurant reviews for a target type of food, e.g., Italian, Fast food, or Japanese, and separately to generate text with a positive sentiment. We find that structured Voronoi sampling outperforms FUDGE [48], MUCoLA, and Langevin Dynamics algorithms in terms of adhering to the control target. Additionally, the samples generated by structured Voronoi sampling are comparably fluent and diverse to those produced by the other methods.
2 Language Models
Let  $\Sigma$  be an alphabet, a finite, non-empty set. By  $\Sigma^{*} \stackrel{\mathrm{def}}{=} \bigcup_{n=0}^{\infty} \Sigma^{n}$ , we denote the Kleene closure of  $\Sigma$ . A probability distribution over  $\Sigma^{*}$  is called a language model (LM). The elements of  $\Sigma$  may be characters, subword pieces, or words; the choice lies with the modeler. Language models can be factored autoregressively by means of the chain rule of probability, i.e., for any string  $w = w_{1} \cdots w_{N} \in \Sigma^{*}$ , we can write
$$
p (\boldsymbol {w}) = p \left(\operatorname {E O S} \mid \boldsymbol {w}\right) \prod_ {n = 1} ^ {N} p \left(w _ {n} \mid \boldsymbol {w} _ {<   n}\right), \tag {1}
$$
where EOS  $\notin \Sigma$  is a distinguished end-of-sequence token and  $\pmb{w}_{<n}$  is the prefix of length  $(n-1)$  of the string  $\pmb{w}$ . We define  $\overline{\Sigma} \stackrel{\mathrm{def}}{=} \Sigma \cup \{\mathrm{EOS}\}$ , and require the conditional distributions  $p(\cdot \mid \pmb{w}_{<n})$  to be defined over  $\overline{\Sigma}$ . While all language models can be factored autoregressively, not all conditionals  $p(\cdot \mid \pmb{w}_{<n})$  can be assembled into a language model. In some cases, probability mass must be placed on infinite sequences [6]. In this work, we assume working with tight language models, i.e., that they indeed define valid probability distributions over  $\Sigma^{*}$ .
2.1 Language Modeling with Embeddings
Most neural language models make use of embeddings. Moreover, in most language models, the weights are shared between the language model head and the embedding layer. Such an embedding-based language model is defined as follows
$$
p \left(w _ {n} \mid \boldsymbol {w} _ {<   n}\right) \stackrel {\text {d e f}} {=} \frac {\exp \mathbf {v} _ {w _ {n}} \cdot \mathsf {e n c} \left(\boldsymbol {w} _ {<   n}\right)}{\sum_ {w \in \bar {\Sigma}} \exp \mathbf {v} _ {w} \cdot \mathsf {e n c} \left(\boldsymbol {w} _ {<   n}\right)}, \tag {2}
$$
where  $\mathbf{v}_w\in \mathbb{R}^d$  is the embedding of  $w$  and  $\mathsf{enc}:\Sigma^{*}\to \mathbb{R}^{d}$  is a real-valued encoding of the context. Notably, the context embedding  $\mathsf{enc}(\pmb{w}_{<  n})\in \mathbb{R}^{d}$  is obtained by inputting the context  $\pmb{w}_{<  n}$  into the language model, converting it to embeddings  $\mathbf{v}_{\pmb{w}_{<  n}}$  , passing it through neural network layers, and extracting the encoding from the model at position  $n - 1$
<sup>1</sup>In fact, a similar projection step is often applied in computer vision applications, which motivated Lou and Ermon [26] to develop a principled approach that avoids projection.
2We call the measures structured Voronoi samplers when the state face is larger and factored.
3We define  $\Sigma^0\stackrel {\mathrm{def}}{=}\{\varepsilon \}$
Interestingly, one can lift an embedding-based language model to be a distribution over the set of base embeddings:  $\mathcal{B} = \{\mathbf{v}_w : w \in \overline{\Sigma}\}$ . Specifically, we can associate a sequence of embeddings  $\mathbf{V} = [\mathbf{v}_1, \dots, \mathbf{v}_N]$  to any string  $\boldsymbol{w} = w_1 \cdots w_N$ . Substituting each  $w$  with the corresponding  $\mathbf{v}_w$ , we can rewrite Eq. (1) as
$$
p (\mathbf {V}) = p \left(\mathbf {v} _ {\mathrm {E O S}} \mid \mathbf {V}\right) \prod_ {n = 1} ^ {N} p \left(\mathbf {v} _ {n} \mid \mathbf {V} _ {<   n}\right). \tag {3}
$$
This transformation encodes a language model as a distribution over real-valued vectors. However,  $p(\mathbf{V})$  only places a positive probability on a countable set, and is zero everywhere else.
2.2 Controlled Language Modeling with Embeddings
In a controlled generation task, we are interested in a subset of strings  $\mathbf{w}$  that have a target property  $t$ . Therefore, we want to model and sample from a conditional distribution  $p(\mathbf{w} \mid t)$ , e.g., sample a sentence given a topic. Following Bayes' rule, one can write  $p(\mathbf{w} \mid t) \propto p(t \mid \mathbf{w}) p(\mathbf{w})$ . Previous papers model  $p(t \mid \mathbf{w})$  with an embedding-augmented classifier. Such a classifier receives embeddings  $\mathbf{V}$  associated with  $\mathbf{w}$  and predicts the probability of the target  $t$ . Notably, if the classifier and the LM share the same base embeddings  $\mathcal{B}$ , the controlled LM can also be lifted as a distribution over the base embeddings
$$
p (\mathbf {V} \mid t) = \frac {1}{Z _ {t}} p (t \mid \mathbf {V}) p (\mathbf {V}), \tag {4}
$$
where  $Z_{t} = \sum_{\mathbf{V}}p(t\mid \mathbf{V})p(\mathbf{V})$  is an intractable normalization factor, that sums over the embeddings of all possible strings.4 Identically to  $p(\mathbf{V}), p(\mathbf{V}\mid t)$  only places a positive probability on a countable set.
3 Voronoi Measures
In this section, we demonstrate how to encode an embedding-based language model as a density that places positive probability on a set with a measure greater than zero. Such encoding allows us to derive a principled gradient-based sampling approach to generate samples in §6. We start with some definitions.
Definition 1. An embedding-augmented probability distribution over the first  $M$  positive integers  $[M]$  is an array  $\mathbf{p} = [p_1, \ldots, p_M]$  such that  $p_m \geq 0$  and  $\sum_{m=1}^{M} p_m = 1$  where we assume that there is a real-valued embedding  $\{\mathbf{v}_m\}_{m=1}^M \subset \mathbb{R}^d$  associated with each  $m \in [M]$ .
Embedding-augmented distributions can be viewed as densities over  $\mathbb{R}^d$  using the following simple encoding
$$
p (\mathbf {x}) = \left\{ \begin{array}{l l} p _ {m}, & \text {i f} \mathbf {x} = \mathbf {v} _ {m} \\ 0, & \text {o t h e r w i s e .} \end{array} \right. \tag {5}
$$
Eq. (5), however, yields a density that is 0 almost everywhere (with respect to the standard Lebesgue measure) and its gradient with respect to  $p_m$  is also zero almost everywhere. Thus, Eq. (5) is not amenable to gradient-based sampling, and to derive a meaningful gradient-based sampling we require a more nuanced encoding.
To provide such an encoding, we introduce the Voronoi measure. Given an embedding-augmented distribution  $\pmb{p} = [p_1,\dots ,p_M]$  with embeddings  $\{\mathbf{v}_m\}_{m = 1}^M$  , and a compact set  $\kappa \subset \mathbb{R}^d$  that covers the embeddings, i.e.,  $\{\mathbf{v}_m\}_{m = 1}^M\subset \mathcal{K}$  , we define the Voronoi cell for the  $m^{\mathrm{th}}$  item with respect to the compact set  $\kappa$  as follows
$$
C _ {m} = \left\{\mathbf {x}: \mathbf {x} \in \mathcal {K}, \| \mathbf {x} - \mathbf {v} _ {m} \| _ {2} ^ {2} \leq \| \mathbf {x} - \mathbf {v} _ {m ^ {\prime}} \| _ {2} ^ {2}, \forall m ^ {\prime} \neq m \right\}. \tag {6}
$$
Now, using the definition of a Voronoi cell  $C_m$  given in Eq. (6), we can define a density that is not zero almost everywhere as follows. The strategy is to spread out the probability mass  $p_m$  over the entirety of the set  $C_m$ . To do so, we assume access to a set of base measures  $\{\mu_m\}_{m=1}^M$  that give us a reference for how to judge the probability mass in each  $C_m$ . We make this encoding formal in the following definition.
<sup>4</sup>We will discuss in §5.2 how gradient-based sampling can help to sample from this distribution without the need to compute  $Z_{t}$ .
Definition 2. Let  $\pmb{p} = [p_1, \dots, p_M]$  be an embedding-augmented distribution with embeddings  $\{\mathbf{v}_m\}_{m=1}^M \subset \mathbb{R}^d$ , and let  $\mathcal{K}$  be a compact set such that  $\{\mathbf{v}_m\}_{m=1}^M \subset \mathcal{K}$ . Furthermore, let  $\{\mu_m\}_{m=1}^M$  be a set of base measures over  $\mathbb{R}^d$  that are absolutely continuous with respect to the standard Lebesgue measure  $\lambda$  over  $\mathbb{R}^d$ , i.e.,  $\mu_m \ll \lambda$ . Define the  $(\mathcal{K}, \mu)$ -Voronoi measure as follows
$$
p _ {\mathrm {V}} (\mathbf {x}) \stackrel {\text {d e f}} {=} \left\{ \begin{array}{l l} \frac {p _ {m ^ {*} (\mathbf {x})}}{\mu_ {m} \left(C _ {m ^ {*} (\mathbf {x})}\right)} \frac {\mathrm {d} \mu_ {m}}{\mathrm {d} \lambda} (\mathbf {x}), & \text {i f} \mathbf {x} \in \mathcal {K} \\ 0, & \text {o t h e r w i s e} \end{array} \right. \tag {7}
$$
where we define projection
$$
m ^ {\star} (\mathbf {x}) \stackrel {\text {d e f}} {=} \underset {m \in [ M ]} {\operatorname {a r g m i n}} \| \mathbf {x} - \mathbf {v} _ {m} \| _ {2} ^ {2}. \tag {8}
$$
with ties broken arbitrarily.5
In the following proposition, we make precise the sense in which a Voronoi measure encodes the original embedding-augmented distribution.
Proposition 1. Let  $\pmb{p} = [p_1, \dots, p_M]$  be an embedding-augmented distribution with embeddings  $\{\mathbf{v}_m\}_{m=1}^M \subset \mathbb{R}^d$ , and let  $p_{\mathrm{V}}$  be the corresponding Voronoi measure Eq. (7). Then,  $p_{\mathrm{V}}(C_m) = p_m$  where  $C_m$  is defined as in Eq. (6). See App. C.1 for proof.
Example 1. Suppose  $\pmb{p} = [p_1, \dots, p_4]$  is a categorical distribution, and there are 4 embeddings in  $\mathbb{R}^2$  associated with each  $p_i$ , namely:  $\mathbf{v}_1 = [1,1], \mathbf{v}_2 = [-1,1], \mathbf{v}_3 = [-1, -1], \mathbf{v}_4 = [1, -1]$ . Given the  $\mathcal{K} = [-2,2] \times [-2,2]$  and the embedding-augmented probability distribution  $\pmb{p}$ , Eq. (7) defines a Voronoi measure over this space, where the Voronoi
cells are visualized in Fig. 1. We will discuss in §6 how Voronoi sampling navigates this space.
Figure 1: The example shows how Voronoi Sampling navigates through the space to sample one embedding in  $\mathbb{R}^2$ . Each Voronoi is annotated with the probability of its center, i.e., Voronoi measure of the cell.
3.1 Structured Voronoi Measures
To encode language models as densities more naturally, we introduce a generalization of the Voronoi measure, which we term a structured Voronoi measure. Now, rather than a distribution over  $M$  elements, we assume to have a sequence of length  $N$ . Each token in the sequence takes value in  $[M]$ . Let  $\boldsymbol{m} = [m_1, \dots, m_N] \in [M]^N$ . We define a structured Voronoi cell as  $C_m = \prod_{n=1}^N C_{m_n}$ , where  $\prod$  denotes the Cartesian product and we define the individual Voronoi cell as
$$
C _ {m _ {n}} = \left\{\mathbf {x}: \mathbf {x} \in \mathcal {K}, \| \mathbf {x} - \mathbf {v} _ {m _ {n}} \| _ {2} ^ {2} \leq \| \mathbf {x} - \mathbf {v} _ {m ^ {\prime}} \| _ {2} ^ {2}, \forall m ^ {\prime} \neq m _ {n} \right\}. \tag {9}
$$
Proposition 2. Let  $\mu$  be a measure on  $\mathbb{R}^d$ . Then, we have the product measure space as  $\mu(C_m) = \prod_{n=1}^{N} \mu(C_{m_n})$ . See App. C.2 for proof.
Definition 3. Let  $\pmb{p}$  be an embedding-augmented distribution over  $[M]^N$ . For  $\pmb{m} \in [M]^N$ , we denote  $\pmb{m}$ 's probability as  $p_m$ , and  $\pmb{m}$ 's embedding as  $\mathbf{V}_m \in \mathbb{R}^{N \times d}$ . Let  $\mathcal{K}$  be a compact set that covers the embeddings  $\mathbf{V}_m$  and let  $\mu \ll \lambda$  be a base measure absolutely continuous with respect to the Lebesgue measure  $\lambda$ . We define the  $(\mathcal{K},\mu)$ -structured Voronoi measure as follows
$$
p _ {\mathrm {V}} (\mathbf {x}) \stackrel {\text {d e f}} {=} \left\{ \begin{array}{l l} \frac {p _ {m ^ {*} (\mathbf {x})}}{\mu \left(C _ {m ^ {*} (\mathbf {x})}\right)} \frac {\mathrm {d} \mu}{\mathrm {d} \lambda} (\mathbf {x}), & i f \mathbf {x} \in \mathcal {K} \\ 0, & o t h e r w i s e \end{array} \right. \tag {10}
$$
where we define the structured projection
$$
\boldsymbol {m} ^ {\star} (\mathbf {x}) \stackrel {\text {d e f}} {=} \operatorname * {a r g m i n} _ {\boldsymbol {m} \in [ M ] ^ {N}} \sum_ {n = 1} ^ {N} \left\| \mathbf {x} _ {n} - \mathbf {V} _ {\boldsymbol {m} _ {n}} \right\| _ {2} ^ {2}. \tag {11}
$$
The set of  $\mathbf{x}$  that induces a tie is a set of measure zero under the Lebesgue measure.
4 Application to Text Generation
Def. 3 gives us the flexibility to use any probability distribution over sequences of embeddings to define a structured Voronoi measure. For example, one can substitute  $p_{m^*(\mathbf{x})}$  with an embedding-augmented LM, i.e., Eq. (3), to encode a language model as a structured Voronoi measure. Another example is to encode a controlled LM as a structured Voronoi measure by substituting  $p_{m^*(\mathbf{x})}$  with Eq. (4).
Base Measure. We have explained how to encode our desired distribution as a structured Voronoi measure. However, in order to actually implement a gradient-based sampler, we need to specify the base probability measures  $\{\mu_m\}_{m=1}^M$ . Given an embedding-augmented probability  $p(\mathbf{V})$  it is natural to follow the gradient of  $\log p(\mathbf{V})$  with respect to the word embedding, i.e.,  $\mathbf{g}_m = \nabla_{\mathbf{V}} \log p(\mathbf{V})$ . Thus, if we want to follow a direction similar to  $\mathbf{g}_m$ , one natural choice for  $\mu$  is a Gaussian measure centered at the gradient  $\mathbf{g}_m$  restricted to Voronoi cell  $C_m$ , which we define:
$$
\mu (A) = \frac {1}{\mu \left(C _ {\boldsymbol {m}}\right)} \int_ {A \cap C _ {\boldsymbol {m}}} \exp \left(- \frac {1}{2} \| \mathbf {g} _ {\boldsymbol {m}} - \mathbf {x} \| _ {2} ^ {2}\right) \mathrm {d} \lambda (\mathbf {x}). \tag {12}
$$
The normalizer ensures the measure is a probability measure. Furthermore, we have that  $\mu$ 's Radon-Nikodym derivative with respect to the Lebesgue measure  $\lambda$  is given by
$$
\frac {\mathrm {d} \mu}{\mathrm {d} \lambda} (\mathbf {x}) \stackrel {\text {d e f}} {=} \left\{ \begin{array}{l l} \frac {1}{\mu \left(C _ {m}\right)} \exp \left(- \frac {1}{2} \| \mathbf {g} _ {m} - \mathbf {x} \| _ {2} ^ {2}\right), & \text {i f} \mathbf {x} \in C _ {m} \\ 0, & \text {o t h e r w i s e} \end{array} \right. \tag {13}
$$
Eq. (13) should be recognizable as the standard Gaussian density, albeit one that is truncated to the Voronoi cell  $C_m$  [27].
Proposition 3. Eq. (13) is absolutely continuous with respect to the Lebesgue measure  $\lambda$ . See App. C.3 for proof.
Proposition 4. The gradient of the log of the Voronoi measure  $p_{\mathrm{V}}$  is given by
$$
\nabla_ {\mathbf {x}} \log p _ {\mathrm {V}} (\mathbf {x}) = \left\{ \begin{array}{l l} \mathbf {g} _ {m} - \mathbf {x}, & i f \mathbf {x} \in \operatorname {i n t} \left(C _ {m}\right) \\ \text {u n d e f i n e d ,} & i f \mathbf {x} \in \partial C _ {m} \\ 0, & o t h e r w i s e \end{array} \right. \tag {14}
$$
where the first two blocks in the case statement apply if there exists some  $\mathbf{m}$  such that  $\mathbf{x} \in \operatorname{int}(C_{\mathbf{m}})$  or  $\mathbf{x} \in \partial C_{\mathbf{m}}$ . See App. C.4 for proof.
5 Gradient-Based Sampling
In this section, we first review gradient-based sampling methods, namely HMC and Langevin dynamics. Then we discuss how they have been used in previous papers on text generation. This will set the stage for our algorithm in §6, which is based on HMC.
5.1 Hamiltonian Monte Carlo
The goal of HMC, originally proposed by Duane et al. [7], is to design a better proposal distribution in the standard Metropolis-Hastings MCMC by taking advantage of the gradient information in a principled way. Concretely, to sample from a given distribution  $p(\mathbf{x})$  where  $\mathbf{x} \in \mathbb{R}^d$ , HMC treats  $\mathbf{x}$  as the coordinates of the particles in some fictitious physical system. It then introduces an auxiliary momentum variable  $\mathbf{r} \in \mathbb{R}^d$  associated with each coordinate and defines a Hamiltonian function  $H(\mathbf{x},\mathbf{r})$ . Here, the Hamiltonian  $H$  has the intuitive physical interpretation of the total energy of some conservative system, and, in classical mechanics, decomposes into the potential energy  $U(\mathbf{x})$  and the kinetic energy  $K(\mathbf{r})$ , i.e.,  $H(\mathbf{x},\mathbf{r}) = U(\mathbf{x}) + K(\mathbf{r})$ . This formulation is convenient in part because if we define the joint distribution  $p(\mathbf{x},\mathbf{r}) \propto e^{-H(\mathbf{x},\mathbf{r})}$  as in energy-based models, then
$$
p (\mathbf {x}, \mathbf {r}) \propto e ^ {- U (\mathbf {x})} \cdot e ^ {- K (\mathbf {r})}, \tag {15}
$$
<sup>6</sup>Please refer to App. B for a discussion on the reasoning behind choosing the Gaussian measure.
which means we can treat  $\mathbf{x}$  and  $\mathbf{r}$  as independent variables. Naturally, we can let  $U(\mathbf{x}) = -\log p(\mathbf{x})$  so that  $\mathbf{x}$  has the marginal of the target distribution. It is also common practice to set  $K(\mathbf{r}) = \mathbf{r}^{\top}M^{-1}\mathbf{r}/2$  so that the momentum variable has a Gaussian distribution. Here,  $M \in \mathbb{R}^{d \times d}$  is called the mass matrix and commonly set to identity.
The Hamiltonian  $H$  determines the equations of motion in a physical system, given by the Hamiltonian equations, which is also known as Hamiltonian dynamics,
$$
\frac {\mathrm {d} \mathbf {x}}{\mathrm {d} t} = \frac {\partial H}{\partial \mathbf {r}}, \quad \frac {\mathrm {d} \mathbf {r}}{\mathrm {d} t} = - \frac {\partial H}{\partial \mathbf {x}}. \tag {16}
$$
We are now ready to give a high-level description of how HMC generates a single sample (see Algorithm 1): First sample a momentum  $\mathbf{r}$  from a Gaussian (line 1), then evolve the system using the Hamiltonian equations Eq. (16) for some predetermined amount of time (lines 4 to 7), and finally accept the new state with the Metropolis-Hastings acceptance criterion (lines 8 to 13). Note that the Hamiltonian equations often admit no closed-form solution in practice, and hence one needs to use numerical integrators for approximation. In particular, the leapfrog integrator, corresponding to lines 5 to 7, is almost always used in HMC.
Ingeniously designed by Duane et al. [7], the efficiency of this elegant procedure is a result
of several favorable properties of Hamiltonian mechanics—namely volume preservation, reversibility, and conservation. Upon exact simulation of the Hamiltonian dynamics, these properties will lead to the acceptance probability of one, i.e., every proposed move will be accepted. In App. D, we will give intuition about these properties. Since the method developed later in §6 will subsume HMC as a special case, we delay the proof of correctness until then.
Langevin Dynamics. Langevin dynamics is a simplification of HMC, where only one leapfrog step is performed. Furthermore, what is usually referred to as Langevin dynamics is in fact the uncorrected Langevin dynamics, where the acceptance criterion is ignored, i.e., every proposed sample is accepted; see Algorithm 2. While uncorrected Langevin dynamics is guaranteed to converge when the energy function is smooth [31, §5.3], there is no such guarantee with the presence of a discontinuity in the energy function. Nevertheless, due to its simplicity, Langevin dynamics is the only gradient-based sampling method that has been applied for text generation applications.
5.2 Applications to Controlled Generation