Venue: NAACL 2018

# Neural Particle Smoothing

for Sampling from Conditional Sequence Models

 Chu-Cheng Lin & Jason Eisner

Center for Language and Speech Processing

Johns Hopkins University, Baltimore MD, 21218

{kitsing,jason}@cs.jhu.edu

###### Abstract

We introduce _neural particle smoothing_, a sequential Monte Carlo method for sampling annotations of an input string from a given probability model. In contrast to conventional particle filtering algorithms, we train a proposal distribution that _looks ahead_ to the end of the input string by means of a right-to-left LSTM. We demonstrate that this innovation can improve the quality of the sample. To motivate our formal choices, we explain how our neural model and neural sampler can be viewed as low-dimensional but nonlinear approximations to working with HMMs over very large state spaces.

## 1 Introduction

Many structured prediction problems in NLP can be reduced to labeling a length-\(T\) input string \(\mathbf{x}\) with a length-\(T\) sequence \(\mathbf{y}\) of tags. In some cases, these tags are annotations such as syntactic parts of speech. In other cases, they represent actions that incrementally build an output structure: IOB tags build a chunking of the input Ramshaw and Marcus (1999), shift-reduce actions build a tree Yamada and Matsumoto (2003), and finite-state transducer arcs build an output string Pereira and Riley (1997).

One may wish to score the possible taggings using a recurrent neural network, which can learn to be sensitive to complex patterns in the training data. A globally normalized conditional probability model is particularly valuable because it quantifies uncertainty and does not suffer from label bias Lafferty et al. (2001); also, such models often arise as the predictive conditional distribution \(p(\mathbf{y}~{}|~{}\mathbf{x})\) corresponding to some well-designed generative model \(p(\mathbf{x},\mathbf{y})\) for the domain. In the neural case, however, inference in such models becomes intractable. It is hard to know what the model actually predicts and hard to compute gradients to improve its predictions.

In such intractable settings, one generally falls back on approximate inference or sampling. In the NLP community, beam search and importance sampling are common. Unfortunately, beam search considers only the approximate-top-\(k\) taggings from an exponential set Wiseman and Rush (2016), and importance sampling requires the construction of a good proposal distribution Dyer et al. (2016).

In this paper we exploit the sequential structure of the tagging problem to do _sequential_ importance sampling, which resembles beam search in that it constructs its proposed samples incrementally--one tag at a time, taking the actual model into account at every step. This method is known as particle filtering Doucet and Johansen (2009). We extend it here to take advantage of the fact that the sampler has access to the entire input string as it constructs its tagging, which allows it to look ahead or--as we will show--to use a neural network to approximate the effect of lookahead. Our resulting method is called _neural particle smoothing_.

### What this paper provides

For \(\mathbf{x}=x_{1}\cdots x_{T}\), let \(\mathbf{x}_{t}\) and \(\mathbf{x}_{t}\): respectively denote the prefix \(x_{1}\cdots x_{t}\) and the suffix \(x_{t+1}\cdots x_{T}\).

We develop _neural particle smoothing_--a sequential importance sampling method which, given a string \(\mathbf{x}\), draws a sample of taggings \(\mathbf{y}\) from \(p_{\theta}(\mathbf{y}~{}|~{}\mathbf{x})\). Our method works for any conditional probability model of the quite general form1

Footnote 1: A model may require for convenience that each input end with a special end-of-sequence symbol: that is, \(x_{T}=\textsc{eos}\).

\[p_{\theta}(\mathbf{y}~{}|~{}\mathbf{x})\stackrel{{\text{ def}}}{{\propto}}\exp G_{T} \tag{1}\]

where \(G\) is an _incremental stateful global scoring model_ that recursively defines scores \(G_{t}\) of prefixes of \((\mathbf{x},\mathbf{y})\) at all times \(0\leq t\leq T\):

\[G_{t} \stackrel{{\text{def}}}{{=}}G_{t-1}+g_{\theta}( \mathbf{s}_{t-1},x_{t},y_{t})\hskip 5.690551pt\mbox{(with $G_{0}\stackrel{{\text{ def}}}{{=}}0$)} \tag{2}\] \[\mathbf{s}_{t} \stackrel{{\text{def}}}{{=}}f_{\theta}(\mathbf{s}_{t- 1},x_{t},y_{t})\hskip 42.679134pt\mbox{(with $\mathbf{s}_{0}$ given)} \tag{3}\]

These quantities implicitly depend on \(\mathbf{x},\mathbf{y}\) and \(\theta\). Here \(\mathbf{s}_{t}\) is the model's _state_ after observing the pair of length-\(t\) prefixes \((\mathbf{x}_{t},\mathbf{y}_{t})\). \(G_{t}\) is the _score-so-far_

of this prefix pair, while \(G_{T}-G_{t}\) is the _score-to-go_. The state \(\mathbf{s}_{t}\) summarizes the prefix pair in the sense that the score-to-go depends only on \(\mathbf{s}_{t}\) and the length-\((T-t)\) suffixes \((\mathbf{x}_{t:},\mathbf{y}_{t:})\). The _local scoring function_\(g_{\theta}\) and _state update function_\(f_{\theta}\) may be any functions parameterized by \(\theta\)--perhaps neural networks. We assume \(\theta\) is fixed and given.

This model family is expressive enough to capture any desired \(p(\mathbf{y}~{}|~{}\mathbf{x})\). Why? Take any distribution \(p(\mathbf{x},\mathbf{y})\) with this desired conditionalization (e.g., the true joint distribution) and factor it as

\[\log p(\mathbf{x},\mathbf{y}) =\sum_{t=1}^{T}\log p(x_{t},y_{t}~{}|~{}\mathbf{x}_{t-1},\mathbf{ y}_{t-1})\] \[=\sum_{t=1}^{T}\underbrace{\log p(x_{t},y_{t}~{}|~{}\mathbf{s}_{ t-1})}_{\text{use as $g_{\theta}(\mathbf{s}_{t-1},\mathbf{x}_{t},y_{t})$}}=G_{T} \tag{4}\]

by making \(\mathbf{s}_{t}\) include as much information about \((\mathbf{x}_{:t},\mathbf{y}_{:t})\) as needed for (2.3) to hold (possibly \(\mathbf{s}_{t}=(\mathbf{x}_{:t},\mathbf{y}_{:t})\)).2 Then by defining \(g_{\theta}\) as shown in (2.3), we get \(p(\mathbf{x},\mathbf{y})=\exp G_{T}\) and thus (1) holds for each \(\mathbf{x}\).

Footnote 2: Furthermore, \(\mathbf{s}_{t}\) could even depend on all of \(\mathbf{x}\) (if \(\mathbf{s}_{0}\) does), allowing direct expression of models such as stacked BiRNNs.

### Relationship to particle filtering

Our method is spelled out in SS4 (one may look now). It is a variant of the popular _particle filtering_ method that tracks the state of a physical system in discrete time (Ristic et al., 2004). Our particular _proposal distribution_ for \(y_{t}\) can be found in equations (5), (6), (25) and (26). It considers not only past observations \(\mathbf{x}_{:t}\) as reflected in \(\mathbf{s}_{t-1}\), but also future observations \(\mathbf{x}_{t:}\), as summarized by the state \(\bar{\mathbf{s}}_{t}\) of a right-to-left recurrent neural network \(\tilde{f}\) that we will train:

\[\hat{H}_{t} \stackrel{{\mbox{\tiny def}}}{{=}}h_{\phi}(\bar{ \mathbf{s}}_{t+1},x_{t+1})+\hat{H}_{t+1} \tag{5}\] \[\bar{\mathbf{s}}_{t} \stackrel{{\mbox{\tiny def}}}{{=}}\bar{f}_{\phi}( \bar{\mathbf{s}}_{t+1},x_{t+1})\hskip 28.452756pt\text{(with $\mathbf{s}_{T}$ given)} \tag{6}\]

Conditioning the distribution of \(y_{t}\) on future observations \(\mathbf{x}_{t:}\) means that we are doing "smoothing" rather than "filtering" (in signal processing terminology). Doing so can reduce the bias and variance of our sampler. It is possible so long as \(\mathbf{x}\) is provided in its entirety before the sampler runs--which is often the case in NLP.

### Applications

Why sample from \(p_{\theta}\) at all? Many NLP systems instead simply search for the _Viterbi sequence_\(\mathbf{y}\) that maximizes \(G_{T}\) and thus maximizes \(p_{\theta}(\mathbf{y}~{}|~{}\mathbf{x})\). If the space of states \(\mathbf{s}\) is small, this can be done efficiently by dynamic programming (Viterbi, 1967); if not, then \(A^{*}\) may be an option (see SS2). More common is to use an approximate method: beam search, or perhaps a sequential prediction policy trained with reinforcement learning. Past work has already shown how to improve these approximate search algorithms by conditioning on the future (Bahdanau et al., 2017; Wiseman and Rush, 2016).

Sampling is essentially a generalization of maximization: sampling from \(\exp\frac{G_{T}}{\mathrm{temperature}}\) approaches maximization as \(\mathrm{temperature}\to 0\). It is a fundamental building block for other algorithms, as it can be used to take expectations over the whole space of possible \(\mathbf{y}\) values. For unfamiliar readers, Appendix E reviews how sampling is crucially used in minimum-risk decoding, supervised training, unsupervised training, imputation of missing data, pipeline decoding, and inference in graphical models.

## 2 Exact Sequential Sampling

To develop our method, it is useful to first consider exact samplers. Exact sampling is tractable for only some of the models allowed by SS1.1. However, the form and notation of the exact algorithms in SS2 will guide our development of approximations in SS3.

An _exact sequential sampler_ draws \(y_{t}\) from \(p_{\theta}(y_{t}~{}|~{}\mathbf{x},\mathbf{y}_{:t-1})\) for each \(t=1,\ldots,T\) in sequence. Then \(\mathbf{y}\) is exactly distributed as \(p_{\theta}(\mathbf{y}~{}|~{}\mathbf{x})\).

For each given \(\mathbf{x},\mathbf{y}_{:t-1}\), observe that

\[p_{\theta}(y_{t} \mid\mathbf{x},\mathbf{y}_{:t-1}) \tag{7}\] \[\propto p_{\theta}(\mathbf{y}_{:t}\mid\mathbf{x})=\sum_{\mathbf{y}_{t }}p_{\theta}(\mathbf{y}~{}|~{}\mathbf{x})\] (8) \[\propto\sum_{\mathbf{y}_{t:}}\exp G_{T}\] (9) \[=\exp\,(G_{t}+\underbrace{\log\sum_{\mathbf{y}_{t:}}\exp\,(G_{ T}-G_{t})}_{\text{call this $H_{t}$}})\] (10) \[=\exp\,(G_{t-1}+g_{\theta}(\mathbf{s}_{t-1},x_{t},y_{t})+H_{t})\] (11) \[\propto\exp\,(g_{\theta}(\mathbf{s}_{t-1},x_{t},y_{t})+H_{t}) \tag{12}\]

Thus, we can easily construct the needed distribution (7) by normalizing (12) over all possible values of \(y_{t}\). The challenging part of (12) is to compute \(H_{t}\): as defined in (10), \(H_{t}\) involves a sum over exponentially many futures \(\mathbf{y}_{t:}\). (See Figure 1.)

We chose the symbols \(G\) and \(H\) in homage to the \(A^{*}\) search algorithm (Hart et al., 1968). In that algorithm (which could be used to find the Viterbi sequence), \(g\) denotes the score-so-far of a partial solution \(\mathbf{y}_{:t}\), and \(h\) denotes the optimal score-to-go. Thus, \(g+h\) would be the score of the _best_ sequence with prefix \(\mathbf{y}_{:t}\). Analogously, our 

\(H_{t}\) is the log of the total exponentiated scores of _all_ sequences with prefix \(\mathbf{y}_{:t}\). \(G_{t}\) and \(H_{t}\) might be called the _logprob-so-far_ and _logprob-to-go_ of \(\mathbf{y}_{:t}\).

Just as \(A^{*}\) approximates \(h\) with a "heuristic" \(h\), the next section will approximate \(H_{t}\) using a neural estimate \(\hat{H}_{t}\) (equations (5)-(6)). However, the specific form of our approximation is inspired by cases where \(H_{t}\) can be computed exactly. We consider those in the remainder of this section.

### Exact sampling from HMMs

A _hidden Markov model_ (HMM) specifies a normalized _joint_ distribution \(p_{\theta}(\mathbf{x},\mathbf{y})=\exp G_{T}\) over state sequence \(\mathbf{y}\) and observation sequence \(\mathbf{x}\),3 Thus the posterior \(p_{\theta}(\mathbf{y}~{}|~{}\mathbf{x})\) is proportional to \(\exp G_{T}\), as required by equation (1).

Footnote 3: The HMM actually specifies a distribution over a pair of infinite sequences, but here we consider the marginal distribution over just the length-\(T\) prefixes.

The HMM specifically defines \(G_{T}\) by equations (2)-(3) with \(\mathbf{s}_{t}=y_{t}\) and \(g_{\theta}(\mathbf{s}_{t-1},x_{t},y_{t})=\log p_{\theta}(y_{t}~{}|~{}y_{t-1} )+\log p_{\theta}(x_{t}~{}|~{}y_{t})\).4

Footnote 4: It takes \(\mathbf{s}_{0}=\textsc{bos}\), a beginning-of-sequence symbol, so \(p_{\theta}(y_{1}~{}|~{}\textsc{bos})\) specifies the initial state distribution.

In this setting, \(H_{t}\) can be computed exactly by the _backward algorithm_[8]. (Details are given in Appendix A for completeness.)

### Exact sampling from OOHMMs

For sequence tagging, a weakness of (first-order) HMMs is that the model state \(\mathbf{s}_{t}=y_{t}\) may contain little information: only the most recent tag \(y_{t}\) is remembered, so the number of possible model states \(\mathbf{s}_{t}\) is limited by the vocabulary of output tags.

We may generalize so that the data generating process is in a latent state \(u_{t}\in\{1,\ldots,k\}\) at each time \(t\), and the observed \(y_{t}\)--along with \(x_{t}\)--is generated from \(u_{t}\). Now \(k\) may be arbitrarily large. The model has the form

\[p_{\theta}(\mathbf{x},\mathbf{y}) =\exp G_{T} \tag{13}\] \[=\sum_{\mathbf{u}}\prod_{t=1}^{T}p_{\theta}(u_{t}~{}|~{}u_{t-1}) \cdot p_{\theta}(x_{t},y_{t}~{}|~{}u_{t})\]

This is essentially a pair HMM [7] without insertions or deletions, also known as an "\(\epsilon\)-free" or "same-length" probabilistic finite-state transducer. We refer to it here as an _output-output HMM_ (OOHMM).5

Footnote 5: This is by analogy with the _input-output HMM_[9], which defines \(p(\mathbf{y}~{}|~{}\mathbf{x})\) directly and conditions the transition to \(u_{t}\) on \(x_{t}\). The OOHMM instead defines \(p(\mathbf{y}~{}|~{}\mathbf{x})\) by conditionalizing (13)—which avoids the _label bias_ problem [10] that in the IOHMM, \(y_{t}\) is independent of future input \(\mathbf{x}_{t:}\) (given the past input \(\mathbf{x}_{:t}\)).

Is this still an example of the general model architecture from SS1.1? Yes. Since \(u_{t}\) is latent and evolves stochastically, it cannot be used as the state \(\mathbf{s}_{t}\) in equations (2)-(3) or (4). However, we _can_ define \(\mathbf{s}_{t}\) to be the model's _belief state_ after observing \((\mathbf{x}_{:t},\mathbf{y}_{:t})\). The belief state is the posterior probability distribution over the underlying state \(u_{t}\) of the system. That is, \(\mathbf{s}_{t}\) deterministically keeps track of all possible states that the OOHMM might be in--just as the state of a determinized FSA keeps track of all possible states that the original nondeterministic FSA might be in.

We may compute the belief state in terms of a vector of _forward probabilities_ that starts at \(\boldsymbol{\alpha}_{0}\),

\[(\boldsymbol{\alpha}_{0})_{u}\stackrel{{\text{\tiny def}}}{{=}} \begin{cases}1&\mbox{if $u=\textsc{bos}$ (see footnote \ref{eq:def})}\\ 0&\mbox{if $u=\mathrm{any\ other\ state}$}\end{cases} \tag{14}\]

and is updated deterministically for each \(0<t\leq T\) by the _forward algorithm_[8]:

\[(\boldsymbol{\alpha}_{t})_{u}\stackrel{{\text{\tiny def}}}{{=}} \sum_{u^{\prime}=1}^{k}(\boldsymbol{\alpha}_{t-1})_{u^{\prime}}\cdot p_{ \theta}(u~{}|~{}u^{\prime})\cdot p_{\theta}(x_{t},y_{t}~{}|~{}u) \tag{15}\]

Figure 1: Sampling a single particle from a tagging model. \(y_{1},\dots,y_{t-1}\) (orange) have already been chosen, with a total model score of \(G_{t-1}\), and now the sampler is constructing a proposal distribution \(q\) (purple) from which the next tag \(y_{t}\) will be sampled. Each \(y_{t}\) is evaluated according to its contribution to \(G_{t}\) (namely \(g_{\theta}\)) and its future score \(H_{t}\) (blue). The figure illustrates quantities used throughout the paper, beginning with exact sampling in equations (7)–(12). Our main idea (§3) is to _approximate_ the \(H_{t}\) computation (a log-sum-exp over exponentially many sequences) when exact computation by dynamic programming is not an option. The form of our approximation uses a right-to-left recurrent neural network but is _inspired_ by the exact dynamic programming algorithm.

 

\({\mathbf{\alpha}_{t}}_{u}\) can be interpreted as the logprob-so-far _if_ the system is in state \(u\) after observing \(({\mathbf{x}}_{:t},{\mathbf{y}}_{:t})\). We may express the update rule (15) by \({\mathbf{\alpha}^{\top}_{t}=\mathbf{\alpha}^{\top}_{t-1}P}\) where the matrix \(P\) depends on \((x_{t},y_{t})\), namely \(P_{u^{\prime}u}\stackrel{{\mbox{\tiny def}}}{{=}}p_{\theta}(u\mid u ^{\prime})\cdot p_{\theta}(x_{t},y_{t}\mid u)\).

The belief state \({\mathbf{s}}_{t}\stackrel{{\mbox{\tiny def}}}{{=}}[\![{\mathbf{ \alpha}_{t}}]\!]\in\mathbb{R}^{k}\) simply normalizes \({\mathbf{\alpha}_{t}}\) into a probability vector, where \([\![{\bf u}]\!]\stackrel{{\mbox{\tiny def}}}{{=}}{\mathbf{u}}/({ \mathbf{u}}^{\top}{\mathbf{1}})\) denotes the _normalization operator_. The state update (15) now takes the form (3) as desired, with \(f_{\theta}\) a normalized vector-matrix product:

\[{\mathbf{s}}_{t}^{\top}=f_{\theta}({\mathbf{s}}_{t-1},x_{t},y_{ t})\stackrel{{\mbox{\tiny def}}}{{=}}[\![{\overline{\mathbf{s}}_{t-1} ^{\top}P}]\!] \tag{16}\]

As in the HMM case, we define \(G_{t}\) as the log of the generative prefix probability,

\[G_{t}\stackrel{{\mbox{\tiny def}}}{{=}}\log p_{ \theta}({\mathbf{x}}_{:t},{\mathbf{y}}_{:t})=\log\sum_{u}({\mathbf{\alpha}_{t}})_{u} \tag{17}\]

which has the form (2) as desired if we put

\[g_{\theta}({\mathbf{s}}_{t-1},x_{t},y_{t}) \stackrel{{\mbox{\tiny def}}}{{=}} G_{t}-G_{t-1} \tag{18}\] \[=\log\frac{{\mathbf{\alpha}^{\top}_{t-1}P}{\mathbf{1}}}{{\mathbf{\alpha}^{ \top}_{t-1}}{\mathbf{1}}}=\log\left({\mathbf{s}}_{t-1}^{\top}P{\mathbf{1}}\right)\]

Again, exact sampling is possible. It suffices to compute (9). For the OOHMM, this is given by

\[\sum_{{\mathbf{y}}_{t}:}\exp G_{T}={\mathbf{\alpha}^{\top}_{t}}{ \mathbf{\beta}_{t}} \tag{19}\]

where \({\mathbf{\beta}_{T}}\stackrel{{\mbox{\tiny def}}}{{=}}{\mathbf{1}}\) and the _backward algorithm_

\[({\mathbf{\beta}_{t}})_{v} \stackrel{{\mbox{\tiny def}}}{{=}} p_{\theta}({\mathbf{x}}_{t:}\mid u_{t}=u) \tag{20}\] \[=\sum_{u_{t}:}\underbrace{p_{\theta}(u^{\prime}\mid u)\cdot p(x_{ t+1}\mid u^{\prime})}_{\text{call this~{}$\bar{P}_{uu^{\prime}}$}}\cdot({\mathbf{\beta}_{t+1}})_{u^{\prime}}\]

for \(0\leq t<T\) uses dynamic programming to find the total probability of all ways to generate the future observations \({\mathbf{x}}_{t:}\). Note that \({\mathbf{\alpha}_{t}}\) is defined for a _specific prefix_\({\mathbf{y}}_{:t}\) (though it sums over all \({\mathbf{u}}_{:t}\)), whereas \({\mathbf{\beta}_{t}}\) sums over _all suffixes_\({\mathbf{y}}_{t:}\) (and over all \({\mathbf{u}}_{:t}\)), to achieve the asymmetric summation in (19).

Define \({\overline{\mathbf{s}}_{t}}\stackrel{{\mbox{\tiny def}}}{{=}}[\![{ \mathbf{\beta}_{t}}]\!]\in\mathbb{R}^{k}\) to be a normalized version of \({\mathbf{\beta}_{t}}\). The \({\mathbf{\beta}_{t}}\) recurrence (20) can clearly be expressed in the form \({\overline{\mathbf{s}}_{t}}=[\![\bar{P}{\overline{\mathbf{s}}_{t+1}}]\!]\), much like (16).

### The logprob-to-go for OOHMMs

Let us now work out the definition of \(H_{t}\) for OOHMMs (cf. equation (35) in Appendix A for HMMs). We will write it in terms of \(\hat{H}_{t}\) from SS1.2. Let us define \(\hat{H}_{t}\) symmetrically to \(G_{t}\) (see (17)):

\[\hat{H}_{t}\stackrel{{\mbox{\tiny def}}}{{=}}\log \sum_{u}({\mathbf{\beta}_{t}})_{u}~{}~{}(=\log{\mathbf{1}}^{\top}{\mathbf{\beta}_{t}}) \tag{21}\]

which has the form (5) as desired if we put

\[h_{\phi}({\overline{\mathbf{s}}_{t+1}},x_{t+1})\stackrel{{ \mbox{\tiny def}}}{{=}}\hat{H}_{t}-\hat{H}_{t+1}=\log\;({\mathbf{1}}^{\top}{ \overline{P}}{\overline{\mathbf{s}}_{t+1}}) \tag{22}\]

From equations (10), (17), (19) and (21), we see

\[H_{t} =\log\big{(}\sum_{{\mathbf{y}}_{t:}}\exp G_{T}\big{)}-G_{t}\] \[=\log\frac{{\mathbf{\alpha}^{\top}_{t}}{\mathbf{\beta}_{t}}}{({\mathbf{ \alpha}^{\top}_{t}}{\mathbf{1}})({\mathbf{1}}^{\top}{\mathbf{\beta}_{t}})}+\log\;({\mathbf{1}}^{ \top}{\mathbf{\beta}_{t}})\] \[=\underbrace{\operatorname*{\underline{\mathrm{sg}}}_{t}^{\top}{ \overline{\mathbf{s}}_{t}}}_{\text{call this~{}$C_{t}$}}+\hat{H}_{t} \tag{23}\]

where \(C_{t}\in\mathbb{R}\) can be regarded as evaluating the _compatibility_ of the state distributions \({\mathbf{s}}_{t}\) and \({\overline{\mathbf{s}}_{t}}\).

In short, the generic strategy (12) for exact sampling says that for an OOHMM, \(y_{t}\) is distributed as

\[p_{\theta}(y_{t}\mid{\mathbf{x}},{\mathbf{y}}_{:t-1})\propto \exp\;(g_{\theta}({\mathbf{s}}_{t-1},x_{t},y_{t})+H_{t})\] \[\propto\exp\;(\underbrace{g_{\theta}({\mathbf{s}}_{t-1},x_{t},y_ {t})}_{\text{depends on }{\mathbf{x}}_{:t},{\mathbf{y}}_{:t}}+\underbrace{C_{t}}_{\text{on }{\mathbf{ x}},{\mathbf{y}}_{:t}}+\underbrace{\hat{H}_{t}}_{\text{on }{\mathbf{ x}}_{:t}})\] \[\propto\exp\;(g_{\theta}({\mathbf{s}}_{t-1},x_{t},y_{t})+C_{t}) \tag{24}\]

This is equivalent to choosing \(y_{t}\) in proportion to (19)--but we now turn to settings where it is infeasible to compute (19) exactly. There we will use the formulation (24) but approximate \(C_{t}\). For completeness, we will also consider how to approximate \(\hat{H}_{t}\), which dropped out of the above distribution (because it was the same for all choices of \(y_{t}\)) but may be useful for other algorithms (see SS4).

## 3 Neural Modeling as Approximation

### Models with large state spaces

The expressivity of an OOHMM is limited by the number of states \(k\). The state \(u_{t}\in\{1,\ldots,k\}\) is a bottleneck between the past \(({\mathbf{x}}_{:t},{\mathbf{y}}_{:t})\) and the future \(({\mathbf{x}}_{t:},{\mathbf{y}}_{:t})\), in that past and future are _conditionally independent_ given \(u_{t}\). Thus, the mutual information between past and future is at most \(\log_{2}k\) bits.

In many NLP domains, however, the past seems to carry substantial information about the future. The first half of a sentence greatly reduces the uncertainly about the second half, by providing information about topics, referents, syntax, semantics, and discourse. This suggests that an accurate HMM language model \(p({\mathbf{x}})\) would require _very large_\(k\)--as would a generative OOHMM model \(p({\mathbf{x}},{\mathbf{y}})\) of _annotated_ language. The situation is perhaps better for discriminative models \(p({\mathbf{y}}\mid{\mathbf{x}})\), since much 

the information for predicting \({\bf y}_{t,:}\) might be available in \({\bf x}_{t,:}\). Still, it is important to let \(({\bf x}_{t,:},{\bf y}_{:t})\) contribute enough additional information about \({\bf y}_{t:}\): even for short strings, making \(k\) too small (giving \(\leq\log_{2}k\) bits) may harm prediction [40].

Of course, (4) says that an OOHMM can express any joint distribution for which the mutual information is finite,6 by taking \(k\) large enough for \(v_{t-1}\) to capture the relevant info from \(({\bf x}_{:t-1},{\bf y}_{:t-1})\).

Footnote 6: This is not true for the language of balanced parentheses.

So why not just take \(k\) to be large--say, \(k=2^{30}\) to allow 30 bits of information? Unfortunately, evaluating \(G_{T}\) then becomes very expensive--both computationally and statistically. As we have seen, if we define \({\bf s}_{t}\) to be the belief state \(\llbracket\boldsymbol{\alpha}_{t}\rrbracket\in\mathbb{R}^{k}\), updating it at each observation \((x_{t},y_{t})\) (equation (3)) requires multiplication by a \(k\times k\) matrix \(P\). This takes time \(O(k^{2})\), and requires enough data to learn \(O(k^{2})\) transition probabilities.

### Neural approximation of the model

As a solution, we might hope that for the inputs \({\bf x}\) observed in practice, the very high-dimensional belief states \(\llbracket\!\boldsymbol{\alpha}_{t}\rrbracket\!\rrbracket\in\mathbb{R}^{k}\) might tend to lie near a \(d\)-dimensional manifold where \(d\ll k\). Then we could take \({\bf s}_{t}\) to be a vector in \(\mathbb{R}^{d}\) that compactly encodes the approximate coordinates of \(\llbracket\!\boldsymbol{\alpha}_{t}\rrbracket\) relative to the manifold: \({\bf s}_{t}=\nu(\llbracket\!\boldsymbol{\alpha}_{t}\rrbracket\!\rrbracket)\), where \(\nu\) is the encoder.

In this new, nonlinearly warped coordinate system, the functions of \({\bf s}_{t-1}\) in (2)-(3) are no longer the simple, essentially linear functions given by (16) and (18). They become nonlinear functions operating on the manifold coordinates. (\(f_{\theta}\) in (16) should now ensure that \({\bf s}_{t}^{\top}\approx\nu(\llbracket(\nu^{-1}({\bf s}_{t-1}))^{\top}P \rrbracket)\), and \(g_{\theta}\) in (18) should now estimate \(\log~{}(\nu^{-1}({\bf s}_{t-1}))^{\top}P\mathbf{1}\).) In a sense, this is the reverse of the "kernel trick" [57] that converts a low-dimensional nonlinear function to a high-dimensional linear one.

Our hope is that \({\bf s}_{t}\) has enough dimensions \(d\ll k\) to capture the useful information from the true \(\llbracket\!\boldsymbol{\alpha}_{t}\rrbracket\), **and** that \(\theta\) has enough dimensions \(\ll k^{2}\) to capture most of the dynamics of equations (16) and (18). We thus proceed to fit the neural networks \(f_{\theta},g_{\theta}\) directly to the data, _without ever knowing_ the true \(k\) or the structure of the original operators \(P\in\mathbb{R}^{k\times k}\).

We regard this as the implicit justification for various published probabilistic sequence models \(p_{\theta}({\bf y}\mid{\bf x})\) that incorporate neural networks. These models usually have the form of SS1. Most simply, \((f_{\theta},g_{\theta})\) can be instantiated as one time step in an RNN [9], but it is common to use enriched versions such as deep LSTMs. It is also common to have the state \({\bf s}_{t}\) contain not only a vector of manifold coordinates in \(\mathbb{R}^{d}\) but also some unboundedly large representation of \(({\bf x},{\bf y}_{:t})\) (cf. equation (4)), so the \(f_{\theta}\) neural network can refer to this material with an attentional [1] or stack mechanism [57].

A few such papers have used _globally_ normalized conditional models that can be viewed as approximating some OOHMM, e.g., the parsers of Dyer et al. [11] and Andor et al. [11]. That is the case (SS1.1) that particle smoothing aims to support. Most papers are _locally_ normalized conditional models (e.g., Kann and Schutze, 2016; Aharoni and Goldberg, 2017); these simplify supervised training and can be viewed as approximating IOHMMs (footnote 5). For locally normalized models, \(H_{t}=0\) by construction, in which case particle filtering (which estimates \(H_{t}=0\)) is just as good as particle smoothing. Particle filtering is still useful for these models, but lookahead's inability to help them is an expressive limitation (known as _label bias_) of locally normalized models. We hope the existence of particle smoothing (which learns an estimate \(H_{t}\)) will make it easier to adopt, train, and decode globally normalized models, as discussed in SS1.3.

### Neural approximation of logprob-to-go

We can adopt the same neuralization trick to approximate the OOHMM's logprob-to-go \(H_{t}=C_{t}+\hat{H}_{t}\). We take \(\bar{{\bf s}}_{t}\in\mathbb{R}^{d}\) on the same theory that it is a low-dimensional reparameterization of \(\llbracket\beta_{t}\rrbracket\), and define \((\bar{J}_{\phi},h_{\phi})\) in equations (5)-(6) to be neural networks. Finally, we must replace the definition of \(C_{t}\) in (23) with another neural network \(c_{\phi}\) that works on the low-dimensional approximations:7

Footnote 7: \(C_{T}=0\) is correct according to (23). Forcing this ensures \(H_{T}=0\), so our approximation becomes exact as of \(t=T\).

\[C_{t}\stackrel{{\text{\tiny def}}}{{=}}c_{\phi}({\bf s}_{t},\bar{{ \bf s}}_{t})\quad\mbox{ (except that $C_{T}\stackrel{{\text{\tiny def}}}{{=}}0$)} \tag{25}\]

The resulting approximation to (24) (which does not actually require \(h_{\phi}\)) will be denoted \(q_{\theta,\phi}\):

\[q_{\theta,\phi}(y_{t}\mid{\bf x},{\bf y}_{:t-1})\stackrel{{\text{ \tiny def}}}{{\propto}}\exp~{}(g_{\theta}({\bf s}_{t-1},x_{t},y_{t})+C_{t}) \tag{26}\]

The neural networks in the present section are all parameterized by \(\phi\), and are intended to produce an estimate of the logprob-_to-go_\(H_{t}\)--a function of \({\bf x}_{t:}\), which sums over all possible \({\bf y}_{t:}\).

By contrast, the OOHMM-inspired neural networks suggested in SS3.2 were used to specify 

actual model of the logprob-_so-far \(G_{t}\)_--a function of \(\mathbf{x}_{:t}\) and \(\mathbf{y}_{:t}\)--using separate parameters \(\theta\).

Arguably \(\phi\) has a harder modeling job than \(\theta\) because it must implicitly sum over possible futures \(\mathbf{y}_{:t}\). We now consider how to get corrected samples from \(q_{\theta,\phi}\) even if \(\phi\) gives poor estimates of \(H_{t}\), and then how to train \(\phi\) to improve those estimates.

## 4 Particle smoothing

In this paper, we assume nothing about the given model \(G_{T}\) except that it is given in the form of equations (1)-(3) (including the parameter vector \(\theta\)).

Suppose we run the exact sampling strategy but approximate \(p_{\theta}\) in (7) with a _proposal distribution_\(q_{\theta,\phi}\) of the form in (25)-(26). Suppressing the subscripts on \(p\) and \(q\) for brevity, this means we are effectively drawing \(\mathbf{y}\) not from \(p(\mathbf{y}~{}|~{}\mathbf{x})\) but from

\[q(\mathbf{y}~{}|~{}\mathbf{x})=\prod_{t=1}^{T}q(y_{t}~{}|~{}\mathbf{x}, \mathbf{y}_{:t-1}) \tag{27}\]

If \(C_{t}\approx H_{t}+\text{const}\) within each \(y_{t}\) draw, then \(q\approx p\).

_Normalized importance sampling_ corrects (mostly) for the approximation by drawing _many_ sequences \(\mathbf{y}^{(1)},\ldots\mathbf{y}^{(M)}\) IID from (27) and assigning \(\mathbf{y}^{(m)}\) a relative _weight_ of \(w^{(m)}\stackrel{{\mbox{\tiny def}}}{{=}}\frac{\hat{p}(\mathbf{ y}^{(m)}|\mathbf{x})}{q(\mathbf{y}^{(m)}|\mathbf{x})}\). This _ensemble of weighted particles_ yields a distribution

\[\hat{p}(\mathbf{y})\stackrel{{\mbox{\tiny def}}}{{=}}\frac{\sum_{ m=1}^{M}w^{(m)}[\mathbf{Y}(\mathbf{y}=\mathbf{y}^{(m)})}{\sum_{m=1}^{M}w^{(m)}} \approx p(\mathbf{y}~{}|~{}\mathbf{x}) \tag{28}\]

that can be used as discussed in SS1.3. To compute \(w^{(m)}\) in practice, we replace the numerator \(p(\mathbf{y}^{(m)}~{}|~{}\mathbf{x})\) by the unnormalized version \(\exp G_{T}\), which gives the same \(\hat{p}\). Recall that each \(G_{T}\) is a sum \(\sum_{t=1}^{T}g_{\theta}(\cdots)\).

_Sequential importance sampling_ is an equivalent implementation that makes \(t\) the _outer_ loop and \(m\) implementativity. It computes a _prefix ensemble_

\[Y_{t}\stackrel{{\mbox{\tiny def}}}{{=}}\{(\mathbf{y}^{(1)}_{: t},w^{(1)}_{t}),\dots,(\mathbf{y}^{(M)}_{:t},w^{(M)}_{t})\} \tag{29}\]

for each \(0\leq t\leq T\) in sequence. Initially, \((\mathbf{y}^{(m)}_{:0},w^{(m)}_{0})=(\epsilon,\exp C_{0})\) for all \(m\). Then for \(0<t\leq T\), we extend these particles in parallel:

\[\mathbf{y}^{(m)}_{:t} =\mathbf{y}^{(m)}_{:t-1}y^{(m)}_{t}\quad\text{(concatenationation)} \tag{30}\] \[w^{(m)}_{t} =w^{(m)}_{t-1}\;\frac{\exp\left(g_{\theta}(\mathbf{s}_{t-1},x_{t} ,y_{t})+C_{t}-C_{t-1}\right)}{q(y_{t}|\mathbf{x},\mathbf{y}_{:t-1})} \tag{31}\]

where each \(y^{(m)}_{t}\) is drawn from (26). Each \(Y_{t}\) yields a distribution \(\hat{p}_{t}\) over prefixes \(\mathbf{y}_{:t}\), which estimates the distribution \(p_{t}(\mathbf{y}_{:t})\stackrel{{\mbox{\tiny def}}}{{\propto}} \exp\;(G_{t}+C_{t})\). We return \(\hat{p}\stackrel{{\mbox{\tiny def}}}{{=}}\hat{p}_{T}\approx p_{T}=p\). This gives the same \(\hat{p}\) as in (28): the final \(\mathbf{y}^{(m)}_{T}\) are the same, with the same final weights \(w^{(m)}_{T}=\frac{\exp G_{T}}{q(\mathbf{y}^{(m)}|\mathbf{x})}\), where \(G_{T}\) was now summed up as \(C_{0}+\sum_{t=1}^{T}g_{\theta}(\cdots)+C_{t}-C_{t-1}\).

That is our basic _particle smoothing_ strategy. If we use the naive approximation \(C_{t}=0\) everywhere, it reduces to _particle filtering_. In either case, various well-studied improvements become available, such as various resampling schemes Douc and Cappe (2005) and the particle cascade Paige et al. (2014).8

Footnote 8: The particle cascade would benefit from an estimate of \(\hat{H}_{t}\), as it (like A* search) compares particles of different lengths.

An easy improvement is _multinomial resampling_. After computing each \(\hat{p}_{t}\), this replaces \(Y_{t}\) with a set of \(M\) new draws from \(\hat{p}_{t}\) (\(\approx p_{t}\)), each of weight 1--which tends to drop low-weight particles and duplicate high-weight ones.9 For this to usefully focus the ensemble on good prefixes \(\mathbf{y}_{:t}\), \(p_{t}\) should be a good approximation to the true marginal \(p(\mathbf{y}_{:t}~{}|~{}\mathbf{x})\propto\exp\;(G_{t}+H_{t})\) from (10). That is why we arranged for \(p_{t}(\mathbf{y}_{:t})\propto\exp\;(G_{t}+C_{t})\). Without \(C_{t}\), we would have only \(p_{t}(\mathbf{y}_{:t})\propto\exp G_{t}\)--which is fine for the traditional particle filtering setting, but in our setting it ignores future information in \(\mathbf{x}_{t}\): (which we have assumed is available) and also favors sequences \(\mathbf{y}\) that happen to accumulate most of their global score \(G_{T}\) early rather than late (which is possible when the globally normalized model (1)-(2) is _not_ factored in the generative form (4)).

Footnote 9: While resampling mitigates the degeneracy problem, it could also reduce the diversity of particles. In our experiments in this paper, we only do multinomial resampling when the effective sample size of \(\hat{p}_{t}\) is lower than \(\frac{M}{2}\). Doucet and Johansen (2009) give a more thorough discussion on when to resample.

## 5 Training the Sampler Heuristic

We now consider training the parameters \(\phi\) of our sampler. These parameters determine the updates \(\bar{f}_{\phi}\) in (6) and the compatibility function \(c_{\phi}\) in (25). As a result, they determine the proposal distribution \(q\) used in equations (27) and (31), and thus determine the stochastic choice of \(\hat{p}\) that is returned by the sampler on a given input \(\mathbf{x}\).

In this paper, we simply try to tune \(\phi\) to yield good proposals. Specifically, we try to ensure that \(q_{\phi}(\mathbf{y}~{}|~{}\mathbf{x})\) in equation (27) is close to \(p(\mathbf{y}~{}|~{}\mathbf{x})\) from equation (1). While this may not be necessary for the sampler to perform well downstream,10 it does 

guarantee it (assuming that the model \(p\) is correct). Specifically, we seek to minimize

Footnote 10: The “real” goal of sampling. This objective can tolerate inaccurate local proposal distributions in cases where the algorithm could recover from them through resampling. Looking even farther downstream, we might merely want \(\hat{p}\)—which is typically used to compute expectations—to provide accurate guidance to some decision or training process (see Appendix E). This might not require fully matching the model, and might even make it desirable to deviate from an inaccurate model.

\[(1-\lambda)\text{KL}(p||q_{\phi})+\lambda\text{KL}(q_{\phi}||p)~{}~{}(\text{ with $\lambda\in[0,1]$}) \tag{32}\]

averaged over examples \(\mathbf{x}\) drawn from a training set.11 (The training set need not provide true \(\mathbf{y}\)'s.)

Footnote 11: Training a single approximation \(q_{\phi}\) for all \(\mathbf{x}\) is known as _amortized inference_.

The _inclusive KL divergence_\(\text{KL}~{}(p||q_{\phi})\) is an expectation under \(p\). We estimate it by replacing \(p\) with a sample \(\hat{p}\), which in practice we can obtain with our sampler under the current \(\phi\). (The danger, then, is that \(\hat{p}\) will be biased when \(\phi\) is not yet well-trained; this can be mitigated by increasing the sample size \(M\) when drawing \(\hat{p}\) for training purposes.)

Intuitively, this term tries to encourage \(q_{\phi}\) in future to re-propose those \(\mathbf{y}\) values that turned out to be "good" and survived into \(\hat{p}\) with high weights.

The _exclusive KL divergence_\(\text{KL}(q_{\phi}||p)\) is an expectation under \(q_{\phi}\). Since we can sample from \(q_{\phi}\) exactly, we can get an unbiased estimate of \(\nabla_{\phi}\text{KL}(q_{\phi}||p)\) with the likelihood ratio trick Glynn (1990).12 (The danger is that such "REINFORCE" methods tend to suffer from very high variance.)

Footnote 12: The normalizing constant of \(p\) from (1) can be ignored because the gradient of a constant is 0.

This term is a popular objective for variational approximation. Here, it tries to discourage \(q_{\phi}\) from re-proposing "bad" \(\mathbf{y}\) values that turned out to have low \(\exp G_{T}\) relative to their proposal probability.

Our experiments balance "recall" (inclusive) and "precision" (exclusive) by taking \(\lambda=\frac{1}{2}\) (which Appendix F compares to \(\lambda\in\{0,1\}\)). Alas, because of our approximation to the inclusive term, neither term's gradient will "find" and directly encourage good \(\mathbf{y}\) values that have never been proposed. Appendix B gives further discussion and formulas.

## 6 Models for the Experiments

To evaluate our methods, we needed pre-trained models \(p_{\theta}\). We experimented on several models. In each case, we trained a _generative_ model \(p_{\theta}(\mathbf{x},\mathbf{y})\), so that we could try sampling from its posterior distribution \(p_{\theta}(\mathbf{y}\mid\mathbf{x})\). This is a very common setting where particle smoothing should be able to help. Details for replication are given in Appendix C.

### Tagging models

We can regard a tagged sentence \((\mathbf{x},\mathbf{y})\) as a string over the "pair alphabet" \(\mathcal{X}\times\mathcal{Y}\). We train an RNN language model over this "pair alphabet"--this is a neuralized OOHMM as suggested in SS3.2:

\[\log p_{\theta}(\mathbf{x},\mathbf{y})=\sum_{t=1}^{T}\log p_{\theta}(x_{t},y_ {t}\mid\mathbf{s}_{t-1}) \tag{33}\]

This model is locally normalized, so that \(\log p_{\theta}(\mathbf{x},\mathbf{y})\) (as well as its gradient) is straightforward to compute for a given training pair \((\mathbf{x},\mathbf{y})\). Joint sampling from it would also be easy (SS3.2).

However, \(p(\mathbf{y}\mid\mathbf{x})\) is globally renormalized (by an unknown partition function that depends on \(\mathbf{x}\), namely \(\exp H_{0}\)). Conditional sampling of \(\mathbf{y}\) is therefore potentially hard. Choosing \(y_{t}\) optimally requires knowledge of \(H_{t}\), which depends on the future \(\mathbf{x}_{t}\).

As we noted in SS1, many NLP tasks can be seen as tagging problems. In this paper we experiment with two such tasks: **English stressed syllable tagging**, where the stress of a syllable often depends on the number of remaining syllables,13 providing good reason to use the _lookahead_ provided by particle smoothing; and **Chinese NER**, which is a familiar textbook application and reminds the reader that our formal setup (tagging) provides enough machinery to treat other tasks (chunking).

Footnote 13: English, like many other languages, assigns stress from right to left Hayes (1995).

English stressed syllable taggingThis task tags a sequence of phonemes \(\mathbf{x}\), which form a word, with their stress markings \(\mathbf{y}\). Our training examples are the stressed words in the CMU pronunciation dictionary Wei et al. (1998). We test the sampler on held-out unstressed words.

Chinese social media NERThis task does named entity recognition in Chinese, by tagging the characters of a Chinese sentence in a way that marks the named entities. We use the dataset from Peng and Dredze (2015), whose tagging scheme is a variant of the BIO scheme mentioned in SS1. We test the sampler on held-out sentences.

### String source separation

This is an artificial task that provides a discrete analogue of speech source separation Zibulevsky and Pearlmutter (2001). The generative model is that \(J\) strings (possibly of different lengths) are generated 

IID from an RNN language model, and are then combined into a single string according to a random _interleaving_ string \({\bf y}\).14 The posterior \(p({\bf y}~{}|~{}{\bf x})\) predicts the interleaving string, which suffices to reconstruct the original strings. The interleaving string is selected from the uniform distribution over all possible interleavings (given the \(J\) strings' lengths). For example, with \(J=2\), a possible generative story is that we first sample two strings Foo and Bar from an RNN language model. We then draw an interleaving string 112122 from the aforementioned uniform distribution, and interleave the \(J\) strings deterministically to get FoBoar.

Footnote 14: We formally describe the generative process in Appendix G.

\(p({\bf x},{\bf y})\) is proportional to the product of the probabilities of the \(J\) strings. The only parameters of \(p_{\theta}\), then, are the parameters of the RNN language model, which we train on clean (non-interleaved) samples from a corpus. We test the sampler on random interleavings of held-out samples.

The state \({\bf s}\) (which is provided as an input to \(c_{\theta}\) in (25)) is the concatenation of the \(J\) states of the language model as it independently generates the \(J\) strings, and \(g_{\theta}({\bf s}_{t-1},x_{t},y_{t})\) is the log-probability of generating \(x_{t}\) as the next character of the \(y_{t}\)th string, given that string's language model state within \({\bf s}_{t-1}\). As a special case, \({\bf x}_{T}=\textsc{eos}\) (see footnote 1), and \(g_{\theta}({\bf s}_{T-1},\textsc{eos},\textsc{eos})\) is the total log-probability of termination in all \(J\) language model states.

String source separation has good reason for lookahead: appending character "o" to a reconstructed string "_gh" is only advisable if "s" and "t" are coming up soon to make "ghost." It also illustrates a powerful application setting--posterior inference under a generative model. This task conveniently allowed us to construct the generative model from a pre-trained language model. Our constructed generative model illustrates that the state \({\bf s}\) and transition function \(f\) can reflect interesting problem-specific structure.

CMU Pronunciation dictionaryThe CMU pronunciation dictionary (already used above) provides sequences of phonemes. Here we use words no longer than \(5\) phonemes. We interleave the (unstressed) phonemes of \(J=5\) words.

Penn TreebankThe PTB corpus Marcus et al. (1993) provides English sentences, from which we use only the sentences of length \(\leq 8\). We interleave the words of \(J=2\) sentences.

## 7 Experiments

In our experiments, we are given a pre-trained scoring model \(p_{\theta}\), and we train the parameters \(\phi\) of a particle smoothing algorithm.15

Footnote 15: For the details of the training procedures and the specific neural architectures in our models, see Appendices C and D.

We now show that our proposed neural particle smoothing sampler does better than the particle filtering sampler. To define "better," we evaluate samplers on the _offset KL divergence_ from the true posterior.

### Evaluation metrics

Given \({\bf x}\), the "natural" goal of conditional sampling is for the sample distribution \(\hat{p}({\bf y})\) to approximate the true distribution \(p_{\theta}({\bf y}~{}|~{}{\bf x})=\exp G_{T}/\exp H_{0}\) from (1). We will therefore report--averaged over all held-out test examples \({\bf x}\)--the KL divergence

\[\text{KL}(\hat{p}||p)=\mathbb{E}_{{\bf y}\sim\hat{p}}\left[\log \hat{p}({\bf y})\right]\\ -(\mathbb{E}_{{\bf y}\sim\hat{p}}\left[\log\tilde{p}({\bf y}~{}| ~{}{\bf x})\right]-\log Z({\bf x})),\]

where \(\tilde{p}({\bf y}~{}|~{}{\bf x})\) denotes the _unnormalized_ distribution given by \(\exp G_{T}\) in (2), and \(Z({\bf x})\) denotes its normalizing constant, \(\exp H_{0}=\sum_{\mathcal{Y}}\tilde{p}({\bf y}~{}|~{}{\bf x})\).

As we are unable to compute \(\log Z({\bf x})\) in practice, we replace it with an estimate \(z({\bf x})\) to obtain an _offset KL divergence_. This change of constant does not change the measured difference between two samplers, \(\text{KL}(\hat{p}_{1}||p)-\text{KL}(\hat{p}_{2}||p)\). Nonetheless, we try to use a reasonable estimate so that the reported KL divergence is interpretable in an absolute sense. Specifically, we take \(z({\bf x})=\log\sum_{{\bf y}\in\mathcal{Y}}\tilde{p}({\bf y}~{}|~{}{\bf x})\leq \log Z\), where \(\mathcal{Y}\) is the full set of distinct particles \({\bf y}\) that we ever drew for input \({\bf x}\), including samples from the beam search models, while constructing the experimental results graph.16 Thus, the offset KL divergence is a "best effort" lower bound on the true exclusive KL divergence \(\text{KL}(\hat{p}||p)\).

Footnote 16: Thus, \(\mathcal{Y}\) was collected across all samplings, iterations,and ensemble sizes \(M\), in an attempt to make the summation over \(\mathcal{Y}\) as complete as possible. For good measure, we added some extra particles: whenever we drew \(M\) particles via particle smoothing, we drew an additional \(2M\) particles by particle filtering and added them to \(\mathcal{Y}\).

### Results

In all experiments we compute the offset KL divergence for both the particle filtering samplers and the particle smoothing samplers, for varying ensemble sizes \(M\). We also compare against a beam search baseline that keeps the highest-scoring \(M\) particles at each step (scored by \(\exp G_{t}\) with no lookahead). The res 

Given a fixed ensemble size, we see the smoothing sampler consistently performs better than the filtering counterpart. It often achieves comparable performance at a fraction of the ensemble size.

Beam search on the other hand falls behind on three tasks: stress prediction and the two source separation tasks. It does perform better than the stochastic methods on the Chinese NER task, but only at small beam sizes. Varying the beam size barely affects performance at all, across all tasks. This suggests that beam search is unable to explore the hypothesis space well.

We experiment with resampling for both the particle filtering sampler and our smoothing sampler. In source separation and stressed syllable prediction, where the right context contains critical information about how viable a particle is, resampling helps particle filtering _almost_ catch up to particle smoothing. Particle smoothing itself is not further improved by resampling, presumably because its effective sample size is high. The goal of resampling is to kill off low-weight particles (which were overproposed) and reallocate their resources to higher-weight ones. But with particle smoothing, there are fewer low-weight particles, so the benefit of resampling may be outweighted by its cost (namely, increased variance).

## 8 Related Work

Much previous work has employed sequential importance sampling for approximate inference of intractable distributions (e.g., Thrun, 2000; Andrews et al., 2017). Some of this work learns adaptive proposal distributions in this setting (e.g. Gu et al., 2015; Paige and Wood, 2016). The key difference in our work is that we consider future inputs, which is impossible in online decision settings such as robotics. Klaas et al. (2006) did do particle smoothing, like us, but they did not learn adaptive proposal distributions.

Just as we use a right-to-left RNN to guide _posterior sampling_ of a left-to-right generative model, Krishnan et al. (2017) employed a right-to-left RNN to guide _posterior marginal inference_ in the same sort of model. Serdyuk et al. (2018) used a right-to-left RNN to regularize training of such a model.

## 9 Conclusion

We have described neural particle smoothing, a sequential Monte Carlo method for approximate sampling from the posterior of incremental neural scoring models. Sequential importance sampling has arguably been underused in the natural language processing community. It is quite a plausible strategy for dealing with rich, globally normalized probability models such as neural models--particularly if a good sequential proposal distribution can be found. Our contribution is a neural proposal distribution, which goes beyond particle filtering in that it uses a right-to-left recurrent neural network to "look ahead" to future symbols of \({\bf x}\) when proposing each symbol \(y_{t}\). The form of our distribution is well-motivated.

There are many possible extensions to the work in this paper. For example, we can learn the generative model and proposal distribution jointly; we can also infuse them with hand-crafted structure, or use more deeply stacked architectures; and we can try training the proposal distribution end-to-end (footnote 10). Another possible extension would be to allow each step of \(q\) to propose a _sequence_ of actions, effectively making the tagset size \(\infty\). This extension relaxes our \(|{\bf y}|=|{\bf x}|\) restriction from SS1 and would allow us to do general sequence-to-sequence transduction.

### Acknowledgements

This work has been generously supported by a Google Faculty Research Award and by Grant No. 1718846 from the National Science Foundation.

Figure 2: Offset KL divergences for the tasks in §§ 6.1 and 6.2. The logarithmic \(x\)-axis is the size of particles \(M\) (\(8\leq M\leq 128\)). The \(y\)-axis is the offset KL divergence described in §7.1 (in bits per sequence). The smoothing samplers offer considerable speedup; for example, in Figure 1(a), the non-resampled smoothing sampler achieves comparable offset KL divergences with only \(\nicefrac{{1}}{{4}}\) as many particles as its filtering counterparts. Abbreviations in the legend: PF=particle filtering. PS=particle smoothing. BEAM=beam search. ‘:R’ strikes indicate resampled variants. For readability, beam search results are omitted from Figure 1(d), but appear in Figure 3 of the appendices.

 



## References

* Aharoni and Goldberg (2017) Roee Aharoni and Yoav Goldberg. 2017. Morphological inflection generation with hard monotonic attention. In _ACL_.
* Andor et al. (2016) Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. In _ACL_.
* Andrews et al. (2017) Nicholas Andrews, Mark Dredze, Benjamin Van Durme, and Jason Eisner. 2017. Bayesian modeling of lexical resources for low-resource settings. In _ACL_.
* Bahdanau et al. (2017) Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. An actor-critic algorithm for sequence prediction. In _ICLR_.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _ICLR_.
* Bengio and Frasconi (1996) Yoshua Bengio and Paolo Frasconi. 1996. Input-output HMMs for sequence processing. _IEEE Transactions on Neural Networks_, 7(5):1231-1249.
* Boser et al. (1992) Bernhard E. Boser, Isabelle M. Guyon, and Vladimir N. Vapnik. 1992. A training algorithm for optimal margin classifiers. In _COLT_.
* Bouchard-Cote et al. (2007) Alexandre Bouchard-Cote, Percy Liang, Thomas Griffiths, and Dan Klein. 2007. A probabilistic approach to diachronic phonology. In _EMNLP-CoNLL_, pages 887-896.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _EMNLP_.
* Coterell et al. (2017) Ryan Coterell, John Sylak-Glassman, and Christo Kirov. 2017. Neural graphical models over strings for principal parts morphological paradigm completion. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 759-765.
* Douc and Cappe (2005) Randal Douc and Olivier Cappe. 2005. Comparison of resampling schemes for particle filtering. In _Image and Signal Processing and Analysis, 2005. ISPA 2005. Proceedings of the 4th International Symposium on_, pages 64-69. IEEE.
* Doucet and Johansen (2009) Arnaud Doucet and Adam M. Johansen. 2009. A tutorial on particle filtering and smoothing: Fifteen years later. _Handbook of Nonlinear Filtering_, 12(656-704):3.
* Dreyer and Eisner (2009) Markus Dreyer and Jason Eisner. 2009. Graphical models over multiple strings. In _EMNLP_.
* Dreyer et al. (2008) Markus Dreyer, Jason R. Smith, and Jason Eisner. 2008. Latent-variable modeling of string transductions with finite-state methods. In _EMNLP_.
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In _ACL_.
* Dyer et al. (2016) Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In _HLT-NAACL_.
* Finkel et al. (2006) Jenny Rose Finkel, Christopher D. Manning, and Andrew Y. Ng. 2006. Solving the problem of cascading errors: Approximate Bayesian inference for linguistic annotation pipelines. In _EMNLP_.
* Glynn (1990) Peter W. Glynn. 1990. Likelihood ratio gradient estimation for stochastic systems. _Communications of the ACM_, 33(10):75-84.
* Gu et al. (2015) Shixiang Gu, Zoubin Ghahramani, and Richard E. Turner. 2015. Neural adaptive sequential Monte Carlo. In _NIPS_.
* Hart et al. (1968) Peter E. Hart, Nils J. Nilsson, and Bertram Raphael. 1968. A formal basis for the heuristic determination of minimal cost paths. 4(2):100-107.
* Hayes (1995) Bruce Hayes. 1995. _Metrical Stress Theory: Principles and Case Studies_. University of Chicago Press.
* Ihler and McAllester (2009) Alexander T. Ihler and David A. McAllester. 2009. Particle belief propagation. In _AISTATS_.
* Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with Gumbel-softmax. In _ICLR_.
* Kann and Schutze (2016) Katharina Kann and Hinrich Schutze. 2016. Single-model encoder-decoder with explicit morphological representation for reinflection. In _ACL_.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _ICLR_.
* Klaas et al. (2006) Mike Klaas, Mark Briers, Nando de Freitas, Arnaud Doucet, Simon Maskell, and Dustin Lang. 2006. Fast particle smoothing: If I had a million particles. In _ICML_.

* Krishnan et al. (2017) Rahul G. Krishnan, Uri Shalit, and David Sontag. 2017. Structured inference networks for nonlinear state space models. In _AAAI_.
* Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In _ICML_.
* Lienart et al. (2015) Thibaut Lienart, Yee Whye Teh, and Arnaud Doucet. 2015. Expectation particle belief propagation. In _NIPS_

* Little and Rubin (1987) Roderick J. A. Little and Donald B. Rubin. 1987. _Statistical Analysis with Missing Data_. J. Wiley & Sons, New York.
* Maddison et al. (2017) Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The concrete distribution: A continuous relaxation of discrete random variables. In _ICLR_.
* Marcus et al. (1993) Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatric Santorini. 1993. Building a large annotated corpus of English: The Penn treebank. _Computational Linguistics_, 19(2):313-330.
* McCallum et al. (2000) Andrew McCallum, Dayne Freitag, and Fernando Pereira. 2000. Maximum entropy Markov models for information extraction and segmentation. In _Machine Learning: Proceedings of the 17th International Conference (ICML 2000)_, pages 591-598, Stanford, CA.
* Mikolov et al. (2010) Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In _Interspeech_, volume 2, page 3.
* Paige and Wood (2016) Brooks Paige and Frank D. Wood. 2016. Inference networks for sequential Monte Carlo in graphical models. In _ICALL_.
* Paige et al. (2014) Brooks Paige, Frank D. Wood, Arnaud Doucet, and Yee Whye Teh. 2014. Asynchronous anytime sequential Monte Carlo. In _NIPS_.
* Peng and Dredze (2015) Nanyun Peng and Mark Dredze. 2015. Named entity recognition for Chinese social media with jointly trained embeddings. In _EMNLP_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In _EMNLP_.
* Pereira and Riley (1997) Fernando C. N. Pereira and Michael D. Riley. 1997. Speech recognition by composition of weighted finite automata. _Finite-State Language Processing_, page 431.
* Rabiner (1989) Lawrence R. Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. _Proceedings of IEEE_, 77(2):257-285.
* Ramshaw and Marcus (1999) Lance A. Ramshaw and Mitchell P. Marcus. 1999. Text chunking using transformation-based learning. In _Natural Language Processing Using Very Large Corpora_, pages 157-176. Springer.
* Ristic et al. (2004) Branko Ristic, Sanjeev Arulampalam, and Neil James Gordon. 2004. _Beyond the Kalman Filter: Particle Filters for Tracking Applications_. Artech House.
* Robbins and Monro (1951) Herbert Robbins and Sutton Monro. 1951. A stochastic approximation method. _The Annals of Mathematical Statistics_, pages 400-407.
* Serdyuk et al. (2018) Dmitris Serdyuk, Nan Rosemary Ke, Alessandro Sordoni, Adam Trischler, Chris Pal, and Yoshua Bengio. 2018. Twin networks: Matching the future for sequence generation. In _ICLR_.
* Stuhlmuller et al. (2013) Andreas Stuhlmuller, Jacob Taylor, and Noah Goodman. 2013. Learning stochastic inverses. In _NIPS_.
* Thrun (2000) Sebastian Thrun. 2000. Monte Carlo POMDPs. In _NIPS_.
* Viterbi (1967) Andrew J. Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. _IEEE Transactions on Information Theory_, IT-13(2):260-269.
* Wei and Tanner (1990) Greg C. G. Wei and Martin A. Tanner. 1990. A Monte Carlo implementation of the EM algorithm and the poor man's data augmentation algorithms. _Journal of the American Statistical Association_, 85(411):699-704.
* Weide (1998) Robert L. Weide. 1998. The CMU pronunciation dictionary, release 0.6.
* Williams (1992) Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Machine Learning_, 8(23).
* Wiseman and Rush (2016) Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In _EMNLP_.
* Yamada and Matsumoto (2003) Hiroyan Yamada and Yuji Matsumoto. 2003. Statistical dependency analysis with support vector machines. In _Proceedings of IWPT_, volume 3, pages 195-206.
* Zibulevsky and Pearlmutter (2001) Michael Zibulevsky and Barak A. Pearlmutter. 2001. Blind source separation by sparse decomposition in a signal dictionary. _Neural Computation_, 13(4):863-882.

 

The logprob-to-go for HMMs

As noted in SS2.1, the logprob-to-go \(H_{t}\) can be computed by the backward algorithm. By the definition of \(H_{t}\) in equation (10),

\[\exp H_{t} =\sum_{\mathbf{y}_{t:}}\exp~{}(G_{T}-G_{t}) \tag{35}\] \[=\sum_{\mathbf{y}_{t:}}\exp\sum_{j=t+1}^{T}g_{\theta}(\mathbf{s}_ {j-1},x_{j},y_{j})\] (36) \[=\sum_{\mathbf{y}_{t:}}\prod_{j=t+1}^{T}p_{\theta}(y_{j}\mid y_{ j-1})\cdot p_{\theta}(x_{j}\mid y_{j})\] \[=(\boldsymbol{\beta}_{t})_{y_{t}}~{}~{}\text{(backward prob of $y_{t}$ at time $t$)}\]

where the vector \(\boldsymbol{\beta}_{t}\) is defined by base case \((\boldsymbol{\beta}_{T})_{y}=1\) and for \(0\leq t<T\) by the recurrence

\[(\beta_{t})_{y} \stackrel{{\mathrm{def}}}{{=}}\sum_{\mathbf{y}_{t: }}p_{\theta}(\mathbf{x}_{t:},\mathbf{y}_{t:}\mid y_{t}=y) \tag{37}\] \[=\sum_{y^{\prime}}p_{\theta}(y^{\prime}\mid y)\cdot p_{\theta}( x_{t+1}\mid y^{\prime})\cdot(\boldsymbol{\beta}_{t+1})_{y^{\prime}}\]

The backward algorithm (20) for OOHMMs in SS2.2 is a variant of this.

## Appendix B Gradients for Training the Proposal Distribution

For a given \(\mathbf{x}\), both forms of KL divergence achieve their minimum of 0 when \((\forall\mathbf{y})~{}q_{\phi}(\mathbf{y}\mid\mathbf{x})=p(\mathbf{y}\mid \mathbf{x})\). However, we are unlikely to be able to find such a \(\phi\); the two metrics penalize \(q_{\phi}\) differently for mismatches. We simplify the notation below by writing \(q_{\phi}(\mathbf{y})\) and \(p(\mathbf{y})\), suppressing the conditioning on \(\mathbf{x}\).

Inclusive KL DivergenceThe inclusive KL divergence has that name because it is finite only when \(\text{support}(q_{\phi})\supseteq\text{support}(p)\), i.e., when \(q_{\phi}\) is capable of proposing any string \(\mathbf{y}\) that has positive probability under \(p\). This is required for \(q_{\phi}\) to be a valid proposal distribution for importance sampling.

KL \[(p||q_{\phi})\] (38) \[\qquad=\mathbb{E}_{\mathbf{y}\sim p}\left[\log p\left(\mathbf{y }\right)-\log q_{\phi}(\mathbf{y})\right]\] \[\qquad=\mathbb{E}_{\mathbf{y}\sim p}\left[\log p\left(\mathbf{y }\right)\right]\] \[\qquad\quad-\mathbb{E}_{\mathbf{y}\sim p}\left[\log q_{\phi} \left(\mathbf{y}\right)\right]\]

The first term \(\mathbb{E}_{\mathbf{y}\sim p}\left[\log p\left(\mathbf{y}\right)\right]\) is a constant with regard to \(\phi\). As a result, the gradient of the above is just the gradient of the second term:

We cannot directly sample from \(p\). However, our weighted mixture \(\hat{p}\) from equation (28) (obtained by sequential importance sampling) could be a good approximation:

\[\nabla_{\phi}\text{KL}(p||q_{\phi}) \approx\nabla_{\phi}\mathbb{E}_{\mathbf{y}\sim\hat{p}}\left[- \log q_{\phi}\left(\mathbf{y}\right)\right] \tag{39}\] \[=\sum_{t=1}^{T}\mathbb{E}_{\hat{p}}\left[-\nabla_{\phi}\log q_{ \phi}(y_{t}\mid y_{t-1},\mathbf{x})\right]\]

Following this approximate gradient downhill has an intuitive interpretation: if a particular \(y_{t}\) value ends up with high relative weight in the final ensemble \(\hat{p}\), then we will try to adjust \(q_{\phi}\) so that it would have had a high probability of proposing that \(y_{t}\) value at step \(t\) in the first place.

Exclusive KL DivergenceThe exclusive divergence has that name because it is finite only when \(\text{support}(q_{\phi})\subseteq\text{support}(p)\). It is defined by

KL \[(q_{\phi}||p)=\mathbb{E}_{\mathbf{y}\sim q_{\phi}}\left[\log q_{ \phi}(\mathbf{y})-\log p(\mathbf{y})\right]\] (40) \[=\mathbb{E}_{\mathbf{y}\sim q_{\phi}}\left[\log q_{\phi}(\mathbf{ y})-\log\tilde{p}(\mathbf{y})\right]+\log Z\] \[=\sum_{\mathbf{y}}q_{\phi}(\mathbf{y})\underbrace{\left[\log q_{ \phi}(\mathbf{y})-\log\tilde{p}(\mathbf{y})\right]}_{\text{call this $d_{\phi}(\mathbf{y})$}}+\log Z\]

where \(p(\mathbf{y})=\frac{1}{Z}\hat{p}(\mathbf{y})\) for \(\tilde{p}(\mathbf{y})=\exp G_{T}\) and \(Z=\sum_{\mathbf{y}}\tilde{p}(\mathbf{y})\). With some rearrangement, we can write its gradient as an expectation that can be estimated by sampling from \(q_{\phi}\).17 Observing that \(Z\) is constant with respect to \(\phi\), first write

Footnote 17: This is an extension of the REINFORCE trick Williams (1992), which estimates the gradient of \(\mathbb{E}_{\mathbf{y}\sim q_{\phi}}\left[\text{reward}(\mathbf{y})\right]\) when the reward is independent of \(\phi\). In our case, the expectation is over a quantity that does depend on \(\phi\).

\[\nabla_{\phi}\text{KL} (q_{\phi}||p) \tag{41}\] \[=\sum_{\mathbf{y}}\nabla_{\phi}\left(q_{\phi}(\mathbf{y})\,d_{ \phi}(\mathbf{y})\right)\] (42) \[=\sum_{\mathbf{y}}\left(\nabla_{\phi}q_{\phi}(\mathbf{y})\right)\, d_{\phi}(\mathbf{y})\] \[\qquad+\sum_{\mathbf{y}}\underbrace{q_{\phi}(\mathbf{y})\nabla_{ \phi}\log q_{\phi}(\mathbf{y})}_{=\nabla_{\phi}q_{\phi}(\mathbf{y})}\] \[=\sum_{\mathbf{y}}\left(\nabla_{\phi}q_{\phi}(\mathbf{y})\right)\, d_{\phi}(\mathbf{y})\]

where the last step uses the fact that \(\sum_{\mathbf{y}}\nabla_{\phi}q_{\phi}(\mathbf{y})=\nabla_{\phi}\sum_{\mathbf{y }}q_{\phi}(\mathbf{y})=\nabla_{\phi}1=0\). We can turn this into an expectation with a second use of Glynn (1990)'s observation that 

\(\nabla_{\phi}q_{\phi}(\mathbf{y})=q_{\phi}(\mathbf{y})\nabla_{\phi}\log q_{\phi }(\mathbf{y})\) (the "likelihood ratio trick"):

\[\nabla_{\phi}\text{KL} (q_{\phi}||p)\] \[=\sum_{\mathbf{y}}q_{\phi}(\mathbf{y})d_{\phi}(\mathbf{y})\nabla_{ \phi}\log q_{\phi}(\mathbf{y})\] \[=\mathbb{E}_{\mathbf{y}\sim q_{\phi}}\left[d_{\phi}(\mathbf{y}) \nabla_{\phi}\log q_{\phi}(\mathbf{y})\right] \tag{43}\]

which can, if desired, be further rewritten as

\[=\mathbb{E}_{\mathbf{y}\sim q_{\phi}}\left[d_{\phi}(\mathbf{y}) \nabla_{\phi}\,d_{\phi}(\mathbf{y})\right]\] \[=\mathbb{E}_{\mathbf{y}\sim q_{\phi}}\left[\nabla_{\phi}\left( \tfrac{1}{2}d_{\phi}(\mathbf{y})^{2}\right)\right] \tag{44}\]

If we regard \(d_{\phi}(\mathbf{y})\) as a signed error (in the log domain) in trying to fit \(q_{\phi}\) to \(\tilde{p}\), then the above gradient of KL can be interpreted as the gradient of the mean squared error (divided by 2).18

Footnote 18: We thank Hongyuan Mei, Tim Vieira, and Sanjeev Khudanpur for insightful discussions on this derivation.

We would get the same gradient for any rescaled version of the unnormalized distribution \(\tilde{p}\), but the formula for obtaining that gradient would be different. In particular, if we rewrite the above derivation but add a constant \(b\) to both \(\log\tilde{p}(\mathbf{y})\) and \(\log Z\) throughout (equivalent to adding \(b\) to \(G_{T}\)), we will get the slightly generalized expectation formulas

\[\mathbb{E}_{\mathbf{y}\sim d_{\phi}}\left[(d_{\phi}(\mathbf{y})- b)\nabla_{\phi}\log q_{\phi}(\mathbf{y})\right] \tag{45}\] \[\mathbb{E}_{\mathbf{y}\sim q_{\phi}}\left[\nabla_{\phi}\left( \tfrac{1}{2}\left(d_{\phi}(\mathbf{y})-b\right)^{2}\right)\right] \tag{46}\]

in place of equations (43) and (44). By choosing an appropriate "baseline" \(b\), we can reduce the variance of the sampling-based estimate of these expectations. This is similar to the use of a baseline in the REINFORCE algorithm Williams (1992). In this work we choose \(b\) using an exponential moving average of past \(\mathbb{E}\left[d_{\phi}(\mathbf{y})\right]\) values: at the end of each training minibatch, we update \(b\gets 0.1\cdot b+0.9\cdot\bar{d}\), where \(\bar{d}\) is the mean of the estimated \(\mathbb{E}_{\mathbf{y}\sim q_{\phi}(\cdot|\mathbf{x})}\left[d_{\phi}(\mathbf{ y})\right]\) values for all examples \(\mathbf{x}\) in the minibatch.

## Appendix C Implementation Details

We implement all RNNs in this paper as GRU networks Cho et al. (2014) with \(d=32\) hidden units (state space \(\mathbb{R}^{32}\)). Each of our models (SS6) always specifies the logprob-so-far in equations (2) and (3) using a 1-layer left-to-right GRU,19 while the corresponding proposal distribution (SS3.3) always specifies the state \(\overline{\mathbf{s}}_{t}\) in (6) using a 2-layer right-to-left GRU, and specifies the compatibility function \(C_{t}\) in (23) using a 4-layer feedforward ReLU network.20 For the Chinese social media NER task (SS6.1), we use the Chinese character embeddings provided by Peng and Dredze (2015), while for the source separation tasks (SS6.2), we use the 50-dimensional GloVe word embeddings Pennington et al. (2014). In other cases, we train embeddings along with the rest of the network. We optimize with the Adam optimizer using the default parameters Kingma and Ba (2015) and \(L_{2}\) regularization coefficient of \(10^{-5}\).

Footnote 19: For the tagging task described in §6.1, \(g_{\theta}(\mathbf{s}_{t-1},x_{t},y_{t})\stackrel{{\text{def}}}{{ =}}\log p_{\theta}(x_{t},y_{t}~{}|~{}\mathbf{s}_{t-1})\), where the GRU state \(\mathbf{s}_{t-1}\) is used to define a softmax distribution over possible \((x_{t},y_{t})\) pairs in the same manner as an RNN language model Mikolov et al. (2010). Likewise, for the source separation task (§6.2), the source language models described in Appendix G are GRU-based RNN language models.

## Appendix D Training Procedures

In all our experiments, we train the incremental scoring models (the tagging and source separation models described in SS6.1 and SS6.2, respectively) on the training dataset \(T\). We do early stopping, using perplexity on a held-out development set \(D_{1}\) to choose the number of epochs to train (maximum of \(3\)).

Having obtained these model parameters \(\theta\), we train our proposal distributions \(q_{\theta,\phi}\) on \(T\), keeping \(\theta\) fixed and only tuning \(\phi\). Again we use early stopping, using the KL divergence from SS7.1 on a separate development set \(D_{2}\) to choose the number of epochs to train (maximum of \(20\) for the two tagging tasks and source separation on the PTB dataset, and maximum of \(50\) for source separation on the phoneme sequence dataset). We then evaluate \(q_{\theta^{*},\phi^{*}}\) on the test dataset \(E\).

**Appendices E-G appear in the supplementary material file.]**

Figure 3: Offset KL divergence for the source separation task on phoneme sequences.