Venue: NAACL 2019

# Neural Finite State Transducers: Beyond Rational Relations

 Chu-Cheng Lin

Computer Science Department

Johns Hopkins University

Baltimore, MD 21218, USA kitsing@cs.jhu.edu Hao Zhu

Dept. of Computer Science and Technology

Tsinghua University

Beijing, China zhuhao15@mails.tsinghua.edu.cn Matthew R. Gormley

Machine Learning Department

Carnegie Mellon University

Pittsburgh, PA 15213, USA mgormley@cs.cmu.edu Jason Eisner

Computer Science Department

Johns Hopkins University

Baltimore, MD 21218, USA jason@cs.jhu.edu

###### Abstract

We introduce neural finite state transducers (NFSTs), a family of string transduction models defining joint and conditional probability distributions over pairs of strings. The probability of a string pair is obtained by marginalizing over all its accepting paths in a finite state transducer. In contrast to ordinary weighted FSTs, however, each path is scored using an arbitrary function such as a recurrent neural network, which breaks the usual conditional independence assumption (Markov property). NFSTs are more powerful than previous finite-state models with neural features [7]. We present training and inference algorithms for locally and globally normalized variants of NFSTs. In experiments on different transaction tasks, they compete favorably against seq2seq models while offering interpretable paths that correspond to hard monotonic alignments.

## 1 Introduction

Weighted finite state transducers (WFSTs) have been used for decades to analyze, align, and transduce strings in language and speech processing [6, 4]. They form a family of efficient, interpretable models with well-studied theory. A WFST describes a function that maps each string pair \(({\bf x},{\bf y})\) to a weight--often a real number representing \(p({\bf x},{\bf y})\) or \(p({\bf y}~{}|~{}{\bf x})\). The WFST is a labeled graph, in which each path \({\bf a}\) represents a sequence of operations that describes how some \({\bf x}\) and some \({\bf y}\) could be jointly generated, or how \({\bf x}\) could be edited into \({\bf y}\). Multiple paths for the same \(({\bf x},{\bf y})\) pair correspond to different analyses (labeled alignments) of that pair.

However, WFSTs can only model certain functions, known as the rational relations [9].The weight of a path is simply the product of the weights on its arcs. This means that in a random path of the form \(a\leadsto b\leadsto c\), the two subpaths \(\leadsto\) are conditionally independent given their common state \(b\): a Markov property.

In this paper, we propose neural finite state transducers (NFSTs), in which the weight of each path is instead given by some sort of neural network, such as an RNN. Thus, the weight of an arc can depend on the context in which the arc is used. By abandoning the Markov property, we lose exact dynamic programming algorithms, but we gain expressivity: the neural network can capture dependencies among the operations along a path. For example, the RNN might give higher weight to a path if it is "internally consistent": it might thus prefer to transcribe a speaker's utterance with a path that maps similar sounds in similar contexts to similar phonemes, thereby adapting to the speaker's accent.

Figure 1: A marked finite-state transducer \({\cal T}\). Each arc in \({\cal T}\) is associated with input and output substrings, listed above the arcs in the figure. The arcs are not labeled with weights as in WFSTs. Rather, each arc is labeled with a sequence of _marks_ (shown in brown) that featurize its qualities. The neural scoring model scores a path by scoring each mark in the context of all marks on the entire path. The example shown here is from the G2P application of §4.1; for space, only a few arcs are shown. \(\varepsilon\) represents the empty string.



Consider a finite-state transducer \({\cal T}\) as in Figure 1 (see Appendix A for background). Using the composition operator \(\circ\), we can obtain a new FST, \({\bf x}\circ{\cal T}\), whose accepting paths correspond to the accepting paths of \({\cal T}\) that have input string \({\bf x}\). Similarly, the accepting paths of \({\cal T}\circ{\bf y}\) correspond to the accepting paths of \({\cal T}\) that have output string \({\bf y}\). Finally, \({\bf x}\circ T\circ{\bf y}\) extracts the paths that have both properties. We define a joint probability distribution over \(({\bf x},{\bf y})\) pairs by marginalizing over those paths:

\[p({\bf x},{\bf y})=\!\!\sum_{{\bf a}\in{\bf x}\circ{\cal T}\circ{\bf y}}\!\!\! p({\bf a})=\frac{1}{Z({\cal T})}\sum_{{\bf a}\in{\bf x}\circ{\cal T}\circ{\bf y}}\!\! \tilde{p}({\bf a}) \tag{1}\]

where \(\tilde{p}({\bf a})\) is the weight of path \({\bf a}\) and \(Z({\cal T})=\sum_{{\bf a}\in{\cal T}}\tilde{p}({\bf a})\) is a normalization constant.

We define \(\tilde{p}({\bf a})\triangleq\exp G_{\boldsymbol{\theta}}({\bf a})\) with \(G_{\boldsymbol{\theta}}({\bf a})\) being some parametric scoring function. In our experiments, we will adopt a fairly simple left-to-right RNN architecture (SS2.2), but one could easily substitute fancier architectures. We will also consider defining \(G_{\boldsymbol{\theta}}\) by a _locally normalized_ RNN that ensures \(Z({\cal T})=1\).

In short, we use the finite-state transducer \({\cal T}\) to compactly define a set of _possible_ paths \({\bf a}\). The number of paths may be exponential in the size of \({\cal T}\), or infinite if \({\cal T}\) is cyclic. However, in contrast to WFSTs, we abandon this combinatorial structure in favor of neural nets when defining the probability distribution over \({\bf a}\). In the resulting marginal distribution \(p({\bf x},{\bf y})\) given in equation (1), the path \({\bf a}\) that _aligns_\({\bf x}\) and \({\bf y}\) is a latent variable. This is also true of the resulting conditional distribution \(p({\bf y}~{}|~{}{\bf x})\).

We explore training and inference algorithms for various classes of NFST models (SS3). Classical WFSTs Mohri et al. (2008) and BiRNN-WFSTs Rastogi et al. (2016) use restricted scoring functions and so admit exact dynamic programming algorithms. For general NFSTS, however, we must resort to approximate computation of the model's training gradient, marginal probabilities, and predictions. In this paper, we will use sequential importance sampling methods Lin and Eisner (2018), leaving variational approximation methods to future work.

Defining models using FSTs has several benefits:

**Output-sensitive encoding** Currently popular models of \(p({\bf y}~{}|~{}{\bf x})\) used in machine translation and morphology include seq2seq Sutskever et al. (2014), seq2seq with attention Bahdanau et al. (2015); Luong et al. (2015), the Transformer Vaswani et al. (2017). These models first encode \({\bf x}\) as a vector or sequence of vectors, and then condition the generation of \({\bf y}\) on this encoding. The vector is determined from \({\bf x}\) only. This is also the case in the BiRNN-WFST Rastogi et al. (2016), a previous finite-state model to which we compare. By contrast, in our NFST, the state of the RNN as it reads and transduces the second half of \({\bf x}\) is influenced by the first halves of both \({\bf x}\)_and_\({\bf y}\) and their alignment.
**Inductive bias** Typically, a FST is constructed with domain knowledge (possibly by compiling a regular expression), so that its states reflect interpretable properties such as syllable boundaries or linguistic features. Indeed, we will show below how to make these properties explicit by "marking" the FST arcs. The NFST's path scoring function then sees these marks and can learn to take them into account. The NFST also inherits any hard constraints from the FST: if the FST omits all \(({\bf x},{\bf y})\) paths for some "illegal" \({\bf x},{\bf y}\), then \(p({\bf x},{\bf y})=0\) for any parameter vector \(\boldsymbol{\theta}\) (a "structural zero").
**Interpretability** Like a WFST, an NFST can "explain" why it mapped \({\bf x}\) to \({\bf y}\) in terms of a latent path \({\bf a}\), which specifies a hard monotonic labeled alignment. The posterior distribution \(p({\bf a}~{}|~{}{\bf x},{\bf y})\) specifies which paths \({\bf a}\) are the best explanations (e.g., Table 5).

We conduct experiments on three tasks: grapheme-to-phoneme, phoneme-to-grapheme, and action-to-command Bastings et al. (2018). Our results on these datasets show that our best models can improve over neural seq2seq and previously proposed hard alignment models.

## 2 Neuralizing Finite-State Transducers

### Neuralized FSTs

An NFST is a pair \(({\cal T},G_{\boldsymbol{\theta}})\), where \({\cal T}\) is an unweighted FST with accepting paths \({\cal A}\) and \(G_{\boldsymbol{\theta}}:{\cal A}\rightarrow\mathbb{R}\) is a function that **scores** these paths. As explained earlier, we then refer to \(\tilde{p}({\bf a})=\exp G_{\boldsymbol{\theta}}({\bf a})\) as the **weight** of path \({\bf a}\in{\cal A}\). A **weighted relation** between input and output strings is given by \(\tilde{p}({\bf x},{\bf y})\), which is defined to be the total weight of all paths with input string \({\bf x}\in\Sigma^{*}\) and output string \({\bf y}\in\Delta^{*}\), where where \(\Sigma\) and \(\Delta\) are the input and output alphabets of \({\cal T}\). The real parameter vector \(\boldsymbol{\theta}\) can be adjusted to obtain different weighted relations. We 

can normalize \(\tilde{p}\) to get a probability distribution as shown in equation (1).

### A basic scoring architecture

Weighted FST.A WFST over the \((+,\times)\) semiring can be regarded as the special case in which \(G_{\mathbf{\theta}}(\mathbf{a})\triangleq\sum_{t=1}^{[\mathbf{a}]}g_{\mathbf{\theta}}(a _{t})\). This is a sum of scores assigned to the arcs in \(\mathbf{a}=a_{1}a_{2}\cdots\).

Marked FST.Our innovation is to allow the arcs' scores to depend on their context in the path. Now \(\mathbf{\theta}\) no longer associates a fixed score with each arc. Rather, we assume that each arc \(a\) in the FST comes labeled with a sequence of **marks** from a **mark alphabet**\(\Omega\), as illustrated in Figure 1. The marks reflect the FST constructor's domain knowledge about what arc \(a\) does (see SS4.2 below). We now define \(G_{\mathbf{\theta}}(\mathbf{a})=G_{\mathbf{\theta}}(\mathbf{\omega}(\mathbf{a}))\), where \(\mathbf{\omega}(\mathbf{a})=\mathbf{\omega}(a_{1})\mathbf{\omega}(a_{2})\cdots\in\Omega^{*}\) is the concatenated sequence of marks from the arcs along path \(\mathbf{a}\).

It is sometimes helpful to divide marks into different classes. An arc can be regarded as a possible "edit" that aligns an input substring with an output substring in the context of transitioning from one FST state to another. The arc's **input marks** describe its input substring, its **output marks** describe its output substring, and the remaining marks may describe other properties of the arc's aligned input-output pair or the states that it connects.

Recall that an FST encodes domain knowledge. Its paths represent alignments between input and output strings, where each alignment specifies a segmentation of \(\mathbf{x}\) and \(\mathbf{y}\) into substrings labeled with FST states. Decorating the arcs with marks furnishes the path scoring model with domain-specific information about the alignments.

RNN scoring.If \(\mathbf{\theta}\) merely associated a fixed score with each mark, then the marked FST would be no more powerful than the WFST. To obtain contextual mark scores as desired, one simple architecture is a recurrent neural network:

\[G_{\mathbf{\theta}}(\mathbf{\omega}) \triangleq\sum_{t=1}^{|\mathbf{\omega}|}g_{\mathbf{\theta}}(\mathbf{s}_{ t-1},\omega_{t}) \tag{2}\] \[\mathbf{s}_{t} =f_{\mathbf{\theta}}(\mathbf{s}_{t-1},\omega_{t}),~{}\text{with}~{} \mathbf{s}_{0}=\mathbf{0} \tag{3}\]

where \(\mathbf{s}_{t-1}\in\mathbb{R}^{d}\) is the hidden state vector of the network after reading \(\omega_{1}\cdots\omega_{t-1}\). The \(g_{\mathbf{\theta}}\) function defines the score of reading \(\omega_{t}\) in this left context, and \(f_{\mathbf{\theta}}\) defines how doing so updates the state.

In our experiments, we chose \(f_{\mathbf{\theta}}\) to be the GRU state update function Cho et al. (2014). We defined \(g_{\mathbf{\theta}}(\mathbf{s},\omega_{t})\triangleq(\mathbf{W}\mathbf{s}+ \mathbf{b})^{\top}\mathrm{emb}(\omega_{t})\). The parameter vector \(\mathbf{\theta}\) specifies the GRU parameters, \(\mathbf{W},\mathbf{b}\), and the mark embeddings \(\mathrm{emb}(\omega)\).

One could easily substitute much fancier architectures, such as a stacked BiLSTM with attention Tilk and Alumae (2016), or a Transformer Vaswani et al. (2017).

### Partitioned hidden vectors

In hopes of improving the inductive bias of the learner, we partitioned the hidden state vector into three sub-vectors: \(\mathbf{s}_{t}=[\mathbf{s}_{t}^{\rm a};\bar{\mathbf{s}}_{t}^{\rm x};\mathbf{ s}_{t}^{\rm y}]\). The mark scoring function \(f_{\mathbf{\theta}}(\mathbf{s}_{t-1},\omega_{t})\) was as before, but we restricted the form of \(g_{\mathbf{\theta}}\), the state update function. \(\mathbf{s}_{t}^{\rm a}\) encodes _all_ past marks and depends on the _full_ hidden state so far: \(\mathbf{s}_{t}^{\rm n}=g_{\mathbf{\theta}}^{\rm a}(\mathbf{s}_{t-1},\omega_{t})\). However, we make \(\mathbf{s}_{t}^{\rm x}\) encode only the sequence of past _input_ marks, ignoring all others. Thus, \(\mathbf{s}_{t}^{\rm x}=g_{\mathbf{\theta}}^{\rm g}(\mathbf{s}_{t-1}^{\rm x},\omega_ {t})\) if \(\omega_{t}\) is an input mark, and \(\mathbf{s}_{t}^{\rm x}=\mathbf{s}_{t-1}^{\rm x}\) otherwise. Symmetrically, \(\mathbf{s}_{t}^{\rm y}\) encodes only the sequence of past _output_ marks. This architecture is somewhat like Dyer et al. (2016), which also uses different sub-vectors to keep track of different aspects of the history.

### Local normalization

A difficulty with the general model form in equation (1) is that the normalizing constant \(Z(\mathcal{T})=\sum_{\mathbf{a}\in\mathcal{T}}\tilde{p}(\mathbf{a})\) must sum over a large set of paths--in fact, an infinite set if \(\mathcal{T}\) is cyclic. This sum may diverge for some values of the parameter vector \(\mathbf{\theta}\), which complicates training of the model Dreyer et al. (2016).

\begin{table}
\begin{tabular}{c l c c} \hline Model & Training Algorithms & Long-Term Output-Output Dependency & Left-to-Right Factorization \\ \hline WFSTs & Dynamic Programming & ✗ & ✓ \\ BiRNN-WFSTs & Dynamic Programming & ✗ & ✓ \\ Local NFSTs & Importance Sampling & ✓ & ✓ \\ Global NFSTs & Importance Sampling & ✓ & ✗ \\ \hline \end{tabular}
\end{table}
Table 1: Comparison between WFSTs, BiRNN-WFSTs Rastogi et al. (2016), and NFSTs.

 

2011). Even if the sum is known to converge, it is in general intractable to compute it exactly. Thus, estimating the gradient of \(Z({\mathcal{T}})\) during training involves approximate sampling from the typically high-entropy distribution \(p({\mathbf{a}})\). The resulting estimates are error-prone because the sample size tends to be too small and the approximate sampler is biased.

A standard solution in the WFST setting (e.g. Cotterell et al., 2014) is to use a locally normalized model, in which \(Z({\mathcal{T}})\) is guaranteed to be 1.1 The big summation over all paths \({\mathbf{a}}\) is replaced by small summations--which can be computed explicitly--over just the outgoing edges from a given state.

Footnote 1: Provided that every state in \({\mathcal{T}}\) is co-accessible, i.e., has a path to a final state.

Formally, we define the unnormalized score of arc \(a_{i}\) in the context of path \({\mathbf{a}}\) in the obvious way, by summing over the contextual scores of its marks:

\[\tilde{g}_{\boldsymbol{\theta}}(a_{i})\triangleq\sum_{t=j+1}^{k}g_{ \boldsymbol{\theta}}({\mathbf{s}}_{t-1},\omega_{t}) \tag{4}\]

where \(j=|\boldsymbol{\omega}(a_{1})\cdots\boldsymbol{\omega}(a_{i-1})|\) and \(k=|\boldsymbol{\omega}(a_{1})\cdots\boldsymbol{\omega}(a_{i})|\). Its normalized score is then

\[g_{\boldsymbol{\theta},{\mathcal{T}}}(a_{i})\triangleq\log\left(\exp\tilde{ g}_{\boldsymbol{\theta}}(a_{i})/{\sum}_{a^{\prime}}\exp\tilde{g}_{\boldsymbol{ \theta}}(a^{\prime})\right)\]

where \(a^{\prime}\) ranges over all arcs in \({\mathcal{T}}\) (including \(a_{i}\) itself) that emerge from the same state as \(a_{i}\) does. We can now score the paths in \({\mathcal{T}}\) using

\[G_{\boldsymbol{\theta},{\mathcal{T}}}({\mathbf{a}})=\sum_{i=1}^{|{\mathbf{a}} |}g_{\boldsymbol{\theta},{\mathcal{T}}}(a_{i}) \tag{5}\]

This gives rise to a proper probability distribution \(p({\mathbf{a}})\triangleq\tilde{p}({\mathbf{a}})=\exp G_{\boldsymbol{\theta} ,{\mathcal{T}}}({\mathbf{a}})\) over the paths of \({\mathcal{T}}\). No global normalization constant is necessary. However, note that the scoring function now requires \({\mathcal{T}}\) as an extra subscript, because it is necessary when scoring \({\mathbf{a}}\) to identify the competitors in \({\mathcal{T}}\) of each arc \(a_{i}\). Thus, when \(p({\mathbf{x}},{\mathbf{y}})\) is found as usual by summing up the probabilities of all paths in \({\mathbf{x}}\circ{\mathcal{T}}\circ{\mathbf{y}}\), each path is still scored using its arcs' competitors from \({\mathcal{T}}\). This means that each state in \({\mathbf{x}}\circ T\circ{\mathbf{y}}\) must record the state in \({\mathcal{T}}\) from which it was derived.

## 3 Sampling, Training, and Decoding

### Sampling from conditioned distributions with amortized inference

Many algorithms for working with probability distributions--including our training and decoding algorithms below--rely on conditional sampling. In general, we would like to sample a path of \({\mathcal{T}}\) given the knowledge that its input and output strings fall into sets \(X\) and \(Y\) respectively.2 If \(X\) and \(Y\) are regular languages, this is equivalent to defining \({\mathcal{T}}^{\prime}=X\circ{\mathcal{T}}\circ Y\) and sampling from

Footnote 2: When \(X\) or \(Y\) is larger than a single string, it is commonly all of \(\Sigma^{*}\) or \(\Delta^{*}\) respectively, in which case conditioning on it gives no information.

\[p({\mathbf{a}}\mid{\mathcal{T}}^{\prime})\triangleq\frac{\hat{p}({\mathbf{a} })}{\sum_{{\mathbf{a}}^{\prime}\in{\mathcal{T}}^{\prime}}\tilde{p}({\mathbf{ a}}^{\prime})}, \tag{6}\]

Due to the nonlinearity of \(G_{\boldsymbol{\theta}}\), the denominator of equation (6) is generally intractable. If \({\mathcal{T}}^{\prime}\) is cyclic, it cannot even be computed by brute-force enumeration. Thus, we fall back on normalized importance sampling, directly adopting the ideas of Lin and Eisner (2018) in our more general FST setting. We employ a proposal distribution \(q\):

\[p({\mathbf{a}}\mid{\mathcal{T}}^{\prime}) =\mathbb{E}_{{\mathbf{a}}\sim q}[\frac{p({\mathbf{a}}\mid{ \mathcal{T}}^{\prime})}{q({\mathbf{a}})}], \tag{7}\] \[\approx\sum_{m=1}^{M}\frac{\tilde{p}({\mathbf{a}}^{(m)})}{q({ \mathbf{a}}^{(m)})\cdot\hat{Z}}\cdot\mathbb{I}({\mathbf{a}}={\mathbf{a}}^{(m)})\] \[=\tilde{p}({\mathbf{a}}\mid{\mathcal{T}}^{\prime}),\]

where \(\hat{Z}=\sum_{m^{\prime}=1}^{M}\frac{\tilde{p}({\mathbf{a}}^{(m^{\prime})})}{q( {\mathbf{a}}^{(m^{\prime})})}\), and \(q\) is a locally normalized distribution over paths \({\mathbf{a}}\in{\mathcal{T}}^{\prime}\). In this paper we further parametrize \(q\) as

\[q_{\boldsymbol{\phi}}({\mathbf{a}};{\mathcal{T}}^{\prime})=\prod_{t=1}^{T}q_{ t}(a_{t}\mid{\mathbf{a}}_{1...t-1};\boldsymbol{\phi},{\mathcal{T}}^{\prime}), \tag{8}\]

\[q_{t}(a\mid{\mathbf{a}}_{t-1};\boldsymbol{\phi},{\mathcal{T}}^{\prime})\propto \exp(g({\mathbf{s}}_{t-1},a_{t};\boldsymbol{\theta},{\mathcal{T}})+C_{ \boldsymbol{\phi}}),\]

where \(C_{\boldsymbol{\phi}}\triangleq C({\mathbf{s}}^{\prime}_{t},X,Y,\boldsymbol{ \phi})\in\mathbb{R}\), \({\mathbf{s}}^{\prime}_{t}\triangleq f({\mathbf{s}}_{t-1},\boldsymbol{\omega}(a ))\) is a _compatibility function_ that is typically modeled using a neural network. In this paper, one the following three cases are encountered:

* \(X={\mathbf{x}}\), is a string, and \(Y=\Delta^{*}\): in this case \({\mathcal{T}}^{\prime}={\mathbf{x}}\circ{\mathcal{T}}\). We let \(C_{\boldsymbol{\phi}}=C^{\text{x}}({\mathbf{s}}^{\prime}_{t},\text{RNN}^{ \text{x}}({\mathbf{x}},i,\boldsymbol{\phi});\boldsymbol{\phi})\), where \(i\) is the length of the input prefix in \({\mathbf{a}}_{1...t}\).\(a\), \(\text{RNN}^{\text{x}}({\mathbf{x}},i,\boldsymbol{\phi})\) is the hidden state of the \(i\)-th position after reading \({\mathbf{x}}\) (not \({\mathbf{a}}\) nor \(\boldsymbol{\omega}\)) backwards, and \(C^{\text{x}}(\cdot,\cdot)\) is a feed-forward network that takes the concatenated vector of all arguments, and outputs a real scalar. We describe the parametrization of \(C^{\text{x}}\) in Appendix C.

 

* \(X=\Sigma^{*}\), and \(Y=\mathbf{y}\) is a string: in this case \(\mathcal{T}^{\prime}=\mathcal{T}\circ\mathbf{y}\). We let \(C_{\mathbf{\phi}}=C^{\mathrm{y}}(\mathsf{s}_{t}^{\prime},\text{RNNN}^{\mathrm{x}}( \mathbf{x},i,\mathbf{\phi}),\text{RNNV}^{\mathrm{y}}(\mathbf{y},j,\mathbf{\phi});\mathbf{ \phi})\), where \(j\) is the length of the output prefix in \(\mathbf{a}_{1...t}.a\), and \(\text{RNNN}^{\mathrm{y}},C^{\mathrm{y}}\) are similarly defined as in \(\text{RNN}^{\mathrm{x}}\) and \(C^{\mathrm{x}}\).
* \(X\) and \(Y\) are both strings -- \(X=\mathbf{x},Y=\mathbf{y}\): in this case we let \(C_{\mathbf{\phi}}=C^{\mathrm{xy}}(\mathsf{s}_{t}^{\prime},\text{RNNN}^{\mathrm{ x}}(\mathbf{x},i,\mathbf{\phi}),\text{RNNN}^{\mathrm{y}}(\mathbf{y},j,\mathbf{\phi});\mathbf{ \phi})\).

Given a path prefix \(\mathbf{a}_{t-1}\), \(q_{t}(a~{}|~{}\mathbf{a}_{t-1};\mathbf{\phi},\mathcal{T}^{\prime})\) is defined over arcs \(a\) such that \(\mathbf{a}_{t-1}.a\) is a valid path prefix in \(\mathcal{T}^{\prime}\). To optimize \(\mathbf{\phi}\) with regard to \(q_{\mathbf{\phi}}\), we follow Lin2018 and seek to find \(\mathbf{\phi}^{*}=\operatornamewithlimits{argmin}_{\mathbf{\phi}}\text{KL}[\hat{p}|| q_{\mathbf{\phi}}]\), where \(\hat{p}\) is the approximate distribution defined in equation (7), which is equivalent to maximizing the log-likelihood of \(q_{\mathbf{\phi}}(\mathbf{a})\) when \(\mathbf{a}\) is distributed according to the approximation \(\hat{p}\).

### Training

In this paper, we consider joint training. The loss function of our model is defined as the negative log joint probability of string pair \((\mathbf{x},\mathbf{y})\):

\[\mathcal{L}(\mathbf{x},\mathbf{y})=-\log p(\mathbf{x},\mathbf{ y})=-\log\sum_{\mathbf{a}\in\mathbf{x}\circ\mathcal{T}\circ\mathbf{y}}p( \mathbf{a}). \tag{9}\]

Since \(p\) is an exponential family distribution, the gradients of \(\mathcal{L}\) can be written as Bishop2006

\[\nabla\mathcal{L}(\mathbf{x},\mathbf{y})=-\mathbb{E}_{\mathbf{a }\sim p(\cdot|\mathbf{x}\circ\mathcal{T}\circ\mathbf{y})}[\nabla\log p( \mathbf{a})], \tag{10}\]

where \(p(\cdot~{}|~{}\mathbf{x}\circ\mathcal{T}\circ\mathbf{y})\) is a conditioned distribution over paths. Computing equation (10) requires sampling from \(p(\cdot~{}|~{}\mathbf{x}\circ\mathcal{T}\circ\mathbf{y})\), which, as we discuss in SS3.1, is often impractical. We therefore approximate it with

\[\begin{split}\nabla_{\mathbf{\theta}}\mathcal{L}(\mathbf{x}, \mathbf{y})&=-\mathbb{E}_{\mathbf{a}\sim p(\cdot|\mathbf{x} \circ\mathcal{T}\circ\mathbf{y})}[\nabla_{\mathbf{\theta}}\log p(\mathbf{a})]\\ &\approx-\mathbb{E}_{\mathbf{a}\sim\hat{p}(\cdot|\mathbf{x}\circ \mathcal{T}\circ\mathbf{y})}[\nabla_{\mathbf{\theta}}\log p(\mathbf{a})]\\ &=-\sum_{m=1}^{M}w^{(m)}\nabla_{\mathbf{\theta}}G_{\mathbf{\theta}}( \mathbf{a}^{(m)}),\end{split} \tag{11}\]

where \(q\) is a proposal distribution parametrized as in equation (8) (discussed in SS3.1,) \(\mathbf{a}^{(1)}\dots\mathbf{a}^{(M)}\sim q\) are i.i.d. samples of paths in \(\mathbf{x}\circ\mathcal{T}\circ\mathbf{y}\), and \(w^{(m)}\) is the **importance weight** of the \(m\)-th sample satisfying \(w^{(m)}\propto\frac{\exp G_{\mathbf{\theta}}(\mathbf{a}^{(m)})}{q(\mathbf{a}^{(m)} )},\sum_{m=1}^{M}w^{(m)}=1\). Pseudocode for calculating equation (12) is listed in Algorithm 1.

```
0:\(G_{\mathbf{\theta}}:\mathcal{A}\rightarrow\mathbb{R}\) is an NFST scoring function, \(q\) is a distribution over paths, \(M\in\mathbb{N}\) is the sample size
1:functionGet-Gradient(\(G_{\mathbf{\theta}}\), \(M\), \(q\))
2:for\(m\) in \(1\ldots M\)do
3:\(\mathbf{a}^{(m)}\sim q\)
4:\(\tilde{w}^{(m)}\leftarrow\frac{\exp G_{\mathbf{\theta}}(\mathbf{a}^{(m)})}{q( \mathbf{a})}\)
5:endfor
6:\(\hat{Z}\leftarrow\sum_{m=1}^{M}\tilde{w}^{(m)}\)
7:for\(m\) in \(1\ldots M\)do
8:\(w^{(m)}\leftarrow\frac{\tilde{w}^{(m)}}{Z}\)
9:endfor
10:return\(-\sum_{m=1}^{M}w^{(m)}\nabla_{\mathbf{\theta}}G_{\mathbf{\theta}}(\mathbf{a}^{(m)})\)
11:endfunction
```

**Algorithm 1** Compute approximate gradient for updating \(G_{\mathbf{\theta}}\)

### Decoding most probable strings

Besides finding good _paths_ in a conditioned distribution as we discuss in SS3.1, we are also often interested in finding good _output strings_, which is conventionally referred to as the _decoding_ problem, which we define to be finding the best output string \(\mathbf{y}^{*}\triangleq\operatornamewithlimits{argmax}_{\mathbf{y}\in L( Y)}p_{\mathbf{Y}}(\mathbf{y}\mid\mathcal{T}^{\prime})\), where

\[p_{\mathbf{Y}}(\mathbf{y}\mid\mathcal{T}^{\prime})\triangleq\frac{\sum_{ \mathbf{a}\in\mathcal{T}^{\prime}\circ\mathbf{y}}\tilde{p}(\mathbf{a})}{\sum_{ \mathbf{a}^{\prime}\in\mathcal{T}^{\prime}}\tilde{p}(\mathbf{a}^{\prime})}. \tag{13}\]

\(\hat{\mathbf{y}}^{*}\triangleq\operatornamewithlimits{argmax}_{\mathbf{y}}\hat{P}_ {\mathbf{Y}}(\mathbf{y}\mid\mathcal{T}^{\prime})\) is a consistent estimator of \(\mathbf{y}^{*}\), which can directly be used to find the best string. However, making this estimate accurate might be expensive: it requires sampling many paths in the machine \(\mathcal{T}^{\prime}\), which is usually cyclic, and therefore has infinitely many more paths, than \(\mathcal{T}^{\prime}\circ\mathbf{y}_{k}\), which has finitely many paths when \(A\) is acyclic. On the other hand, for the task of finding the best string among a pool candidates, we do not need to compute (or approximate) the denominator in equation (13), since

\[\mathbf{y}^{*}=\operatornamewithlimits{argmax}_{\mathbf{y}\in L(Y)}\sum_{ \mathbf{a}\in\mathcal{T}^{\prime}\circ\mathbf{y}}\tilde{p}(\mathbf{a}). \tag{14}\]

As in the case for paths, the language \(L(Y)\) is usually infinitely large. However given an output candidate \(\mathbf{y}_{k}\in L^{\prime}\subseteq L(Y)\), we can approximate the summation in equation (14) using importance sampling:

\[\sum_{\mathbf{a}\in\mathcal{T}^{\prime}\circ\mathbf{y}_{k}}\tilde{p}( \mathbf{a})=\mathbb{E}_{\mathbf{a}\sim q(\cdot|\mathcal{T}^{\prime}\circ\mathbf{y }_{k})}[\frac{\tilde{p}(\mathbf{a})}{q(\mathbf{a}\mid\mathcal{T}^{\prime}\circ \mathbf{y}_{k})}],\] (15) 

where \(q(\cdot\mid{\cal T}^{\prime}\circ{\bf y}_{k})\) is a proposal distribution over paths in \({\cal T}^{\prime}\circ{\bf y}_{k}\). In this paper we parametrize \(q(\cdot\mid{\cal T}^{\prime}\circ{\bf y}_{k})\) following the definition in equation (8). When \(L^{\prime}\) is finitely large, we reduce the decoding task into a reranking task.

To populate \(L^{\prime}\), one possibility is to marginalize over paths in the approximate distribution \(\hat{p}({\mathbf{a}}\mid{\cal T}^{\prime})\) discussed in SS3.1 to obtain an estimate \(\hat{p}_{\bf Y}({\mathbf{y}}\mid{\cal T}^{\prime})\), and use its support as \(L^{\prime}\). Note that it's possible to populate the candidate pool in other ways, each with its advantages and drawbacks: for example, one can use a top-\(k\) path set from a weighted (Markovian) FST. This approach guarantees exact computation, and the pool quality would no longer depend on the qualities of the smoothing distribution \(q_{\mathbf{\phi}}\). However it is also a considerably much weaker model and may yield uninspiring candidates. In the common case where the conditioned machine \({\cal T}^{\prime}=X\circ{\cal T}\circ Y\) has \(X={\mathbf{x}}\in\Sigma^{*}\) as the input string, and \(Y\) is the universal acceptor that accepts \(\Delta^{*}\), one can obtain a candidate pool from seq2seq models: seq2seq models can capture long distance dependencies between input and output strings, and are typically fast to train and decode from. However they are not applicable in the case where \(L(Y)\neq\Delta^{*}\). Experimental details of decoding are further discussed in SS4.3.

## 4 Experiments

Our experiments mainly aim to: (1) show the effectiveness of NFSTs on transduction tasks; (2) illustrate that how prior knowledge can be introduced into NFSTs and improve the performance; (3) demonstrate the interpretability of our model. Throughout, we experiment on three tasks: (i) grapheme-to-phoneme, (ii) phoneme-to-grapheme, and (iii) actions-to-commands. We compare with competitive string transduction baseline models in these tasks.

### Tasks and datasets

We carry out experiments on three string transduction tasks:

**Graphene-to-phoneme and phoneme-to-graphene (G2P/P2G)** refer to the transduction between words' spelling and phonemic transcription. English has a highly irregular orthography Venezky (2011), which necessitates the use of rich models for this task. We use a portion of the standard CMUDict dataset: the Sphink-compatible version of CMUDict Weide (2005). As for metrics, we choose widely used exact match accuracy and edit distance.

**Action-to-command (A2C)** refers to the transduction between an action sequence and imperative commands. We use NACS Bastings et al. (2018) in our experiment. As for metrics, we use exact match accuracy (EM). Note that the in A2C setting, a given input can yield different outputs, e.g. I_JUMP I_WALK I_WALK corresponds to both "jump and walk twice" and "walk twice after jump". NACS is a finite set of action-command pairs; we consider a predicted command to be correct if it is in the finite set and its corresponding actions is exactly the input. We evaluate on the _length_ setting proposed by Bastings et al. (2018), where we train on shorter sequences and evaluate on longer sequences 



### FST designs

NFSTs require an unweighted FST \({\mathcal{T}}\) which defines a scaffold for the relation it recognizes. In this paper we experiment with two versions of \({\mathcal{T}}\): the first is a simple 'general' design \({\mathcal{T}}_{0}\), which contains only three states \(s_{\{0,1,2\}}\), where the only arc between \(q_{0}\) and \(q_{1}\) consumes the mark <BOS>; and the only arc between \(q_{1}\) and \(q_{2}\) consumes the mark <EOS>. \({\mathcal{T}}_{0}\) has exactly one accepting state, which is \(q_{2}\). To ensure that \({\mathcal{T}}_{0}\) defines relation for all possible string pairs \((\mathbf{x},\mathbf{y})\in\Sigma^{*}\times\Delta^{*}\), we add all arcs of the form \(a=(s_{1},s_{1},\mathbf{\omega},\sigma,\delta),\forall(\sigma,\delta)\in\Sigma \times\Delta\) to \({\mathcal{T}}\).

To recognize transduction rules defined in the Wikipedia English IPA Help page, we define \({\mathcal{T}}_{\text{IPA}}\), which has all states and arcs of \({\mathcal{T}}_{0}\), and additional states and arcs to handle multi-graphene and multi-phoneme transductions defined in the IPA Help:3 for example, the transduction \(\mathsf{th}\to\emptyset\) is encoded as two arcs \((s_{1},s_{3},\mathbf{\omega},\mathfrak{t},\theta)\) and \((s_{3},s_{1},\mathbf{\omega},\mathtt{h},\varepsilon)\). Because of the lack of good prior knowledge that can be added to A2C experiments, we only use general FETs in those experiments for such experiments. Nor do we encode special marks that we are going to introduce below.4

Footnote 3: [https://en.wikipedia.org/wiki/Help](https://en.wikipedia.org/wiki/Help): IPA/English

#### 4.2.1 Design of mark sequences

As with regular WFSTs, the arcs can often be hand engineered to incorporate prior knowledge. Recall that as we describe in SS2.2, each arc is associated with a mark sequence. In this paper, we will always derive the mark sequence on an arc \(a=(s^{\prime},s,\mathbf{\omega}^{\prime},\sigma,\delta)\) of the transducer \({\mathcal{T}}\) as \(\mathbf{\omega}=[\sigma,\mathbf{\omega}^{\prime},\delta,s]\), where \(\mathbf{\omega}^{\prime}\in\Omega^{*}\) can be engineered to reflect FST- and application-specific properties of a path, such as the IPA Help list we mentioned earlier. One way to encode such knowledge into mark sequences is to have special mark symbols in mark sequences for particular transductions. In this paper we experiment with two schemes of marks:

* **IPA Help (IPA).** We define the IPA mark \(\omega_{\text{IPA}}=\{\mathsf{c}~{}|~{}\forall\}\), where the symbol \(\mathsf{c}\) indicates that this arc is part of a transduction rule listed in the consonant section of the Wikipedia English IPA Help page. Similarly, the mark \(\mathsf{V}\) indicates that the transduction rule is listed in the vowel section.
* **Phoneme Classes (Phone).** We define Phone marks \(\omega_{\text{PHONE}}=\Phi(\delta)\), where \(\Phi\) is a lookup function that returns the phoneme class of \(\delta\) defined by the CMUDict dataset.5 Footnote 5: The NACS dataset was actually generated from a regular transducer, which we could in principle use, but doing so would make the transduction fully deterministic and probably not interesting/hard enough.

In this paper we experiment with the following three FST and mark configurations for G2P/P2G experiments:

* **-IPA-Phone** in which case \(\mathbf{\omega}^{\prime}=\varnothing\) for all arcs. \({\mathcal{T}}={\mathcal{T}}_{0}\).
* **+IPA-Phone** in which case \(\mathbf{\omega}^{\prime}=[\omega_{\text{IPA}}]\) when the transduction rule is found in the IPA Help list, otherwise \(\mathbf{\omega}^{\prime}=\varnothing\). \({\mathcal{T}}={\mathcal{T}}_{\text{IPA}}\).
* **+IPA+Phone** in which case \(\mathbf{\omega}^{\prime}=[\omega_{\text{IPA}}\omega_{\text{PHONE}}]\) when the transduction rule is found in the IPA Help list, otherwise \(\mathbf{\omega}^{\prime}=[\omega_{\text{PHONE}}]\). \({\mathcal{T}}={\mathcal{T}}_{\text{IPA}}\).

As we said earlier, we only use \({\mathcal{T}}={\mathcal{T}}_{0}\) with no special marks for A2C experiments. Experimental results on these different configurations are in SS5.3.

### Decoding methods

We experiment with the following methods to decode the most probable strings:

* **Approximate Posterior (AP).** We approximate the posterior distribution over output strings \(\hat{p}_{\bf Y}(\mathbf{y}~{}|~{}{\mathcal{T}}^{\prime})\), and pick \(\hat{\mathbf{y}}^{*}=\operatornamewithlimits{argmax}_{\bf y}\hat{p}_{\bf Y}( \mathbf{y}~{}|~{}{\mathcal{T}}^{\prime})\) as the output.
* **Reranking AP.** As we discuss in SS3.3, improving \(\hat{\mathbf{y}}^{*}\) by taking more path samples in \({\mathcal{T}}^{\prime}\) may be expensive. The reranking method uses the support of \(\hat{p}_{\bf Y}\) as a candidate pool \(L^{\prime}\), and for each \(\mathbf{y}_{k}\in L^{\prime}\) we estimate equation (15) using path samples in \({\mathcal{T}}^{\prime}\circ\mathbf{y}_{k}\).
* **Reranking External.** This decoding method uses \(k\)-best lists from external models. In this paper, we make use of sequence-to-sequence baseline models as the candidate pool \(L^{\prime}\).
* **Reranking AP + External.** This decoding method uses the union of the support of \(\hat{p}_{\bf Y}\) and \(k\)-best lists from the sequence-to-sequence baseline models as the candidate pool \(L^{\prime}\).

In this paper, we take \(128\) path samples per candidate for all **Reranking** methods.





## 5 Results

### Baselines

We compare NFSTs against the following baselines:

BiRNN-WFSTsproposed by Rastogi et al. (2016), were weighted finite-state transducers whose weights encode input string features by the use of recurrent neural networks. As we note in Table 1, they can be seen as a special case of NFSTs, where the Markov property is kept, but where exact inference is still possible.

Seq2seq modelsare the standard toolkit for transduction tasks. We make use of the attention mechanism proposed by Luong et al. (2015), which accomplishes 'soft alignments' that do not enforce a monotonic alignment constraint.

Neuralized IBM Model 1is a character transduction model recently proposed by Wu et al. (2018), which marginalizes over non-monotonic hard alignments between input and output strings. Like (Luong et al., 2015), they did not enforce monotonic alignment constraints; but unlike them, they did not make use of the **input feeding** mechanism,6 where past alignment information is fed back into the RNN decoder. This particular omission allows (Wu et al., 2018) to do exact inference with a dynamic programming algorithm.

Footnote 6: We discuss this further in Appendix B.1.

All baseline systems are tuned on the validation sets. The seq2seq models employ GRUs, with word and RNN embedding size = 500 and a dropout rate of \(0.3\). They are trained with the Adam optimizer (Kingma and Ba, 2014) over \(50\) epochs. The Neuralized IBM Model 1 models are tuned as described in (Wu et al., 2018).

### The effectiveness of NFSTs

#### 5.2.1 Does losing the Markov property help?

Table 2 indicates that BiRNN-WFST models (Rastogi et al., 2016) perform worse than other models. Their Markovian assumption helps enable dynamic programming, but restricts their expressive power, which greatly hampers the BiRNN-WFST's performance on the P2G/G2P task. The NACS task also relies highly on output-output interactions, and BiRNN-WFST performs very poorly there.

#### 5.2.2 Effectiveness of proposed decoding methods

Table 3 shows results from different decoding methods on the G2P/P2G tasks, configuration **+IPA+Phone**. AP performs significantly worse than **Reranking AP**, suggesting that the estimate \({\hat{\mathbf{y}}^{*}}\) suffers from the variance problem. Interestingly, of decoding methods that employ external models, **Reranking External** performs better than **Reranking AP + External**, despite having a smaller candidate pool. We think there is some product-of-experts effect in **Reranking External** since the external model may not be biased in the same way as our model is. But such benefits vanish when candidates from **AP** are also in the pool -- our learned approximation learns the bias in the model -- and hence the worse performance in **Reranking AP + External**. This suggests an interesting regularization trick in practice: populating the candidate pool using external models to hide our model bias. However when we compare our method against non-NFST baseline methods we do not make use of such tricks, to ensure a more fair comparison.

### Prior knowledge: does it help?

In Table 4 we see that combining both +IPA and +Phone improves model generalizability over the general FST (-IPA -Phone). We also note that using only the IPA marks leads to degraded performance

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{4}{c}{GD2P /P2G} & \multicolumn{2}{c}{NACS} \\ \cline{2-7}  & EM Accuracy & Edit Distance & EM Accuracy \\ \cline{2-7}  & Dev & Test & Dev & Test & Test \\ \hline BiRNN-WFST & 16.9 & 15.9 & 1.532 & 1.645 & 5.6 \\ Seq2seq & 30.7 & 28.9 & 1.373 & 1.426 & 9.0 \\ Neuralized IBM Model & 31.6 & 30.2 & 1.366 & 1.398 & — \\ Local NFSTs & 32.7 & 31.8 & 1.319 & 1.332 & 15.64 \\ \end{tabular}
\end{table}
Table 2: Average exact match accuracy (%, higher the better) and edit distance (lower the better) on G2P and P2G as well as exact match accuracy on NACS. Comparison between our models with baselines. For NFST models, we make use of the **Reranking AP** decoding method described in §4.2.

\begin{table}
\begin{tabular}{l l l l l} \hline \hline  & \multicolumn{2}{c}{EM Accuracy} & \multicolumn{2}{c}{Edit Distance} \\ \cline{2-5}  & Dev & Test & Dev & Test \\ \hline AP & 28.2 & 28.2 & 1.513 & 1.467 \\ Reranking AP & 32.7 & 31.8 & 1.319 & 1.332 \\ Reranking External & 33.3 & 32.7 & 1.297 & 1.298 \\ Reranking AP + External & 32.9 & 32.0 & 1.309 & 1.303 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Average exact match accuracy (%, higher the better) and edit distance (lower the better) on G2P and P2G. The effectiveness of different decoding methods.



compared to the general FST baseline. This is a surprising result -- one explanation is the IPA marks are not defined on all paths that transduce the intended input-output pairs: NFSTs are capable of recognizing phoneme-grapheme alignments in different paths,7 but only one such path is marked by +IPA. But we leave a more thorough analysis to future work.

Footnote 6: This is discussed further in Appendix B.2.

## 6 Related Work

Recently, there has been work relating finite-state methods and neural architectures. For example, Schwartz et al. (2018) and Peng et al. (2018) have shown the equivalence between some neural models and WFSAs. The most important differences of our work is that in addition to classifying strings, NFSTs can also transduce strings. Moreover, NFSTs also allow free topology of FST design, and breaks the Markovian assumption. In addition to models we compare against in SS4, we note that (Aharoni and Goldberg, 2017; Deng et al., 2018) are also similar to our work; in that they also marginalize over latent alignments, although they do not enforce the monotonicity constraint. Work that discusses globally normalized sequence models are relevant to our work. In this paper, we discuss a training strategy that bounds the partition function; other ways to train a globally normalized model (not necessarily probabilistic) include (Wiseman and Rush, 2016; Andor et al., 2016). On the other hand, our locally normalized FSTs bear resemblance to (Dyer et al., 2016), which was also locally normalized, and also employed importance sampling for training.

## 7 Conclusions and Future Work

Neural finite state transducers (NFSTs) are able to model string pairs, considering their monotonic alignment but also enjoying RNNs' power to handle non-finite-state phenomena. They compete favorably with state-of-the-art neural models on transduction tasks. At the same time, it is easy to inject domain knowledge into NFSTs for inductive bias, and they offer interpretable paths.

In this paper, we have used rather simple architectures for our RNNs; one could experiment with multiple layers and attention. One could also experiment with associating marks differently with arcs--the marks are able to convey useful domain information to the RNNs. For example, in a P2G or G2P task, all arcs that cross a syllable boundary might update the RNN state using a syllable mark. We envision using regular expressions to build the NFSTs, and embedding marks in the regular expressions as a way of sending useful features to the RNNs to help them evaluate paths.

In this paper, we have studied NFSTs as standalone systems. But as probabilistic models, they can be readily embedded in a bigger picture: it should be directly feasible to incorporate a globally/locally normalized NFST in a larger probabilistic model (Finkel and Manning, 2009; Chiang et al., 2010).

The path weights of NFSTs could be interpreted simply as scores, rather than log-probabilities. One would then decode by seeking the 1-best path with input \({\bf x}\), e.g., via beam search or Monte Carlo Tree Search. In this setting, one might attempt to train the NFST using methods similar to the max-violation structured perceptron or the structured SVM.

## Acknowledgments

This work has been generously supported by a Google Faculty Research Award and by Grant No. 1718846 from the National Science Foundation, both to the last author. Hao Zhu is supported by Tsinghua University Initiative Scientific Research Program. We thank Shijie Wu for providing us IBM Neuralized Model 1 experiment results.

## References

* Aharoni and Goldberg (2017) Roee Aharoni and Yoav Goldberg. 2017. Morphological inflection generation with hard monotonic attention. In _ACL_.
* Andor et al. (2016) Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. In _Association for Computational Linguistics_.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proceedings of ICLR_.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{3}{c}{EM Accuracy} & Edit Distance \\ \cline{2-5}  & Dev & Test & Dev & Test \\ \hline -IPA -Phone & 31.8 & 29.3 & 1.38 & 1.373 \\ +IPA -Phone & 31.3 & 29.2 & 1.367 & 1.431 \\ +IPA +Phone & 32.7 & 31.8 & 1.319 & 1.332 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Average exact match accuracy (%, higher the better) and edit distance (lower the better) on G2P and P2G. The effectiveness of different FST designs.

 

J. Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela (2018) Jump to better conclusions: scan both left and right. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 47-55. Cited by: SS3.
* J. Berstel, Jr. and C. Reutenauer (1988)Rational series and their languages. Springer-Verlag, Berlin, Heidelberg. External Links: ISBN 978-3-642-46556-5 Cited by: SS1.
* D. Chiang, Jonathan Graehl, Kevin Knight, Adam Pauls, and Sujith Ravi (2010) Bayesian inference for finite-state transducers. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 447-455. Cited by: SS1.
* K. Cho, B. van Merrienboer, C. Gulcehre, Dzmitry Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio (2014)Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, ACL, pp. 1724-1734. Cited by: SS1.
* R. Coterell, N. Peng, and J. Eisner (2014)Stochastic contextual edit distance and probabilistic FSTs. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Baltimore, pp. 625-630. Cited by: SS1.
* Y. Deng, Y. Kim, J. Chiu, D. Guo, and A. Rush (2018)Latent alignment and variational attention. In Advances in Neural Information Processing Systems 31, W. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. , pp. 9735-9747. External Links: Link, Document Cited by: SS1.
* T. Chung, H. Pham, and C. D. Manning (2015)Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421. Cited by: SS1.
* S. Choy, G. Kuncoro, M. Ballesteros, and N. A. Smith (2016)Recurrent neural network grammars. In HLT-NAACL, Cited by: SS1.
* J. R. Finkel and C. D. Manning (2009)Hierarchical bayesian domain adaptation. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 602-610. Cited by: SS1.
* D. P. Kingma and J. Ba (2014)Adam: a method for stochastic optimization. CoRRabs/1412.6980. Cited by: SS1.
* C. Lin and J. Eisner (2018)Neural particle smoothing for sampling from conditional sequence models. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), New Orleans, pp. 929-941. External Links: Link, Document Cited by: SS1.
* T. Luong, H. Pham, and C. D. Manning (2015)Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421. Cited by: SS1.
* M. Mohri, Fernando Pereira, and Michael Riley (2008)Speech recognition with weighted finite-state transducers. In Springer Handbook of Speech Processing, pp. 559-584. Cited by: SS1.
* H. Peng, R. Schwartz, S. Thomson, and N. A. Smith (2018)Rational recurrences. In EMNLP, pp. . Cited by: SS1.
* P. Rastogi, R. Cotterell, and J. Eisner (2016)Weighting finite-state transductions with neural context. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), San Diego, pp. 623-633. External Links: Link, Document Cited by: SS1.
* E. Roche and Y. Schabes (1997)Finite-state language processing.. MIT Press. Cited by: SS1.
* Volume 2, NIPS'14, Cambridge, MA, USA, pp. 3104-3112. External Links: Link, Document Cited by: SS1.
* O. Tilk and T. Alumae (2016) Bidirectional recurrent neural network with attention mechanism for punctuation restoration. In INTERSPEECH, Cited by: SS1.
* A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017)Attention is all you need. Cited by: SS1.
* R. L. Venezuela (2011)The structure of english orthography. Vol. 82, Walter de Gruyter. Cited by: SS1.
* R. Weide (2005)The carnegie mellon pronouncing dictionary [cmudict. 0.6]. Cited by: SS1.
* S. Wiseman and A. M. Rush (2016)Sequence-to-sequence learning as beam-search optimization. In EMNLP, Cited by: SS1.
* S. Wu, P. Shapiro, and R. Cotterell (2018)Hard non-monotonic attention for character-level transduction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4425-4438.

 

Finite-state transducers

### Rational Relations

A **relation** is a set of pairs--in this paper, a subset of \(\Sigma^{*}\times\Delta^{*}\), so it relates strings over an "input" alphabet \(\Sigma\) to strings over an "output" alphabet \(\Delta\).

A **weighted relation** is a function \(R\) that maps any string pair \((\mathbf{x},\mathbf{y})\) to a **weight in \(\mathbb{R}_{\geq 0}\)**.

We say that the relation \(R\) is **rational** if \(R\) can be defined by some weighted finite-state transducer (FST) \(\mathcal{T}\). As formalized in Appendix A.3, this means that \(R(\mathbf{x},\mathbf{y})\) is the total weight of all accepting paths in \(\mathcal{T}\) that are labeled with \((\mathbf{x},\mathbf{y})\) (which is 0 if there are no such accepting paths). The weight of each accepting path in \(\mathcal{T}\) is given by the product of its arc weights, which fall in \(\mathbb{R}_{>0}\).

The set of pairs \(\operatorname{support}(R)\triangleq\{(\mathbf{x},\mathbf{y}):R(\mathbf{x}, \mathbf{y})>0\}\) is then said to be a **regular relation** because it is recognized by the unweighted FST obtained by dropping the weights from \(\mathcal{T}\). In this paper, we are interested in defining _non-rational_ weighting functions \(R\) with this same regular support set.

### Finite-state transducers

We briefly review finite-state transducers (FSTs). Formally, an FST is a tuple \(\mathcal{T}_{0}=(\Sigma,\Delta,Q,A_{0},I,F)\) where

* \(\Sigma\) is a finite input alphabet
* \(\Delta\) is a finite output alphabet
* \(Q\) is a finite set of states
* \(A_{0}\subseteq Q\times Q\times(\Sigma\cup\{\epsilon\})\times(\Delta\cup\{ \epsilon\})\) is the set of weighted arcs
* \(I\subseteq Q\) is the set of initial states (conventionally \(|I|=1\))
* \(F\subseteq Q\) is the set of final states

Let \(\mathbf{a}=a_{1}\ldots a_{T}\) (for \(T\geq 0\)) be an accepting path in \(\mathcal{T}_{0}\), that is, each \(a_{i}=(q_{i-1},q_{i},\sigma_{i},\delta_{i})\in A_{0}\) and \(q_{0}\in I,q_{T}\in F\). We say that the input and output strings of a are \(\sigma_{1}\cdots\sigma_{T}\) and \(\delta_{1}\cdots\delta_{T}\).

### Real-valued weighted FSTs

Weighted FSTs (WFSTs) are defined very similarly to FSTs. A WFST is formally defined as a \(6\)-tuple, just like an (unweighted) FST: \(\mathcal{T}=(\Sigma,\Delta,Q,A,I,F)\), with arcs carrying weights: \(A\subseteq Q\times Q\times(\Sigma\cup\{\epsilon\})\times(\Delta\cup\{ \epsilon\})\times\mathbb{R}\). Compared to FST arcs in Appendix A.2, a WFST arc each \(a_{i}=(q_{i-1},q_{i},\sigma_{i},\delta_{i},\kappa_{i})\in A\) has weight \(\kappa_{i}\). We also define the weight of \(\mathbf{a}\) to be \(w(\mathbf{a})\triangleq\bigotimes_{i=1}^{T}\kappa_{i}\in\mathbb{R}\).

The weight of the entire WFST \(\mathcal{T}\) is defined as the total weight (under \(\oplus\)) of all accepting paths:

\[\mathcal{T}[\,]\triangleq\bigoplus_{\mathbf{a}}w(\mathbf{a})\in\mathbb{R} \tag{16}\]

More interestingly, the weight \(\mathcal{T}[\mathbf{x},\mathbf{y}]\) of a string pair \(\mathbf{x}\in\Sigma^{*},\mathbf{y}\in\Delta^{*}\) is given by similarly summing \(w(\mathbf{a})\) over just the accepting paths \(\mathbf{a}\) whose input string is \(\mathbf{x}\) and output string is \(\mathbf{y}\).

## Appendix B More analysis on the effectiveness of NFSTs

### Does feeding alignments into the decoder help?

In particular, we attribute our models' outperforming Neuralized IBM Model 1 to the fact that a complete history of past alignments is remembered in the RNN state. Wu et al. (2018) noted that in character transduction tasks, past alignment information seemed to barely affect decoding decisions made afterwards. However, we empirically find that there is performance gain by explicitly modeling past alignments. This also shows up in our preliminary experiments with non-input-feeding seq2seq models, which resulted in about \(1\%\) of lowered accuracy and about 0.1 longer edit distance.

### Interpretability of learned paths

The model is not required to learn transduction rules that conform to our linguistic knowledge. However, we expect that a well-performing one would tend to pick up rules that resemble what we know. To verify this, we obtain samples (listed in Table 4) from \(\hat{p}(\mathbf{a}\mid\mathbf{x},\mathbf{y})\) using the importance sampling algorithm described in SS3.3. We find that our NFST model has learned to align phonemes and graphemes, generating them alternately. It has no problem picking up obvious pairs in the English orthography (e.g. (J, c   h), and (u, n   g)). We also find evidence that the model has picked up how context affects alignment: for example, the model has learned that the bigram 'gh' is pronounced differently in different contexts: in 'onslaugh**t**,' it is aligned with o in the sequence 'augh;' in 'Willingham,' it spans over two phonemes u hi; and in '**ghczzi**,' it is aligned with the phoneme g. We also find that our NFST has no problem learning phoneme-graphene alignments that span over two 

arcs, which is beyond the capability of of ordinary WFSTs.

## Appendix C Implementation Details

### Model parametrization details

As mentioned before, the type of RNN that we use is GRU. The GRU parameterizing \(G_{\mathbf{\theta}}\) has \(500\) hidden states. The embedding sizes of tokens, including the input symbol, output symbol and states, and marks are all 500. During inference we make use of proposal distributions \(q_{\phi}(\mathbf{a}~{}|~{}\mathcal{T^{\prime}})\), where \(\mathcal{T^{\prime}}\in\{\mathsf{x}\circ\mathcal{T},\mathcal{T}\circ\mathbf{y },\mathbf{x}\circ\mathcal{T}\circ\mathbf{y}\}\). All RNNs used to parametrize \(q_{\mathbf{\phi}}\) are also GRUs, with \(125\) hidden states. \(q_{\phi}\) makes use of input/output embeddings independent from \(G_{\mathbf{\theta}}\), which also have size \(125\) in this paper. The feed-forward networks \(C^{\text{x},\text{y},\text{xy}}\) are parametrized by \(3\)-layer networks, with ReLU as the activation function of the first two layers. The output dimension sizes of the first and second layers are \(\lfloor D/2\rfloor\) and \(\lfloor D/4\rfloor\), where \(D\) is the input vector dimension size.

### Training procedure details

We use stochastic gradient descent (SGD) to train \(G_{\mathbf{\theta}}\). For each example, we compute the gradient using normalized importance sampling over an ensemble of \(512\) particles (paths), the maximum that we could compute in parallel. By using a large ensemble, we reduce both the bias (from normalized importance sampling) and the variance of the gradient estimate; we found that smaller ensembles did not work as well. Thus, we used only one example per minibatch.

We train the 'clamped' proposal distribution differently from the 'free' ones \(q_{\phi}(\mathbf{a}~{}|~{}\mathbf{x}\circ\mathcal{T}\circ\mathbf{y})\) and \(q_{\phi}(\mathbf{a}~{}|~{}\mathcal{T}\circ\mathbf{y})\). The clamped distribution is trained alternately with \(G_{\mathbf{\theta}}\), as listed in Algorithm 2. We evaluate on the development dataset at the end of each epoch using the **Reranking External** method described in SS4.3. When the EM accuracy stops improving, we fix the parameters of \(G_{\mathbf{\theta}}\) and start training \(q_{\phi}(\mathbf{x}\circ\mathcal{T})\) and \(q_{\phi}(\mathcal{T}\circ\mathbf{y})\) on the inclusive KL divergence objective function, using methods described in Lin and Eisner (2018). We then initialize the free distributions' RNNs using those of the clamped distributions. We train the free proposal distributions for \(30\) epochs, and evaluate on the development dataset at the end of each epoch. Results from the best epochs are reported in this paper.