SlowGuess/ACL_2020_OCR huggingface dataset:
Xiang Lisa Li
Department of Computer Science Johns Hopkins University
xli150@jhu.edu
Department of Computer Science Cornell Tech
arush@cornell.edu
Text generation often requires high-precision output that obeys task-specific rules. This fine-grained control is difficult to enforce with off-the-shelf deep learning models. In this work, we consider augmenting neural generation models with discrete control states learned through a structured latent-variable approach. Under this formulation, task-specific knowledge can be encoded through a range of rich, posterior constraints that are effectively trained into the model. This approach allows users to ground internal model decisions based on prior knowledge, without sacrificing the representational power of neural generative models. Experiments consider applications of this approach for text generation. We find that this method improves over standard benchmarks, while also providing fine-grained control.
A core challenge in using deep learning for NLP is developing methods that allow for controlled output while maintaining the broad coverage of data-driven methods. While this issue is less problematic in classification tasks, it has hampered the deployment of systems for conditional natural language generation (NLG), where users often need to control output through task-specific knowledge or plans. While there have been significant improvements in generation quality from automatic systems (Mei et al., 2016; Dusek and Jurcicek, 2016; Lebret et al., 2016b), these methods are still far from being able to produce controlled output (Wiseman et al., 2017). Recent state-of-the-art system have even begun to utilize manual control through rule-based planning modules (Moryossef et al., 2019; Puduppully et al., 2019).
Consider the case of encoder-decoder models for generation, built with RNNs or transformers.
These models generate fluent output and provide flexible representations of their conditioning. Unfortunately, auto-regressive decoders are also globally dependent, which makes it challenging to incorporate domain constraints.
Research into controllable deep models aims to circumvent the all-or-nothing dependency tradeoff of encoder-decoder systems and expose explicit higher-level decisions. One line of research has looked at global control states that represent sentence-level properties for the full decoder. For example, Hu et al. (2017) uses generative adversarial networks where the attributes of the text (e.g., sentiment, tense) are exposed. Another line of research exposes fine-level properties, such as phrase type, but requires factoring the decoder to expose local decisions, e.g. Wiseman et al. (2018).
This work proposes a method for augmenting any neural decoder architecture to incorporate fine-grained control states. The approach first modifies training to incorporate structured latent control variables. Then, training constraints are added to anchor the state values to problem-specific knowledge. At test time, the control states can be ignored or utilized as grounding for test-time constraints. Technically, the approach builds on recent advances in structured amortized variational inference to enforce additional constraints on the learned distribution. These constraints are enforced through efficient structured posterior calculations and do not hamper modeling power.
We demonstrate that the method can improve accuracy and control, while utilizing a range of different posterior constraints. In particular on two large-scale data-to-text generation datasets, E2E (Novikova et al., 2017) and WikiBio (Lebret et al., 2016a), our method increases the performance of benchmark systems while also producing outputs that respect the grounded control states. Our code is available at https://github.com/XiangLi1999/
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2731-2743 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
PosteriorControl-NLG.
2 Control States for Blackbox Generation
Consider a conditional generation setting where the input consists of an arbitrary context  $x$  and the output  $y_{1:T}$  is a sequence of target tokens. We are interested in modeling latent fine-grained, discrete control states  $z = z_{1:T}$  each with a label in  $\mathcal{C}$ . We assume that these states are weakly-supervised at training through problem-specific constraints. The goal is to induce a model of  $p(y \mid x) = \sum_{z} p(y, z \mid x)$ . Concretely, our experiments will focus on a data-to-text generation problem where  $x$  corresponds to a table of data, and  $y_{1:T}$  is a textual description. We hope to induce control states  $z$  that indicate which table fields are being described, and our weak supervision corresponds to indicators of known alignments.
We assume the generative model is a blackbox auto-regressive decoder that produces both  $y$  and  $z$ . Define this general model as:
$$
\begin{array}{l} p _ {\theta} (y, z \mid x) = \prod_ {t = 1} ^ {T} p _ {\theta} (y _ {t} \mid x, y _ {<   t}, z _ {\leq t}) \cdot \\ p _ {\theta} \left(z _ {t} \mid x, y _ {<   t}, z _ {<   t}\right) \\ \end{array}
$$
For a neural decoder, where  $h_t(y_{1:t-1}, z_{1:t-1})$  is the hidden state at time-step  $t$ , we might generate the latent class  $z_t \in \mathcal{C}$  and next token  $y_t$  as,
$$
\begin{array}{l} p _ {\theta} \left(z _ {t} \mid z _ {<   t}, y _ {<   t}\right) = \operatorname {s o f t m a x} \left(W _ {0} h _ {t} + b _ {0}\right) \\ p _ {\theta} (y _ {t} \mid z _ {\leq t}, y _ {<   t}) = \operatorname {s o f t m a x} \left(W _ {1} \left[ h _ {t}, g _ {\theta} (z _ {t}) \right] + b _ {1}\right) \\ \end{array}
$$
Here  $g_{\theta}$  is a parameterized embedding function and  $W, b$  are model parameters from  $\theta$ . The log-likelihood of the model is given by  $\mathcal{L}(\theta) = \log p_{\theta}(y \mid x)$ .
The key latent term of interest is the posterior distribution  $p_{\theta}(z \mid x, y)$ , i.e. the probability of over state sequences for a known output. The decoder parameterization makes this distribution intractable to compute in general. We instead use variational inference to define a parameterized variational posterior distribution,  $q_{\phi}(z \mid x, y)$ , from a preselected family of possible distributions  $\mathcal{Q}$ . To fit the model parameters  $\theta$ , we utilize the evidence lower bound (for any variational parameters  $\phi$ ),
$$
\begin{array}{l} \mathcal {L} (\theta) \geq \operatorname {E L B O} (\theta , \phi) \\ = \mathbb {E} _ {z \sim q _ {\phi} (z | x, y)} [ \log p _ {\theta} (y, z \mid x) ] + \mathrm {H} [ q _ {\phi} (z \mid x, y) ] \\ \end{array}
$$
Several recent works have shown methods for effectively fitting neural models with structured variational inference (Johnson et al., 2016; Krishnan et al., 2017; Kim et al., 2019). We therefore use these techniques as a backbone for enforcing problem-specific control. See §4 for a full description of the variational family used.
3 Posterior Regularization of Control States
Posterior regularization (PR) is an approach for enforcing soft constraints on the posterior distribution of generative models (Ganchev et al., 2010). Our goal is to utilize these soft constraints to enforce problem specific weak supervision. Traditionally PR uses linear constraints which in the special case of expectation maximization for exponential families leads to convenient closed-form training updates. As this method does not apply to neural generative models, we resort to gradient-based methods. In this section, we develop a form of posterior regularization that accommodates the neural variational setting.
Starting with the log-likelihood objective,  $\mathcal{L}(\theta)$ , PR aims to add distributional constraints on the posterior. These soft constraints are expressed as a distributional penalty,  $R_{p}(x,y) \geq 0$ . For example, if we have partial information that a specific control state takes on label  $c$  we can add a constraint  $R_{p}(x,y) = 1 - p(z_{t} = c \mid x,y)$ . We might also consider other distributional properties, for instance penalizing the entropy of a specific posterior marginal,  $R_{p}(x,y) = \mathrm{H}_{z^{\prime}}(z_{t} = z^{\prime} \mid x,y)$ . See §5 for more constraint examples.
PR uses these soft constraints to regularize the model. Ideally we would penalize the posterior directly, but as noted above, computing this term in a blackbox model is intractable. We therefore follow Ganchev et al. (2010) and use a relaxed version with a surrogate posterior  $q_{\phi}(z \mid x, y)$ ,
$$
\begin{array}{l} \mathcal {L} _ {P R} (\theta) = \mathcal {L} (\theta) - \tag {1} \\ \min  _ {\phi} \left[ \mathrm {K L} \left[ q _ {\phi} \mid \mid p _ {\theta} (z \mid x, y) \right] + \lambda R _ {q _ {\phi}} (x, y) \right] \\ \end{array}
$$
We can write this in terms of a variational lower-bound on the relaxed PR objective.
$$
\begin{array}{l} \mathcal {L} _ {P R} (\theta) \geq \operatorname {P R L B O} (\theta , \phi) = \mathcal {L} (\theta) - \tag {2} \\ \left[ \mathrm {K L} [ q _ {\phi} | | p _ {\theta} (z | x, y) ] + \lambda R _ {q _ {\phi}} (x, y) \right] \\ \end{array}
$$
This allows us to relate the  $q$  in the PRLBO to the variational posterior in the ELBO simply by
<sup>1</sup>Since our family is over a combinatorial set of  $z_{1:T}$ , this corresponds to a structured variational inference setting.
Figure 1: Model training. Assumes we are given conditioning  $x$  (not shown) and output sentence  $y$ . (Middle) An inference network  $\phi$  is used to parameterize a structured segmental conditional random field  $q_{\phi}(z \mid x, y)$  over control states  $z$ . (Right) Sample from  $q_{\phi}$  (colored circles) is used to provide control state labels for a blackbox generation model  $p_{\theta}(y, z \mid x)$ . (Left) To ground the control states to represent problem-specific meaning, posterior regularization is used to enforce distributional constraints through penalties  $R_{q}(x, y)$ . The whole system is optimized end-to-end to learn latent properties of the final output tokens.
expanding the KL and rearranging terms,
$$
\operatorname {P R L B O} (\theta , \phi) = \operatorname {E L B O} (\theta , \phi) - \lambda R _ {q _ {\phi}} (x, y)
$$
To train, we jointly maximize over both terms in the PRLBO: the model parameters  $\theta$  and the variational parameters  $\phi$  (which tightens the bounds). Following standard practice, we use an amortized inference network, i.e. a variational autoencoder (Kingma and Welling, 2014; Mnih and Gregor, 2014; Rezende et al., 2014), to define  $\phi$ .
4 Structured Variational Family for Segmental Generation
We now discuss how to efficiently compute the PRLBO under a structured variational family.
$$
\mathrm {P R L B O} = \underbrace {\mathbb {E} _ {z \sim q _ {\phi}} [ \log p _ {\theta} ]} _ {(1)} + \underbrace {\mathrm {H} [ q _ {\phi} ]} _ {(2)} - \underbrace {\lambda R _ {q _ {\phi}} (x , y)} _ {(3)}
$$
We need a  $q_{\phi}(z \mid x, y)$  for which we can efficiently (1) take samples, (2) compute entropy, and (3) compute the distributional penalties. This motivates the use of a factored conditional random field (CRF), defined by a potential function  $\phi(x, y, z)$ . At training time,  $x, y$  are observed and  $z$  is the latent variable that denotes the control states. We then specify a variational posterior distribution:  $q_{\phi}(z \mid x, y) = \frac{\phi(x, y, z)}{\sum_{z'} \phi(x, y, z')}$ .
In this work, we focus on the semi-Markov CRF (Gales and Young, 1993; Sarawagi and Cohen, 2005), a common CRF family used in generation (Wiseman et al., 2018). It divides tokens into segmental spans, which are useful for generating entity mentions and commonly used phrases. This model divides the potential function into three parts: the emission potential for a span of tokens given
Algorithm 1: Generic Semi-Markov Algorithm.
Given  $\phi$  and generic semiring  $(\oplus ,\otimes ,\mathbf{0},\mathbf{1})$
Set  $\beta_T(c) = \mathbb{1} \forall c \in \mathcal{C}$
for  $i = T - 1,\dots ,0$  do
for  $c\in \mathcal{C}$  do
$$
\beta_{i}^{\prime}(c) = \bigoplus_{d = 1}^{\min (L,T - i)}\begin{array}{c}\beta_{i + d}(c)\otimes \phi_{(l)}(d)\otimes \\ \phi_{(e)}(x,y_{i,i + d},c) \end{array}
$$
$$
\beta_ {i} (c) = \bigoplus_ {c ^ {\prime} \in \mathcal {C}} \beta_ {i} ^ {\prime} (c ^ {\prime}) \otimes \phi_ {(t)} (c, c ^ {\prime})
$$
return  $Z = \bigoplus_{c\in \mathcal{C}}\beta_0'(c)\otimes \phi_{(t)}(0,c)$
a state, denoted as  $\phi_{(e)}$ ; the transition potential between states,  $\phi_{(t)}$ ; and the length potential of span length given a state,  $\phi_{(l)}$ . Suppose our control states define a span from  $i$  (inclusive) to  $j$  (exclusive) labeled by  $c$ , we denote it as  $z_{i:j} = c$ . The potential function of a labeled sequence is defined:
$$
\begin{array}{l} \phi (x, y, z) = \prod_ {i <   j <   k} \phi_ {(t)} \left(z _ {i: j}, z _ {j: k}\right) \cdot \phi_ {(l)} (j - i) \cdot \\ \phi_ {(e)} (x, y _ {i: j}, z _ {i: j}) \tag {3} \\ \end{array}
$$
For computational efficiency, we restrict all segment length to be  $\leq L$ .<sup>2</sup>
With this model, we can use the forward-backward algorithm for all required inferences: exact sampling, computing partition function, entropy, and posterior marginals  $q_{\phi}(z_{i:j} = c \mid x,y)$ , useful for term (3). In Algorithm 1, we give a
2 The time complexity to compute the posterior moments of the full semi-Markov CRF is  $O(|\mathcal{C}|^2 nL)$
One-to-One
Name	Penalty	One-to-Many
Name	Penalty
Inclusion	For (i, j, f) ∈ A(x, y),
Rq = 1 - q(zi:j = σ(f) | x, y)	Sparsity	For f ∈ F,
Rq = H[σ(c | f)]
Exclusion	For f ∈ x and (i, j, f)∉ A(x, y),
Rq = q(zi:j = σ(f) | x, y)	Fit	For (i, j, f) ∈ A(x, y)
Rq = H[σ(c | f), q(zi:j | x, y)]
Coverage	For f ∈ F,
Rq = |∑i<j q(zi:j = σ(f) | x, y) - 1|f ∈ x|	Diversity	Let pagg(ˆz) ∝ ∑t=1T q(zt = ˆz | x, y)
Rq = H[Unif(ˆz)] - H[pagg(ˆz)]
generic semi-Markov algorithm (Sarawagi and Cohen, 2005). We store two tables  $\beta$  and  $\beta'$ , both of size  $T \times |\mathcal{C}|$ .  $\beta_{t}(c)$  denotes the event that there is a transition at time  $t$  from state  $c$ .  $\beta_{t}'(c)$  denotes the event that there is a emission starting from time  $t$  at state  $c$ . Then we have the recursion for  $\beta_{t}'(c)$  by "summing" over different span length, and we have the recursion for  $\beta_{t}(c)$  that sums over all different state transitions.
The algorithm is generic in the sense that different  $(\otimes, \oplus)$  operators allow us to compute different needed terms. For example, computing the partition function  $Z = \sum_{z'} \phi(x, y, z')$  requires the  $(+, \times)$  semiring (Goodman, 1999; Li and Eisner, 2009), other distributional terms can be computed by using the same algorithm with alternative semirings and backpropagation<sup>3</sup>.
5 Posterior Constraints from Data Alignment
To make the PR model concrete, we consider the problem of incorporating weak supervision from heuristic alignment in a data-to-text generation task. Assume that we are tasked with describing a table  $x$  consisting of global field names  $\mathcal{F}$  each with a text value  $v$ , e.g.  $x_{f} = v$ . Not all global fields may be used in a given  $x$ , we use  $f \in x$  to indicate an
Table 1: Posterior penalties utilized in the One-to-One and One-to-Many setting. These constraints softly enforce an alignment between control states and text spans by penalizing posterior violations. The objective sums over the three  $R_{q}$  in both cases.
x	name[Clowns] eatType[coffee shop], rating[1 out of 5], near[Clare Hall]
f ∈ x	name, eatType, rating, near
y	Clowns1 is2 a3 coffee4 shop5 near6 Clare7 Hall8 with9 a10 111 out12 of13 514 rating15
A(x,y)	(1, 2, name), (4, 6, eatType), (7, 9, near), (11, 15, rating)
Table 2: Example of data alignment notation. Here  $x$  is a table of data, and  $f$  are its fields. For a given output  $y$  we enforce a soft alignment  $A$ .
active field.
We would like control states to indicate when each field is used in generation. Our alignment heuristic is that often these fields will be expressed using the identical text as in the table. While this heuristic obviously does not account for all cases, it is very common in natural language generation tasks as evidence by the wide use of copy attention based approaches (Gu et al., 2016; Gulcehre et al., 2016). To utilize these alignments, we use the notation  $(i,j,f) \in A(x,y)$  to indicate that a span  $i:j$  in the training text  $y$  overlaps directly with a field  $f \in x$ . Table 2 gives an example of the notation.
One-to-One Constraints We first consider one-to-one constraints where we assume that we have a static, mapping from fields to states  $\sigma : \mathcal{F} \mapsto \mathcal{C}$ . Given this mapping, we need to add penalties to encourage the semi-Markov model to overlap with the given weak supervision.
To enforce soft alignments, we define three posterior constraint types and their computation as shown in Table 1 (Left). The three constraints are i) Inclusion: if a span in  $y$  aligns with a field value
3We need four terms: (a) log-partition term  $\log \sum_{z'}\phi (x,y,z')$  requires the log semiring (logsumexp, +). The posterior margins  $q(z\mid x,y)$  requires backpropagating from the log-partition term; (b) max score  $\max_z\phi (x,y,z)$ : (max, +) max semiring and argmax  $\arg \max_z\phi (x,y,z)$  by (subgradient) backpropagation, (c) entropy through an expectation semiring  $\langle p_1,r_1\rangle \otimes \langle p_2,r_2\rangle = \langle p_1p_2,p_1r_2 + p_2r_1\rangle$ , and  $\langle p_1,r_1\rangle \oplus \langle p_2,r_2\rangle = \langle p_1 + p_2,r_1 + r_2\rangle$ , with  $\mathbb{1} = \langle 1,0\rangle$ . To initialize, all the emission, transition and length scores take the form  $\langle \phi , - \log \phi \rangle$ . The algorithm returns  $\langle Z,R\rangle$ , and the true entropy is  $\frac{R}{Z} +\log Z$ . (d) exact sampling through one backward pass and one forward filtering backward sampling, where forward uses the log-partition semiring and backpropagation is by categorical sampling.
$f$ , then label that span  $\sigma(f)$  the state allocated to that field; ii) Exclusion: A span should only have a state  $\sigma(f)$ , if it aligns with the field value of type  $f$ ; iii) Coverage. The usage count of state  $\sigma(f)$  should be 1 if  $f$  in  $x$ .
One-to-Many Constraints We also consider the case when it is infeasible to specify a hard mapping  $\sigma$  between the fields and the states. For example,  $\mathcal{F}$  could be unbounded or large, whereas we hope to keep the cardinality of states small for computational efficiency.
We propose a method of inducing a dynamic soft mapping  $\sigma(c \mid f)$  as we train the model, and impose constraints on the mapping from table field to the state names. First, we would like the distribution of state given table field to be consistent, so one table field is mapped to roughly 1 state. Second, we want to make use of the state space as much as possible by requiring a diverse usage of states.
In order to enforce these properties we introduce the dynamic mapping as a second amortized variational distribution  $\sigma (c\mid f;M) = \mathrm{softmax}(Mf)$  which gives the probability that a table field  $f$  takes on state  $c$ . As shown in Table 1 (Right), we define three constraints that regularize the local  $q$  with respect to the global  $\sigma$ : i) Sparsity: Each vocabulary entry in  $\sigma$  should have low entropy; ii) Fit: The global  $\sigma$  should represent the class name distribution posterior of each table field by minimizing the cross entropy between types  $\sigma (c\mid f)$  and tokens  $q(z_{i:j}\mid x,y)$  for all  $(i,j,f)\in A(x,y)$ ; iii) Diversity: the aggregate class label distribution over all the token in a sentence should have high entropy.
In addition to previously mentioned work, other researchers have noted the lack of control of deep neural networks and proposed methods at sentence-level, word-level, and phrase-level. For example Peng et al. (2018) and Luo et al. (2019) control the sentiment in longer-form story generation. Others aim for sentence-level properties such as sentiment, style, tense, and specificity in generative neural models (Hu et al., 2017; Oraby et al., 2018; Zhang et al., 2018; Shen et al., 2017). Closest to this work is that of Wiseman et al. (2018) who control phrase-level content by using a neuralized hidden semi-Markov model for generation itself. Our work differs in that it makes no independence assumption on the decoder model, uses a faster
training algorithm, and proposes a specific method for adding constraints. Finally, there is a line of work that manipulates the syntactic structure of generated texts, by using some labeled syntactic attribute (e.g., parses) or an exemplar (Deriu and Cieliebak, 2018; Colin and Gardent, 2018; Iyyer et al., 2018; Chen et al., 2019). While our work uses control states, there is no inherent assumption of compositional syntax or grammar.
Posterior regularization (PR) is mostly used in standard EM settings to impose constraints on the posterior distribution that would otherwise be intractable (or computationally hard) in the prior. Ganchev et al. (2010) applies posterior regularization to word alignment, dependency parsing, and part-of-speech tagging. Combining powerful deep neural networks with structured knowledge has been a popular area of study: Xu et al. (2019) applies PR to multi-object generation to limit object overlap; Bilen et al. (2014) focuses on object detection, and uses PR features to exploit mutual exclusion. In natural language processing; Hu et al. (2016a,b) propose an iterative distillation procedure that transfers logic rules into the weights of neural networks, as a regularization to improve accuracy and interpretability.
Finally, the core of this work is the use of amortized inference/variation autoencoder to approximate variational posterior (Kingma and Welling, 2014; Mnih and Gregor, 2014; Rezende et al., 2014). We rely heavily on a structure distribution, either linear chain or semi-Markov, which was introduced as a structured VAEs (Johnson et al., 2016; Krishnan et al., 2017; Ammar et al., 2014). Our setting and optimization are based on Kim et al. (2019), who introduce a latent tree variable in a variational autoencoding model with a CRF as the inference network, and on Yin et al. (2018) who use an encoder-decoder model as the inference network.
7 Experimental Setup
Data and Metrics We consider two standard neural generation benchmarks: E2E (Novikova et al., 2017) and WikiBio (Lebret et al., 2016a) datasets, with examples shown in Figure 1. The E2E dataset contains approximately 50K examples with 8 distinct fields and 945 distinct word types; it contains multiple test references for one source table. We evaluate in terms of BLEU (Papineni et al., 2002), NIST (Belz and Reiter, 2006), ROUGE-L
Table  $(x)$  :name[Clowns] eatType[coffee shop] food[Chinese] customer-rating[1 out of 5] area[riiverside] near[Clare Hall]
Ref.1: Clowns is a coffee shop in the riverside area near Clare Hall that has a rating 1 out of 5. They serve Chinese food.  
Ref.2: The Chinese coffee shop by the riverside near Clare Hall that only has a customer rating of 1 out of 5 is called Clowns.  
Ref.3: There is a Chinese coffee shop near Clare Hall in the riverside area called Clowns its not got a good rating though.
Figure 2: Generation benchmarks. Model is given a table  $x$  consisting of semantic fields and is tasked with generating a description  $y_{1:T}$  of this data. Two example datasets are shown. Left: E2E, Right: WikiBio.
Ref.1: Frederick Parker-Rhodes (21 March 1914 - 21 November 1987) was an English linguist, plant pathologist, computer scientist, mathematician, mystic, and mycologist.
(Lin, 2004), CIDEr (Vedantam et al., 2015) and METEOR (Lavie and Agarwal, 2007), using the official scoring scripts<sup>4</sup>. The WikiBio dataset contains approximately 700K examples, 6K distinct table field types, and 400K word types approximately; it contains one reference for one source table. We follow the metrics from (Lebret et al., 2016a) and evaluate the BLEU, NIST, and ROUGE-4 scores.