SlowGuess/ACL_2020_OCR huggingface dataset:
Xiaoyu Shen $^{1,2}$ , Ernie Chang $^{3}$ , Hui Su $^{4}$ , Cheng Niu $^{4}$  and Dietrich Klakow $^{1}$
$^{1}$ Spoken Language Systems (LSV)  $^{2}$ Max Planck Institute for Informatics  $^{3}$ Department of Language Science and Technology, Saarland Informatics Campus
$^{4}$ Pattern Recognition Center, Wechat AI, Tencent Inc, China
xshen@mpi-inf.mpg.de, cychang@coli.uni-saarland.de {aaronsu, niucheng}@tencent.com, dklakow@lsv.uni-saarland.de
The neural attention model has achieved great success in data-to-text generation tasks. Though usually excelling at producing fluent text, it suffers from the problem of information missing, repetition and "hallucination". Due to the black-box nature of the neural attention architecture, avoiding these problems in a systematic way is non-trivial. To address this concern, we propose to explicitly segment target text into fragment units and align them with their data correspondences. The segmentation and correspondence are jointly learned as latent variables without any human annotations. We further impose a soft statistical constraint to regularize the segmental granularity. The resulting architecture maintains the same expressive power as neural attention models, while being able to generate fully interpretable outputs with several times less computational cost. On both E2E and WebNLG benchmarks, we show the proposed model consistently outperforms its neural attention counterparts.
Data-to-text generation aims at automatically producing natural language descriptions of structured database (Reiter and Dale, 1997). Traditional statistical methods usually tackle this problem by breaking the generation process into a set of local decisions that are learned separately (Belz, 2008; Angeli et al., 2010; Kim and Mooney, 2010; Oya et al., 2014). Recently, neural attention models conflate all steps into a single end-to-end system and largely simplify the training process (Mei et al., 2016; Lebret et al., 2016; Shen et al., 2017; Su et al., 2018, 2019; Chang et al., 2020). However, the black-box conflation also renders the generation uninterpretable and hard to control (Wiseman et al., 2018; Shen et al., 2019a). Verifying the generation correctness in a principled way is nontrivial. In practice, it often suffers from the problem
of information missing, repetition and "hallucination" (Dušek et al., 2018, 2020).
Source data:
Name[Clowns], PriceRange[more than £30], EatType[pub], FamilyFriendly[no]
Generation:
①Name  $\rightarrow$  ②(Clowns)
③FamilyFriendly  $\rightarrow$  ④(is a child-free)
⑤PriceRange  $\rightarrow$  ⑥(，expensive)
⑦ EatType → ⑧ (pub.)
Figure 1: Generation from our model on the E2E dataset. Decoding is performed segment-by-segment. Each segment realizes one data record.  $①\sim ⑧$  mark the decision order in the generation process.
In this work, we propose to explicitly exploit the segmental structure of text. Specifically, we assume the target text is formed from a sequence of segments. Every segment is the result of a two-stage decision: (1) Select a proper data record to be described and (2) Generate corresponding text by paying attention only to the selected data record. This decision is repeated until all desired records have been realized. Figure 1 illustrates this process.
Compared with neural attention, the proposed model has the following advantages: (1) We can monitor the corresponding data record for every segment to be generated. This allows us to easily control the output structure and verify its correctness<sup>1</sup>. (2) Explicitly building the correspondence between segments and data records can potentially reduce the hallucination, as noted in (Wu et al., 2018; Deng et al., 2018) that hard alignment usually outperforms soft attention. (3) When decoding each segment, the model pays attention only to the
<sup>1</sup>For example, we can perform a similar constrained decoding as in Balakrishnan et al. (2019) to rule out outputs with undesired patterns.
7155
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7155-7165 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
selected data record instead of averaging over the entire input data. This largely reduces the memory and computational costs<sup>2</sup>.
To train the model, we do not rely on any human annotations for the segmentation and correspondence, but rather marginalize over all possibilities to maximize the likelihood of target text, which can be efficiently done within polynomial time by dynamic programming. This is essentially similar to traditional methods of inducing segmentation and alignment with semi-markov models (Daumé III and Marcu, 2005; Liang et al., 2009). However, they make strong independence assumptions thus perform poorly as a generative model (Angeli et al., 2010). In contrast, the transition and generation in our model condition on all previously generated text. By integrating an autoregressive neural network structure, our model is able to capture unbounded dependencies while still permitting tractable inference. The training process is stable as it does not require any sampling-based approximations. We further add a soft statistical constraint to control the segmentation granularity via posterior regularization (Ganchev et al., 2010). On both the E2E and WebNLG benchmarks, our model is able to produce significantly higher-quality outputs while being several times computationally cheaper. Due to its fully interpretable segmental structure, it can be easily reconciled with heuristic rules or hand-engineered constraints to control the outputs.
Data-to-text generation is traditionally dealt with using a pipeline structure containing content planning, sentence planning and linguistic realization (Reiter and Dale, 1997). Each target text is split into meaningful fragments and aligned with corresponding data records, either by hand-engineered rules (Kukich, 1983; McKeown, 1992) or statistical induction (Liang et al., 2009; Koncel-Kedziorski et al., 2014; Qin et al., 2018). The segmentation and alignment are used as supervision signals to train the content and sentence planner (Barzilay and Lapata, 2005; Angeli et al., 2010). The linguistic realization is usually implemented by template mining from the training corpus (Kondadadi et al., 2013; Oya et al., 2014). Our model adopts a similar pipeline generative process, but
integrates all the sub-steps into a single end-to-end trainable neural architecture. It can be considered as a neural extension of the PCFG system in Konstas and Lapata (2013), with a more powerful transition probability considering inter-segment dependence and a state-of-the-art attention-based language model as the linguistic realizer. Wiseman et al. (2018) tried a similar neural generative model to induce templates. However, their model only captures loose data-text correspondence and adopts a weak markov assumption for the segment transition probability. Therefore, it underperforms the neural attention baseline as for generation. Our model is also in spirit related to recent attempts at separating content planning and surface realization in neural data-to-text models (Zhao et al., 2018; Puduppully et al., 2019; Moryossef et al., 2019; Ferreira et al., 2019). Nonetheless, all of them resort to manual annotations or hand-engineered rules applicable only for a narrow domain. Our model, instead, automatically learn the optimal content planning via exploring over exponentially many segmentation/correspondence possibilities.
There have been quite a few neural alignment models applied to tasks like machine translation (Wang et al., 2018; Deng et al., 2018), character transduction (Wu et al., 2018; Shankar and Sarawagi, 2019) and summarization (Yu et al., 2016; Shen et al., 2019b). Unlike word-to-word alignment, we focus on learning the alignment between data records and text segments. Some works also integrate neural language models to jointly learn the segmentation and correspondence, e.g., phrase-based machine translation (Huang et al., 2018), speech recognition (Wang et al., 2017) and vision-grounded word segmentation (Kawakami et al., 2019). Data-to-text naturally fits into this scenario since each data record is normally verbalized in one continuous text segment.
3 Background: Data-to-Text
Let  $X, Y$  denote a source-target pair.  $X$  is structured data containing a set of records and  $Y$  corresponds to  $y_{1}, y_{2}, \ldots, y_{m}$  which is a text description of  $X$ . The goal of data-to-text generation is to learn a distribution  $p(Y|X)$  to automatically generate proper text describing the content of the data.
The neural attention architecture handles this task with an encode-attend-decode process (Bahdanau et al., 2015). The input  $X$  is processed into a sequence of  $x_{1}, x_{2}, \ldots, x_{n}$ , normally by flatten
$^{2}$ Coarse-to-fine attention (Ling and Rush, 2017; Deng et al., 2017) was proposed for the same motivation, but they resort to reinforcement learning which is hard to train, and the performance is sacrificed for efficiency.
7156
ing the data records (Wiseman et al., 2017). The encoder encodes each  $x_{i}$  into a vector  $h_i$ . At each time step, the decoder attends to encoded vectors and outputs the probability of the next token by  $p(y_t | y_{1:t-1}, A_t)$ .  $A_t$  is a weighted average of source vectors:
$$
A _ {t} = \sum_ {i} \alpha_ {t, i} h _ {i}
$$
$$
\alpha_ {t, i} = \frac {e ^ {f \left(h _ {i} , d _ {t}\right)}}{\sum_ {j} e ^ {f \left(h _ {j} , d _ {t}\right)}} \tag {1}
$$
$d_{t}$  is the hidden state of the decoder at time step  $t$ .  $f$  is a score function to compute the similarity between  $h_i$  and  $d_{t}$  (Luong et al., 2015).
Suppose the input data \( X \) contains a set of records \( r_1, r_2, \ldots, r_K \). Our assumption is that the target text \( y_{1:m} \) can be segmented into a sequence of fragments. Each fragment corresponds to one data record. As the ground-truth segmentation and correspondence are not available, we need to enumerate over all possibilities to compute the likelihood of \( y_{1:m} \). Denote by \( S_y \) the set containing all valid segmentation of \( y_{1:m} \). For any valid segmentation \( s_{1:\tau_s} \in S_y \), \( \pi(s_{1:\tau_s}) = y_{1:m} \), where \( \pi \) means concatenation and \( \tau_s \) is the number of segments. For example, let \( m = 5 \) and \( \tau_s = 3 \). One possible segmentation would be \( s_{1:\tau_s} = \{\{y_1, y_2, \\), \\), \\), \\), \\), \\), \\), \\), \\), \\), \\), \\), \\), \\), \\)
The end-of-segment symbol and is removed when applying the \( \pi \) operator. We further define \( c(*) \) to be the corresponding data record(s) of \( * \). The likelihood of each text is then computed by enumerating over all possibilities of \( s_{1:\tau_s} \) and \( c(s_{1:\tau_s}) \):
$$
\begin{array}{l} p (y _ {1: m} | X) = \sum_ {s _ {1: \tau_ {s}} \in \mathcal {S} _ {y}} p (s _ {1: \tau_ {s}} | X) \\ = \sum_ {s _ {1: \tau_ {s}} \in \mathcal {S} _ {y}} \prod_ {o = 1} ^ {\tau_ {s}} \sum_ {c (s _ {o}) = r _ {1}} ^ {r _ {K}} p \left(s _ {o} \mid \pi \left(s _ {<   o}\right), c \left(s _ {o}\right)\right) \tag {2} \\ \times p \left(c \left(s _ {o}\right) \mid \pi \left(s _ {<   o}\right), c \left(s _ {<   o}\right)\right) \\ \end{array}
$$
Every segment is generated by first selecting the data record based on the transition probability  $p(c(s_o) | \pi(s_{<o}), c(s_{<o}))$ , then generating tokens based on the word generation probability  $p(s_o | \pi(s_{<o}), c(s_o))$ . Figure 2 illustrates the generation process of our model.
Generation Probability We base the generation probability on the same decoder as in neural attention models. The only difference is that the model
Figure 2: Generation process of our approach. Segment end symbol  $\$$  is ignored when updating the state of the decoder. Solid arrows indicate the transition model and dashed arrows indicate the generation model. Every segment  $s_o$  is generated by attending only to the corresponding data record  $c(s_o)$ .
can only pay attention to its corresponding data record. The attention scores of other records are masked out when decoding  $s_o$ :
$$
\alpha_ {t, i} = \frac {e ^ {f (h _ {i} , d _ {t})} \mathbb {1} \left(x _ {i} \in c \left(s _ {o}\right)\right)}{\sum_ {j} e ^ {f (h _ {j} , d _ {t})} \mathbb {1} \left(x _ {j} \in c \left(s _ {o}\right)\right)}
$$
where  $\mathbb{1}$  is the indicator function. This forces the model to learn proper correspondences and enhances the connection between each segment and the data record it describes.
Following the common practice, we define the output probability with the pointer generator (See et al., 2017; Wiseman et al., 2017):
$$
p _ {g e n} = \sigma \left(\mathrm {M L P} _ {g} ([ d _ {t} \circ A _ {t} ]))\right)
$$
$$
p _ {v o c a b} = \operatorname {s o f t m a x} \left(W _ {1} d _ {t} + W _ {2} A _ {t}\right)
$$
$$
\begin{array}{l} p _ {\theta} \left(y _ {t} \mid y _ {<   t}\right) = p _ {g e n} p _ {v o c a b} \left(y _ {t}\right) \\ + \left(1 - p _ {g e n}\right) \sum_ {i: y _ {t} = x _ {i}} a _ {t, i} \\ \end{array}
$$
$d_{t}$  is the decoder's hidden state at time step  $t$ .  $\circ$  denotes vector concatenation.  $A_{t}$  is the context vector. MLP indicates multi-layer perceptron and  $\sigma$  normalizes the score between  $(0,1)$ .  $W_{1}$  and  $W_{2}$  are trainable matrices.  $p_{gen}$  is the probability that the word is generated from a fixed vocabulary distribution  $p_{vocab}$  instead of being copied. The final decoding probability  $p_{\theta}(y_t)$  is marginalized over  $p_{vocab}$  and the copy distribution. The generation probability of  $s_{o}$  factorizes over the words within it and the end-of-segment token:
$$
p \left(s _ {o} \mid \pi \left(s _ {<   o}\right), c \left(s _ {o}\right)\right) = p _ {\theta} (\$ \mid y _ {1: t}) \prod_ {y _ {t} \in s _ {o}} p _ {\theta} \left(y _ {t} \mid y _ {<   t}\right)
$$
Transition Probability We make a mild assumption that  $c(s_{o})$  is dependent only on  $c(s_{o - 1})$  and  $\pi (s_{1:o - 1})$  but irrelevant of  $c(s_{< o - 1})$ , which is a common practice when modelling alignment (Och
et al., 1999; Yu et al., 2016; Shankar and Sarawagi, 2019). The transition probability is defined as:
$$
\begin{array}{l} p \left(c \left(s _ {o}\right) = r _ {i} \mid c \left(s _ {<   o}\right), \pi \left(s _ {<   o}\right)\right) \\ \approx p \left(c \left(s _ {o}\right) = r _ {i} \mid c \left(s _ {o - 1}\right), \pi \left(s _ {<   o}\right)\right) \tag {3} \\ \propto f \left(r _ {i}\right) ^ {T} \left[ M ^ {T} A _ {s _ {o - 1}} + N ^ {T} d _ {s _ {o - 1}} \right] \\ \end{array}
$$
A softmax layer is finally applied to the above equation to normalize it as a proper probability distribution.  $f(r_{i})$  is a representation of  $r_i$ , which is defined as a max pooling over all the word embeddings contained in  $r_i$ .  $A_{s_{o-1}}$  is the attention context vector when decoding the last token in  $s_{o-1}$ , defined as in Equation 1. It carries important information from  $c(s_{o-1})$  to help predict  $c(s_o)$ .  $d_{s_{o-1}}$  is the hidden state of the neural decoder which goes through all history tokens  $\pi(s_{1:o-1})$ .  $M, N$  are trainable matrices to project  $A_{s_{o-1}}$  and  $d_{s_{o-1}}$  into the same dimension as  $f(r_i)$ .
We further add one constraint to prohibit self-transition, which can be easily done by zeroing out the transition probability in Equation 3 when  $c(s_{o}) = c(s_{o - 1})$ . This forces the model to group together text describing the same data record.
Since Equation 3 conditions on all previously generated text, it is able to capture more complex dependencies as in semi-markov models (Liang et al., 2009; Wiseman et al., 2018).
Null Record In our task, we find some frequent phrases, e.g., "it is", "and", tend to be wrongly aligned with some random records, similar to the garbage collection issue in statistical alignment (Brown et al., 1993). This hurt the model interpretability. Therefore, we introduce an additional null record  $r_0$  to attract these non-content phrases. The context vector when aligned to  $r_0$  is a zero vector so that the decoder will decode words based solely on the language model without relying on the input data.
Training Equation 2 contains exponentially many combinations to enumerate over. Here we show how to efficiently compute the likelihood with the forward algorithm in dynamic programming (Rabiner, 1989). We define the forward variable  $\alpha(i,j) = p(y_{1:i}, c(y_i) = j|X)$ . With the base  $\alpha(1,j) = p(y_1|c(y_1) = j)$ . The recursion goes as
follows for  $i = 1,2,\ldots ,m - 1$
$$
\begin{array}{l} \alpha (i + 1, j) = \sum_ {p = 1} ^ {i} \sum_ {q = r _ {0}} ^ {r _ {K}} \alpha (p, q) \\ \times p \left(c \left(y _ {p + 1}\right) = j \mid c \left(y _ {p}\right) = q, y _ {1: p}\right) \tag {4} \\ \times p \left(y _ {p + 1: i + 1} \mid c \left(y _ {p + 1: i + 1}\right) = q, y _ {1: p}\right) \\ \times p (\$ | c (y _ {p + 1: i + 1}) = q, y _ {1: i + 1}) \\ \end{array}
$$
The final likelihood of the target text can be computed as  $p(y_{1:m}|X) = \sum_{j=r_0}^{r_K} \alpha(m, j)$ . As the forward algorithm is fully differentiable, we maximize the log-likelihood of the target text by backpropagating through the dynamic programming. The process is essentially equivalent to the generalized EM algorithm (Eisner, 2016). By means of the modern automatic differentiation tools, we avoid the necessity to calculate the posterior distribution manually (Kim et al., 2018).
To speed up training, we set a threshold  $L$  to the maximum length of a segment as in Liang et al. (2009); Wiseman et al. (2018). This changes the complexity in Equation 4 to a constant  $O(LK)$  instead of scaling linearly with the length of the target text. Moreover, as pointed out in Wang et al. (2017), the computation for the longest segment can be reused for shorter segments. We therefore first compute the generation and transition probability for the whole sequence in one pass. The intermediate results are then cached to efficiently proceed the forward algorithm without any re-computation.
One last issue is the numerical precision, it is important to use the log-space binary operations to avoid underflow (Kim et al., 2017).
Near[riverside], Food[French], EatType[pub], Name[Cotto]
1. [Near]Near[the]Null[riverside]Near[is a]Null[French]Food [pub]Eaftypecalled]Null[Cotto]Name[.]Null
2. [Near the riverside]Near[is]Null[a French]Food[pub]EatType [called Cotto]Name[.]Null
3. [Near the riverside]Near[is a French]Food[pub]EATType [called Cotto .]Name
4. [Near the riverside]Near[is a French pub]Food [called Cotto.]Name
Table 1: Segmentation with various granularities. 1 is too fine-grained while 4 is too coarse. We expect a segmentation like 2 or 3 to better control the generation.
Segmentation Granularity There are several valid segmentations for a given text. As shown in Table 1, when the segmentation (example 1) is too fine-grained, controlling the output information becomes difficult because the content of
7158
one data record is realized in separate pieces  $^{3}$ . When it is too coarse, the alignment might become less accurate (as in Example 4, "pub" is wrongly merged with previous words and aligned together to the "Food" record). In practice, we expect the segmentation to stay with accurate alignment yet avoid being too brokenly separated. To control the granularity as we want, we utilize posterior regularization (Ganchev et al., 2010) to constrain the expected number of segments for each text  $^{4}$ , which can be calculated by going through a similar forward pass as in Equation 4 (Eisner, 2002). Most computation is shared without significant extra burden. The final loss function is:
$$
- \log \mathbb {E} _ {\mathcal {S} _ {y}} p (s _ {1: \tau_ {s}} | X) + \max  (\left| \mathbb {E} _ {\mathcal {S} _ {y}} \tau_ {s} - \eta \right|, \gamma) \tag {5}
$$
$\log \mathbb{E}_{S_y}p(s_{1:\tau_s}|X)$  is the log-likelihood of target text after marginalizing over all valid segmentations.  $\mathbb{E}_{S_y}\tau_s$  is the expected number of segments and  $\eta ,\gamma$  are hyperparameters. We use the maxmargin loss to encourage  $\mathbb{E}_{S_y}\tau_s$  to stay close to  $\eta$  under a tolerance range of  $\gamma$
Decoding The segment-by-segment generation process allows us to easily constrain the output structure. Undesirable patterns can be rejected before the whole text is generated. We adopt three simple constraints for the decoder:
1. Segments must not be empty.
2. The same data record cannot be realized more than once (except for the null record).
3. The generation will not finish until all data records have been realized.
Constraint 2 and 3 directly address the information repetition and missing problem. When segments are incrementally generated, the constraints will be checked against for validity. Note that adding the constraints hardly incur any cost, the decoding process is still finished in one pass. No post-processing or reranking is needed.
Computational Complexity Suppose the input data has  $M$  records and each record contains  $N$  tokens. The computational complexity for neural attention models is  $O(MN)$  at each decoding step where the whole input is retrieved. Our model, similar to chunkwise attention (Chiu and Raffel, 2018) or coarse-to-fine attention (Ling and Rush, 2017), reduces the cost to  $O(M + N)$ , where we select the record in  $O(M)$  at the beginning of each segment and attend only to the selected record in  $O(N)$  when decoding every word. For larger input data, our model can be significantly cheaper than neural attention models.
Dataset We conduct experiments on the E2E (Novikova et al., 2017b) and WebNLG (Colin et al., 2016) datasets. E2E is a crowd-sourced dataset containing 50k instances in the restaurant domain. The inputs are dialogue acts consisting of three to eight slot-value pairs. WebNLG contains 25k instances describing entities belonging to fifteen distinct DBpedia categories. The inputs are up to seven RDF triples of the form (subject, relation, object).
Implementation Details We use a bi-directional LSTM encoder and uni-directional LSTM decoder for all experiments. Input data records are concatenated into a sequence and fed into the encoder. We choose the hidden size of encoder/decoder as 512 for E2E and 256 for WebNLG. The word embedding is with size 100 for both datasets and initialized with the pre-trained Glove embedding  $^5$  (Pennington et al., 2014). We use a drop out rate of 0.3 for both the encoder and decoder. Models are trained using the Adam optimizer (Kingma and Ba, 2014) with batch size 64. The learning rate is initialized to 0.01 and decays an order of magnitude once the validation loss increases. All hyperparameters are chosen with grid search according to the validation loss. Models are implemented based on the open-source library PyTorch (Paszke et al., 2019). We set the hyperparameters in Eq. 5 as  $\eta = K,\gamma = 1$  (recall that  $K$  is the number of records in the input data). The intuition is that every text is expected to realize the content of all  $K$  input records. It is natural to assume every text can be roughly segmented into  $K$  fragments, each corresponding to one data record. A deviation of
5nlp.stanford.edu/data/glove.6B.zip
3The finer-grained segmentation might be useful if the focus is on modeling the detailed discourse structure instead of the information accuracy (Reed et al., 2018; Balakrishnan et al., 2019), which we leave for future work.
4We can also utilize some heuristic rules to help segmentation. For example, we can prevent breaking syntactic elements obtained from an external parser (Yang et al., 2019) or match entity names with handcrafted rules (Chen et al., 2018). The interpretability of the segmental structure allows easy combination with these rules. We focus on a general domain-agnostic method in this paper, though heuristic rules might bring further improvement under certain cases.
7159
$K \pm 1$  is allowed for noisy data or text with complex structures.