SlowGuess/ACL_2020_OCR huggingface dataset:
Hayate Iso†* Chao Qiao‡ Hang Li‡
†Nara Institute of Science and Technology ‡ByteDance AI Lab
hyate.iso@gmail.com,{qiaochao, lihang.lh}@bytedance.com
We propose a novel text editing task, referred to as fact-based text editing, in which the goal is to revise a given document to better describe the facts in a knowledge base (e.g., several triples). The task is important in practice because reflecting the truth is a common requirement in text editing. First, we propose a method for automatically generating a dataset for research on fact-based text editing, where each instance consists of a draft text, a revised text, and several facts represented in triples. We apply the method into two public table-to-text datasets, obtaining two new datasets consisting of 233k and 37k instances, respectively. Next, we propose a new neural network architecture for fact-based text editing, called FACTEDITOR, which edits a draft text by referring to given facts using a buffer, a stream, and a memory. A straightforward approach to address the problem would be to employ an encoder-decoder model. Our experimental results on the two datasets show that FACTEDITOR outperforms the encoder-decoder approach in terms of fidelity and fluency. The results also show that FACTEDITOR conducts inference faster than the encoder-decoder approach.
Automatic editing of text by computer is an important application, which can help human writers to write better documents in terms of accuracy, fluency, etc. The task is easier and more practical than the automatic generation of texts from scratch and is attracting attention recently (Yang et al., 2017; Yin et al., 2019). In this paper, we consider a new and specific setting of it, referred to as fact-based text editing, in which a draft text and several facts (represented in triples) are given, and the system
Set of triples
{Baymax, creator, Douncan_Rouleau), (Douncan_Rouleau, nationality, American), (Baymax, creator, Steven_T_Seagle), (Steven_T_Seagle, nationality, American), (Baymax, series, Big_Hero_6), (Big_Hero_6, starring, Scott_Adsit)}
Baymax was created by Duncan_Rouleau, a winner of Eagle_Award. Baymax is a character in Big_Hero_6.
Baymax was created by American creators Duncan_Rouleau and Steven_T.Seagle. Baymax is a character in Big_Hero_6 which stars Scott_Adsit.
Table 1: Example of fact-based text editing. Facts are represented in triples. The facts in green appear in both draft text and triples. The facts in orange are present in the draft text, but absent from the triples. The facts in blue do not appear in the draft text, but in the triples. The task of fact-based text editing is to edit the draft text on the basis of the triples, by deleting unsupported facts and inserting missing facts while retaining supported facts.
aims to revise the text by adding missing facts and deleting unsupported facts. Table 1 gives an example of the task.
As far as we know, no previous work did address the problem. In a text-to-text generation, given a text, the system automatically creates another text, where the new text can be a text in another language (machine translation), a summary of the original text (summarization), or a text in better form (text editing). In a table-to-text generation, given a table containing facts in triples, the system automatically composes a text, which describes the facts. The former is a text-to-text problem, and the latter a table-to-text problem. In comparison, fact-based text editing can be viewed as a 'text&table-to-text' problem.
* The work was done when Hayate Iso was a research intern at ByteDance AI Lab.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 171-182 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
First, we devise a method for automatically creating a dataset for fact-based text editing. Recently, several table-to-text datasets have been created and released, consisting of pairs of facts and corresponding descriptions. We leverage such kind of data in our method. We first retrieve facts and their descriptions. Next, we take the descriptions as revised texts and automatically generate draft texts based on the facts using several rules. We build two datasets for fact-based text editing on the basis of WEBNLG (Gardent et al., 2017) and ROTOWIRE, consisting of  $233\mathrm{k}$  and  $37\mathrm{k}$  instances respectively (Wiseman et al., 2017) $^{1}$ .
Second, we propose a model for fact-based text editing called FACTEDITOR. One could employ an encoder-decoder model, such as an encoder-decoder model, to perform the task. The encoder-decoder model implicitly represents the actions for transforming the draft text into a revised text. In contrast, FACTEDITOR explicitly represents the actions for text editing, including Keep, Drop, and Gen, which means retention, deletion, and generation of word respectively. The model utilizes a buffer for storing the draft text, a stream to store the revised text, and a memory for storing the facts. It also employs a neural network to control the entire editing process. FACTEDITOR has a lower time complexity than the encoder-decoder model, and thus it can edit a text more efficiently.
Experimental results show that FACTEDITOR outperforms the baseline model of using encoder-decoder for text editing in terms of fidelity and fluency, and also show that FACTEDITOR can perform text editing faster than the encoder-decoder model.
2.1 Text Editing
Text editing has been studied in different settings such as automatic post-editing (Knight and Chander, 1994; Simard et al., 2007; Yang et al., 2017), paraphrasing (Dolan and Brockett, 2005), sentence simplification (Inui et al., 2003; Wubben et al., 2012), grammar error correction (Ng et al., 2014), and text style transfer (Shen et al., 2017; Hu et al., 2017).
The rise of encoder-decoder models (Cho et al., 2014; Sutskever et al., 2014) as well as the attention (Bahdanau et al., 2015; Vaswani et al., 2017)
and copy mechanisms (Gu et al., 2016; Gulcehre et al., 2016) has dramatically changed the landscape, and now one can perform the task relatively easily with an encoder-decoder model such as Transformer provided that a sufficient amount of data is available. For example, Li et al. (2018) introduce a deep reinforcement learning framework for paraphrasing, consisting of a generator and an evaluator. Yin et al. (2019) formalize the problem of text edit as learning and utilization of edit representations and propose an encoder-decoder model for the task. Zhao et al. (2018) integrate paraphrasing rules with the Transformer model for text simplification. Zhao et al. (2019) proposes a method for English grammar correction using a Transformer and copy mechanism.
Another approach to text editing is to view the problem as sequential tagging instead of encoder-decoder. In this way, the efficiency of learning and prediction can be significantly enhanced. Vu and Haffari (2018) and Dong et al. (2019) conduct automatic post-editing and text simplification on the basis of edit operations and employ Neural Programmer-Interpreter (Reed and De Freitas, 2016) to predict the sequence of edits given a sequence of words, where the edits include KEEP, DROP, and ADD. Malmi et al. (2019) propose a sequential tagging model that assigns a tag (KEEP or DELETE) to each word in the input sequence and also decides whether to add a phrase before the word. Our proposed approach is also based on sequential tagging of actions. It is designed for fact-based text editing, not text-to-text generation, however.
2.2 Table-to-Text Generation
Table-to-text generation is the task which aims to generate a text from structured data (Reiter and Dale, 2000; Gatt and Krahmer, 2018), for example, a text from an infobox about a term in biology in wikipedia (Lebret et al., 2016) and a description of restaurant from a structured representation (Novikova et al., 2017). Encoder-decoder models can also be employed in table-to-text generation with structured data as input and generated text as output, for example, as in (Lebret et al., 2016). Puduppully et al. (2019) and Iso et al. (2019) propose utilizing an entity tracking module for document-level table-to-text generation.
One issue with table-to-text is that the style of generated texts can be diverse (Iso et al., 2019). Re-
1The datasets are publicly available at https:// github.com/isomap/factedit
(a) Example for insertion. The revised template  $\pmb{y}^{\prime}$  and the reference template  $\hat{\pmb{x}}^{\prime}$  share subsequences. The set of triple templates  $\mathcal{T}\backslash \tilde{\mathcal{T}}$  is  $\{(BRIDGE - 1, \text{operator, PATIENT-2})\}$ . Our method removes "that was operated by PATIENT-2" from the revised template  $\pmb{y}^{\prime}$  to create the draft template  $\pmb{x}^{\prime}$ .
(b) Example for deletion. The revised template  $\mathbf{y}'$  and the reference template  $\hat{\mathbf{x}}'$  share subsequences. The set of triple templates  $\hat{\mathcal{T}}\backslash \mathcal{T}$  is  $\{(\text{AGENT-1}, \text{fullName}, \text{PATIENT-1})\}$ . Our method copies "whose full name is PATIENT-1" from the reference template  $\mathbf{x}'$  to create the draft template  $\mathbf{x}'$ .
Table 2: Examples for insertion and deletion, where words in green are matched, words in gray are not matched, words in blue are copied, and words in orange are removed. Best viewed in color.
searchers have developed methods to deal with the problem using other texts as templates (Hashimoto et al., 2018; Guu et al., 2018; Peng et al., 2019). The difference between the approach and fact-based text editing is that the former is about table-to-text generation based on other texts, while the latter is about text-to-text generation based on structured data.
3 Data Creation
In this section, we describe our method of data creation for fact-based text editing. The method automatically constructs a dataset from an existing table-to-text dataset.
3.1 Data Sources
There are two benchmark datasets of table-to-text, WEBNLG (Gardent et al., 2017) and ROTOWIRE (Wiseman et al., 2017). We create two datasets on the basis of them, referred to as WEBEDIT and ROTOEDIT respectively. In the datasets, each instance consists of a table (structured data) and an associated text (unstructured data) describing almost the same content.
For each instance, we take the table as triples of facts and the associated text as a revised text, and we automatically create a draft text. The set of triples is represented as  $\mathcal{T} = \{t\}$ . Each triple  $t$  consists of subject, predicate, and object, denoted
as  $t = (\text{subj}, \text{pred}, \text{obj})$ . For simplicity, we refer to the nouns or noun phrases of subject and object simply as entities. The revised text is a sequence of words denoted as  $\mathbf{y}$ . The draft text is a sequence of words denoted as  $\mathbf{x}$ .
Given the set of triples  $\mathcal{T}$  and the revised text  $\pmb{y}$ , we aim to create a draft text  $\pmb{x}$ , such that  $\pmb{x}$  is not in accordance with  $\mathcal{T}$ , in contrast to  $\pmb{y}$ , and therefore text editing from  $\pmb{x}$  to  $\pmb{y}$  is needed.
3.2 Procedure
Our method first creates templates for all the sets of triples and revised texts and then constructs a draft text for each set of triples and revised text based on their related templates.
Creation of templates
For each instance, our method first delexicalizes the entity words in the set of triples  $\mathcal{T}$  and the revised text  $\pmb{y}$  to obtain a set of triple templates  $\mathcal{T}'$  and a revised template  $\pmb{y}'$ . For example, given  $\mathcal{T} = \{\text{(Baymax, voice, Scott\_Adsit)}\}$  and  $\pmb{y} = \text{"Scott\_Adsit does the voice for Baymax"}$ , it produces the set of triple templates  $\mathcal{T}' = \{\text{(AGENT-1, voice, PATIENT-1)}\}$  and the revised template  $\pmb{y}' = \text{"AGENT-1 does the voice for PATIENT-1"}$ . Our method then collects all the sets of triple templates  $\mathcal{T}'$  and revised templates  $\pmb{y}'$  and retains them in a key-value store with  $\pmb{y}'$  being a key and  $\mathcal{T}'$  being a value.
Creation of draft text
Next, our method constructs a draft text  $\pmb{x}$  using a set of triple templates  $\mathcal{T}'$  and a revised template  $\pmb{y}'$ . For simplicity, it only considers the use of either insertion or deletion in the text editing, and one can easily make an extension of it to a more complex
2The data is available at https://github.com/ ThiagoCF05/webnlg. We utilize version 1.5. 3We utilize the ROTOWIRE-MODIFIED data provided by Iso et al. (2019) available at https://github.com/ aistairc/rotowire-modified. The authors also provide an information extractor for processing the data. 4In ROTOWIRE, we discard redundant box-scores and unrelated sentences using the information extractor and heuristic rules.
setting. Note that the process of data creation is reverse to that of text editing.
Given a pair of  $\mathcal{T}'$  and  $\pmb{y}^{\prime}$ , our method retrieves another pair denoted as  $\hat{\mathcal{T}}'$  and  $\hat{x}^{\prime}$ , such that  $\pmb{y}^{\prime}$  and  $\hat{\pmb{x}}^{\prime}$  have the longest common subsequences. We refer to  $\hat{\pmb{x}}^{\prime}$  as a reference template. There are two possibilities;  $\hat{\mathcal{T}}'$  is a subset or a superset of  $\mathcal{T}'$ . (We ignore the case in which they are identical.) Our method then manages to change  $\pmb{y}^{\prime}$  to a draft template denoted as  $\pmb{x}^{\prime}$  on the basis of the relation between  $\mathcal{T}'$  and  $\hat{\mathcal{T}}'$ . If  $\hat{\mathcal{T}}' \subsetneq \mathcal{T}'$ , then the draft template  $\pmb{x}^{\prime}$  created is for insertion, and if  $\hat{\mathcal{T}}' \supsetneq \mathcal{T}'$ , then the draft template  $\pmb{x}^{\prime}$  created is for deletion.
For insertion, the revised template  $\pmb{y}^{\prime}$  and the reference template  $\hat{\pmb{x}}^{\prime}$  share subsequences, and the set of triples  $\mathcal{T}\backslash \hat{\mathcal{T}}$  appear in  $\pmb{y}^{\prime}$  but not in  $\hat{\pmb{x}}^{\prime}$ . Our method keeps the shared subsequences in  $\pmb{y}^{\prime}$ , removes the subsequences in  $\pmb{y}^{\prime}$  about  $\mathcal{T}\backslash \hat{\mathcal{T}}$ , and copies the rest of words in  $\pmb{y}^{\prime}$ , to create the draft template  $\pmb{x}^{\prime}$ . Table 2a gives an example. The shared subsequences "AGENT-1 performed as PATIENT-3 on BRIDGE-1 mission" are kept. The set of triple templates  $\mathcal{T}\backslash \hat{\mathcal{T}}$  is {BRIDGE-1, operator, PATIENT-2)}. The subsequence "that was operated by PATIENT-2" is removed. Note that the subsequence "served" is not copied because it is not shared by  $\pmb{y}^{\prime}$  and  $\hat{\pmb{x}}^{\prime}$ .
For deletion, the revised template  $\mathbf{y}'$  and the reference template  $\hat{\mathbf{x}}'$  share subsequences. The set of triples  $\hat{\mathcal{T}}\backslash \mathcal{T}$  appear in  $\hat{\mathbf{x}}^{\prime}$  but not in  $\mathbf{y}'$ . Our method retains the shared subsequences in  $\mathbf{y}'$ , copies the subsequences in  $\hat{\mathbf{x}}^{\prime}$  about  $\hat{\mathcal{T}}\backslash \mathcal{T}$ , and copies the rest of words in  $\mathbf{y}'$ , to create the draft template  $\mathbf{x}'$ . Table 2b gives an example. The subsequences "AGENT-1 was created by BRIDGE-1 and PATIENT-2" are retained. The set of triple templates  $\hat{\mathcal{T}}\backslash \mathcal{T}$  is { (AGENT-1, fullName, PATIENT-1)}. The subsequence "whose full name is PATIENT-1" is copied. Note that the subsequence "the character of" is not copied because it is not shared by  $\mathbf{y}'$  and  $\hat{\mathbf{x}}'$ .
After getting the draft template  $\boldsymbol{x}^{\prime}$ , our method lexicalizes it to obtain a draft text  $\boldsymbol{x}$ , where the lexicons (entity words) are collected from the corresponding revised text  $\boldsymbol{y}$ .
We obtain two datasets with our method, referred to as WEBEDIT and ROTOEDIT, respectively. Table 3 gives the statistics of the datasets.
In the WEBEDIT data, sometimes entities only appear in the subj's of triples. In such cases, we also make them appear in the obj's. To do so, we
WEBEDIT	ROTOEDIT
TRAIN	VALID	TEST	TRAIN	VALID	TEST
#D	181k	23k	29k	27k	5.3k	4.9k
#Wd	4.1M	495k	624k	4.7M	904k	839k
#Wr	4.2M	525k	649k	5.6M	1.1M	1.0M
#S	403k	49k	62k	209k	40k	36k
Table 3: Statistics of WEBEDIT and ROTOEDIT, where  $\# D$  is the number of instances,  $\# \mathcal{W}_{\mathrm{d}}$  and  $\# \mathcal{W}_{\mathrm{r}}$  are the total numbers of words in the draft texts and the revised texts, respectively, and  $\# S$  is total the number of sentences.
introduce an additional triple (ROOT, IsOf, subj) for each subj, where ROOT is a dummy entity.
4 FACTEDITOR
In this section, we describe our proposed model for fact-based text editing referred to as FACTEDITOR.
FACTEDITOR transforms a draft text into a revised text based on given triples. The model consists of three components, a buffer for storing the draft text and its representations, a stream for storing the revised text and its representations, and a memory for storing the triples and their representations, as shown in Figure 1.
FACTEDITOR scans the text in the buffer, copies the parts of text from the buffer into the stream if they are described in the triples in the memory, deletes the parts of the text if they are not mentioned in the triples, and inserts new parts of next into the stream which is only presented in the triples.
The architecture of FACTEDITOR is inspired by those in sentence parsing Dyer et al. (2015); Watanabe and Sumita (2015). The actual processing of FACTEDITOR is to generate a sequence of words into the stream from the given sequence of words in the buffer and the set of triples in the memory. A neural network is employed to control the entire editing process.
4.2 Neural Network
FACTEDITOR first initializes the representations of content in the buffer, stream, and memory.
There is a feed-forward network associated with the memory, utilized to create the embeddings of triples. Let  $M$  denote the number of triples. The
embedding of triple  $t_j, j = 1, \dots, M$  is calculated as
$$
\pmb {t} _ {j} = \mathrm {t a n h} (\pmb {W} _ {\mathrm {t}} \cdot [ \pmb {e} _ {j} ^ {\mathrm {s u b j}}; \pmb {e} _ {j} ^ {\mathrm {p r e d}}; \pmb {e} _ {j} ^ {\mathrm {o b j}} ] + \pmb {b} _ {\mathrm {t}}),
$$
where  $W_{t}$  and  $\pmb{b}_{\mathrm{t}}$  denote parameters,  $e_j^{\mathrm{subj}},e_j^{\mathrm{pred}},e_j^{\mathrm{obj}}$  denote the embeddings of subject, predicate, and object of triple  $t_j$ , and  $[\cdot ]$  denotes vector concatenation.
There is a bi-directional LSTM associated with the buffer, utilized to create the embeddings of words of draft text. The embeddings are obtained as  $\pmb{b} = \mathrm{BiLSTM}(\pmb{x})$ , where  $\pmb{x} = (x_{1},\dots,x_{N})$  is the list of embeddings of words and  $\pmb{b} = (b_{1},\dots,b_{N})$  is the list of representations of words, where  $N$  denotes the number of words.
There is an LSTM associated with the stream for representing the hidden states of the stream. The first hidden state is initialized as
$$
\boldsymbol {s} _ {1} = \tanh  \left(\boldsymbol {W} _ {\mathrm {s}} \cdot \left[ \frac {\sum_ {i = 1} ^ {N} \boldsymbol {b} _ {i}}{N}; \frac {\sum_ {j = 1} ^ {M} \boldsymbol {t} _ {j}}{M} \right] + \boldsymbol {b} _ {\mathrm {s}}\right)
$$
where  $W_{s}$  and  $\pmb{b}_{s}$  denotes parameters.
Action prediction
FACTEDITOR predicts an action at each time  $t$  using the LSTM. There are three types of action, namely Keep, Drop, and Gen. First, it composes a context vector  $\tilde{t}_t$  of triples at time  $t$  using attention
$$
\tilde {\boldsymbol {t}} _ {t} = \sum_ {j = 1} ^ {M} \alpha_ {t, j} \boldsymbol {t} _ {j}
$$
where  $\alpha_{t,j}$  is a weight calculated as
$$
\alpha_ {t, j} \propto \exp \left(\boldsymbol {v} _ {\alpha} ^ {\top} \cdot \tanh  \left(\boldsymbol {W} _ {\alpha} \cdot [ \boldsymbol {s} _ {t}; \boldsymbol {b} _ {t}; \boldsymbol {t} _ {j} ]\right)\right)
$$
where  $\pmb{v}_{\alpha}$  and  $W_{\alpha}$  are parameters. Then, it creates the hidden state  $\pmb{z}_t$  for action prediction at time  $t$
$$
\boldsymbol {z} _ {t} = \tanh  \left(\boldsymbol {W} _ {z} \cdot \left[ \boldsymbol {s} _ {t}; \boldsymbol {b} _ {t}; \tilde {\boldsymbol {t}} _ {t} \right] + \boldsymbol {b} _ {z}\right)
$$
where  $W_{z}$  and  $\pmb{b}_{z}$  denote parameters. Next, it calculates the probability of action  $a_{t}$
$$
P \left(a _ {t} \mid \boldsymbol {z} _ {t}\right) = \operatorname {s o f t m a x} \left(\boldsymbol {W} _ {a} \cdot \boldsymbol {z} _ {t}\right)
$$
where  $W_{a}$  denotes parameters, and chooses the action having the largest probability.
(a) The Keep action, where the top embedding of the buffer  $\pmb{b}_t$  is popped and the concatenated vector  $[\tilde{\pmb{t}}_t; \pmb{b}_t]$  is pushed into the stream LSTM.
(b) The Drop action, where the top embedding of the buffer  $\pmb{b}_t$  is popped and the state in the stream is reused at the next time step  $t + 1$ .
Figure 1: Actions of FACTEDITOR.
(c) The Gen action, where the concatenated vector  $[\tilde{t}_t;W_p\pmb {y}_t]$  is pushed into the stream, and the top embedding of the buffer is reused at the next time step  $t + 1$
Action execution
FACTEDITOR takes action based on the prediction result at time  $t$ .
For Keep at time  $t$ , FACTEDITOR pops the top embedding  $\pmb{b}_t$  in the buffer, and feeds the combination of the top embedding  $\pmb{b}_t$  and the context vector of triples  $\tilde{\pmb{t}}_t$  into the stream, as shown in Fig. 1a. The state of stream is updated with the LSTM as  $s_{t+1} = \mathrm{LSTM}([\tilde{\pmb{t}}_t; \pmb{b}_t], s_t)$ . FACTEDITOR also copies the top word in the buffer into the stream.
For Drop at time  $t$ , FACTEDITOR pops the top embedding in the buffer and proceeds to the next state, as shown in Fig. 1b. The state of stream is updated as  $s_{t+1} = s_t$ . Note that no word is inputted into the stream.
For Gen at time  $t$ , FACTEDITOR does not pop the top embedding in the buffer. It feeds the
Draft text x	Bakewell_pudding is Dessert that can be served Warm or cold .
Revised text y	Bakewell_pudding is Dessert that originates from Derbyshire_Dales .
Action sequence a	Keep Keep Keep Keep Gen(originates) Gen(from) Gen(Derbyshire_Dales)
Drop Drop Drop Drop Keep
Table 4: An example of action sequence derived from a draft text and revised text.
combination of the context vector of triples  $\tilde{t}_t$  and the linearly projected embedding of word  $w$  into the stream, as shown in Fig. 1c. The state of stream is updated with the LSTM as  $s_{t + 1} =$  LSTM  $([\tilde{t}_t;W_p\pmb {y}_t],s_t)$  , where  $\pmb{y}_{t}$  is the embedding of the generated word  $y_{t}$  and  $W_{p}$  denotes parameters. In addition, FACTEDITOR copies the generated word  $y_{t}$  into the stream.
FACTEDITOR continues the actions until the buffer becomes empty.
Word generation
FACTEDITOR generates a word  $y_{t}$  at time  $t$ , when the action is Gen,
$$
P _ {\mathrm {g e n}} (y _ {t} \mid \boldsymbol {z} _ {t}) = \operatorname {s o f t m a x} (\boldsymbol {W} _ {y} \cdot \boldsymbol {z} _ {t})
$$
where  $W_{y}$  is parameters.
To avoid generation of OOV words, FACTEDITOR exploits the copy mechanism. It calculates the probability of copying the object of triple  $t_j$
$$
P _ {\mathrm {c o p y}} (o _ {j} \mid \boldsymbol {z} _ {t}) \propto \exp (\boldsymbol {v} _ {c} ^ {\top} \cdot \tanh (\boldsymbol {W} _ {c} \cdot [ \boldsymbol {z} _ {t}; \boldsymbol {t} _ {j} ]))
$$
where  $\mathbf{v}_c$  and  $W_{c}$  denote parameters, and  $o_j$  is the object of triple  $t_j$ . It also calculates the probability of gating
$$
p _ {\text {g a t e}} = \operatorname {s i g m o i d} \left(\boldsymbol {w} _ {\mathrm {g}} ^ {\top} \cdot \boldsymbol {z} _ {t} + b _ {\mathrm {g}}\right)
$$
where  $w_{\mathfrak{g}}$  and  $b_{\mathfrak{g}}$  are parameters. Finally, it calculates the probability of generating a word  $w_{t}$  through either generation or copying,