SlowGuess/ACL_2020_OCR huggingface dataset:
Yichen Huang (黄溢辰) $^{12*}$ , Yizhe Zhang $^{1*}$ , Oussama Elachqar $^{1}$ , Yu Cheng $^{1}$
<sup>1</sup>Microsoft Corporation, Redmond, Washington 98052, USA
$^{2}$ Center for Theoretical Physics, MIT, Cambridge, Massachusetts 02139, USA
yichuang@mit.edu, {yizhang, ouelachq, yu.cheng}@microsoft.com
Missing sentence generation (or sentence infilling) fosters a wide range of applications in natural language generation, such as document auto-completion and meeting note expansion. This task asks the model to generate intermediate missing sentences that can syntactically and semantically bridge the surrounding context. Solving the sentence infilling task requires techniques in natural language processing ranging from understanding to discourse-level planning to generation. In this paper, we propose a framework to decouple the challenge and address these three aspects respectively, leveraging the power of existing large-scale pre-trained models such as BERT and GPT-2. We empirically demonstrate the effectiveness of our model in learning a sentence representation for generation and further generating a missing sentence that fits the context.
Generating a span of missing tokens in a text chunk, known as "text infilling," has attracted many attention recently (Zhu et al., 2019; Song et al., 2019; Liu et al., 2019; Ippolito et al., 2019; Joshi et al., 2020). Here we study the related but somewhat different task of "sentence infilling." Specifically, as illustrated in Figure 1, intermediate sentences (or chunks of text) are removed from long-form text (e.g., paragraphs, documents), and the task is to generate the missing pieces that can smoothly blend into and fit the context both syntactically and semantically. The generation can be either based only on context, or based on both context and side information such as constraint keywords. Compared with text infilling, sentence infilling requires the model to handle inter-sentential correlation and to reason about missing semantic information. Developing models for sentence infilling can potentially
Beautiful beachside boutique hotel with great views and modern decoration. My favorite part about this hotel is definitely the restaurant, UVA. I recently visited UVA to attend a friend's birthday party.
She was extremely happy with our hotel and we had a complimentary buffet.
The food was just phenomenal! I can't recall what everything was called, but we rolled out of there stuffed and happy. My husband had the rib eye dumpling as an appetizer and he said it was the best dumpling he has ever had.
Figure 1: Sentence infilling: generating an intermediate sentence that provides a smooth semantic transition from the preceding to the following context. This example is generated by our model on the TripAdvisor dataset. The colors mark the correspondence between the generated sentence and the context.
facilitate many text generation applications. Possible scenarios include, but are not limited to: document auto-completion by detecting and suggesting missing bridging sentences in the surrounding context; collaborative document writing by modifying and unifying different writing styles from multiple authors; meeting note expansion by extending a set of keywords (lexical constraints) to a full sentence, leveraging the surrounding context.
There are many challenges associated with this long-form sentence infilling task, which is typically a one-to-many problem in that the possible outputs can be diverse. As the generated sentence should connect separate text pieces in a syntactically and semantically smooth and coherent manner, the task requires a wide range of understanding, planning, and generation techniques. Large-scale pre-trained language models such as BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019) have dramatically enhanced the understanding and generation modules. However, how to in
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2502-2515 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
tegrate them in a holistic manner, and to analyze and establish the long-range dependence structure by high-level semantic planning is still challenging and yet to explore, as semantic appropriateness is usually subtler than syntactic appropriateness, which can be well characterized by autoregressive language models.
Several works have been done in this direction. MASS (Song et al., 2019) obtains sentence representations by predicting a span of missing tokens. It can be used to generate missing text, but the missing span length needs to be pre-specified. Other related works (Liu et al., 2019; Joshi et al., 2020) also require knowledge of the span length as an input to their models, and thus are different from our work. Text infilling (Zhu et al., 2019) sequentially generates tokens for the missing part of a sentence until an end-of-blank token is generated. Its generation can be of arbitrary length. By design, all these previous approaches operate at the token level, and thus arguably focus more on lexical appropriateness than the global semantics.
In this paper, we propose INter-SEntential Transformer (INSET), a novel approach to sentence infilling. Our model first produces sentence-level semantic features that capulate the missing high-level information. Then, grounded on the predicted semantic features, the model generates the syntactic and lexical features to embody the predicted sentence. Specifically, understanding, planning, and generation are handled by three modules in a synergistic manner:
- a BERT-based encoder to map each sentence to the latent semantic space.
- a sentence-level semantic planner to infer the missing information that can bridge the semantics of preceding and following context.
- a GPT-based generator (decoder) to map semantic features back to the text domain.
The main contributions and advantages of this work are summarized as follows:
- We study the task of sentence infilling, which requires the model to handle inter-sentential correlation and to predict missing semantic information. This goes beyond text infilling (Zhu et al., 2019), which asks the model to fill in the missing part of a single sentence.
- Our approach decouples understanding, planning, generation, and leverages existing large-scale pre-trained understanding and generation models (BERT, GPT-2). The components
of our model can be separately examined and improved with additional data.
- Our model predicts a feature vector in the latent semantic space for the missing sentence and maps the vector to text. Thus, it takes care of semantic smoothness and appropriateness.
- Our model allows the generation to be of arbitrary length, as in (Zhu et al., 2019).
- Compared with directly processing text, our approach significantly reduces computation time and memory usage during training, as (after pre-computing sentence features) the sequence length is the number of sentences rather than that of tokens.
Pre-Trained Language Model. Language models pre-trained on a large corpus improve natural language understanding and generation through transferable contextualized word representations (Devlin et al., 2019; Lample et al., 2019) and models (Howard and Ruder, 2018). Large transformer models (Vaswani et al., 2017) like GPT-2 (Radford et al., 2019), Megatron (https://github.com/NVIDIA/Megatron-LM), and T5 (Raffel et al., 2019) can achieve state-of-the-art results without training on any particular language modeling benchmark. (Keskar et al., 2019) proposes a conditional generation model, trained to condition on control codes that govern style, content, and other task-specific properties. Different from them, our model builds sentence representations via autoencoding with a pair of BERT and GPT-2.
Context-Aware Text Generation. There are some related works on context-aware text generation (Mikolov and Zweig, 2012; Tang et al., 2016; Mangrulkar et al., 2018). Most previous works on language modeling with contextual information (Wang and Cho, 2016; Wang et al., 2018; Sordoni et al., 2015b; Wen et al., 2015; Vinyals and Le, 2015) treat the preceding sentences as context. Compared with these sequential generation tasks, our task is constrained by bidirectional context, and is more challenging.
Text infilling (Zhu et al., 2019) aims at filling in the missing part, given the rest of a sentence. (Liu et al., 2019) proposes an iterative inference algorithm based on gradient search for text infilling. For story infilling, (Ippolito et al., 2019) first predicts rare words in the missing span, and then generates text conditioned on these words. SpanBERT (Joshi
et al., 2020) masks random contiguous spans and (pre-)trains a language model to predict tokens in the span. XL-Editor (Shih et al., 2019) adapts XLNet (Yang et al., 2019) to text infilling and other editing tasks.
(Kang and Hovy, 2019) models logic connections between sentences and generates intermediate sentences grounded on inter-sentential "flow." (Bhagavatula et al., 2020) formulates abductive commonsense reasoning as a natural language inference task to decide the appropriate reason that could explain the observation in one sentence given the background described by another sentence. (Cheng et al., 2020) proposes a text style transfer task to translate a sentence in the context of a paragraph into the desired style. These three works study generation tasks that address inter-sentential relationship, and thus may be conceptually related to our motivation.
Compared with (Zhu et al., 2019; Liu et al., 2019; Ippolito et al., 2019; Joshi et al., 2020; Shih et al., 2019; Kang and Hovy, 2019; Bhagavatula et al., 2020; Cheng et al., 2020), our approach is clearly different. We fully exploit existing large-scale pretrained models BERT and GPT-2 to learn smooth sentence embeddings in the latent semantic space, and then process sentence-level information in this space.
Hierarchical Text Generation. Hierarchical text generation with high-level semantic planning has been studied in many previous works. (Sordoni et al., 2015a) presents a hierarchical recurrent encoder-decoder architecture for context-aware query suggestion. (Zhang et al., 2019) proposes a framework to infer semantic features for response generation using self-supervised learning. Previous works have used multi-level LSTM encoders (Yang et al., 2016; Hu et al., 2020) and hierarchical autoencoders (Li et al., 2015) to learn hierarchical representations for long text. (Shen et al., 2019) uses a variational autoencoder to encode an entire paragraph into a single latent variable, from which the paragraph can be generated hierarchically. In comparison, our task is to generate intermediate sentences in the surrounding context.
The task of sentence infilling is formally defined as follows. Consider a dataset of  $N$
paragraphs  $\{p^{(k)}\}_{k = 1}^{N}$ . Each paragraph  $p^{(k)} = (s_1^{(k)}, s_2^{(k)}, \ldots, s_{M_k}^{(k)})$  consists of  $M_k$  consecutive sentences. For each  $k$ , we are given a positive integer  $m_k \leq M_k$  and the context  $(s_1^{(k)}, s_2^{(k)}, \ldots, s_{m_k - 1}^{(k)}, s_{m_k + 1}^{(k)}, \ldots, s_{M_k}^{(k)})$ , but the  $m_k$ 'th sentence  $s_{m_k}^{(k)}$  is missing. The task is to generate a sentence  $\hat{s}_{m_k}^{(k)}$  in the missing position such that it fits the context. For simplicity and without any confusion, we drop the index  $k$  from now on (note that  $M$  and  $m$  may depend on  $k$ ).
The criteria for successful generation are:
- The sentence  $\hat{s}_m$  is fluent and meaningful.
- Inserting the generated sentence into the context, we obtain a semantically coherent paragraph  $(s_1, s_2, \ldots, s_{m-1}, \hat{s}_m, s_{m+1}, \ldots, s_M)$ .
-  $\hat{s}_m$  is written in the same style as contextual sentences  $\{s_j\}_{j \neq m}$ .
Since there could be multiple semantically different sentences that fit the same context well, it is not necessary for  $\hat{s}_m$  to be close to the ground truth  $s_m$ . Rather,  $\hat{s}_m$  is considered successful as long as it satisfies the criteria above.
3.2 INSET: Inter-Sentential Transformer
Model Overview. At a high level, our model consists of two components: a (denoising) autoencoder and a sentence-level transformer. The former maps each sentence to a fixed-length feature vector in the latent semantic space, and reconstructs the sentence from the representation. The latter predicts the semantic features of the missing sentence from those of contextual sentences. We call our model INter-SEntential Transformer (INSET).
Formally, let  $(\mathcal{E},\mathcal{D})$  be an autoencoder, where  $\mathcal{E}$ $(\mathcal{D})$  is the encoder (decoder) such that  $\mathcal{E} \circ \mathcal{D}$  and  $\mathcal{D} \circ \mathcal{E}$  are supposed to be identity maps. Let  $\mathcal{T}$  be a sentence-level transformer with positional encoding  $\mathcal{P}$ . The transformer  $\mathcal{T}$  takes the contextual information as input and outputs a hypothetical representation of the missing sentence. Specifically,
$$
\begin{array}{l} \hat {s} _ {m} = \mathcal {D} \big (\mathcal {T} (f _ {1} + \mathcal {P} (1), f _ {2} + \mathcal {P} (2), \dots , \\ f _ {m - 1} + \mathcal {P} (m - 1), \vec {0} + \mathcal {P} (m), \\ \left. f _ {m + 1} + \mathcal {P} (m + 1), \dots , f _ {M} + \mathcal {P} (M)\right) [ m ]\bigg), \tag {1} \\ \end{array}
$$
where  $f_{j} = \mathcal{E}s_{j}$  is the encoding of the sentence  $s_j$ ,  $\vec{0}$  is the zero vector representing the missing sentence, and  $\mathcal{T}(\dots)[m]$  is output of the transformer  $\mathcal{T}$  in the missing position  $m$ .
The autoencoder and the sentence-level transformer can be trained separately. We first train the
Figure 2: Model overview. Left panel: Denoising autoencoder. The encoder  $\mathcal{E}$  takes a corrupted sentence (with each token  $w_{i}$  for  $i = 1,2,\dots,l$  masked randomly) as input and outputs a representation of the sentence. The decoder  $\mathcal{D}$  should reconstruct the original uncorrupted sentence. The training parameters of  $\mathcal{E}$  and  $\mathcal{D}$  are initialized with those of BERT and GPT-2, respectively. Right panel: Sentence-level transformer. Using the encoder  $\mathcal{E}$ , we obtain the representation of every contextual sentence. These sentence representations are fed into a sentence-level transformer  $\mathcal{T}$ , which outputs a representation of the missing sentence.
former on individual sentences. Then, we precompute and save the feature vectors of all sentences. While training the latter, it is not necessary to load the former. This makes training more efficient.
Sentence Representation Learning via Denoising Autoencoding. Large-scale pre-training approaches (e.g., BERT) lead to superior performance in many language understanding tasks related to sentence representation learning (Reimers and Gurevych, 2019). However, the features learned by BERT (or fine-tuned on downstream tasks) cannot be directly used for generation tasks, as the masked language model objective of BERT does not enforce the reconstruction of the original sentence from the extracted features. Instead of directly using BERT features, we learn sentence representations via autoencoding. This naturally integrates BERT and GPT-2, and combines sentence representation learning and generation.
As shown in the left panel of Figure 2, we pad the [CLS] token at the beginning of each sentence  $s_j$ . We initialize the encoder  $\mathcal{E}$  with BERT, and extract the output  $f_j$  corresponding to the [CLS] token as the embedding of  $s_j$ . We initialize the decoder  $\mathcal{D}$  with GPT-2, and feed  $f_j$  as the embedding of the zeroth token. Then, we have  $\mathcal{D}$  generate a sequence of tokens in the hope that the sequence matches  $s_j$  (padded with special tokens [SOS] at the beginning and [EOS] at the end). To train the autoencoder, we use teacher forcing and minimize the negative log-likelihood loss by (fine-)tuning the parameters of  $\mathcal{E}$  and  $\mathcal{D}$  jointly.
An autoencoder embeds sentences into vectors in the latent space. We hope that the embedding is smooth in the sense that semantically similar
sentences are mapped to vectors that are close to each other. In particular, interpolation between two points in the latent space should correspond to a smooth semantic transition in the text domain. To this end, we use the following two tricks.
First, we employ a denoising autoencoder, which is known to yield a smoother embedding (Vincent et al., 2008). To add noise, we randomly mask each token in  $s_j$  with probability  $15\%$  by replacing the masked tokens with a special token [MASK]. During training, we use the "noisy"  $s_j$  with masks as input to the encoder, and use the "clean"  $s_j$  without masks to compute the loss function. Of course, one could try more sophisticated noise-adding strategies (Lewis et al., 2019).
Second, we use early stopping. In our experiments, we observe that as training proceeds, the validation loss of the autoencoder keeps decreasing. In the absence of masks, presumably it would eventually decay to zero so that the autoencoder perfectly reconstructs every sentence. However, this does not necessarily imply that the embedding is smooth. On the contrary, an overtrained autoencoder often tries to remember every individual token and thus fails to achieve smoothness in the latent semantic space. Moreover, it can catastrophically forget some of the information in the initial pre-trained model (GPT-2) and partially lose the power of generating fluent sentences. In practice, we select a checkpoint by monitoring its validation performance on sentence interpolation. Some examples of sentence interpolation are shown in Table 1.
Sentence Feature Prediction. After encoding sentences into feature vectors, we use a sentence
level transformer  $\mathcal{T}$  to predict the feature vector of the missing sentence from those of contextual sentences. This is analogous to the task of predicting masked tokens for (pre-)training BERT (Devlin et al., 2019), but now it is at the sentence level. Indeed, sentence feature vectors in  $\mathcal{T}$  correspond to token embeddings in BERT, and sentence position ID in  $\mathcal{T}$  corresponds to position ID in BERT.
We train the transformer  $\mathcal{T}$  with the objective
$$
\mathcal {L} _ {\mathrm {S e n t T r a n s}} = 1 - \cos (f _ {m}, \mathcal {T} (\dots) [ m ]), \qquad (2)
$$
where  $\cos (\dots)$  is the cosine similarity between the ground truth sentence feature vector  $f_{m}$  and the prediction  $\mathcal{T}(\cdot \cdot \cdot)[m]$  in Eq. (1). Note that  $\cos (\cdot \cdot \cdot)$  is a good similarity measure only when its arguments are unit vectors. This is guaranteed by the technical trick of fixing the parameters of the last LayerNorm of the transformers  $\mathcal{E}$  and  $\mathcal{T}$ , i.e., do not compute the gradients of these parameters in backpropagation.
Generating Sentences from Features. At test time, we use the decoder  $\mathcal{D}$  to generate the missing sentence by mapping the predicted feature vector to the text domain. Note that standard generation schemes such as top- $k$  sampling, beam search, and nucleus sampling (Holtzman et al., 2020) can be used without additional modeling effort.
Computational Efficiency. Compared with vanilla GPT-2, our model can process and analyze a document containing many sentences at the discourse level with dramatically lower time and space complexity. To estimate quantitatively, suppose that a document contains  $N_{s}$  sentences, each of which has  $N_{t}$  tokens. Then, the time complexity is reduced from  $\mathcal{O}(N_s^2 N_t^2)$  to  $\mathcal{O}(N_s^2 + N_s N_t^2)$ . Moreover, sentence features can be precomputed once and then reused for every epoch or even in other tasks on the same dataset. If sentence features have been precomputed and are already directly available, the time complexity is further reduced to  $\mathcal{O}(N_s^2)$ .
3.3 Sentence Infilling with Lexical Constraints
We further introduce a related task called sentence infilling with lexical constraints, which is the same as sentence infilling except that now we are given some keywords of the missing sentence as an additional input to hint the generation. The keywords are treated as soft constraints (aka priming): The
generated sentence is not directly enforced to contain the exact keywords. It may contain a synonym or share some semantics with the keywords.
We expect that the presence of keyword constraints makes the task more difficult rather than easier, although incorporating keywords can significantly improve the BLEU score of the generation with respect to the ground truth. Intuitively, keywords force the model to speculate the semantics of the ground truth sentence, and significantly reduce the number of possible solutions. In the absence of keywords, the model has the freedom of completing the task according to its own way of thinking.
To handle keyword constraints, we introduce a new component called the constraint feature encoder to our architecture. It is a transformer encoder  $\mathcal{K}$  that maps a set  $S$  of keywords to a feature vector that lives in the same latent space of sentence embeddings. We train  $\mathcal{K}$  with knowledge distillation (Kim and Rush, 2016). The teacher model is the sentence encoder  $\mathcal{E}$ , which maps a sentence containing the keywords in  $S$  to a feature vector. We use the cosine similarity loss between these two feature vectors to teach the student model  $\mathcal{K}$ .
For implementation details, suppose we have two keywords  $w_{1}$  and  $w_{2}$ . Then, the input to  $\mathcal{K}$  is three tokens ([CLS],  $w_{1}, w_{2}$ ). We replace the zero vector in Eq. (1), which represents the missing sentence, with the output of  $\mathcal{K}$  above the [CLS] token. We do not use positional encoding in  $\mathcal{K}$  because keywords do not have order.
We evaluate our model on two datasets (TripAdvisor and Recipe). We have released the source code to facilitate future research (https://github.com/dreasynail/INSET).
Dataset and Preprocessing. We conduct experiments on the TripAdvisor and Recipe datasets. For the TripAdvisor dataset of hotel reviews (Wang et al., 2010), we partially follow the preprocessing in (Cho et al., 2019). Our preprocessing includes, but is not limited to: (i) discarding reviews containing non-English tokens; (ii) removing duplicate reviews so that only one copy is retained. We set the maximum number of tokens in a sentence to be 32 and the minimum number of sentences in a review to be 7 (so that the context is not too short). Any review with longer sentences or having
a smaller number of sentences is discarded.
We use the following strategy to mask sentences. For a paragraph consisting of  $M \geq 7$  consecutive sentences, we split it into  $M - 6$  data points, each of which has exactly 7 sentences. The  $j$ 'th data point spans from the  $j$ 'th to the  $(j + 6)$ 'th sentence (inclusive) of the paragraph, for  $j = 1, 2, \ldots, M - 6$ . We mask the middle (i.e., 4th) sentence for each data point so that the masking rate is  $1/7 \approx 14.3\%$ , which is close to that  $(15\%)$  of BERT. After preprocessing, the size of the dataset (training, validation, test) is (1108134, 62543, 533) data points.
Our strategy of always masking the middle sentence out of 7 sentences is not only the simplest but also without loss of generality. Our model is directly applicable to the situation where we randomly mask, e.g., 3 out of 20 sentences. However, the quality of human evaluation may be affected because the patience and attention of human evaluators may decrease as the context length increases. For the effectiveness of human evaluation, we use the simplest strategy to mask sentences.