SlowGuess/ACL_2020_OCR huggingface dataset:
The Recipe dataset is obtained from (https://commoncrawl.org), where the metadata is formatted according to Schema.org (https:// schema.org/Recipe). We use the same preprocessing as that of the TripAdvisor dataset except that instructions with less than 5 sentences are discarded. After preprocessing, the size of the dataset (training, validation, test) is (1073886, 56055, 500) data points. Recipe instructions usually describe a time-ordered procedure, and thus are ideal for testing the reasoning capability of the model.
Evaluation Metrics. Following (Galley et al., 2019; Zhang et al., 2020), we perform automatic evaluation using standard machine translation metrics, including BLEU (Papineni et al., 2002), NIST (Doddington, 2002), and METEOR (Lavie and Agarwal, 2007). As a variant of BLEU, NIST weights  $n$ -gram matches by their information gain, and thus penalizes uninformative  $n$ -grams. We also use Entropy (Zhang et al., 2018) and Dist- $n$  (Li et al., 2016) to evaluate lexical diversity. See (Galley et al., 2019) for more details.
BLEU, NIST, and METEOR measure the similarity between the generated sentence and the ground truth. They are not ideal scores for our task because a sentence that is semantically very different from the ground truth could possibly fit the context perfectly. However, it may still be helpful to compute these commonly used scores. It
is an important and challenging open problem to design an automatic score that faithfully measures the overall quality of the generation in our task.
Baseline. Our baseline is the self-attention model for text infilling (Zhu et al., 2019). It is a transformer language model with novel positional encoding. The traditional approach of encoding the absolute position of each token is not directly applicable to our task because we do not know in advance the absolute positions of contextual tokens after the missing sentence. To resolve this issue, (Zhu et al., 2019) divides the text into segments. In the case of only one masked sentence, the first (third) segment consists of contextual tokens before (after) the mask, and the second corresponds to the mask. Then, each token is indexed by its segment ID and its position ID within the segment. The missing tokens are sequentially generated from these IDs and the current surrounding context.
Training the baseline model on our dataset, we use the same set of hyperparameters as in the original reference except that the batch size is set to 250 (it is 400 in (Zhu et al., 2019)). This avoids out-of-memory errors. Note that we are handling much longer sequences (usually  $>100$  tokens) than (Zhu et al., 2019), in which the maximum number of tokens in a sequence is only 16.
The baseline model is trained for a sufficient number (30) of epochs until the validation (negative log-likelihood) loss and perplexity clearly saturate. We report the results of the checkpoint with the smallest validation loss and perplexity. Note that we observe that other checkpoints in the saturation regime behave very similarly on the test set.
Keyword Extraction. In the task of sentence infilling with lexical constraints, we need to extract keywords from the masked sentence. Keyword extraction is a classical problem in information retrieval. Standard methods include, but are not limited to, tfidf (term frequency-inverse document frequency) (Ramos, 2003). We have tried tfidf, but it does not work well for the TripAdvisor dataset of hotel reviews. One reason is that this dataset has quite a few typos, and unfortunately tfidf favors them because typos occur less frequently than normal words. This issue can be resolved by manually filtering out all typos. After the fix, however, we observe that the quality of extracted keywords remains unsatisfactory.
We use the following strategy to extract key-
words. We first define a list of stop words. To this end, we use the stop word list from NLTK (Bird et al., 2009) and manually add a number of words (e.g., "hotel") that are not very informative for the particular dataset of hotel reviews. For each sentence, we select non-stop words that appear most frequently in the entire dataset. We usually select two keywords per sentence, but occasionally select one or even zero if few words remain after filtering out stop words and typos. We observe that the keywords extracted with this strategy can pivot the gist of most sentences well.
Model Size and Hyperparameters. Our architecture has several components. The encoder  $\mathcal{E}$  and the sentence-level transformer  $\mathcal{T}$  have the same size as BERT<sub>BASE</sub>. The decoder  $\mathcal{D}$  has the same size as GPT-2 (117M). In the presence of lexical constraints, the constraint feature encoder  $\mathcal{K}$  has the same size as BERT<sub>BASE</sub>. During decoding, we use beam search with beam size 5.
Sentence Representation Learning. We first qualitatively evaluate the smoothness of the latent-space sentence embeddings learned via denoising autoencoding. Table 1 shows two examples of sentence interpolation on the TripAdvisor dataset. In each example, the first and last sentences are inputs by hand, and the 3 intermediate ones are interpolations generated by our (denoising) autoencoder. We observe that the interpolations not only combine words from input sentences, but are readable, meaningful, and show a smooth semantic transition from the first to the last sentence. We speculate that the power of generating fluent and semantically coherent sentence interpolations is derived from BERT and GPT-2. Inherited from these large-scale pre-trained models, the latent-space sentence embedding is reasonably smooth as our sentence interpolation results show.
Automatic Evaluation. Table 2 shows the BLEU, NIST, METEOR, Entropy, Dist- $n$  scores, and the average length (number of words) of the generated sentences. For the TripAdvisor dataset, we also present results in the presence of keyword constrains.
Table 2 compares the baseline (Zhu et al., 2019), our results, and the ground truth. In the absence of keyword constraints, INSET outperforms the baseline in terms of all scores on both datasets. This indicates that our results are semantically closer
example 1
A	The pool area was nice and sunbathing was great.
-	The pool area was nice and staff was great.
-	The pool area staff was nice and very helpful.
-	Front desk staff were very helpful and friendly.
B	Front desk staff were very nice and helpful.
example 2
A	The service was attentive and we had the best food in town.
-	The service was attentive and we had a great room with plenty of food.
-	The room was spacious with good service and we had a queen bed.
-	The room was very spacious with queen beds.
B	The room was very spacious with 2 queen beds.
Table 1: Sentence interpolation. "A" and "B" are two sentences in the test set. The intermediate sentences are generated by interpolating between the latent-space representations of A and B.
to the ground truth and are more diverse than the baseline. In terms of the average generation length, our results are much closer to the ground truth than the baseline is.
Table 2 also presents two ablation studies. The first shows the performance decrease with less context. Recall that each data point in the TripAdvisor dataset has 6 contextual sentences (full context). We train INSET on the same set of data points but truncate the context to 4 sentences (less context). The second ablation study shows the effect of context in the presence of keywords. We compare two models. The first (INSET w/ context) is the model described in Subsection 3.3. Its generation is based on both keywords and context. The second model (INSET w/o context) is  $\mathcal{D} \circ \mathcal{K}$ , which directly decodes the output of the constraint feature encoder  $\mathcal{K}$  using the decoder  $\mathcal{D}$ . Its generation is only based on keywords but not context. We observe that the scores of the first model are higher than those of the second. Both ablation studies show that our model can make full use of context to improve the generation.
Human Evaluation. We performed human evaluation of our method on the TripAdvisor dataset. We used a crowd evaluation platform to compare two systems and assess their fluency, informativeness, and relevance to the surrounding context (coherence) on 500 random samples from the test set. Following recommended best practices, each sample was evaluated by five judges. We performed simple spam detection by excluding judges that were too fast or performed too low on a gold set. To avoid bias, we randomized the position of each system while asking judges to compare our systems (with and without keywords) with the ground truth
Dataset	Method	NIST	BLEU	MET- EOR	Ent. E-4	Dist	Len.
N-2	N-4	B-2	B-4	D-1	D-2
Trip	Without keyword constraints:
baseline	0.54	0.54	4.29%	0.54%	5.85%	3.10	1.32%	2.23%	6.97
INSET (full context)	1.23	1.23	6.08%	0.96%	7.04%	8.13	16.30%	46.64%	10.70
INSET (less context)	1.02	1.02	4.74%	0.51%	5.83%	7.85	12.98%	41.39%	11.26
With keyword constraints:
INSET (w/ context)	3.09	3.15	20.14%	6.57%	16.48%	8.34	22.61%	63.60%	11.23
INSET (w/o context)	3.00	3.04	19.47%	6.07%	16.00%	8.16	20.51%	57.41%	11.12
ground truth (human)	-	-	-	-	-	8.40	33.96%	79.84%	11.36
Recipe	baseline	0.67	0.68	3.91%	0.88%	5.23%	3.12	0.37%	0.47%	15.32
INSET (ours)	1.36	1.37	7.24%	1.33%	7.07%	7.99	20.12%	55.13%	9.63
ground truth (human)	-	-	-	-	-	8.22	29.21%	74.97%	10.55
Table 2: Automatic evaluation. "w/ context" indicates that the generation is based on both keywords and context. "w/o context" indicates that the generation is only based on keywords but not context. "Ent." and "Len." stand for Entropy and the average generation length, respectively.
system A	system B	criterion	prefer A (%)	same (%)	prefer B (%)
INSET (ours)	baseline	coherence	54.16	13.76	32.07
fluency	43.38	26.98	29.64
informativeness	53.48	18.79	27.72
INSET (ours)	ground truth	coherence	27.87	15.69	56.44
fluency	21.78	31.38	46.84
informativeness	27.49	21.92	50.59
INSET w/ keywords	ground truth	coherence	18.50	23.45	58.04
fluency	17.82	29.78	52.39
informativeness	20.54	26.13	53.33
INSET w/ keywords	INSET w/ keywords	coherence	37.71	37.62	24.68
fluency	36.16	37.87	25.97
informativeness	35.93	39.86	24.21
INSET w/ keywords	INSET w/o keywords	coherence	34.97	17.06	47.97
fluency	29.30	28.04	42.65
informativeness	31.73	23.24	45.03
Table 3: Human evaluation. "w/(w/o) keywords" and "w/(w/o) context" indicate whether the generation is based on keywords and context, respectively. All numbers are percentages.
and the text infilling baseline (Zhu et al., 2019).
Table 3 shows the human evaluation results. The judges strongly prefer our results (without keywords) to the baseline in all aspects: coherence, fluency, and informativeness. They also strongly prefer the ground truth to our results. Moreover, our results with keywords and context are compared with three other systems: (i) the ground truth; (ii) our results with keywords but not context; (iii) our results with context but not keywords. The second comparison shows that in the presence of keywords, our model can use context to improve all aspects of the generation. The third comparison shows that the presence of keywords reduces the performance of our model, probably because keywords are constraints that the model must take care of.
Generated Examples. To qualitatively demonstrate the effectiveness of our model, Table 4 shows some examples from the TripAdvisor and Recipe datasets. We observe that the baseline (Zhu et al., 2019) tends to generate generic sentences, while our results (either with or without keywords) are more informative and can fit the surrounding context reasonably well. Table 5 shows examples generated by our model in the same context but with different keywords. Our model can extend keywords to a full sentence, adapting to the context. More examples generated by our model on both datasets are given in Appendix A.
5 Conclusions and Outlook
We study the task of sentence infilling, which is analogous to the masked language modeling task for (pre-)training BERT, but now it is at the sen
example from the TripAdvisor dataset	example from the TripAdvisor dataset	example from the Recipe dataset
preceding context	It was such a pleasure to see something new every night. It was not very crowded so we were able to get great seats at either the pool or the beach. The VIP service was great for dinner reservations and pillow service.	The walls are very thin. Since this is a family vacation type of hotel, people are up at the pool/bbg area/hallways during all hours of the night. Do not stay here if you need a quite night of sleep.	After another 15 minutes or so the mixture should thicken up. The mixture will continue to thicken as it cools.
following context	Enjoyed the shrimp cocktail and seafood salad delivered to us while enjoying the pool. All of us would not want to stay at another resort and are planning to go back again. Enjoy and Hola!Karen and FriendsMilford, CT	You have to take multiple elevators to go all the way to the 5th floor. My other complaint is that the hotel staff seemed a bit un-professional. Not what I'm used to when I stay at Marriott properties.	Sterilize your jars and lids and while still hot fill with the jam leaving about a 1/2 inch headspace. Place lids onto the jars and boil in a water bath with jars covered by 3 inches of water for 10 minutes.
ground truth	We did bring a lot of $1 for tipping and of course the service stepped up a notch more.	Also, the elevator situation is weird.	Remove from the heat and stir in your amaretto.
baseline	The staff was friendly and helpful.	The rooms are very clean and well kept.	Add the flour mixture to the dry ingredients and mix well.
INSET	The buffet dinner was amazing and we had the best food in the resort.	There is only one elevator block in the hotel.	Carefully remove the jars from hot water and keep going until a thick sauce is formed.
+ keywords	$, service	elevator, situation	-
INSET (w/keywords)	Service fee for the buffet dinner was $5.00 and we paid $5.00 extra for food service.	The elevator situation is extremely frustrating.	-
Table 4: Examples generated by our model and the baseline.
preceding context	My room was a very good size. Tiled floors and woodchip painted walls. The tv did not work - so what.
following context	Great places to eat close by and very reasonable. No air con -so summer could be sticky. My concern is the left luggage room not supervised.
human oracle	The location is terrific beside Sevilla metro stn so only 2 to get by metro all the way to airport.
+ (walk, shopping)	Walking distance to shopping mall and Circular Quay.
+ (internet, $)	Internet cost $20.00 per day.
Table 5: Examples generated by our model in the same context but with different keywords. “+ (…)” is keywords.
tence level. Sentence infilling requires the model to handle long-range inter-sentential correlation and to process high-level semantic information. It is complementary to (token-level) masked language modeling, which focuses more on syntactic appropriateness and short-range correlation. We propose a framework called INSET to decouple three aspects of the task (understanding, planning, and generation) and address them in a unified manner. We demonstrate the effectiveness of our approach using automatic and human evaluation.
Our approach can be modified or extended in some ways. (i) We use a denoising autoencoder to obtain sentence embeddings. One can try to use a variational autoencoder (Kingma and Welling, 2014) instead. A large-scale pre-trained variational autoencoder (Li et al., 2020) could possibly improve the smoothness of sentence embeddings. (ii) Our model predicts a feature vector for the missing sentence. This vector can be fed into and serve as a guide to token-level models including the baseline (Zhu et al., 2019).
Since sentence infilling is analogous to masked language modeling, we expect that it can also be used as a pre-training task. For example, in ma
chine translation of long texts, it is often the case that sentences are translated independently from each other. This sometimes leads to incoherence or even inconsistency between the translated sentences. A post-editor to fix the issue (Voita et al., 2019) should be able to understand inter-sentential relationship and to generate fluent sentences in the surrounding context, both of which can be learned from sentence infilling.
Note. After this paper was posted on arXiv, some related works appeared. (Shen et al., 2020) proposes Blank Language Model for text infilling and other tasks. (Donahue et al., 2020) trains (finetunes) a language model (GPT-2) for text and sentence infilling. (Li et al., 2020) pre-trains a large-scale variational autoencoder with a pair of BERT and GPT-2. (Ippolito et al., 2020) uses a sentence-level language model, which operates on sentence embeddings obtained from BERT, to predict story endings.
We thank Bill Dolan, Chris Quirk, and Jingjing Liu for helpful discussions and suggestions.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In International Conference on Learning Representations.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O'Reilly Media Inc.
Yu Cheng, Zhe Gan, Yizhe Zhang, Ouussama Elachqar, Dianqi Li, and Jingjing Liu. 2020. Contextual text style transfer. arXiv:2005.00136.
Woon Sang Cho, Pengchuan Zhang, Yizhe Zhang, Xiujun Li, Michel Galley, Chris Brockett, Mengdi Wang, and Jianfeng Gao. 2019. Towards coherent and cohesive long-form text generation. In Proceedings of the First Workshop on Narrative Understanding, pages 1-11, Minneapolis, Minnesota. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram cooccurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research, HLT '02, pages 138-145, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Chris Donahue, Mina Lee, and Percy Liang. 2020. *Enabling language models to fill in the blanks*. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*. Association for Computational Linguistics.
Michel Galley, Chris Brockett, Xiang Gao, Jianfeng Gao, and Bill Dolan. 2019. Grounded response generation task at DSTC7. http://workshop.colips.org/dstc7/papers/DSTC7_Task_2_overview_paper.pdf.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Junjie Hu, Yu Cheng, Zhe Gan, Jingjing Liu, Jianfeng Gao, and Graham Neubig. 2020. What makes a good story? Designing composite rewards for visual storytelling. In AAAI Conference on Artificial Intelligence.
Daphne Ippolito, David Grangier, Chris Callison-Burch, and Douglas Eck. 2019. Unsupervised hierarchical story infilling. In Proceedings of the First Workshop on Narrative Understanding, pages 37-43, Minneapolis, Minnesota. Association for Computational Linguistics.
Daphne Ippolito, David Grangier, Douglas Eck, and Chris Callison-Burch. 2020. Toward better storylines with sentence-level language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Dongyeop Kang and Eduard Hovy. 2019. Linguistic versus latent relations for modeling coherent flow in paragraphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5809-5815, Hong Kong, China. Association for Computational Linguistics.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. arXiv:1909.05858.
Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327, Austin, Texas. Association for Computational Linguistics.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In International Conference on Learning Representations.
Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herve Jegou. 2019. Large memory layers with product keys. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8548-8559. Curran Associates, Inc.
Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228-231, Prague, Czech Republic. Association for Computational Linguistics.