SlowGuess/ACL_2020_OCR huggingface dataset:
Yen-Chun Chen $^{1}$ , Zhe Gan $^{1}$ , Yu Cheng $^{1}$ , Jingzhou Liu $^{2}$ , Jingjing Liu $^{1}$
<sup>1</sup>Microsoft Dynamics 365 AI Research  
<sup>2</sup>Carnegie Mellon University
{yen-chun.chen, zhe.gan, yu.cheng, jinjl}@microsoft.com; liujingzhou@cs.cmu.edu
Large-scale pre-trained language model such as BERT has achieved great success in language understanding tasks. However, it remains an open question how to utilize BERT for language generation. In this paper, we present a novel approach, Conditional Masked Language Modeling (C-MLM), to enable the finetuning of BERT on target generation tasks. The finetuned BERT (teacher) is exploited as extra supervision to improve conventional Seq2Seq models (student) for better text generation performance. By leveraging BERT's idiosyncratic bidirectional nature, distilling knowledge learned in BERT can encourage auto-regressive Seq2Seq models to plan ahead, imposing global sequence-level supervision for coherent text generation. Experiments show that the proposed approach significantly outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization. Our proposed model also achieves new state of the art on IWSLT German-English and English-Vietnamese MT datasets.
Large-scale pre-trained language model, such as ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), has become the de facto first encoding step for many natural language processing (NLP) tasks. For example, BERT, pre-trained with deep bidirectional Transformer (Vaswani et al., 2017) via masked language modeling and next sentence prediction, has revolutionized the state of the art in many language understanding tasks, such as natural language inference (Bowman et al., 2015) and question answering (Rajpurkar et al., 2016).
However, beyond common practice of finetuning BERT for language understanding (Wang et al., 2019), applying BERT to language generation still remains an open question. Text generation aims to generate natural language sentences conditioned on certain input, with applications ranging from machine translation (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015), text summarization (Nallapati et al., 2016; Gehring et al., 2017; Chen and Bansal, 2018), to image captioning (Vinyals et al., 2015; Xu et al., 2015; Gan et al., 2017). In this work, we study how to use BERT for better text generation, which is still a relatively unexplored territory.
Intuitively, as BERT is learned with a generative objective via Masked Language Modeling (MLM) during the pre-training stage, a natural assumption is that this training objective should have learned essential, bidirectional, contextual knowledge that can help enhance text generation. Unfortunately, this MLM objective is not auto-regressive, which encumbers its direct application to auto-regressive text generation in practice.
We tackle this challenge by proposing a novel and generalizable approach to distilling knowledge learned in BERT for text generation tasks. We first propose a new Conditional Masked Language Modeling (C-MLM) task, inspired by MLM but requiring additional conditional input, which enables finetuning pre-trained BERT on a target dataset. In order to extract knowledge from the finetuned BERT and apply it to a text generation model, we leverage the finetuned BERT as a teacher model that generates sequences of word probability logits for the training samples, and treat the text generation model as a student network, which can effectively learn from the teacher's outputs for imitation. The proposed approach improves text generation by providing a good estimation on word probability distribution for each token in a sentence, consum
$^{1}$ Code is available at https://github.com/ChenRocks/Distill-BERT-Textgen.
7893
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7893-7905 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ing both the left and the right context, the exploitation of which encourages conventional text generation models to plan ahead. At inference time, the teacher model (BERT) is not required thus the decoding speed is as fast as the underlying student model.
Text generation models are usually trained via Maximum Likelihood Estimation (MLE), or teacher forcing (Bengio et al., 2015): at each time step, it maximizes the likelihood of the next word conditioned on its previous ground-truth words. This corresponds to optimizing one-step-ahead prediction. As there is no explicit signal towards global planning in the training objective, the generation model may incline to focusing on local structure rather than global coherence. With our proposed approach, BERT's looking into the future ability can act as an effective regularization method, capturing subtle long-term dependencies that ensure global coherence and in consequence boost model performance on text generation.
An alternative way to leverage BERT for text generation is to initialize the parameters of the encoder or decoder of Seq2Seq with pretrained BERT, and then finetuning on the target dataset. However, this approach requires the encoder/decoder to be identical to BERT, inevitably making the final text generation model too large. Our approach, on the other hand, is modular and compatible to any text-generation model, and has no restriction on model size or model architecture (e.g., LSTM or Transformer).
The main contributions of this work are three-fold: (i) We present a novel approach to utilizing BERT for text generation. The proposed method induces sequence-level knowledge into the conventional one-step-ahead and teacher-forcing training paradigm, by introducing an effective regularization term to MLE training loss. (ii) We conduct comprehensive evaluation on multiple text generation tasks, including machine translation and text summarization. Experiments show that our proposed approach significantly outperforms strong Transformer baselines and is generalizable to different tasks. (iii) The proposed model achieves new state of the art on both IWSLT14 German-English and IWSLT15 English-Vietnamese datasets.
Pre-trained Language Models Prior to large-scale pre-trained language model, word embed
dings (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017) were widely used for NLP tasks. Recently, CoVe (McCann et al., 2017) introduced (conditional) language models pre-trained on paired machine translation corpus. ELMo (Peters et al., 2018) learned a contextual language model on a large corpus with bidirectional RNN. GPT (Radford et al., 2018) used unidirectional Transformer to achieve better contextualized word representation. By fine-tuning pre-trained language models, ULMFit (Howard and Ruder, 2018) also achieved promising results on text classification.
In our study, we focus on BERT due to its superior performance on multiple language understanding tasks. However, different from previous work exploiting BERT for language understanding tasks, here we aim to apply BERT to text generation. To the best of our knowledge, this is still a relatively unexplored space. The proposed approach is also model-agnostic and can be applied to other pretrained language models as well.
BERT for Text Generation There has been some recent attempt on applying BERT to text generation. Specifically, Lample and Conneau (2019) trained cross-lingual MLM and demonstrated promising results for cross-lingual natural language inference (Conneau et al., 2018) and unsupervised neural machine translation (NMT) (Lample et al., 2018). Wang and Cho (2019) formulated BERT as a Markov Random Field LM and showed preliminary results on unsupervised text generation with improved diversity. Zhang et al. (2019a) utilized an encoder with BERT and a two-stage decoder for text summarization. Song et al. (2019) proposed Masked Seq2Seq (MASS) pre-training, demonstrating promising results on unsupervised NMT, text summarization and conversational response generation. Concurrent with our work, Ghazvininejad et al. (2019) proposed a similar conditional MLM for constant-time translation, and Yang et al. (2019) studied how to fine-tune BERT for NMT.
Our approach is novel in the sense that we do not directly use the parameters of BERT in the Seq2Seq model. Instead, BERT acts as an effective regularization to the MLE training loss, by proactively injecting future information for predicting the present.
Right-to-Left Generation Our work also shares a high-level intuition with those approaches that try to regularize left-to-right generative models with
Figure 1: Illustration of distilling knowledge from BERT for text generation. See Section 3.2 and 3.3 for details.
a right-to-left counterpart. Specifically, Liu et al. (2016) trained a separate reverse NMT and performed joint decoding at inference time to enforce agreement between forward and reverse models. Twin Networks (Serdyuk et al., 2018) used a backward RNN jointly trained with a forward RNN decoder by matching their hidden states. Zhang et al. (2019b) further extended the idea to Transformer with joint training, so that the forward and the backward models iteratively improve each other. Our proposed approach stems from a similar intuition. However, we focus on using pre-trained language model such as BERT to regularize an auto-regressive generation model.
Knowledge Distillation Our method shares the same loss formulation as Knowledge Distillation (KD) proposed in Bucilua et al. (2006); Hinton et al. (2015); Kim and Rush (2016), where a smaller student model is trained on soft labels provided by a larger teacher model. More recently, Tan et al. (2019) applied KD to multilingual NMT, and Sun et al. (2019) proposed patient KD for BERT model compression. Compared with these previous studies, where both the teacher and the student are trained on the same task, our approach is different in the sense that the BERT teacher is not designed to perform the student's generation task. We focus on using KD to leverage the learned knowledge in BERT for text generation, while previous work mostly focused on model compression.
In this section, we present our proposed approach to distilling the knowledge in BERT for text generation in generic sequence-to-sequence (Seq2Seq)
setting. We first review Seq2Seq learning in Section 3.1, and then describe the proposed approach in Section 3.2 and 3.3.
3.1 Sequence-to-Sequence Learning
Seq2Seq learning (Sutskever et al., 2014) aims to generate a sequence of discrete output  $Y = (y_{1},\ldots ,y_{N})$  of length  $N$ , conditioned on a sequence of discrete input  $X = (x_{1},\dots,x_{M})$  of length  $M$ . A Seq2Seq model learns parameters  $\theta$  to estimate the conditional likelihood  $P_{\theta}(Y|X)$ , typically trained via Maximum Likelihood Estimation (MLE), or equivalently, minimizing the cross-entropy loss:
$$
\begin{array}{l} \mathcal {L} _ {x e} (\theta) = - \log P _ {\theta} (Y | X) \tag {1} \\ = - \sum_ {t = 1} ^ {N} \log P _ {\theta} (y _ {t} | y _ {1: t - 1}, X), \\ \end{array}
$$
where each conditional probability can be calculated via an attention-based recurrent neural network (RNN) (Bahdanau et al., 2015; Luong et al., 2015), Transformer (Vaswani et al., 2017), or any other neural sequence-generation models.
3.2 Finetune BERT with Conditional MLM
This generic Seq2Seq learning framework is the state of the art on a wide range of text generation tasks. Using modern deep neural networks, the conditional probabilities can be readily modeled as a sequence of classifications over the word vocabulary. However, during training, in order to generate the  $t$ -th token  $y_{t}$ , the model only sees a partial sentence  $y_{1:t-1}$  from the ground-truth training data. Intuitively, it is reasonable to assume that a bidirectional model can be more informative than a left-
7895
to-right generation model, since additional context from the right (or future) is also incorporated to predict the current word. Unfortunately, this additional information is not utilized in a standard Seq2Seq model, since it can only be trained in a left-to-right manner, where the future context is masked out to prevent each word from indirectly "seeing itself". To compensate this single-directional limitation of Seq2Seq setting, we propose a new conditional language model (C-MLM) to enable the finetuning of BERT on target generation task, in hope that the finetuned bidirectional BERT can be utilized for better text generation.
BERT (Devlin et al., 2019) is a deep bidirectional Transformer trained via Masked Language Modeling (MLM). In a similar setting, where the input is a sequence pair  $(X,Y)$ ,  $15\%$  of the tokens are randomly masked. Formally, we denote the masked token sets as  $X^m$  and  $Y^m$ , and the disjoint counterpart (i.e., the unmasked tokens) as  $X^u$  and  $Y^u$ , respectively. The trained BERT model aims to estimate the joint probability:
$$
P \left(x _ {1} ^ {m}, \dots , x _ {i} ^ {m}, y _ {1} ^ {m}, \dots , y _ {j} ^ {m} \mid X ^ {u}, Y ^ {u}\right), \tag {2}
$$
where  $i$  and  $j$  denote the number of masked tokens in  $X$  and  $Y$ , respectively. Each  $x_{\star}^{m} \in X^{m}$ , and each  $y_{\star}^{m} \in Y^{m}$ . Eqn. (2) can be trained with the standard word-level cross-entropy loss.
We aim to marry MLM pre-training with Seq2Seq learning, to leverage bidirectional language model for text generation. To this end, we propose a conditional-MLM, a variant of MLM that allows further finetuning of pre-trained BERT on target dataset. For example, for machine translation,  $X$  and  $Y$  represent the source and the target sentence, respectively. We first concatenate them together and randomly mask  $15\%$  of the tokens only in  $Y$ , then train the network to model the joint probability:
$$
P \left(y _ {1} ^ {m}, \dots , y _ {j} ^ {m} \mid X, Y ^ {u}\right). \tag {3}
$$
The above C-MLM objective is similar to the conditional language modeling (LM) objective in Eqn. (1), but conditional LM only permits predicting a word based on its left context. C-MLM is also related to Masked Seq2Seq (MASS) pretraining (Song et al., 2019). However, in MASS,
the encoder takes a sentence with randomly masked fragment (several consecutive tokens) as input, and the decoder tries to predict this masked fragment, which is different from our model design. The final goal is also different: MASS focuses on Seq2Seq pre-training, while we focus on leveraging BERT for text generation. In our experiments, we observe that the C-MLM task can obtain high accuracy and good generalization on word prediction. However, it is not feasible to generate sequential output directly from C-MLM. Instead, we use knowledge distillation to distill the knowledge learned from the finetuned BERT into a Seq2Seq model for direct text generation, which will be explained in the next sub-section.
3.3 Knowledge Distillation for Generation
Our inspiration springs from the observation that the probability distribution of the masked word  $y_{t}^{m}$  is estimated using both  $y_{1:t-1}^{u}$  and  $y_{t+1:N}^{u}$  from  $Y^{u}$ . In other words, the distribution for a given word  $P(y_{t}^{m}|X,Y^{u})$  contains information from both backward and forward contexts, which is a desirable benefit for providing sequence-level global guidance. This probability distribution can be considered as soft targets for a text generation model to mimic from, which potentially contains more useful and fine-grained information than the usual hard-assigned, one-hot label, therefore enhancing conventional left-to-right generation models to look into the future.
In a knowledge distillation setting, the BERT model can be considered as a teacher, while the Seq2Seq model acts as a student. Specifically, the Seq2Seq model can be trained with the following objective function:
$$
\begin{array}{l} \mathcal {L} _ {b i d i} (\theta) = - \sum_ {w \in \mathcal {V}} \left[ P _ {\phi} \left(y _ {t} = w \mid Y ^ {u}, X\right) \cdot \right. \tag {4} \\ \left. \log P _ {\theta} (y _ {t} = w | y _ {1: t - 1}, X) \right], \\ \end{array}
$$
where  $P_{\phi}(y_t)$  is the soft target estimated by the finetuned BERT with learned parameters  $\phi$ , and  $\mathcal{V}$  denotes the output vocabulary. Note that  $\phi$  is fixed during the distillation process. An illustration of this learning process is provided in Figure 1, which aims to match the word probability distribution  $P_{\theta}(y_t)$  provided by the student with  $P_{\phi}(y_t)$  provided by the teacher (i.e., distillation).
To further improve the Seq2Seq student model, hard-assigned labels are also utilized. The final
2Besides MLM, Devlin et al. (2019) also introduced the next sentence prediction task for training BERT. We omit this task since it is unrelated to our work.
3The two sequences are consecutive paragraphs sampled from a very large corpus such as Wikipedia.
7896
model is trained with the following compound objective:
$$
\mathcal {L} (\theta) = \alpha \mathcal {L} _ {b i d i} (\theta) + (1 - \alpha) \mathcal {L} _ {x e} (\theta), \tag {5}
$$
where  $\alpha$  is a hyper-parameter for tuning the relative importance of the two training targets: soft estimation from finetuned BERT, and ground-truth hard label. Note that our proposed approach only has a minimal requirement on the architecture of the incorporated Seq2Seq model. As long as the model is trained to estimate word-level probability as in Eqn. (1), it can be trained jointly with the proposed objective function Eqn. (5).
At a higher level, the additional loss term  $\mathcal{L}_{bidi}$  can be interpreted as a sequence-level objective function. Our auto-regressive (or causal) model  $\theta$  tries to predict the probability distribution that matches the estimation the bidirectional teacher model predicts, hence encouraging the planning of future (right context) for generation.
In this section, we describe our experiments on two well-studied text generation tasks: machine translation, and abstractive text summarization.
Machine Translation We consider two relatively small-scale datasets, IWSLT15 English-Vietnamese (En-Vi, 113k training samples) and IWSLT14 German-English (De-En, 160k training samples), and one medium-scale dataset, WMT14 English-German (En-De, 4.5M training samples). For IWSLT15 En-Vi, we use the pre-processed dataset provided by Luong and Manning (2015). We use tst2012 as dev set and test on tst2013. For IWSLT14 De-En, we follow the pre-processing steps and the same train/dev/test split as in Wu et al. (2019). For WMT14 En-De, we follow the preprocessing steps in Vaswani et al. (2017) for fair comparison. We use newtest2013 as the dev set and newtest2014 as the test set. We report BLEU scores (Papineni et al., 2002) for evaluation of MT performance following the Moses script. $^{4}$
Abstractive Summarization For summarization, we conduct experiments on the Gigaword summarization dataset (Rush et al., 2015). Note that
the original train-valid/test split of Gigaword is  $3.8\mathrm{M} / 190\mathrm{k} / 2\mathrm{k}$ . In our experiments, we observed severe distribution mismatch between the validation and test data. See Table 4, 5, and Sec. 4.4 for detailed discussion. Therefore, we further sampled  $5\mathrm{k} / 5\mathrm{k}$  dev/test-dev splits from the validation set and tuned hyper-parameters on the dev set only. We report ROUGE scores (Lin, 2004) on test-dev for the evaluation of our proposed approach, and include results on the standard test split for the comparison with prior work.
Our implementation is based on the PyTorch (Paszke et al., 2017) version of OpenNMT (Klein et al., 2018) seq2seq toolkit. We use the 'base' model of 6-layer Transformer with 512-hidden 8-head attention blocks and 2048-hidden feed-forward layer for all experiments, with label smoothing regularization (LSR) (Szegedy et al., 2016) of  $0.1$ . We batch examples with similar sequence length, and count batch size by the number of tokens. For MT we use the pre-trained BERT-base-multilingual-cased model, and for summarization we use BERT-base-uncased as the starting point of BERT finetuning. We use the corresponding pre-trained byte-pair-encoding (Sennrich et al., 2016) shipped together with the BERT model for tokenization.
For all training methods of all Transformer models, the learning rate schedule is set to  $lr = \eta \cdot d_{model}^{-0.5} \cdot \min (step^{-0.5}, step \cdot \text{warmup\_steps}^{-1.5})$ , where  $d_{model} = 512$  is the attention representation size (Vaswani et al., 2017). For all BERT finetuning, we follow Devlin et al. (2019) and use a triangular learning rate schedule with maximum learning rate  $\eta$ . The parameters are updated with the Adam optimizer (Kingma and Ba, 2015). In the distillation stage, we pre-compute BERT's prediction logits of the training data and use top-  $K$  distillation (Tan et al., 2019) to reduce computation overhead and memory footprint, where  $K$  is set to 8 across all the experiments.
Our method can also be viewed as a 'learned LSR'. The results reported of our proposed method are trained together with regular LSR, showing the effectiveness of our teacher.
6BERT pre-trained models are available at https://github.com/google-research/bert. Our finetuning implementation is modified from code available at https://github.com/huggingface/pytorch-pretrained-BERT.
7The masking strategy is described in the supplementary. 8We also tune the temperature  $T$  for the softmax applied to the teacher's logits. Different from the original KD, we
4For fair comparison to previous work, we report tokenized BLEU scores using https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl, and for WMT14 En-De, we further split the compound words after tokenization.
De-En Models	dev	test
Our Implementations
Transformer (base)	35.27	34.09
+ BERT teacher	36.93	35.63
Other Reported Results
ConvS2S + MRT‡	33.91	32.85
Transformer (big)◇	-	34.4†
Lightweight Conv◇	-	34.8†
Dyn. Convolution◇	-	35.2†
Table 1: BLEU scores for IWSLT14 German-English translation.  $(\dagger)$  tuned with checkpoint averaging.  $(\ddagger)$  from Edunov et al. (2018).  $(\diamond)$  from Wu et al. (2019).
En-Vi Models	tst2012	tst2013
Our Implementations
RNN	23.37	26.80
+ BERT teacher	25.14	27.59
Transformer (base)	27.03	30.76
+ BERT teacher	27.85	31.51
Other Reported Results
RNN†	-	26.1
Seq2Seq-OT*	24.5	26.9
ELMo◇	-	29.3
CVT◇	-	29.6
For the detailed values of the hyper-parameters for each experiment, please refer to the supplementary material. We found it necessary to train longer with  $\mathcal{L}_{bidi}$ , since it is still improving after the step at which the baseline Transformer starts to plateau. At inference time, we use beam search with beam size 4 and length penalty (Wu et al., 2016) of 0.6 across all the models. All the hyper-parameters are tuned on the development set. Note that our Transformer baselines achieve higher scores than the reference implementation on each dataset (in most cases comparable to the state-of-the-art).
4.3 Results on Machine Translation
We first validate our proposed text generation approach on machine translation task. Experimental results are summarized in Table 1, 2 and 3, which show that our model significantly improves over the strong Transformer baseline across all three
Table 2: BLEU scores for IWSLT15 English-Vietnamese translation.  $(\dagger)$  from Luong et al. (2017).  $(\star)$  from Chen et al. (2019).  $(\diamond)$  from Clark et al. (2018).
En-De Models	NT2013	NT2014
Our Implementations
Transformer (base)	25.95	26.94
+ BERT teacher	26.22	27.53
Other Reported Results
Transformer (base)◇	25.8	27.3†
Transformer (big)*‡	26.5	29.3†
Dyn. Convolution●‡	26.9±0.2	29.7†
Table 3: BLEU scores for WMT14 English-German translation.  $(\dagger)$  tuned with checkpoint averaging.  $(\ddagger)$  trained on WMT16, a slightly different version of training data.  $(\diamond)$  from Vaswani et al. (2017).  $(\star)$  from Ott et al. (2018).  $(\bullet)$  from Wu et al. (2019).
datasets. Note that our baseline is the 'base' model of Transformer, which has 44M trainable parameters, and the reference implementation by Wu et al. (2019) of the 'big' model with 176M parameters. $^{9}$