Dynamic and Efficient Inference for Text Generation via BERT Family
Year: 2023

Abstract:
Despite the excellent performance of Pretrained Language Models on many text generation tasks, they suffer from inefficient inference on computation and memory due to their largescale parameters and the universal autoregressive decoding paradigm. In this work, we propose a novel fine-tuning method DEER, which can make a single pre-trained model support Dynamic and Efficient infERence and achieve an adaptive trade-off between model performance and latency. In particular, our critical insight is to jointly utilize the non-autoregressive (NAR) generation and dynamic parameter pruning techniques, which can flexibly control the decoding iteration steps and model sizes according to memory and latency limitations. Besides, we also explore the effectiveness of the pre-trained MLMs (i.e., the BERT family) for text generation tasks since their bidirectional attention nature is more suitable for the NAR training objective. Extensive experiments on both monolingual and multilingual pre-trained MLMs demonstrate the effectiveness of our proposed DEER method by consistently achieving (1) higher BLEU scores than the strong autoregressive Transformer model on three neural machine translation tasks with 3 → 12 times speedup, (2) competitive performance (but with much faster inference speed) compared with the BART model on four GLGE benchmark tasks. Our code will be publicly available at GitHub1.

heading: 1 Introduction
text: Large-scale pre-trained language models (Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020; Chowdhery et al., 2022) have shown great potential in achieving impressive performance; however, they are accompanied by substantial computational complexities and occupy significant memory space. These factors pose obstacles to their practical implementation in real-world applications.
∗Corresponding Author 1https://github.com/dropreg/DEER
While recent studies (Sanh et al., 2019; Jiao et al., 2020) have made attempts to address the challenges associated with compressing and accelerating inference for pre-trained Transformer models, the majority of these efforts have concentrated on techniques such as knowledge distillation (Song et al., 2020), quantization (Bai et al., 2021; Tao et al., 2022), and parameter pruning (Xia et al., 2022). The pre-trained non-autoregressive generation paradigm has received limited attention and remains relatively unexplored.
To fill this blank, we first summarize two main difficulties in the deployment and application of large generative models. Firstly, the prevailing generative models currently employ an autoregressive approach to generate target tokens incrementally, as seen in models like BART (Lewis et al., 2020) and T5 (Raffel et al., 2020). While these models have gained popularity and demonstrated effectiveness, their autoregressive nature hinders efficient inference through parallelization, resulting in inefficiencies. Secondly, task-specific fine-tuning is crucial when deploying pre-trained models on diverse edge devices (Sun et al., 2020; Xu et al., 2021). It is impractical to adopt a single model for all devices due to variations in memory capacity and latency constraints. Consequently, multiple models with different architectural configurations need to be trained to meet these device-specific requirements, leading to additional resource consumption and increased carbon emissions. To address these challenges, we propose a novel joint training strategy called DEER. This strategy offers fast inference by employing a non-autoregressive generation approach and provides flexibility in model size through the utilization of dynamic block pruning.
Concretely, we choose the BERT family models to implement our DEER method because their bidirectional attention mechanism is more suitable for non-autoregressive generation tasks. To allow encoder-based models for text generation and re-
2883
duce the error accumulation in length prediction, we combine the training objective of Connectionist Temporal Classification (Graves et al., 2006; Libovickỳ and Helcl, 2018) (CTC) and Levenshtein Transformer (Gu et al., 2019) for multi-task training. Compared with previous methods, this approach has a better result than the iterative approach at the first generation step and can further improve the iteration refinement performance with the obtained good initialization. Moreover, to easily adapt the BERT family to non-autoregressive generation without introducing extra parameters or cumbersome post-training, we design task-specific input formats and self-attention masks (Dong et al., 2019). Different input formats and self-attention masks can dynamically control the source and target information interaction and remedy the structural defects of the encoder-based model, making it competent for text generation.
Our DEER also incorporates dynamic block pruning for model training and inference to make the BERT family with adaptive model size. Meanwhile, we use score-based parameter mask and sparsity regularization to choose and train the suitable model size for current devices, referring to movement pruning (Sanh et al., 2020; Lagunas et al., 2021; Xia et al., 2022). Unlike current pruning works, DEER is a one-stage training method without two-stage fine-tuning for sub-models and can dynamically choose a model size instead of a fixed size. In inference, we gather the weight from the trained model for different devices when its importance score is larger than the global threshold. The sparsity regularization is also crucial, which can encourage the model to decrease the importance of weight score and control the sparsity level.
We conducted extensive experiments to validate and analyze the effectiveness of our proposed DEER method on both monolingual and multilingual models from the BERT family. In particular, our DEER method outperforms the AR model, achieving a 3× → 12× speedup on three neural machine translation tasks. Additionally, DEER overcomes the limitations of memory and latency, enabling support for various hardware devices without compromising the task performance of the original model. These results demonstrate the efficacy of our DEER method in improving inference speed and compatibility with diverse hardware devices, while maintaining or surpassing the task performance of the original models.
In a nutshell, our contributions are as follows:
• DEER leverages the combination of nonautoregressive training and the pre-trained BERT family to enhance performance while maintaining fast inference by modifying the iteration step.
• DEER integrates the CTC generator and Levenshtein editor to empower the Transformer encoder-based model with the ability to generate and produce favorable results for iterative refinement, eliminating the need for taskspecific length prediction modules.
• DEER utilizes dynamic block pruning to reduce the model size with only a marginal decrease in performance, enabling deployment on diverse hardware devices and overcoming limitations related to memory and latency.
• Benefits from the NAR generation and dynamic block pruning, we demonstrate that DEER achieves excellent performance on multiple text generation tasks, showcasing its remarkable generalization capability.

heading: 2 Related Works

heading: 2.1 Structured Pruning
text: Structured pruning methods (He et al., 2017; Molchanov et al., 2019; Guo et al., 2020) aim to search a sub-model for large-size models by pruning unimportant dimensions (McCarley et al., 2019; Prasanna et al., 2020), heads (Renda et al., 2019; Wang et al., 2020), and layers (Fan et al., 2019; Sajjad et al., 2020). Movement Pruning (Sanh et al., 2020; Lagunas et al., 2021; Xia et al., 2022) is a representative method that introduces a flexible parameter mask to obtain significant weights by scoring parameters during training. However, this approach only tries to find a high-performance submodel with target sparsity rather than a model that can adaptively adjust the model size. It is an urgent need to explore dynamic and efficient models for various common mobile platforms (Li et al., 2021), such as self-driving cars, smartphones, drones, and robots. Hou et al. (2020) propose a dynamic BERT model called DynaBERT, allowing both adaptive width and depth to satisfy the requirements of different edge devices. In order to make the model adaptable to different hardware devices and push sub-models to achieve competitive performance,
our DEER combines the advantage of movement pruning and dynamic training to fine-tune the pretrained generative model.

heading: 2.2 Non-autoregressive Generation
text: Recently, there has been a wide range of studies (Gu et al., 2018; Qi et al., 2021; Li et al., 2022a) for Non-autoregressive text generation to improve inference efficiency. The commonly used nonautoregressive methods can be categorized into two types, i.e., single-step generation (Qian et al., 2021; Ghazvininejad et al., 2020; Du et al., 2021) and iterative generation (Kasai et al., 2020; Gu et al., 2019; Saharia et al., 2020; Huang et al., 2021). For example, Libovickỳ and Helcl (2018) introduced CTC to the single-step non-autoregressive framework that models latent alignments with dynamic programming. Ghazvininejad et al. (2019) introduced the masked language modeling objective to non-autoregressively model predict and refine translations iteratively. Gu et al. (2019) proposed a new sequence generation model called Levenshtein Transformer, composed of the insertion and deletion operations, which facilitates not only generation but also sequence refinement by allowing dynamic length changes. However, the iterative model does not produce satisfactory results for single-step decoding and needs multiple-step refinement to improve performance. As a concurrent work, XLM-D (Wang et al., 2022) also delved into the implicit alignment and pre-trained models for non-autoregressive generation. However, we employed distinct methods and model architectures in research. Additionally, we conducted further exploration by incorporating model pruning to achieve additional compression of the model size, enhancing its suitability for a broader range of scenarios.

heading: 3 Methods
text: In this section, we first exhibit how to fine-tune the BERT family model (e.g., XLM-R and RoBERTa) as a NAR text generator, which supports single-step generation (§ 3.1) and iterative-based generation (§ 3.2), as shown in Figure 1. Then we introduce the dynamic block pruning for model training to reduce the computation and memory consumption in inference with dynamic model size (§ 3.3).

heading: 3.1 Single-step CTC Generator
text: The BERT family models comprise stacked bidirectional Transformer encoder blocks (Vaswani et al., 2017), in which each block contains two sub-layers: the multi-head self-attention layer and the fully connected feed-forward layer. For a given BERT variant MBERT, the l-th encoder block takes the representation of the (l-1)-th block as input Hl−1, and sequentially processes it as:
S l = Self_Attention(Hl−1) +Hl−1, Hl = Feed_Forward(S l) + S l,
(1)
where Hl is the output of the encoder layer l, and there is also a residual connection and layer normalization for each sub-layer.
Given the paired training data D=(X ,Y), the BERT family models can easily obtain the contextualized vector representation for source sentence X , but their bidirectional attention mask mechanism makes them difficult to be applied to text generation tasks. Thus, we use the latent alignment model to train our model, which utilizes the Connectionist Temporal Classification (CTC) to model the token alignment A between X and Y . In this way, the model does not need to predict
the length of the target sequence. The latent alignment assumption requires that the length of the source sentence is at least as long as the target. To satisfy this requirement, we utilize specific input formats and self-attention masks to control context information and generate target sentences in a NAR manner. As shown in Figure 1, we combine the source X and pseudo target Ŷ as input and build a specific attention mask when the source sentence length is close with the target, which makes the Ŷ attend to X , but X cannot attend to Ŷ , such as machine translation task. For example, we copy the source sentence twice uniformly as Ŷ , e.g., Ŷ = {x1, x1, x2, x2, . . . , xm, xm}, given the X = {x1, x2, . . . , xm}. Finally, we will compute the log-likelihood of the target and CTC loss function by marginalizing the latent alignments:
logP(Y|X ) = log ∑
a∈β(Y)
∏
i
P(ai|Ŷ,X ),
LCTC = − logP(Y|X ), (2)
where function β(Y) can generate the set of all possible alignments from X to Y , which can implement with an efficient dynamic programming algorithm (Graves et al., 2006).
It is worth noting that we have discovered that in tasks with rich resources, the model’s exclusive reliance on implicit alignment does not adequately capture the alignment patterns inherent in the dataset. The existence of numerous intricate patterns amplifies the challenges associated with model learning. Consequently, we adopt the Glancing strategy (Qian et al., 2021) to facilitate a progressive learning approach for the model.

heading: 3.2 Iterative-based Levenshtein Editor
text: Although the CTC model supports fast inference with the single-step generation, it relies on the conditional independence assumption for token alignments, which is incapable of handling multi-modal scenarios. Therefore, we introduce the iterative refinement mechanism using Levenshtein Editor (Gu et al., 2019), which shares parameters with the CTC model to correct the text error.
During training, we first build training data to imitate insertion and deletion behaviors in the text editor, which are basic operations from the Levenshtein Transformer. In particular, we corrupt the target as an initial state YDEL by random deleting each token from Y and then reconstruct the original target sequence by three classifiers: 1) the place-
holder classifier can predict the number of insertion tokens via the adjacent two tokens of YDEL:
ŶPLH = PLH_CLS(MBERT(HX ,YDEL)), LPLH = Cross_Entropy(YPLH, ŶPLH),
(3)
where the placeholder target label YPLH is calculated by comparing Y and YDEL. Meanwhile, we concatenate the hidden states of the source sequence HX and target sequence hidden states HYDEL as the attention key/value for Transformer selfattention layer, as shown in Figure 1. Especially, HX is the cached hidden states from the CTC generation step; 2) we insert placeholder for YDEL as the insertion classifier input YINS, and predict the missing token for each placeholder:
ŶINS = INS_CLS(MBERT(HX ,YINS)), LINS = Cross_Entropy(Y, ŶINS);
(4)
3) the deletion classifier can predict whether the current token needs to be kept or removed for previous step results ŶINS:
ŶDEL = DEL_CLS(MBERT(HX , ŶINS)), LDEL = Cross_Entropy(ȲDEL, ŶDEL),
(5)
where the delete label ȲDEL is calculated by ŶINS ̸= Y . During inference, we take the CTC result as input to feed the Levenshtein Editor sequentially through different classifiers (deletion classifier → placeholder classifier → insertion classifier) to obtain the target sequence. We refer the reader to Gu et al. (2019) for more details.

heading: 3.3 Dynamic Block Pruning
text: To achieve dynamic computation scales, we introduce the dynamic block pruning to fine-tune the BERT family with a task-specific dataset refer to movement pruning (Sanh et al., 2020). We select important weight from the pre-trained model by introducing the score-based parameter mask M(S) in each forward pass, i.e., W = W ⊙ M(S). S is the score parameter for each parameter, which is calculated by the straight-through estimator (Bengio et al., 2013). The importance score can guide us to adjust the model size dynamically by setting a specific threshold τ , e.g., M(S) = 1 when S > τ . Different from the pruning method, our method needs to modify the threshold value according to fixed model sparsity (such as {0%, 25%, 50%, 75%}) during training. The
threshold τ is not needed to be updated every training step as it is time-consuming, and we found that setting the updating number to 200 works better in experiments. It is worth noting that we set two global thresholds for the self-attention layer and the feed-forward layer, respectively, considering their different designs and functions for Transformers.
The masked weight is required for each multihead self-attention and the fully connected feedforward layer in model training:
Q = Hl−1Wq ⊙M(Sq), K = Hl−1Wk ⊙M(Sk), V = Hl−1Wv ⊙M(Sv), A = Softmax(QK T
√ d ),
S l = AVWo ⊙M(So) +Hl−1, Hl = gelu(S lWf1)⊙M(Sf )⊙Wf2 + S l,
(6)
where d is the dimension of hidden states, Wq, Wk, Wv, Wo, Wf1, and Wf2 are the projection matrices. We use two kinds of block-wise score parameter (Lagunas et al., 2021): square blocks (32×32) for the self-attention layer, and dimension blocks (1 × d and d × 1) for feed-forward layer. We also add the L1 norm as a regularization item in training objectives to encourage more sparsity:
Lreg = λ∥σ(S)∥, (7) where λ is the hyper-parameter, σ is the sigmoid function to limit the score boundary.

heading: 3.4 Joint Training Algorithm
text: The detailed training process of DEER is shown in Algorithm 1. Lines 2 to 5 are the dynamic block pruning process, i.e., randomly selecting target sparsity from the model size list Lm to initialize the weight mask. Lines 6 to 9 initialize the specific input to train the CTC generator for the first-step generation. Lines 11 to 20 will switch the self-attention mask and input formats to train the iterative-based Levenshtein Editor through three classifiers. The final training objective is the sum of all items: CTC loss, Levenshtein classifier loss, and weight sparsity regularization term (line 21).

heading: 4 Experiments
text: Datasets We evaluate DEER on multiple widely used text generation tasks to verify its effectiveness: 1) neural machine translation (NMT), we conduct experiments on three benchmark translation
Algorithm 1 Training model with DEER Require: Given data D={(X ,Y)}, BERT family model MBERT and model size list Lm, for example {0.25, 0.5, 0.75, 1.0}.
1: while not converged do 2: ▷ Dynamic Block Sparsity 3: Sample model size m ∼ Lm 4: Calculate threshold by sorted weight 5: Initialize M(S) when τ > sort(θ)[m|θ|] 6: ▷ Train Single-step CTC Generator 7: switch self-attention mask for CTC 8: Initialize Ŷ by uniformly copy X 9: LCTC = criterion(Y,MBERT(X , Ŷ))
10: ▷ Train Levenshtein Editor 11: reswitch self-attention mask for Levenshtein
12: Initialize YDEL by random delete token from Y and calculate placeholder label YPLH 13: ŶPLH = PLH_CLS(MBERT(Hx,YDEL)) 14: LPLH = criterion(YPLH, ŶPLH) 15: Initialize YINS by insert mask token for X 16: ŶINS = INS_CLS(MBERT(Hx,YINS)) 17: LINS = criterion(Y, ŶINS) 18: Initialize ȲDEL as delete label 19: ŶDEL = DEL_CLS(MBERT(Hx, ŶINS)) 20: LDEL = criterion(ȲDEL, ŶDEL) 21: L = LCTC + LPLH + LINS + LDEL + Lreg 22: Compute gradients and update weights 23: end while
datasets: IWSLT’14 German→English2 (De→En), WMT’16 English→Romanian3 (En→Ro), and WMT’14 English→German4 (En→De). For all translation tasks, we report the results of raw (RAW) data and knowledge distilled (KD) data, respectively. We use the same training/validation/test sets as in previous works and the BELU score as the evaluation metric for a fair comparison. 2) monolingual text generation scenarios, we evaluate the efficacy of the proposed DEER on four GLGE benchmarks5, including text summarization (XSum (Narayan et al., 2018) and MSNews) and question generation tasks (SQuAD 1.1 (Rajpurkar et al., 2016) and MSQG). For each dataset, we first train BART Base as a teacher model and gener-
2https://github.com/facebookresearch/fairseq/ tree/main/examples/translation
3https://github.com/facebookresearch/DisCo/ issues/5
4https://github.com/facebookresearch/fairseq/ tree/main/examples/nonautoregressive_translation
5https://github.com/microsoft/glge
ate the distilled data as DEER training data, which can reduce the multi-modality problem (Zhou et al., 2019) to facilitate the learning of NAR models. The official script6 is used for evaluation. Descriptions and data statistics are shown in Appendix A.
Training Setups We use diverse BERT variants as backbone models for different tasks, e.g., XLMR (Conneau et al., 2020) Base for NMT tasks and RoBERTa (Liu et al., 2019) for monolingual text generation. All pre-trained model contains 12 layers of encoder layer with 12 head for multi-head self-attention layer. The embedding size is 768; the feed-forward layer dimension is 3072; dropout and attention dropout is 0.1, and 85M model parameters are in total. For all experiments, we adopt the Adam (Kingma and Ba, 2014) as an optimization algorithm with an initial learning rate 5e− 5, with learning rate schedule polynomial_decay. Label smoothing is utilized in the loss function with a value of 0.1. We set hyper-parameter λ as 10 for all tasks. We select the best checkpoint based on the model performance on the validation set. We train models with target sparsity of {25%, 50%, 75%} for each dataset. We set batch size as 1 for all models and evaluate them on the corresponding test set with the same hardware setup on a single NVIDIA V100 GPU to measure inference speedup. All experiments are done using the sequence mod-
6https://github.com/microsoft/ProphetNet/blob/ master/GLGE_baselines/script/eval.py
eling toolkit Fairseq library (Ott et al., 2019).
Baselines We compare DEER against several baselines, including vanilla AR-based Transformers, single-step NAR models, and iterative-based NAR models. We also take several pre-trained language models as the strong baseline, e.g., pretrained AR model BART, ProphetNet, and CeMAT, and pre-trained NAR model BANG and ELMER.

heading: 5 Main Results
text: In this section, we explore whether DEER can provide dynamic and efficient inference on multiple tasks and datasets by evaluating its nonautoregressive capabilities and model performance with adaptive model sizes.

heading: 5.1 Neural Machine Translation
text: Table 1 shows the performance of our DEER compared with base models on three NMT datasets. DEER consistently achieves higher performance on the KD dataset by fine-tuning the BERT family model compared to the model trained from scratch. Remarkably, our model can improve nearly 2 to 3 BLEU scores for every dataset through single-step iterative refinement using Levenshtein Editor. Significantly, DEER exceeds the vanilla Transformer (AR model) by 2 BLEU score (37.46 v.s. 35.05) on the IWSLT’14 De→En dataset and nearly 1 BLEU score (35.53 v.s. 34.6) on WMT’16 En→Ro dataset with 4 iteration steps. For the fully NAR
setting (single-step generation), our method also achieves comparable performance compared with strong baseline GLAT by only using CTC alignment training objective. Benefiting from the NAR speedup, DEER obtains efficient inference with faster 3 → 12 × than the AR model, even though the BERT family model has more parameters and layers. For the raw data scenario, DEER obtains acceptable results on low-resource datasets but fails on the rich-resource dataset (WMT’14 En→De). Obviously, the CTC-based model cannot handle the multi-modality problem in large-scale data, which confuses the model in learning the alignment effectively. Considering its complexity, we will leave it as future work.

heading: 5.2 Text Generation
text: Table 2 presents the experimental results for the monolingual text generation datasets. Compared to the pre-trained NAR model BANG (Qi et al., 2021) and ELMER (Li et al., 2022a), DEER obtains better performance on question generation task SQuAD 1.1 under the fully NAR setting. Besides, DEER also achieves 9.3 ×, 5.8 ×, 6.3 ×, and 4.6 × inference speedup for XSUM, MSNews, SQuAD, and MSQG, respectively. Compared to the pre-trained AR model, DEER surpasses the ProphetNet (Qi et al., 2020) and achieves a comparable result with BART. These results well demonstrate that DEER
supports dynamic and efficient inference and good trade-offs between performance and latency with flexible iteration steps.

heading: 5.3 Dynamic Model Size for Inference
text: We conducted further experiments to evaluate the performance of the models under different sizes pruning, to verify whether the models are overparameterized for various tasks. We partitioned the backbone networks of RoBERTa-base and XLMRbase into different proportions: 100%, 75%, 50%, and 25% (excluding the parameters of the embedding layer). In the experiments, it can be observed that our approach maintains satisfactory performance even after reducing the parameter size by half. Thus, we can effectively deploy DEER on different edge devices by adjusting the model sizes.
In Table 3, we compare the scalability for DEER and Scalable Transformer (Gao et al., 2021) (AR model) on the WMT’14 En→De dataset, which contains multiple sub-Transformer that can be eas-
ily obtained from full Transformer by parameters pruning. Under the same memory constraint, DEER outperforms Scalable Transformer by comparing the sub-model performance with competitive parameters, which demonstrates the superiority of our dynamic block pruning.

heading: 6 Analysis and Discussion

heading: 6.1 Ablation Study
text: To confirm the effectiveness of the CTC model and Levenshtein Editor combination, we separately train them by using the RoBERTa as the backbone model on the IWSLT’14 De→En dataset. Table 4 shows that DEER achieves better performance than Levenshtein Transformer (w/o CTC) with nearly 3 BLEU scores, which benefits from the good CTC initialization at the first iteration step. We also observe that DEER performs better than a single CTC generator under the fully NAR setting, which indicates that their combination can enhance each other without sacrificing the model performance.

heading: 6.2 Sparsity Regularization
text: We continue to explore the effect of sparsity on dynamic block pruning, which is also the notable dissimilarity between DEER and related work DynaBERT (Hou et al., 2020). Figure 2 displays the results of DEER without sparsity regularization term
Lreg. We can observe that the model performance drops significantly with the increase of the pruning scale. Experiments show that sparse regularization is crucial for model training, which ensures that the model performs well without post-tuning.

heading: 6.3 Structures of Pruned Units
text: Furthermore, we study the pruned structures produced by DEER and show the proportion of kept weights on WMT’14 En→De (please refer to Appendix B for other datasets) for each multi-head self-attention (MHA) layer and feed-forward (FFN) layer respectively, as shown in Figure 3. The model tends to prune the parameters of the top layer of the stacked transformer block rather than the bottom layer, which is consistent with the phenomenon in NLU model pruning (Xia et al., 2022). In addition, there is not much distinction for pruned structures on each MHA layer. We also test the model performance with a single mix threshold instead separately for different layers. Unfortunately, we do not obtain better results in experiments. The mixed threshold reduces numerous essential parameters in the MHA layer and seriously impairs the model inference because the FFN layer has much more parameters than the MHA layer.

heading: 7 Conclusion
text: In this work, we propose DEER, a novel fine-tuning method that supports dynamic and efficient inference to adapt to the memory and latency limitations during deployment. Our approach has achieved impressive results on multiple natural language processing tasks, including the GLGE benchmark and three machine translation datasets. Furthermore, we have observed that the issue of length prediction consistently limits the performance of the
model, especially when dealing with raw datasets. The model struggles to accurately determine the length of the target data, which somewhat affects the model evaluation. In our future work, we will prioritize addressing the challenge of length prediction, aiming to make it more convenient and applicable to a wider range of tasks and scenarios.

heading: 8 Limitation
text: Although DEER has shown excellent performance on multiple datasets and tasks, we still found some limitations affecting its usability and efficiency: (1) The latent alignment model (such as CTC) cannot deal with the multi-modality problem in the largescale dataset, which also leads DEER to underfitting the multiple latent alignment targets that need to be aligned. (3) Although DEER does not need to perform length prediction, it relies on the assumption that the input length is large than the output, which causes the model to lose flexibility in length control. (3) We compared sequence-tosequence models such as BART and ProphetNet in the experimental part of this work. In fact, BART only through six layers on each forward pass, while the BERT family model needs to go through 12 layers, leading the inefficient inference due to latency accumulation of multiple iteration steps.

heading: 9 Ethics Statement
text: DEER relies on the pre-trained language models, e.g., RoBERTa and XLM-R, which may inherit problematic biases. However, we only use these models as a backbone rather than using their predictions. DEER is also a task-specific method that performs the fine-tuning process at the task-specific dataset, which also makes the generated result depend on the input of the dataset and reduces the inherent bias.

heading: Acknowledgements
text: This work is supported by the National Science Foundation of China (NSFC No. 62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488). This work is also supported by Beijing Academy of Artificial Intelligence (BAAI).

heading: A Dataset Statistics
text: The statistic of each dataset is shown in Table 5. We exhibit the number of examples in the train/dev/test set and the average number of words for the source and target sentence. In particular, the XSUM dataset consists of 227K online articles from the British Broadcasting Corporation (BBC), which contains professionally written single-sentence summaries. MSNews is a new News headline generation dataset, which contains online news articles, and each article contains a professionally written single-sentence headline. SQuAD 1.1 contains over 100K crowd-worker created questions in 536 Wikipedia articles. MSQG contains 220K passages as source sentences from a real-world search engine, and each passage contains a highlighted span as the target.

heading: B Structures of Pruned Models
text: Figure 5 and Figure 4 show the structures of the pruned model on IWSLT’14 De→En dataset and WMT’16 En→Ro dataset respectively. We can summarize from the experimental results that the pruning ratio of each layer (multi-head selfattention layer and feed-forward layer) in the model is similar even in different tasks.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: We provide the limitations in Section 8.
7 A2. Did you discuss any potential risks of your work? We think our general training method will not lead to any negative societal impact.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: We summarize our contribution in section 7.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments?

heading: In section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? We provide computational information in section 4 training setup, which contains the computational budget, i.e., NVIDIA V100 GPU.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? We provide experimental setup including hyper-parameter setting and best-found in section 4.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We report the average results (number) for multiple runs of most experiments instead of the error bars.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We report the toolkit version in section 4.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In our future work, we will prioritize addressing the challenge of length prediction, aiming to make it more convenient and applicable to a wider range of tasks and scenarios