言語処理学会 第30回年次大会 発表論文集（2024年3月）

Non-autoregressive Pre-trained Sequence-to-Sequence
Modeling with BERT-NAR-BERT
Mohammad Golam Sohrab1 Masaki Asada1 Matı̄ss Rikters1 Makoto Miwa1,2
1 National Institute of Advanced Industrial Science and Technology
2 Toyota Technological Institute, Japan{firstname.lastname}@aist.go.jp makoto-miwa@toyota-ti.ac.jp

Abstract
We introduce BERT-NAR-BERT (BnB) – a pre-trained
non-autoregressive sequence-to-sequence model, which
employs BERT as the backbone for the encoder and decoder for natural language generation tasks in general andbiomedical domains. We adopt length classiﬁcation andconnectionist temporal classiﬁcation models to control the
output length of BnB. Evaluation results on language understanding, abstractive summarization, question generation, machine translation and biomedical text summarization show substantial improvements in inference speed(∼10x) with a slight deﬁciency in output quality compared to our autoregressive baseline. Code is releasedon GitHub1）under the Apache 2.0 License.

1

Introduction

Sequence-to-sequence (S2S) models have recently been
widely used for natural language processing problems. TheS2S architecture is ﬁrst introduced in the ﬁeld of machinetranslation (MT) [1] and later used for pre-trained generative language models (LM), such as BART [2], Optimus [3]and BERT2BERT [4]. These models usually adopt an autoregressive (AR) decoding strategy to generate texts fromleft to right, token by token. AR decoding can perform
high-quality inference, but it has the limitation that it cannot decode tokens in parallel and requires more time andcomputational cost for inference.
In this paper, we propose a novel S2S non-autoregressive(NAR) model based on existing Transformer [5] architectures to allow parameter initialization from publicly
available PLM checkpoints. Speciﬁcally, we extend the
BERT2BERT (B2B) [4] model to build a NAR S2S model1） https://github.com/aistairc/BERT-NAR-BERT

Figure 1

The S2S BERT-NAR-BERT (BnB) architecture.

using BERT as the backbone for both encoder and decoder models. The NAR modeling allows fast decoding
and generation of longer texts. Using BERT as the backbone, we can start training with reliable parameters byloading the pre-trained BERT checkpoints. In addition,
unlike the B2B model, we perform one epoch of additional
pre-training starting at the BERT checkpoints and investigate the eﬀectiveness of the additional pre-training2）. We
adopt the Length Classiﬁcation (LC) [6] or Connectionist
Temporal Classiﬁcation (CTC) [7] models to control the
output length of BnB.
We ﬁne-tune and evaluate the BnB model on the general language understanding evaluation (GLUE), abstractive summarization, question generation, MT, and biomedical text summarization. We compare its performance withseveral AR and NAR models.

2

BERT-NAR-BERT Framework

Our model extends the B2B model into a NAR decoderwith additional pre-training and modeling output length. It
is our direct AR baseline. The overview of BnB is shownin Figure 1.
2） We use the term additional pre-training for pre-training our BnB
model. We load the pre-trained checkpoints and do not pre-train
from scratch.

― 274 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

2.1

tokens are predicted in a non-autoregressive manner.

Model Architecture

Permutation Language Modeling

We randomly

The BnB model comprises a multi-layer Transformer-

permute tokens in a sequence and predict the original order

based encoder and decoder, in which the embedding layerof the tokens, which is inspired by the idea in XLNet [9].

and the stack of transformer layers are initialized with
BERT [8]. To leverage the expressiveness power of ex-

2.2

Modeling Output Length

isting pre-trained BERT models, we initialize our encoder

To control the generation output length of BnB, we im-and decoder parts with the pre-trained BERT parameters.

plement two length prediction (LP) models, namely: 1)

We denote the number of layers (i.e., Transformer blocks)

Length Classiﬁcation [6], and 2) CTC [7] that implicitly

as 𝐿, the hidden size as 𝐻, and the number of self-attention

determines the target length from the token alignment.

heads as 𝐴.
BnB Encoder

Length Classiﬁcation

The encoder part of BnB is the samearchitecture as the BERT model. The BnB model ﬁrst
feeds the source input sequence 𝑿 = 𝒙1 , . . . , 𝒙 𝑛 to the BnB
encoder layer. In the BnB embedding layers, the input representation is constructed by summing the corresponding
token (X), position (P), and type (T) embeddings.
The embeddings are fed into the BERT self-attention
and feed-forward layers. The hidden representation of the
ﬁnal layer, 𝒉, is passed to the subsequent layer for obtaining
latent representations.
Latent Representations

We construct the latent rep-resentation 𝒛 = 𝑾𝐸 𝒉 + 𝒃 based on token-level representation from the encoder hidden state 𝒉 where 𝒛 ∈ ℝ 𝑃 is

The length classiﬁcation for-mulates the LP as a classiﬁcation task and utilizes the latent
representations to predict the target length:𝑝 𝜃 (𝒚 |𝒛 )=∑𝑙 𝑝 𝜃 (𝒚,𝑙 |𝒛 )= 𝑝 𝜃( 𝒚,𝑙𝑦 |𝒛 ) = 𝑝 𝜃 ( 𝒚 |𝒛,𝑙𝑦 ) 𝑝 𝜃 ( 𝑙𝑦 |𝒛 ) ,where 𝑙 𝑦 denotes the length of 𝒚 that is the gold length()
in training and 𝑝 𝜃 𝑙 𝑦 |𝒛 = L𝐿 is the length predictor thatpredicts the length of target sentence 𝒚. Once the sentence
length is predicted, the model predicts each output token
with a token-level categorical cross-entropy loss.
Connectionist Temporal Classiﬁcationrior performance and ﬂexibility for latent alignment. The
CTC loss is independently computed after the decoder by
replacing the length classiﬁcation and CE loss.

matrix. A visualization of BnB encoder and the latent

3

BnB Decoder

The decoder part is also based on the

BERT architecture, and we can directly initialize the decoder with the pre-trained BERT model. The crossattention mechanism is adopted from BERT2BERT and theExperiments
In this section, we evaluate the BERT-NAR-BERT bothin the general and biomedical domains over the diﬀerent downstream tasks including GLUE, abstractive summarization, question generation, MT, and biomedical text

encoder hidden representation of the ﬁnal layer 𝒉 is usedsummarization. (details in Appendix A).

for cross-attention. Our model diﬀers from BERT2BERT

3.1

in input representation and attention masks to enable NAR
decoding. In our BnB decoder, input representation isconstructed without providing any target tokens. The input
representation is constructed by summing the corresponding P and T embeddings and the latent embedding z fromthe encoder. The attention masks are the normal masksthat give access to all future tokens. The resulting decoder output representations of the ﬁnal layer are fed to thesubsequent generation layer.
Masked Language Modeling

We employ the self-

supervised learning strategy adopted in BERT. We randomly mask the tokens in each sequence, and all masked

In contrast

to LC, we also adopt the CTC [7] considering its supe-a 𝑃-dimensional vector and 𝑾𝐸 ∈ ℝ 𝑃×𝐻 is the weight
representations can be seen in the left part of Figure 1.

(1)

GLUE

In Table 1, we compare our fully non-autoregressive
BnB approach with the OpenAI GPT 3） [10], encoderbased BERT, and variational auto encoder-based Optimus
models. In this table, we compare the BnB based on the
permutation LM objectives. Our best BnB model with permutation LM objective function outperforms the OpenAI
GPT [10], and BERT models on averaged scores, but shows
comparable results with of Optimus. The Optimus model
follows the sentence-level pre-training procedure which allows training the model longer than document-level.
3） The ﬁrst version since the model parameters are comparable withthe proposed BnB model.

― 275 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Model

MNLI
392k
ACC

QQP
363k
F1/ACC

QNLI
108k
ACC

SST-2
67k
ACC

COLA
8.5k
MCC

STS-B
5.7k
P.C.

MRPC
3.5k
F1

RTE
2.5k
ACC

Avg.

OpenAI GPT [10]BERT [8]Optimus [3]BnB Encoder + additional pre-training

82.1
84.6
83.4
82.2

70.3
71.2
90.9
90.3

87.4
90.5
90.8
89.8

91.3
93.5
92.4
90.7

45.4
52.1
57.3
54.3

80.0
85.8
88.8
87.9

82.3
88.9
87.3
88.0

56.0
66.4
69.7
66.8

75.1
79.6
82.5
81.3

Table 1Comparison of OpenAI GPT, BERT, Optimus, and BnB on the validation set of GLUE. ACC and P.C stand for accuracy andPearson correlation coeﬃcient, respectively. The OpenAI GPT scores are reported by Devlin et al. [8]. Bold and underlined scoresdenote the best and second-best results.
Type

AR
Semi-AR

NAR

Model
Transformer [5]BART [2]BERT2BERT [4]iNAT [11]CMLM [12]NAT [13]
CMLM [12]ELMER-Hard [14]ELMER-Soft [14]BnB
BnB + additional pre-training

R-1

R-2

XSum
R-L Latency (↓)

R-L

SQuAD v1.1
B-4 Latency (↓)

30.7
38.8
37.5
27.0
29.1
24.0
23.8
34.5
38.3
32.7
36.1

10.8
16.2
15.2
6.9
7.7
3.9
3.6
9.8
14.2
11.6
13.4

24.5
30.6
30.1
22.4
23.0
20.3
20.2
26.1
29.9
27.8
30.0

29.4
42.6
39.3
32.3
29.6
31.5
31.6
37.9
40.2
36.8
41.7

4.6
17.1
13.5
3.2
3.9
2.5
2.5
11.8
13.5
9.1
13.8

18.33x12.92x10.67x
2.17x
7.92x
1.25x
1.17x
0.83x
0.83x
1.0x
1.0x

11.17x8.00x
10.67x
2.25x
7.50x
1.17x
1.17x
0.83x
0.83x
1.0x
1.0x

Table 2

Performance comparison on XSum and SQuAD v1.1. R-1/2/L and B-4 stand for ROUGE-1/2/L and BLEU-4, respectively.
Bold and underlined scores denote the best and second-best results within NAR models. We include existing method latency values
reported by Li et al. [14].

3.2

Summarization and Question Generation

Table 2 shows the performance comparison of BnB
over the XSum and SQuAD v1.1 datasets for summarization and question generation tasks, respectively. Our bestmodel outperforms all the semi-autoregressive (semi-AR)
and most of the NAR approaches but closely competes
with the AR approaches. Under the NAR setting, we
achieve second-best results over the XSum and SQuAD
v1.1 datasets. Our model outperforms the ELMER-Hardon the XSum and SQuAD v1.1 datasets, while ELMERSoft remains the deﬁnitive state-of-the-art. ELMER-Hardand ELMER-soft denote ﬁne-tuning ELMER with hardand soft early exit strategies, respectively. Unlike BnB thatneeds to determine length by integrating an extra LP model,
ELMER dynamically adjusts the output length by emitting
an end token at any position with early exit strategies.

3.3

Machine Translation

Model

EN - DE→←EN - RO→←

Transformer
B2B mBERTBnB random
BnB mBERTBnB mBERT dist.

27.30
25.80
7.15
6.81
27.49

21.53
23.24
4.12
5.92
18.94

25.36
23.40
8.02
11.07
27.06

27.81
30.67
7.11
9.36
30.42

Table 3

MT experiment results in BLEU scores of training
Transformer, B2B, and BnB initialized with multilingual BERT(mBERT); trained on original and distilled (dist.) data from
WMT 2014 (German) and WMT 2016 (Romanian). Bold andunderlined scores denote the best and second-best results.

with initialization from mBERT, as well as knowledge distillation [15], which has previously proven to be highly
beneﬁcial for NAR MT models [16]. Results of MT experiments are summarized in Table 3. The results show that
our model can compete with the baseline Transformer andB2B models after knowledge distillation. It is expected as
the BnBs are compared with the autoregressive baselinetranslation models.

For MT, we compare the scores with autoregressive baselines - Transformer MT models and BERT2BERT, initialized with random parameters or pre-trained multilingual
BERT. We compare random initialization of parameters

3.4

Biomedical Text Summarization

Table 4 shows the performance comparison between AR
methods and our BnB. First, our model is about 18 times
faster than AR models in inference. Regarding genera-― 276 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Modelencoder (VAE) language model, Optimus. They connect

iCliniq
R-1

R-2

R-L

Latencya BERT encoder and a GPT-2 decoder using a universal

61.43

48.68

59.71

18.5x

latent embedding space. The model is ﬁrst pre-trained on a

BioBART base [17] 61.07

48.47

59.42

18.7x

large text corpus and then ﬁne-tuned for various language

57.43 43.99

55.24

1.0x

generation and understanding tasks. It achieves SOTA on

BART base [2]BnB

VAE language modeling benchmarks. While the general

HealthCareMagic
R-1

R-2

R-L

Latencyidea of our work is similar, there are several core diﬀerences

46.81

26.19

44.34

18.1x

from this paper. Our model does not have a VAE andBioBART base [17] 46.67

26.03

44.11

18.9x

instead of the GPT-2 decoder we use the same BERT as in

40.05 19.38

38.27

1.0x

the encoder.

BART base [2]BnB

Rothe et al. [4] developed Transformer-based S2S mod-

Table 4The main results on the summarization tasks in thebiomedical domain. R-1/2/L stand for ROUGE-1/2/L.

els by describing several combinations of model initialization that include BERT2BERT, a BERT-initialized en-tive performance, biomedical BnB showed the ROUGE1 scores of about 94% and 85% of the performance bycoder paired with a BERT-initialized AR decoder. Our
implementation of BnB is similar, except for the main

BioBART on the iCliniq and HealthCareMagic datasets,

diﬀerences of having a length prediction model, a latent

respectively.

representation from the encoder output layer, and a NAR
decoder. The NAR decoder can decode tokens in parallel

3.5

Inference Speedup

which drastically reduce the inference computational cost.
Sohrab et al. [20] describe BERT-NAR-BERT (BnB)

We also compare the diﬀerences in inference speed between the models. While training speed is mostly similar– a pre-trained non-autoregressive sequence-to-sequence

for all, BnB can generate output 17x faster on average due

model, which employs BERT as the backbone for the en-

to its non-autoregressive nature (tested on NVIDIA V100

coder and decoder for natural language understanding andand A100 GPUs) as shown in the Latency column of Ta-

generation tasks. BnB shows substantial improvements inble 2. For MT, translating 2,000 sentences in the WMT16

inference speed over the downstream tasks. This work is

test data set with BnB takes 87 seconds on a GPU and 587

a paper summarizing on the basis of the original researchseconds on a CPU. The same took 234 seconds on a GPU

methods and ﬁndings of BERT-NAR-BERT [20].

and 1,234 seconds on a CPU for an equivalent with thedirect B2B baseline model. BERT-NAR-BERT consists of5

Conclusion

110M parameters for the encoder and decoder, includingThis paper introduces an eﬃcient non-autoregressive

12 layers, 768 hidden sizes, and 12 self-attention heads.

S2S model BERT-NAR-BERT that outperforms baselines

Our direct baseline BERT2BERT model follows the same

in most of the language understanding, summarization andparameters, 220M totals for both the encoder and decoder

question generation tasks. Still, it remains competitive inmodels.

the quality of outputs when evaluated on MT tasks. However, our model with distilled data shows improvement

4

Related Workover the baseline approaches. We ﬁnd that using pre-Pre-trained Language Models such as GPT-2 [18], XL-

trained BERT models as the encoder and decoder, along

Net [9], and XLM [19] are neural networks trained on large-

with CTC for LP and knowledge distillation for MT, helps

scale datasets that can be ﬁne-tuned on problem-speciﬁc

improve the performance of language generation tasks.

data. They became widely adopted after BERT [8], which

In future work, we plan to experiment with replacing

reported state of the art (SOTA) results for 11 NLP tasks.

the BERT models with other pre-trained language models

Our PLM BnB is also trained on a large-scale dataset andwhich can be used as encoders/decoders, as well as running
broader evaluations on other S2S NLP tasks. We will also

further ﬁne-tuned on 12 speciﬁc NLP tasks, 10 of which
overlap with BERT.
Li et al. [3] proposed the ﬁrst large-scale variational auto

address large language models and decoder only models in
our future direction.

― 277 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Acknowledgment
This paper is based on results obtained from a project
JPNP20006, commissioned by the New Energy and Indus-[15]

trial Technology Development Organization (NEDO).
[16]

References[1] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry
Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.
Learning phrase representations using RNN encoder–decoder for
statistical machine translation. In EMNLP 2014), pp. 1724–1734,
Doha, Qatar, October 2014. ACL.
[2] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad,
Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke
Zettlemoyer. BART: Denoising sequence-to-sequence pre-training
for natural language generation, translation, and comprehension.
In ACL 2020, pp. 7871–7880, Online, July 2020. ACL.
[3] Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe
Zhang, and Jianfeng Gao. Optimus: Organizing sentences via
pre-trained modeling of a latent space. In EMNLP 2020, pp.
4678–4699, Online, November 2020. ACL.
[4] Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. Leveraging
pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics,
Vol. 8, pp. 264–280, 2020.
[5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit,
Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.
Attention is all you need. Advances in neural information processing systems, Vol. 30, , 2017.
[6] Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho.
Latent-variable non-autoregressive neural machine translation withdeterministic inference using a delta posterior. Proceedings of
the AAAI Conference on Artiﬁcial Intelligence, Vol. 34, pp.
8846–8853, 04 2020.
[7] Jindřich Libovický and Jindřich Helcl.
End-to-end nonautoregressive neural machine translation with connectionist temporal classiﬁcation. In EMNLP 2018, pp. 3016–3021, Brussels,
Belgium, October-November 2018. ACL.
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova. BERT: Pre-training of deep bidirectional transformers
for language understanding. In NAACL-HLT 2019, pp. 4171–4186,
Minneapolis, Minnesota, June 2019. ACL.
[9] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R
Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive
pretraining for language understanding. In Advances in Neural Information Processing Systems, Vol. 32, pp. 5753–5763,
2019.
[10] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever,
et al. Improving language understanding by generative pre-training.
OpenAI Blog, 2018.
[11] Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative reﬁnement. In EMNLP 2018, pp. 1173–1182, Brussels, Belgium,
October-November 2018. ACL.
[12] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In EMNLP-IJCNLP 2019, pp. 6112–6121, Hong
Kong, China, November 2019. ACL.
[13] Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, andRichard Socher. Non-autoregressive neural machine translation.
In International Conference on Learning Representations,
2018.
[14] Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong

[17]

[18]

[19]

[20]

[21]

[22]

Wen. ELMER: A non-autoregressive pre-trained language modelfor eﬃcient and eﬀective text generation. In EMNLP 2022, pp.
1044–1058, Abu Dhabi, United Arab Emirates, December 2022.
ACL.
Geoﬀrey Hinton, Oriol Vinyals, and Jeﬀ Dean. Distilling theknowledge in a neural network. NIPS 2014 Deep Learning Workshop, 2014.
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In EMNLP 2016, pp. 1317–1327, Austin, Texas, November 2016. ACL.
Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie,and Sheng Yu. BioBART: Pretraining and evaluation of a biomedical generative language model. In Proceedings of the 21st
Workshop on Biomedical Language Processing, pp. 97–109,
Dublin, Ireland, May 2022. ACL.
Alec Radford, Jeﬀrey Wu, Rewon Child, David Luan, Dario
Amodei, and Ilya Sutskever. Language models are unsupervised
multitask learners.
https://d4mucfpksywv.cloudfront.
net/better-language-models/language models areunsupervised multitask learners.pdf, 2019.
Accessed:
2023-08-19.
Alexis Conneau and Guillaume Lample. Cross-lingual language
model pretraining. In Advances in neural information processing systems, Vol. 32, pp. 7059–7069, 2019.
Mohammad Golam Sohrab, Masaki Asada, Matı̄ss Rikters, and
Makoto Miwa. Bert-nar-bert: A non-autoregressive pre-trained
sequence-to-sequence model leveraging bert checkpoints. IEEE
Access, Vol. 12, pp. 23–33, 2024.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim,
Sunkyu Kim, Chan Ho So, and Jaewoo Kang. BioBERT: a pretrained biomedical language representation model for biomedical
text mining. Bioinformatics, Vol. 36, No. 4, pp. 1234–1240, 09
2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer
Levy, and Samuel Bowman. GLUE: A multi-task benchmark andanalysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing
and Interpreting Neural Networks for NLP, pp. 353–355, Brus-sels, Belgium, November 2018. ACL.
[23] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t giveme the details, just the summary! topic-aware convolutional neuralnetworks for extreme summarization. In EMNLP 2018, pp. 1797–
1807, Brussels, Belgium, October-November 2018. ACL.
[24] Chin-Yew Lin. ROUGE: A package for automatic evaluation ofsummaries. In Text Summarization Branches Out, pp. 74–81,
Barcelona, Spain, July 2004. ACL.
[25] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy
Liang. SQuAD: 100,000+ questions for machine comprehensionof text. In EMNLP 2016, pp. 2383–2392, Austin, Texas, November
2016. ACL.
[26] Xinya Du, Junru Shao, and Claire Cardie. Learning to ask: Neuralquestion generation for reading comprehension. In ACL 2017, pp.
1342–1352, Vancouver, Canada, July 2017. ACL.
[27] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.
Bleu: a method for automatic evaluation of machine translation.
In ACL 2002, pp. 311–318, Philadelphia, Pennsylvania, USA, July
2002. ACL.
[28] Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186–191, Brussels, Belgium, October 2018.

ACL.
[29] Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang, SichengWang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoyu
Zhang, Hongchao Fang, Penghui Zhu, Shu Chen, and Pengtao Xie.
MedDialog: Large-scale medical dialogue datasets. In EMNLP
2020, pp. 9241–9250, Online, November 2020. ACL.

― 278 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Abstractive SummarizationA

Experimental Settings

A.1

Pre-training: Data and Task Settings

Abstractive text sum-

marization aims to produce a short version of a document
while preserving its salient information content. We evaluate the models based on the BBC extreme [23] (XSum)

We pre-train our BERT-NAR-BERT (additional pre-dataset. This is a news summarization dataset containing

training) by loading the PLM checkpoints as S2S task.

227K news articles and single-sentence summary pairs.

The additional pre-training procedure follows existing lit-

We set the number of training epochs to 100 and adopt early

erature on PLM pre-training. The entire Wikipedia data

stopping. The evaluation metric is ROUGE [24], including

dump with the version of 20220301.en from Huggingface

ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L).

datasets4） is used by only considering the text ﬁeld for

For latency comparison, we evaluate the time required to

pre-training without any data ﬁltering scheme. As a re-generate all the samples in the validation set with the same

sult, 6,458,670 input texts that include multiple sequencesmachine settings for our BnB and ELMER [14] and calcu-

are truncated to a maximum sequence length of 512, andlate the ratios of latency. We include the reported latency

3.2B tokens are used for our additional pre-training. We
initialize BnB with bert-base-cased and update all thevalues of other existing models from ELMER.
Question Generation SQuAD v1.1 [25] is a dataset

parameters of BnB’s encoder and decoder, while the check-created for machine reading comprehension. The dataset

points are saved from both encoder and decoder outputs.

contains 98K triples of {passage, question, answer} where

We set the 15% probability of masking for masked LM andthe input is formatted as answer [SEP] passage. We use

50% probability of permuting for permutation LM.

this data as a question generation dataset, in which a model

‌

‌

‌

‌

Biomedical Pre-training‌‌

As a parameter initializa-

receives an answer and a passage and generates the corre-tion, we loaded BioBERT [21] v1.1 base-cased checkpoints

sponding question. We follow the same train, validation,

for the initial values of the encoder and decoder parts ofand test data split setting as Du et al. [26]. The evalua-

BnB. We then performed the additional pre-training on thetion metrics are ROUGE-L and BLEU-4 (B-4). We follow

PubMed/MEDLINE abstract corpus5）.

the same settings as the abstractive summarization task forThis corpus con-tains 5.4B tokens of research article abstracts from the 2021
version of PubMed/MEDLINE. We used the same vocab-

latency comparison.
Machine Translation

We evaluate our models using

ulary as BioBERT [21] to tokenize the texts. We truncateddata sets from the WMT shared tasks on news translation

all the input texts to 512 maximum sequence lengths fol-

- English (EN) ↔German (DE) data from WMT 2014 andEnglish↔Romanian (RO) data from WMT 2016. We also

lowing BioBART [17], which is an autoregressive biomedical pre-trained model. CTC loss and permutation language

experiment with distilled versions of these data sets gen-modeling are adopted for additional pre-training.

erated by vanilla transformer models [5] trained on the

A.2

Fine-tuning: Data and Task Settingsnormal data. We load the WMT datasets from Huggingface datasets7）,8）and use them directly to train the models

For ﬁne-tuning, we describe the data and task settings

without ﬁltering, back-translation, or any other kinds ofof the benchmark downstream tasks. We initialize all thesynthetic data generation. We evaluate the performance bydownstream tasks with additional pre-training checkpoints

computing BLEU [27] scores using sacreBLEU [28].
Biomedical Text Summarization

generated from BnB.
GLUE

For ﬁne-tuning

We consider the General Language Under-

the model on downstream tasks, we follow the same set-

standing Evaluation (GLUE) benchmark [22], where wetings as BioBART [17]. We ﬁne-tuned our model on two

employ the Optimus evaluation script6） to evaluate thebiomedical text summarization tasks in English: iCliniq

scores and select the best performances among diﬀerent

and HealthCareMagic [29] datasets.

runs to report all the scores following Optimus.
4） https://huggingface.co/datasets/wikipedia5） https://huggingface.co/datasets/pubmed6） https://github.com/ChunyuanLI/Optimus7） https://huggingface.co/datasets/wmt148） https://huggingface.co/datasets/wmt16― 279 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).