BioNART: A Biomedical Non-AutoRegressive Transformer for Natural Language Generation
Year: 2023

Abstract:
We propose a novel Biomedical domainspecific Non-AutoRegressive Transformer model for natural language generation: BioNART. Our BioNART is based on an encoder-decoder model, and both encoder and decoder are compatible with widely used BERT architecture, which allows benefiting from publicly available pre-trained biomedical language model checkpoints. We performed additional pre-training and finetuned BioNART on biomedical summarization and doctor-patient dialogue tasks in English. Experimental results show that our BioNART achieves about 94% of the ROUGE score to the pre-trained autoregressive model while realizing an 18 times faster inference speed on the iCliniq dataset.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 369–376 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: In biomedical natural language processing, several domain-specific Pre-trained Language Models (PLMs) such as BioBERT (Lee et al., 2019), SciBERT (Beltagy et al., 2019) and PubMedBERT (Gu et al., 2021) have been proposed by training on domain-specific text in PubMed/MEDLINE, and these domain-specific models have shown remarkable performance on the in-domain downstream tasks. Similarly, following the success of generative encoder-decoder-based sequence-to-sequence PLMs such as BART (Lewis et al., 2020), T5 (Raffel et al., 2020), and BERT2BERT (Rothe et al., 2020), pre-trained biomedical generative models such as SciFive (Phan et al., 2021) and BioBART (Yuan et al., 2022) have also been proposed, and these models can tackle the generation tasks in the biomedical domain, such as clinical dialogue, biomedical question answering and biomedical text summarization.
The generative models usually adopt an AutoRegressive (AR) strategy to decode the target sequence token-by-token from left to right. AR
decoding can perform high-quality inference, but it has the limitation that it cannot be easily parallelized and requires significant time and computational cost for inference. To alleviate this problem, several Non-AutoRegressive (NAR) methods (Lee et al., 2018; Qi et al., 2021) that can simultaneously predict all target tokens have been proposed in the general domain. However, there has been no discussion and verification of NAR methods in the biomedical domain.
This paper proposes a novel biomedical domainspecific NAR model called BioNART. In constructing the biomedical NAR model, we achieve NAR generation with only minimal modifications from the BERT (Devlin et al., 2019)-based encoderdecoder model BERT2BERT (Rothe et al., 2020). This allows our model to benefit from already performed large-scale pre-training by initializing both the encoder and decoder of our model with parameters checkpoints of PLMs such as BioBERT (Lee et al., 2019) and SciBERT (Maheshwari et al., 2021). In addition, we further verify the effect of additional pre-training to fit the parameters from checkpoints to the encoder-decoder NAR. Finally, we perform our pre-training on PubMed/MEDLINE abstracts with permutation language modeling (Yang et al., 2019) task.
Our contributions are summarized as follows:
• We are the first to propose a nonautoregressive language model that enables fast inference for the biomedical domain.
• The encoder and decoder of our nonautoregressive model are compatible with existing BERT architecture to allow the parameter initialization from publicly available biomedical PLMs checkpoints such as BioBERT.
• We perform an additional pre-training that starts with the checkpoints to verify that PLMs checkpoints can be fitted for NAR decoding.
369
• We evaluate our BioNART models on several biomedical natural language generation tasks and show that leveraging parameter checkpoints and additional pre-training are effective for NAR natural language generation.

heading: 2 Method
text: In this section, we first describe our BioNART model architecture and then explain the objective function used during training and NAR text generation for inference. Finally, we show the parameter initialization of our BioNART. The overview of BioNART is shown in Figure 1.

heading: 2.1 BioNART Architecture
text: Encoder The encoder part of BioNART is exactly the same as the BERT architecture. The source input sentence is split into sub-word tokens wi by the BERT tokenizer, then converted to word embeddings, positional embeddings, and token type embeddings. These embeddings are fed into the self-attention and feed-forward layers of BERT.
Latent vectors Each token representation of the encoder’s final layer hi is converted into the latent vector zi ∈ Rdz by a single linear layer: zi = W1hi + b1, where W1 and b1 are the weight and bias, and dz is the dimension of the latent vector. The latent vectors have the same sequence length as the input representations of the decoder.
Decoder In an AR model, the decoder takes the target sentence as input and controls the attention
mask so that future tokens are not seen during training. In NAR decoding, the decoder does not take the target sentence as input because target tokens are predicted simultaneously and independently. In the decoder part of BioNART, first, the dimension of the latent vector is matched to the dimension of the hidden representation vectors of the decoder by a single linear layer: h′i = W2zi+b2. Then, these resulting embeddings and positional embeddings are fed into the decoder. The final layer representation from the encoder hi is also fed into the decoder as a cross-attention. Finally, we calculate the prediction vector p ∈ RdV from the final layer representation of the decoder, where dV is the vocabulary size.

heading: 2.2 Training
text: Standard AR models are trained with the cross entropy (CE) loss, which compares the prediction p with the target token y at each corresponding position. However, as NAR ignores the dependency in the output token space, it cannot accurately predict token positions. To ease such limitations, several methods (Lee et al., 2018; Ghazvininejad et al., 2019; Gu and Kong, 2021; Li et al., 2022) have been proposed to consider the latent alignments between the target positions. In this work, we adopt Connectionist Temporal Classification (CTC) (Graves et al., 2006) as the latent alignments, considering its flexibility of variable length prediction. CTC can efficiently find all valid aligned sequences at which the target y can be recovered and marginalize the log-likelihood.

heading: 2.3 Inference
text: Our decoding method during inference is simple and fast. The predicted vector p ∈ RdV is converted to token ids by applying argmax function. The BERT special tokens such as [CLS] and [PAD] are removed from the generated texts. Following the previous methods (Gu and Kong, 2021), we removed repetitive tokens. The decoding of BioNART can be easily parallelized and has an advantage in generation speed over AR models.

heading: 2.4 Parameter Initialization
text: Loading Parameters from Publicly Available Checkpoints Each of the encoder and decoder is compatible with the widely used BERT, so we can initialize the parameters of BioNART with publicly available biomedical domain-specific PLMs such as BioBERT (Lee et al., 2019) and SciBERT (Beltagy et al., 2019). This parameter initialization is expected to boost the performance from a randomly initialized model and lower the cost of subsequent additional pre-training.
Additional Pre-training In order to familiarize our BioNART with NAR decoding, we performed an additional pre-training. We adopt a permutation language modeling strategy proposed in XLNet (Yang et al., 2019). BioNART is pre-trained on the PubMed/MEDLINE abstracts so that it can correctly rearrange the permutated tokens.

heading: 3 Experimental Settings

heading: 3.1 Loading PLMs Checkpoints
text: We initialized each of the encoder and decoder parts of BioNART with BioBERT (Lee et al., 2019) v1.1 base-cased checkpoints.

heading: 3.2 Our additional pre-training
text: We then performed the additional pre-training on the PubMed/MEDLINE abstract corpus1. This corpus contains 5.4B tokens of research article abstracts from the 2021 version of PubMed/MEDLINE. We used the same vocabulary as BioBERT (Lee et al., 2019) to tokenize the texts. We truncated all the input texts to 512 maximum sequence lengths following BioBART. We permuted the 30% of the input tokens, and the model is pre-trained with the CTC loss so that the tokens can be rearranged into the correct orders.
1https://huggingface.co/datasets/pubmed
We set the mini-batch size to 2,560 by adopting gradient accumulation, which is the same size as BioBART. We used a learning rate scheduler with 0.1 warmup ratio and linear decay. The peak of the learning rate was 1e-04. We set the number of training epochs to 5, which corresponds to 45K steps. The number of training epochs is approximately 37.5% of the BioBART. We perform the additional pre-training on 16 40GB A100 GPUs for about 40 hours.

heading: 3.3 Datasets for Downstream Task
text: For fine-tuning on downstream tasks, we follow the same settings as BioBART. We fine-tuned our model on two biomedical language generation tasks in English: text summarization and clinical dialogue. For biomedical text summarization, we trained and evaluated BioNART on iCliniq and HealthCareMagic (Zeng et al., 2020), and for the clinical dialogue task, we employed CovidDialog (Ju et al., 2020) dataset. The details and statistics of the datasets are shown in Appendix A.1.

heading: 4 Results

heading: 4.1 Summarization
text: Table 1 shows the performance comparison between AR methods and our BioNART. First, our model is about eighteen times faster than AR models in inference. As measured with Green Algorithms (Lannelongue et al., 2021)2, we can achieve a reduction of 0.13kg CO2 emission compared to BioBART model during the inference on HealthCareMagic test set. Regarding generative performance, BioNART showed the ROUGE-1 scores of about 81% and 78% of the performance by BioBART on the iCliniq and HealthCareMagic datasets, respectively. We present generative examples on iCliniq dataset in Appendix A.2 Table 3 shows the performance comparison of parameter initialization methods for the encoder and decoder parts of BioNART. These results show that loading the parameter checkpoints of BioBERT improved the ROUGE-score on iCliniq dataset.
By performing our additional pre-training, ROUGE-1/2/L scores improved 7.93/6.45/7.07 points and 3.38/2.62/2.88 points on the iCliniq and HealchCareMagic datasets, respectively. BioNART with additional pre-training model showed the ROUGE-1 scores of about 94% and 85% of the performance by BioBART on the iCliniq and Health-
2http://calculator.green-algorithms.org
CareMagic datasets. These results show that our additional pre-training is effective for NAR generation tasks.

heading: 4.2 Dialogue
text: We show the performance comparison with AR models on the CovidDialogue dataset in Table 2. On the dataset, our model had a more significant performance difference from the AR model compared to the biomedical summarization datasets. We consider one of the factors is the small data size.

heading: 5 Related Work
text: Existing work has shown that pre-training the language models on the domain-specific texts can bring better performance on corresponding domain downstream tasks. BioBERT (Lee et al., 2019) is pre-trained on the PubMed abstracts and PubMed Central (PMC) full-text articles. Instead of continuous training from the general BERT checkpoint, SciBERT (Beltagy et al., 2019) and PubMedBERT (Gu et al., 2021) are trained from scratch from scientific papers on Semantic Scholar (Ammar et al., 2018) and PubMed papers, respectively.
BioBART (Yuan et al., 2022) further analyzed the influence of the pre-training task on the biomedical natural language generation (NLG) tasks and collated existing biomedical NLG tasks along with corresponding data and experimental settings, but all biomedical NLG models are AR-based, and there has been no discussion and verification of the NAR methods in the biomedical domain.

heading: 6 Conclusion
text: In this study, we proposed a non-autoregressive language model for the biomedical domain for the first time. Our method is based on the encoder-decoder model, and both the encoder and decoder are compatible with widely used BERT architecture, which allows for boosting generation performance and reducing additional pre-training costs. In the experiments on biomedical summarization tasks, BioNART achieved an 18 times inference speedup without a large performance drop compared to AR models. Our code and models are available at https://github.com/aistairc/BioNART
In future work, we would like to approach other downstream tasks such as relation extraction and entity linking as sequence-to-sequence problems and realize a high-speed and versatile model for the biomedical domain.
Limitations
This paper shows that NAR generation can be achieved from publicly available PLMs parameters. However, there is still room to be validated for additional pre-training. The performed pre-training in this study is small, and further investigation of whether the training steps and the size of the corpus are sufficient, and whether the self-training tasks other than permutation language modeling are effective.

heading: Acknowledgements
text: This paper is based on results obtained from a project JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO).

heading: A Appendix
text: In this section, we provide the details of the biomedical generation task datasets. The statistics of the datasets are shown in Table 4.
A.1 Dataset Details
A.1.1 Summarization
iCliniq and HealthCareMagic datasets are extracted from the MedDialog dataset (Zeng et al., 2020), collected from the online healthcare platform. iCliniq contains 31,062 samples, and HalthCareMagic contains 226,405 samples. Each sample comprises a summary and corresponding dialogues between a patient and a doctor. The summaries in HealthCareMagic are more abstractive, and they are written in a formal style, while the summaries in iCliniq are written by the patients. We follow previous work (Mrini et al., 2021; Yuan et al., 2022) for training, developing, and testing data separations of both datasets.
A.1.2 Dialogue CovidDialog (Ju et al., 2020) is a medical dialogue task that aims to imitate the human doctor to communicate with human patients in a natural way. The objective of this task is to correctly generate the replies by the doctors from the primitive descriptions of the patients and dialogue. The dataset is collected from online healthcare forums. Each consultation starts with a description related to the medical conditions of the patients, then follows the conversation between a doctor and a patient. The number of samples is 614, which is very small compared to the summarization task datasets.
A.2 Generated Examples We show three randomly chosen examples from iCliniq dataset in Table 5. In the first example, both BioBART and BioNART generate the summary of asking about the causes of chest pain. However, the example generated by BioNART has a grammatical error. In the second example, both BioBART and BioNART summarized only the first part of the context and do not generate an adequate summary. In the third example, while BioBART model contains errors on the subject, BioNART model provides a more accurate summary.

heading: Future Work
text: com/aistairc/BioNART
In future work, we would like to approach other downstream tasks such as relation extraction and entity linking as sequence-to-sequence problems and realize a high-speed and versatile model for the biomedical domain. Limitations
This paper shows that NAR generation can be achieved from publicly available PLMs parameters. However, there is still room to be validated for additional pre-training. The performed pre-training in this study is small, and further investigation of whether the training steps and the size of the corpus are sufficient, and whether the self-training tasks other than permutation language modeling are effective