SlowGuess/NAACL_2022_OCR huggingface dataset:
In-BoxBART: Get Instructions into Biomedical Multi-Task Learning
Mihir Parmar<sup>1</sup>
Swaroop Mishra
Mirali Purohit
Man Luo
M. Hassan Murad<sup>2</sup>
Chitta Baral
<sup>1</sup>Arizona State University, USA
$^{2}$ Mayo Clinic, USA
$^{1}$  {mparmar3, srmishr1, mpurohi3, mluo26, chitta} @asu.edu
$^{2}$ murad.mohammad@mayo.edu
Single-task models have proven pivotal in solving specific tasks; however, they have limitations in real-world applications where multitasking is necessary and domain shifts are exhibited. Recently, instructional prompts have shown significant improvement towards multitask generalization; however, the effect of instructional prompts and Multi-Task Learning (MTL) has not been systematically studied in the biomedical domain. Motivated by this, this paper explores the impact of instructional prompts for biomedical MTL. We introduce the BoX, a collection of 32 instruction tasks for Biomedical NLP across (X) various categories. Using this meta-dataset, we propose a unified model termed as In-BoXBART, that can jointly learn all tasks of the BoX without any task-specific modules. To the best of our knowledge, this is the first attempt to propose a unified model in the biomedical domain and use instructions to achieve generalization across several biomedical tasks. Experimental results indicate that the proposed model: 1) outperforms single-task baseline by  $\sim 3\%$  and multitask (without instruction) baseline by  $\sim 18\%$  on an average, and 2) shows  $\sim 23\%$  improvement compared to single-task baseline in few-shot learning (i.e., 32 instances per task) on an average. Our analysis indicates that there is significant room for improvement across tasks in the BoX, implying the scope for future research direction. $^{1}$
For long, task-specific models have played a central role in achieving state-of-the-art performance in both general and biomedical NLP (Wang et al., 2021a; Banerjee et al., 2021). During 2017-2019, pre-train and fine-tune paradigm (Liu et al., 2021) became the prevalent approach in NLP. Due to success of Language Models (LMs) in the biomedical
Figure 1: Schematic representation of multi-tasking in biomedical domain using instructional prompts. In this approach, a model is allowed to utilize tasks to get familiar with instructions and use them to map a given input to its corresponding output.
domain such as BioBERT (Lee et al., 2020), ClinicalXLNET (Huang et al., 2020), and others (Alrowili and Shanker, 2021; Kraljevic et al., 2021; Phan et al., 2021), this paradigm is widely used for creating many task-specific models (Wang et al., 2021a; Banerjee et al., 2021). However, task-specific models have limitations to real-world applications because this approach is computationally expensive (i.e., require large computational resources) and time-consuming (Strubell et al., 2019; Schwartz et al., 2020). Hence, there is a need for generalization where a single model can perform various tasks leading to a computationally efficient approach. Past attempts have been made in general-domain NLP to achieve generalization across tasks such as MQAN (McCann et al., 2018), UNICORN (Lourie et al., 2021), and UnifiedQA (Khashabi et al., 2020). However, approaches to achieve generalization across various biomedical NLP tasks have not been systematically studied. Hence, this paper studies the multi-tasking approach that can generalize over different biomedical NLP tasks. Figure 1 shows the overview of our proposed multitasking approach for various biomedical NLP tasks.
1https://github.com/Mihir3009/In-BoxBART
Findings of the Association for Computational Linguistics: NAACL 2022, pages 112 - 128
Recently, prompt-based models have been widely used because of their ability to achieve generalization instead of task-specific models (Liu et al., 2021). Mishra et al. (2021b); Wei et al. (2021) and Sanh et al. (2021) show the effectiveness of instructional prompts in generalizing on seen as well as unseen general-domain NLP tasks. In this paper, we adapt this instructional prompt-based approach for the first time to achieve generalization across various biomedical NLP tasks. To this extent, this paper introduces a collection of 32 instruction tasks for Biomedical NLP across (X) various categories (BoX) and proposes a unified model that can generalize over 32 different biomedical NLP tasks. The proposed unified model (i.e., In-BoXBART) is trained on the instruction-based meta-dataset (i.e., BoX) and evaluated on each task individually from the BoX.
To evaluate the proposed approach, we compare our model (i.e., In-BoxXBART) with two baselines: (1) single-task models (i.e., models trained on one task and evaluated on the same task), and (2) multitask model (i.e., a single model trained on a combination of all tasks) without instructions. Experimental results show that In-BoxXBART outperforms single-task baseline by  $\sim 3\%$ , and multi-task baseline by  $\sim 18\%$ . We also analyze few-shot learning scenario using In-BoxXBART since obtaining annotated data in the biomedical domain is costly and time-consuming (Luo et al., 2022b). In the few-shot setting (i.e., 32 instances per task), In-BoxXBART outperforms the single-task baseline by  $23.33\%$ . This indicates that Multi-Task Learning (MTL) and instruction-tuning have an advantage in the low resources settings. Although the performance of the In-BoxXBART is promising, our analysis reveals that there is still room for improvement on some tasks, implying the scope for future research direction. Concisely, our contributions can be summarized in three folds:
1. This paper introduces the first benchmark metadata dataset in biomedical domain, i.e., BoX: a collection of 32 instruction tasks for Biomedical NLP across (X) various categories. Each task is processed in a unified format and equipped with instructions that can be used to train sequence-to-sequence models.
2. Using this meta-dataset, we propose an instruction-tuned Bidirectional and Auto-Regressive Transformer (BART) model, termed as In-BoXBART. The comparison of
In-BoxBART and two baselines shows that In-BoXBART outperforms single-task baseline by  $\sim 3\%$  and multi-task (without instruction) baseline by  $\sim 18\%$ .
3. In the few-shot setting, we show that In-BoxBART significantly outperforms the single-task baseline by  $\sim 23\%$ . This indicates the potential application of instruction-tuning in the biomedical domain where annotated data is difficult to obtain.
Multi-task Learning Owing to the problems associated with single-task learning in terms of their space and time requirements, several multi-task learning approaches have been proposed over the years. DecaNLP (McCann et al., 2018) built a multi-tasking model by converting format of each tasks to question answering format. Several other works have followed similar approach, for example, by converting tasks to reading comprehension (Mishra et al., 2022) and textual entailment format (Wang et al., 2021b). The multitasking model T5 (Raffel et al., 2020) was built with the help of a unified framework that converts all text-based language problems into a text-to-text format. SCI-FIVE (Phan et al., 2021) involved building a text-to-text model for the biomedical literature. Aghajanyan et al. (2021) introduced pre-finetuning, an additional large-scale learning stage between language model pre-training and fine-tuning to improve multitask learning performance. Models empowered by multi-task learning have achieved SOTA in many different tasks, e.g., Question Answering (QA) (Khashabi et al., 2020), commonsense reasoning (Lourie et al., 2021) and structured knowledge grounding tasks (Xie et al., 2022).
Instruction Learning The turking test (Efrat and Levy, 2020) was proposed to measure the efficacy of models to follow instructions. Studies have been made to investigate the effect of natural language instructions on model performance (Hase and Bansal, 2021; Ye and Ren, 2021b; Zhong et al., 2021; Weller et al., 2020). Moreover, Mishra et al. (2021b) proposed Natural Instructions which break down each task to multiple sub-tasks that help models in following instructions and subsequently generalize to unseen tasks (i.e., cross-task generalization). FLAN (Wei et al., 2021) and T0 (Sanh et al., 2021) models were built by leveraging instruction/prompt-tuning on diverse range
Figure 2: Schematic representation of 9 categories of tasks: each block represents one category with various tasks equipped with instruction.
of tasks and achieving zero-shot generalization on target unseen tasks. Task reframing (Mishra et al., 2021a) proposed several guidelines to reframe task instructions to improve model response to follow instructions. Analysis introduced to understand in-context learning better on a large set of training tasks (Min et al., 2021, 2022). InstructGPT model (Ouyang et al., 2022) is proposed, which is fine-tuned with human feedback to follow natural instructions. Furthermore, many works focused on investigating whether LMs understand meaning of natural language and prompts (Webson and Pavlick, 2021; Zhao et al., 2021). Weller et al. (2020) and Ye and Ren (2021a) use task descriptions to achieve generalization to new tasks. Puri et al. (2022) introduced instruction augmentation to improve model performance and sample complexity. Wang et al. (2022) has developed instruction-based multi-task framework for few-shot Named Entity Recognition (NER) task. Prasad et al. (2022) introduced Gradient-free Instructional Prompt Search (GrIPS) for improving task instructions for large LMs. Recently, many approaches have been proposed to improve model performance using instructions (Wu et al., 2021, 2022; Lin et al., 2021; Kuznia et al., 2022).
3 BoX
We use 29 existing, widely adopted biomedical NLP datasets collected from various challenges, platforms and organizations to create BoX. We define the BoX as a benchmark dataset for biomedical MTL across 9 different categories. In the BoX,
Category	# of training samples
NER	82503
De-identification	106
POS Tagging	16323
QA	5778
RE	23359
Sentiment Analysis	2860
Systematic Review	5761
Document Classification	3119
Risk Factor Identification	986
Total	140795
Table 1: Size of training samples in each category
we reframed all the datasets as text generation tasks (see examples in Appendix B) and created 32 instruction tasks. BoX consists of high-quality human-authored Biomedical Instructions (BIs) for all 32 tasks. Figure 2 shows the 9 different categories and corresponding generated tasks. Each category is defined as colored box and each box contains instruction tasks re-purposed from original datasets.
Table 1 shows the number of training samples we have used for each category. Further details of each instruction task statistics is shown in Appendix A. Each category and corresponding tasks from the BoX are defined as below:
Named Entity Recognition (NER) NER has been considered a necessary first step in processing literature for biomedical text mining where the model helps in identifying named entities such as protein, gene, chemical, disease, treatment. We use fifteen publicly available biomedical NER datasets (Crichton et al., 2017) to create instruction tasks.
De-Identification (DI) In this task, the model takes medical discharge records of a patient as input and identify Private Health Information (PHI) such as organizations, persons, locations, dates. We use n2c2 2006 de-identification challenge dataset (Uzuner et al., 2007) to perform this task.
Part-Of-Speech (POS) Tagging The goal of this task is to identify various POS tags from the biomedical text. We use GENIA corpus (Tateisi et al., 2005) built from MEDLINE abstracts for the POS tagging task.
Question-Answering (QA) QA models receive a question and a corresponding context as input and output the relevant answer from the given context. To execute this task, we used the BioASQ-8b dataset (Nentidis et al., 2020) for different question types, i.e., yes/no, factoid, and list type questions. We created three different tasks from this dataset. Also, we use PubMedQA dataset (Jin et al., 2019) for this task.
Relation Extraction (RE) We used two datasets for this task: (1) CHEMPROT corpus from biocreative VI precision medicine track (Islamaj Dogan et al., 2019), and (2) Drug-Drug Interaction (DDI) corpus from SemEval 2013 DDI Extraction challenge (Herrero-Zazo et al., 2013). Here, we only consider binary RE tasks without any label describing the type of the relation.
Systematic Review (SR) We have included data from the following five Systematic Reviews (SRs) that were conducted using the traditional (manual) process and published in relevant venues by Mayo Clinic physicians: (1) Hormone Replacement Therapy (HRT), (2) Cooking, (3) Accelerometer, (4) Acromegaly, and (5) COVID for this task (Parmar, 2021). More details about these datasets creation and statistics are given in Appendix C.
Sentiment Analysis (SA) Analyzing the sentiment of people towards medical drugs is an essential task in the biomedical domain. To that effect, we use medical drug sentiment analysis dataset² to identify one of three sentiments: (1) positive, (2) negative, and (3) neutral.
Document Classification (DC) We have used the Hallmarks of Cancer (HoC) dataset (Baker et al., 2016) for this task.
Risk Factor Identification (RFI) The goal of this task is to identify risk factors for Coronary Artery Disease (CAD) in diabetic patients over time. For this, we used n2c2 2014 shared task track 2 dataset (Kumar et al., 2015) with two different purposes: (1) identify if the risk factor is presented in the medical discharge summary and (2) time of risk factor present in the discharge records.
3.2 Biomedical Instructions
Motivated by Mishra et al. (2021b), we have used a similar approach to create Biomedical Instruc
Figure 3: Unified schema used to create a Biomedical Instruction (BI).
Figure 4: Example of Biomedical Instruction (BI) and task instances from BioNLP11ID (NER) dataset.
tions (BIs). BI consists of natural language instructions that describe a task and contain instances of that task. Here, we introduce a unified schema to present BI and described how we can construct BI for each task given in the BoX. Figure 3 illustrates the schematic representation of the schema, and Figure 4 shows an example of BI that describes a "Named Entity Recognition (NER)" task accompanied with a few positive examples.
3.2.1 Unified Schema
All BIs are mapped to the unified schema. As shown in Figure 3, unified schema consists of a definition, prompt, and positive examples. This schema helps in organizing each BI. Each of the elements of the schema is explained below:
$^{2}$ https://www.kaggle.com/arbazkhan971/analyticvidhyadatasetsentiment
Definition contains the core explanation about the task and detailed instruction to the model that what needs to be done in the given task.
Prompt is the short explanation of the task that needs to be done.
Examples contain the input/output pairs of the task instance along with the explanation of how the output is generated. Generally, we provide 2-3 examples for each task.
Instances contain the input/output pairs of training samples from the task datasets.
3.2.2 Construction of BI
We have created a BI for each dataset given in the BoX. To create BI, we manually fill in the fields of unified instruction schema (Figure 3). For each dataset, the BI is created by one author and were verified by other authors.
Quality of BIs In the instruction verification process, we edit BIs if needed in terms of grammar, typos, ambiguity, etc. to improve the quality. According to (Beltagy et al., 2020), concise instructions are more beneficial compared to repetition, hence, we also redact repetition from BIs. As a result, our BIs consists of high-quality, short, and meaningful task definition, and prompts.
Positive examples and its explanation For each dataset, we have provided 2-3 positive examples and corresponding explanations to give an idea of how to perform the given task. As we know, the selection of examples has an impact on model performance (Lu et al., 2021). To that extent, we have been careful in selecting examples for text generation and classification tasks. For text generation, we have provided 2-3 examples with a detailed explanation about how the output is generated. For text classification tasks, we have included examples corresponding to each class with an explanation of why the particular class is assigned to a given input instance. All positive examples are drawn from training instances and have been removed from training in order to avoid repetition. All the explanations of examples pass through the verification process to maintain high quality.
Collection of input/output instances Since each biomedical NLP dataset included in the BoX has its own annotated input/output instances, we converted them into text-to-text format (Lourie et al., 2021). Example of instances converted for each
task is given in Appendix B. After this, we appended all instances tuple (i.e., <input, output>) with instruction schema (as shown in Figure 3).
4 Problem Setup and Models
Let us assume, we have input/output instances pair  $(X_{t},Y_{t})$  for given task  $t$ . Along with that, each task is described in terms of its instruction  $BI_{t}$ .
Single-task models Traditional supervised models learn a mapping function  $(f_{M})$  between input  $(x)$  and output  $(y)$ , where  $(x,y)\in (X_{t}^{\mathrm{train}},Y_{t}^{\mathrm{train}})$  and are evaluated on the same task  $(X_{t}^{\mathrm{test}},Y_{t}^{\mathrm{test}})$ . We refer this setup as single-task learning.
Multi-task models In this setup, we combined training data and corresponding biomedical instruction of all tasks together. The goal of multitask learning models is to learn mapping function  $(f_{M})$  between input  $(x)$ , output  $(y)$  and biomedical instruction  $BI_{t}$ , i.e.,  $f_{M}(BI_{t}, x) = y$ , where  $(x, y) \in (X_{t}, Y_{t})$ . This model is evaluated on task-specific instances  $(x, y) \in (X_{t}^{\mathrm{test}}, Y_{t}^{\mathrm{test}})$  In contrast to single-task models, a single model is used here to solve various tasks, hence, achieving generalization. We refer this setup as MTL.
We propose an instruction-based model to achieve multi-tasking and compare it with two baselines: (1) single-task models, and (2) multi-task models without instructions. We have fine-tuned the BART (base) model (Lewis et al., 2020) to build baselines as well as the proposed model.
Single-Task models As formulated in the single-task problem setup, we have trained the BART model on each task from the BoX and evaluated it on the same task.
Multi-task without instruction To build this baseline, we have combined training data of each task from the BoX together without appending BIs and trained a single model on the combined data. We refer this model as Vanilla-BoXBART. This model is evaluated on each task of the BoX.
4.2.2 Proposed Model
As formulated in the multi-task problem setup, we have combined training data and the corresponding BI of each task. To combine instruction with
input instances, we map a BI and an input  $(x)$  into the textual format and obtain  $enc(BI_t, x)$ . After that, BART model is used to predict an output  $(y)$  using a mapping function  $f_M: enc(BI_t, x) \to y$ . To perform encoding, a standard NLP paradigm of mapping is used, i.e., mapping an input to text. Here, we map each element of BI (i.e., definition and positive examples as shown in the schema) to a textual format and append it before the input instances. After appending BI of each task to instances, we combined all training data of each task. Now, we fine-tuned the BART model with this combined instruction meta-dataset. We refer this instruction-tuned model as In-BoXBART.
We have used BART (base) model to build all baselines and proposed model. All the experiments are performed using Quadro RTX 8000 GPU. All models are trained for 3 epochs. In particular, we have used huggingface implementation (Wolf et al., 2020) of the BART and its pre-defined functions for the training and evaluation with default parameters.
Instance Selection As we know, BART (base) can accept the input of a maximum 1024 token length. Since there are few instances in some datasets that exceed this limit (after including instructions), we have discarded those instances while creating instruction tasks. We have also removed the same instances while training two baselines to do a fair comparison. We have discarded long samples (>1024 token length) from validation and testing data as well.
Example Selection As discussed in Lu et al. (2021), the selection and order of the examples included in instructions matters for mainly classification tasks and affects the performance of the model. We empirically conclude that the proposed model benefits from ignoring examples from biomedical instructions for classification tasks during training and evaluation. Hence, we have discarded all examples from the BIs associated with the classification instruction tasks.
Instance Sampling Some classification datasets used to create the BoX are imbalanced. To balance these datasets, we have applied the sampling techniques (Poolsawad et al., 2014) before using datasets to create BoX. In particular, we have
analyzed three sampling techniques: (1) undersampling, (2) average-sampling, and (3) oversampling. In under-sampling, we have reduced instances for all the classes to the class with the lowest number of instances. In contrast, we have over-sampled instances via replication of random instances to the class with the highest number of instances to achieve over-sampling. In average sampling, we calculated mean of number of instances across all the classes and over-sampled or undersampled instances accordingly for each class.
Few-shot setting Similar to the (Schick and Schütze, 2021), we have started with 32 randomly selected instances for each instruction task from the BoX to exhibit few-shot learning. After that, we have increased randomly selected instance instances per task to  $100 / 1k / 4k$ . If any task has already less number of instances than the threshold (i.e.,  $100 / 1k / 4k$ ), we keep all the instances from that task. While selecting the instances, we made sure that we select balanced data for the classification tasks. Moreover, the BoX contains an average  $6k$  instances per task.
Evaluation Metric We use Rouge-L (Lin, 2004) as our evaluation metric since we treat all the tasks as text generation problems. We also use  $F_{1}$ -Score for evaluations.
5.2 Results and Findings
Effect of Sampling As mentioned above, we conduct three experiments to analyze the effect of sampling on In-BoXBART. We train our model using training data obtained from (1) under-sampling, (2) average-sampling, and (3) over-sampling. We achieve on an average (across all instruction tasks) 69.62, 70.23 and 73.49 Rouge-L for under-, average- and over-sampling, respectively. Here, we observe from the experimental results that over-sampling gives better performance compared to under- and average-sampling since there is a loss of training data samples for under- and average-sampling. Hence, we report results of oversampling as the main result in Table 2.
Performance comparison Table 2 presents the results for single-task model, Vanilla-BoXBART and In-BoXBART. We can see from Table 2 that the single-task model, Vanilla-BoXBART, and In-BoXBART achieve on an average (across all tasks) Rouge-L of 70.51, 55.55, and 73.49, respectively. They achieve  $70.15\%$ ,  $55.21\%$ , and  $73.01\%$