SlowGuess/NAACL_2022_OCR huggingface dataset:
This section describes the brief data creation process for Systematic Reviews (SRs) that are used in this study. The relentless growth in clinical research and published articles have created a need for automation to expedite the process of SRs and to enable Living Systematic Reviews (LSRs). A crucial step in both SRs and LSRs is the title and abstract-based screening of the articles. A new dataset was developed from six SRs in the clinical domain by Mayo clinic physicians. In this study, we used data from the following five SRs that were conducted using the traditional (manual) process and published in relevant venues: (1) Hormone Replacement Therapy (HRT), (2) Cooking, (3) Accelerometer, (4) Acromegaly, and (5) COVID. The initial bibliographic search was designed and conducted by an experienced librarian with guidance from the principal investigators for the respective studies. The search was conducted in different bibliographic databases like PubMed, PubMed Central (PMC), Embase, EBM Reviews, and Ovid MEDLINE(R). Each article in the bibliographic search results was categorized by two physicians with domain expertise as “Include” or “Exclude”, by reading the title and abstract of the article. When there was a disagreement between two annotators, a positive class (i.e., “Include”) was preferred.
D Few-Shot Learning results
This section presents the results of few-shot learning for all instruction tasks in Table 5.
E State-of-the-art results
In Table 6, we present State-Of-The-Art (SOTA) results for 21 tasks. To compare the SOTA results with the proposed model, we calculate the corresponding metric used in particular research from our model predictions. For each task, we gather the best performance, and specifically, they are BioASQ-8b (Nentidis et al., 2020), Chemprot (Peng et al., 2019), DDI (Peng et al., 2019). In Chemprot and DDI, we compare results with the base LMs instead of large for a fair comparison. SOTA results for all 15 NER datasets are obtained from (Banerjee et al., 2021). Best performance for the HoC dataset is obtained from (Peng et al., 2019). Here, we have considered the result of the best system submitted to (Stubbs et al., 2015) as SOTA result.
Category	Tasks	# of Instances
Train	Dev	Test
NER	AnatEM	3507	1121	2303
BC2GM	6427	1291	2570
BC4CHEMD	14466	14568	12397
BC5CDR	4940	4940	5158
BioNLP11EPI	3796	1242	2836
BioNLP11ID	2466	780	1869
BioNLP13CG	4591	1489	2759
BioNLP13GE	1503	1663	1937
BioNLP13PC	2945	1070	1997
BioNLP09	4710	1013	1699
CRAFT	12839	4423	8882
Ex-PTM	855	278	1160
JNLPBA	15124	1533	3152
NCBI	2922	488	538
linnaeus	1484	524	993
DI	DI 2006	106	22	27
POS	Genia	16323	2174	2035
QA	BioASQ8b (factoid)	695	16	115
BioASQ8b (list)	373	8	45
BioASQ8b (yesno)	543	16	115
PubMedQA	4167	500	473
RE	ChemProt	3350	2415	2660
DDI	20009	2780	2660
SA	Medical Drugs	2860	526	804
SR	Accelerometer	499	58	142
Acromegaly	663	80	192
COVID	2385	300	675
Cooking	735	84	205
HRT	1479	171	410
DC	HoC	3119	445	890
RFI	RFHD 2014 (yesno)	834	360	451
RFHD 2014 (time-riskfactor)	152	177	69
Total	-	140795	46554	64561
Table 3: Statistics of training (i.e., Train), validation (i.e., Dev) and evaluation (i.e., Test) data for all instruction tasks from the BoX. RFHD: Risk Factor for Heart Disease.
Category	Task	Input	Output
NER	BC5CDR	Such interactions may result in serious cardiovascular complications even after cessation of an infusion of ritodrine.	cardiovascular complications <Disease>, ritodrine <Chemical>
de-identification	DI2006	757085252 HLGMC 1228824 18705/6o5b 3/25/1993 12:00:00 AM CONGESTIVE HEART FAILURE . Unsigned DIS Report Status : Unsigned ADMISSION DATE : 3/25/93 DISCHARGE DATE : 4/4/93 PRINCIPAL DIAGNOSIS : congestive heart failure . AS-SOCIATED DIAGNOSIS : aortic stenosis ; coronary artery disease , status post multi vessel coronary artery bypass graft surgery , ..., M.D. TR : go / bmot DD : 4/4/93 TD : 04/06/93 CC : [report_end]	3/25 <DATE>, 18705/6o5b <ID>, 757085252 <ID>, go / bmot <DOCTOR>, 4/4 <DATE>, 04/06 <DATE>
POS-Tagging	Genia	Binding sites were mapped for each factor .	Binding <VBG> sites <NNS> were <VBD> mapped <VBN> for <IN> each <DT> factor <NN>. <>
QA	BioASQ8b (factoid)	Context: Hyperosmia is suspected in pregnancy; however, no empirical study using validated measures of olfactory function has clearly confirmed the anecdotal reports of this phenomenon. subjective hyperosmia is associated with primarily negative odor-related experiences. Hyperosmia is increased olfactory acuity \n Question: What is hyperosmia	Hyperosmia is increased olfactory acuity.
RE	Drug-Drug Interaction	Context: Antacids may interfere with the ab-sorption of LEVSIN. Drug_1: Antacids Drug_2: LEVSIN	true
Sentiment Analysis	Medical Drugs	Why don't more folk opt for Cladribine? \n Drug: cladribine \n Option1: Neutral Option2: Positive Option3: Negative	Positive
Systematic Review	Acromegaly	No greater incidence or worsening of cardiac valve regurgitation with somatostatin analog treatment of acromegaly CONTEXT: Excess GH and IGF-I in acromegaly are associated with reduced life expectancy due to cardiovascular complications. Option_1: Include, Option_2: Exclude.	Include
Document Classification	Hallmarks of Cancer (HoC)	Studies of cell-cycle progression showed that the anti-proliferative effect of Fan was associated with an increase in the G1/S phase of PC3 cells.	Evading growth suppressors, Sustaining proliferative signaling
Risk Factor Identification	n2c2 - Risk Factors Heart Disease 2014 (yesno)	Context: Record date: 2157-08-27 History of Present Illness ID:Admitted from cardiac cath lab. HPI:Mr. Doty is a 80 y.o. male with h/o HTN, DM, PVD, elevated cholesterol who presents with 6 month h/o chest and upper extremity discomfort on exertion along with SOB. He has limited his activities to prevent symptoms. ... \n Risk Factor: Diabetes	Yes
Table 4: Examples of one instruction tasks converted into text-to-text format for each category
Category	Task	32	100	1k	4k
S	I-BB	S	I-BB	S	I-BB	S	I-BB
NER	AnatEM	12.74	60.73	20.68	79.34	87.81	86.76	84.88	83.44
BC2GM	16.92	65.65	21.31	70.39	82.92	77.19	77.66	74.11
BC4CHEMD	10.55	71.05	14.93	73.85	86.53	83.75	88.85	86.19
BC5CDR	11.75	60.37	12.58	67.51	69.62	73.66	74.83	74.34
BioNLP11EPI	31.14	78.64	42.31	81.51	85.71	85.57	84.64	86.68
BioNLP11ID	11.00	62.38	10.06	68.92	71.41	71.62	71.08	71.96
BioNLP13CG	12.39	49.15	12.53	52.68	55.23	63.15	64.19	67.23
BioNLP13GE	26.10	78.80	25.00	81.82	84.77	84.29	83.74	85.58
BioNLP13PC	12.40	69.29	12.59	71.89	68.11	68.49	70.42	71.97
BioNLP09	32.51	78.17	30.51	82.71	87.48	86.39	85.16	86.33
CRAFT	8.07	37.35	8.60	40.38	49.67	51.56	63.72	63.35
Ex-PTM	16.06	74.32	47.93	76.15	82.92	84.11	82.32	83.81
JNLPBA	20.15	57.61	19.77	59.54	64.46	63.63	71.65	70.45
NCBI	38.69	68.82	30.46	79.35	93.02	90.36	89.51	86.46
linnaeus	28.75	58.69	36.94	67.29	93.81	92.50	94.43	70.57
Average	19.28	64.74	23.08	70.22	77.56	77.54	79.14	77.50
DI	DI 2006	12.67	50.19	13.30	49.54	13.54	55.28	12.60	50.10
POS	Genia	51.48	13.41	48.26	30.65	66.27	61.93	71.45	70.57
QA	BioASQ8b (factoid)	36.63	35.99	41.89	40.77	51.96	49.84	52.95	51.72
BioASQ8b (list)	14.99	20.91	19.66	29.38	40.14	29.59	38.96	34.68
BioASQ8b (yesno)	43.48	61.11	39.13	57.94	66.96	60.32	56.52	52.17
PubMedQA	17.32	19.28	25.16	23.26	27.68	25.86	27.12	24.96
Average	28.11	34.32	31.46	37.84	46.68	41.40	43.89	40.88
RE	ChemProt	61.64	72.02	66.07	64.91	66.01	55.22	76.86	77.38
DDI	85.53	77.37	85.53	81.37	46.99	55.41	87.39	73.04
Average	73.59	74.70	75.80	73.14	56.50	55.31	82.12	75.21
SA	Medical Drugs	33.29	63.48	24.51	63.66	43.41	31.58	37.31	49.50
SR	Accelerometer	76.76	77.78	75.35	68.06	83.80	73.61	72.54	70.83
Acromegaly	80.21	80.71	81.25	75.63	76.56	79.19	76.04	77.66
COVID	87.85	88.36	87.85	84.85	61.93	86.96	73.93	78.12
Cooking	88.29	87.08	87.80	87.56	81.95	87.08	80.98	82.78
HRT	85.86	86.02	85.61	75.12	89.08	81.99	83.87	80.81
Average	83.79	83.99	83.57	78.24	78.66	81.77	77.47	78.04
DC	HoC	17.06	19.87	17.98	27.13	46.94	52.36	88.53	81.51
RFI	RFHD 2014 (yesno)	57.21	51.78	57.21	51.50	43.02	66.35	43.86	66.46
RFHD 2014 (time-riskfactor)	54.51	64.22	52.75	63.37	66.18	59.60	66.18	62.70
Average	55.86	58.00	54.98	57.43	54.60	62.98	54.93	64.58
Average	-	37.31	60.64	39.24	63.38	66.75	67.98	69.81	70.23
Table 5: Comparison of few-shot learning results in terms of Rouge-L between single-task models and In-BoxBART for 32/100/1000 training samples per instruction tasks. All results are presented in%. S: Single-task model, I-BB: In-BoxBART, RFHD: Risk Factor for Heart Disease.
Category	Task	Metric	SOTA	Multi-Task
V-BB	I-BB
NER	AnatEM	F	91.61	33.50	84.61
BC2GM	F	83.47	50.86	75.03
BC4CHEMD	F	92.39	71.44	86.97
BC5CDR	F	90.50	70.11	75.24
BioNLP11EPI	F	88.66	52.85	88.04
BioNLP11ID	F	87.36	60.15	73.39
BioNLP13CG	F	90.16	53.88	65.09
BioNLP13GE	F	85.81	51.78	87.39
BioNLP13PC	F	91.65	51.61	67.77
BioNLP09	F	91.94	54.31	88.48
CRAFT	F	90.12	52.31	64.03
Ex-PTM	F	87.08	52.07	84.49
JNLPBA	F	79.19	68.60	70.26
NCBI	F	89.82	75.55	86.91
linnaeus	F	95.68	44.59	93.77
QA	BioASQ8 (list)	F	52.99	17.74	35.59
BioASQ8 (yesno)	F	89.95	62.61	68.25
RE	Chemprot	F	74.40	52.17	63.22
DDI	F	79.40	82.97	89.35
DC	HoC	F	85.30	49.51	82.53
RFI	RFHD 2014 (time-riskfactor)	F	92.76	0.97	85.28
Average	-	-	85.55	50.36	72.24
Table 6: The state-of-the-art (SOTA) results for each task compared with Vanilla-BoXBART and In-BoXBART. All results are in %. F:  $F_{1}$ -score, V-BB: Vanilla-BoXBART, I-BB: In-BoXBART, RFHD: Risk Factor for Heart Disease.