Can NLI Provide Proper Indirect Supervision for Low-resource Biomedical Relation Extraction?
Year: 2023

Abstract:
Two key obstacles in biomedical relation extraction (RE) are the scarcity of annotations and the prevalence of instances without explicitly pre-defined labels due to low annotation coverage. Existing approaches, which treat biomedical RE as a multi-class classification task, often result in poor generalization in low-resource settings and do not have the ability to make selective predictions on unknown cases but give a guess from seen relations, hindering the applicability of those approaches. We present NBR, which converts biomedical RE as a natural language inference formulation to provide indirect supervision. By converting relations to natural language hypotheses, NBR is capable of exploiting semantic cues to alleviate annotation scarcity. By incorporating a ranking-based loss that implicitly calibrates abstinent instances, NBR learns a clearer decision boundary and is instructed to abstain on uncertain instances. Extensive experiments on three widely-used biomedical RE benchmarks, namely ChemProt, DDI, and GAD, verify the effectiveness of NBR in both full-shot and low-resource regimes. Our analysis demonstrates that indirect supervision benefits biomedical RE even when a domain gap exists, and combining NLI knowledge with biomedical knowledge leads to the best performance gains.1

heading: 1 Introduction
text: In silico studies of biology and medicine have primarily relied on machines’ understanding of relations between various molecules and biomolecules. For instance, disease-target prediction requires accurate identification of the association between the drug target and the disease (Bravo et al., 2015), and drug-drug interaction recognition is essential for polypharmacy side effect studies (Herrero-Zazo et al., 2013). Due to the complexity and high cost
1Code is released at https://github.com/luka-group/ NLI_as_Indirect_Supervision
of human curation of such biomedical knowledge (Krallinger et al., 2017; Bravo et al., 2015), there has been a growing interest in the field of biomedical relation extraction (RE), a task of automatically inferring the relations between biomedical entities described in domain-specific corpora.
However, two obstacles remain in training a reliable biomedical RE model. First, biomedical RE often suffers from insufficient and imperfect annotations, due to that the annotation process is very challenging and requires expert annotators to identify complex structures from lengthy and sophisticated biomedical literature. The existing biomedical learning resources either require very costly expert annotations (Krallinger et al., 2017) or resort to weak supervision (Bravo et al., 2015). The insufficiency and imperfection of annotations inevitably cause existing state-of-the-art (SOTA) biomedical RE systems (Yasunaga et al., 2022; Peng et al., 2019; Tinn et al., 2021, inter alia), though showing satisfactory results in a fully supervised setting, to result in poor generalization regarding the more common low-resource regime in this domain. For example, Han et al. (2018) showed that model performance deteriorated quickly as the number of instances for each relation drops, hindering the applicability of those approaches in real-world scenarios. Second, given that biomedical RE annotations tend to be incomplete or have low coverage, it is difficult for models to learn a clear decision boundary (Gardner et al., 2020). Specifically, in many scenarios where the described biomedical entities are not related in the context, the model may fail to abstain but give a guess from seen relations (Xin et al., 2021; Kamath et al., 2020). An overconfident model can be particularly harmful in high-stakes fields such as medicine, where incorrect predictions can have severe direct consequences for patients.
Recently, indirect supervision (Roth, 2017; He et al., 2021; Levy et al., 2017; Lu et al., 2022; Li et al., 2019) is proposed that leverages supervision
2450
light color . (2) For training NBR calculates the entailment scores for each relation candidate and optimizes the score of the ground-truth relation to rank over the scores of other candidates. (3) For inference NBR computes entailment scores of each relation and returns the one with the maximum entailment score.
signals from resource-rich source tasks to enhance resource-limited target tasks. In this approach, the training and inference pipeline of the target task is transformed into the formulation of the source task, thus introducing additional supervision signals not accessible in the target task. Recent works (Li et al., 2022; Yin et al., 2020; Sainz et al., 2021) transfer cross-task learning signals from the Natural Language Inference (NLI) task. The NLI task aims at determining whether the hypothesis can be entailed given the premise, and inductive bias of NLI models learns adaptive generalized logical reasoning which aligns well with the goal of biomedical RE. On the other hand, traditional direct supervision on the biomedical RE fails to capture semantic information of relations since they are merely transformed to logits of a classifier. By converting relations to meaningful hypotheses in NLI, the indirectly supervised method bypasses this shortage and can adapt the the preexisting inductive bias of NLI-finetuned models to make meaningful predictions based on relation semantics (Huang et al., 2022; Chen et al., 2020). This critically benefits the generalizability of the model in low-resource regimes where limited direct supervision signals are provided (Sainz et al., 2021) to remedy insufficient annotations. However, previous studies focus on general domain tasks and explore little in specific domains such as biomedical. Moreover, to maximize the utility of indirect supervision, it is found that incorporating task knowledge into the model, i.e. NLI model that is trained on NLI data, yields the best performance (Li et al., 2022; Sainz et al., 2021). Yet, biomedical NLI is rarely available and whether general domain NLI can provide strong indirect supervising signals to specific target domains remains unexplored. This study presents a general learning framework, dubbed NLI improved Biomedical Relation Extraction (NBR), to enhance biomedical RE with indirect supervision from general domain NLI task. Fig. 1 illustrates the structure of NBR. Specifically, given an input sentence, NBR reformulates RE to NLI by treating the input as the premise while verbalizing each relation label into template-based natural language hypotheses. NBR learns to rank the relations based on the entailment scores such that the hypothesis of a correct relation should be scored higher than those of any incorrect ones. Furthermore, to learn a fine-grained, instance-aware decision boundary, NBR deploys ranking-based loss for implicit abstention calibration that handles abstinent relations in the dataset. During inference, the relation whose verbalized hypothesis achieved the highest score becomes the prediction. NBR fully exploits indirect supervision from NLI and performs exceptionally well even in low-resource scenarios. Our contributions are three-fold: First, to the best of our knowledge, this is the first work to leverage indirect supervision from NLI on biomedical RE. Instead of solely relying on provided RE annotations, NBR leverages additional supervision signals from NLI indirect supervision and
can generalize well in low resource regimes. Second, we show that NBR provides a proper indirect supervision signal even if there is a domain gap between general NLI knowledge NBR trained on and biomedical downstream task. Third, we propose a new ranking-based loss that implicitly handles abstinent relations ubiquitous in biomedical RE by contrastively calibrating the score of abstinent instances. By extensive experiments on three commonly-used biomedical RE benchmarks, namely, ChemProt (Krallinger et al., 2017), DDI (Herrero-Zazo et al., 2013) and GAD (Bravo et al., 2015), we verify our contributions and show that general domain NLI can provide a proper supervision signal, especially in low resource settings where annotations are scarce. NBR provides consistent improvements on three datasets (1.10, 1.79, and 0.96 points of F1 improvement respectively), and up to 34.25 points of F1 improvement in low-resource settings. Further analysis demonstrates that combing NLI knowledge with biomedical knowledge leads to the best performance gains.

heading: 2 Related Works
text: Biomedical relation extraction. Despite the growing availability of biomedical corpora on Web repositories, the main challenge remains in transforming those unstructured textual data into a rigidly-structured representation that includes interested entities and relations between them (Peng et al., 2019; Lee et al., 2020; Tinn et al., 2021). However, knowledge curation for this purpose is often costly and requires expert involvement (Krallinger et al., 2017; Herrero-Zazo et al., 2013; Bravo et al., 2015). To address this issue, biomedical RE techniques are developed to automate this process. Most existing works mainly conduct supervised fine-tuning language models pretrained on relevant corpus e.g. PubMed abstracts and MIMIC-III clinical notes, on annotated biomedical RE corpora (Tinn et al., 2021; Peng et al., 2019; Beltagy et al., 2019; Lee et al., 2020; Shin et al., 2020; Yasunaga et al., 2022). Two drawbacks of the aforementioned approach are: (1) it fails to capture the semantic interaction between relations and entities as relations are represented as integer indices (Chen et al., 2020; Huang et al., 2022), and (2) performance deteriorates as the number of training instances drops (Han et al., 2018).
Indirect supervision. Indirect supervision (Roth, 2017; He et al., 2021) transfers supervision sig-
nals from a more resource-rich task to enhance a specific more resource-limited task. Often this line of work reformulates the training and inference pipeline of the target task into the form of the source task to facilitate the cross-task signal transfer. Levy et al. (2017) demonstrate that relation extraction can be solved using machine reading comprehension formulation. Similarly, Li et al. (2019) and Lu et al. (2022) further show that relation extraction performance can be improved by multi-turn question answering and summarization, respectively. Recently Sainz et al. (2021) and Li et al. (2022) propose to leverage indirect supervision from the NLI task. LITE (Li et al. (2022)) enhances entity typing by incorporating NLI and a learning-to-rank training objective while Sainz et al. (2021) observes the benefits of indirect supervision in low-resource relation extraction. As discussed, NLI aligns well with relation extraction, but to the best of our knowledge, there is no prior work that investigates the effectiveness of indirect supervision when there is a domain gap between the target task and the source task, e.g. biomedical domain and general domain in this study.

heading: 3 Method
text: We hereby present NBR. We discuss how to frame relation extraction as a NLI task in §3.2, illustrate how to leverage cross-domain NLI knowledge in §3.3, and lastly provide an optional explicit abstention detector to handle abstinent instances in §3.4.

heading: 3.1 Problem Formulation
text: The RE model takes a sentence x with two mentioned entities e1, e2 as input, and predicts the relation y between e1, e2 from the label space Y that includes all considered relations. The dataset D consists of both non-abstinent instances where y ∈ Y , and abstinent instances2 where y =⊥. A successful RE model should abstain for abstinent instances and accurately predict y for non-abstinent instances.

heading: 3.2 Relation Extraction with NLI
text: Following Sainz et al. (2021), we reformulate the RE task as a NLI task, allowing cross-task transfer of indirect supervision signals from NLI resources. An overview of our pipeline is visualized in Fig. 1.
2Indicating that either there is no relation between e1, e2 or the relation is not one of the relation labels defined in Y .
Decompose RE to NLI queries. The NLI model takes in a premise and a hypothesis, both in natural language, and outputs a logit indicating if the premise either “entails,” “contradicts” the hypothesis or the inference relation is “neutral.” We decompose an instance (x, e1, e2) into |Y| + 1 NLI queries, each about a candidate relation. We formulate the RE input sentence x as the premise and a verbalized sentence describing the candidate relation as the hypothesis.
Verbalizing relations to hypotheses. For each relation y ∈ Y ∪ {⊥}, we verbalize y as a natural language hypothesis ν(y). Contextual textual representations of labels provide more semantic signals and are thus more understandable by a language model (LM) compared to the relation name itself or discrete relation label index used in standard classification methods (Chen et al., 2020; Huang et al., 2022).
Entity mentions in biomedical RE are mostly domain-specific terms that rarely appear in the LM’s pre-training corpus. The relations are always defined between entities of certain types, e.g. between a gene complex and another chemical in ChemProt (Krallinger et al., 2017) or between two drugs in DDI (Herrero-Zazo et al., 2013). Thus, each entity mention is replaced by typed entity masks such as @GENE$ following Gu et al. (2021) and Peng et al. (2019).3 The replacement enables the LM to capture semantic information of the types and avoid using poorly trained representations for rare biomedical terms.
As demonstrated by recent studies (Yeh et al., 2022; Li et al., 2022; Sainz et al., 2021), picking a good verbalizer for each relation may affect performance. Specifically, we design several types of templates (details and performances are provided in Appx. §D) listed below, each containing the two typed entity masks:
1. Simple Template verbalizes relation between two entities with “is-a” phrase.
2. Descriptive Template provides a contextual description of the relation.
3. Demonstration Template includes a randomly sampled trainset exemplar with the same relation.
4. Descriptive+Demonstration Template combines both the Descriptive description and the sampled exemplar.
3We choose to use our typed entity mask design instead of the “entity mask” (Zhou and Chen, 2022) as it has been observed to produce better performance in those tasks with NLI. We do not consider the entity masks as special tokens.
5. Learned Prompt Template (Yeh et al., 2022) learns optimal discrete tokens for description. We observe that Descriptive Template performs the best empirically (Tab. 7).
Confidence scoring. For each relation label y ∈ Y ∪ {⊥}, we calculate the confidence score of whether relation y holds by s(y) = fNLI(x [SEP] ν(y)) where [SEP] is a special token separating x (premise) and ν(y) (hypothesis). fNLI is a transformer-based NLI model that encodes the input and produces logits that correspond plausibility of premise entailing hypothesis.
Abstention as a separate label. We treat ⊥ as a separate relation label and verbalize it explicitly, which is analogous to how supervised biomedical RE treats ⊥ as an additional label (Yasunaga et al., 2022; Peng et al., 2019). An explicit template relieves the burden of incorporating both stop condition and label discriminative power into scores of Y labels. Training objective. Recent works in contrastive learning show that InfoNCE loss benefits efficient learning from negative examples (Robinson et al., 2021; Wang et al., 2022; Zhang and Stratos, 2021; Zhou et al., 2021; Ma et al., 2023, 2021). Motivated by the intuition that positive instances should be ranked higher than negative instances with regard to the anchor instance, in each step we sample n negative relations {y1, . . . , yn} ⊆ Y ∪ {⊥ } \ {y} and compute s(y1), . . . , s(yn), and optimize ground truth relation’s entailment score to be ranked higher. Specifically, we optimize the following InfoNCE loss
LNCE = ∑
(x,y)∈D `NCE(x, y) (1)
, ∑
(x,y)∈D − ln
 
exp(s(y)/τ)
exp(s(y)/τ) + n∑
i=1
exp(s(yi)/τ)
 ,
in which temperature τ controls focus on harder negatives. In practice, learning from all possible negatives performs the best.
In pilot experiments, we observed that the model was prone to be misled by the vast number of abstinent instances in the dataset, leading to deteriorated performance. To alleviate such abstinent v.s. nonabstinent imbalance, we introduce a margin-based Abstention Calibration regularization to penalize over-confident abstinent instances while encouraging non-abstinent instances. Concretely, if relation
is not⊥, we calibrate the score of⊥ such that s(⊥) is suppressed; otherwise, we control⊥ to be ranked higher than other relations.
LAC = ∑
(x,y)∈D `AC(x, y) (2)
`AC(x, y) ,    n∑ i=1 `rank(s(y), s(yi); γ), if y =⊥
`rank(s(y), s(⊥); γ), otherwise
where the ranking loss `rank(x1, x2; γ) learns to project x1 higher than x2 by a margin γ. Training with this objective, NBR can be viewed as combining an implicit abstention calibrator and s(⊥) as a learnable instance-aware threshold. The final training loss is LNCE + λLAC where non-negative hyperparameter λ controls the strength of abstention calibration.
Inference. NBR gathers hypotheses verbalized from every relation and performs ranking among the entailment scores of each hypothesis. Then the relation whose verbalized hypothesis achieves the highest score is selected as the final prediction.

heading: 3.3 Cross-Domain NLI Fine-tuning
text: In order to maximize the benefit of NLI formulation, it is advised to use models trained on targetdomain NLI dataset (Li et al., 2022; Sainz et al., 2021). However, available biomedical NLI training resource is limited. As a remedy, we experiment with fine-tuning NLI models on two commonly used general domain NLI datasets, namely MNLI (Williams et al., 2018) and SNLI (Bowman et al., 2015), instead. Empirically we found strong evidence (§4.2, §4.4) that general-domain NLI knowledge can still be beneficial in the biomedical domain even if a domain gap exists.

heading: 3.4 Explicit Abstention Detector
text: Training with aforementioned LAC (Eq. 2) makes NBR an implicit abstention calibrator. As an optional post-process step, we can further improve NBR by introducing an Explicit Abstention Dector (EAD). This is analogous to the “no-answer reader” component used in previous works that detect abstinent instances explicitly (Back et al., 2020; Hu et al., 2019; Kundu and Ng, 2018).
EAD is essentially another instance of NBR trained separately on the same train set, but changing relation labels into binary “has relation” versus “no relation” (⊥). A new verbalization template is created for “has relation”. For inference, we collect
all differences sEAD(⊥)−sEAD(“has relation”) on the dev set. Then we iterate each difference as a threshold, and for one instance in the test set, EAD predicts ⊥ only if the difference of such instance exceeds the threshold. Once EAD is trained, NBR and EAD are combined using a simple heuristic: resort to NBR only when EAD prediction is not ⊥ (Appx. §C). In this manner, even if EAD makes a false positive prediction, since NBR still retains the ability to flag ⊥, such error can be recovered. Otherwise, we trust EAD prediction since it specializes in abstention prediction.

heading: 4 Experiments
text: In this section, we discuss our experiment setup (§4.1) and evaluation results (§4.2), followed by detailed ablation studies (§4.3) and analyses (§4.4).

heading: 4.1 Experimental Setup
text: Dataset and evaluation metric. We conduct experiments on three sentence-level biomedical RE datasets contained in the widely-used BLURB benchmark (Gu et al., 2021). ChemProt (Krallinger et al., 2017) consists of PubMed abstracts corpora with five high-level chemicalprotein interaction annotations. DDI (HerreroZazo et al., 2013) studies drug-drug interaction and specializes in pharmacovigilance built from PubMed abstracts. GAD (Bravo et al., 2015) is a semi-labeled dataset created using Genetic Association Archive and consists of gene-disease associations.
There are multiple variants of the datasets used by existing literature that differ by data statistics or evaluation protocol (Dong et al., 2021; Phan et al., 2021; Beltagy et al., 2019; Yeh et al., 2022; Peng et al., 2020; Xu et al., 2022) as described in Appx. §B, we adopt the most popular setting used by Gu et al. (2021) and give dataset statistics in Tab. 5. Most of entity pairs are labeled as ⊥ without an explicit relation label.4 This setting is realistic since the model must identify a relation’s existence first. Following Gu et al. (2021), we use the micro F1 score calculated across all nonabstinent instances as the evaluation metric.
Baselines. We compare against the various baselines (Appx. §A), mostly classification-based approaches that use |Y|+ 1-way classification head on top of a biomedical-pretrained LM. Sci-Five
4In train set, ChemProt contains 77% abstinent while DDI contains 85%.
results in red and the best results of direct supervision in cyan .
(Phan et al., 2021) generates the relation label as a seq-to-seq conditional generation formulation.
Our method. We term three variants of NBR: • NBRNLI using NLI formulation (§3.2) with BioLinkBERTlarge (Yasunaga et al., 2022) backbone that pretrained on biomedical corpus.
• NBRNLI+FT further cross-domain fine-tunes (§3.3) BioLinkBERT on two general domain NLI datasets. The model retains biomedical domain knowledge and learns relevant NLI knowledge.
• NBRNLI+FT+EAD assembles NBRNLI+FT with a separately trained EAD component (§3.4).
We choose BioLinkBERT as the pretrained LM due to its supremacy in performance on various biomedical domain tasks, but we emphasize that our approach is agnostic to backbone models.

heading: 4.2 Experimental Results
text: NLI provides helpful indirect supervision. We report the comparison between NBR and baselines in Tab. 1. Overall, NBRNLI+FT+EAD achieves SOTA performance on all three datasets, with 1.10, 1.79, and 0.96 points F1 improvement on ChemProt, DDI, and GAD respectively. Strong
performance gains verify the effectiveness of reformulating biomedical RE as NLI. NLI supervision signals from the general domain are transferred to enhance the biomedical RE learning signals. By verbalizing relations into natural language hypothesis, NBR leverages the preexisting inductive bias of NLI-finetuned models to make informed predictions based on relation semantics.
We further compare the performance of our model’s variants. First, due to the prevalence of abstinent instances on the datasets, we notice that by explicitly detecting the abstinent instances, assembling EAD (§3.4) with NBRNLI+FT improves performance on ChemProt and DDI. This is likely because explicitly detecting ⊥ by a separate EAD model reduces the burden on NBRNLI+FT to predict relations and identify abstinent instances at the same time. Second, we show that cross-domain fine-tuning (§3.3) is vital. Compared to NBRNLI, which is not trained on NLI datasets, NBRNLI+FT resulted in significant improvements in F1 across three datasets. This demonstrates that having prior NLI knowledge allows better utilization of the NLI formulation. Lastly, we note that NBRNLI is outperformed by its direct supervision counterpart,
the best model in red and the best of direct supervision in cyan . Columns are ordered by the number of training instances. 3 indicates the results are from our re-implementation to conform to our evaluation metric.
namely BioLinkBERT on ChemProt and GAD. The possible reason could be that the model needs to learn to perform NLI tasks on top of the RE task without NLI training, which leads to shallower supervision signals. However we observe that generally, and especially in low-resource regimes, NBRNLI improves over direct supervision (§4.4).
Indirect supervision from NLI shines particularly under low-resource. We evaluate the NBR under zero- and few-shot settings in Tab. 2. Following existing works (Peng et al., 2020; Xu et al., 2022), we train the model with 0, 8 and 50 shots and 1% and 10% of training instances. We note that classification-based methods could not adapt to the zero-shot setting.
Our experimental results show that all three variants of NBR consistently achieve strong performance across all few-shot settings on all datasets, e.g. 34.25 points F1 improvement on 8-shot ChemProt. The performance of direct supervision models deteriorates dramatically as the num-
ber of training instances decreases, due to the limited learning signals. On the contrary, NBR effectively leverages indirect supervision to transform richer NLI signals to improve the RE performance. Additionally verbalized hypotheses provide valuable semantic cues for prediction. We also observe similar patterns as the full-set experiments: using NLI knowledge learned from NLI training data improves the performance of NBRNLI, and combing EAD with NBRNLI+FT leads to further performance gains.
Lastly, we note that as the number of training instances increases, the benefits of indirect supervision tend to decrease. This suggests that given sufficient training signals, direct supervision can learn effectively, and the marginal returns of introducing additional NLI signals become smaller. In practical settings where biomedical annotations are scarce, learning with indirect supervision can lead to better performance.

heading: 4.3 Ablation Study
text: We perform ablation studies on model components on ChemProt and DDI using 1% and 100% training data in Tab. 3. (1) InfoNCE LNCE (Eq. 1) is essential. Replacing LNCE with ranking loss sum i.e. ∑n i=1 `rank(s(y), s(yi); γ) deteriorate performance. These results confirm the effectiveness of InfoNCE in learning from negative samples (Robinson et al., 2021; Wang et al., 2022). (2) LAC (Eq. 2) is vital. Given the prevalence of abstinent relations in the two datasets, it is easy for models to be misled by abstinent instances since they impose stronger learning signals. We specifically notice 1% settings have a larger performance drop, which might be caused by the fact that detecting abstention is harder when the quantity of other labels and their associated learning signals is reduced. (3) We further consider a variant that replaces LNCE with ranking loss sum, removes LAC and uses only one negative sample, which corresponds to LITE (Li et al., 2022) that uses NLI indirect supervision for the general domain entity typing task. We observe further performance degradation, which again verifies the effectiveness of the two losses. Lastly (4) we fine-tune BioLinkBERT on the biomedical MedNLI (Romanov and Shivade, 2018). Despite being domain-relevant, we observe performance drops compared to fine-tuning on general domain NLI datasets. We hypothesize that perform drops might be caused by (a) MedNLI being relatively small as MNLI is 35x larger and (b) low coverage on relevant knowledge e.g. only 11.77% of ChemProt entities are mentioned in MedNLI. Therefore even if MedNLI provides both NLI knowledge and biomedical knowledge, the gain is insignificant.

heading: 4.4 Analysis
text: In this section, we first show the benefits of indirect supervision, then illustrate two key ingredients for
effective indirect supervision gains: biomedical domain knowledge and NLI knowledge.
NLI formulation benefits, even without additional NLI resources. In Tab. 4, we demonstrate the effectiveness of NLI formulation using two backbones without NLI knowledge: RoBERTa (Liu et al., 2019) and BioLinkBERT.
We observe that even if models lack NLI formulation adaption, NLI formulation outperforms original RE formulation in most settings, particularly in low-resource settings. When data is limited, it is challenging for direct supervision methods to access sufficient supervision signals. In contrast, the model can leverage the semantic information in the natural language hypothesis with the NLI formulation. Additionally, BioLinkBERT consistently outperformed RoBERTa in the same settings, despite RoBERTalarge having larger parameters, suggesting the importance of domain knowledge.
Two key ingredients of indirect supervision for biomedical RE. We identify two potential factors that contribute to the effective usage of indirect supervision for biomedical RE: 1) biomedical domain-specific knowledge; and 2) NLI knowledge to adapt to the NLI formulation. To test the importance of these two kinds of knowledge, in Fig. 2 we evaluate on 1% and 100% of ChemProt and DDI the four combinations: RoBERTa and RoBERTa fine-tuned on NLI, and BioLinkBERT and BioLinkBERT fine-tuned on NLI.
We first observe that BioLinkBERT fine-tuned on NLI datasets behaves the best across all four settings, indicating the importance of both pieces of knowledge. When the learning signal is limited, the model can dynamically load-balance both forms of knowledge to make educated predictions. Secondly, we note that RoBERTa, which lacks both biomedical and NLI knowledge, consistently performs the worst, except for 1% ChemProt. Finally,
it is difficult to determine whether the domain or NLI knowledge is more important in biomedical RE, as the relative importance may depend on the specific dataset or the knowledge requirements of each input.

heading: 5 Conclusion
text: We present a novel method NBR that leverages indirect supervision by cross-task transfer learning from NLI tasks to improve the biomedical RE task. NBR verbalizes relations to natural language hypotheses so that model is able to exploit semantic information to make informed predictions. Furthermore, NBR adopts a ranking-based abstinent calibration loss that penalizes overconfident abstinent instances while encouraging non-abstinent instances, thus being capable of abstaining on uncertain instances. Extensive experiments on three widely-used biomedical RE benchmarks demonstrate that NBR is effective in both full-set and low-resource settings. We further investigate two key ingredients for effective NLI indirect supervision on biomedical RE. Future work could involve further investigation of other indirect supervision approaches and automatic relation template generation based on prompt learning.

heading: Acknowledgement
text: We appreciate the reviewers for their insightful comments and suggestions. Jiashu Xu was supported by the Center for Undergraduate Research in Viterbi Engineering (CURVE) Fellowship. Mingyu Derek Ma was supported by the AFOSR MURI grant #FA9550-22-1-0380, the Defense Advanced Research Project Agency (DARPA) grant #HR00112290103/HR0011260656, and a Cisco
Research Award. Muhao Chen was supported by the NSF Grant IIS 2105329, by the Air Force Research Laboratory under agreement number FA8750-20-2-10002, by a subaward of the INFER Program through UMD ARLIS, an Amazon Research Award and a Cisco Research Award. Computing of this work was partly supported by a subaward of NSF Cloudbank 1925001 through UCSD.
Limitations
This work investigates using NLI as indirect supervision for biomedical RE. Experiments suggest two key ingredients in high-performing indirect supervision biomedical RE are biomedical knowledge and NLI knowledge. To this goal, we need to access a language model that is pretrained on biomedical domain corpus, which requires computational resources. Compared to general domain ones, models pretrained on a specific domain are often limited in variety. Further to learn NLI knowledge additional cross-domain fine-tuning needs to be conducted, which results in additional computational overhead.
During inference NBR requires #label times of forward passes to yield prediction since NBR needs to evaluate entailment scores for each verbalized relation. Compared to standard supervision which only requires one pass for every instance, inference cost and training cost are higher in a factor of # label. Higher inference cost hinders applicability in a number of scenarios e.g. real-time applications. Additionally, the high inference cost makes it difficult to deploy machine learning models in resource-constrained environments, such as edge devices with limited processing power.
Lastly, since NBR is sensitive to templates, designing an effective template is crucial for perfor-
mance. However, currently human involvement is required to design templates for each relation. As the number of relations increases, human involvement might become costly and time-consuming. Moreover, it is not easy to test the effectiveness of templates as no objective metric exists, and the only way to assess the quality is to test the templates.

heading: A Models
text: Baselines We categorize compared baselines by the pretrain corpus. • PubMed abstracts: BioM-ELECTRA (Alrowili
and Vijay-Shanker, 2021). • PubMed abstracts and PMC full-text articles:
Bio-BERT (Lee et al., 2020); BioM-BERT (Alrowili and Vijay-Shanker, 2021); BioMegatron (Shin et al., 2020) pretrain on commercialcollection subset of PMC; PubMed-BERT (Tinn et al., 2021) fine-tune model released by Gu et al. (2021), which is pretrain on those corpus;
Sci-Five (Phan et al., 2021) is T5 based model that learns to conditionally generate relation labels in textual form directly; BioLinkBERT (Yasunaga et al., 2022) further proposes a pretraining task of link prediction, which enables the model to learn multi-hop knowledge.
• PubMed abstracts and MIMIC-III clinical notes: BLUE-BERT (Peng et al., 2019).
• Semantic Scholar: Sci-BERT (Beltagy et al., 2019) pretrain BERT on scientific corpus consists of 1.14M full-text papers from Semantic Scholar; BioRE-Prompt (Yeh et al., 2022) initializes from RoBERTa trained on the Semantic Scholar and learns a three-token prompt for each relation and infers by finding the best matching prompt. We use model checkpoints released by huggingface (Wolf et al., 2020). Specifically, we use bionlp/bluebert_pubmed_mimic_uncased_L24_H1024_A-16 for BLUE-BERT (Peng et al., 2019), allenai/scibert_scivocab_uncased for Sci-BERT (Beltagy et al., 2019), dmis-lab/biobert-basecased-v1.2 for BioBERT (Lee et al., 2020), microsoft/BiomedNLPPubMedBERT-base-uncased-abstract-fulltext for PubMed-BERT (Tinn et al., 2021), razent/SciFive-large-Pubmed_PMC for Sci-Five (Phan et al., 2021), sultan/BioM-ALBERT-xxlarge-PMC for BioMALBERT (Alrowili and Vijay-Shanker, 2021), sultan/BioM-BERT-PubMed-PMC-Large for BioM-BERT (Alrowili and Vijay-Shanker, 2021), michiyasunaga/BioLinkBERT-large for BioLink-BERT (Yasunaga et al., 2022), and cnut1648/biolinkbert-large-mnli-snli for BioLink-BERT that is fine-tuned on SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018).
NBR We run experiments on Quadro RTX 8000 GPU. AdamW optimizer (Loshchilov and Hutter, 2019) with learning rate 1e-5 is used, and we set margin γ = 0.7, temperature τ = 0.01 and calibration (Eq. 2) strength λ in sweep from 0.001 to 10. We train models for 300 epochs. Models are
evaluated every ten epochs on the dev set, and the best checkpoint is selected to infer on the test set.

heading: B Evaluation Difference
text: As mentioned in §4, several previous works use a different evaluation metric and variants of the datasets, rendering it hard to compare with previous work. In this section, we describe the main differences in the dataset. We first report the statistics of the dataset we use in this work in Tab. 5. For other works that use variants of the datasets:
• BLUE-BERT (Peng et al., 2019)’s variant of ChemProt and DDI. Their ChemProt contains 4,154/2,416/3458 train/val/test instances and five relations, while their DDI contains 2,937/1,004/979 train/val/test instances and four relations.
• Sci-BERT (Beltagy et al., 2019) uses a variant of ChemProt with 4,169/2,427/3,449 train/val/test instances and contains 13 relations.
• Dong et al. (2021) and (Peng et al., 2020) use a variant of ChemProt with 4,168/2,427/3,469 train/val/test instances and 13 relations.
• Xu et al. (2022) use a variant of ChemProt with 14 relations
• BioRE-Prompt (Yeh et al., 2022) also use ChemProt provided by Gu et al. (2021), but does not exclude abstinent instances.

heading: C EAD Details and Variants
text: Since only relations for EAD is “has relation” versus “no relation”, instead of Eq. 1 and Eq. 2 used in NBR, EAD learns only via ranking loss `rank(s(y), s(y
′); γ) where y is the ground-truth while y′ is the opposite relation.
We discuss several heuristics in assembling NBR and EAD. The best performing heuristic is simple: only resort to NBR when EAD prediction is not ⊥. In other words, the final prediction is ⊥ only if EAD prediction is ⊥; otherwise, return the prediction of NBR. We evaluate other more sophisticated heuristics:
• Voting: Predict ⊥ only when both NBR and EAD predict ⊥; otherwise, return NBR’s prediction.
• Confident: Predict⊥ only when EAD predicts⊥ and confidence score sEAD(⊥) is higher than confidence score sNBR(⊥); otherwise, return NBR’s prediction. Note that if EAD makes a false positive, NBR is still able to recover if sNBR(⊥) is the highest.
• Super-confident: Predict ⊥ when EAD predicts ⊥; if sEAD(⊥) > sNBR(⊥) return highest-scored non-abstinent relation argmaxy∈Y sNBR(y); otherwise prediction of NBR.
• Classification: Use a classification-based model (with the same backbone as NBRNLI+FT), and use logits for confidence score under the simple heuristic.
In Tab. 6, we observe that a more complicated heuristic does not entail better performance gains. Note that designing a contextual description for “has relation” is challenging and our template is a simple phrase such as “relation exists between.” Surprisingly, we still found assembling NBR with EAD empirically outperforms classification-based abstention detector. We credit enhanced performance to additional semantic information captured by the verbalized template.

heading: D Template for datasets
text: We provide details for each of the templates investigated in this work.
1. Simple Template: This template verbalizes the relation between two entities as a “is-a” phrase, e.g. “@CHEMICAL$ is a downregulator to @GENE$.”
2. Descriptive Template: We manually curate a description for each relation that contains more context, e.g. “Downregulator @CHEMICAL$ is designed as an inhibitor of @GENE$.”
3. Demonstration Template: Motivated by fewshot exemplars used for in-context learning, the demonstration template includes a randomly sampled context sentence whose entities hold the same relation, e.g. “Relation described between @CHEMICAL$ to @GENE$ is similar to <example sentence>.”
4. Descriptive + Demonstration: We include both a contextual description and an incontext exemplar by simple concatenating.
5. Learned Prompt Template: Borrowed from Yeh et al. (2022), which leverage prompt tuning with rules (Han et al., 2022) to learn optimal discrete tokens to fill in [MASK] within the template such as “@CHEMICAL$ [MASK] [MASK] [MASK] @GENE$.”
We further provide templates for NBR on three datasets: ChemProt (Tab. 10), DDI (Tab. 9) and GAD (Tab. 8).
Lastly, Tab. 7 shows the effect of template design. The descriptive template, which involves manual efforts, leads to the best performance. The simple template preserves the relation name semantics and yields strong performance. On the other hand, while popular in in-context learning works, we find that the demonstration template or descriptive + demonstration template consistently underperforms the descriptive template, indicating that incorporating examples in NLI hypothesis is not helpful potentially due to limited diversity. The learned prompt template used by Yeh et al. (2022) does not outperform the manually constructed descriptive template. Finally, we note that changing templates can lead to significant performance perturbations, our experiments suggest that evaluating the quality of templates in low-resource settings such as 1% can be effective and efficient. We note that the contextual template might not be optimal and we leave how to automatically pick the optimal template as future work.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: No section number, after Section 5 Conclusion
7 A2. Did you discuss any potential risks of your work? We do not see significant risks in our work

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract before Section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 Experiments

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 Experiments

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 4 Experiments

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4 Experiments and Appendix B Evaluation Difference
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We refer readers who interested in those information to the original paper
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We refer readers who interested in those information to the original paper
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Table 5
C 3 Did you run computational experiments? Section 4 Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A Models
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4 Experiments and Appendix A Models

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 Experiments

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 Experiments
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We note that the contextual template might not be optimal and we leave how to automatically pick the optimal template as future work. ACL 2023 Responsible NLP Checklist