SlowGuess/NAACL_2021_OCR huggingface dataset:
Fangyu Liu*, Ehsan Shareghi\*, Zaiqiao Meng*, Marco Basaldella\*, Nigel Collier
*Language Technology Lab, TAL, University of Cambridge
$\diamond$ Department of Data Science & AI, Monash University  $\diamond$ Amazon Alexa
\*{f1399，zm324，nhc30}@cam.ac.uk
$\diamond$ ehsanshareghi@monash.edu  $\diamond$ mbbasald@amazon.co.uk
Despite the widespread success of self-supervised learning via masked language models (MLM), accurately capturing fine-grained semantic relationships in the biomedical domain remains a challenge. This is of paramount importance for entity-level tasks such as entity linking where the ability to model entity relations (especially synonymy) is pivotal. To address this challenge, we propose SAPBERT, a pretraining scheme that self-aligns the representation space of biomedical entities. We design a scalable metric learning framework that can leverage UMLS, a massive collection of biomedical ontologies with  $4\mathrm{M}+$  concepts. In contrast with previous pipeline-based hybrid systems, SAPBERT offers an elegant one-model-for-all solution to the problem of medical entity linking (MEL), achieving a new state-of-the-art (SOTA) on six MEL benchmarking datasets. In the scientific domain, we achieve SOTA even without task-specific supervision. With substantial improvement over various domain-specific pretrained MLMs such as BIOBERT, SCIBERT and PUB-MEDBERT, our pretraining scheme proves to be both effective and robust.
Biomedical entity $^2$  representation is the foundation for a plethora of text mining systems in the medical domain, facilitating applications such as literature search (Lee et al., 2016), clinical decision making (Roberts et al., 2015) and relational knowledge discovery (e.g. chemical-disease, drug-drug and protein-protein relations, Wang et al. 2018). The heterogeneous naming of biomedical concepts
Figure 1: The t-SNE (Maaten and Hinton, 2008) visualisation of UMLS entities under PUBMEDBERT (BERT pretrained on PubMed papers) & PUBMEDBERT+SAPBERT (PUBMEDBERT further pretrained on UMLS synonyms). The biomedical names of different concepts are hard to separate in the heterogeneous embedding space (left). After the self-alignment pretraining, the same concept's entity names are drawn closer to form compact clusters (right).
poses a major challenge to representation learning. For instance, the medication Hydroxychloroquine is often referred to as Oxichlorochine (alternative name), HCQ (in social media) and Plaquenil (brand name).
MEL addresses this problem by framing it as a task of mapping entity mentions to unified concepts in a medical knowledge graph. The main bottleneck of MEL is the quality of the entity representations (Basaldella et al., 2020). Prior works in this domain have adopted very sophisticated text pre-processing heuristics (D'Souza and Ng, 2015; Kim et al., 2019; Ji et al., 2020; Sung et al., 2020) which can hardly cover all the variations of biomedical names. In parallel, self-supervised learning has shown tremendous success in NLP via leveraging the masked language modelling (MLM)
<sup>3</sup>Note that we consider only the biomedical entities themselves and not their contexts, also known as medical concept normalisation/disambiguation in the BioNLP community.
*Work conducted prior to joining Amazon.
  1For code and pretrained models, please visit: https: //github.com/cambridgeltl/sapbert.
  2In this work, biomedical entity refers to the surface forms of biomedical concepts, which can be a single word (e.g. fever), a compound (e.g. sars-cov-2) or a short phrase (e.g. abnormal retinal vascular development).
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4228-4238 June 6-11, 2021. ©2021 Association for Computational Linguistics
objective to learn semantics from distributional representations (Devlin et al., 2019; Liu et al., 2019). Domain-specific pretraining on biomedical corpora (e.g. BIOBERT, Lee et al. 2020 and BIOMEGA-TRON, Shin et al. 2020) have made much progress in biomedical text mining tasks. Nonetheless, representing medical entities with the existing SOTA pretrained MLMs (e.g. PUBMEDBERT, Gu et al. 2020) as suggested in Fig. 1 (left) does not lead to a well-separated representation space.
To address the aforementioned issue, we propose to pretrain a Transformer-based language model on the biomedical knowledge graph of UMLS (Bodenreider, 2004), the largest interlingua of biomedical ontologies. UMLS contains a comprehensive collection of biomedical synonyms in various forms (UMLS 2020AA has  $4\mathrm{M}+$  concepts and  $10\mathrm{M}+$  synonyms which stem from over 150 controlled vocabularies including MeSH, SNOMED CT, RxNorm, Gene Ontology and OMIM). We design a self-alignment objective that clusters synonyms of the same concept. To cope with the immense size of UMLS, we sample hard training pairs from the knowledge base and use a scalable metric learning loss. We name our model as Self-aligning pretrained BERT (SAPBERT).
Being both simple and powerful, SAPBERT obtains new SOTA performances across all six MEL benchmark datasets. In contrast with the current systems which adopt complex pipelines and hybrid components (Xu et al., 2020; Ji et al., 2020; Sung et al., 2020), SAPBERT applies a much simpler training procedure without requiring any pre- or post-processing steps. At test time, a simple nearest neighbour's search is sufficient for making a prediction. When compared with other domain-specific pretrained language models (e.g. BIOBERT and SCIBERT), SAPBERT also brings substantial improvement by up to  $20\%$  on accuracy across all tasks. The effectiveness of the pretraining in SAPBERT is especially highlighted in the scientific language domain where SAPBERT outperforms previous SOTA even without fine-tuning on any MEL datasets. We also provide insights on pretraining's impact across domains and explore pretraining with fewer model parameters by using a recently introduced ADAPTER module in our training scheme.
Figure 2: The distribution of similarity scores for all sampled PUBMEDBERT representations in a minibatch. The left graph shows the distribution of + and - pairs which are easy and already well-separated. The right graph illustrates larger overlap between the two groups generated by the online mining step, making them harder and more informative for learning.
2 Method: Self-Alignment Pretraining
We design a metric learning framework that learns to self-align synonymous biomedical entities. The framework can be used as both pretraining on UMLS, and fine-tuning on task-specific datasets. We use an existing BERT model as our starting point. In the following, we introduce the key components of our framework.
Formal Definition. Let  $(x,y) \in \mathcal{X} \times \mathcal{Y}$  denote a tuple of a name and its categorical label. For the self-alignment pretraining step,  $\mathcal{X} \times \mathcal{Y}$  is the set of all (name, CUI $^5$ ) pairs in UMLS, e.g. (Remdesivir, C4726677); while for the fine-tuning step, it is formed as an entity mention and its corresponding mapping from the ontology, e.g. (scratchy throat, 102618009). Given any pair of tuples  $(x_i, y_i), (x_j, y_j) \in \mathcal{X} \times \mathcal{Y}$ , the goal of the self-alignment is to learn a function  $f(:, \theta): \mathcal{X} \to \mathbb{R}^d$  parameterised by  $\theta$ . Then, the similarity  $\langle f(x_i), f(x_j) \rangle$  (in this work we use cosine similarity) can be used to estimate the resemblance of  $x_i$  and  $x_j$  (i.e., high if  $x_i, x_j$  are synonyms and low otherwise). We model  $f$  by a BERT model with its output [CLS] token regarded as the representation of the input. $^6$  During the learning, a sampling procedure selects the informative pairs of training samples and uses them in the pairwise metric learning loss function (introduced shortly).
Online Hard Pairs Mining. We use an online hard triplet mining condition to find the most
In UMLS, CUI is the Concept Unique Identifier.
<sup>6</sup>We tried multiple strategies including first-token, mean-pooling, [CLS] and also NOSPEC (recommended by Vulić et al. 2020) but found no consistent best strategy (optimal strategy varies on different *BERTs).
4https://www.nlm.nih.gov/research/uml5s/knowledge_sources/metathesaurus/release/statistics.html
informative training examples (i.e. hard positive/negative pairs) within a mini-batch for efficient training, Fig. 2. For biomedical entities, this step can be particularly useful as most examples can be easily classified while a small set of very hard ones cause the most challenge to representation learning. We start from constructing all possible triplets for all names within the mini-batch where each triplet is in the form of  $(x_{a}, x_{p}, x_{n})$ . Here  $x_{a}$  is called anchor, an arbitrary name in the minibatch;  $x_{p}$  a positive match of  $x_{a}$  (i.e.  $y_{a} = y_{p}$ ) and  $x_{n}$  a negative match of  $x_{a}$  (i.e.  $y_{a} \neq y_{n}$ ). Among the constructed triplets, we select out all triplets that violate the following condition:
$$
\left\| f \left(x _ {a}\right) - f \left(x _ {p}\right) \right\| _ {2} <   \left\| f \left(x _ {a}\right) - f \left(x _ {n}\right) \right\| _ {2} + \lambda , \tag {1}
$$
where  $\lambda$  is a pre-set margin. In other words, we only consider triplets with the negative sample closer to the positive sample by a margin of  $\lambda$ . These are the hard triplets as their original representations were very far from correct. Every hard triplet contributes one hard positive pair  $(x_{a}, x_{p})$  and one hard negative pair  $(x_{a}, x_{n})$ . We collect all such positive & negative pairs and denote them as  $\mathcal{P}, \mathcal{N}$ . A similar but not identical triplet mining condition was used by Schroff et al. (2015) for face recognition to select hard negative samples. Switching-off this mining process, causes a drastic performance drop (see Tab. 2).
Loss Function. We compute the pairwise cosine similarity of all the BERT-produced name representations and obtain a similarity matrix  $\mathbf{S} \in \mathbb{R}^{|\mathcal{X}_b| \times |\mathcal{X}_b|}$  where each entry  $\mathbf{S}_{ij}$  corresponds to the cosine similarity between the  $i$ -th and  $j$ -th names in the mini-batch  $b$ . We adapted the Multi-Similarity loss (MS loss, Wang et al. 2019), a SOTA metric learning objective on visual recognition, for learning from the positive and negative pairs:
$$
\begin{array}{l} \mathcal {L} = \frac {1}{| \mathcal {X} _ {b} |} \sum_ {i = 1} ^ {| \mathcal {X} _ {b} |} \left(\frac {1}{\alpha} \log \left(1 + \sum_ {n \in \mathcal {N} _ {i}} e ^ {\alpha \left(\mathbf {S} _ {i n} - \epsilon\right)}\right) \right. \tag {2} \\ \left. + \frac {1}{\beta} \log \left(1 + \sum_ {p \in \mathcal {P} _ {i}} e ^ {- \beta (\mathbf {S} _ {i p} - \epsilon)}\right)\right), \\ \end{array}
$$
where  $\alpha, \beta$  are temperature scales;  $\epsilon$  is an offset applied on the similarity matrix;  $\mathcal{P}_i, \mathcal{N}_i$  are indices
of positive and negative samples of the anchor  $i$ .<sup>8</sup>
While the first term in Eq. 2 pushes negative pairs away from each other, the second term pulls positive pairs together. This dynamic allows for a re-calibration of the alignment space using the semantic biases of synonymy relations. The MS loss leverages similarities among and between positive and negative pairs to re-weight the importance of the samples. The most informative pairs will receive more gradient signals during training and thus can better use the information stored in data.
3 Experiments and Discussions
Data Preparation Details for UMLS Pretraining. We download the full release of UMLS 2020AA version.<sup>9</sup> We then extract all English entries from the MRCONSO.RFF raw file and convert all entity names into lowercase (duplicates are removed). Besides synonyms defined in MRCONSO.RFF, we also include tradenames of drugs as synonyms (extracted from MRREL.RRF). After pre-processing, a list of 9,712,959 (name, CUI) entries is obtained. However, random batching on this list can lead to very few (if not none) positive pairs within a mini-batch. To ensure sufficient positives present in each mini-batch, we generate offline positive pairs in the format of (name<sub>1</sub>, name<sub>2</sub>, CUI) where name<sub>1</sub> and name<sub>2</sub> have the same CUI label. This can be achieved by enumerating all possible combinations of synonym pairs with common CUIs. For balanced training, any concepts with more than 50 positive pairs are randomly trimmed to 50 pairs. In the end we obtain a training list with 11,792,953 pairwise entries.
UMLS Pretraining Details. During training, we use AdamW (Loshchilov and Hutter, 2018) with a learning rate of  $2\mathrm{e} - 5$  and weight decay rate of  $1\mathrm{e} - 2$ . Models are trained on the prepared pairwise UMLS data for 1 epoch (approximately 50k iterations) with a batch size of 512 (i.e., 256 pairs per mini-batch). We train with Automatic Mixed Precision  $(\mathrm{AMP})^{10}$  provided in PyTorch 1.7.0. This takes approximately 5 hours on our machine (con
<sup>8</sup>We explored several loss functions such as InfoNCE (Oord et al., 2018), NCA loss (Goldberger et al., 2005), simple cosine loss (Phan et al., 2019), max-margin triplet loss (Basaldella et al., 2020) but found our choice is empirically better. See App. §B.2 for comparison.
9 https://download.nlm.nih.gov/uml5s/kss/2020AA/uml5s-2020AA-full.zip
10. https://pytorch.org/docs/stable/amp.html
7Most of Hydroxychloroquine's variants are easy: Hydroxychlorochin, Hydroxychloroquine (substance), Hidroxicloroquina, but a few can be very hard: Plaquenil and HCO.
model	scientific language	social media language
NCBI	BC5CDR-d	BC5CDR-c	MedMentions	AskAPatient	COMETA
@1	@5	@1	@5	@1	@5	@1	@5	@1	@5	@1	@5
vanilla BERT (Devlin et al., 2019)	67.6	77.0	81.4	89.1	79.8	91.2	39.6	60.2	38.2	43.3	40.4	47.7
+ SAPBERT	91.6	95.2	92.7	95.4	96.1	98.0	52.5	72.6	68.4	87.6	59.5	76.8
BIOBERT (Lee et al., 2020)	71.3	84.1	79.8	92.3	74.0	90.0	24.2	38.5	41.4	51.5	35.9	46.1
+ SAPBERT	91.0	94.7	93.3	95.5	96.6	97.6	53.0	73.7	72.4	89.1	63.3	77.0
BLUEBERT (Peng et al., 2019)	75.7	87.2	83.2	91.0	87.7	94.1	41.6	61.9	41.5	48.5	42.9	52.9
+ SAPBERT	90.9	94.0	93.4	96.0	96.7	98.2	49.6	73.1	72.4	89.4	66.0	78.8
CLINICALBERT (Alsentzer et al., 2019)	72.1	84.5	82.7	91.6	75.9	88.5	43.9	54.3	43.1	51.8	40.6	61.8
+ SAPBERT	91.1	95.1	93.0	95.7	96.6	97.7	51.5	73.0	71.1	88.5	64.3	77.3
SCIBERT (Beltagy et al., 2019)	85.1	88.4	89.3	92.8	94.2	95.5	42.3	51.9	48.0	54.8	45.8	66.8
+ SAPBERT	91.7	95.2	93.3	95.7	96.6	98.0	50.1	73.9	72.1	88.7	64.5	77.5
UMLSBERT (Michalopoulos et al., 2020)	77.0	85.4	85.5	92.5	88.9	94.1	36.1	55.8	44.4	54.5	44.6	53.0
+ SAPBERT	91.2	95.2	92.8	95.5	96.6	97.7	52.1	73.2	72.6	89.3	63.4	76.9
PUBMEDBERT (Gu et al., 2020)	77.8	86.9	89.0	93.8	93.0	94.6	43.9	64.7	42.5	49.6	46.8	53.2
+ SAPBERT	92.0	95.6	93.5	96.0	96.5	98.2	50.8	74.4	70.5	88.9	65.9	77.9
supervised SOTA	91.1	93.9	93.2	96.0	96.6	97.2	OOM	OOM	87.5	-	79.0	-
PUBMEDBERT	77.8	86.9	89.0	93.8	93.0	94.6	43.9	64.7	42.5	49.6	46.8	53.2
+ SAPBERT	92.0	95.6	93.5	96.0	96.5	98.2	50,8	74.4	70.5	88.9	65.9	77.9
+ SAPBERT (ADAPTER13%)	91.5	95.8	93.6	96.3	96.5	98.0	50.7	\(75.0^†\)	67.5	87.1	64.5	74.9
+ SAPBERT (ADAPTER1%)	90.9	95.4	\(93.8^†\)	\(96.5^†\)	96.5	97.9	\(52.2^†\)	74.8	65.7	84.0	63.5	74.2
+ SAPBERT (FINE-TUNED)	92.3	95.5	93.2	95.4	96.5	97.9	50.4	73.9	\(89.0^†\)	\(96.2^†\)	\(75.1\ (\text{81.1}^{\dagger})\)	\(85.5\ (\text{86.1}^{\dagger})\)
BioSYN	91.1	93.9	93.2	96.0	96.6	97.2	OOM	OOM	82.6	87.0	71.3	77.8
+(init. w/) SAPBERT	\(92.5^†\)	\(96.2^†\)	93.6	96.2	96.8	\(98.4^†\)	OOM	OOM	87.6	95.6	77.0	84.2
Table 1: Top: Comparison of 7 BERT-based models before and after SAPBERT pretraining (+ SAPBERT). All results in this section are from unsupervised learning (not fine-tuned on task data). The gradient of green indicates the improvement comparing to the base model (the deeper the more). Bottom: SAPBERT vs. SOTA results. Blue and red denote unsupervised and supervised models. Bold and underline denote the best and second best results in the column. “†” denotes statistically significant better than supervised SOTA (T-test,  $\rho < 0.05$ ). On COMETA, the results inside the parentheses added the supervised SOTA's dictionary back-off technique (Basaldella et al., 2020). “-”: not reported in the SOTA paper. “OOM”: out-of-memory (192GB+).
figurations specified in App.  $\S \mathrm{B}.4)$  . For other hyperparameters used, please view App.  $\S \mathrm{C}.2$
Evaluation Data and Protocol. We experiment on 6 different English MEL datasets: 4 in the scientific domain (NCBI, Doğan et al. 2014; BC5CDR-c and BC5CDR-d, Li et al. 2016; MedMentions, Mohan and Li 2018) and 2 in the social media domain (COMETA, Basaldella et al. 2020 and AskAPatient, Limsopatham and Collier 2016). Descriptions of the datasets and their statistics are provided in App.  $\S A$ . We report Acc@1 and Acc@5 (denoted as @1 and @5) for evaluating performance. In all experiments, SAPBERT denotes further pretraining with our self-alignment method on UMLS. At the test phase, for all SAPBERT models we use nearest neighbour search without further fine-tuning on task data (unless stated otherwise). Except for numbers reported in previous papers, all results are the average of five runs with different random seeds.
Fine-Tuning on Task Data. The red rows in Tab. 1 are results of models (further) fine-tuned on the training sets of the six MEL datasets. Similar to pretraining, a positive pair list is generated through traversing the combinations of mention and all ground truth synonyms where mentions are from
the training set and ground truth synonyms are from the reference ontology. We use the same optimiser and learning rates but train with a batch size of 256 (to accommodate the memory of 1 GPU). On scientific language datasets, we train for 3 epochs while on AskAPatient and COMETA we train for 15 and 10 epochs respectively. For BIOSYN on social media language datasets, we empirically found that 10 epochs work the best. Other configurations are the same as the original BIOSYN paper.
3.2 Main Results and Analysis
*BERT + SAPBERT (Tab. 1, top). We illustrate the impact of SAPBERT pretraining over 7 existing BERT-based models (*BERT = {BIOBERT, PUBMEDBERT, ...})*. SAPBERT obtains consistent improvement over all *BERT models across all datasets, with larger gains (by up to  $31.0\%$  absolute Acc@1 increase) observed in the social media domain. While SCIBERT is the leading model before applying SAPBERT, PUBMEDBERT+SAPBERT performs the best afterwards.
SAPBERT vs. SOTA (Tab. 1, bottom). We take PUBMEDBERT+SAPBERT (w/wo fine-tuning) and compare against various published SOTA results (see App. §C.1 for a full listing of 10 baselines)
which all require task supervision. For the scientific language domain, the SOTA is BIOSYN (Sung et al., 2020). For the social media domain, the SOTA are Basaldella et al. (2020) and GEN-RANK (Xu et al., 2020) on COMETA and AskAP- patient respectively. All these SOTA methods combine BERT with heuristic modules such as tfidf, string matching and information retrieval system (i.e. Apache Lucene) in a multi-stage manner.
Measured by  $\mathrm{Acc}_{@1}$ , SAPBERT achieves new SOTA with statistical significance on 5 of the 6 datasets and for the dataset (BC5CDR-c) where SAPBERT is not significantly better, it performs on par with SOTA (96.5 vs. 96.6). Interestingly, on scientific language datasets, SAPBERT outperforms SOTA without any task supervision (fine-tuning mostly leads to overfitting and performance drops). On social media language datasets, unsupervised SAPBERT lags behind supervised SOTA by large margins, highlighting the well-documented complex nature of social media language (Baldwin et al., 2013; Limsopatham and Collier, 2015, 2016; Basaldella et al., 2020; Tutubalina et al., 2020). However, after fine-tuning on the social media datasets (using the MS loss introduced earlier), SAPBERT outperforms SOTA significantly, indicating that knowledge acquired during the self-aligning pretraining can be adapted to a shifted domain without much effort.
The ADAPTER Variant. As an option for parameter efficient pretraining, we explore a variant of SAPBERT using a recently introduced training module named ADAPTER (Houlsby et al., 2019). While maintaining the same pretraining scheme with the same SAPBERT online mining + MS loss, instead of training from the full model of PUBMEDBERT, we insert new ADAPTER layers between Transformer layers of the fixed PUBMEDBERT, and only train the weights of these ADAPTER layers. In our experiments, we use the enhanced ADAPTER configuration by Pfeiffer et al. (2020). We include two variants where trained parameters are  $13.22\%$  and  $1.09\%$  of the full SAPBERT variant. The ADAPTER variant of SAPBERT achieves comparable performance to full-model-tuning in scientific datasets but lags behind in social media datasets, Tab. 1. The results indicate that more parameters are needed in pretraining for knowledge transfer to a shifted domain, in our case, the social media datasets.
The Impact of Online Mining (Eq. (1)). As suggested in Tab. 2, switching off the online hard pairs mining procedure causes a large performance drop in @1 and a smaller but still significant drop in @5. This is due to the presence of many easy and already well-separated samples in the mini-batches. These uninformative training examples dominated the gradients and harmed the learning process.
configuration	@1	@5
Mining switched-on	67.2	80.3
Mining switched-off	52.3↓14.9	76.1↓4.2
Table 2: This table compares PUBMED-BERT+SAPBERT's performance with and without online hard mining on COMETA (zeroshot general).
Integrating SAPBERT in Existing Systems. SAPBERT can be easily inserted into existing BERT-based MEL systems by initialising the systems with SAPBERT pretrained weights. We use the SOTA scientific language system, BIOSYN (originally initialised with BIOBERT weights), as an example and show the performance is boosted across all datasets (last two rows, Tab. 1).
We present SAPBERT, a self-alignment pretraining scheme for learning biomedical entity representations. We highlight the consistent performance boost achieved by SAPBERT, obtaining new SOTA in all six widely used MEL benchmarking datasets. Strikingly, without any fine-tuning on task-specific labelled data, SAPBERT already outperforms the previous supervised SOTA (sophisticated hybrid entity linking systems) on multiple datasets in the scientific language domain. Our work opens new avenues to explore for general domain self-alignment (e.g. by leveraging knowledge graphs such as DBpedia). We plan to incorporate other types of relations (i.e., hypernymy and hyponymy) and extend our model to sentence-level representation learning. In particular, our ongoing work using a combination of SAPBERT and ADAPTER is a promising direction for tackling sentence-level tasks.
We thank the three reviewers and the Area Chair for their insightful comments and suggestions. FL is supported by Grace & Thomas C.H. Chan Cambridge Scholarship. NC and MB would like to
acknowledge funding from Health Data Research UK as part of the National Text Analytics project.