Solving the Right Problem is Key for Translational NLP: A Case Study in UMLS Vocabulary Insertion
Authors: Bernal Jimenez Gutierrez
Yuqing Mao
Vinh Nguyen
KIN WAH FUNG
Yu Su
Olivier Bodenreider
Venue: EMNLP 2023 Findings (2023)
Keywords: biomedical NLP
translational NLP
synonymy prediction
knowledge base construction
TL;DR: In this work, we highlight the importance of well-aligned problem formulation for translational NLP through the UMLS vocabulary insertion (UVI) task in which our re-formulation leads to significant progress in the development of practical systems.

Abstract: As the immense opportunities enabled by large language models become more apparent, NLP systems will be increasingly expected to excel in real-world settings. 
However, in many instances, powerful models alone will not yield translational NLP solutions, especially if the formulated problem is not well aligned with the real-world task.
In this work, we study the case of UMLS vocabulary insertion, an important real-world task in which hundreds of thousands of new terms, referred to as atoms, are added to the UMLS, one of the most comprehensive open-source biomedical knowledge bases. 
Previous work aimed to develop an automated NLP system to make this time-consuming, costly, and error-prone task more efficient. 
Nevertheless, practical progress in this direction has been difficult to achieve due to a problem formulation and evaluation gap between research output and the real-world task. 
In order to address this gap, we introduce a new formulation for UMLS vocabulary insertion which mirrors the real-world task, datasets which faithfully represent it and several strong baselines we developed through re-purposing existing solutions.
Additionally, we propose an effective rule-enhanced biomedical language model which enables important new model behavior, outperforms all strong baselines and provides measurable qualitative improvements to editors who carry out the UVI task.
We hope this case study provides insight into the considerable importance of problem formulation for the success of translational NLP solutions.

Solving the Right Problem is Key for Translational NLP: A Case Study in UMLS Vocabulary Insertion
Bernal Jiménez Gutiérrez' Yuqing Mao', Vinh Nguyen?, Kin Wah Fung", Yu Su', Olivier Bodenreider 'Tee Ohio State University, NNational Library of Medicine
tmmeezgut rerrez. 1, su. 809) @osu.edu (yugling ng..ing.a ingiinn.. mao, vf nh. nguyen, kwfung) ên hh. gov, 61 wwer@nlm.nnhhg...
Abstract
Updated UMLS
As the immense opportunities enabled by large language models become more apparent, NLP systems will be increasingly expected to excel real-world settings. However, in many in- stances, powerful models alone will not yield translational NLP solutions, especially if the formulated problem not well aligned with the real-world task. In this work, we study the case of UMLS vocabulary insertion, an important real-world task in which hundreds of thousands of new terms, referred to as atoms, are added to the UMLSS, one the most comprehensive open-source biomedical knowledge bases (Bo- denreider, 2004), Previous work aimed to de- velop an automated NLP system to make this time-consuming, costly, and error-prone task more efficient. Nevertheless, practical progress in this direction has been difficult to achieve due to problem formulation and evaluation gap between research output and the rorldworld task. In order to address this gap, introduce a new formulation for UMLS vocabulary inser- tion which mirrors the real-world task, datasets which faithfully represent it and several strong baselines we developed through re-purposing existing solutions. Additionally, we propose an effective rule-enhanced biomedical language model which enables important new model be- havior, outperforms all strong basclines and provides measurable qualitative improvements editors who carry out the UVI task, We hope this case study provides insight into the consid- erable importance of problem formulation for the success of translational NLP solutions.'
New Atoms (Q)
atom;
UMLS (K)
concept, :
atom
atom
atom
concept:
atom
concept,
atom Fx
atom,
atom
atom K
concept,
attom
atomm concept. atom?
(a)
Synonymous
F UvA (
atom;
atom,
Synonymous
concept,
F Lvi
UMLS
atom,
New Concept
Figure I: The UMLS update process in (a) introduces atoms from individual sources into the original UMLS as synonyms of existing concepts or entirely new con- cepts. The UVA task is formulated as binary synonymy prediction (b) and thus unable tackle the real- world update task addressed by our UVI formulation.
these technologies, NLP systems will be expected to perform in a wide variety of real-world scenar- ios. However, even LLMs get increasingly more capable, it is unlikely that they will lead trans- lational solutions alone. Although many aspects are crucial for an NLP system's success, we use this work to highlight one key aspect of building real-world systems which is sometimes taken for granted: formulating a problem in a way that is well-aligned with its real-world counterpart. To explore the effect of this key step in building real- world NLP systems, provide case study the important task of UMLS vocabulary insertion.
1 Introduction
The public release of large language model (LLM) products like ChatGPT has triggered a wave of en- thusiasm for NLP technologies. As more people discover the wealth of opportunities enabled by
The Unified Medical Language System (UMLS) (Bodenreider, 2004) is a large-scale biomedical knowledge base that standardizes over 200 med- ical vocabularies. The UMLS contains approxi- mately 16 million source-specific terms, referred
*Part of this work was done while interning at the NLM. 'Our code is available at https: / thub. con/ OSU-NLP NLP-Group.
ry- insertion.

---

as atoms, grouped into over million unique concepts, making it one of the most comprehen- sive publicly available biomedical knowledge bases and a crucial resource for biomedical interoperabil- ity. Many of the vocabularies which make up the UMLS are independently updated to keep up with the rapidly advancing biomedical research field. In order for this essential public resource to remain up-to-date, team of expert editors painstakingly identify which new atoms should be integrated into existing UMLS concepts or added as new concepts, as shown in Figure la. This process, which we refer to as UMLS vocabulary insertion (UVI), in- volves inserting an average of over 300,000 new atoms into the UMLS and is carried out twice a year before each new UMLS version release.
work (Bajaj al., 2022), we find that biomedical language models (LMs) outperform previous UVA models. Thirdly, we discover that rule-based and deep learning frameworks greatly improve each other's performance. Finally, inspired by biomed- ical entity linking and the complementary nature of our baseline systems, we propose a null-aware and rule-enhancee re-ranking model which outper- forms all other methods and achieves low error rates on all five UMLS update datasets. To show our model's practical utility, we quantitatively eval- uate its robustness across UMLS update versions and semantic domains, conduct a comparative eval- uation against the second best method and carry out qualitative error analysis to more deeply under- stand its limitations. We hope that our case study helps researchers and practitioners reflect on the importance of problem formulation for the transla- tional success of NLP systems.
Despite its importance, scale and complexity, this task is accomplished by editors using lexical information (McCray al., 1994), synonymy in- formation provided by the source vocabularies and their own expertise. order to improve this pro- cess, much work has been done to augment it with modern NLP techniques. In Nguyen et al. (2021), the authors introduce datasets and models which explore the task of UMLS vocabulary alignment (UVA). As seen in Figure b, the authors formulate the UVA task binary synonymy prediction task between two UMLS atoms, while the real-world task requires the whole UMLS to be considered and a concept to be predicted for each new atom (unless it is deemed a new concept atom). Un- fortunately, while the UVA task has successfully explored biomedical synonymy prediction, its for- mulation has made it unable to yield practical im- provements for the UVI process.
2 Related Work
2.1 UMLS Vocabulary Alignment
Previous work improve UMLS editing formu- lates the problem as biomedical synonymy predic- tion through the UMLS vocabulary alignment task (Nguyen et al., 2021, 2022; Wijesiriwardene et al., 2022). These investigations find that deep learning methods are effective at predicting synonymy for biomedical terms, obtaining F1 scores above 90% (Nguyen et al., 2021). Although this formulation can help explore biomedical synonymy prediction, does not consider the larger UMLS updating task and thus the strong performance of these models does not transfer to wallworld tasks such as UVI.
Apart from the clear difference in scope between UVA and UVI shown in Figure 1b, major differ- ences in evaluation datasets contribute to the gap in UVA's applicability to the UVI task. In Nguyen et al. (2021), the authors built a synonymy pre- diction dataset with almost 200 million training and test synonym pairs to approximate the large- scale nature of UMLS editing. UVA dataset statis- tics can be found in Appendix A. Since the UVA test set was created using lexical similarity aware negative sampling, it does not hold the same dis- tribution as all the negative pairs in the UMLS. Since the UVI task considers all of the UMLS, UVA sampling leads to significant distribution shift between these tasks. This unfortunately di- minishes the usefulness of model evaluation on the UVA dataset for the real-world task. Surprisingly.
In this work, we attempt to address this gap with a novel UVI problem formulation, also depicted in Figure lb. Our formulation follows the real- world task exactly by predicting whether a new atom should be associated with an existing concept or identified as a new concept atom. We introduce five datasets taken directly from actual UMLS up- dates starting from the second half of 2020 until the end of 2022. These datasets enabled us to mea- sure the real-world practicality of our systems and led us to findings we could not have discovered otherwise. First, we find that adapting UVA mod- els perform the UVI task yields much higher error rates than in their original task, showing that their strong performance does not transfer to the woal-world setting. Second, contrary to previous

---

this gap results in biomedical language models like BioBERT (Lee et al., 2019) and SapBERT (Liu et al., 2021) underperforming previous UVA mod- els in the UVA dataset Bajaj et al. (2022) while outperforming them in our experiments.
medical vocabularies into the UMLS. In this task. each new term encountered in medical source vocabulary introduced into the UMLS as either a synonym of an existing UMLS concept or as an entirely new concept. In this section, we describe our formulation of the UVI task, the baselines we adapted from previous work, well thorough description of our proposed approach.
2.2 Biomedical Entity Linking
In the task of biomedical entity linking, terms men- tioned within text must be linked to existing con- cepts in a knowledge base, often UMLS. Our own task, UMLS vocabulary insertion, follows a similar process except for three key differences: 1) relevant terms come from biomedical vocabularies rather than text, 2) some terms can be new to the UMLS and 3) each term comes with source-specific infor- mation. Many different strategies have been used for biomedical entity linking such as expert-written rules (D'Souza and Ng, 2015), soaming-to-rrane methods (Leaman al., 2013), models that com- bine NER and
3.1 Problem Formulation
First, we define the version of the UMLS before the update as K == {cI, [c., Cn a set of unique UMLS concepts c. Each concept oa is defined as C == {at o where each atom aj. as they referred to by the UMLS, is defined as the jth source-specific synonym for the 7th concept in the UMLS.
In the UMLS Vocabulary Insertion (UVD) task, a set of m new atoms Q == {q1.... ..... must be integrated into the current set of concepts K. Thus, we can now define the UVI task as the following function I which maps new atom zj to its gold labelled concept Cyj if it exists in the old UMLS K or a null value if is a new concept atom, described by the following Equation 1.
signals (Leaman and Lu, 2016; Furrer et al., 2022) and language model fine-tuning (Liu et al., 2021; Zhang et al., 2022; Yuan et al., 2022). Due to the strong parallels be- tween biomedical entity- Miniky and our task, we leverage the best performing LM based methods for the UVI task Liu et al. (2021); Zhang et al. (2022); Yuan al. (2022). These methods fine- tune an LM to represent synonymy using embed- ding distance, enabling a nearest neighbor search to produce likely candidates for entity linking.
( 1 Cis Ø
if Cr, K otherwise
T(K, (j)
(1)
Experimental Setup
The first difference between biomedical entity linking and UVI is addressed by ignoring textual context done in Liu al. (2021), which we adopt as strong baseline. The second difference, that some new atoms can be new to the UMLS, is addressed by work which includes un-linkable entitics in the scope of their task (Ruas and Couto, 2022; Dong et al., 2023). In these, a cross-encoder candidate module introduced by Wu et al. (2020) is used to re-rank the nearest neighbors suggested by embedding methods like Liu et al. (2021) with extra candidate which represents that the entity is unlinkable, or in our case, a new concept atom. The third difference has no parallel in biomedical entity linking since mentions do not originate from specific sources and is therefore one of our contri- butions in $4.6.
4.1 Datasets
To evaluate the UVT task in the most realistic way possible, we introduce a set of five insertion sets Q which contain all atoms which are inserted into the UMLS from medical source vocabularies by expert editors twice a year. Due to their real-world nature, these datasets vary in size and new concept distribution depending on the number and type of atoms that are added to source vocabularies before every update as shown in Table We note that the version of the UMLS we use contains 8.5 rather than 16 million atoms because we follow previous work and only use atoms that are in English, come from active vocabularies and are non-suppressible, features defined by UMLS editors.
While most of our experiments focus on the UMLS 2020AB, we use the other four as test sets to evaluate temporal generalizability. We split the 2020AB insertion dataset into training, dev and test sets using a 50:25:25 ratio and the other in- sertion datasets using a 50:50 split into dev and
3 UMLS Vocabulary Insertion
We refer to UMLS Vocabulary Insertion (UVI) as the process of inserting atoms from updated or new

---

Original
Insertion
New Concepts
4.3 UVA Baselines
UMLS K
Set Q
We adapted several UVA specific system baselines for our UMLS vocabulary insertion task.
2020AB
8,521, ,20
430,135
260,058
2021AA
8,839,907
226,210
91,834
2021AB
3,835,147
455,493
218,933
Rule-based Approximation (RBA). (Nguyen et al., 2021) This system was designed to ap- proximate the decisions made by UMLS editors regarding atom synonymy using three simple rules. Two atoms were deemed synonymous if 1) they were labelled as synonyms in their source vocabularies, 2) their strings have identical normalized forms and compatible semantics (McCray et al., 1994) and 3) the transitive closure of the other two strategies. We thus define the function for the UVI task as follows. We first obtain an unsorted list of atoms a; in K deemed synonymous with qj by the RBA. We then group these atoms by concept to make smaller set of unique concepts Cj. Since this predicted concept list is unsorted, if it contains more than one potential concept, we randomly select one of them as the predicted concept & the RBA synonym list is empty, we deem the new atom as not existing in the current UMLS version.
2022AA
9,175,923
175,989
111,853
2022AB
9,082,515
275,842
188,984
Table I: UMLS Statistics from 2020AB to 2022AB. Our models are trained on the 2020AB insertion dataset.
test sets. We do stratified sampling to keep the distribution of semantic groups, categories defined by the UMLS, constant across splits within each insertion set. This is important since the distri- bution of semantic groups changes significantly across insertion datasets and preliminary studies showed that performance can vary substantially across categories. For details regarding the number examples in each split and the distribution of se- mantic groups across different insertion sets, refer to Appendix B.
4.2 Metrics
We report several metrics to evaluate our methods comprehensively the UVT task: accuracy, new concept metrics and existing concept accuracy. Accuracy. It measures the percentage of correct predictions over the full insertion set Q.
LexLM. (Nguyen al., 2021) The Lexical- Learning Model (LexL.M) system was designed as the deep learning alternative to the RBA and trained for binary synonymy prediction using the UVA training dataset. Their proposed model consists of an LSTM encoder over Weo (Zhang et al., 2019) embeddings which encodes two strings and calculates a similarity score between them. A threshold is used over the similarity score deter- mine the final synonymy prediction.
New Concept Metrics. These measure how well models predict new atoms as new concepts and they are described in Equation 2. The terms in Equation 2, subscripted by ne, refer to the number of true positive (TP), false positive (FP) and false negative (FNN examples, calculated by using the concept label as the positive class.
To adapt this baseline to the UVI task, we define the insertion function / as mapping new atom qj to the concept in K, Eag. containing the atom with the highest similarity score to zj based the LexLM representations. To allow the function I predict that zj does not exist in the current UMLS and should mapped to the empty set 2), we select a similarity threshold for the most similar concept under which qj is deemed a new atom. For faimeess in evaluation, the similarity threshold is selected using the 2020AB UVI training set.
TPhc
Pnc TP. + FPPc
(2)
TPnc
Rnc TPnc + FNnc
Existing Concept Accuracy. This metric shows model performance on atoms in Q which were linked by annotators to the previous version of UMLS K, as shown in Equation 3. Let Ne be the number of concepts in which were linked to concepts in K.
4.4 LM Baselines
Cgz
Cgs
if Coj K otherwise
Aer
Noc
Previous work finds that language models do not improve UVA performance (Bajaj et al., 2022). However, given our new formulation, we evaluate
BEEQ
Eyj == 1(Kgg))

---

concep
conceptapxe
conceptmpx
atom (q)
RBA
Re-Ranker RBA Signal
conce
0.25
concep
conceptana
+
concepT
0.20
conceptspaERTa
conceptapaaTT.
conceptsupaenra
0.30
UMLS (K)
SapBERT
concep
concept
New Cancapt
New Concept
0.10
Figure 2: Overall architecture for best performing approach the UVI task formulation. Our methodology leverages the best distance- based ranking model (SapBERT) as well as RBA signal. Additionally, our design allows new atoms be identified as new concepts by introducing 'New Concept' placeholder into candidate list given to the re-ranking module as shown above.
two language models in the more realistic UVI task using the same strategy described for the LexLM model above. For implementation details, we refer the interested reader to Appendix C.
atoms and candidates simultancously using lan- guage models, allowing for the encoding of one to be conditioned on the other. Our cross-encoder is based on PubMedBERT 2 and we use the most sim- ilar 50 atoms which represent unique concepts measured by the best bascline, the RBA system aug- mented with SapBERT ranking. More concretely, the atom which represents each candidate concept Oc, is appended new atom Qj and encoded as follows: [CLS] gj [SEP] ac.- Since the number of RBA candidates differs for every new atom, if the RBA produces less that 50 candidates, the re- maining candidates are selected from SapBERT's nearest neighbor candidates. We use the BLINK codebase (Wu al., 2020) train re-ranking module. More information about our implementa- tion can be found in Appendix C.
PubMedBERT (Gu et al., 2021). PubMedBERT is one of the most capable biomedical specific language models available due its from scratch pre-training on biomedical data as well as its specialized biomedical tokenizer.
SapBERT (Liu et al., 2021). SapBERT is a lan- guage model designed for biomedical entity linking concept normalization. was developed by fine- tuning the original PubMedBERT the 2020AA version of UMLS using contrastive learning ob- jective. This objective incentivizes synonymous entity reprexentations in UMLS to be more similar than non-synonymous ones.
4.6.1 Null Injection
In contrast with standard entity linking settings where every mention can linked to a relevant en- tity, UVI requires some mentions or new atoms to be deemed absent from the relevant set of entities. To achieve this in our re-ranking framework, we closely follow unlinkable biomedical entity link- ing methods (Dong et al., 2023; Ruas and Couto, 2022) and introduce new candidate, denoted by the NULL token, to represent the possibility that the atom is new the UMLS.
4.5 Augmented RBA
Given that the neural representation baselines dis- cussed above provide a ranking system missing from the RBA, we create a strong baseline by aug- menting the RBA system with each neural ranking baseline. In these simple but effective baselines, the concepts predicted by the RBA are ranked based on their similarity qj using cach neural baseline sys- tem. New concept prediction uses the same method employed by the original RBA model.
4.6.2 RBA Enhancement
4.6 Our Approach: Candidate Re-Ranking
Finally, given the high impact of the RBA system preliminary experiments, we integrate rule-based information into the candidate re-ranking learning. The RBA provides information in primarily two ways: 1) the absence of RBA synonyms sends atom being novel con-
Our candidate re-ran re-ranking approach is inspired by some entity linking systems which use two distinct steps: 1) candidate generation, which uses bi- encoder like the baselines described above, and candidate re-ranking, in which a more compu- tationally expensive model is used to rank the most similar concepts obtained by the bi-encoder. Other work (Wu et al., 2020) encodes both new
strong signal for cept the UMLS and 2) the candidate concepts
Prelliinaay results showed that PubMedBERT outper- forms SapBERT as a re-ranker.

---

New Concept
Existing Concept
Accuracy
Accuracy
Recall
Precision
F1
Rule Based Approximation (RBA)
70.1
99.0
90.5
94.6
26.3
LexLM
63.2
89.5
92.4
90.9
22.4
PubMedBERT
68.4
99.1
67.3
80.2
20.7
SapBERT
77.4
94.1
79.2
86.0
52.0
RBA + LexLM
80.4
99.0
90.5
94.6
51.6
RBA + PubMedBERT
83.7
99.0
90.5
91.6
60.0
RBA + SapBERT
90.7
99.0
90.5
94.6
76.1
Re- Re-Ranker (PubMedBERT)
85.5
96.3
91.6
93.9
68.4
RBA Signal
93.2
96.1
97.1
85.5
Table Comparison for rule-based, distance- and combined baselines against our re-ranking approaches both with and without RBA-signal over all metrics. All results reported above calculated the 2020AB UMLS insertion dataset. We find that all improvements of our best approach over the RBA+SapBERT baseline are very highly significant (p-value 0.001) based on a paired t-test with bootstrap resampling.
which the RBA predicted, rather than the ones pre- dicted based solely on lexical similarity, have higher chance of being the most appropriate con- cept for the new atom. Thus, we integrate these information elements into the cross-encoder by I) when no RBA synonyms exist, we append the string "(No Preferred Candidate)" to the new atom and 2) every candidate that was predicted by the RBA is concatenated with the string "(Preferred)" This way, the cross-encoder obtains access to vital RBA information while still being able to learn the decision-making flexibility which UML.S editors introduce through their expert knowledge.
Nevertheless, simple baseline which com- bines the strengths of neural models and the rule-based system obtains surprisingly strong results. This especially the case for augment- ing the RBA with SapBERT which obtains a num90% overall accuracy and existing concept accuracy of 76%. We note that the new concept recall and precision all RBA baselines the same since the same rule-based mechanism is used.
Our Approach. For the PubMedBERT-based re- ranking module, we find that the NULL injection mechanism enables it to outperform the models that rely solcly on lexical information (LcxLM, Pub- MedBERT and SapBERT) by wide margin. How- ever, it underperforms the best augmented RBA baseline substantially, underscoring the importance of RBA signal for the UVI task. Finally, we note that RBA enhancement allows the re-ranking mod- ule to obtain a 93.2% accuracy due to boosts existing concept accuracy and new concept preci- sion of almost 10% and 1% respectively. These improvements comes from several important fea- tures of our best approach which we discuss in more detail in $5.3, namely the ability flexibly determine when a new atom exists in the current UMLS even when has no RBA synonyms and to apply rules used by UMLS editors seen the model's training data. This substantial error reduc- tion indicates our method's potential as a useful tool for supporting UMLS editors.
5 Results & Discussion
In this section, we first discuss performance of our baselines and proposed methods on the UMLS 2020AB test set. We then evaluate the generaliz- ability of our methods across UMLS versions and biomedical subdomains. Finally, we provide com- parative evaluation and qualitative error analysis to understand our model's potential benefits and limitations.
5.1 Main Results
Baselines. As seen in Table 2, previous baselines such as RBA, LexLM and biomedical language models like PubMedBERT and SapBERT stay under the 80% mark in overall accuracy, with specially low performance in the existing concept accuracy metric. Even SapBERT, which is fine-tuned for the biomedical entity linking task, is unable obtain high existing concept and new concept prediction scores when using a simple optimal similarity threshold method.
5.2 Model Generalization
In this section, we note the robust generalization our re-ranking module across both UMLS versions

---

and semantic groups (semantic categories defined by the UMLS).
RBA + SapBERT
Re-Ranker
Semantic Group
RBA Signal
Across Versions. In Figure 3, we see that the best performing baseline RBA + SapBERT and our best method obtain strong performance across all five UMLS insertion datasets. Even though our pro- posed approach obtains the largest gains in the 2020AB set in which it trained, it achieves stable existing concept accuracy and new concept Fl score improvements across all sets and shows no obvious deterioration over time, demonstrating its practicality for future UMLS updates. Unfor- tunately. we do observe a significant dip in new concept F1 for all models in the 2021 AA dataset mainly due to the unusually poor performance of the RBA in one specific source, Current Procedural Terminology (CPT), for that version.
Living Beings
97.2
98.0
Chemicals & Drugs
81.1
93.7
Genes & Molecular Seq.
74.3
77.7
Disorders Procedures
92.1
97.7
82.6
84.3
Physiology
92.8
99.0
Concepts & Ideas Devices
89.1
97.2
90.7
97.4
Anatomy
95.1
98.3
Table 3: Accuracy by semantic group for the two highest performing UVI systems averaged over all development insertion sets from 2020AB to 2022AB.
the dataset details in Appendix B, our model's im- proved semantic group robustness vital for its potential in improving the efficiency of the UMLS update process.
Existing Concept Accuracy
New Concept F1
100
100
As for the categories in which our approach re- mained below 90% like Genes and Procedures, we find that they are mainly due to outlier insertion sets. Both the Genes and Procedures categories have one insertion set, 2022AA and 2021 AA re- spectively, in which the performance of both sys- tems drops dramatically due to weak RBA signal which our methodology was unable to correct for. We refer the interested reader to Appendix E for these results and more detailed discussion around this limitation.
75
75-
50
50
25
25
20A8 21AA
21AB
22AA
22AB
20A8 21AA 21AB 22AA 22AB
RBA SapBERT
RBA SapBERT
Re-Ranker RBA Signal
Figure 3: Existing concept accuracy (left) and con- cept Fl (right) of the best model from each baseline type and our best approach across UVI datasets from 2020AB 2022AB. All improvements over the best baseline are very highly significant (p-value < 0.001).
5.3 Comparative Evaluation
Across Subdomains. Apart from evaluating whether our proposed approach generalizes across UMLS versions, we evaluate how model perfor- mance changes across different semantic groups. Table 3 shows the results of our best baseline (RBA SapBERT) compared against our best proposed approach (Re-Ranker + RBA Signal) on the nine most frequent semantic groups averaged over all development insertion sets. We report the results in detail over all insertion sets in Appendix F. Our evaluation reveals that even though our best base- line performs quite well across several semantic groups, performance drops in challenging cate- gories like Drugs, Genes, Procedures and the more general Concepts & Ideas category. Our approach is able improve performance across most groups to above 90%, with the exception of Genes and Pro- cedures. Since the distribution of semantic groups can vary widely across UMLS updates, as seen in
As mentioned the main results, our best model outperforms the best baseline mainly through im- provements in existing concept accuracy and new concept precision. In Table 4, we report the distri- bution of 2,943 examples incorrectly predicted by RBA + SapBERT amended by our best approach. We note that a large majority, around 60%, of the corrections are concept linking corrections, new atoms which are linked to an existing concept cor- rectly while they were wrongly predicted as new concept atoms by the baseline. Most of the remain-
Correction Type
Correction %
Concept Linking Re-Ranking
59.5
35.9
New Concept Identification
1.6
Table 4: Distribution of examples incorrectly predicted by the best baseline amended by our best model.

---

Correction type
Atoms
Top RBA SupHERT Candidates
Error Type
New Atom
Top Best Model
Left ocbital region (Preferted)
Ausorpha cautllo (Preferrel)
(NEW CONCEPT]
Amorpha
Anerpho <angisapeent> (Prefervod)
Left orbital region
Structure petiorbiia region left eye othital grojer
condiostio
Amerphas sp.
Amnmphas
UMLS Emar
Left orbital
Re- Ranking
ctarabing/lloggaiin (Preforref)
Assosphottica
(Dplicatic Concepts)
Genostomatidae (Preferred)
Cyasthise- Thiogaatine
Caraaioo-Thioquuaa Rayinees (Preforrd)
Gomsstonatiine aailiatees
Gonseommnter (Proloneet)
(NEW CONCEPT)
Cytnaiine Mitorsttycne/ Thioguaiine
Gonichihys
Cyurshue Thloguanine
Protediglostomazidae
(NEW CONCEPT)
CONCEPT)
hysterccomm
removal right overy
abdominal bystesoctomyy with ressovzl right warry 'lotal hystereciomy right salpengo oxphorectomy
insterectonry right seploorectooyy
UEEa 400 MGMML
wreate % TOPICALLSOLLLUTT
unna 40% UREA 40RE TOP GEL
mighnal cutameons
Concept Linkiig WARFARIN
Tocal lnstoroctesyy with ressoval both
ovanies
Tiue Errors
Marffrra LNEW CONCEaPT)
INEW CONCEPT) VERUSES ACCOMPANIED EXANTHEM exantheate viral Esanthem
(ANTOVENN) 7.5MG TAB
WARFARIN (TARO)
Chai LEANTOMEN)
Esamthen caused buman echovizus (disorder)
Wurtsto Sodium 7.5 MG Oral Tustec (COUMADEN)
Whatl exantiicn duee to echoviras
Table 5: Some examples which were incorrectly pre- dicted by our best baseline (RBA+ + SapBERT), shown above in red, but corrected by our best proposed re- ranking model, shown above in green.
Table 6: Some examples which were incorrectly pre- dicted by our best proposed model, shown in red. Gold label concepts are marked with green. The first two rows show two errors caused by UMLS annotations while the final two are legitimate errors caused by com- plexity and ambiguity.
ing corrections, 35.9%, are re-ranking corrections based model's ability to re-rank gold con- cept over other candidate concepts. The final 5% comes from new concept identification corrections in which a new atom correctly identified as a new concept atom when it was incorrectly linked to an existing one by the best baseline.
5,4 Error Analysis
Given that our work focuses on specific practical application, in this section, we aim to more deeply understand how our approach can be effectively adopted by UMLS editors in their vocabulary in- sertion task. To this end, we recruited a biomedical terminology expert familiar with the UMLS vo- cabulary insertion process to analyze the practical effectiveness and limitations of our system.
The examples shown in Table 5 illustrate the benefits of our proposed approach more clearly. In the first two rows, we see two re-ranking correc- tions. In the first example, SapBERT incorrectly identifies '<eudicots>' as being closer to '<moth>' than '<angiosperm>' but our model has learned to interpret the disambiguation tags and correctly associates 'eudicots" with 'angiosperm' as levels of plant family classifications. In the second ex- ample, we observe that our trained model learns to link new atoms to concepts which have more comprehensive information such as the addition of the "Regimen" phrase. Although this is an editorial rule rather than an objective one, it is important to note that our model can adequately encode these.
We first studied the calibration of our best model's output as a way to understand its eгror detection abilities. As shown in detail in Appendix F, we see substantial drop in performance when model confidence, a softmax over candidate logit scores, drops below 90%. This drop could indi- cate that our model is well calibrated, however, our qualitative experiments reveal that this signal comes from a large number of annotation errors in the UMLS which are easily detected by our prob- lem formulation.
The final two rows in Table 5 show concept link- ing corrections. These examples illustrate the most important feature of our proposed model, the abil- ity to link new atoms to concepts even when the RBA would consider them new concept atom. In these instances, the model must determine whether all the features in the new atom are present in any potential candidates without support from the RBA. In these two examples, the model is able to cor- rectly identify synonymy by mapping 'removal of the right ovary' to 'right oophorectomy", 'NA' to 'Sodium' and 'TAB' to 'Oral Tablet.
We discovered this through a qualitative error analysis carried out with the help of the aforemen- tioned biomedical terminology expert. We chose three sets of 30 randomly chosen example errors with different model confidence scores: high (90%- 100%), medium (60%-70%) and low (30%-40%). Our expert editor reports several important findings. First, there was no substantial difference in exam- ple difficulty between different model confidence bins. Second, 70% of model errors are caused by the existence of UMLS concepts which have

---

phrases that are equivalent the new atoms, lead- ing to ambiguous examples which can be found the first section of Table 6. This arises from two types of annotation errors within the UMLS, either the new atom was incorrectly introduced into the UMLS or the phrase that is representing that concept was previouly introduced into UMLS incorrectly. Out of this study, the expert found 15 out of the 90 instances where model's sug- gestions lead detecting incorrect associations in the original UMLS vocabulary insertion process. This evaluation suggests that our model could be quite useful supporting quality assurance for the UMLS.
that our proposed approach is highly translational by providing evidence for its robustness across UMLS versions and biomedical subdomains, ex- ploring the reasons behind its superior performance over our baselines and carrying out a qualitative er- ror analysis to understand its limitations. We hope our case study highlights the significance of prob- lem formulation and offers valuable insights for researchers and practitioners for building effective and practical NLP systems.
Limitations
We acknowledge several limitations to our investi- gation, which we propose to address future work. First, while our formulation aligns exactly with part of the insertion process, there are aspects of the full insertion of new terms into the UMLS which are out of our scope. While we do identify terms that not linked existing UMLS concepts, we do not attempt group these terms into new concepts. The identification of synonymous terms for new concepts will be addressed in future work. Second, except for the RBA approach that leverages lexical information and source synonymy, our approach does not take advantage of contextual information available for new terms (e.g.. hierarchical informa- tion provided by the source vocabulary). We plan to follow (Nguyen et al., 2022) and integrate this kind of information that has been shown increase precision without detrimental effect on recall in the UVA task. Third, our approach uses a single term, the term closest to the new atom, as the represen- tative for the concept for linking purposes. While this approach drastically simplifies processing, it also restricts access to the rich set of synonyms available for the concept. We plan to explore al- ternative trade off's in performance when including more concept synonyms. Finally, reliance on the RBA information had the potential for incorrectly identifying new concepts when RBA signal is not complete. Even though RBA signal is quite useful for this task, it is important to build systems robust to its absence. We plan to explore this robustness more actively in future work by including such incomplete signal in the training process.
Even though most model errors are caused by annotation issues in the UMLS, there are still some which are due to complexity and ambiguity. In the bottom half of Table 6, we see examples that our model still struggles with. First, the new atom "urea 400 MG/ML" should have been mapped to "urea 10%" since the percentage is calculated as the number of grams in 100 mL. However, this decision requires not only the knowledge of this definition but also mathematical reasoning abilities. Finally, the last error in our table is caused by the ambiguity deciding whether "human echovirus" and "echovirus" should be deemed equivalent. We note that both of these error types as well as the previously mentioned annotation errors show that our model's errors are occurring scenarios which are either vnsolvable very challenging, shedding light its potential practical system support UMLS editors.
6 Conclusion
In conclusion, this paper emphasizes the impor- tance of formulating NLP problems that align well with real-world scenarios in the midst of growing enthusiasm for NLP technologies. Focusing on the real-world task of UMLS vocabulary insertion, we demonstrate the importance of problem formu- lation by showcasing the differences between the UMLS vocabulary alignment formulation and our own UVI formulation. We evaluate existing UVA models as baselines and find that their performance differs significantly in the real-worke setting. Addi- tionally, show that our formulation allows to not only discover straightforward but exceptionally strong new baselines but also develop a novel null- aware and rule-enhanced re-ranking model which outperforms all other methods. Finally, we show
Acknowledgements
The authors would like thank the expert UMLS annotators from the NLM for their detailed error analysis. We also appreciate constructive com- ments from anonymous reviewers and our NLM

---

and OSU NLP group colleagues. This research was supported in part by NIH R01LM014199, the Ohio Supercomputer Center (Center, 1987) and the Intramural Research Program of the NIH, National Library of Medicine.
Jinhyuk Lee. Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformaticx, 36(4):1234-1244)
Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. 2021. Self-alignment pretraining for biomedical entity representations. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 4228-4233 Online. Association for Computa- tional Linguistics.
References
Goonmeet Bajaj, Vinh Nguyen, Thilini Wijesiriwardene, Hong Yung Yip, Vishesh Javangula, Amit Sheth, Srinivasan Parthasarathy, and Olivier Bodenreider. 2022. Evaluating biomedical word embeddings for vocabulary alignment at scale the UMI.S Metathe- saurus using Siamese networks. In Proceedings of the Third Workshop on Insightsfrom Negative Results NLP, pages 82-87, Dublin, Ireland. Association for Computational L Linguistics.
Alexa McCray, Suresh Srinivasan, and Allen C. Browne. 1994. Lexical methods for managing varia- tion in biomedical terminologies. Proceedings. Sym- posium on Computer Applications in Medical Care, pages 235-9. 9.
Olivier Bodenreider. 2004. The Unified Medical Lan- guage System (UMLS): Integrating Biomedical Ter- minology, Nucleic acids research, 32 Database isxue:D267-70.
Vinh Phu Nguyen, Hong Yung Yip, Goonmeet Bajaj, Thilini Wijesiriwardenee Vishesh Javangula, Srini- vas Parthasarathy, Amit P. Sheth, and Olivier Bo- denreider. 2022. context-enriched learning models for aligning biomedical vocabularies at scale in the umls metathesaurus. Proceedings of the ACM Web Conference 2022.
Ohio Supercomputer Center. 1987. Ohio supercomputer center.
Hang Dong, Jiaoyan Chen, Yuan He, Yinan Liu, and Ian Horrocks. 2023. Reveal the unknown: Out-of- knowledge- base mention discovery with entity link- ing.
Vinh Phu Nguyen, Hong Yung Yip. and Olivier Boden- reider. 2021. Biomedical vocabulary alignment at scale in the umls metathesaurus. Proceedings of the ... International World- Wide Web Conference. Inter- national WWW Conference, 2021:2672 2683.
Jennifer D' and Vincent Ng. 2015. Sieve-based entity linking for the biomedical domain. In Proceed- ings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Interna- tional Joint Conference on Natural Language Pro- cessing ( (Volume 2: Short Papers), pages 297-302, Beijing, China. Association for Computational Lin- guistics.
Pedro Ruas and Francisco M. Couto. 2022. Nilinker: Attention-based approach to nil entity linking. Jour- nal of Biomedical Informatics, 132: 104137.
Thilini Wijesiriwardene, Vinh Phu Nguyen, Goon- meet Bajaj. Hong Yung Yip. Vishesh Javangula, Yuqing Mao, Kin Wah Fung, Srinivas Parthasarathy, Amit Sheth, and Olivier Bodenreider. 2022. Ubert: A novel language model for synonymy predic- tion at scale in the umls metathesaurus. ArXiv, abs/2204.12716.
Lenz Furrer, Joseph Cornelius, and Fabio Rinaldi. 2022. Parallel sequence tagging for concept recognition. BMC Bioinformatics, 22.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-Spest Specifie Lan- guage Model Pretraining for Biomedical Natural Lan- guage Processing. ACM Trans. Comput. Healthcare, 3(1).
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- rie Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of of the-art art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational Linguistics.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with gpus. JEEE Transactions on Big Data 7(3):535-55 547.
Robert Leaman, Rezarta Islamaj Dogan, and Zhiyong Lu. 2013. DNom:: disease name normalization with pairwise learning to rank. Bioinformaticx, 29:2909 2917.
Ledell Wu, Fabio Petroni, Martin Josifoski. Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
Robert Leaman and Zhiyong Lu. 2016. TaggerOne: joint named entity recognition and normaliza- tion with semi-Markov Models, Bioinformatics, 22118)283992846

---

pages 6397-6447, Online, Association for Computa- tional Linguistics.
Zheng Yuan, Zhengyun Zhao, Haixia Sun, Jiao Li, Fei Wang, and Sheng Yu. 2022. Coder: Knowledge- infused cross-lingual medical term embedding for term normalization. Journal of Biomedical Informat- icx, page 103983.
Sheng Zhang, Hao Cheng, Shikhar Vashishth, Cliff Wong, Jinfeng Xiao, Xiaodong Liu, Tristan Nau- mann, Jianfeng Gao, and Hoifung Poon. 2022.
showledge-rich self-supervision for biomedical en- tity linking. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 868- 880, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yijia Zhang. Qingyu Chen, Zhihao Yang. Hongfei Lin, and Zhiyong Lu. 2019. BioWord improving biomedical word embeddings with subword informa- tion and MeSH. Scientifie Data, 6.

---

A Original UVA Dataset
in semantic group distribution across insertion sets, as seen in as well as the high variance in model performance across semantic categories, as seen $5.2 and Appendix E.
Table 7 lists the basic statistics for the UMLS vo- cabulary alignment datasets. Since the UVA task was formulated and evaluated only as a binary clas- sification task, the dataset is divided into positive and negative pairs. For more details about how the negative pairs were sampled from the UMLSS refer the interested reader $4.2 of Nguyen al. (2021).
C Implementation Details
In this section we discuss the implementation de- tails for our baselines as well as our proposed ap- proach. For the UMLS vocabulary alignment base- lines, we use the same implementation of the Rule- Based Approximation (RBA) and LexLM used by the authors Nguyen et al. (2021). To imple- ment our language model baselines we use the Hug- gingFace Transformers library (Wolf et al., 2020). We use the FAISS library (Johnson et al., 2021) to speed up nearest neighbor search using GPUs when experimenting with LcxLM, SapBERT and PubMedBERT embeddings (Johnson et al., 2021). We train our cross-encoder re-ranking module us- ing BLINK (Wu et al., 2020), which uses a cross- entropy loss to maximize the score of the correct candidate over the rest the candidates. We use default hyperparameters listed in Table 9 to train our re-ranking module but perform early stopping using the accuracy metric on our 2020AB valida- tion set. All experiments used an NVIDIA V100 GPU with 16 GB of VRAM. The models we used and the approximate amount of GPU hours used for each is listed Table 10.
UVA Pairs
Positive Pairs
Negative Pairs
Train
192,400,466 173,035,862
22,324,834 5,581,209
170,075,628 167,4 167,454,653
Test
Table Original UVA dataset statisties.
B UVI Dataset Details
In Table 8, we report the size of our five UMLS vocabulary insertion dataset splits. We note that only the 2020AB version contains training set, all other insertion sets only have development and test sets.
Train
Dev
Test
2020AB
215,402
105,796
108,937
2021AA
112,647
113,563
2021AB
227, 227,440
228,053
2022AA
88,186 186
87, 87,803
2022AB
138,107
137,735
Table 8: Experimental split statistics for UMLS inser- tion dataset Q from 2,020 to 2,022.
Learning Rate
Total
Batch Size
Warmup
Epochs
Ratio
2e-5
3
0.1
In terms of datasct construction, we reitcrate that stratified sampling based on semantic groups was used to keep the original distributions intact. We adopt this technique due to the substantial changes
Table 9: Hyperparameters selected for our cross-encoder re-ranking training for reproducibility.
Semantic Group Distribution
# Parameters (millions)
Total GPU
Hours
2020AB
0.4
2021AA
LexLM
0.2
2021AB
PubMedBERT 100
140
0.3
2022AA
SapBERT
100
40
2022AB
Table 10: Total GPU Hours associated with experi- ments. PubMedBERT GPU hours include both UMLS encoding and fine-tuning for our re-ranking module.
0.1
0.0
Living Beings
Chemicals
Disorders
Procedures
Physiolegy
Genes
Concepts
Devices
D Latency Comparison
In Table 11, we report the inference latency for each baseline as well as our proposed approaches
Figure& 4: This figure shows incidence each of the most frequent semantic groups across the 5 insertion sets explored in this work.

---

on the UVI task. As seen in the table, our ap- proach has significantly slower inference than pre- vious baselines. Nevertheless, since the UMLS insertion task happens only twice a year, variations in inference latency are not a significant concern long as the process can be run within rea- sonable amount of time on available computing resources. We hope that these numbers can help other researchers and practitioners understand the computing requirements on this or similar tasks.
atoms to link. For the Procedures category, we see similar pattern in the 2021AA insertion set while the other sets see small but regular improve- ments with our system. These results indicate that, although our proposed approach can lever- age the RBA signal more consistently when it is sufficiently strong, it fails to correct for it when it is very weak to begin with. It is therefore important to continue working on ways correct at least alert annotators about potential system failures in specific concept sub-groups.
Time for 300k Atoms (mins)
Inference Latency (ms)
Model
Model Calibration Details
As discussed above, our re-ranker model's output confidence, defined softmax over candidate logit scores produced by our model, seemed corre- lated with model accuracy. In Table 12, we show model accuracy across different model confidence scores. We find that model confidence score is highly correlated with model accuracy, which drops to around 50% when model confidence drops be- low 90% and continues to drop after that. Through qualitative analysis, dicate successful model calibration but actually mainly caused by annotation errors within UMLS which result in duplicate and ambiguous concepts.
RBA
0.01
0.05
LexLM
1.28
6.40
SapBERT
2.50
12.50
RBA LexLM
1.29
6.45
RBA SapBERT
2.51
12.55
Re-Ranker (RBA Signal)
35.51
177.5
Table 11: Time spent on inference for each baseline as well as our proposed approach.
find that this does not in-
E Detailed Semantic Group Evaluation
As mentioned in 5.2, different UMLS updates often contain completely different semantic group distri- butions since they depend entirely on independent source updates. Due to this, generalization across different semantic categories (semantic groups in the UMLS) crucial feature for a system to be successful in real-world UMLS vocabulary inser- tion. Table 13 provides a detailed report of the performance of our strongest baseline and our best proposed approach on all development insertion sets across the 9 most frequent semantic groups. As seen in these detailed results, our proposed ap- proach obtains stronger and more consistent results across all semantic groups compared to our best baseline.
Model Confidence
Number of Examples
Accuracy
(%)
0
23
8.7
10
22.5
20
206
32.5
30
397
36.0
40
1, 1,282
56.0
50
964
55.1
60
511
48.7
70
411
50.6
80
590
55.3
90
38,076
92.1
100
62,743
99.8
Nevertheless, as discussed in the main text, our approach remained below 90% on average cat- egories like Genes and Procedures. In the broken down results in Table 13, we can more clearly see that these averaged results are caused by outlier insertion sets. For the Genes semantic group, our proposed approach improves performance consid- erably for all insertion sets except for 2022AA, in which its performance drops by more than 10 points. We note that the performance of the best bascline is also much lower than usual, potentially indicating a weak RBA signal and challenging
Table 12: The output probability of our best re-ranking approach (the probability of the highest scoring candi- date concept) seemed to be correlated with high predic- tion accuracy but actually indicates annotation errors.

---

Semantic Group
2020AB
2021AA
2021AB
2022AA
2022AII
RBA
Re-Ranker
RBA
Re- Rankan
REA
Re- Re-Ranker
RBA
Re- Re-Ranker
RBA
Re-Ranker
SapBERT
RBA Signal
SapBERT
RBA Signal
SapHERT
RBA Signal
SapBERT
RBA Signal
SapBERT
RBA Signal
Beings Chemicals & Drugs
87.7
94.8
73.8
89.2
$7.9
95.3
81.5
96
74.8
92.4
Genes Molecular Sequences
86.8
97.0
78.2
87.4
71.2
79.2
Disorders
91.7
98.0
90.2
97.2
96.0
98.3
91.8
97.0
9018
98.0
Procedures
91.1
96.9
$1.6
51.8
95.3
97.6
95.0
74.2
75.0
Physiology
99.2
99.1
Concepts & Idens
91.5
97.4
70.5
96.1
98.4
98.5
92.6
96.4
92.5
97.5
Devices
Anatomy
92.7
96.1
94,2
97.9
92.2
95.1
98.3
99.0
97.8
99.1
Table 13: Breakdown for Table 3 all insertion development sets and the most frequent semantic groups. These detailed results can help us more closely understand model failures across semantic groups compared to the aggregated results.


Reviews:
[{"review_id": "Nb8hoZHvrD", "reviewer": "Reviewer_wXDg", "summary": "", "questions": "Do you have any idea of the latency (inference time in ms) of your proposed systems ? And how does it compare to other systems ?\n\nA question about L. 238, have you tried your systems on other languages to know if it is replicable on them ? Or if the specificities of other languages make it more complicated to transfer performances ? For example, Chinese, Hebrew, French or Turkish ?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are delighted that the reviewer found our paper well-presented, strongly motivated and interesting. Their recognition of our strong empirical results and thorough qualitative error analysis is especially encouraging. We would like to thank the reviewer for the amount of effort they spent familiarizing themselves with and evaluating our work. We address their questions in the sections below:\n \n **Inference Latency**\n\nGiven that the UVI process happens only twice a year, variations in inference latency are not a significant concern as long as the process can be run within a reasonable amount of time on standard computing resources. \n\nNevertheless, we concur with the reviewer that this information could be empirically valuable for other tasks and datasets. We will thus add the table below listing the inference latencies of all baselines and our own approach.\n\n| Model                               | Inference Latency (ms) | Total Time for 300,000 Atoms (minutes) |\n| --------------------------------  | ------------------------------- | ---------------------------------------------------- |\n| RBA                                 | 0.01                               | 0.05                                                        |\n| LexLM                              | 1.28                               | 6.40                                                        |\n| SapBERT                         | 2.50                               | 12.50                                                      |\n| RBA + LexLM                  | 1.29                               | 6.45                                                         |\n| RBA + SapBERT             | 2.51                               |12.55                                                        | \n| Re-Ranker (RBA Signal) | 35.51                             |177.5                                                        |\n\n**Multilingual Generalization**\n\nAlthough we have not tried our approach for other languages within the UMLS Metathesaurus, it is likely to generalize successfully as long as a strong language model which captures that language is used. However, performance will be limited by the amount of training data available since only a small percentage of all terms in the UMLS Metathesaurus are in languages other than English (Spanish is the second most common language with 1.9M terms and all other languages consist of fewer than 50,000 terms each).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and present extremely well the subject of study despite being a difficult task. They introduce the previous works perfectly and share constantly their motivations behind this work without denigrate any previous contributions.\n\nThe different experiments are interesting and give extraordinary results compared to the selected baselines. The best proposed system give constant and reliable performances across a large set of datasets curated from a large time frame (5 versions). The expert analysis give good hints about the reasons why the system are sometime badly performing and will be interesting improvements for the future.\n\nEvaluating both accuracy and ranking accuracy is a very good way of combining both academia standard metrics and real-world applications metrics, since the tool can be used as an assistance tool for experts editors.\n", "reasons_to_reject": "None", "questions_to_authors": "Do you have any idea of the latency (inference time in ms) of your proposed systems ? And how does it compare to other systems ?\n\nA question about L. 238, have you tried your systems on other languages to know if it is replicable on them ? Or if the specificities of other languages make it more complicated to transfer performances ? For example, Chinese, Hebrew, French or Turkish ?\n", "paper_topic_and_main_contributions": "The authors present a new SOTA model for solving UMLS vocabulary insertion (UVI) and helping experts editors in their tasks. The paper start by introducing the UVI task and introduce its multiple contributions (language models, entity linking, candidate re-ranking, augmented RBA, NULL-injection and RBA enhancement). Then, they give context about previous contributions to the subject, why they are for most of them treating the problem in a very different manner than the one from real world usages and each pro & cons on them. Finally, they are concluding by the results, a deep error analysis and a study of model generalization over time and subjects.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Adding a citation for L. 90 \"prediction, its formulation has made it unable to yield practical improvements for the UVI process.\" can be very nice."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "L. 608 \"While models like ChatGPT have vast potential, they are unlikely to single-handedly provide translational NLP solutions.\". I found this remark inappropriate according to the thematic of the paper. You are not evaluating LLMs and ChatGPT model. It looks more like a buzzword than to be an interesting remark.\n"}}, {"review_id": "BpSd8T2hop", "reviewer": "Reviewer_SKdh", "summary": "", "questions": "\nPlease provide a more accurate description of the evaluation metrics in section 4.2.  I suggest to use formulas.\n\nIn particular the \"ranking accuracy\", which is a central metric in the paper because it is the one where the most\nimprovements are seen, is very superficially defined. It is not clear if it is a proper ranking metric, as the name\nsuggest, or a measure of accuracy. If it is a ranking metric, please explain in which sense it provides a measure\nof the quality of the ranking. If it is not, please name it differently. \n\nI also struggled a bit to understand the definition of New Concept Precision\" because it is not immediately\nobvious what is the different between \"correct new concept predictions\" and \"true new concept atoms\", but\nI believe I understood it. \n\nIn any case, formulas would help understanding and remove some ambiguity.\n\nFigure 2, not clear what the \"New Concept\" as input to the Re-Ranker represents.\n\n\n\n\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would first like to thank the reviewer for the time and effort spent assessing our paper. We are glad that the reviewer deems our work well-developed, our experiments well-designed and that they find our methodology for combining rule-based and neural models interesting. \n\nWe will address the reviewer’s concerns regarding our work’s contribution, scope and evaluation metric clarity in the following sections:\n\n**Contribution & Scope**\n\nOur work strives to provide a real-world case study which highlights the importance of problem formulation in the successful development of NLP applications. In the case of UVI,  slightly different problem formulations in previous works led to the development of datasets, models and conclusions which were not well-aligned with the real-world task and were therefore less practically applicable. The effects of this gap serves to remind readers that problem framing is a vitally important part of NLP practice and deserves more attention in the applied NLP literature.\n\nIn addition to this more abstract contribution,  as the reviewer correctly points out, our methodological contributions are directly applicable to others working in similar problems such as other knowledge base update tasks. Furthermore, as the reviewer also mentions in their “Reasons to Accept”, our work provides an interesting approach for integrating rule-based and neural signals for this task successfully. Our approach provides evidence that simple methods can be used to augment neural NLP models with external rule-based signals. Given the amount of signal which does not come from the text itself in real-world NLP applications, this insight can be quite valuable to the applied NLP community.\n\n**Metric Clarity**\n\nWe greatly appreciate the reviewer’s constructive criticism on our metric descriptions, clarifying this important point is crucial for our work. We will be sure to add formulas and more thorough descriptions for each metric in the camera-ready version. \n\nIn particular, ranking accuracy is the percentage of correct predictions computed only on atoms in the insertion set which exist within the previous UMLS version (equivalent to “Accuracy” except that “new concept atoms” are excluded). Since it is a standard “Accuracy” metric rather than a ranking based metric, we will take the reviewer’s suggestion and change its name to “Existing Concept Accuracy” throughout the paper for clarity.\n \n**“Figure 2, not clear what the \"New Concept\" as input to the Re-Ranker represents.”**\n\nThe “New Concept” block in Figure 2 represents that our approach allows the re-ranking module  to label an atom as a “New Concept” instead of as one of the candidate concepts. We will make this much clearer in the caption and the figure itself in the camera-ready version.\n\n**“line 127-129 emphasis inappropriate in my view”**\n\nWe concur with the reviewer that the emphasis is somewhat inappropriate and we will remove it.\n\n**Missing Reference**\n\nWe thank the reviewer for bringing our attention to this important work, we will include it in our camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\nWell developed study of the problem of UMLS update, which is framed as a problem similar but not identical to (biomedical) entity linking.\nWell designed experimental setup, with one particular UMLS update used as a reference for training/development/testing, and other\nupdates used for further testing of the results. Interesting combination of rule-based and BERT-based methods. Good evaluation.\n", "reasons_to_reject": "\nThe problem tackled by the authors is extremely specific, and has a very narrow application, although the methods could probably be generalized to similar problems of knowledge base update.\n\nThe evaluation metrics are not sufficiently clearly described, but it might be just a question of providing a more formal definition.", "questions_to_authors": "\nPlease provide a more accurate description of the evaluation metrics in section 4.2.  I suggest to use formulas.\n\nIn particular the \"ranking accuracy\", which is a central metric in the paper because it is the one where the most\nimprovements are seen, is very superficially defined. It is not clear if it is a proper ranking metric, as the name\nsuggest, or a measure of accuracy. If it is a ranking metric, please explain in which sense it provides a measure\nof the quality of the ranking. If it is not, please name it differently. \n\nI also struggled a bit to understand the definition of New Concept Precision\" because it is not immediately\nobvious what is the different between \"correct new concept predictions\" and \"true new concept atoms\", but\nI believe I understood it. \n\nIn any case, formulas would help understanding and remove some ambiguity.\n\nFigure 2, not clear what the \"New Concept\" as input to the Re-Ranker represents.\n\n\n\n\n", "paper_topic_and_main_contributions": "\nThis paper deals with a practical problem, namely the regular updates of the UMLS, which involve the integration of new terminological vocabularies. The papers proposes a methodology which could be used to speed up the updates, by automatically finding whether\na new term is a potential synonymous of a term already in the UMLS, or it can be attributed to a novel concept, not yet present in the UMLS.\n\nThe paper proposes a different conceptualization of the problem compared to previous approaches. Rather then simply evaluating the similarity of a new term to existing terms, the paper proposes an approach where each new term is assessed in relation to the entire\nUMLS, and either a concept is found to which the term can be assigned, or the term is maked as novel, and a new concept will have\nto be created. I am not completely convinced that this different conceptualization can be considered innovative, as it seems to me to\nbe entirely derivable from the original one.\n\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "\nline 127-129 emphasis inappropriate in my view\n"}, "extra_text": {"Missing_References": "\nRelevant to biomedical entity linking, section 2.2\n\nFurrer L, Cornelius J, Rinaldi F. Parallel sequence tagging for concept recognition. BMC Bioinformatics. 2022 Mar 24;22(Suppl 1):623. doi: 10.1186/s12859-021-04511-y. PMID: 35331131; PMCID: PMC8943923."}}, {"review_id": "X0FuG6Z7pa", "reviewer": "Reviewer_F1a9", "summary": "", "questions": "1. Lines 067-068: How do they know about this number 300,000 related to UVI (unless of course, they work at NIH)? Providing a citation will significantly strengthen the claim and the motivation for the work. Also, Table 1 shows the statistics of the 5 datasets used with the Insertion sets which likely average to 300,000, however, can this be generalized beyond these 5?\n\n2. Lines 243-244: Was there any rationale for choosing the 2020AB dataset for training? \n\n3. Lines 248-252: The authors mention significant performance variability across concept categories. How different are these semantic categories across the five datasets? \n\n4. Lines 390-393: How is the higher chance of being the most appropriate concept for the new atom determined? Any citation for this?\n\n5. Lines 475-492: Can these findings be generalized for the other datasets?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are pleased that the reviewer considers our work to be nicely written, clearly motivated, experimentally thorough and that the high-quality of our error analysis was appreciated. We are very grateful for the time and effort the reviewer spent evaluating our paper as well as their thoughtful questions and concerns, as they will surely improve the quality of our work. \n\nWe hope to address their concerns in the revisions discussed below:\n\\\n\\\n**Significance Testing**\n\nWe thank the reviewer for pointing out the lack of significance testing. We carry out a Paired T-Test using Bootstrap Resampling and find that all improvements for our approach over the best baseline (RBA+SapBERT) are very highly significant (p-value < 0.001) on all metrics. This includes improvements in our generalization experiments in Figure 3. We will add this to the paper and note it in Table 2 and Figure 3.\n\\\n\\\n**Subdomain Variance**\n\n*“Lines 248-252: The authors mention significant performance variability across concept categories. How different are these semantic categories across the five datasets?”*\n\nWe will add a plot with the semantic category distribution across versions to our paper to illustrate their high variability. To give the reviewer a better idea of this variance across insertion sets in this rebuttal, we note that the coefficient of variation for the most frequent semantic groups ranges between 33% and 110%. These large values indicate that the standard deviation compared to the mean percentage of each semantic group is high.\n\n*“Findings from comparison across subdomains (Lines 475-492) are provided in the appendix. This looks like an important result since the authors mentioned significant performance variations across semantic categories. Also in the appendix data is provided only for the 2020AB dataset.”*\n\nWe concur with the reviewer that Table 10, which shows the subdomain performance variation in 2020AB, would be a good addition to the main paper. We will include the aggregate of subdomain performance variation across all UMLS updates and include a larger table with the data for each insertion set separately in the appendix for clarity.\n\n*“Lines 475-492: Can these findings be generalized for the other datasets?”*\n\nOur analysis concerning the low performance of the baseline in the “Genes” and “Drugs” categories as well as the strong improvement of our approach generalizes across all insertion sets except for “Genes” in 2022AA. In this update, the best baseline’s performance on “Genes” is much worse than in other updates due to a weak RBA signal. This domain shift could explain the lack of improvements from our approach in this specific instance. We will discuss this analysis and explanation in the camera-ready version of our paper. \n\\\n\\\n**Main Update Choice**\n\n*“Lines 243-244: Was there any rationale for choosing the 2020AB dataset for training?”*\n\nWe chose the 2020AB dataset as our starting point for this study since it is the earliest possible version for which we are able to fairly compare with related baselines. Given that both the LexLM model and SapBERT were trained on large sections of the 2020AA version of UMLS, we could not choose any update prior to 2020AB. On the other hand, choosing any update after 2020AB for training would reduce the number of datasets we could use to test for model generalization across versions.\n\\\n\\\n**Insertion Set Size Information**\n\nWe calculated the average number of new atoms as 300,000 as the mean of all 5 insertion sets we study in the paper. A more long-term average for new english atom which ignores filters described in Section 4.1 can be computed from the “Statistics” section of each release taken from the publicly available “Release Documentation Archive” of the UMLS (https://www.nlm.nih.gov/research/umls/archive/archive_home.html). Using this resource older and post-2018 updates the average is still high but decreases to around 200,000 atoms. We will include this description in the camera-ready version.\n\\\n\\\n**RBA Signal Clarification**\n\n*“Lines 390-393: How is the higher chance of being the most appropriate concept for the new atom determined? Any citation for this?”*\n\nIn lines 390-393, we mention how the RBA can go beyond lexical similarity in indicating whether an atom should be linked to a specific concept. This difference comes from the RBA’s use of source synonymy as described in section 4.3. This signal lets the RBA check if the new atom has a source-determined synonym already in the UMLS which makes predictions more accurate in cases where these synonyms are available. We understand the reviewer’s confusion and will clarify this accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors motivated the work nicely. They discuss the research gap in detail and how their work attempts to address this gap. The section on problem formulation defines the problem of UVI clearly.\n\n2. Thorough experiments - comparison with baselines and across datasets for generalization.\n\n3. Qualitative error analysis was done by experts", "reasons_to_reject": "1. The proposed approach is shown to have higher accuracy/f1, but no statistical significance is provided. \n\n2. Findings from comparison across subdomains (Lines 475-492) are provided in the appendix. This looks like an important result since the authors mentioned significant performance variations across semantic categories. Also in the appendix data is provided only for the 2020AB dataset.", "questions_to_authors": "1. Lines 067-068: How do they know about this number 300,000 related to UVI (unless of course, they work at NIH)? Providing a citation will significantly strengthen the claim and the motivation for the work. Also, Table 1 shows the statistics of the 5 datasets used with the Insertion sets which likely average to 300,000, however, can this be generalized beyond these 5?\n\n2. Lines 243-244: Was there any rationale for choosing the 2020AB dataset for training? \n\n3. Lines 248-252: The authors mention significant performance variability across concept categories. How different are these semantic categories across the five datasets? \n\n4. Lines 390-393: How is the higher chance of being the most appropriate concept for the new atom determined? Any citation for this?\n\n5. Lines 475-492: Can these findings be generalized for the other datasets?", "paper_topic_and_main_contributions": "The focus of this work is better integration of realistic scenarios in the task of UMLS vocabulary insertion. The authors introduce the UVI task of how a new UMLS atom can be inserted - if it is related to an existing concept or if it is a new concept altogether. The authors contribute five datasets of UMLS updates over a period of time for the task. They integrate domain knowledge by using biomedical LMs. They also propose a model for the task and show through comparisons with baselines and across the datasets the improvement brought by the model. \n\nThis is actually a nicely written paper and reads well. I think with some revisions this paper should be accepted", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is well-written, presents the subject clearly, provides strong experimental results, combines academic and real-world metrics, and offers valuable insights, though some aspects of the evaluation and findings could be improved.
Decision: Accept-Findings