Multimodal Medical Code Tokenizer
Authors: Xiaorui Su
Shvat Messica
Yepeng Huang
Ruth Johnson
Lukas Fesser
Shanghua Gao
Faryad Sahneh
Marinka Zitnik
Venue: ICML 2025 poster (2025)
Keywords: Medical Code
Tokenization
EHR
Primary area: applications->health_medicine
TL;DR: Multimodal Medical Code Tokenizer

Abstract: Foundation models trained on patient electronic health records (EHRs) require tokenizing medical data into sequences of discrete vocabulary items. Existing tokenizers treat medical codes from EHRs as isolated textual tokens. However, each medical code is defined by its textual description, its position in ontological hierarchies, and its relationships to other codes, such as disease co-occurrences and drug-treatment associations. Medical vocabularies contain more than 600,000 codes with critical information for clinical reasoning.  We introduce MedTok, a multimodal medical code tokenizer that uses the text descriptions and relational context of codes. MedTok processes text using a language model encoder and encodes the relational structure with a graph encoder. It then quantizes both modalities into a unified token space, preserving modality-specific and cross-modality information.  We integrate MedTok into five EHR models and evaluate it on operational and clinical tasks across in-patient and out-patient datasets, including outcome prediction, diagnosis classification, drug recommendation, and risk stratification. Swapping standard EHR tokenizers with MedTok improves AUPRC across all EHR models, by 4.10% on MIMIC-III, 4.78% on MIMIC-IV, and 11.32% on EHRShot, with the largest gains in drug recommendation. Beyond EHR modeling, we demonstrate using MedTok tokenizer with medical QA systems. Our results demonstrate the potential of MedTok as a unified tokenizer for medical codes, improving tokenization for medical foundation models.

Multimodal Medical Code Tokenizer
Xiaorui Su
Shvat Messica
Yepeng Huang',
Ruth Johnson Lukas Fesser' Shanghua Gao
Faryad Sahneh
Marinka Zitnik
Abstract
Fiitheeee mailliix Withou complication
Foundation models trained on patient electronic health records (EHRs) require tokenizing med- ical data into sequences of discrete vocabulary items. Existing tokenizers treat medical codes from EHRs as isolated textual tokens. However, each medical code is defined by its textual descrip- tion, position in ontological hierarchies, and its relationships to other codes, such as disease co-occurrences and drug-treatment associations.
of drugge cood in lover Mose glarnes levels
ATC A1SAD
Fisuiinznn ijoo.. ling anngg
Messure
Medical patient record
MedTok Tokenizer
Medical vocabularies contain more than 600,000 codes with critical information for clinical rea- soning. We introduce MED TOK, a multimodal medical code tokenizer that uses the text descrip- tions and relational context of codes. MEDTOK processes text using language model encoder and encodes the relational structure with graph encoder. It then quantizes both modalities into unified token space, preserving modality-specific and cross moos-moddllty information. We integrate MED into five EHR models and evaluate it on operational and clinical tasks across in-patient and out-patient datasets, including outcome pre- diction, diagnosis classification, drug recommen- dation, and risk stratification. Swapping standard EHR tokenizers with MEDTOK improves AUPRC across all EHR models, by 4.10% on MIMIC-III, 4.78% on MIMIC-IV, and 11.32% on EHRShot, with the largest gains in drug recommendation. Beyond EHR modeling, we demonstrate using MEDTOK tokenizer with medical QA systems. Our results demonstrate the potential of MEDTOK unified tokenizer for medical codes, improv- ing tokenization for medical foundation models.
Muttimodal tokens of medical codes
Medical QA
based models
Figure I. MEDTOK is multimodal tokenizer of medical codes that combines text descriptions of codes with graph-based rep- resentations of dependencies between codes derived from clin- ical ontologies and standard medical terminologies. MEDTOK is general-purpose tokenizer that be integrated into any transformer- based model or system that requires tokenization.
1. Introduction
Electronic health records (EHRs) are the backbone of mod- em healthcare, capturing a person's 's health state with in- creasing precision across diverse modalities. Structured EHR data, encoded through standardized medical codes, support wide range of applications, from personalized risk prediction (Goldstein al., 2016; Yu al., 2024b) and disease trajectory modeling (Jensen et al., 2017; Heumos et al., 2024) to emulation of clinical trials (Katsoulakis et al., 2024; Kraljevic et al., 2024). The cornerstone of struc- tured EHRs is medical coding systems, which assign stan- dardized alphanumeric codes to various aspects of patient health, including diseases, procedures, medications, and laboratory tests. These codes come from widely used termi- nologies such as ICD-9, ICD-10, SNOMED CT, CPT, and ATC, among others (Foley et al., 1992; Organization et al., 1988; Organization, 2004; Donnelly et al., 2006; Dotson, 2013; Miller & Britt, 1995). Although essential for inter- operability, medical codes introduce challenges for models, particularly in the tokenization process, which transforms structured EHR data into token sequences that foundation
'Department of Biomedical Informatics, Harvard Uni- versity, Boston, MA, USA Digital Data, Sanofi, Cam- bridge. MA, USA. Correspondence to: Marinka Zimik <marinka@ hms hawaddaaard.........
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Multimodal Medical Code Tokenizer
models can process.
coding systems remains an open challenge.
Transformer-baset models for structured EHRs (Poulain & Beheshti, 2024; Yang et al., 2023b; Jiang et al., 2023b; Rene et al.. 2024) rely on tokenizers to map raw data into discrete vocabulary items. However, standard tokenization strategies inherited from general-purpose language models fail to capture the complexity of medical codes, leading to six key challenges: (1) Scalability of medical vocabular- ies Medical coding systems contain over 600,000 unique codes, far exceeding standard tokenizer capacities. Treating each code as a separate token leads to inefficient vocabulary expansion, increasing memory demands and fragmenting rare codes (e.g.. splitting "ICD9: 250.0" into arbitrary sub- words). (2) Loss of hierarchical and relational structure Many coding systems encode structured dependencies, such ATC codes, which classify drugs based pharma- cological and chemical properties (Miller & Britt, 1 1995). Standard tokenizers, relying only on co-occurrence statistics, fail to capture hierarchical relationships, losing dependen- cies like disease co-occurrences and drug contraindications. (3) Redundancy across coding systems- Identical clinical concepts often appear under different codes across termi- nologies (e.g., ICD vs. SNOMED). Standard tokenization treats them as separate tokens, creating redundancy and complicating cross-system data integration. (4) Inefficiency in token storage Expanding vocabulary sizes to accom- modate medical codes results in bloated embedding tables that degrade computational efficiency, particularly for low- resource codes that appear infrequently but still require dedicated tokens. (5) Sparse and inconsistent usage- Many medical codes are rarely used or inconsistently documented, making it difficult for standard tokenizers to learn meaning- ful representations. Low-frequency codes suffer from poor embeddings, reducing performance on underrepresented conditions. (6) Lack of multimodal representations - Ex- isting methods (Jiang et al., 2023b; Zhu et al., 2024; Xu et al., 2024) treat medical codes as isolated textual tokens, discarding graph-based relationships that encode essential links between diagnoses, treatments, and medications. A robust tokenizer must integrate both textual and relational information to fully represent medical codes.
Present work. We introduce MEDToK (bttps://github.com mims- (atisshharaar/MedTok)) multimodal medical code tok- enizer that integrates textual descriptions and graph-based dependencies from biomedical ontologies (Figure 1). Un- like standard tokenization methods that treat medical codes as isolated textual tokens, MEDTOK captures both semantic meaning and structured relationships by encoding multiple modalities into unified token space. MEDTOK operates in three stages. Multimodal encoding first extracts text em- beddings from medical code descriptions and graph-based representations from biomedical knowledge graphs using separate encoders. Next, vector quantization maps both modalities into a shared token space, generating distinct text-informed and graph-informed token embeddings while preserving cross- moss-modallty interactions. Finally, optimiza- tion for expressivity ensures that token representations cap- ture hierarchical relationships, semantic equivalence across different coding systems, and dependencies such as comor- bidities and drug interactions.
We integrate MEDTOK into five EHR models and evaluate it in clinical and operational tasks that span the inpatient (MIMIC-III III, MIMIC-IV) and outpatient (EHRShot) settings. These tasks include disease prediction, operational outcome modeling, drug recommendation, patient risk stratification, and operational outcomes. Our key contributions are:
Multimodal tokenization of medical codes - MEDTOK tokenizer jointly encodes both textual descriptions and graph-based representations of medical codes, enabling richer and structured embeddings.
Improved cross-system generalization By incorporating ontological knowledge, MEDTOK bridges semantic gaps between different coding systems.
Demonstrated performance gains Replacing standard EHR tokenizers with MEDTOK improves AUPRC by 4.10% on MIMIC-III, 4.78% on MIMIC-IV, and 11.30% on EHRShot, with the largest gains in drug recommen- dation tasks. MEDTOK is general purpose tokenizer that can be integrated into any transformer- model or system that requires tokenization. Beyond EHR mod- els, we demonstrate its applicability in medical question- answering systems, further highlighting the benefit of optimized tokenization of structured medical data.
Several models attempt to enrich the representations of medical codes by incorporating external knowledge from Large Language Models (LLMs) (Jiang et al., 2023b; Zhu et al., 2024; Xu al., 2024). Methods like GraphCare and RAM-EHR prompt LS to generate structured knowledge triplets of medical codes or summarize retrieved knowledge. Although effective in specific tasks, these approaches suffer from limited generalizability, a heavy reliance on knowledge generated by L.Ms, and a lack of a unified framework for handling various medical coding systems. Despite advances in medical representation learning, a unified tokenizer that integrates textual and structured relational knowledge across
2. Related work
Domain-specific tokenizers. Tokenizers tailored for spe- cific domains have been employed to process various types of data, including language (Sennrich et al., 2016; Kudo & Richardson, 2018: Song et al., 2021; Wang et al., 2024b; Minixhofer et al., 2024), images (Zhou et al., 2022; Yu et al., 2022; 2024a; Zha al., 2024), videos (Choudhury et al.,
2

---

Multimodal Medical Code Tokenizer
2024), graphs (Perozzi al., 2024; Yang et al., 2024a), and molecular and material sciences (Fu et al., 2024; Tah- mid et al., 2024; Qiao et al., 2024). While these tokenizers perform well within their respective domains, they are not directly applicable to medical codes, which contains spe- cialized medical semantics. Medical codes reside in relation contexts and are accompanied by textual descriptions. Di- rectly using the tokenizers for languages risks flattening the relationships among codes and failing to preserve the biomedical information. This will lead to fragmented tok- enization of medical codes, resulting in loss of contextual information during encoding. Meanwhile, visual tokenizer typically focus on local pixel-level relationships, which are insufficient for capturing the complex semantics inherent in medical codes. Graph tokenizers are designed to encode structured information from graphs into discrete token, then enabling LLMs to process relational and topological knowledge effectively. However, graph tokenizers may suf- fer from information loss when applied to graphs in other domains, making them less flexible and efficient for large, dynamic, and cross cross-domain graphs. In contrast, our MED- Tok tokenizer explicitly incorporates the relevant medical semantics by integrating textual descriptions with graph- based relational contexts.
to learn representations for clinical prediction and oper- ational healtheare tasks. These models differ from med- ical LLMs (Singhal et al., 2025; Tu et al., 2024; Sing- hal et al., 2023), which are typically trained on free-text clinical notes (Jiang et al., 2023a) and biomedical litera- ture rather than structured EHR data. BEHRT (Li et al., 2020) applies deep bidirectional learning to predict fu- ture medical events, encoding discase codes, age, and visit sequences using self-attention. TransformEHR (Yang et al., 2023b) adopts an encoder-decoder transformer with visit-level masking to pretrain on EHRs, enabling multi- task prediction. GT-BEHRT (Poulain & Beheshti, 2024) models intra-visit dependencies as a graph, using a graph transformer to learn visit representations before processing patient-level sequences with transformer encoder. Other models enhance EHR representations with external knowl- edge. GraphCare (Jiang et al., 2023b) integrates LLMs and biomedical knowledge graphs to construct patient-specific graphs processed via a Bi-attention Augmented Graph Neu- ral Network. MulT-EHR (Chan et al., 2024) introduces multi-task heterogencous graph learning with causal denois- ing to address data heterogeneity and confounding effects. ETHOS (Renc et al., 2024) tokenizes patient health time- lines for munsformer-based pretraining, achieving zero-shot performance. While these models focus on learning pa- tient representations, MEDTOK serves a different role as a medical code tokenizer. It can be integrated into any struc- tured EHR, transformer- or other foundation model, improving how medical codes are tokenized before being processed. Unlike these models, which rely on predefined tokenization schemes, MEDTOK optimizes the tokenization process itself.
Vector-quantized tokenizers. Tokenization strategies often vary according to the problem domain and data modality where recent work has highlighted the benefits of discrete tokenization (Du et al., 2024). This process involves parti- tioning the input according to finite set of tokens, often held in codebook (this concept is independent of medi- cal coding despite the similar name), and the quantization process involves learning mapping from input data to the optimal set of tokens according to a pre-defined objective such as reconstruction loss (Van Den Oord et al., 2017). Recent work has highlighted the ability of vector quantized (VQ-based) tokenization to effectively compress semantic information (Gu al., 2024). This approach particularly successful for tokenizing inputs with an inherent semantic structure such as graphs (Yang et al., 2023a; Wang et al., 2024c), speech (Zeghidour et al., 2021; Baevski et al., 2019), and time (Yu et al., 2021) as well as complex tasks like rec- ommendation retrieval (Wang et al., 2024d; Rajput et al., 2023; Sun et al., 2024) and image synthesis (Zhang et al., 2023; Yu et al., 2021). Another significant advantage to VQ-based tokenization is the natural integration of multiple modalities. By learning a shared latent space across modal- ities, each modality can jointly modeled using a common token vocabulary (Agarwal et al., 2025; Yu et al., 2023). To- kenFlow leverages a coul-co design that allows for correlations across modalities through a dual encoder (Qu al., 2024).
3. Approach
MEDTOK a multimodal medical tokenizer that leverages both text descriptions and relational contexts of medical codes. MEDTOK operates as a tokenization function f(-) that maps medical code E -M to sequence of elements T in the vocabulary V with a size of N by leveraging both its textual definition D(m) and a subgraph (m) extracted from a biomedical knowledge graph G. Here, M is a set of 617,490 medical codes from eight medical coding systems: ICD-9, ICD-10-CM, ICD-10-PCS, SNOMED CT, ATC, NDC, CPT, and RxNORM.
Problem definition. Our goal is to train multimodal tokenizer f()) so that T = (D((m), (m)), where T [4, (r] and 4 E V, < i < Then the generated T for medical code m could be integrated to any EaRebased models h(-)) and LMs or LLMs p(-) to perform predictive or generative tasks.
Structured EHR, transformer-based and foundation models. Structured EHR models leverage patient records
Figure illustrates the architecture of MEDTOK, which takes both the medical code description and contextual
3

---

Multimodal Medical Code Tokenizer
Codebook
Top k
Greph-specifi
Query
Graph Encoder
ICD9: 250.0
Greph-shared
Local subgraph of the medical code
B Saoine d Iesggbeen
foywar p (okean
X-attn
ICD9: 250.0
Text-shared
Medical taken embedding
Text Encoder
'A chronic disease that occurs when the body doesn't produce enough insulin."
sext-specific
Medical code description
Figure 2. MEDTTK is general multimodal tokenizer of medical codes that can be integrated into transformer-based model or system that requires tokenization. 'X-attn' denotes cross-attention module.
knowledge from biomedical KGs as input. MEDTOK takes two steps, multimodal tokenization and token packing.
where W E Rdxd,, W E Rddd, and W E Rddx,, represents the query, key, and value weight matrix.
Tokenization. After generating modality-speciff and cross- modality embeddings, for each embedding. MEDTOK quan- tizes the embedding into K tokens by querying unified codebook C E R xd The K tokens are identified by the top K nearest vectors in the codebook.
3.1. Multimodal tokenization
Given a medical code paired with its description / and its biological subgraph G, MEDTOK first adopts the text en- coder, denoted as E, and the graph encoder, denoted as Ey. to generate two embeddings: the text semantic embedding x, E d for and the graph-level embedding Xg R.d, for G. These embeddings are computed as x, E5 (t) and Xg = (G), where E, and Ey represent the text and graph encoders, respectively.
In detail, for any modality- or cross-modality em- bedding e,, its quantized tokens I(e.) is formulated by:
I(e.) = argming (Idist(e, C.))
(3)
-odelity-speciffc embeddings. MEDTOK then adopts two linear projectors: fr Rd + Rd and fg Rd. + Rd to generate mopelity- embeddings ef Rd and of E Rdd respectively, where of (x,), og fx(x)), and d the dimension of specific embeddings.
where dist(:,:) :) denotes the Euclidean distance, [I(e.)] K, and C; =li, Then MEDTOK assigns weight each token & E I(e.) based on the distance between and its corresponding vector C₁ Clk, :]. These weighted tokens are then summed together to obtain the quantized vector for e, denoted as é., which is given by:
Cross-modality embeddings. Moreover, MEDTOK incor- porates cross- cross-attention module to derive cross- moddlloy embeddings ef E Rd and eç E Rd, Specifically, the embed- ding ef is computed as:
é.
Σ -dottmax(dist(e, Cx)) C2 kEI(E)
(4)
Following vector quantization conventions, we employ a straight-through gradient estimator: e. = sg(e. - è. +
W: (WXx(WXX, (W
ef softmax
Vd
é, where sg[:] denotes the goaddgtadent operation. The
codebook learning objective is L(e., ê.) ||sg[ē] c
where W! Rddxd W Rdxds, and Wg E Rdxd,
||e sgle: where is hyperparameter.
represent the query, key and value weight matrix. Similarly, the embedding of is given by:
To preserve the distinctiveness of modality-specifie and cross-modality embeddings, MEDTOK divides the entire codebook into three regions: a text-specific region, a graph- specific region, and a shared region. The shared region
Wix, (TXx,(W(x)) Wxxx)" (Wix.) (2)
cg softmax
Vd

---

Multinodal Medical Code Tokenizer
includes the graph-shared and text-shared region and is shared with both two modalities. MEDTOK then queries distinct regions of the codebook: to generate their respec- tive tokens and quantized vectors, which are represented by: (I(ee), [( I[ I(e[)) and (êf, 6 eg 6.). The final codebook objective is as follows:
eg arg max (1(E) 6 gr ( eg) A M(e B)1)
Based on the derivation of Wang et al., the loss for packing shared information across two modalities is formulated by: token = LIntONCE(ē;, eg) + LintoNCE es) 2 es (e; .g)) where LInfONCE denotes the In- foNCE loss. Additionally, the loss for packing specific information across two modalities is formulated by: Lioken LIntoNCE(éf, ē.) + ACorthogonal (ēi, ef) + LintoNCE(C, (g)) + Aootthogonal (%) .g) where Lorthogonal denotes the orthogonal loss.
LC -(f ( Cio cg) + -(of,
+ L(og og) (5)
3.2. Token packing
Unlike image-text paired data, where modalities have sub- stantial overlap, the two modalities considered here (text and graph representations of medical codes) are more dis- tinct but also highly complementary. Text provides the clinical definitions and describes each medical code in natu- ral language. In contrast, the graph representation encodes domain-specific relationships such as disease co-occurrence, hierarchical groupings, and other medical ontologies. These relationships are not fully captured by text alone and intro- duce structured, expert-driven knowledge that is critical for many clinical and scientific applications. By considering both modalities, MEDTOK generates a representation that captures the shared information and also preserves informa- tion that is unique to each modality. MEDTOK achieves this by extracting modality-specifie features during the tokeniza- tion process, rather than relying on standard approaches that may blend or discard valuable distinctions. This ensures that MEDTOK's 's tokens reflect both clinical language and the structured relationships present in the graph.
We combine modality and modality-speciffi losses into an overall token packing loss as: Lioken Lioken + Lioken where 3 and À are hyperparameters set be equal. This approach allows MEDTOK to leverage both modality- shared and modality-specific information.
3.3. Training and inference for MEDTOK
During the training stage, MED'TO is trained by the sum of codebook loss Lc. KL divergency loss LKL, token pack- ing loss Ltoken. where L = Le + LKL + Ltoken. After pre-training, MEDTOK can be integrated into any model or pipeline dealing with medical codes, providing unified medical tokens for downstream tasks.
4. Experiments
Medical coding systems. We collected a total of 617,490 medical codes from eight commonly used coding systems:
We pack tokens (Z(e)), I(e)), Z(ef). I(ee)) by optimiz- ing both modality shared and pondallty-specifff information between these tokens and their corresponding quantized vectors. To capture modality- sharred information, we fo- cus on the tokens I(e() and Z(e)). The objective uses Kullback-Leible divergence to align their distance matri- ces, dist(ef, C) and (e C), such that they follow a similar distribution: LKL= = DKL (softmax(-dist(cf, C)) || softmax (-dist(e g. C))). Next, we optimize the quantized vectors éû and ēG to maximize the information they carry about the other modality, while minimizing redundancy with their own modality. Specifically, we solve:
ICD-9 (Organization et al., 1988), ICD-10-CM (Fung et al., 2020), ICD-10-PCS (Averill et al., 2001), SNOMED CT (Donnelly et al., 2006), ATC (Miller & Britt, 1995), NDC (Palmer, 2006), CPT (Dotson, 2013), and RxNORM (Nel- son al., 201 1), as shown in Table 1. These codes cover various events, including procedures, diagnoses, and medi- cations. Each code is paired with textual description from official documents and a subgraph from PrimeKG (Chandak et al., 2023). Details are available in Appendix A.
Code system
Count
Code system
Count
SNOMED
303,325
ICD9
18,365
0.th = arg max (I(e); e - 3. I(ee; e)le))) &7
(6)
ICDIO-CM
81,184
CPT
10,602
RxNorm
81,151
ATC
6,659
ICDIO-PCS
61,644
NDC
54,560
6- = arg max (T(O;; - B I(oe: (l
E
Table 1. Summary of the datasct's code systems distribution.
For modality-specific information, MEDTOK optimizes to- kens Z(ee) and I(e)) by ensuring that the quantized vectors of and 6g retain maximal information about their respective modalities while minimizing shared information between modalities. The optimal solutions are given by:
Patient EHR datasets. We used three publicly available EHR datasets: MIMIC-III (Johnson et al., 2016), MIMIC- IV (Johnson et al., 2024), and EHRShot (Wornow et al., 2023). MIMIC-III and MIMIC-IV are in-patient datasets with medical records for ICU patients, while EHRShot is a dataset containing longitudinal medical records that include both out-patients and ICU/ED patients. MIMIC datasets
6-- = arg max (T(ē, o6 e)) - X A-(o7; o())
(8)
5

---

Multimodal Medical Code Tokenizer
include NDC medications and ICD-9/ ICD-10 codes for diagnoses and procedures. In contrast, EHRShot mainly uses RxNorm codes for medications, SNOMED codes for diagnoses, and CPT, SNOMED, ICD-9, and 10 codes for procedures. Table summarizes the statistics of three EHR datasets.
4.1. MEDTOK tokenizer with in-patient EHR models
Table 3 presents the AUPRC values for each baseline and their integration with our MEDTOK for five tasks in two in-patient datasets. Compared to baselines that treat each medical code as an individual token, integrating our MED- TOK consistently improves performance across all five tasks, achieving an average improvement of 3.29% on MIMIC-II IIII and 2.67% on MIMIC-IV. This improvement comes from more informative tokens generated by MEDTOK, which strengthen the EHR--ased models. Among five tasks, MED- ToK demonstrates the most significant impact on drug rec- ommendation tasks, highlighting the value of incorporating prior knowledge into our tokenizer.
#patients
#visits
#visits/patient
#events/patient
MIMIC-III MIMIC-IV
35.707
44,399
1.24
51.14
123,488
232,263
1.88
70.33
EHRShot
6,739
921,499
136.74
6182.17
Table 2. Statistics of EHR datasets.
Baselines. We consider two tokenizers and five -Hsed models based on EHR. The first type of tokenizer is text- based (e.g., bert-be (Devlin et al., 2018)), while the second is graph-based (e.g., VQGraph (Yang et al., 2024b)). Five EHR-bases models are ETHOS (Renc et al., 2024), GT-BEHRT (Poulain & Beheshti, 2024), MulT-EHR (Chan et al., 2024), TransformEHR (Yang et al., 2023b), and BEHRT (Li et al., 2020). Details on implementation can be found in the Appendix B.
To further assess the effectiveness of MED TOK, we compare it against two tokenization methods: the text-based BERT tokenizer and the graph- based VQGraph tokenizer. Figure presents the performance of each tokenizer when integrated with Transformer-basedd EHR model (TransformEHR) across five tasks on two in in-patient datasets. MEDTOK con- sistently outperforms BERT and VQGraph in all tasks and datasets, demonstrating the superiority of its tokenization strategy.
Evaluation setup. We consider two evaluation setups:
In-patient evaluation: This setting combines the MED- Tok tokenizer with patient prediction models, using two in-patient datasets that include individuals admitted a hospital. The evaluation encompasses five tasks: C mor- tality prediction (MT), 2 readmission prediction (RA), c length-of-stay prediction (LOS), ④ phenotype predic- tion (Pheno), and ( drug recommendation (DrugRec). The first three tasks focus on predicting a patient's ' future health status using their historical medical records. Phe- notype prediction involves the identification of the pheno- type of a patient's disease based on their medical history. We identified 24 phenotypes for diseases in MIMIC-III and MIMIC-IV, as follows (Harutyunyan et al., 2019). Drug recommendation aims to suggest appropriate medi- cations for patient, considering their historical medical records and the diseases identified during their current visit. For drug recommendation, we focus on five specific drug candidates, including Vancomycin, Levofloxacin, Heparin Sodium, Metoprolol, and Atorvastatin, rather than considering the entire range of available medications. AUPRC is adopted to evaluate the model's performance on the above classification tasks.
MMVI-1II
MINIC-IV
EHRShot
MT
MT
RA-
LOS
1
LOS
10
Phono
Pheno
EXO
ND
Q8
DrugRec
DrugRec
0.2
0.4
0.8
0.2
0.4
0.6
0.3
0.4 AUPRC
0.5
AUPRC
AUPRC
Bott Waton
V03rrph
MedTak
Figure 3. The AUPRC values of three types of tokenizers on in- patient and out-patient datasets, where 00 means Operational Outcomes and ND means assignment of new diagnoses.
4.2. MEDTOK tokenizer with out-patient EHR models
Table 4 presents the AUPRC values for each baseline and its integration with MEDTOK across two task types on the out-patient EHRShot dataset. The results reveal that our tokenizer has the most significant impact on mortality pre- diction in Operational Outcomes, achieving an average im- provement of 11 1.32%. It also significantly improves the detection of new diagnoses of Hyperlipidemia, with an aver- age improvement of 6.00%. As shown in Figure 3, com- parison of three types of tokenizers further demonstrates the effectiveness of MEDTOK in integrating both graph and tex- tual modalities. Additionally, when comparing performance across two in in-patient datasets, we observe that MEDTOK is particularly beneficial for longitudinal data.
Out-patient evaluation: We evaluate MED together with patient prediction models on a dataset of patients who are not admitted to hospital and consider two categories of tasks: ① Operational Outcomes (OO), and 2 new diagnosis assignments (ND), following (Wornow et al., 2023). The 00 includes MT, RA, and prolonged LOS. The new diagnosis assignments are used to predict the first diagnosis of a disease. Details are in Appendix C.
6

---

Multinodal Medical Code Tokenizer
Task I: MT"
Task 2: RA days))
Task 3: LOS"
Task 4: Pheno"
Task S: DrugRee"
Model
MIMIC-III
MIMIC-IV
MIMIC-III
MIMIC-IV AUPRC
MIMIC-III AUPRC
MIMIC-IV AUPRC
MIMIC-III AUPRC
MIMIC-IV AUPRC
MIMIC-III
MIMIC-IV AUPRC
AUPRC
AUPRC
AUPRC
AUPRC
ETHOS
0.617 (0.010)
0.282 (0.001)
0.421 (0.007)
(0.005
NIA
NIA
NIA
NIA
(0.0008
0.131 (0.005)
MEDTOK
(0,020)
(4.2(0.030)
(0,077)
(0.007)
NIA
N/A
(0,004)
(0,0022
GI-BEHRT
0,160 (0.037)
0.028 (0.004)
0..86(00070)) (0.058)
(0,000)
0.103 (0.001)
(0,23)
(0,05)
(0.0000002)
(0,00))
+ MEDTOX
0.193 (0.146)
0.034 (0.005)
(0.052)
0.609 (0.064)
0,287 (0.039)
0.114 (0.00))
0.459 (0.028)
0,512 (0.006)
(0.004
0.783 (0.010)
MuIT-EHR
0,136 (0.021)
(0.003)
(0.008)
0.515 (07)
(0.018)
0.118 (0.052)
(12)
0.498 (0.001)
0.445 (0.000))
(0.027)
+ MEDTOK
(0.025)
0.141 (0.013)
0.565(0.016) (0.016)
0.565 (0.002)
0.198 (0.911)
0.136 (0.030)
0.480 (0,002)
0,504 (0.001)
0,571 (0.006)
0.465 (0.003)
TransformEHR
0.207 (0.012)
0.022 (0.012)
0.527 (0X030)
0.152 0.152(0.021) (0.08)
(0 (0.00T
0,507 (0.007)
0.533 (0.030)
(D.46)
MEDTOK
0,246 (0,044)
0,058 (0.0477
0.568 (0.036)
(0.017)
0.159 (0,031)
0.121 (0.002)
0.513 (0.024)
0,518 (0.012)
0,580 (0.035)
0.661 (0.092)
BEHRT
0.163 (0.037)
0.028 (0.)033)
0.529 (0.053)
(0.0L)
0.232 (0.015)
(0.003)
0,587 (0.004)
0.493 (0.006)
0.539 (0.013)
0.778 (0.014)
MEDTOK
(0,025)
0J032 (06)
(0,040)
0.515 (0.005)
(0,030)
0.137 (0.004)
0.603(000000)
0.504 (0.006)
0.558 (0.006)
0.792 (0.007
Improvementt// (%)
+3.32%
3,54%
3,00%
2.46 %
3.13%
140%%
2,90%
1.8%
4.10%
4.78%
+: inbolanced bisary classficctioon e: multi-class clasiffiation, macro-tverriell. of multistabel classification, N/A indientes that the model was not configured for this trsk.
Table 3. The results of MEDTOK with all baseline models across five tasks on two in-patient datasets.
Task 1: Operational Outcomes (00)
Task 2: Assigument of New Diagnoses (ND)
Model
Long LOS AUPRC
RA (<15 days) AUPRC
MT
Hypertension AUPRC
Hyperlipidemia AUPRC
Pancreatic Cancer AUPRC
Acute MI
AUPRC
AUPRC
ETHOS
NA
0.079 (0.017)
0..02(0.018)
(0.60()
0,155 (0.031)
0.056 (0.006)
(0.011)
MEDTTO
0.128 (0.025)
0.339(0.010)
0.175 (0.019)
0.163 (0.025)
0.056 (0.0133
(0.077))
GT-BEHRT
(0.02))
0.115 (0.012)
(0.012)
0.303 (0.018)
0.239
0.044 (0.008)
(0.000)
MEDTTO
0.739 (0.025)
0.154 (0.013)
(0.055)
0.360 (0.012)
0.441 (0.005)
0.074 (0.010)
0.031 (0.015)
MuIT-EHR
(0.025)
0.125 (0.014)
0.397(0.016)
(0.00(
0.243 (0.0005 (0.005)
0.022 (0.008)
(0.000
MEDTTO
0.571 (0.015)
0.188 (0.021)
0.444 (0.012)
0.226 (0.006)
(0.02))
0.037 (0.015)
0.028 (0.014)
'rransormEHH
(0.02()
0.197 (0.016)
(0.030)
(0.76(
0.305 (0.021)
0.053 (0.006)
(0.006)
MEDTOX
0.675(00015)
0.243 (0.016)
0,379 (0.034)
(0.03
0,333 (0.018)
0.082 (0.012)
(.052(00017)
BEHRT
(0.03))
(0.022)
0.389 (0.018)
(0.03(00022))
0,251 (0.019)
0.036 (0.008)
(0003()
MEDTOX
0.723 (0.028)
0.397 (0.036)
0.431 (0.017)
0.287 (0.018)
0.302 (0.015)
0.057 (0.012)
(0.06))
Improvement (%)
+5.52%
+5.24%
+11.32%
+3.30%
+6.00%
+1.90%
+1.76%
Table 4. The results of MEDTOK with all baseline models across two tasks on the EHRShot dataset.
4.3. Ablation studies
outcomes in EHRShot. These findings emphasize the im- portance of incorporating the underlying information linked to medical codes.
To comprchensively understand the contributions of various components in MEDTOK, we conduct ablation studies on: (1) the effect of input modalities, and (2) the effect of shared and moodalitysspeciff optimization strategies. To ensure that our analysis is not confounded by architectural differences, integrate MEDTOK with standardized Transformer- based backbone (e.g.. TransformHHRR.. This setup allows us to attribute performance differences directly to modalities and optimization strategies, rather than model architecture.
Effects of modality-shared and modality-specifff infor- mation on MEDTOK. MEDTOK is built on two modalities, and we have analyzed the impact of each modality in Figure 4. The results show that the model performs best when using both modalities. Additionally, MEDTOK optimizes to- kens by maximizing shared information between modalities while preserving modality-speeife. information. To assess the contribution of each loss component, we conducted ab- lation studies by retraining MEDTOK with different loss function combinations. We then apply the pretrained MED- Tok to all tasks across datasets to obtain its performances. The results (average AUPRC across all tasks) in Table 5 demonstrate that both shared and specific information opti- mization enhance performance, with the full optimization achieving the best results across all datasets. The vector quantization loss Lo is the basic loss for tokenization. By optimizing both shared and specific information across two modalities, the performances are improved by 3.9%, 5.7%, and 9.1% % on three datasets, respectively. The experimen- tal results also show that shared and specific information contribute more to out-patient datasets.
Multimodal learning in MEDTOK. To evaluate the im- pact of the two modalities (text, graph) used in MEDTOK medical code definitions and biological subgraphs derived from biomedical knowledge graph-we assess its perfor- mance by removing the text and graph modalities separately. As shown in Figure 4, MEDTOK, when leveraging both modalities, achieves the best performance across all tasks on three datasets. By comparing the performance of MED- Tok without the graph modality and MEDTOK without the text modality, we observe that both modalities contribute significantly to EHR-based prediction tasks. The graph modality benefits drug recommendation and new disease detection tasks, while the text modality proves essential for readmission prediction on MIMIC-III and operational
7

---

Multimodal Medical Code Tokenizer
MIMIC-II
MIMIC-IV
EHRShol
0.6
WCD graph
0.59
0.8
We graph
0.6
HWO toxt MixTTak
Web graph
Med Tok
0:49
MedTok
0.81
AUFRC
0.6
0.69
0.47
0.40
0.37
2:23
0.4
0.29
0.1
0.2
0.2
0.2
0.14
0:116
0.0
0.0
MT.
RA.
LOS
Pheno
DrugRec
RA.
LOS
Pheno
DrugRec
00
New Disease
Figure 4. The AUPRC values obtained by removing the text and graph modalities across all tasks on two in-patient datascts and one out-patient dataset.
Optimization
MIMIC III
MIMIC IV
EHRShot
A
B
Le
0.373
0.387
0.287
0.5
1MMCAV
0.5
BHRShit
Lo + Lioken LKL
0.379
0.409
0.314
AUPRO
Le + Lioken
0.382
0.402
0.366
0.4
0.412
0.444
0.378
C Lloren PKL Cloken
0.01
0.1
0.2
0.30
0.4
0.5
3000
8000
9309
12000
1500
(603
21000
24000
Table averaged AUPRC across tasks with different loss function combinations.
The Codebook
Figure 5. A, The AUPRC values of MEDTOK with different weighting coefficients X, 8: B. The AUPRC values of MEDToK with different codebook size N.
4.4. Hyperparameter sensitivity analysis
We next investigate the impact of hyperparameters Med- Tok's s performance across all datasets. MedTook is trained with three key hyperparameters: two weighting coefficients, and 3, which control the contribution of shared and spe- cific information loss components, and a codebook size pa- rameter, N, which determines the number discrete tokens available for representation.
the codebook size set to N 12, 000, indicating that this size strikes an optimal balance between sufficient coverage of the medical vocabulary and avoiding overfitting
In contrast, when analyzing the performance on EHRShot, dataset consisting of patients with longer visit histories than those in MIMICIII and MIMIC-IV, we observe that MED- TOK benefits from a larger codebook size. For EHRShot, the highest average performance achieved when the codebook size is increased to N 24, 000. This suggests that for datasets with more extensive patient visit histories, larger codebook may be more effective in capturing the underlying complexity of the medical information, thus improving the model's predictive capabilities.
Effects of loss weight À and 3. To ensure that MedTok treats shared and specific information equally, we set À 3, where À and B are the weighting coefficients for the shared and specific information loss terms, respectively. The average AUPRC scores across all tasks are presented in below Figure 5A. The results demonstrate the influence of hyperparameter choices model performance across the three datasets. Based on findings, recommend setting 3 0.1 for in-patient settings and À = 3 0.01 for out- nat-patient settings.
4.5. Using MEDTOK tokenizer for medical QA
MEDTOK demonstrates strong performance in EHR-based tasks, as shown in Tables 3-4. To further assess its capa- bilities, we explore its effectiveness in a generation task, specifically multiple-choice medical question answering (MedicalQA), where the goal is select the correct answer to given clinical question (Singhal et al., 2023).
Effects of codebook size N. We further evaluate the impact of the codebook size on the performance MEDTOK by training it with varying sizes and assessing its effectiveness across three distinct datasets integrated with TransformEHR. Figure 5B presents the results for various codebook sizes across all tasks on the three datasets. The performance trends observed on MIMIC-III and MIMIC-IV are quite con- sistent, demonstrating a clear pattern where increasing the codebook size enhances the model's performance. Specifi- cally, the most stable average performance achieved when
We evaluate whether MEDTOK enhances few-shot learn- ing in MedicalQA by integrating its tokenized represen- tations with three LLMs (LLaMA3. 1-BB (Dubey et al., 2024). Qwen2.5-7B (Hui et al., 2024). MMedLM (Qiu et al., 2024)). WEDDToK-geernann tokens are used as prefix to- 8

---

Multimodal Medical Code Tokenizer
A
MMLU
MedDDX
AfrimedQA
Pure LLMs
Pura LLMs
Pura LLMs
0.8
MedSoknLLMS
0.770
0.8-
MedTokrLLMS
0.7
MedTokrLLMS
Accuracy
Accuracy
0.5-
0.494
Accuracy
0.612
0.653
0.7-
0.004
0.8
0.634
0.040
0.4
0.586
0.5
1.000
0.3-
LLaMA3. 1-88
Swen2.5-7B
MMcdLM
LLaMA3.1-88
Owen2.5-2 5-78
MMMdLM
LLMMA311-8
Owen2.5-78
MMccLM
Figure 6. The accuracy LLMs vs. MEDTOK+LLMS on three medical QA datasets.
kens, which provide structured medical context before the main input, allowing the LLM to incorporate medical codes.
4718
201
732
230
than EDs tequaritt appoor in patkents
215
1093
For this evaluation, we use three medical QA datasets, in- cluding MMLU (Hendrycks et al., 2021), MedDDx (Su et al., 2024), and AfrimedQA (Olatunji al., 2024). addition, MedDDx dataset contains questions at three diffi- culty levels: Basic, Intermediate, and Expert.
.21
100
atnig rek phypertatrema
170
368
174
1482
121
371
546
4678
103
129
The process consists of three steps: (1) Disease code map- ping Extract disease mentions from each question and retrieve their corresponding medical codes, (2) Tokeniza- tion via MEDTOK Convert medical codes into structured tokens using MEDTOK. (3) Prefix token fine-tuning Fine- tune LLMs using MEDTOK tokens as prefix inputs before the question text. We fine-tune the model on the MedMCQA dataset (Pal et al., 2022), and then evaluate the fine-tuned model on MMLU, MedDDx, and AfrimedQA.
1054
115
368
114
4564
113
271
111
2403
110
108
4610
100
200
300
The number of times that takens cocumed the record posients prodicted at the high hyperipaiemii
Figure 7. Top 100 frequent token IDs appearing in patients at high risk of Hyperlipidemia.
The results in Figure 6 show an accuracy improvement across all datasets compared with LLMs, suggesting that MEDTOK can enhance medical QA when used struc- tured representation of medical codes and integrated with an LLM through prefix tuning.
5. Conclusion
Tokenizing medical codes a critical yet challenging step in developing foundation models for EHRs. Existing tok- enizers treat medical codes as isolated textual units, failing to capture their structured relationships within large-scale medical ontologies. With more than 600,000 codes that span multiple terminologies, standard tokenization meth- ods struggle to scale while preserving the rich semantic and relational context necessary for downstream clinical and operational tasks. We introduced MEDTOK, a multi- modal tokenizer of medical codes that integrates textual definitions and relational ontologies of medical codes to create a unified token representation. MEDTOK applies vec- tor quantization to encode both modalities in structured token space, preserving cross- cross-modality relationships. We integrated MEDTOK with five EHR models, evaluating its impact across inpatient (MIMICIIII MIMIC-IV and outpa- tient (EHRShot) settings, as well as in fine- fine-tuning a medical question-anse answering system. Our results establish MEDTOR as generalizable tokenizer for medical codes, shedding light on how optimizing the tokenization process can benefit medical foundation models.
4.6. Interpretability of MEDTOK
To this end, we selected subset of patients classified as high risk for Hyperlipidemia by where these patients have no records of Hyperlipidemia before. We then counted the tokens assigned to these pa- tients and identified those appearing more than 100 times, as shown in Figure We then mapped these frequent tokens to medical codes, with the most overlapping codes being Rosuvastatin mg Oral Tablet (RxNorm 2669980), Burn of skin (SNOMED CT 147087003), Type diabetes mellitus without complication (disorder) (SNOMED CT 373555004), and Hyperlipidemia (SNOMED CT 285605009). They are closely related to Hyperlipidemia. Rosuvastatin corresponds to medications commonly prescribed for lipid disorders. The other three medical codes represent clinical diagnoses or findings associated with hyperlipidemia-rellee related cardiovas- cular risk. It suggests that MEDTTOK captures key medical concepts related to Hyperlipidemia, supporting its predictive capability.

---

Multimodal Medical Code Tokenizer
Impact statement
Balsa-Canto, E., Brush, M. H., Carbon, S., et al. Mondo: Unifying diseases for the world, by the world. Nucleic Acids Research, (1(DI):D1122DDI 1093/nar/gkac1062.
This work presents a tokenizer for medical codes designed to assist other models in better encoding semantic knowledge. While our tokenizer complements other models, it does not directly raise ethical concerns. Instead, it enhances the trustworthiness of these models by providing text and graph- based information as references for tokenization. In future work, we will explore ways to better integrate our tokenizer with other models to further improve both their performance and trustworthiness.
2023, doi: 10.
Bodenreider, O.
The unified medical language system (umls): integrating biomedical terminology. Nucleic Acids Research, 32(Database issue)) 2004. doi: 10.1093/na//kkh061..
Chan, T. H., Yin, G., Bac, K., and Yu, L. Multi-task het- erogeneous graph learning on electronic health records. Neural Networks, 180:106644, 2024.
Acknowledgement
We sincerely appreciate the valuable discussions with Intae Moon and Zhenglun Kong, whose insights and feedback have contributed to the development of this work. We grate- fully acknowledge the support of NIH RO1-HD NSF CAREER 2339524, US DoD 1-8702-15-D-0000 ARPA-H BDF program, awards from Chan Zuckerberg Initiative, Bill & Melinda Gates Foundation INV-079038, Amazon Faculty Research, Google Research Scholar Program, AstraZeneca Research, Roche Alliance with Distinguished Scientists, Sanofi iDEA-iTECH, Pfizer Research, John and Virginia Kaneb Fellowship at Harvard Medical School, Biswas Com- putational Biology Initiative partnership with the Milken Institute, Harvard Medical School Dean's 's Innovation Fund for the Use of Artificial Intelligence, Harvard Data Science Initiative, and Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University. Any opin- ions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessar- ily reflect the views of the funders.
Chandak, P., Huang, K., and Zitnik, M. Building a knowl- edge graph to enable precision medicine. Scientific Data, 10(1):67, 2023.
Choudhury, R., Zhu, G., Liu, S., Ninuma, K., Kitani, K. M., and Jeni, L. Don't look twice: Faster video transform- ers with run-length tokenization. November 2024. doi: 10. 10448500a2Xi..2411..5333 URL http://arxiv.orr//bb//22/ .05222. 1iii::24111002222
1rXXi::2411.05222
Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for lan- guage understanding, CoRR, abs/ Abss/810.04405, 2018. URL http://axxi.orrgaab///11...44
Donnelly, K. et al. Snomed-ct: The advanced terminol- ogy and coding system for ehealth. Studies in health technology and informatics, 121:279, 2006.
Dotson, P. Cpt codes: what are they, why are they neces- sary, and how are they developed?, 2013.
Conflict of interest
Du, T., Wang, Y., and Wang, Y. On the role of discrete tok- enization in visual representation learning. arXiv preprint urXiv:2407.09055 2024.
Faryad Sahneh is Sanofi employee and may hold shares and/or stock options in the company, Other authors declare no conflict of interest.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., and Fan, A. The llama 3 herd of models, 2024. URL https:
References
Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E., Cai, T., Chattopadhyyy, P., Chen, Y., Cui, Y., Ding. Y., et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.0357. 2025.
Hanxiv.org/ah/2447..173...
Foley, S. M., Daley, J., Hughes, J., Fisher, E. S., Heeren, T., al. Comorbidities, complications, and coding bias: does the number of diagnosis codes matter in predicting in- nn-hospital mortality? Jama, 267(16):2197-222)
Averill, R. F., Mullin, R. L., Steinbeck, B. A., Goldfield, N. I., and Grant, T. M. Development of the icd-10 proce- dure coding system (ied- 10-pes). Topics in health infor- mation management, 21(3):54-88, 2001.
1992.
Fu, C., Li, X., Olson, B., Ji, H., and Ji, S. Frag- ment and geometry aware tokenization of molecules for cannture-based drug design using language mod- els. (arXiv:2408.0972 August 2024. doi: 10.48550/ arXiv.2408 09730. URL http://arxiv.rr/aab//44...899 arXiv:2408.097:: 09730.
Baevski, A., Schneider, S., and Auli, M. vq-wav2vec: Self- supervised learning of discrete speech representations. arXiv preprint arXiv: /510:/999005533 2019.
10

---

Multimodal Medical Code Tokenizer
Fung, K. W. Xu, J., and Bodenreider, O. The new inter- national classification of diseases 11th edition: a com- parative analysis with icd-10 and icd- 10--10 Journal of the American Medical Informatics Association, 27(5): 738-746, 2020.
Johnson, A., Pollard, T., and Mark, R. MIMIC-III Clinical
Database (version 1.4), 2016. URL https://oii.org// 100 13026/C2XW26. Available at: https://doi.orr/10.110 10.10/.00.00/10100206.6 C2XW26.
Johnson, A., Bulgarelli, L.,, Pollard, T., Gow, B., Moody, B., Horng. S., Celi, L. A., and Mark, R. MIMIC-IV (version 3.1), 2024. URL 00tp 10t.1...6//.p.mmmmm.t.
Goldstein, B. A., Navar, A. M., and Pencina, M. J. Risk prediction with electronic health records: The importance of model validation and clinical context. JAMA Cardiol- ogy, 1(9):976, December 2016. ISSN 2380-6583, doi: 10.100//jemmccario.2000...
Katsoulakis, E., Wang. Q.. Wu, H., Shahriyari, L.,, Fletcher, R., Liu, J., Achenie, L., Liu, H., Jackson, P., Xiao, Y., Lyeda-Mahmooo. T., Tuli, R., and Deng, J. Digital twins for health: scoping review. npj Digital Medicine, 7 (1):77, March 2024. ISSN 2398-6352, doi: 10.1038/ $41746-024-01073-0
Gu, Y., Wang, X., Ge, Y., Shan, Y., and Shou, M. Z. Re- thinking the objectives of quator-quantized tokenizers for image synthesis. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp- 7631-7640, 2024.
Kraljevic, Z., Bean, D., Shek, A., Bendayan, R., Heming- way, H., Yeung, J. A., Deng, A., Balston, A., Ross, J., Idowu, E.,, Teo, J. T., and Dobson, R. B. Foresight-a generative pretrained transformer for modelling of pa- tient timelines using electronic health records: a retro- spective modelling study. The Lancet Digital Health, 6(4):e281 April 2024. ISSN 25897500. doi: 10. 10.1016652589--5002240000036
Harutyunyan, H., Khachatrian, H., Kale, D. C.,, Ver Steeg, G., and Galstyan, A. Multitask learning and benchmark- ing with clinical time series data. Scientific data, 6(1):96, 2019.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Kudo, T. and Richardson, J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. (arXiv: 0188800026668 August 2018. doi: 10.48550/arXiv. 062266. URL http://arxiv. org/abs/1 https///880006222660 1eXiv:1808.0660
Heumos, L., Ehmele, P., Treis, T., Upmeier Zu Belzen, J., Roellin, E., May, L., Namsaraeva, A., Horlava, N., Shitov, V. A., Zhang, X., Zappia, L., Knoll, R., Lang, N. J., Hetzel, L., Virshup, I., Sikkema, L.,, Curion, F., Eils, R., Schiller, H. B., Hilgendorff, A., and Theis, F. J. An open- source framework for end-to-end analysis of electronic health record data. Nature Medicine, November 2024. ISSN 1078-8956, 1546-170X doi: 10.1038//411591-024-00222
Li, Y., Rao, S., Solares, J. R. A., Hassaine, A., Ramakr- ishnan, R., Canoy, D., Zhu, Y., Rahimi, K., and Salimi- Khorshidi, G. Behrt: transformer for electronic health records. Scientific reports, 10(1):7155, 2020.
-3380,
Miller, G. and Britt, H. A new drug classification for com- puter systems: the atc extension code. International journal of bio-medical computing, 40(2):121-124, 1995.
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, I.., Liu, T., Zhang, J., Yu, B., Lu, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12166, 2024.
Minixhofer, B., Ponti, E. M., and Vulié, I. Zero-shot to- kenizer transfer. Sttii::20050078833 May 2024. doi: 10. 10.48550/arxiv.440..00 07780883. URL http://arxxivorg/abbs/ 2405.07883. arXiv:2405.07883.
Jensen, K., Soguero-Ruiz, C., Oyvind Mikalsen, K., Lindsetmo, R.-O., Kouskoumvekaki, I., Girolami, M., Olav Skrovseth, S., and Augestad, K. M. Analysis of free text in electronic health records for identification of cancer patient trajectories. Scientific Reports, 7(1):46226, April 2017. ISSN 2045-2322.
Nelson, S. J., Zeng, K., Kilbourne, J., Powell, T., and Moore, R. Normalized names for clinical drugs: Rxnorm at 6 years. Journal of the American Medical Informatics Association, 18(4):441-445 2011.
doi: 10. 10.1038/srep46626.
Jiang, L. Y., Liu, X. C., Nejatian, N. P., Nasir-Moin, M., Wang, D., Abidin, A., Haton, K., Riina, H. A., Laufer, I., Punjabi, P., et al. Health system-scale language models are pur-oue prediction engines. Nature, 619(7969): 357-362, 2023a.
Olatunji, T., Nimo, C., Owodunni, A., Abdullahi, T., Ay- odele, E., Sanni, M., Aka, C., Omofoye, F., Yuehgoh, F. Faniran, T., et al. Afrimed-qa: A pan-african, multi- specialty, medical inestion-answering benchmark dataset. arXiv preprint arXiv:2411. 15640, 2024.
Jiang, P., Xiao, C., Cross, A., and Sun, J. Graphcare: En- hancing healthcare predictions with personalized knowl- edge graphs. arXiv preprint arXiv:2305. /2788, 2023b.
Organization, W. H. International Statistical Classification of Diseases and related health problems: Alphabetical index, volume 3. World Health Organization, 2004.

---

Multimodal Medical Code Tokenizer
Organization, W. H. al. International classification of diseases ninth revision (icd-9). Weekly Epidemiological Record= Relevé épidémiologique hebdomadaire, 63(45); 343-344, 1988.
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. Large language models encode clinical knowledge. Nature, 620(7972) 2023.
Pal, A., Umapathi, L. K., and Sankarasubbu, M. Medmeqa: large-scale multi-subject multi-choice dataset for medi- cal domain question answering. In Conference on health, inference, and learning, pp. 260. PMLR, 2022.
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, I.., Clark, K., Pfohl, S. R., Cole-Leewis, H., et al. Toward expert-level medical question answering with large language models. Nature Medicine, pp. 1-8, 2025.
Palmer, E. What is ndc? Pension reform: Issues and prospects for non-financial defined contribution (NDC) schemes, pp. 17-34, 2006.
Song, X., Salcianu, A., Song, Y., Dopson, D., and Zhou, D. Fast wordpiece tokenization. (arXiv:0012.15524) October 2021. doi: 10488500arXx.222222155444 URL http//arriv.oor/abb2022.... arXiv:2012.15524.
Perozzi, B., Fatemi, B., Zelle, D., Tsitsulin, A., Kazemi, M., Al-Rfou, R., and Halcrow, J. Let your graph do the talking: Encoding structured data for Ilms. (arXiv:2402.0586)) February 2024. doi: 10.48550/ arXiv.2402.0586. URL http://arxiv..rr//bs/440..222.. arXiv:2402.05862.
Su, X., Wang, Y., Gao, S., Liu, X., Giunchiglia, V., Clevert, D.-A., and Zitnik, M. Knowledge graph based agent for complex, knowledge-inteniive qa in medicine. arXiv preprint arXiv:2410.04660, 2024.
Poulain, R. and Beheshti, Graph transformers on chrs: Better representation improves downstream performance. In The Twelfth International Conference on Learning Representations, 2024.
Sun, W., Yan, L., Chen, Z., Wang. S., Zhu, H., Ren, P., Chen, Z., Yin, D., Rijke, M., and Ren, Z. Learning tokenize for generative retrieval. Advances in Neural Information Processing Systems, 36, 2024.
Qiao, L., Ye, P., Ren, Y., Bai, W., Liang, C., Ma, X., Dong, N., and Ouyang, W. Model decides how to- kenize: Adaptive dna sequence tokenization with mxdna. (arXiv:2412.1371)) December 2024. doi: 10.48550/ arXiv.2412.13716. URL http://axxivorr//a//2442211111. arXiv:2412.13716.
Tahmid, M. T., Shahgir, H. S., Mahbub, S., Dong, Y., and Bayzid, M. S. Birna-bert allows efficient rna language modeling with adaptive tokenization. November 2024. doi: 10. 10.12002222407 022.601703. URL https://www. biorxiv.org/conten///Oo http:////.....000000220000000000
Tu, T., Azizi, S., Driess, D., Schackermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. Towards generalist biomedical ai. NEJM AI, 1(3): Aloa2300138, 2024.
Qiu, P., Wu, C., Zhang, X., Lin, W., Wang. H., Zhang, Y., Wang, Y., and Xie, W. Towards building multilingual language model for medicine. Nature Communications, 15(1):3384, 2024.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
Qu, L.,, Zhang, H., Liu, Y., Wang, X., Jiang, Y., Gao, Y., Ye, H., Du, D. K., Yuan, Z., and Wu, X. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069 2024.
Wang, C., Gupta, S., Zhang, X., Tonekaboni, S., Jegelka, S., Jaakkola, T., and Uhler, C. An information criterion for controlled disentanglement of multimodal data. arXiv preprint arXiv:2410.2 23999, 2024a.
Rajput, S., Mehta, N., Singh, A., Hulikal Keshavan, R., Vu, T., Heldt, L., Hong, L., Tay, Y., Tran, V., Samost, J., et al. Recommender systems with generative retrieval. Advances in Neural Information Processing Systems, 36: 10299-10315, 2023.
Wang, D., Li, Y., Jiang, J., Ding, Z., Jiang, G., Liang, J., and Yang. D. Tokenization matters! degrading large language models through challenging their tokeniza- tion. (arXiv:2405. 17067), May 2024b. doi: 10.48550/ arXiv.2405. 17067. URL http://ariiv.ooyyaa//205...66 ar arXiv:2405.17067.
Renc, P., Jia, Y., Samir, E., Was, J., Li, Q., Bates, D. W. and Sitek, A. Zero shot health trajectory prediction using transformer. NPJ Digital Medicine, 7(1):256, 2024.
Sennrich, R., Haddow, B., and Birch, A. Neural machine translation rare words with subword units. June 2016. doi: 10.48550//rXiv.15500 001111008079000
Wang. L., Hassani, K., Zhang, S., Fu, D., Yuan, B., Cong, W., Hua, Z., Wu, H., Yao, N., and Long, B. Learn- ing graph quantized tokenizers for transformers. arXiv preprint arXiv:2410. 13798, 2024c.
1ww.......000099,,,,, URL http://arxiv.org/
12

---

Multimodal Medical Code Tokenizer
Wang, W., Bao, H., Lin, X., Zhang, J., Li, Y., Feng, F., Ng. S.-K., and Chua, T.-S. Learnable item tokenization for generative recommendation. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 2400-2409, 2024d.
Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, Ku, A.,
Xu, Y., Baldridge, J., and Wu, Y. Vector-quantized image modeling with improved vqgan. 1..Xiv:2110004221 June 2022. doi: 10448500/rXxv.21100.44666
URL http:
/arxiv.org/abs/21
10.p:///a//111000.22222 10.04627. arXiv:2110.046))
Wishart, D. S., Feunang, Y. D., Guo, A. C., Lo, E. J., Marcu, A., Grant, R., Sajed, T., Johnson, D., Li, C., Sayeeda, Z., et al. Drugbank 5.0: a major update to the drug- bank database for 2018. Nucleic Acids Research, 46(D1); doi: 10. 10.1093/nar/gkx1003
Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y.. Birodkar, V., Gupta, A.. Gu, X., et al. Language model beats diffusion- visual generation. arXiv preprint arXiv:2310.05737, 2023.
key
#1074-D1082 2018.
Yu, L.,, Lezama, J., Gundavarapu, N. B., Versari, L.., Sohn, K., Minnen, D., Cheng, Y., Birodkar, V., Gupta, A., Gu, X., Hauptmann, A. G., Gong, B., Yang, M.-H., Essa, I., Ross, D. A., and Jiang, L. Language model beats diffusion - tokenizer is key to visual generation. (arXiv:2310.05737) March 2024a. doi: 10.48550/ arXiv.2310.05737. URL http://ariv.oo//as2231000077 arXiv:2310.05737.
Wornow, M., Thapa, R., Steinberg, E., Fries, J. A., and Shah, N. EHRSHOT: An EHR benchmark for few-shot evalua- tion of foundation models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2023.
Xu, R., Shi, W., Yu, Y., Zhuang, Y., Jin, B., Wang, M. D., Ho, J., and Yang, C. RAM-EHR: Retrieval augmen- electronic health
tation meets clinical predictions records. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 754-765, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/2024. ac.-short.68.
Yu, Z., Zhang, C., Wang, Y., Tang, W., Wang, J., and Ma, L. Predict and interpret health risk using ehr through typical patients. In ICASSP 2024 2024 IEEE In- ternational Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), pp. 1506-1510, 2024b. doi: 10. N0.100//AASS44445550000 10447313.
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 30:495-507, 2021.
Yang, L., Tian, Y., Xu, M., Liu, Z., Hong, S., Qu, W., Zhang, W., Cui, B., Zhang, M., and Leskovec, J. Vggraph: Rethinking graph representation space for bridging gnns and mlps. arXiv preprint arXiv:2308.02111 2023a.
Zha, K., Yu, L., Fathi, A., Ross, D. A., Schmid, C., Katabi, D., and Gu, X. Language-guided image tokenization for generation. (arXivv2412..05796) December 2024. doi: 10.48850VarXxw.22122.07799 URL http://arxiv.or//abs/ 2412.05796. arXiv:2412.05796.
Yang, L., Tian, Y., Xu, M., Liu, Z., Hong, S., Qu, W.,
Zhang. W., Cui, B., Zhang, M., and Leskovec, J. Vegraph: Rethinking graph representation space for bridging gnns and mlps. (arXiv:2308.02) 17), March 2024a. doi: 10. 08550/arxxi.22008.01111 URL http://arkiv..rg/ahs/23.0. 02117. arXiv:2308.02111.
Zhang. J., Zhan, F., Theobalt, C., and Lu, S. Regularized vector quantization for tokenized image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18467-18476, 2023,
Yang, L., Tian, Y., Xu, M., Liu, Z., Hong, S., Qu, W., Zhang, CUI, B.. Zhang, M., and Leskovec, J. Vggraph: Re- thinking graph representation space for bridging gnns and mlps. In International Conference on Learning Represen- tations, 2024b.
Zhou, J., Wei, C., Wang. H., Shen, W., Xie, C., Yuille, A., and Kong, T. ibot: Image bert pre-training with online tokenizer. January 2022. doi: 10.48550/arXiv. 2111.07832. URL http://arxivorr/abb2211.1..
Yang, Z., Mitra, A., Liu, W., Berlowitz, D., and Yu, H. Transformehr: transformer-base encoder-decoder gen- erative model to enhance prediction of disease outcomes using electronic health records. Nature communications, 14(1):7857, 2023b.
Zhu, Y., Ren, C., Wang, Z., Zheng, X., Xie, S., Feng, J., Zhu. X., Li, Z.. Ma, L., and Pan, C. Emerge: En haneing multimodal electronic health records predictive modeling with retrieval-augne -uugeeted generation. In Pro- ceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM '24, pp. 3549-3559, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704369, doi: 10. 10.1145/622773333699388
Yu, J., Li. X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y. Vector-quantized image modeling with improved vggan. arXiv preprint a 10. (0.04627,20 2021.
13

---

Multimodal Medical Code Tokenizer
A. Data preprocessing
A.1. A multimodal text-graph dataset of medical codes
The medical codes dataset consists of medical codes, their descriptions, and associated knowledge subgraphs, encompassing eight commonly used health coding systems: ICD-9-CM (procedures and diagnoses), ICD-10-CM, ICD-10-PCS, NDC (National Drug Codes), SNOMED CT, ATC (Anatomical Therapeutic Chemical Classification), CPT (Current Procedural Terminology), and RxNorm. All code lists were obtained from official sources. Specifically, ICD-9 and ICD-10 (CM and PCS) were sourced from the CMS website; NDC codes from the U.S. Food and Drug Administration (FDA) database; and CPT (Level HCPCS) from the Physician Fee Schedule (PFS) Relative Value Files at CMS. SNOMED CT, RxNorm (active codes only), and ATC were downloaded via the National Library of Medicine (NIM),, part of the National Institutes of Health (NIH).
1. BUILDING A KNOWLEDGE GRAPH OF MEDICAL CODES
In the final dataset, each medical code is linked to a knowledge graph capturing relevant medical insights and relationships. We constructed these subgraphs in two steps: mapping each code to graph (Chandak al., 2023); and extracting node-centered subgraphs to represent the code's s associated knowledge and connections. To facilitate mapping, we leveraged several external resources, notably the UMLS database (Bodenreider, 2004) and MONDO Disease Ontology files (Balsa-Canto et al., 2023). Medical codes were first mapped to Concept Unique Identifiers (CUIs) in the UMLS database, then linked to PrimeKG nodes via a custom PrimeKG includes MONDO annotations, we also aligned medical codes to MONDO terms using the mondo. oowl file, thus achieving direct integration with PrimeKG nodes. Additionally, a custom entity linker was employed enhance coverage by translating medical codes into descriptive text (via PyHealth's 's MedCode InnerMap) and matching these descriptions PrimeKG node names. When exact matches were unavailable, we resorted to NLP-based linker (SciSpacy with UMLS) to measure semantic similarity. For drug codes, the rxnav.nlm.nn www.:llm.iih.go.. nov. API was used to map RxNorm codes to ATC identifiers, which were then associated with DrugBank entities (Wishart et al., 2018) 1) through a predefined (CC-to-DrugBane mapping.
more nodes in the PrimeKG knowledge
file. Because
A.1.2. ASSEMBLING TEXT DEFINITIONS OF MEDICAL CODES
Initially, each medical code's description was taken from its official source. For medication codes (e.g., NDC) where the original text was sparse, additional details were derived from attributes such as trade name, proprictary name, and pharmacological classification. These preliminary definitions were then refined and enriched using GPT-4 (turbo), with prompts tailored to each coding system but sharing common goal of elaborating on clinical uses (for drugs), procedural steps (for procedures), or mechanistic and clinical context (for diagnoses).
B. Implementation details
B.1. Experimental environments
MEDTOK is training on a machine equipped with 4 NVIDIA H100. All experiments were conducted with NVIDIA H100.
We implement MED'TOK using Python 3.9.19, PyTorch 2.3.1, Transformers 4.43.1, and Tokenizers 0.19.1. All LMs and LLMs adopted in this study are retrieved from Hugging Face, except for OpenAl models.
B.2. Details in MED training
MEDTTK is trained on 4 NVIDIA H100 GPUs by using the loss defined in Section 3.2. During the training stage, we set the training step as 3000 with a global batch size of 1024, the dimension of quantized vectors is 64. In terms of the models' weights, we freeze the text encoder in MEDTOK and the graph encoder is trainable during the training stage.
B.3. Implementation details of baseline models
All results presented in this study were obtained using the same machine which the MEDTOK trained.
ETHOS (Renc et al., 2024) experiments were conducted using the authors' original repository. For each experimental setting, three models were trained on the MIMIC-IV dataset with different random seeds, and their predictions were averaged during inference to ensure robustness. In the "MEDTok + ETHOS" configuration, the original vocabulary was extended to
14

---

Multinodal Medical Code Tokenizer
incorporate MED'OK's 's tokens for diagnoses, procedures, and prescriptions. The lab measurements were excluded from the analysis. Training and dataset splitting on MIMIC-IV adhered the methodology outlined in the ETHOS paper. During inference, the number of generated tokens was limited to 2048, and the timeline duration was adjusted based on the specific task: fifteen days for readmission, two weeks for mortality, and up to six months for other tasks. Each model was executed five times, and the resulting predictions were averaged to produce a continuous output, as described in the ETHOS study. Inference on the MIMIC-III dataset was performed on the entire dataset, excluding BMI, ICU stay tables, blood pressure, and lab data. For the EHRShot dataset, inference was conducted on the full dataset for mortality and disease-related tasks, and on randomly selected, stratified samples of ten thousand instances for other tasks.
As for the other baselines adopted in this work, we first downloaded their code and deploy these models on working machine. For BEHRT (Li et al., 2020) and GT-BEHRT (Poulain & Beheshti, 2024), we re-trained it in an end-to-end way and integrated the tokens for time, visit, and patient's info as that in their original work. For MulT-EHR (Chan et al., 2024), we first pre-train it on MIMIC-III, MIMIC-IV, and EHRShot, respectively, to get the embedding of medical codes, and next fine-tune it on multi-task learning. For the "MEDTOK+ experiments, we use our token embeddings to initialize the nodes or tokens the original work adopted and then train or pre-train the model. It should be noted that we adopt unified epoch number for all baselines, which is 50.
C. Task definitions and data preparation under in-patient setting
C.1. Mortality prediction
Task definition. Mortality (MT) prediction estimates the mortality label of the subsequent visit for each sample, with the last sample dropped. Formally,
J: (V1, ; + y(ve].
where y|vx] E {0,1] } is binary label indicating the patient's 's survival status recorded in visit ug.
C.2. Readmission prediction
Task definition. Readmission prediction checks if the patient will be readmitted to the hospital within days. Formally, (V1, U2,..., v2-1) + u - r(()-1)), where y E {0, 1} and T(2)) denotes the encounter time of visit v/. Specifically,
[1 if T(x,) (w1-1) S o,
v(T(0)) ((h-11]]
{ otherwise.
In our study, we set 15 days.
C.3. Length-of-Stay (LOS) prediction
Task definition. Length-of-Star (LOS) prediction follows the formulation of (Harutyunyan et al., 2019), estimating ICU stay length for each visit. Formally, f (11, , Ut) + y/uz], where y(vx] E R1xC' is a one-hot vector indicating its class among C' possible categories. We define 10 classes, {0, 1, ..., 7, 7,8,99 9}. representing the following durations: 0 for one day or less, 1-7 for within one week, 8 for one to two weeks, and 9 for at least two weeks.
C.4. Phenotype prediction
Task definition. Phenotype prediction aims to classify which acute care conditions are present in given patient record: fi (U1, U2 ... v2) > y(u.). where ylu:] e IxC is one-hot vector indicating its class among C' possible categories. This task is a multilabel classification problem with macro-averaged AUC-ROC being the main metric.
C.5. Drug recommendation
Task definition. Drug recommendation aims to recommend drugs for a patient according to the patient's visit history and diagnosis in current visit: f : (V1, 12,..., vx) + s/vx), where y/vx] E IRI is one-hot vector indicating its class among C possible categories. This task is multilabel classification problem with macro-averaged AUC-ROC being the main metric.
Data preprocessing. In this study, we adopted a data preprocessing approach similar to that used in previous research (Haru- tyunyan al., 2019), which defined 25 acute care conditions. Each diagnosis code was mapped to one of these 25 phenotype
15

---

Multimodal Medical Code Tokenizer
categories. Finally, we got 24 diagnosis codes. Since ICD-9 codes in MIMIC-III associated with hospital visits rather than specific ICU stays, linked diagnoses to ICU stays using the hospital admission identifier. To reduce ambiguity, excluded hospital admissions involving multiple ICU stays, ensuring that each diagnosis corresponded to single ICU stay admission. It's important to note that our phenotype classification was retrospective; we analyzed the complete ICU stay before predicting the presence of specific diseases.
C.6. Out-patient setting
Under this setting, we adopt two types of tasks in EHRShot: operational outcomes prediction and assignment of new diagnoses. In the field of operational outcomes, we follow the same task definitions in long length of stay prediction, which only considers if patient stay the hospital for less than days or more than days. In terms ofthe readmission task, we set the time window as 15 days, which is the same as that under in-patient setting. We also add another operational outcome task, which is mortality prediction. The definition of mortality prediction is the same that under the in in-patient setting.
16


Reviews:
[{"review_id": "JUlHXGCr89", "reviewer": "Reviewer_EVKN", "summary": "In this paper, the authors present MEDTOK, a multimodal medical code tokenizer that integrates textual descriptions of medical codes with graph-based relational contextual information. The proposal employs separate encoders to process each modality, and the resulting representations are mapped into a shared space through vector quantization, ensuring the preservation of both modality-specific and modality-shared information. The authors evaluate MEDTOK on three electronic health record (EHR) datasets using five different backbone models to assess its effectiveness.\n\n## update after rebuttal\nI appreciate the authors' responses to my concerns, particularly the newly supplemented results. Accordingly, I have updated my original assessment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Claims:**\nWe added the following discussion to related work: *\"Rather than treating medical codes in isolation, some methods incorporate additional knowledge to enhance their representation using structures like knowledge graphs (Choi et al., 2017; Burger et al., 2023\\) or ontology trees (Shang et al., 2019). These methods build relationships between medical codes, improving performance on EHR tasks.\"*\n\nMedTok differs from existing methods:\n* MedTok is a **tokenizer**, converting raw input into **discrete tokens** for transformer models. It doesn't directly learn high-dimensional embeddings for downstream tasks but maps **inputs to a fixed set of tokens**.  \n* MedTok maintains a codebook during training, unlike traditional representation learning methods that maintain a set of medical code embeddings.\n\nTo address concerns, we analyzed alternative methods and integrated MMUGL (Burger et al., 2023\\) as MedTok’s encoder, training it with a vector quantization loss. We compared MMUGL-driven MedTok with Transformer-based models on EHR tasks. GRAM (Choi et al., 2017\\) was excluded due to its dependence on EHR settings, and G-Bert (Shang et al., 2019\\) only considers medication and diagnostic codes, which do not align with our setting. Results are:\n||MIMIC III|MIMIC IV|EHRShot|\n|:----|:----|:----|:----|\n|MMUGL-MedTok|0.370|0.305|0.361|\n|MedTok|0.412 (+4.2%)|0.444 (+13.9%)|0.378 (+1.7%)|\n### **E1:**\nMedTok can be used with any transformer-based EHR predictive model. Our benchmarking included comparisons with five such models. TransformEHR is the best-performing backbone across 4 tasks, and thus it was chosen.\n### **E2:**\nMedTok uses two modalities and results show optimal performance when using both modalities (Fig. 4). We additionally conduct ablation studies, demonstrating that shared and specific information optimization enhances performance:\n||MIMIC III|MIMIC IV|EHRShot|\n|:----|:----|:----|:----|\n|VQ|0.373|0.387|0.287|\n|VQ \\+ shared|0.379|0.409|0.314|\n|VQ \\+ specific|0.382|0.402|0.366|\n|VQ \\+ shared \\+ specific|0.412|0.444|0.378|\n### **E3:**\nFollowing your advice, we examine the impact of hyperparameters on MedTok performance. To make MedTok consider shared and specific information equally, we assume λ\\= β, where λ is the weight for shared information loss and β for specific information. Results are:\n|λ \\= β|MIMIC III|MIMIC IV|EHRShot|\n|:----|:----|:----|:----|\n|0.01|0.356|0.376|0.425|\n|0.1|0.412|0.444|0.378|\n|0.2|0.344|0.388|0.381|\n|0.3|0.330|0.357|0.418|\n|0.4|0.382|0.327|0.409|\n|0.5|0.403|0.401|0.404|\n### **E4:**\nTo this end, we selected a subset of patients classified as high risk for Hyperlipidemia by MedTok \\+ TransformEHR, where these patients have no records of Hyperlipidemia before. We then counted the tokens assigned to these patients and identified those appearing more than 100 times (https://anonymous.4open.science/r/MedTok-8DEE/E4.png). We then mapped these frequent tokens to medical codes, with the most overlapping codes being Rosuvastatin 5 mg Oral Tablet (RxNorm 2669980), Burn of skin (SNOMED CT 147087003), Type 2 diabetes mellitus without complication (disorder) (SNOMED CT 373555004), and Hyperlipidemia (SNOMED CT 285605009). They are closely related to Hyperlipidemia. Rosuvastatin corresponds to medications commonly prescribed for lipid disorders. The other three medical codes represent clinical diagnoses or findings associated with hyperlipidemia-related cardiovascular risk. It suggests that MedTok effectively captures key medical concepts related to Hyperlipidemia, supporting its predictive capability.\n### **W1:**\nWe identify 24 phenotypes following (Harutyunyan et al., 2019). Please refer to Table 2 in https://doi.org/10.1038/s41597-019-0103-9 for details.\n### **W2:**\nWe limited the scope to five drugs to assess MedTok on well-defined, clinically relevant drug recommendations across diverse therapies. Each selected drug was prescribed to ~20–30% of patients, underscoring their relevance in clinical decision-making. Besides, the selection spans a broad range of categories: antibiotics (Vancomycin and Levofloxacin), anticoagulant (Heparin Sodium), beta-blocker (Metoprolol), and lipid-lowering agent (Atorvastatin). This range ensures that the evaluation covers multiple clinical scenarios, while focusing on conditions readily identifiable from patient data.\n### **W3:**\nWe follow https://arxiv.org/abs/2307.02028 in categorizing seven tasks. Operational Outcomes (OO) includes length of stay, readmission, and mortality prediction, while Assignment of New Diagnoses (ND) includes four new disease diagnoses.\n### **W4**\nSince EHRShot is a longitudinal dataset with 921,499 visits and per-visit readmission predictions, we use stratified sampling to reduce redundancy while ensuring consistency with the ETHOS training setting.\n### **Comments:**\nWe revise it to ‘EHRShot’.\n\n---\nWe appreciate your suggestions for improvement. Please reach out with any questions or for further clarification. Thank you\\!", "claims_and_evidence": "The proposed MEDTOK functions as a broadly defined \"tokenizer\" rather than a conventional one. Specifically, its tokenization process integrates both tokenization and embedding through dedicated encoders. Given this design, the extensive body of research on medical code embeddings should be discussed in the related work section and incorporated as baselines in the experimental evaluation to provide a more comprehensive comparison.\n\nFurthermore, MEDTOK primarily adapts and integrates existing techniques, including the designed encoders, vector quantization, and the modality fusion mechanism inspired by Wang et al. (2024a). As a result, the degree of novelty in the proposal is limited.\n\nAdditionally, prior studies have extensively leveraged graph-based knowledge information to enhance EHR data analytics, including [a, b, c], among others. The authors should explicitly articulate how MEDTOK differentiates itself from these works in terms of methodology, contributions, and empirical performance. A thorough comparative analysis, both conceptually and through experimental validation, would strengthen the claims of the paper.\n\n[a] Choi, Edward, et al. \"GRAM: graph-based attention model for healthcare representation learning.\" Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 2017.\n\n[b] Shang, Junyuan, et al. \"Pre-training of graph augmented transformers for medication recommendation.\" IJCAI. 2019.\n\n[c] Burger, Manuel, Gunnar Rätsch, and Rita Kuznetsova. \"Multi-modal graph learning over umls knowledge graphs.\" Machine Learning for Health (ML4H). PMLR, 2023.", "methods_and_evaluation_criteria": "The selection of datasets for evaluation, specifically MIMIC-III, MIMIC-IV, and EHRShot, is well-justified given their relevance to the problem under investigation.", "theoretical_claims": "I have reviewed the theoretical claims in detail.", "experimental_designs_or_analyses": "The evaluation should include existing methods for both medical code embedding and graph-based knowledge exploitation in EHR data as baselines to ensure a more comprehensive comparison.\n\nSeveral aspects of the experimental evaluation require further discussion or additional experiments to strengthen the findings:\n\n(i) In both the comparison between MEDTOK and the baselines, as well as in the ablation study, only the backbone TransformEHR is used for evaluation. Is TransformEHR the best-performing backbone model? The rationale behind this choice should be explicitly justified.\n\n(ii) Additional ablation studies should be conducted to further analyze the contribution of individual components within MEDTOK.\n\n(iii) The influence of critical hyperparameters, such as $\\beta$ and $\\lambda$ in the loss functions, should be investigated to assess their impact on the performance of MEDTOK.\n\n(iv) Given the focus on EHR data analytics, it would be valuable to provide interpretable findings and medical validation to demonstrate how the proposed MEDTOK can benefit healthcare practitioners in real-world applications.", "relation_to_broader_scientific_literature": "The proposed MEDTOK builds on prior research in multimodal learning for EHR analytics by leveraging the intra-modality and inter-modality relationships to improve predictive performance.\n\nMEDTOK extends these ideas by introducing a multimodal tokenization component for encoding different modalities separately and a token packing mechanism to integrate complementary information. This structured approach enhances analytic performance, contributing to the advancement of EHR-based predictive modeling.", "essential_references_not_discussed": "The paper should discuss related works on medical code embedding and graph-based knowledge exploitation in EHR data (such as [a, b, c]).\n\n[a] Choi, Edward, et al. \"GRAM: graph-based attention model for healthcare representation learning.\" Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 2017.\n\n[b] Shang, Junyuan, et al. \"Pre-training of graph augmented transformers for medication recommendation.\" IJCAI. 2019.\n\n[c] Burger, Manuel, Gunnar Rätsch, and Rita Kuznetsova. \"Multi-modal graph learning over umls knowledge graphs.\" Machine Learning for Health (ML4H). PMLR, 2023.", "other_strengths_and_weaknesses": "**Other Strengths:**\nThe experimental evaluation encompasses a comparison with two baseline methods across three EHR datasets using five different backbone models, an ablation study on modalities, a hyperparameter sensitivity analysis, and a case study on medical Q&A.\n\n**Other Weaknesses:**\nThe clarity of the paper could be improved by addressing the following points:\n\n(i) The evaluation setup does not display the 24 phenotypes used for in-patient evaluation.\n\n(ii) In the drug recommendation task for in-patient evaluation, the justification for restricting the recommendation scope to five specific drug candidates should be clearly articulated.\n\n(iii) In Figure 3, for the results on the EHRShot dataset, it is unclear which two tasks out of the seven are selected for demonstration.\n\n(iv) In Appendix B.3, the rationale for conducting inference on sampled datasets rather than the entire test set for certain tasks on the EHRShot dataset should be explicitly explained.", "other_comments_or_suggestions": "In Section 4.2, “HRShot.”", "questions_for_authors": "Please refer to my detailed comments and suggestions outlined in the questions above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p8pz7omyhz", "reviewer": "Reviewer_1aYX", "summary": "This paper introduces MEDTOK, a multimodal medical code tokenizer that combines text descriptions and relational information from medical ontologies to improve the processing of medical codes in electronic health records (EHRs). MEDTOK encodes both text and graph information into a unified token space, enhancing model performance on various clinical tasks, such as disease prediction and drug recommendation. Experimental results show MEDTOK’s efficiency across multiple datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Claims**:\n**Please refer to the results in ‘Reviewer EVKN: E2’.** The obtained results demonstrate that both shared and specific information optimization enhance performance, with the full optimization achieving the best results across all datasets. \n### **E1**:\nTo address your concerns, we added two new LLMs to the paper (Qwen2.5-7B and MMedLM), generated tokens using MedTok, and prefix-tuned them on the MedMCQA dataset. We then evaluated these fine-tuned models on three other established QA datasets. Results are:\n||MMLU|MedDDx|AfrimedQA|\n|:----|:----|:----|:----|\n|Llama3.1-8B|0.634|0.424|0.502|\n|+ MedTok|0.664|0.494|0.540|\n|Qwen2.5-7B|0.667|0.334|0.618|\n|+ MedTok|0.770|0.437|0.657|\n|MMedLM|0.564|0.308|0.491|\n|+ MedTok|0.640|0.403|0.653|\n### **W1:**\nGraph/text-shared information refers to the shared information between graph and text modalities, while graph/text-specific information refers to information specific to each modality. These features are learned by maximizing the shared information and minimizing the overlap between shared information. By considering both modality-shared and modality-specific information, tokens in MedTok can distinguish medical codes that have similar names (i.e., same text descriptions) but different relational structures in corresponding medical code taxonomies (i.e., different graph structures). For example, consider E11.9 (Type 2 Diabetes Mellitus Without Complications) and E11.12 (Type 2 Diabetes Mellitus With Chronic Kidney Disease). These codes both refer to Type 2 Diabetes but are used for patients with different disease progression. E11.9 is linked to general diabetes management and metabolic pathways, while E11.12 is associated with kidney-specific pathways and nephroprotective drugs. MedTok’s optimization strategy can use the shared and specific information between these codes, which improves modeling of these seemingly similar yet distinct codes with different clinical relevance.\n\nTo further address your concerns, we performed new experiments to quantify the impact of modality-shared and modality-specific information on MedTok’s. For that, **please refer to new results in ‘Reviewer EVKN: E2’.** \n\n### **W2:**\nWe evaluated MedTok using three EHR datasets that vary considerably in scale and scope. \n* EHRShot is a longitudinal dataset containing **41.6M observations** from **921K visits** across **6,739 patients** (1909–2023). Patients have an average timeline of **59 years**(max **88**) and **136 visits** (max **2,397**).  \n* MIMIC IV includes **185.8M observations**, **546K visits**, and **364K patients**. Patients have **1.5 years** of data on average (max **14.7**) with **2.4 visits** (max **238**). \n* MIMIC III contains **32.9M observations**, **58,976 visits**, and **46,520 patients**. The average record spans **4 months** (max **11.5 years**) with **1.3 visits** (max **42**). \n\nOur results across these three datasets demonstrate strong performance, efficiency, and robustness of MedTok across a wide variety of clinical environments (outcome and diagnostic prediction tasks), both in-patient and out-patient contexts, both acute and chronic medical conditions, and patients with varying volumes of clinical data (small vs. large EHR record). \n### **W3**:\nThat is an excellent suggestion. All five baselines we adopted are models that directly incorporate patient electronic health records into their vocabulary. We compared the performance of these baselines with and without our MedTok tokenizer. Results show that integrating MedTok improves the performance of these baseline models by 3.29%, 2.67%, and 5.01% across three datasets (w2\\) relative to using baseline models with standard tokenization methods. \n### **W4:**\nWe have added ablation studies on different module combinations and loss functions to examine the utility of each module and loss function component.  **please refer to the response to Reviewer EVKN: E2-E3.** Results show that the performance of full MedTok increases by 3.9%, 5.7%, and 9.1% across three datasets relative to the simplified version of MedTok with modality-shared and modality-specific modules turned off.\n### **W5**:\nWe appreciate your acknowledging the importance of considering both graph and text features for developing a comprehensive tokenizer of medical codes. As we illustrate in our response to W1 and as you nicely point out, multimodality is particularly relevant to model codes that are similar across both graph-text modalities as well as codes that are similar in one modality but not in the other. \n\n—-  \n*Thank you again for your thoughtful commentary. If you feel our responses are insufficient to motivate increasing your score, we would love to hear from you further about how we can better address your concerns. Thank you again\\!*", "claims_and_evidence": "The paper shows a strong understanding of the field, with convincing motivation and model design. It highlights the limitations of traditional tokenizers and presents MEDTOK as a solution that combines text and graph encoding, supported by experimental results. However, the proposed loss functions lack ablation experimental validation to prove their effectiveness.", "methods_and_evaluation_criteria": "The proposed method is novel, combining text and graph encoding to address the limitations of traditional medical code tokenizers. The datasets and downstream tasks selections are well-aligned with domain standards, ensuring the method’s relevance and applicability to real-world clinical scenarios. Tasks designs are critical for evaluating the model’s performance and its potential impact in healthcare applications", "theoretical_claims": "The theoretical underpinnings of the proposed modules are presented clearly, with well-defined mathematical formulas that explain the process in detail. The authors effectively justify the design choices, such as the use of modality-specific and cross-modality embeddings, using formal representations to support their approach.", "experimental_designs_or_analyses": "To demonstrate the effectiveness of MEDTOK, the authors have selected appropriate downstream tasks and datasets that are professionally relevant. The ablation design effectively shows the impact of different modality data and the codebook size on the model's performance. Additionally, the MedicalQA experiment adds depth to the overall experimental design. However, in the generative tasks, the choice of the LLM and the datasets used are relatively limited, authors should expand more datasets to further strengthen the findings and provide a more robust evaluation of MEDTOK’s capabilities in diverse scenarios.", "relation_to_broader_scientific_literature": "The authors clearly position their work within the context of existing literature, highlighting the challenges faced by traditional methods in tokenizing medical codes in the patient electronic health records (EHRs) context.", "essential_references_not_discussed": "The research covers most of the essential related works, especially in the context of tokenization for medical codes in EHRs. It discusses relevant studies on traditional tokenization methods, transformer-based models, and multimodal approaches, providing a solid foundation for its contributions.", "other_strengths_and_weaknesses": "Strengths:\n1.\tThe authors address a practical problem in healthcare, specifically the tokenization of medical codes in EHRs, which has significant real-world relevance.\n2.\tThe proposed MEDTOK shows improvements across various downstream tasks and datasets, demonstrating its potential in real-world applications.\n\nWeaknesses:\n1.\tGraph-specific, Graph-shared, Text-shared, and Text-specific mentioned in the method lack sufficient explanation, particularly regarding why these features are chosen and how they positively impact the results. This section left me confused. I want the authors include a visualization of these embedding methods and provide a clearer explanation to help readers understand how these different embedding features improve the model's performance. Additionally, authors should supplement the experiments by analyzing the specific impact of different embedding methods (such as Graph-specific and Text-specific) on task results and provide both quantitative and qualitative analyses.\n2.\tWhile the paper performs well on the evaluated datasets, authors should explore how MEDTOK scales with even larger or more diverse EHR datasets.\n3.\tAlthough the paper compares MEDTOK to standard tokenization methods, it should compare the result with other baseline models, including a comparison with a model where patient electronic health records are directly added to the vocabulary. \n4.\tThe paper mentions an ablation study evaluating the impact of removing the text or graph modality on model performance, but the experimental details and analysis are brief. It lacks a more detailed evaluation of different module combinations and loss functions.\n\n5.\tEven though transforming multi-modal information into shared features and unique features has already been applied in many models (e.g., [1] and [2]), the use of graph features and text features from biomedical knowledge graphs in this paper is sufficiently novel, with strong motivation that addresses a critical pain point in medical multimodality.\n[1] Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling \n[2] DeCUR: DECOUPLING COMMON & UNIQUE REPRESENTATIONS FOR MULTIMODAL SELF-SUPERVISION", "other_comments_or_suggestions": "n/a", "questions_for_authors": "Please see the Weakness session", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "n/a"}}, {"review_id": "a1NydXLike", "reviewer": "Reviewer_P3y8", "summary": "Medical codes in electronic health records (EHRs) contain rich textual descriptions and structured relationships. However, existing tokenization methods treat them as isolated textual tokens, failing to capture their ontological and relational context. The authors propose MEDTOK, a multimodal medical code tokenizer that creates a codebook to select the top k modality-specific or cross-modality embedding tokens, integrating both text-based and graph-based representations of medical codes. They evaluate MEDTOK across five clinical tasks on MIMIC-III and MIMIC-IV, as well as two additional tasks on EHRShot, and conduct an ablation study to demonstrate the effectiveness of the proposed multimodal approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Theoretical Claims:**\nThank you for your comment. Equations 8-9 contribute to the overall optimization process by representing the optimal values of graph embeddings (Eq. 8\\) and text embeddings (Eq. 9), which are designed to model modality-specific and modality-shared information. To approximate these optimal values, we use a method inspired by Wang et al., 2024 ([https://openreview.net/forum?id=r7Xnetd0Pq\\#discussion](https://openreview.net/forum?id=r7Xnetd0Pq#discussion)), where the optimization is achieved through InfoNCE loss and orthogonal loss.\n\nThe contribution of modality-specific and modality-shared information is shown in the response to **Reviewer EVKN: E2-E3.** By optimizing both shared and specific information across two modalities, the performances are improved by 3.9%, 5.7%, and 9.1% on three datasets, respectively. \n\n### **E1:**\n**Please refer to the responses to Reviewer EVKN: E2-E3.** By optimizing both shared and specific information across two modalities, the performances are improved by 3.9%, 5.7%, and 9.1% on three datasets, respectively. \n\n### **E2:**\nOne possible explanation for the codebook usage is as follows: at 18,000 tokens, the usage was around 32%, compared to \\~40% when the size was 6,000. As the codebook expands, the model may struggle to allocate representation capacity effectively, leading to less efficient learning and potential overfitting to specific token patterns. Such observations could also be observed by other tokenizer work (https://arxiv.org/abs/2308.02117, https://arxiv.org/html/2406.11837v1), which makes this a good open problem and future direction for the field.\n\n### **References:**\nWe add the following two references to strengthen our argument and make our explanation more convincing.\n\n*\\[1\\] Shang, Junyuan, et al. \"Pre-training of graph augmented transformers for medication recommendation.\" IJCAI. 2019\\.*  \n*\\[2\\] Xia L, Kao B, Huang C. Opengraph: Towards open graph foundation models\\[J\\]. arXiv preprint arXiv:2403.01121, 2024\\.*\n### **Q1:**\nFor modality-specific embeddings in the graph, we first use a graph encoder to learn the graph embedding, followed by a projector to learn the graph-specific embedding. The graph-specific embedding is optimized separately using a loss function and is not combined with the medical textual descriptions. Please refer to the 'specific embedding' function (Line 188-218) in [https://anonymous.4open.science/r/MedTok-8DEE/vector\\_quantization\\_soft\\_one\\_new.py](https://anonymous.4open.science/r/MedTok-8DEE/vector_quantization_soft_one_new.py). \n\n### **Q2:**\nExample 1: Heart Failure (I50.9)  \nThe text description for this code provides information about heart failure, its core diagnostic criteria, and primary clinical features (for example, impaired ventricular function). However, it does not include information on risk factors, complications, and treatment options–critical information that is not included in the text description but exists in the knowledge graph. Complementing this text definition with relational, graph-based information reveals related conditions such as hypertension, coronary artery disease, and diabetes; suggests treatments like beta-blockers, ACE inhibitors, and diuretics; and highlights potential complications, including pulmonary edema and kidney dysfunction.\n\nExample 2: Type 2 Diabetes Mellitus, Without Complications (E11.9)  \nThe text description captures a type 2 diabetes diagnosis along with lab values (fasting blood glucose and an HbA1c) and notes that the patient is prescribed metformin. However, it does not provide information on long-term risks or alternative treatment options. In contrast, the knowledge graph enriches this snapshot by linking diabetes to related conditions such as obesity, hypertension, and dyslipidemia; by highlighting potential complications like diabetic retinopathy, neuropathy, and chronic kidney disease; and by suggesting additional treatments such as insulin therapy or GLP-1 receptor agonists.\n\nIn both examples, the text modality provides an immediate clinical snapshot, while the knowledge graph adds a broader context that supports comprehensive and personalized care decisions.\n\n### **Q3:**\nThe output of MedTok includes both tokens and their corresponding embeddings. Specifically, the token refers to the indices in the codebook, and the embedding is the corresponding vector for that index. When integrating with other models, we first obtain the tokens from MedTok and then use these tokens to query the codebook and retrieve the corresponding embeddings, which are used as input to the other models. \n\n---\n\n*Thank you for your helpful feedback. If you feel our responses are insufficient to motivate increasing your score, we would love to hear from you further about how we can better address your questions. Thank you again\\!*", "claims_and_evidence": "- The claims presented in the paper are supported by experimental results and theoretical proofs.\n - Please also check the __Theoretical Claims:__", "methods_and_evaluation_criteria": "Yes. The authors use two simple linear projection layers for modality-specific embeddings and a cross-attention module for cross-modality embeddings, both of which are common and appropriate choices in multimodal tasks.", "theoretical_claims": "- Yes, I have carefully checked all equations presented in the \"Approach\" section.\n   - The Equations (8) and (9) define modality-specific quantized vectors $\\hat{e}_t^s, \\hat{e}_g^s$, but they do not appear to be explicitly used in any other equation. Could you please clarify how they contribute to the overall optimization process?\n   - In the token packing section, why do the authors use \n$\\mathcal{L}_{\\text{InfoNCE}} (\\hat{e}_t^c, \\hat{e}_t^c) + \\lambda \\mathcal{L}_{\\text{orthogonal}}(\\hat{e}_t^c, e_t^c) + \\mathcal{L}_{\\text{InfoNCE}}(\\hat{e}_g^c, \\hat{e}_g^c) + \\lambda \\mathcal{L}_{\\text{orthogonal}}(\\hat{e}_g^c, e_g^c)$ \nto represent $\\mathcal{L}_{\\text{token}}^{s}$ instead of directly using $\\hat{e}_t^s, e_t^s$? Especially considering that the loss function already includes $\\mathcal{L}_{\\text{token}}^{c}$. While the definition of $\\mathcal{L}_{\\text{token}}^{c}$ is clear based on Wang et al.'s previous work, the reasoning for the specific formulation of $\\mathcal{L}_{\\text{token}}^{s}$ is less evident. Could the authors elaborate on the motivation behind choosing this particular formulation?", "experimental_designs_or_analyses": "- I have reviewed the experimental designs carefully. The authors thoroughly evaluate their MEDTOK approach using five tasks across two inpatient datasets and two additional tasks on the EHRShot dataset. They also comprehensively compare their proposed tokenizer with alternative tokenizers, such as the VQGraph tokenizer and a standard BERT-based tokenizer.\n- However, I have some concerns regarding certain experimental analyses:\n - - In the ablation experiments, MEDTOK demonstrates strong performance. However, simply removing the text-specific embeddings $e_t^s$ or graph-specific embeddings $e_g^s$ might be insufficient to fully assess the impact of each modality, as the model still retains shared embeddings $e_t^c$ and $e_g^c$. Have you tested an ablation setting where only $e^c$ embeddings** are used (i.e., removing both $e_t^s$ and $e_g^s$)? This would provide further insight into how much information is encoded in the cross-modality tokens alone.\n - - Additionally, in the codebook size analysis, there is an unexpected performance drop on both MIMIC datasets when the codebook size is around 18,000. Could the authors provide an explanation or hypothesis for this anomaly? Specifically, this sudden drop is quite substantial, with the AUPRC dropping to nearly one-third compared to when the codebook size is around 6,000. Clarifying this would strengthen the analysis.", "relation_to_broader_scientific_literature": "- The proposed approach extends traditional vector quantization methods by explicitly dividing the codebook into modality-specific and shared regions, which helps preserve modality distinctions while improving cross-modal interactions.", "essential_references_not_discussed": "Are there related works that are essential to understanding the (context for) key contributions of the paper, but are not currently cited/discussed in the paper? Be specific in terms of prior related findings/results/ideas/etc. (Example: \"The key contribution is a linear-time algorithm, and only cites a quadratic-time algorithm for the same problem as prior work, but there was also an O(n log n) time algorithm for this problem discovered last year, namely Algorithm 3 from [ABC'24] published in ICML 2024.\")\n\n   - Line 112 left column \"Directly using the tokenizers for languages risks flattening the relationships among codes and failing to preserve the biomedical information\", please add reference for this sentence.\n   - Line 122 left column \"However, graph tokenizers may suffer from information loss when applied to graphs in other domains, please add reference", "other_strengths_and_weaknesses": "__Strengths:__\n - The paper is clearly written, making it easy to follow and understand.\n - The authors validate their approach across seven tasks using three datasets. The experimental results outperform all baseline methods. The work is solid.\n    \n__Weakness:__ please check the __Experimental Designs Or Analyses__ and __Theoretical Claims__.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Could the authors clarify how the modality-specific embeddings for knowledge graph $e_g^s$ are constructed? While the paper briefly mentions using a linear projector, upon reviewing the provided code, it appears that the graph data might be converted directly into text and then embedded together with other medical textual descriptions.\n\n- Given that this is a multimodal task, could the authors provide a case study or example illustrating situations where one modality compensates for limitations in another?\n   \n- Could the authors provide more specific details about how their proposed tokenizer is applied in downstream tasks? While the appendix provides a detailed description of selecting the label $y$, I am interested in how the tokenizer integrates with the inputs in these experiments. \n\n- other questions pleas check the __Experimental Designs Or Analyses__ and __Theoretical Claims__", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xto2geHdOV", "reviewer": "Reviewer_Eig7", "summary": "MEDTOK is a tokenizer designed specifically for medical codes, improving upon traditional approaches (that treat each code as isolated textual units) by considering the textual description of each medical code and its ontological hierarchy and relationships across different medical coding standards. It employs a language model encoder to process textual descriptions and a graph encoder to represent relational structures among medical codes. Both modalities—textual and relational—are combined into a unified token space suitable for input into various medical models.\n\nIntegrating MEDTOK into state-of-the-art EHR models shows substantial improvements across diverse medical tasks, including outcome prediction, diagnosis classification, drug recommendation, and risk stratification.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **W1:**\nWe apologize for the misunderstanding. In the final version, we will clarify that our graph construction approach is relatively simple and straightforward. Specifically, for each medical code, we define the code’s graph as a subgraph of a knowledge graph centered on the node representing the code plus its 2-hop neighborhood. Therefore, graph construction is straightforward and computationally efficient. \n\n### **W2:**\nThank you for your insightful feedback. The figure (https://anonymous.4open.science/r/MedTok-8DEE/drug_embedding.png) illustrates different drug codes (NDC, ATC, RxNorm) for the five selected drugs, demonstrating how MedTok effectively captures drug semantics. We observe that different codes representing the same drug cluster together, indicating that MedTok learns meaningful tokens of drugs across various coding systems.\n\nAdditionally, NDC and RxNorm are highly granular, meaning a single drug can have multiple codes reflecting different dosages or formulations. Despite this granularity, MedTok successfully groups related tokens, preserving their underlying medical meaning. This highlights MedTok’s ability to generalize across different drug coding schemes.\n\n### **Comments:**\nThank you for your careful review. We will change it to ‘EHRShot’ in the next version.\n\n### **Q1:**\nThank you for your comment. We ensure the accuracy of the knowledge by extracting subgraphs from a well-established biomedical knowledge graph (e.g., PrimeKG). We include nodes within two hops of the medical code to build the subgraph. However, since PrimeKG is dense, we use the PageRank algorithm to select the top 2,000 most influential nodes connected to the central medical code. Based on these processes, we could make sure the extracted subgraph is of high quality. \n\n### **Q2:**\nThank you for your comment. Appendix 1.1 explains how we extract the subgraph centered on the medical code from PrimeKG, including using CUI code to map medical code and entities in KGs. It is about extracting knowledge from an existing KG, not constructing a new KG.\n\n### **Q3:**\nYes\\! We will release our tokenizer and all source data, including code description and corresponding subgraph.\n\n### **Q4:**\nThank you for your valuable suggestion. **Please refer to the responses to Reviewer EVKN: E2-E3 .**\n\n---\n*We sincerely thank the reviewer for the thoughtful and encouraging feedback on the originality, quality, clarity, and significance of MedTok. We appreciate your positive evaluation of our work and the opportunity to address your comments. In response, we have provided detailed explanations to address your questions. Please do not hesitate to let us know if additional clarifications would be helpful. Thank you again for your valuable input.*", "claims_and_evidence": "The paper clearly identifies shortcomings in standard EHR tokenizers and effectively motivates the need for MedTok. The claims about improved performance by incorporating domain knowledge through code descriptions and relational structures are convincingly supported by empirical results. The experiments showing improvements simply by replacing standard tokenizers with MedTok are particularly compelling.", "methods_and_evaluation_criteria": "The methods proposed, including leveraging both language models and graph encoding to integrate textual and relational data, are well justified for the medical domain. The evaluation criteria using standard benchmarks like MIMIC-III, MIMIC-IV, and EHRShot are appropriate and relevant.", "theoretical_claims": "This paper does not explicitly present theoretical proofs.", "experimental_designs_or_analyses": "The experimental analyses appear sound and clearly demonstrate improvements over state-of-the-art models. The provided ablation studies are valuable, though an additional ablation study excluding cross-modality embeddings would be beneficial to fully understand their contribution.", "relation_to_broader_scientific_literature": "The paper’s contributions clearly extend existing literature on EHR tokenizers by introducing multimodal embeddings leveraging both textual descriptions and relational contexts. This approach offers a meaningful advancement compared to previous studies focusing primarily on isolated textual tokens.", "essential_references_not_discussed": "The paper seems to cover well relevant related works.", "other_strengths_and_weaknesses": "Strengths:\n\n* Clearly motivated and well-presented rationale for MedTok.\n\n* Straightforward integration with existing state-of-the-art models.\n\n* Robust empirical support showing improvements across various medical prediction tasks.\n\nWeaknesses:\n\n* Graph construction methodology seems complex and potentially difficult to reproduce.\n\n* It would enhance the paper to include visualizations of the learned embeddings to better illustrate how MedTok captures relational and textual information.", "other_comments_or_suggestions": "Typo:\n\nLine 303-right column: \"HRShot\" -> \"EHRShot\"", "questions_for_authors": "1. How did you assess the quality of the final constructed knowledge graph? Were quantitative or qualitative evaluations performed?\n\n2. Did you conduct ablation studies specifically targeting each step detailed in appendix A.1.1 regarding graph construction? Are all these steps critical?\n\n3. Is the code for graph construction and MedTok tokenizer publicly available or planned to be released?\n\n4. Have you done an ablation study excluding the cross-modality embeddings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a multimodal medical code tokenizer that uses the text descriptions and relational context of codes to tackle patient electronic health records. The motivation is meaningful, and the method is well-estimated. During the review and rebuttal periods, all reviewers' concerns are solved. Therefore, I recommend the acceptance.
Decision: Accept (poster)